A method and system for describing consumer electronics using separate task and device descriptions

A method and system for describing consumer electronics using separate task and device descriptions

1A is a block diagram showing an example of the structure of an embodiment of a task organization system according to the present invention.

1B is a flow diagram illustrating an example of a task organization process according to the system of FIG. 1A ;

2 is a block diagram showing an example of the structure of another embodiment of the task organization system including the task organization module according to the present invention.

3 is a diagram illustrating an example of a task proposal using a pseudo-language in the system of FIG. 2 .

FIG. 4 is a diagram illustrating obtaining device function descriptions from a device to generate a job proposal in the system of FIG. 2 ;

5 is a diagram illustrating an example of logically separating operation manuals and device manuals according to the present invention.

6 is a flow diagram illustrating an example of steps for generating and displaying a work proposal.

FIG. 7 is a flow diagram illustrating an example of steps for displaying a task suggestion for a user to select and performing by the task organizing module in the system of FIG. 2 ;

<backgroundart><p>The present invention relates to the provision of a user interface for user control of a device, and more particularly to presenting and displaying user actions for a group of consumer electronic (CE) devices.</p><p>Typical user interfaces for user control of consumer electronics devices such as TVs, VCRs, DVD players, CD players, etc. do not display user actions on the device. Instead, it presents the user with a set of device functions and allows the user to select a combination of these device functions to perform a task. For example, to view a videotape, the user must select an input (tape), rewind the tape, and press the play button on the VCR. As a result, the user cannot automate such an operation by simply specifying "I want to watch a videotape". Therefore, the user cannot directly express the desired operation, but must directly control the device to perform the desired functions (eg, select an input, rewind the tape, press the play button on the VCR).</p><p>This is similar to a traditional software architecture, in which tasks are mapped to the concept of an application program. For example, if a user wants to write a letter, the user must run a word processing program with the appropriate function. However, in this process, functions well-known to users are not mapped to well-known names.</p><p>Another method is to present several options to the user in the form of a menu. Many systems allow you to select well-known actions by presenting them on menu options (eg, "Spell Check Document" or "Record Instantly on VCR"). However, even in such a system, only the functions of the device are provided to the user.</p><p>Another way is to allow users to graphically represent their work. For example, a series of devices to perform a desired task are connected using a line. The problem with this method is that it does not hide the complexity of using the device from the user. That is, in this way, the user is presented with a function graphically, the user selects it, and the user is required to specify the connection required to perform the task.</p><p>Furthermore, traditionally, consumer electronics are described using interface description methods such as language APIs. Middlewares such as UPNP, OSGi, and Jini have standardized the use of procedural interfaces of such computers. However, these interfaces do not describe the device in detail to the extent that it can be used meaningfully by other computers/devices other than in the case of calling the defined interface. Other approaches focus on describing capabilities or preferences in greater detail, such as CC/PP, where sets of attributes can be expressed for computing devices that represent capabilities and preferences. Furthermore, approaches in the realm of the semantic web are moving towards more detailed technical capabilities, such as the tasks implemented by DAML and RDF. However, these approaches only describe the core for describing entities rather than specific approaches. Using the Semantic Web and XML technologies at its core, past work such as User-Centric Appliance Aggregation in HPL has tried to describe consumer electronics using device descriptions that merely associate the task with every device. For example, a DVD 'plays DVDs' and a TV 'shows a video'. The problem with these approaches is that they do not adequately describe devices and their interactions, since many user actions have more detail than can be captured by a single device. For example, playing a DVD may require dimming the lights at the same time.</p></backgroundart><abstractproblem><p>Therefore, the technical problem to be achieved by the present invention is to describe the interaction of devices described in a flexible manual rather than as a controller program operating as a black box, and the interaction of devices that more closely match the user's recognition structure for home device usage. A user that provides a job description that allows interaction, does not allow interactions to be centralized on a single device (i.e. does not map a device directly to a job), and also allows interactive descriptions and interchange of manuals. To provide a method and system for providing a description of operations.</p></abstractproblem>

<p>In the method for expressing a user task performed by interaction of a plurality of electronic devices in a task organizing system for achieving the technical task, the function of the device is described as a device description that describes a function that the device can perform. indicating step; and generating, based on the device description, task suggestions, each representing a user task as an abstraction of one or more device descriptions, whereby the user tasks are preferably separated from the devices.</p><p>A task orchestration system for a network of interconnected devices for accomplishing the technical task obtains function descriptions from one or more devices, and generates task suggestions based on the obtained device function descriptions to each Allow task suggestions to represent a user task as an abstraction of one or more acquired device descriptions, wherein a device function is expressed as a device description that represents a function that the device can perform, thereby decoupling user actions from devices. a task-driven controller; and a user interface device for presenting task suggestions on a display from which the user can select from, such that the user task indicated by the selected task suggestion is performed by the one or more devices.</p><p>In one embodiment, the present invention provides a task orchestration system that implements a method for representing user tasks performed on one or more of a plurality of electronic devices on a network. Each device includes function descriptions that describe the functions that the device can perform. The capabilities of a device are expressed by device descriptions that describe the functions that the device can perform. Functional descriptions are obtained from one or more devices. Further task descriptions are also obtained that describe the user tasks and the functions necessary to perform those tasks. Then, task suggestions for the user are generated using the function descriptions and task descriptions thus obtained. Each task proposal is abstractly expressed as a pseudo-sentence of a user task, obtained from the task description, and may use one or more of the obtained device descriptions. Task suggestions are presented on the display, allowing the user to select from them. In this way, the task requested by the user-selected task proposal is performed in one or more devices. In this way, the task suggestions hide the functionality of the device from the user of the system.</p><p>In one embodiment, task suggestions are expressed using a pseudo-language structure composed of a typed set of terms, describing user tasks as an abstraction of task descriptions and device function descriptions. A user interface device is used to display task suggestions for the user to select, which task suggestions are selected to perform tasks by one or more devices under the control of a task organizing (automation) module.</p><p>In selecting from task suggestions, the user is allowed to select pseudo-sentence elements/terms, which task suggestions may be performed by devices placed under the control of the task organizing module of the system. The task organization module may be a logical module of the system.</p><p>In one example, the user task description includes a task external description, task properties, task functionalities, and task actions. The task external description outlines task suggestions (ie pseudo-sentences) for interacting with the user. Task properties represent information about user tasks. Task functions represent the functions required to perform a user task. And, the work action describes the arrangement and combinations of devices that fulfill the required function. The device description includes a device functionality description and device grounding, wherein the device capability description provides an overview of the functions of the device, and the device basic knowledge describes the control of the device based on the function description. .</p><p>In one embodiment, as such, the task organization module comprises a task-driven controller, which obtains device function descriptions from one or more devices, obtain job descriptions describing functions, generate a job proposal from the obtained job descriptions and device descriptions, and when the job proposal is selected, a selected job based on job descriptions using the functions described in the obtained device function descriptions carry out the proposal </p><p>These features, aspects, and advantages of the present invention will be understood with reference to the following description and appended claims and appended drawings.</p><p>In one embodiment, the present invention provides a system for coordinating user tasks to be performed on a set of various devices, such as in a household appliance in which the devices are interconnected by a network. Coordination of user tasks means automating the process of selecting the device used to perform the task. In one example, it obtains device function descriptions and task descriptions and generates task suggestions based on the obtained device function descriptions and task descriptions, ranks the task suggestions, and allows the user to select among the task suggestions. allowing selection and organizing (automating) the device to perform the selected/required task suggestion. For example, as mentioned, task proposals may be described as pseudo-sentences consisting of a series of elements/terms that modify each other.</p><p>The present invention allows for the representation of user actions in a human-readable, incremental and flexible manner while using pseudo-sentences. As such, the present invention does not define how the description maps to the operation of the device, but instead focuses on how to express the operation operations in a way that is interpreted by the user. It provides an abstraction layer that is isolated from the devices in which pseudo-sentences are embedded. Tasks are described using pseudo-sentences in much the same way that handwritten scribble-recognized symbols come close to real Roman characters (eg, scribbles of user tasks compared to natural language and full cursive recognition). In one implementation, this allows a compact program to determine user actions without going through complex natural language processing.</p><p>Task organization acts as a high-level, task-centric, remote control of the device, minimizing selection and configuration by the user. As such, a combination of several techniques is used to provide an organization system suitable for automating the organization of user tasks.</p><p>1A and 1B , the task organizing system 10 according to the present invention allows user tasks to be automated as a set of operations of the organizing device with minimal input from the user. In one version, the task organization system 10 is configured as three types of logical devices. The first are client devices 20 comprising a device showing a user interface, second controller devices 30 comprising a device controlling other devices, and third control comprising a device controlled by the controllers. devices 40 that become</p><p>Each logical device has a specific function that enables task organization according to the present invention. Specifically, the controlled devices 40 include an enhanced description of their functionality using a language such as a semantic markup language or the like (which does not describe their interface, but instead describes their functionality). abstractly described). Also, in addition to a description of the device's functions, the device may include a description of one or more user actions. The job description may list only jobs that use their function, jobs that use their function and potential functions of other devices, or list jobs for other devices.</p><p>Controller devices 30 include task orchestration (TO) modules 50 , including the ability to orchestrate devices to perform user-selected task suggestions (ie task demands). In addition to the control access software, the controller 30 includes an inference system (eg, an expert system, a rule system, etc.) within the task organization module 50, which provides a set of operations and devices necessary to perform the task request. can be inferred. The controller 30 obtains task and device function descriptions and uses this information to determine (1) which tasks are valid in the system, given the current equipment, their function, and the tasks described, and (2) the requested task. Infer what combinations of devices can be used to satisfy.</p><p>Client devices 20 include the ability to request and display job suggestions to a user. As such, it includes a user interface to be connected with the task organizing module 50 of the controller 30 for presenting tasks to the user. This connection allows the user interface (eg GUI) to display only possible task suggestions given the current device 40 , tasks and functions. Furthermore, when a portion of a task proposal has been made (ie, a portion of a task request has been specified), the user interface may use the controller 30 to narrow down the applicable user options. Using this functionality, the client device 20 can present available tasks and options using pseudo-sentences. The task suggestion is selected by the user, and the request to perform the selected task suggestion is processed by the controller 30 to perform the selected/required task. This allows the controller 30 to determine, in a concise manner, what user actions are expected, match them with the job description, and eventually match device selections and settings to complete the job. User preferences, such as in which room to play the video, can be specified as modifiers as part of a task suggestion pseudo-sentence (eg, play the pseudo-sentence 'in the bedroom' or play the video 'using headphones'). do. Portions of this search space for selecting task suggestions (ie, available portions of pseudo-sentences) will be further described below by describing the appliance as separate task and device descriptions.</p><p>2 shows an example of the functional structure of the task organization system 10 according to the present invention. The task organization system 10 according to the present invention includes a user interface (GUI) module 60 and a controller 30, and the controller 30 is a task organization module ( 50). A middleware layer/module 70 between the controller 30 and the client devices 40 provides intermediate layer functions (eg, API for controlling the devices, etc.) to control the devices. Since a description of the operation of the middleware module 70 is outside the scope of the present invention, a detailed description thereof will be omitted.</p><p>Via the user interface 60 , the task organizing module 50 presents and displays task suggestions that can be performed by the group of consumer electronics devices 40 . As such, task orchestration module 50 hides how such task suggestions are mapped to devices 40 , and presents only task suggestions to the user. A way of describing tasks as task proposals is provided in an incremental, flexible and human-readable manner. This method does not focus on how the task proposal 95 maps to the actions of the devices 40 , but instead focuses on expressing the task actions in a way that can be interpreted by the user (eg, , using the terminology of pseudo-sentences for work suggestions).</p><p>Referring now to FIG. 3 , in one example, a task proposal 95 may be described by a simplified language structure 200 representing a pseudo-sentence. For convenience of description below, the illustrated structure consists of a set of terms that modify each other, and in this example, a standardized set of 'verbs' that define an action is used as the root of the hierarchical language tree 200 . . For example, "play", "show", "view", "adjust", "watering", etc. make up a set of 'verbs', each of which is followed by "picture", "lawn", A 'subject word' such as 'movie' appears. 'Subject' modifies 'verbs' by reducing the scope of the action defined by the 'verbs'.</p><p>A 'subject' is followed by a combination of 'modifiers' such as instances that further narrow the scope of a particular action. For example, "playback", "movie" and "matrix" are cases in which "matrix" is added as a specific instance. Other 'modifiers' such as "in the cave" or "at 8 pm" can be specified to control the operation. In addition, other modifiers may specify the medium on which the movie "The Matrix" is stored. Note that this structure specifies tasks, not operation of the device. Operations of the device, such as changing the volume or channel, are not treated here as tasks, instead they become operations in the existing task (eg changing the volume while "watching" "TV"). Combinations of words in this structure 200 are populated by a standardized description of the tasks located in the device 40 rather than by a set of terms defined for a particular context (eg, at home). These devices are searched to generate a set of terms and possible modifiers for each level based on the devices currently connected and accessible on the system. This approach makes it much easier to specify user tasks, since a large, unrelated word base makes it difficult to determine the correct pseudo-sentences.</p><p>Referring now also to the example diagram shown in FIG. 4 , devices 40 are currently connected or connected to a home network 85 such as a set of terms and task descriptions at each level (ie, position in a pseudo-sentence) or Search for possible modifiers based on accessible devices. In this way, it simplifies specifying the task proposal, as otherwise the irrelevant but large word base makes it difficult to determine the correct pseudo-sentences.</p><p>An improvement on this embodiment according to the present invention is to determine only a set of possible terms or modifiers based on the current selection of other terms or modifiers. For example, if "Play" or "From CD Player" is selected, even if "Play" "Video" is a possible operation, the video cannot be played on the CD player, so the system will not fit the device operation manual. . Instead, only "music" is presented to the user if that is the only possible option based on the devices 40 present.</p><p>Another improvement on this embodiment according to the present invention is to allow the task to be only partially specified (eg "play" "music" "song name"). If multiple tasks match, the task is instantiated by determining an option for the user. This can be done by random selection, by learning based on past needs, or by a more complex approach. Doing so allows users to reduce decision making and user interface options when they know there are few options or that task orchestration module 50 can determine an appropriate choice for the user based on user input so far. The task orchestration module 50 may also determine them based on available devices and resources, thus populating possible options. For example, if there is only one music device (CD player), the modifiers for the room and the selections for the device are removed during execution as part of the underlying machine automation.</p><p>Thus, the present invention allows users to specify tasks rather than device operations, enables applications or devices to specify the functions they perform, and allows a user interface to express/perform rather than all tasks that exist. It allows only tasks to be searched, provides a standardized vocabulary for users to quickly express tasks, and allows concise programs to determine user tasks without complex natural language processing.</p><p>As mentioned, in another aspect, the present invention provides a method for describing consumer electronics devices using separate job descriptions and device function descriptions. As such, the interactions of devices are expressed using device function descriptions and abstract device interactions. In this way, device descriptions (ie device functions) and user actions (ie device usage) are logically separated into rich sets of descriptions. Using such instructions it becomes possible to describe a set of household appliances (such as those used in the home) and a set of possible ways in which they interact. Based on the complexity of the task descriptions and device descriptions, complex sequential, parallel and cooperative interactions may be expressed to orchestrate the functions by which the consumer electronics devices may perform user tasks. Thereby, the interaction of the device described by the manual in a flexible way rather than the controller program acting as a black box, the interaction of the device more conforming to the structure of the user's feeling for the use of the home appliance, and the interaction are combined into one It provides job descriptions that do not focus on the device alone (ie that the device does not map directly to the job), and descriptions of user actions as job proposals that enable the exchange of interactive descriptions and descriptions.</p><p>Referring to FIG. 5 , in one example, the manual 300 of the home appliances 40 is provided as two logical manuals, a job manual 310 indicating a user operation and a device manual 320 indicating a function of the device. do. These instructions 310 , 320 provide an understanding of which task proposals are available so that the controller 30 can perform the task proposal based on the task description 310 to which devices 40 in the home network 85 . It is related in a way that allows it to be done. This type of description 310 has many uses, but primarily allows tasks to be separated from the devices 40 of the home network 85 . And it allows tasks to be decoupled from black box controller style applications that programmatically perform tasks in a predefined and predefined order for well known and existing devices.</p><p>5 , the job description 310 includes (1) a job external description 330 , (2) a job nature 340 , (3) a job function 350 , and (4) a job action 360 . ) contains four parts. Task external description 330 provides an outline of pseudo-sentences (note that in this example task proposal 95 of FIG. 2 and task external description 330 of FIG. 5 are identical). Task properties 340 represent useful information about the task (eg, it may dim the lights arbitrarily when a DVD is playing, or allow this task to be used for a particular user). Task functions 350 represent a function that a task needs to achieve its goal. These functions are described by sharing terms used in the device description 320 and abstractly describe the functions of the device 40 . For example, a DVD player may be expressed as an "AV source". These functions are described as hierarchical structures of functions. Furthermore, if a DVD player has a form of multi-function operation, it can be described as an AV source. Or it can simply be expressed as "DVDAV source" which is a subclass of "AV source" that describes DVD players. Finally, task actions 360 describe the sequence and combinations of devices that perform the enumerated functions.</p><p>Furthermore, the device description 320 has a structure similar to the operation description and shares the function description. The device descriptions 320 may include a function 350a , where the job function 350 on the job description 310 corresponds to the function 350a on the device description 320 . Function 350a may include a function description 370 , device basics 380 , and device properties (eg, screen size, features, etc.). Function description 370 provides an outline of the functionality of device 40 . For example, a CD player is an audio sink, whereas a transcode can function as an MPEG audio sink and as an MPEG audio source. Device basic knowledge 380 describes how to control device 40 based on function description 380 . For example, it may include a URL called for an operation such as changing a volume level or establishing a connection of a device. Alternatively, it may have the basic knowledge to control a UPNP device by expressing the UPNP interactions necessary to control the device. As will be appreciated by those skilled in the art, other device control protocols may also be utilized.</p><p>At runtime, task descriptions 310 and device descriptions 320 are read and analyzed by task orchestration module 50 to determine whether a set of devices 40 matching task functions 350 exists. do. If present, the task organizing module 50 determines the device function 350 matching the task as a candidate device for the task. The task orchestration module 50 then iteratively searches the available devices 40 to determine the combination of devices 40 that can achieve the task function 350 . Several sets of matching task/device combinations in this search are given as answers and presented to the user as output solutions 75 on GUI 60 . The user selects his preferred pseudo-sentence, which maps to a task/device combination and uses a task action that in turn requires device actions that use the device basic knowledge 380 on the device description 320 to accomplish the task. Instantiate the task by</p><p>Since the device descriptions 320 are only logically separated from the job description 310 on the manual 300, there are no restrictions on the device 40 participating in a larger task and describing its function as a task description. There is no such thing, and the instructions may be physically coupled. An alternative implementation is to create device descriptions 320 and operation descriptions 310 using semantic web technology, so that a shared semantic ontology (language) has the semantics of all devices and operations in a machine-readable form. It allows for a rich description of aspects. Although the above description has been described in terms of consumer electronics, those skilled in the art will recognize that the present invention may be used in mobile devices, desktop computing, enterprise computing, and other applications.</p><p>As such, the module 50 combines the job descriptions with the device descriptions to determine if the device can be used for a particular job. To perform a user task, the module 50 obtains information on a combination of the task and device descriptions to combine the basic knowledge of the device, where the task description is a set of action scripts that the module 50 uses to orchestrate device actions. includes permutations. Further, the module 50 combines the basic knowledge information from the plurality of device descriptions with the job description script to generate a specific script for controlling the devices for the user job. The device basic knowledge information is specified based on the function from the device description. Further, device-based direct information specifies interactions and parameter settings for determining control of the device. The device basic knowledge can be specified in the form of a Uniform Resource Identifier (URI) that describes the action and parameters to be used. Furthermore, the device basic knowledge information may include a plurality of basic knowledge information sections that are combined to achieve a required function. The device description may also include a plurality of grounding information sections that map other protocols used by the controller or device software.</p><p>An example of an implementation of a task organization system 10 that enhances the functionality and intelligence of the underlying devices 40 using the above methods according to an embodiment of the present invention will now be described. As shown in FIG. 2 , The task organization system 10 implements a technique for a user to request a task proposal from the system 10 and a technique for interpreting and processing the task proposal. In this embodiment, several Semantic Web-based tools and techniques are applied to create an efficient system, including evolving ontologies for objects (eg, devices) in a semantically rich markup language. Ontologies are structured to display consumer electronic devices 40 and user action suggestions 95, which will be described in more detail below.</p><p>The task organizing module 50 provides task suggestions to the user and accepts requests from the user to organize the performance of the user-selected task suggestions. The task orchestration module 50 processes this by parsing the selected task proposal 95 using the inference engine (65 in Fig. 1b) as the top layer in combination with the user interface (60 in Fig. 2), and performs the selected task proposal It finds suitable devices 40 capable of doing so, and passes this information to lower layers (eg, module 70) for carrying out. In one example, a suitable markup language is used in conjunction with the inference engine 65 to reason about the content expressed by that language, whereby each entity communicates and understands each other. In one version, semantic techniques are used to describe device functions and operations to facilitate functional-centric coupling of device functions and devices.</p><p>For example, the following types of devices 40 are used on the home network 40 .</p><p>TV - show audio/video</p><p>DVD - Play audio/video, show pictures</p><p>PC - Show Audio/Video, Show Photos, Play Games, AV Source</p><p>Receiver - connect source to sink, play audio, play radio</p><p>Refrigerator - web surfing, audio/video play, control</p><p>light - control</p><p>Phone - Audio Play, User Call, User Database</p><p>PVR - Record audio/video, stop, play, save picture, show picture</p><p>Camera - Record audio/video, play audio/play, show audio/video streaming, etc.</p><p>Examples of possible operations on the home network 85 include the following.</p><p>"watering" "garden"</p><p>"Playback" "Movie" "The Matrix" "With Surround Sound"</p><p>"Show" "Photos" "Brazil"</p><p>"Synchronize" "MP3 Player"</p><p>"Dark" "Light"</p><p>"Play" "My Music" "Living Room"</p><p>Examples (scenarios) such as the above use pseudo-sentence language to, for example, express a task proposal, whereas a semantic language may be used, for example, to describe devices and functions, reasoning about knowledge and demanding tasks. An inference engine may be used to perform In one example, the language chosen to express the ontology/rules provides a semantic level of conformance, a high degree of flexibility and expressiveness, support for types and subsumption relationships, support for data types and expression of constraints, and the like. The knowledge base 105 is included in the task organization module ( 50 in FIG. 2 ).</p><p>In another example, semantic web technologies may be utilized in task organization module 50 . The Semantic Web is an extension of the current Web so that the meaning of information is well defined, and it allows computers and people to work better together. A markup language designed for the Semantic Web provides a common understanding between various entities. Examples of Semantic Web technologies include DAML+OIL, DAML-S, and OWL, which were developed for the Semantic Web. In one example of the task organization module 50 , the device and task ontology 110 are structured using DAML+OIL and loaded into the knowledge base 105 . Various properties and functions of the devices 40 are identified and an ontology that captures the relationship between those properties is structured. Furthermore, device instances 115 in DAML+OIL are structured based on an ontology, where an instance is essentially a semantic description of the present device in terms of its interfaces and functions. As shown in the task organization flow of FIG. 1B , the task proposal selected by the user is divided into one or more tasks as required and mapped to the functions of tasks that can be performed by available devices. This mapping is made possible by inferring the information expressed on the ontology and applying additional rules to the status/preference of the device/user.</p><p>Referring now to the example of FIGS. 2 and 4 , task orchestration module 50 is located within controller 30 that interacts with a module that provides an interface to device orchestration module 50 so that device descriptions can be obtained whenever necessary. exist. In one embodiment, the DAML+OIL job instances 120 provided by existing devices are asserted in the knowledge base 105, and a predefined user written in Jess Rule and/or DAML+OIL. Preferences 130 are also loaded into knowledge base 105 . Further, device preferences for the requested task and attribute preferences for the functionality required for the task are consolidated. All possible 'verbs' (eg, "play") and 'subjects' (eg, "movie") are retrieved from the knowledge base 105 and displayed on the GUI 60 .</p><p>As such, the user task request is obtained as a set of fields (eg, 135 in FIG. 2 ) via the GUI 60 and provided as input to the knowledge base 105 of the task organization module 50 . In one example, the input obtained from the user includes all selection fields. In another example, the input obtained from the user includes (a) required fields: verb, key word and content, place, terminal device, and (b) optional fields: attributes (eg, surround sound, widescreen, etc.) do.</p><p>In this example, after the user selects a 'verb' and a 'subject word', the task organizing module 50 communicates with the protocol stack in the module 70 to obtain all possible content relevant in the context of the 'subject word'.</p><p>Upon obtaining the complete task proposal selected by the user, the task organizing module 50 parses the task proposal 95 and the corresponding task description and calculates all possible combinations of devices 40 capable of servicing a solution to the request. do. The task orchestration module 50 then applies the preference and ranks the solutions and sends it to the GUI 60 for display 75 for the user.</p><p>As mentioned, the task orchestration module 50 interacts with the user through the GUI 60 , interacts with the inference engine ( 65 in FIG. 1B ), and interacts with the rules engine ( 130 in FIG. 2 ). The task organization module 50 further comprises a knowledge base 105 that stores ontology in the form of facts and applies rules 130 to the data.</p><p>Referring now to FIG. 6, an example of an operation scenario will be described.</p><p>One. In step 400, during initialization, device and task ontologies 110 are asserted to knowledge base 105, where the content of the ontology is a fact having the format ('property value' 'subclass' 'person' 'individual'). is converted to</p><p>2. At step 405 , instances 115 , 120 of devices and tasks are also asserted in knowledge base 105 . In this example, the ontology 110 is implemented for AV devices such as a TV, a CD player, a DVD player, an IPAQ, a speaker, and a transcoder.</p><p>3. In step 410, after the ontology 110 and instances 115, 120 are loaded, 'verbs', 'subjects' related to 'verbs' and 'tasks' supported by the devices 40 ' is extracted from the facts in this knowledge base 105 .</p><p>4. At step 415, in a similar manner, the 'functions' necessary to perform the tasks identified above are also extracted from the facts in the knowledge base 105 and stored as new facts for easy retrieval.</p><p>5. In step 420, using the GUI 60, the user selects, for example, a 'verb', a 'subject word' related to the context of the 'verb', and related content (category to which the selected subject word belongs), thereby suggesting a task 95 ) to specify/select. The user may also specify a preferred location and/or the terminal device 40 on which the requested/selected task proposal is to be performed.</p><p>6. In step 425, the functions of the device required to perform the task are identified (preferably, this information should be pre-computed and stored as facts that can be easily detected so that the time spent in this step is small).</p><p>7. In step 430 , the task organizing module 50 identifies devices 40 having the necessary functions as a solution. If the task requires a variety of different functions, the task organizing module 50 collectively identifies a set of devices 40 that can perform/service the task.</p><p>8. At step 435 , after identifying the set of devices 40 capable of performing the user-selected task suggestion, the user preference 130 is applied and ranked to the set of solutions.</p><p>9. At step 440 , the user is finally presented with a set of solutions as output 75 in increasing order of their rank. An example of a ranking for a home network is that the living room in the home contains a plasma TV, PC, speaker and IPAQ. The user sets preferences such as setting the device's screen size to 40 inches or larger when playing movies, or preferring speakers over IPAQ when playing audio. For a task request such as "Play Movie Matrix in the living room", the task organizing module 50 identifies a combination (solutions) of the following devices that can perform the requested/selected task proposal: PC and speaker , PC and IPAQ, and Plasma TV. Solutions are ranked according to user preference as above, and displayed to the user as (1) plasma TV, (2) PC and speaker, and (3) PC and IPAQ.</p><p>The user selects the most preferred combination of devices 40 among the solutions, and uses task actions to instantiate the task, the task actions on the device basic knowledge (320 in Fig. 5) on the device description (320 in Fig. 5) to accomplish the task 380) in turn. As mentioned, when the user selects a set of devices 40 to perform a requested task, the task organizing module 50 calls up interfaces suitable for the devices required for the task.</p><p>With reference to each step in the flowchart of FIG. 7 , an example of operation in the embodiment of the user interface module 60 will be described next. Users can incrementally create job requests using the GUI module in four steps as described below.</p><p>One. At step 500 , valid 'verbs' are displayed on the GUI 60 . The user may select a 'verb' from the list and click 'Next' to proceed to the next step (ie, step 505). The user can change the selected 'verb' at any time before clicking 'next'.</p><p>2. In step 505 , a list of all 'subject words' related to the selected 'verb' is displayed on the GUI 60 . The user may select a 'subject word' from the list and click 'Next' to proceed to the next step (ie, step 510). The user can change the selected 'subject' at any time before clicking 'Next'. The user can also click 'back' or 'start over' to go back to the previous step (ie 500 steps) and change the selected 'verb'.</p><p>3. In step 510, a list of all 'contents' related to the selected 'subject word' is displayed. The user may select 'content' from the list and click 'next' to proceed to the next step (ie, step 515). The user can change the selected 'content' at any time before clicking 'next'. The user can also click 'back' to go back to the previous step (ie step 505) and change the selected 'subject', or click 'from scratch' to go back to step 500 and change the selected 'verb'. In any of the above processes or at the end of step 510, the user may set a desired location for performing the task and/or set the terminal device to perform the entire task or part of the task.</p><p>4. In step 515, the task organizing module 50 selects 'verb', 'subject word', 'content', user preference (place, final terminal set by GUI and other preferences specified by facts and rules, etc.) and Infers internally about facts about existing devices and tasks that you have on the knowledge base 105 . After inference, task orchestration module 50 determines all possible combinations (ie solutions) of devices capable of performing the requested task. The obtained set of solutions is ranked within the task organizing module 50 and displayed to the user through the GUI 60 (eg, in increasing order of ranking). An example of the processing in step 515 will be described next. The task organizing module 50 reads the task and device descriptions and stores these information as facts in the knowledge base 105 . The inference engine 65 interprets the task and device descriptions and derives new information that is not specified directly in the task and device descriptions, but is specified as the type of classes or properties, or relationships between them. The task organizing module 50 processes and manipulates the information.</p><p>In one example, the basic steps associated with task organization module 50 include:</p><p>One. Read operation and device manuals</p><p>2. Applying semantics to task and device manuals </p><p>3. Transform job and device descriptions loaded in knowledge base 105 using parser</p><p>4. Applying other rules or query terms as described above</p><p>As mentioned, ontologies 110 are structured as a mechanism for describing tasks, devices, and user preferences. In general, an ontology is a specification of a conceptualization. It is a description of a concept (specific to a formal program) and a relationship that may exist to an entity (eg, devices, users, etc.) or a set of entities. In this example, devices and tasks are described based on their basic attribute/properties and/or functions provided or required. The description is encoded using a markup language, on the basis of which the user's job needs are understood, processed and served.</p><p>Furthermore, 'function + content type' attributes may be included in the ontology 110 . In the implementation of the ontology 110, for each function provided by the device 40, what kind of data is accepted as input, from which sources the device 40 can accept the input, and what It is also specified whether the kind of data (if present) is to be generated as an output. This suffices as an answer to a request that is very general and does not specify the type of content. However, if the task organizing module 50 intends to provide a content type-based lookup for devices, how to express a function including the content type (eg, mp3render) as not a separate attribute this is required This can be achieved by including the 'function + content type' properties in the ontology 110 .</p><p>In addition, an ontology expressing the user preference 130 may be utilized. In an implementation, preference 130 may be expressed in markup language in a manner similar to how device instances 115 and task instances 120 are specified. To write such a description in a standardized/generic format and to better express the relationship between different attributes of a set of preferences, an ontology can be utilized based on which instances of user preferences can be structured.</p><p>A mechanism for understanding preferences may also be implemented. For example, the task organizing module 50 may understand preferences such as "select a device with a larger screen size when playing a movie". Preferably, the task organizing module 50 can understand that "plasma TV is always better than any other type of TV, and has the largest screen size". Furthermore, more general ontology can be included, which can be applied to all kinds of devices, even to devices unknown at the time of implementation.</p><p>Although the examples so far have been described with respect to home appliances and home networks, those skilled in the art will appreciate that the present invention can be applied to mobile personal devices, desktop tasks, complex enterprise systems, and the like. Further, although embodiments of the task organization system have been described in the context of home appliances in the present description, those skilled in the art will recognize that the present invention provides a simple method for user organization to which known sets of objects interact with tasks and those objects. It will be appreciated that other applications where necessary (eg, a production line for specifying the settings of various operations or assembly robots) are also useful. Furthermore, since they are logically separated from the devices, there is no reason why they cannot be physically coupled with the actual devices. Also, although only one controller/organization module is described as an embodiment, a set of devices may participate in determining a control operation or organization of devices.</p><p>The present invention has been described in considerable detail with respect to certain optimal versions. However, other versions are possible. Therefore, the spirit or scope of the appended claims should not be limited to the scope of the description of the best version contained herein.</p>

<p>According to the method and system for describing home appliances using separate task and device instructions according to the present invention, it describes the interaction of the devices described in the flexible manual rather than as a controller program operating as a black box, and the user's home device usage User actions that allow interaction of devices that more closely match the recognition structure of By providing their instructions, the user can directly select the desired operation without having to make complicated selections about the functions of the device.</p>