A method and system for describing consumer electronics using separate task and device descriptions
38 claims: 2 independent, 36 dependent
- 1制御器と、ユーザーインタフェース装置を含む作業編成システムにおいて、複数の電子装置の相互作用により行われるユーザー作業を表現する方法であって、 前記制御器が、前記電子装置が行う機能を説明する装置説明書を 装置オントロジーとして構造化 するステップと、 前記制御器が、ユーザー作業を行うために必要な機能を説明する作業説明書を 作業オントロジーとして構造化 するステップと、 前記制御器が、 前記装置オントロジー及び前記作業オントロジー に基づき、擬似文章で表現された作業提案を 使用可能な電子装置により行われうる作業にマッピングさせる ステップと、 前記ユーザーインタフェース装置のディスプレイ上に マッピング された作業提案を表示するステップと、を含むことを特徴とするユーザー作業の表現方法。
- 2前記作業説明書は、作業提案の概要を表す作業外部説明書、作業性質、作業機能及び作業動作を含むことを特徴とする請求項1に記載のユーザー作業の表現方法。
- 3前記ユーザーインタフェース装置のディスプレイ上に生成された作業提案を表示するステップは、 前記ディスプレイ上に複数の作業提案を表示するステップと、 ユーザーをして前記複数の作業提案のうち一つを選択せしめるステップと、を含むことを特徴とする請求項1に記載のユーザー作業の表現方法。
- 4前記作業性質は、ユーザー作業についての情報を表現することを特徴とする請求項2に記載のユーザー作業の表現方法。
- 5前記作業機能は、ユーザー作業を行うために要求される機能を表現することを特徴とする請求項2に記載のユーザー作業の表現方法。
- 6前記作業動作は、要求される機能を満たす装置の配列及び組合わせを記述することを特徴とする請求項2に記載のユーザー作業の表現方法。
- 7前記装置説明書は、装置機能説明書及び装置基礎知識を含むことを特徴とする請求項1に記載のユーザー作業の表現方法。
- 8前記装置機能説明書は、装置の機能についての概要を表すことを特徴とする請求項7に記載のユーザー作業の表現方法。
- 9前記装置基礎知識は、機能説明書に基づいた装置の制御を記述することを特徴とする請求項8に記載のユーザー作業の表現方法。
- 10前記作業説明書及び装置説明書を使用して、ユーザー作業を行える装置の集合を決定するステップを更に含むことを特徴とする請求項2に記載のユーザー作業の表現方法。
- 11前記作業説明書は、セマンティック言語で記載されたことを特徴とする請求項2に記載のユーザー作業の表現方法。
- 12前記作業提案を生成するステップは、 順次かつ併行的であり、共同作用できる装置の相互作用を表現するステップを含むことを特徴とする請求項1に記載のユーザー作業の表現方法。
- 13複数の電子装置の相互作用により行われるユーザー作業を表現するための作業編成システムにおいて、 前記電子装置が行う機能を説明する装置説明書を 装置オントロジーとして構造化 し、ユーザー作業を行うために必要な機能を説明する作業説明書を 作業オントロジーとして構造化 し、 前記装置オントロジー及び前記作業オントロジー に基づいて、擬似文章で表現された作業提案を 使用可能な電子装置により行われうる作業にマッピングさせる 作業駆動制御器と、 マッピング された作業提案をディスプレイ上に表示するユーザーインタフェース装置を含むことを特徴とする作業編成システム。
- 14前記作業説明書は、作業提案の概要を表す作業外部説明書、作業性質、作業機能及び作業動作を含むことを特徴とする請求項13に記載の作業編成システム。
- 15前記ユーザーインタフェース装置は、 前記ディスプレイ上に複数の作業提案を表示し、ユーザーをして前記複数の作業提案のうち一つを選択せしめることを特徴とする請求項13に記載の作業編成システム。
- 16前記作業性質は、ユーザー作業についての情報を表現することを特徴とする請求項14に記載の作業編成システム。
- 17前記作業機能は、ユーザー作業を行うために要求される機能を表現することを特徴とする請求項14に記載の作業編成システム。
- 18前記作業動作は、要求される機能を満たす装置の配列及び組合わせを記述することを特徴とする請求項14に記載の作業編成システム。
- 19前記装置説明書は、装置機能説明書及び装置基礎知識を含むことを特徴とする請求項13に記載の作業編成システム。
- 20前記機能説明書は、装置の機能についての概要を表すことを特徴とする請求項19に記載の作業編成システム。
- 21前記装置基礎知識は、前記機能説明書に基づいた装置の制御を記述することを特徴とする請求項19に記載の作業編成システム。
- 22前記作業駆動制御器は、前記作業説明書及び装置説明書を使用して、ユーザー作業を行える装置の集合を決定することを特徴とする請求項14に記載の作業編成システム。
- 23前記作業説明書は、セマンティック言語で記載されることを特徴とする請求項14に記載の作業編成システム。
- 24前記作業駆動制御器は、順次かつ併行的であり、共同作用できる装置の相互作用を表現することを特徴とする請求項13に記載の作業編成システム。
- 25前記作業提案は、ユーザーによって選択される ことを特徴とする請求項14に記載の作業編成システム。
- 26前記装置オントロジーに基づいて装置インスタンスを構造化し、前記作業オントロジーに基づいて作業インスタンスを構造化し、前記装置及び作業オントロジーとインスタンスとを使用して、ユーザーが選択した作業提案を、使用可能な装置により行われうる作業にマッピングさせることを特徴とする請求項14に記載の作業編成システム。
- 27前記作業駆動制御器は、選択されたユーザー作業提案を行う装置の組合わせを制御することを特徴とする請求項14に記載の作業編成システム。
- 28前記ユーザーインタフェース装置は、前記作業駆動制御器と結合して、使用可能な装置、作業及び装置機能が与えられた場合に可能な作業提案を表示することを特徴とする請求項14に記載の作業編成システム。
- 29前記ユーザーインタフェース装置は、前記作業駆動制御器と結合して、使用可能な作業提案及び装置説明書に基づいたオプションを表示することを特徴とする請求項14に記載の作業編成システム。
- 30作業駆動制御器から要求されるユーザー作業を決定し、ユーザー作業を行うために、ユーザー作業説明書をユーザー作業説明書に一致させるために、それぞれの作業説明書がオントロジー言語で表現され、そのオントロジー言語で、装置が作業を制御器に表現することを特徴とする請求項14に記載の作業編成システム。
- 31前記作業提案は、動詞、主題語及び修飾語を含む擬似文章で表現されることを特徴とする請求項1に記載のユーザー作業の表現方法。
- 32前記動詞は、ユーザー作業の動作を定義することを特徴とする請求項31に記載のユーザー作業の表現方法。
- 33前記主題語は、前記動詞を修飾し、前記作業提案の範囲を縮小させることを特徴とする請求項31に記載のユーザー作業の表現方法。
- 34前記修飾語は、前記主題語を修飾し、前記作業提案の範囲を更に縮小させることを特徴とする請求項31に記載のユーザー作業の表現方法。
- 35前記作業提案は、動詞、主題語及び修飾語を含む擬似文章で表現されることを特徴とする請求項13に記載の作業編成システム。
- 36前記動詞は、ユーザー作業の動作を定義することを特徴とする請求項35に記載の作業編成システム。
- 37前記主題語は、前記動詞を修飾し、前記作業提案の範囲を縮小させることを特徴とする請求項35に記載の作業編成システム。
- 38前記修飾語は、前記主題語を修飾し、前記作業提案の範囲を更に縮小させることを特徴とする請求項35に記載の作業編成システム。
Independent claims38
70 paragraphs, as filed
The present invention relates to providing a user interface for user control of a device, and more particularly to the presentation and display of user work on a group of consumer electronics (CE) devices.
The usual user interface for user control of home appliances such as TVs, VCRs, DVD players and CD players does not display user work on the device. Instead, it presents the user with a set of device functions and selects a combination of those device functions for the user to perform the task. For example, to watch a videotape, the user must select an input (tape), rewind the tape, and press the play button on the VCR. As a result, users cannot automate such operations by simply stating "I want to watch a videotape." Therefore, the user cannot directly express the desired work, and must always control the device directly to perform the desired function (for example, input selection, tape rewind, pressing the play button of the VCR).
This is similar to traditional software architectures, in which work is mapped to the concept of application programs. For example, if the user wants to write a letter, the user must run a word processor program with the appropriate functionality. However, in such a process, the functions known to the user are not mapped to the known names.
An alternative method is to present the user with multiple options in the form of a menu. Many systems offer known tasks to be presented and selected on menu options (eg, "document spell checking" or "immediate recording on a VCR"). However, even such a system provides the user with only the functionality of the device.
Another method is to allow the user to graphically represent the work. For example, a series of devices for performing a desired operation are connected by using a wire. The problem with such a scheme is that it does not hide the complexity of using the device from the user. That is, in such a method, the user is presented with a graphic function, which is selected by the user, and is requested to identify the connection required for the user to perform the work.
Moreover, traditionally, home appliances are described using interface description methods such as language APIs. Middleware such as UPNP (Universal Plug and Play), OSGi (Open Service Gateway Initiative), and Jini (Java® Intelligent Network Infrastructure) have standardized the use of procedural interfaces on such computers. However, such an interface does not describe the device in detail to the extent that it can be used significantly by other computers / devices except when calling the defined interface. Other approaches, such as CC / PP, focus on explaining abilities and preferences in more detail, but here a set of attributes, etc. represents a computing device that represents abilities and preferences. Can be done. In addition, approaches in the area of the Semantic Web are moving towards more detailed descriptive capabilities, such as the work embodied by DAML (DARPA Agent Markup Language) and RDF (Resource Description Framework). .. However, such an approach describes only the core in order to describe the individual rather than a particular approach. Using the Semantic Web and XML (Executive Markup Language) technology at its core, past work such as user-centric appliance aggregation at HPL has made home appliances simply a device description that associates the work with any device. I tried to write using. For example, a DVD'plays a DVD'and a TV'shows a video'. The problem with such approaches is that they do not adequately describe the devices and their interactions, which is much more than many user tasks can be supplemented by a single device. This is because it has the details of. For example, in order to play a DVD, it is necessary to dim the lamp.
<p> The technical challenge to be achieved by the present invention is to describe the interaction of the devices described in the elastic manual rather than the controller program that operates as a black box, and to the recognition structure for the user's home device usage. Provides work instructions that allow closer-matched device interactions, allowing interactions not to be concentrated in one device (ie, do not map devices directly to work), and also interact. It is here to provide methods and systems for providing specific instructions and user-work instructions that allow interaction of the instructions.</p>
<p> In the work organization system for achieving the technical problem, the method of expressing the user work performed by the interaction of a plurality of electronic devices includes a step of expressing the function of the device as a device manual describing the function of the device. Based on the device description, each includes a step of generating a work proposal that represents the user's work as one or more abstractions of the device's description, thereby separating the user's work from the device. Is preferable.</p><p> A work organization system for a network of interconnected devices to achieve the technical task obtains functional instructions from one or more devices and proposes work based on the acquired device functional instructions. Is generated so that each work proposal represents the user's work as an abstraction of one or more acquired device manuals, where device functions are represented as device manuals that represent the functions that the device can perform. The work proposal is then displayed on the display so that the work drive controller that separates the user work from the device and the user work displayed by the selected work suggestion is performed by one or more devices. It is preferable to provide a user interface device which is displayed on the screen and allows the user to select from the user interface device.</p><p> In one embodiment, the present invention provides a work organization system that embodies a method of expressing user work performed by any one or more of a plurality of electronic devices on a network. Each device includes a functional description that describes the functions that the device can perform. The function of a device is represented by a device description that describes the function that the device can perform. Functional instructions are obtained from one or more devices. In addition, work instructions are obtained that explain the user work and the functions required to perform such work. Then, using the function manual and the work manual obtained in this way, a work proposal to the user is generated. In each work proposal, the user work is abstractly expressed by pseudo sentences, and any one or more of the device manuals obtained from the work manuals can be used. Work suggestions are displayed on the display and are then selectable by the user. Thus, the work requested by the work proposal selected by the user is performed on one or more devices. In this way, the work proposal hides the functionality of the device from the users of the system.</p><p> In one embodiment, the work proposal is expressed using a pseudo-linguistic structure composed of a set of terms having a type, describing the user work as an abstraction of the work manual and the device function manual. User interface devices are used to display work proposals of user choice, such work proposals being performed by one or more devices under the control of a work organization (automation) module. Selected to do.</p><p> In selecting from work proposals, the user is allowed to select pseudo-sentence elements / terms, but the work proposals selected by the user can be made by a device under the control of the work organization module of the system. .. The work organization module can be a logical module of the system.</p><p> In one example, the user work manual includes a work external manual, work nature, work function and work operation. The work external manual outlines a work proposal (ie, a pseudo-sentence) for interacting with the user. Work properties represent information about user work. The work function represents the function required to perform the user work. The working operation then describes an array and combination of devices that satisfy the required function. The device description includes a device function description and basic device knowledge, the device function description outlines the function of the device, and the device basic knowledge describes the control of the device based on the function description.</p><p> In one embodiment, the work organization module itself comprises a work-driven controller, which in turn obtains device function instructions from one or more devices and is specific. Obtain a work manual that explains the function of the device required by the work, generate a work proposal from the obtained work manual and device manual, and if the work proposal is selected, the obtained device function manual Make selected work proposals based on work instructions that use the functions described in.</p>
<p> According to the method and system for describing home appliances using the separated work and device instructions according to the present invention, the device interaction described in the flexible instructions is more than the controller program that operates as a black box. Provides work instructions that allow device interactions that more closely match the user's perception of home device usage, and allow interactions not to be concentrated in one device, and also provide interactions. By providing actionable instructions and user-work instructions that allow interaction of the instructions, the user can directly select the desired task without having to make complicated choices about the functionality of the device. ..</p>
Such features, aspects, and advantages of the present invention can be understood by referring to the following description, the attached claims, and the attached drawings.
In one embodiment, the present invention provides a system for organizing user work performed on a collection of various devices such as home appliances in a home where the devices are interconnected by a network. Organizing user work means automating the process of selecting the equipment used to perform the work. In one example, it acquires device function instructions and work instructions, generates work proposals based on the acquired device function instructions and work instructions, ranks the work proposals, and allows the user to do the work. Includes organizing (automating) equipment that allows selection in proposals and makes selected / requested work proposals. For example, as mentioned, work proposals can be described in pseudo-sentences consisting of a set of elements / terms that qualify with each other.
The present invention allows user work to be expressed in an increasing and elastic manner that can be read by humans, while using pseudo-sentences. As such, the present invention does not define how the instructions map to device behavior, but instead focuses on how to represent work behavior in a user-interpreted manner. .. Provides an abstract hierarchy away from the device in which the pseudo-text is embedded. The task is described using pseudo-sentences in a manner that is generally similar to the fact that a hand-written typographical recognition symbol is closer to the actual Roman alphabet (for example, for recognition of natural language and perfect cursive). User work to be compared). In one embodiment, this allows a simple program to determine user work without having to go through complex natural language processing.
The work organization acts as a high-level, work-centric, long-range control for the device, allowing the user to minimize selections and settings. As such, a combination of diverse techniques is used to provide a knitting system suitable for automating the knitting of user work.
With reference to the structural examples shown in FIGS. 1A and 1B, the work organization system 10 according to the present invention allows the user work to be automated into a set of operations of the organization device with minimal input from the user. To do. In one version, the work organization system 10 is composed of three types of logical devices. First, it is a client device 20 provided with a device indicating a user interface, secondly, it is a controller device 30 provided with a device for controlling other devices, and thirdly, it is provided with a device controlled by a controller. It is a controlled device 40.
Each logical device has a specific function that enables the work organization according to the present invention. Specifically, the controlled device 40 contains enhanced instructions for their functionality using a language such as the Semantic Markup Language (which does not describe their interface. , Instead, abstractly describe their functionality). Also, in addition to the instructions for the function of the device, the device may include one or more instructions for user work. The operating instructions list only the tasks that use those features, the tasks that use those features and the potential features of the other device, or the tasks for the other device. it can.
The controller device 30 includes a work organization (task orchestration: TO) module 50 and includes a function of organizing the device to make a work proposal (that is, a work request) selected by the user. In addition to the control proximity software, the controller 30 includes an inference system (eg, expert system, rule system, etc.) within the work organization module 50, which is the sequence of actions required to make a work request. And the device can be inferred. The controller 30 obtains a work and device function description and uses that information to (1) give the current device, their functions and the described work, and any work is valid in the system. And (2) infer what device combinations can be used to meet the required work.
The client device 20 includes a function capable of requesting and displaying a work proposal from the user. As such, it includes a user interface that is coupled with the work organization module 50 of the controller 30 to display the work to the user. Such a concatenation can display only work suggestions that are possible with the user interface (eg, GUI (graphical user interface)) currently given the device 40, work and functionality. In addition, if part of the work proposal is formed (ie, part of the work request is identified), the user interface can use the control 30 to reduce the applicable user options. Using such a function, the client device 20 can present available tasks and options using pseudo-text. The work proposal is selected by the user, and the request to execute the selected work proposal is processed by the controller 30 to perform the selected / requested work. This allows the controller 30 to determine what user work is expected in a simplified manner, match it with the work manual, and eventually select and configure the device to complete the work. To match. The user's preference for playing the video in any room is as part of the pseudo-sentence of the work proposal as a modifier (eg, play the video in the pseudo-sentence'bedroom'or'headphones'. Use'Play the video) to be identified. Such parts of the search space for selecting work proposals (ie, the available parts of the pseudo-text) are further described below by describing the appliances that separate the work and device instructions.
FIG. 2 is a diagram showing an example of the functional structure of the work organization system 10 according to the present invention. The work organization system 10 according to the present invention includes a user interface (GUI) module 60 and a controller 30, and the controller 30 includes a work organization module 50 as an upper layer of an internal client device 40. The middleware hierarchy / module 70 between the controller 30 and the client device 40 provides an intermediate hierarchy function for controlling the device (for example, an API for controlling the device). Since the description of the operation of the middleware module 70 is outside the scope of the present invention, the detailed description thereof will be omitted here.
Through the user interface 60, the work organization module 50 presents and displays work proposals that can be made by a group of home appliances 40. As such, the work organization module 50 hides how such work proposals are mapped to device 40 and displays only work proposals to the user. A method is provided in which the work is described as a work proposal in an increasing and elastic manner and by a method that can be read by humans. Such a method does not focus on how the work manual 95 maps to the behavior of device 40, but instead focuses on expressing the work behavior in a manner that can be interpreted by the user. Match (for example, use pseudo-sentence terms in work proposals).
Hereinafter, with reference to FIG. 3, in one example, the work proposal may be described by a simplified language structure 200 representing a pseudo-sentence. In the following, for convenience of explanation, the illustrated structure consists of a set of terms that qualify each other. In this example, the standardized set of'verbs' that define the behavior is a hierarchical language tree. Used as 200 routes. For example, "play", "display", "see", "adjustment", "watering", etc. form a set of "verbs", and each "verb" is followed by "photograph", "lawn", "movie". A'subject word'like this appears. The'subject'modifies the'verb' by reducing the range of actions defined by the'verb'.
After the'subject', there comes a combination of'modifiers', such as instances that further reduce the scope of a particular action. For example, "playback", "movie", and "matrix" are cases where "matrix" is added as a specific instance. Other'modifiers' such as "in the cave" or "8 pm" can be identified to control the work. Not only that, other modifiers can identify the medium in which the movie "The Matrix" is stored. It should be noted that this structure does not specify the work and the operation of the device. The operation of a device that changes a volume or channel is not treated here as an operation, but instead, such an operation is an operation in an existing operation (eg, in the middle of "TV" or "watching"). Change volume to). Such word combinations in structure 200 are not based on a fixed set of terms for a particular context (eg, at home), but are satisfied by standardized instructions for work located at device 40. Such devices are currently linked on the system and searched to generate a set of terms and possible modifiers for each level based on accessible devices. Such an approach can further simplify the identification of user work, because a large but unrelated word base makes it difficult to determine the correct pseudo-sentence.
In reference to the exemplary diagram shown in FIG. 4, device 40 is currently connected by a set of terms at each level (ie, position in the pseudo-sentence), and a home network 85, such as work manual 95. Searched for possible modifiers based on devices that are or accessible. Such a method makes it easier to identify work proposals, because otherwise the irrelevant but large word base makes it difficult to determine the correct pseudo-sentence.
An improvement over such an embodiment of the present invention is to determine only a set of possible terms or modifiers based on the current selection of other terms or modifiers. For example, if "Play" or "On a CD player" is selected, the video cannot be played on a CD player, even if it is a work that can be "played" or "video". It becomes incompatible with the book. Instead, if it is the only possible option based on the existing device 40, then only "music" is presented to the user.
Another improvement method for such embodiments according to the present invention is to allow the work to be specified only partially (eg, "play", "music", "song title"). If multiple tasks match, the tasks are instantiated by determining options for the user. This can be done by random selection, learning based on past demands, or by more complex approaches. By doing so, the user can make decisions and user interface options when it is found that there are a small number of options or that the work organization module 50 can make a suitable choice for the user based on previous user input. Allow to be reduced. Also, the work organization module 50 may meet possible options as it can determine them based on the equipment and resources available. For example, if there is only one music device (CD player), the modifiers for the room and the choices for the device are removed during execution in the underlying machine automation part.
Therefore, the present invention allows the user to identify the work, not the device operation, the application program or device, the function they perform, the user interface, and not any work that exists. Allows users to search only expressive / executable tasks, provides a standardized vocabulary that allows users to express tasks faster, and allows concise programs to determine user tasks without having to deal with complex natural words. ..
As mentioned, in another aspect, the invention provides a method of describing a home appliance using a separate work and device function description. As such, device interactions are expressed using device function descriptions and abstract device interactions. In this way, device instructions (ie, device functions) and user work (ie, device use) are logically separated into a rich set of instructions. Such instructions can be used to describe a set of home appliances (such as products used in the home) and a set of possible ways in which they can interact. Based on the complexity of the work instructions and device instructions, complex sequential, parallel, and collaborative interactions can be expressed to organize the functions by which the home appliance can perform user work. This allows device interactions, device interactions that are more consistent with the structure of the user's perceived use of home appliances, as described in the flexible instructions, rather than a controller program that acts as a black box. Work instructions that do not concentrate the device on only one device (ie, do not map the device directly to the work), and user work descriptions as work suggestions that allow the exchange of interactable instructions with the instructions. Provide a book.
With reference to FIG. 5, in one example, the home appliance 40 manual 300 is provided as two logical manuals: a work manual 310 that represents user work and a device manual 320 that represents the function of the device. .. Such instructions 310, 320 allow work proposals to be used to understand what device 40 can be used to make work proposals based on work instructions 310, with controller 30 at home network 85. It is more relevant depending on the method allowed. This type of manual 310 has many uses, but primarily allows work to be separated from device 40 of the home network 85. It is then allowed to be separated from the black box controller style application program where the work is predetermined for a known existing device and the work is programmatically performed in a predefined order.
As shown in FIG. 5, the work manual 310 includes four parts: (1) work external manual 330, (2) work property 340, (3) work function 350, and (4) work operation 360. The work external manual 330 provides a pseudo-sentence outline (note that in this example, the work manual 95 in FIG. 2 and the work manual 330 in FIG. 5 are the same). The work property 340 represents useful information about the work (eg, it is acceptable to optionally dim the lights when playing a DVD or to allow certain users to use such work). .. The work function 350 represents the function that the work needs to achieve its goal. Such functions are described by sharing the terms used in the device manual 320, and abstractly describe the functions of the device 40. For example, a DVD player can be described as an "AV source". Such a function is described as a hierarchical structure of functions. Furthermore, if the DVD player has a form of multifunctional operation, it can be described as an AV source. Alternatively, it can be simply expressed as a "DVD AV source" which is a subclass of the "AV source" that describes a DVD player. Finally, the work operation 360 describes the order and combination of devices that perform the arranged functions.
Further, the device manual 320 has a structure similar to the work manual and shares the function manual. The device manual 320 can include a function 350a, wherein the work function 350 on the work manual 310 corresponds to the function 350a on the device manual 320. The function 350a may include a function manual 370, device basic knowledge 380 and device attributes (eg, screen size, features, etc.). Functional description 370 provides an outline of the functionality of device 40. For example, a CD player is an audio sync, whereas a transcode can have functions as an MPEG audio sync and an MPEG audio source. The device basic knowledge 380 describes a method of controlling the device 40 based on the function manual 380. For example, it may contain a URL (uniform resource locator) that is called for actions that change the volume level or set device concatenation. Alternatively, it may have the basic knowledge to control an UPNP device by expressing the UPNP (Universal Plug and Play) interaction required to control the device. Other device control protocols may be utilized so that those skilled in the art can recognize them.
At run time, the work instructions 310 and the equipment instructions 320 are read and analyzed by the work organization module 50 to determine if there is a set of equipment 40 that matches the work function 350. If present, the work organization module 50 defines the device function 350 that matches the work as a candidate device for the work. The work organization module 50 iteratively searches the next available device 40 to determine the combination of devices 40 capable of fulfilling the work function 350. Multiple sets of matching work / device combinations are given as an answer in such a search and presented to the user as an output solution 75 on the GUI 60. The user selects a pseudo-sentence that he or she likes, which is mapped to a work / device combination and uses the device basic knowledge 380 on device manual 320 to accomplish the task in sequence. Instantiate the work by using the required work action.
Since the device manual 320 is only logically separated from the work manual 310 on the manual 300, the device 40 participates in a large task and describes its function as a work description. Without limitation, the instructions can be physically combined. Roughly embodied, using Semantic Web technology to create device instructions 320 and work instructions 310, a shared semantic ontology (language) enriches all aspects of equipment and work in a machine-readable form. It is permissible so that it can be described in. Although the above description has been described in terms of home appliances, those skilled in the art will recognize that the present invention can be used in other application fields such as mobile devices, desktop computing, and corporate computing. ..
As such, module 50 combines work instructions with equipment instructions to determine if the equipment can be used for a particular task. In order to perform user work, the module 50 acquires information on the connection between the work and the device manual to determine the basic knowledge of the device, but here in the work manual, the module 50 organizes the device operation. Contains the permutations of the action scripts used for. In addition, module 50 combines basic knowledge information from multiple device manuals with work manual scripts to generate specific scripts for controlling the device for user work. The device basic knowledge information is specified based on the function from the device manual. In addition, device basic knowledge information identifies interactions and parameter settings for determining control of the device. Equipment basic knowledge can be specified in the form of URIs (Uniform Resource Identifiers) that describe the actions and parameters used. In addition, the device basic knowledge information may include multiple basic knowledge information sections that are combined to perform the required function. The device description may also include multiple basic knowledge information sections that map other protocols used by the controller or device software.
Hereinafter, an embodiment of a work organization system 10 for enhancing the function and intelligence of the internal device 40 by using the above method will be described according to an embodiment of the present invention. As shown in FIG. 2, the work organization system 10 embodies a technique in which a user requests a work proposal from the system 10 and a technique for interpreting and processing the work proposal. In this embodiment, several semantic web-based tools and techniques are applied to create efficient systems, including developing ontology for semantically rich markup language individuals (eg, devices). Will be done. The ontology is structured to display the appliance 40 and the user work request 95 and is described in more detail below.
The work organization module 50 provides a work proposal to the user and accepts a request from the user to organize the execution of the work proposal selected by the user. Work organization module 50 handles this by parsing the selected work proposal 95, using the inference engine (65 in Figure 1B) as the top level layer, in combination with the user interface (60 in Figure 2). It finds a suitable device 40 that can make the selected work proposal and communicates this information to the lower hierarchy (eg, module 70). In one example, a suitable markup language with the inference engine 65 is used to infer the content expressed in that language, by which individuals communicate and understand each other. In one version, semantic techniques are used to describe the work to facilitate the coupling of device functions and device function centers.
For example, on the home network 40, the following types of device 40 are used.
TV-Audio / Video display DVD-Audio / Video playback, photo display PC-Audio / video display, photo display, game play, AV source Receiver-Connect Source to Sink, Play Audio, Play Radio Refrigerator-Web surfing, audio / video play, control Lamp-Control Phone-Play audio, call users, database users PVR-Record audio / video, stop, play, save photos, view photos Camera-Record audio / video, play audio / play, view audio / video streaming, etc. Examples of possible work on such a home network 85 include:
"Water supply" "Garden" "Play", "Movie", "Matrix", "Surround Sound" "Display" "Photo" "Brazil" "Synchronization" "MP3 player" "Dark" "Electric light" "Play" "My music" "Living room" An example (scenario) as described above uses, for example, a pseudo-written language to express a work proposal, whereas, for example, a semantic language may be used to describe the device and function, and knowledge. An inference engine may be used to infer about and make work requests. In one example, the language chosen to express an ontology / rule provides a semantic level of agreement, a high degree of elasticity and expressiveness, support for types and embracing relationships, support for data types and expressions of restrictions, and so on. .. The knowledge base 105 is included in the work organization module (50 in Figure 2).
In another example, Semantic Web technology can be leveraged in the Work Organization Module 50. The Semantic Web is an extension of the current Web so that the meaning of information is well defined, making computers and humans work better together. A markup language designed for the Semantic Web provides a common understanding among diverse individuals. Examples of Semantic Web technologies include DAML (DARPA Agent Markup Language) + OIL (Ontology Inference Layer), DAML-S, OWL (Ontology Web Language), etc., which were developed for the Semantic Web. Was done. In one example of the work organization module 50, the device and work ontology 110 are structured using DAML + OIL and loaded onto the knowledge base 105. Various attributes and functions of the device 40 are identified, and an ontology that captures the relationships between these attributes is structured. Further, the device instance 115 in DAML + OIL is structured on an ontology, where the instance is essentially a semantic description of the device in terms of its interface and functionality. As illustrated in the work organization flow of Figure 1B, user-selected work proposals are divided into one or more tasks as requested and mapped to the functions of the tasks that can be performed by the available equipment. Will be done. This mapping is made possible by inferring the information expressed on the ontology and applying additional rules to the device / user state / preference.
With reference to the examples of FIGS. 2 and 4, the work organization module 50 interacts with a module that provides an interface to the device organization module 50 so that the device instructions can be obtained whenever needed. Exists in. In one embodiment, the DAML + OIL work instance 120 provided by the existing device has a predefined user preference 130 expressed in the knowledge base 105 and written as Jess's Law and / or DAML + OIL. It will also be loaded on the Knowledge Base 105. Further, the device preference for the requested work and the attribute preference for the functions required for the work are integrated. All possible'verbs' (eg, "play") and'subject words' (eg, "movies") are searched from the knowledge base 105 and displayed on the GUI 60.
As such, user work requests are captured by the GUI 60 as a set of fields (eg, 135 in Figure 2) and provided as input to the knowledge base 105 of work organization module 50. In one example, the input obtained from the user includes any selection field. In another example, the input obtained from the user is (a) required fields: verb, subject word and content, location, terminal device, and (b) selection fields: attributes (eg surround sound, wide screen, etc.). Including.
In this example, after the user has selected a'verb'and a'subject', the working organization module 50 communicates with the protocol stack in module 70 to do whatever is possible related in the context of the'subject'. Acquire.
Once the complete work proposal of the user's choice is obtained, the work organization module 50 parses the work proposal and the corresponding work description 95 and calculates any possible combination of devices 40 that can service the solution to the request. The work organization module 50 then applies preferences, ranks the solutions, and then transmits to the GUI 60 to display the output 75 for the user.
As mentioned, the work organization module 50 interacts with the user through the GUI 60, with the inference engine (65 in Figure 1B), and with the rules engine (130 in Figure 2). The work organization module 50 also includes a knowledge base 105 that stores the ontology in the form of facts and applies Rule 130 to that data.
Hereinafter, an example of an operation scenario will be described with reference to FIG.
1. In step 400, during initialization, the device and work ontology 110 is expressed in the knowledge base 105, where the content of the ontology is in the format ('property value'' subclass''human''individual'). Is converted into a fact that has.
2. In step 405, device and work instances 115, 120 are also represented in the knowledge base 105. In this example, the ontology 110 is embodied in AV (Audio-Video) devices such as TVs, CD players, DVD players, IPAQs, speakers, and transcoders.
3. In step 410, after the ontology 110 and instances 115, 120 are loaded, the'verb', the'subject' associated with the'verb' and the'work' assisted by device 40 are in the knowledge base 105. Extracted from the facts.
4. In step 415, the'functions' required to perform the task identified above in a similar manner are also extracted from the facts in the knowledge base 105 as new facts for easy retrieval. It will be saved.
5. In step 420, using GUI 60, the user can use, for example, a'verb', a'subject' and related content (the category to which the selected subject belongs) associated with the context of that'verb'. Identify / select work proposal 95 by selecting. The user can also identify the terminal device 40 where the preferred location and / or the requested / selected work proposal is made.
6. In step 425, the function of the device required to perform the work is identified (preferably such information is stored as pre-computed and easily detectable facts and is required in this step. Must not be long).
7. In step 430, the work organization module 50 identifies the device 40 with the required functionality as a solution. If the work requires a variety of functions, the work organization module 50 collectively identifies a set of devices 40 that can perform / service the work.
8. In step 435, after identifying the set of devices 40 that can make work proposals of the user's choice, user preference 130 is applied to the set of solutions and ranked.
9. At step 440, the user sees the set of solutions as output 75, in increasing order of their rank. An example of a ranking relative to a home network is a home living room equipped with a plasma TV, PC, speakers and IPAQ. Users set their preference to have a device screen size of 40 inches or larger when playing movies, or prefer speakers over IPAQ when playing audio. For work requests such as "Play the movie matrix in the living room", the work organization module 50 identifies the next device combination (solution) that can make the requested / selected work proposal: PC and speaker, PC, IPAQ, and plasma TV. Solutions are ranked according to user preference as described above and are displayed to users as (1) Plasma TV, (2) PC and Speaker, (3) PC and IPAQ.
The user selects the combination of the most preferred device 40 of the solution and uses the work action to instantiate the work, but the work action is the device manual (320 in Figure 5) to accomplish the work. ) Request device operation using the above device basic knowledge 380 in order. As mentioned, if the user selects a set of devices 40 to perform the requested work, the work organization module 50 calls the interface suitable for the device required for the work.
Hereinafter, an example of the operation of the user interface module 60 in the embodiment will be described with reference to each step on the flowchart of FIG. 7. Users can incrementally create work requests using GUI modules through four steps as described below.
1. At step 500, a valid'verb'is displayed on the GUI 60. The user can select a'verb'in the list and click'next' to proceed to the next step (ie, step 505). The user can change the selected'verb'at any time before clicking'Next'.
2. In step 505, a list of all'subjects' associated with the selected'verb' is displayed on the GUI 60. The user may select'subject'in the list and click'next' to proceed to the next step (ie, step 510). The user can change the selected'subject'at any time before clicking'Next'. The user can also click'back'or'back to the beginning' to go back to the previous step (ie, step 500) and change the selected'verb'.
3. In step 510, a list of all'contents' related to the selected'subject' is displayed. The user may select'content'on the list and click'next' to proceed to the next step (ie, step 515). The user can change the selected'content'at any time before clicking'Next'. The user can also click'Back'to go back to the previous step (ie, step 505) and change the selected'subject', or click'Back to the beginning'to step 500. You can go back to and change the selected'verb'. At one step in the process, or at the end of step 510, the user can set a desired position for performing the work and / or a terminal device for performing the entire work or part of the work.
4. In step 515, the task organization module 50 is identified by the selected'verb','subject','content', user preference (location, final terminal set by GUI, and facts and rules. Other preferences, etc.), and the facts about the existing equipment and work on the knowledge base 105 are internally inferred. After inference, the work organization module 50 determines any possible combination (ie, solution) of equipment that can perform the requested work. The resulting set of solutions is ranked within the Work Organization Module 50 and displayed to the user by the GUI 60 (eg, in increasing order). Next, an example of the process in step 515 will be described. The work organization module 50 reads the work and equipment instructions and stores this information as facts in the knowledge base 105. The inference engine 65 interprets the work and equipment description and provides new information that is not directly specified in the work and equipment description, but is specified as a form of class or characteristic, or their relationship. Derived. The work organization module 50 processes and manipulates the information.
In one example, the basic steps associated with the work organization module 50 include:
1. Read the work and equipment instructions 2. Apply semantics to work and equipment instructions 3. Use the parser to convert the work and equipment instructions loaded on the knowledge base 105 4. Apply other rules and questions as explained above As mentioned, the ontology 110 is structured as a mechanism for describing work, equipment and user preference. In general, ontology is the identification of conceptualization. This is a conceptual description (identification of a typical program) and is a relationship that can exist for an individual (eg, device, user, etc.) or a set of individuals. In this example, devices and operations are described based on their basic attributes / properties and / or provided or required functionality. The instructions are encoded using a markup language, based on which the user's work requirements are understood, processed, and serviced.
In addition, the'feature + content type'attribute may be included in the ontology 110. In the embodiment of the ontology 110, what kind of data is allowed as input for each function provided by the device 40, what kind of data the device 40 accepts the input from, and what kind of data. Is identified as being produced as output (if any). This is very common and is sufficient as an answer to requests that do not specify the type of content. However, if the work organization module 50 wants to provide a content type-based index for the device, it requires a way to represent the functionality that includes the content type (eg, mp3render) as not being a separate attribute. .. This can be achieved by including the'feature + content type'attribute in the ontology 110.
In addition, an ontology that expresses user preference 130 can be utilized. In the embodiment, the preference 130 can be expressed in markup language in a manner similar to that specified for device instance 115 and work instance 120. Instances of user preferences can be structured to create such instructions in a standardized / general format and better represent the relationships between other attributes of the preference set. Ontology can be utilized based on.
Also, a mechanism for understanding preference can be embodied. As an example, the work organization module 50 can understand preferences such as "When playing a movie, choose a device with a larger screen size." Preferably, the work organization module 50 can be understood that "plasma TV is always better than other types of TV and has the largest screen size". Also, more general ontology may be included, which may be applied to all kinds of devices, and even to devices unknown at the time of implementation.
Although the above examples have been described in the context of home appliances and home networks, those skilled in the art will appreciate that the present invention can be applied to mobile personal devices, desktop work, complex corporate systems, and the like. Further, in this description, embodiments of the work organization system have been described in relation to home appliances, but those skilled in the art will appreciate that a known set of individuals interacts with the work and user organization to those individuals. It will be appreciated that it is also useful in other application areas where a simple method for is required (eg, production lines for identifying various movements and assembly robot settings). Moreover, because they are logically separated from the device, there is no reason why they cannot be physically coupled to the actual device. Also, although only one controller / organization module has been described as an embodiment, the set of devices influences the control operation and the organization of the devices.
The present invention has been described in great detail for a particular best version. However, other versions are possible. Therefore, the ideas and scope of claims should not be limited to the scope of the description of the optimum version contained therein.
The present invention can be suitably applied to technical fields related to user work for controlling electronic devices.
<figref num="1A">It is a block diagram which shows the example of the structure of one Embodiment of the work organization system which concerns on this invention.</figref><figref num="1B">It is a flowchart which shows the example of the work organization process by the system of FIG. 1A.</figref><figref num="2">It is a block diagram which shows the example of the structure of another embodiment of the work organization system including the work organization module which concerns on this invention.</figref><figref num="3">It is a figure which shows the example of the work proposal using the pseudo language in the system of FIG.</figref><figref num="4">It is a figure which shows that the device function manual is acquired from the device in order to generate the work suggestion in the system of FIG.</figref><figref num="5">It is a figure which shows the example which logically separates the work manual and the apparatus manual which concerns on this invention.</figref><figref num="6">It is a flowchart which shows the example of the step which generates and displays a work proposal.</figref><figref num="7">It is a flowchart which shows an example of the steps performed by the work organization module in the system of FIG. 2 by displaying a work proposal and letting a user select it.</figref>
Code description
10 Work organization system 20 Client device 30 Controller device 40 Controlled equipment 50 Work organization module 55 Device control module
8 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP2002063033A | Cites | Japan |
| JP2002533802A | Cites | Japan |
| JP08166945A | Cites | Japan |
14 members in 7 offices
Priority claims2
| Document | Office | Kind | Date |
|---|---|---|---|
| 950121 | United States of America | – | |
| 95012104 | United States of America | A |
Members14
| Document | Office | Kind | |
|---|---|---|---|
| EP1640838A1 | European Patent Office (EPO) | A1 | |
| US2006069602A1 | United States of America | A1 | |
| JP2006092550A | Japan | A | |
| KR20060051202A | Republic of Korea | A | |
| CN1855038A | China | A | |
| KR100739733B1 | Republic of Korea | B1 | |
| EP1640838B1 | European Patent Office (EPO) | B1 | |
| AT376204T | Austria | T | |
| ATE376204T1 | Austria | T1 | |
| DE602005002906D1 | Germany | D1 | |
| DE602005002906T2 | Germany | T2 | |
| CN100541427C | China | C | |
| JP5027397B2This record | Japan | B2 | |
| US8412554B2 | United States of America | B2 |
15 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Cancellation because of no payment of annual feesLAPS | LAPS | |
| Renewal fee payment (event date is renewal date of database)FPAY | FPAY | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Transfer to examiner for re-examination before appeal (zenchi)AppealJAPANESE INTERMEDIATE CODE: A911A911 | A911 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Decision of refusalJAPANESE INTERMEDIATE CODE: A02A02 | A02 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written permission of extension of timeJAPANESE INTERMEDIATE CODE: A602A602 | A602 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 5027397
- Application
- 274135
Titles2
- Japanese
- 分離された作業及び装置説明書を使用した家電製品の記述方法及びシステム
- English
- Descriptive methods and systems for home appliances using separate work and equipment instructions
Classification
- CPC, 4
- G06Q10/06316
- G06Q10/10
- G06Q10/06311
- G06F9/453
- IPC, 3
- G06F3 048
- G06F3 0482
- G06F3 04842
