Apparatus and method for controlling remote device

The present invention relates to the field of a graphic user interface for a computer system, and in particular, an intuitive methodology for an operator to control a remote device with a computer controlled object-oriented user interface using a moving image. It relates to the devices and methods that enable it.

The most popular graphic user interfaces available today include, for example, the Xerox Star (Xerox Star), Apple Macintosh (Apple Macintosh), Microsoft Windows (registered) (Microsoft Windows (registered)) graphic interfaces. , Based on "desktop metaphor". In the desktop metaphor system, the display screen is treated as a virtual desktop. Graphic symbols on virtual desktops (that is, display screens) are used to display common objects (hereinafter referred to as objects) found in office environments such as files, file folders, and printers. These desktop metaphor systems have various restrictions that prevent the effectiveness of the man-machine interface between human operators and computer systems.

The existing desktop metaphor graphics user interface has problems such as the use of a hierarchical file system. Most of the graphic user interfaces in the desktop metaphor are based on computers that have a file system in which computer files are organized in a hierarchical structure. The hierarchical file system structure includes a root directory and a plurality of subdirectories contained in the root directory. Furthermore, each subdirectory may have a subsubdirectory or the like. The hierarchical file system is graphically represented on the screen like some kind of root file cabinet with an internal file folder. Both Microsoft Windows® and Apple Macintosh® operating systems use a hierarchical file system that looks like a file folder.

File folder hierarchies often confuse new users. For example, a system that stores a file in a filefolder, and this filefolder is nested in yet another filefolder (hereinafter referred to as nesting) to have a structure of arbitrary depth is inherently confusing. Many users get lost while entering the hierarchical file system. Users get lost because the filefolders are very similar, and no feedback is given about the user's position within the filesystem hierarchy. Another reason is that the file folder does not provide any information about the contents of the file folder other than the user-defined label. For example, in addition to a short text label that can be defined by another user, whether the opened folder is an accounting file folder containing spreadsheet data or a database file folder containing customer databases. It is generally impossible for the user to judge. Therefore, it is difficult to get into a complex file system hierarchy because the current desktop metaphor graphic user interface gives little feedback on the user's position within the file system hierarchy.

Current graphic user interface systems based on the desktop metaphor often do not fit certain applications ideologically into the desktop metaphor environment. That is, non-office applications may not be suitable for systems based on desktop metaphors. For example, in the desktop metaphor, the application that controls the TV must be stored in a "paper" file folder on the desktop.

Another common problem with the desktop metaphor user interface is that many applications are displayed on the desktop, making it difficult to see. Many users of desktop metaphor systems typically store a large number of applications and files in the form of icons on the display screen. However, if a user wants to open a particular application or file, the user must search the display screen to find the desired icon, much like finding a particular paper file on a cluttered desk. The icons on the desktop do not have a clear hierarchy and are all the same. Such equality confuses users and becomes a problem when they want to switch from one task to another.

Various systems have been developed to improve the organization of the graphic user interface of the desktop metaphor. There is a system of the concept of "room". (See Non-Patent Document 1) In addition, similar techniques are also introduced in Non-Patent Document 2, Non-Patent Document 3, Patent Document 1, and the like.

In a system based on the concept of rooms, users create different rooms that they can perform for a task or related task group. Each room represents its own virtual desktop with associated task groups. Each room has a label that identifies the room by function. For example, a user can create a mail room, a typing room with a word processor, or an accounting room with a spreadsheet. In addition to windows that are shared by all rooms, such as clocks, each room is given only icons and windows that correspond to a specific task room.

To switch between tasks in the system using the room concept, the user moves between rooms associated with different tasks. For example, to move from a spreadsheet to an email search, the user moves from the accounting room to the email room. To move between two emails, the user selects an icon that represents the door connecting the current room to another room. Users can create doors between available rooms.<nplcit num="1"><text>"Learning Consideration in User Interface Design: Room Model", Patrick Chan, Ontario, Canada, Department of Computer Science, University of Waterloo Report CS-84-16, July 1984</text></nplcit><nplcit num="2"><text>"Computer as a Theater", Brenda Laurel, 1991, Addison-Wesley Publishing Co.,</text></nplcit><nplcit num="3"><text>"Science of Human-Computer Interface Design", Brenda Laurel, 1990, Addison Weslley Publishing Co., Ltd.</text></nplcit><patcit num="1"><text>U.S. Pat. No. 5,072,412, "User Interface with Multi-Workspace for Sharing Display System Objects," Henderson, Jr. et al.</text></patcit>

<p> The room concept helps solve problems such as "desktop clutter", but still relies on a common desktop metaphor. For example, the obscure hierarchical file structure used to record information cannot be simplified with the concept of a room interface. In addition, the room concept cannot solve the form-function mismatch that becomes apparent when adding an application that is not suitable for the desktop metaphor to the system.</p><p> In the traditional graphic user interface, the user had to add new control software every time a new hardware component was introduced into the system. For example, if a user of Apple® Macintosh® or Microsoft® Windows® wants to install a new printer in their system, they control the printer they add. You also have to install a new printer driver. Therefore, every time a new device is added to the system, the user has to go through a complicated procedure to add appropriate software.</p><p> Therefore, an improved graphic user interface that solves the problems related to the desktop metaphor system is provided to obtain a remote device user interface that does not require the user to load software. The remote device can be controlled using the user interface of the remote device displayed on the display screen. The user interface of the control device will be described with reference to the control of the remote device, but it can be understood from the following description that it can be applied to various fields such as computer workstations, portable computers, and handheld control systems. Think of it.</p>

<p> The present invention provides a control device and method for a remote device for solving the above problems, and a method for supplying information to a program object that can be used for remote control.</p><p> The first control device of the present invention is a control device that controls at least one remote device, and is a central arithmetic processing unit (CPU), a memory connected to the central arithmetic processing unit, and the central arithmetic processing unit. A display screen connected to the interface and displaying a graphic image, a communication circuit connected to the central arithmetic processing unit and formed to send and receive messages via a communication network, and a program transmitted by each remote device. A means for receiving a user interface program object which is an object and defines a graphic user interface of each remote device via the communication circuit, and a graphic user interface corresponding to the remote device are displayed on the display screen. User Interface Provides a means of calling a user interface method in a program object.</p><p> The second control device of the present invention is a display device that controls at least one remote device connected to a communication network, and is a central arithmetic processing unit (CPU) and the central arithmetic processing unit (CPU). A means of receiving information including a memory connected to the display device, a user interface object used to define the user interface of the display device, and a device driver object used to control the remote device from the display device. A means of accessing the database to obtain the control code for the remote operation, a means of displaying a portal object corresponding to the received user interface object and corresponding to at least one object-oriented method, and a means of displaying the portal object. When the user selects the portal object, the means for displaying the graphic corresponding to the user interface object of the remote device on the display device and the means for sending the control code to the remote device based on the received device driver object. To be equipped with.</p><p> The first control method of the present invention is a control method in which a graphic user interface is acquired from a remote device and the remote device is controlled by using a display device at hand connected to a communication network. A step of connecting to a communication network, a step indicating that the remote device has a graphic user interface capable of transmitting to an object connected to the communication network, and a display device of the remote device to the remote device. A step of requesting a graphic user interface, a step of transferring a graphic user interface program object including a user interface method for displaying the requested graphic interface on the display device from the remote device to the display device, and the display device , Calling the user interface method in the user interface program object to display the graphic user interface on the display device.</p><p> The second control method of the present invention is a method of controlling a remote device using a display device connected to a communication network, which is used to define a user interface of the display device at hand by the display device. A step of receiving information including a user interface object and a device driver object used to control the remote device from the display device, a step of accessing a database to obtain a control code for the remote operation, and the above. The step of displaying the portal object corresponding to the user interface object received by the display device and corresponding to at least one object-oriented method, and when the user selects the portal object, the display device is described. A step of displaying a graphic corresponding to a user interface object of the remote device and a step of sending the control code to the remote device based on the received device driver object are executed.</p><p> The first information supply method of the present invention is a method of supplying information to a program object in an object-oriented graphic user interface that displays an object with a graphic icon, and is a method of supplying information to a program object at a first position on a graphic display screen. A step of displaying a graphic image, a step of accepting user input including a command to move the data object to a space corresponding to the program object, and the graphic which is a position of the program object and a destination of the data object. A ghost image consisting of a step of accepting user input, including a position on the display screen, and a graphic image drawn in a blurred color with an exact copy of the graphic image of the data object at the first position on the graphic display screen. To display that the ghost image and the step of moving the data object in response to the command to move the data object, and to change the appearance of the program object so that the program object has accepted the data object. The steps shown, the step of accepting user input including a command to supply the data object to the space corresponding to the program object when the appearance of the program object is changed, and the ghost image and the graphic image of the data object. And the step of feeding back that the data object has moved are executed.</p><p> The second information supply method of the present invention is a method of supplying information to a program object in an object-oriented graphic user interface that displays an object with a graphic icon, and is a method of supplying information to a program object at a first position on a graphic display screen. A user including a step of displaying a graphic image, a step of accepting user input including a command to move the data object, and a position on the graphic display screen which is a position of the program object and a destination of the data object. A ghost image consisting of a step of accepting input and a graphic image drawn in a blurred color with an exact copy of the graphic image of the data object at the first position on the graphic display screen is displayed, and the data object is displayed. A step of moving the ghost image and the data object in response to a command to move, a step of changing the appearance of the program object to indicate that the program object has accepted the data object, and a step of the program object. When the appearance is changed, the step of accepting user input including the command to supply the data object to the program object and the step of blending the ghost image and the graphic image of the data object are familiar to the user. A step that provides multiple spaces consisting of a graphic background image depicting a real world background, and a portal consisting of graphic icons with a corresponding target space on the graphic display screen at least one space in the space. The user input including the location information including the step of connecting the space where the portal exists to the corresponding target space by the portal is an operation input for holding a predetermined minimum time object on the portal. It is characterized by being.</p><p> The third information supply method of the present invention is a method of supplying information to a program object in an object-oriented graphic user interface that displays an object with a graphic icon, and is a method of supplying information to a program object at a first position on a graphic display screen. A user including a step of displaying a graphic image, a step of accepting user input including a command to move the data object, and a position on the graphic display screen which is a position of the program object and a destination of the data object. A ghost image consisting of a step of accepting input and a graphic image drawn in a blurred color with an exact copy of the graphic image of the data object at the first position on the graphic display screen is displayed, and the data object is displayed. A step of moving the ghost image and the data object in response to a command to move, a step of changing the appearance of the program object to indicate that the program object has accepted the data object, and a step of the program object. A step of accepting user input including a command to supply the data object to the program object when the appearance is changed, a step of fusing the ghost image and a graphic image of the data object, and a command to move the object. Takes input, including, and performs steps to generate a tearing sound from the handheld device.</p>

[Notation and terminology]

Much of the detailed description given below relates to the symbolic representation of display images, algorithms, and operations of data bits in computer memory. These algorithmic descriptions and representations are the means used by those familiar with the field of data processing to most effectively convey the content of their work to people in the same field.

Here, an algorithm is generally considered to be a consistent procedure of a number of steps leading to a desired result. These steps require physical manipulation of physical quantities. Usually, but not always, these physical quantities are in the form of electrical or magnetic signals that can be recorded, transferred, combined, compared, selected, selected, modified or manipulated. The main reason is that they can be used in common, but it turns out that it is sometimes convenient to call these signals bits, values, elements, symbols, letters, images, terms, numbers, and so on. However, it should be remembered that, like other similar terms, these terms all correspond to suitable physical quantities and are merely convenient labels attached to these physical quantities.

Nowadays, arithmetic is performed jointly by human operators and machines. The arithmetic execution machine of the present invention incorporates a general-purpose digital computer or other similar device. In any case, it should be remembered that the method of operating a computer and the method of operation itself are different. The present invention relates to a method step of operating a computer and processing electrical or other physical signals to produce another desired physical signal.

The present invention also relates to a device that executes these operations. The device can be specially configured as needed and can be configured with a general purpose computer that can be selectively started by a computer program recorded on the computer or reconfigured. The algorithms used herein are not specific to any particular computer or other device. In particular, it is possible to programmatically use a variety of general purpose machines according to the methods described herein. Alternatively, special equipment can be configured to perform the steps of the required method for greater convenience. The configurations required for these various machines will be clarified in the following explanations. Machines capable of performing the functions of the present invention include computer system manufacturers including Sun Microsystems, Inc. in Mountain View, California, as well as the Assignee and First Person. There are devices manufactured by.

[Coding details]

This specification does not indicate a specific programming language that performs the various procedures described in the specification. This is partly because not all enumerable languages are available worldwide. The user of a particular computer knows the most suitable language for the purpose at hand. In practice, it has proved useful to implement the invention in advanced languages and then compile this language into machine-executable object code. A detailed program list is not included in this specification because computers and monitoring devices that can be used in the practice of the present invention are composed of a large number of elements of different types. The specification and drawings show sufficient operations and other procedures to the extent that the invention of the present application can be carried out as long as it has general art.

[Outline description of the embodiment]

The graphic user interface of the embodiment shown below is displayed on the handheld display device. The handheld display device consists of at least one central processing unit (CPU) connected to the graphic display system. In the example shown below, the graphic display system is a touch-sensitive type, and the user can input information with a finger without using another input device such as a keypad. A communication circuit is also connected to the CPU, and information is communicated via some kind of communication medium. A communication circuit is used to control the remote device and communicate with the remote device.

The graphic user interface creates a world of Spaces organized in the form of a geographic map structure. It is easy to navigate inside the spatial world because each space constitutes a normal geographical environment. Each space is displayed as a background image, which serves as a navigation marker to identify where this space is located in the geographic map structure. For example, the background image of a living room space includes items commonly found in the living room, such as lamps, chairs, and tables. This makes navigating this "world" much easier, as users can orient themselves by looking around and looking around as people do in the real world.

Within each space, graphic objects that can be selected and manipulated by the user are arranged by the user interface. The graphic object is a cartoon-like video graphic object that can be easily identified like a TV or VCR in the real world. There are at least three types of objects that exist in the user interface space. That is, there are three types: data object, button, and gate (hereinafter referred to as portal). A data object is an object associated with a particular piece of information. A button is an object associated with a particular function or action. That is, when the user selects a button, the user interface performs the function or action that corresponds to that button. A portal is an object corresponding to the second space displayed in the first space. That is, when the user selects a portal, the user enters the portal and goes to the second space corresponding to this portal. The user environment can be further improved by using sounds that match the user's movements and by giving visual feedback.

To control real-world remote devices, many of the graphic objects shown on the screen are associated with real-world remote devices. To control a remote device, the user selects a graphic object on the display screen that corresponds to the particular remote device. When the user selects a graphic object that corresponds to the remote device, a user interface for controlling the remote device is displayed on the handheld display device. In this way, the user interface of the remote device and the user can talk. In order to control the remote device, the device driver method for the remote device is called in the present invention. This device driver method can be located within the remote device or inside the handheld display, depending on the sophistication of the remote device.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

A computer-based remote control system with an intuition graphic user interface will be described. For the purposes of explanation below, specific terms will be described so that the present invention can be fully understood. However, it is obvious to those skilled in the art that these specific detailed explanations are not necessarily necessary for the practice of the present invention. In other cases, well-known circuits, functions, processes and devices are shown in the form of block diagrams and conceptual diagrams so that the invention is not unnecessarily obscured.

[System configuration]

Figure 1 shows an overview of a computer-based remote control system. The main component consists of the handheld display device 170 shown in FIG. In this embodiment, the handheld display device 170 is composed of a computer system with a color graphic touch screen display. The handheld display 170 is designed to control compatible remote devices such as the thermostat 150, videocassette recorder 39, and stereo device 152. The handheld device 170 communicates with the remote device using the communication medium 160.

The remote control system shown in Figure 1 is very different from traditional remote control systems because it is based on modular object-oriented software. Object-oriented software allows you to take advantage of features not available in traditional remote control systems. For example, when an operator (sometimes referred to as a "user") adds a new compatible remote device, such as a thermostat 150, to the system, the new remote device sends its information to the handheld display 170. The transmitted information includes a user interface object that can be used when sending a graphic user interface to a remote device. The remote control system uses a user interface object to display a new graphic user interface for remote devices on the handheld display device 170. To organize all available user interface objects, the handheld display 170 creates a "world" of geographic space that can be easily navigated using a set of gestures.

First, the hardware required to configure the handheld display device 170 and the compatible remote device will be described. Next, a graphic user interface consisting of a geographical space and an object displayed on the handheld display device 170 will be described, and a set of gestures used by the operator when navigating around the user interface will be defined. We will explain how to use navigational gestures to interact with the objects displayed on the handheld device 170 in order to combine the elements of the user interface, exemplifying the user interface concretely. Finally, a system is defined in which the handheld display device 170 communicates with an external remote device and extracts information from the remote device so that the remote device can be controlled.

In the attached drawings, the feature portion of the present invention is displayed in black and white, but as will be described later, in the actual embodiment, the object and the feature portion are displayed in color.

[Hardware configuration]

The handheld display device 170 is composed of a computer system as shown in FIG. The computer system 170 of FIG. 2 is composed of a central processing unit (CPU) 30, a memory unit 33, and an input / output (I / O) unit 34. These components are common components in most general purpose computers, and in fact, the handheld computer system 170 is intended to be typical of a wide range of data processing devices capable of producing graphic representations.

A color graphic screen display 37 is connected to the input / output (I / O) unit 34 of the computer system 170. A part of the screen display 37 is composed of a color raster display device such as a cathode ray tube (CRT) monitor or a liquid crystal display (LCD). The screen display 37 must be of sufficient resolution to display a graphic image.

The screen display 37 preferably has a built-in touch screen display device. The touch screen display has a configuration capable of detecting that a finger is touching the screen display 37. In addition, the touch screen can identify the exact location of the finger touching the screen display 37 so that the screen can sense the gesture of the user's finger on the screen display 37. The touch screen display 37 is composed of one of various types of touch-sensitive display screens on the market.

Alternatively, the handheld computer system 170 can be configured with a conventional graphic display system without using a touch-sensitive display screen. Such a system requires an alternative input system for talking to the user. FIG. 2 shows an alternative input device 42. The user interface of the present invention can be used by using an alternative input device 42 such as a mouse, a trackball, and a joystick, but in the present embodiment, a case where the touch screen display 37 is used will be described.

While the system is running, the CPU 30 generates a graphic user interface by executing the computer code recorded in the memory 33 and displays it on the touch screen display 37. Since the present invention is fully described in the present specification so that a person familiar with computer technology can use the present invention for various computer systems by using any one of many computer languages, the memory 33 is used. The recorded computer code is not disclosed in the specification.

An external communication circuit 35 is connected to the input / output (I / O) unit 34 of the computer system 170 shown in FIG. The external communication circuit 35 functions as an information transmitter / receiver for the handheld computer system 170. In the present embodiment, the communication circuit 35 in the computer system 170 is used to communicate with the remote device such as the video cassette recorder 39 and the thermostat 150 by the communication medium 160. The communication circuit 35 uses any one of many well-known communication techniques such as high frequency, low power, cellular, infrared signals or data signals carried over conventional power lines.

The remote device must have at least one means of receiving information transmitted from the handheld display device 170 so as to be controlled by the handheld display device 170. Ideally, the remote device should be an intelligent device capable of sending and receiving information. However, the handheld display device 170 can control any remote device suitable for any one of the three types of remote devices described below.

The first type of remote device is an "intelligent remote device" specially designed to work with the handheld display device 170. Here, "intelligent remote device" is defined as a remote device having a built-in microprocessor, communication system, and memory. FIG. 3 shows a block diagram of the parts inside the intelligent remote device 150. As shown in FIG. 3, the intelligent remote device 150 includes a CPU 403, a memory 404, a communication circuit 401, and a device control circuit 405.

The handheld display device 170 communicates with the intelligent remote device 150 via the communication circuit 401 of the intelligent remote device. The memory 404 records the programs and data used by the CPU 403, and these data include the user interface object 303 and the device driver object 301. The CPU 403 in the intelligent remote device controls the remote device by calling a method in the device driver object 301.

The second type of remote device is a "simple remote device" specially designed to work with the handheld display device 170. Here, if the remote device does not have a microprocessor, or if the remote device has a built-in processor that is so primitive that the handheld display 170 cannot call the method in the device driver object in the remote device. The remote device is defined as a "simple remote device".

FIG. 4 shows a block diagram of parts in a typical simple remote device 155. As shown in FIG. 4, the simple remote device 155 is composed of an emitter 420, a receiver 421, a memory 424, and a device control circuit 425. Emitter 420 in the simple remote device 155 is used when transferring the contents of memory 424 to the handheld display device 170. Therefore, the simple remote device 155 transmits both the user interface object 353 and the device driver 351 in the memory 424, and the handheld display device receives both of these objects.

To control the simple remote device 155, the display device 170 calls a method in the device driver object 351 to translate the user's interaction into a simple predefined control code. The display device 170 then sends a simple predefined control code to the simple remote device 155. This remote device receives the control code via the receiver 421 and further outputs the information to the device control circuit 425.

The third type of remote device is a conventional electronic device that can be controlled by a transmitted signal such as an infrared signal. That is, the invention of the present application is also compatible with most old-fashioned conventional devices capable of remote control. FIG. 5 shows a general electronic device 157 that can be controlled by a coded signal. The device comprises a receiver 450 that receives the transmitted coded signal and a device control circuit 455 that responds to the coded control signal. The handheld display device 170 controls the electronic device 157 by transmitting a coded control signal. The remote electronic device 157 receives the transmitted signal and responds accordingly.

[Display device user interface]

In order to create a new product that is easy to use, the product designer attempts to operate the new product in the same way as the old product that the intended operator already knows how to operate. For example, audio cassette tape recorders have been in use for so long that no one knows how to operate them. Therefore, most people know the buttons on cassette tape recorders such as play, stop, rewind, fast forward, and record. Manufacturers of videocassette recorders used this general knowledge to create easy-to-use videocassette recorders. The video cassette recorder is easy to use because it uses the same buttons for operating the audio cassette recorder, such as play, stop, rewind, fast forward, and record buttons.

In order to make the handheld display device 170 easier to use, the handheld display device 170 displays an intuition graphic user interface. This chapter defines a graphic user interface that is displayed on the handheld display device 170 for the user.

To design a graphic user interface that is easy to navigate, designers must configure the graphic user interface with a structure that most users already understand. Humans live in a three-dimensional graphic world. To survive in a three-dimensional graphic world, you must learn how to navigate local geographic areas, including geographic spaces such as homes, neighborhoods, and towns. For this reason, humans make their own conceptual maps of the geographical space of the relevant regional world.

An ideal map of a geographic space created by a person is a de facto geographical map structure. Figure 6 shows a typical geographic map structure. For example, everyone on earth knows that he or she lives on earth. We also know that each person lives in a particular town and that there are other towns on the planet. Identifying a particular home on a street in a town on Earth is a natural activity in the geographic map structure shown in Figure 6.

The geographical map structure can be extended into the house. For example, the living room is one of many rooms in the house. Televisions, television listings, and videocassette recorders are objects placed in the living room. Each object has its own graphic space. The map structure can be extended to abstract spaces as well as geographical spaces. Each program is described in the TV programs listed in the TV program list. This program article can be thought of as a space in a television listing. Most people understand both the structure of this geographic space and the details within each geographic space. For this reason, the de facto geographical map structure of a space represents a structure familiar to most people.

The present embodiment uses the familiar geographic map structure of the three-dimensional world within the geographic user interface. Describe a technique for a user to navigate within a graphic map structure of arbitrary size and complexity through a graphic background image representing a geographical space. Such a graphic map structure improves the user memory of a complicated data structure. The advantage of having such a geographic map structure is that a wide range of people can use computer-simplified information records because they are familiar with the structure.

FIGS. 7 and 8 show the geographical sample map structure used in the present invention. Each space in the geographic map structure of FIGS. 7 and 8 is represented as a background image shown on the display screen of the handheld display device of FIG. Within each geographic space are objects that can be selected and manipulated by the user.

User Interface Display Element

The user interface environment in this embodiment is composed of "space" and selectable "two main display elements of the object. Each area of the geographical map structure of FIGS. 7 and 8 represents a space. Spaces are a geographic standard that tells you where you are currently in the geographic map structure. Objects in each space are user selectable and manipulable. Touchscreen Display 37 In the embodiment of the handheld display device 170 provided, the user can select an object by touching the object displayed on the touch screen display 37. The user can select the object by touching the object with the user's finger other than the touch screen or an instruction device other than the input device. It is also possible to process the input differently.

[Space] Each space in FIGS. 7 and 8 is provided with a related background image consisting of appropriate images for the space. As the user enters the space, a background image of the space appears on the display screen. Each background image contains a marker that allows the user to identify their current location. For example, the Earth surface space 32 in FIG. 7 is displayed as a background image as shown in FIGS. 10 and 11. The Earth's surface background images in Figures 10 and 11 resemble the streets of the Earth's surface, indicating that the user is in the Earth's surface space of the map structure. The system displays the background image with desaturated or hazy colors so that you can easily distinguish between the background image and the selectable objects. The background image for spaces may be larger than the display screen. For this reason, the system provides a "panning" gesture for the user to move around in space, as described below.

[object]

Each object is a graphic icon, and the icon is displayed in the background image of the space. Each object is actually an object-oriented programming type object with data methods that can be called and invoked on the data. Any object that appears within the space background technology can be selected and manipulated by the user. All the objects are colored in bright colors and the outlines are drawn with black lines. This bright color distinguishes between the objects in the space and the background image of the space drawn using desaturated colors. When the user points to an object, the user interface produces a "selectable object" sound that informs the user that the object is a selectable and operable object. See Figure 10 again. In the example shown below, house 47 and house 48 are objects that the user can select and manipulate.

In the present invention, "pre-animation" is used to inform the user when the operation occurs. Pre-animation is to display a short video generated by the CPU 30 on the display screen. In most cases, pre-animation is about enlarging an object icon so that it looks closer to the user when the object is selected.

For example, in FIG. 9, selecting the Earth Portal 100 expands the Earth Portal 100 as if it were approaching the user. The user interface pre-videos the anticipate icon to notify the user of the move to the Earth Portal 100, and then moves the user to the Earth surface space 32 corresponding to the Earth Portal 100 as shown in FIG.

When the user points to an object for a given minimum time and selects an object, the CPU generates audio and calls methods within the object that perform actions corresponding to the object while notifying the call of the object.

There are at least three types of objects in the user interface of this embodiment. That is, data objects, buttons and portals. The simplest object is a data object. A data object is an object that corresponds to a set of data. A button is an object that provides a means of accomplishing a function. A portal provides a way to move between different spaces. Hereinafter, each type of object will be described.

[Data Object]

A data object is a simple object used to represent some information. For example, data objects can represent colors and television channels. A data object does not have a method that can be called when a data object is selected to perform the corresponding action or function. However, it is possible to drop a data object on top of another object that uses the information in the data object. For example, a TV program guide data object can be dropped on a video cassette recorder object to program a video cassette recorder and record a TV program.

[button]

A button object is a special object that has a corresponding action or function. When the user selects a button, CPU30 calls the action method of the button object to perform the action or function corresponding to that button. When you select a button, the CPU notifies you that the button has been operated and produces a "button action" sound. This sound feeds back to the user that the button has been selected. For certain buttons, there is also visual feedback, and the CPU informs the user of the result of the called method and performs the action corresponding to the button.

For example, in FIG. 14, a TV power ON button 61 and a TV power OFF button 62 are displayed. When the user selects the TV power "ON" button 61, the CPU 30 notifies that the TV power "ON" button 61 has been operated and produces a "button operation" sound. The CPU30 also calls from within the button object a method that transmits the information needed to turn on the actual television device that corresponds to the television control panel in Figure 14.

If the actual TV device is an "intelligent remote device", the TV device returns a result value to the display device, notifying the user whether the TV has been turned on. Next, the user interface lights up the TV power "ON" button 61 and makes a "boom" sound to let the user know that the actual TV is on.

Buttons can be copied and moved to different spaces so that the "world" of the user interface can be specially created to suit the needs of the user. For example, the TV power "ON" button 61 and the TV power "OFF" button 62 can be copied and the copied buttons can be moved from the TV control space to another space. The user can control the TV using a copy of the TV power "ON" button 61 and the TV power "OFF" button 62 in the new space.

[Portal]

A portal is an object that provides an easy way to move from the current space to another space that corresponds to the portal. Users can simply select a portal and move from the current space to another space that corresponds to the portal. When the user selects a portal, the CPU anticipates the move to the portal and produces a "portal-passing sound". The CPU then calls a method that moves the user to the space corresponding to this portal.

For example, in the geographic map of Figure 7, the user simply selects Home Portal 48 to move from Earth surface space 32 to home space 50. As shown in FIG. 10, on the earth surface 32, the user can move by selecting the house portal 48 in the earth surface 32. When the home portal 48 is selected, the CPU anticipates the move to the home portal 48 and produces a "portal-passing sound". This causes the user to move into the home space 50 as shown in FIG.

Portals can be duplicated and moved to tailor the "world" of the user interface to the needs of the customer. For example, in FIG. 13, the TV program guide portal 71 is displayed in the living room space 60. This TV program guide portal 71 corresponds to a space containing TV program guide information. The user can copy the TV listing portal 71 and move the copy to another space where the television listing may be accessible. The user can use a copy of this TV listing portal to access the same information that is accessible on the original television listing portal. In this way, users can access the same information from different spaces.

There are two special portals that are available to user interface operators almost at any time. That is, there are two portals, the wayback portal and the agent portal.

[Wayback Portal]

The Wayback Portal provides a way to go back along the path a user has taken between different spaces. The Wayback Portal 45 consists of a small rectangular frame that displays a snapshot of the previous space according to the path of the space the user has followed. The background image of a space is larger than what you can see on a single screen, so the wayback portal snaps a picture of the previous space, just as you saw it when the user passed through the previous space. The shot is shown. In this embodiment, the wayback portal is available to the user in the lower right corner of the display screen in any space. If desired, the user can move the Wayback Portal to a new location on the display screen so that the Wayback Portal appears in a new location on the screen for all spaces.

To return to the previous step along the current route, the user selects Wayback Portal 45. When this is selected, the CPU 30 notifies the passage of the wayback portal 45, generates a "wayback portal passage sound", and moves the user to the space shown in the wayback portal. In this embodiment, the "passing the wayback portal" sound is actually the same sound as the "passing the portal" sound, except that it is played backwards. By playing the "pass portal" sound backwards as it passes through the wayback portal, the user is emphasized that he is backtracking the current path.

The user leaves an invisible marker trail as they pass through the portal. Each marker represents a visit to the space. The Wayback Portal manages these markers and keeps them in the order of the spaces visited. The marker and the order of the markers determine the path the user has taken through the world of the user interface. In this way, when the user passes through the wayback portal, the route formed by the markers is regressed by one marker at a time. Each time a user goes through the Wayback Portal to a space, one of the markers currently managed by Wayback is deleted.

For example, in the geographic map structure of FIG. 7, if the user starts from outer space 99, he / she moves to the earth surface 32 and then to the house space 50. As a result, a route is created from outer space 99 to earth surface space 32 and then to home space 50. Following this route, the house space 50 as shown in FIG. 12 is displayed on the display screen. The Wayback Portal 45 is displayed in the lower right corner of the house space 50. As shown in Figure 12, Wayback Portal 45 shows a small snapshot of the previous space along the space path. In this case, the previous space along the space path is Earth surface space 32. Therefore, a small snapshot of Earth's surface space 32 is displayed on the Wayback Portal 45, just as the user saw as he passed through the Home Portal. If the user selects the wayback portal 45, the CPU notifies the move to the wayback portal 45 and generates a "wayback portal passing sound". Then, the user moves to the earth surface space 32 shown in FIG. At Earth surface space 32, as shown in FIG. 10, a small snapshot of outer space 99 is now displayed on the wayback portal. This is the first space in the path. In this way, the Wayback Portal 45 is a means of "backtracking" along the space path that the user has traveled.

In this embodiment, the wayback portal 45 is realized by stacking the data structures, and each data structure is composed of a pointer and a coordinate axis pair. A pointer to each data structure on the stack points to a space that the user has already visited. The coordinate pair of each data structure on the stack pinpoints the user's location within the space pointed to by the pointer as the user leaves the space through the portal. Each time a user traverses the portal from a particular space, a data structure consisting of a pointer to the space and a coordinate pair that pinpoints the user's location in the space as they traverse the portal is moved to the wayback stack by the user interface. It is "pushed". The Wayback Portal 45 displays a small snapshot of the space pointed to by the pointer to the data structure at the top of the stack. To go back along the path of space, the user interface pops the data structure off the stack each time the user selects Wayback Portal 45. And move to the space pointed to by the pointer of the data structure.

[Agent Portal]

An agent portal is an anthropomorphic character that assists in navigating the user interface and handling events that occur in the "world" created by the user interface. In the Earth surface space of FIG. 10, Agent 41 is displayed in the lower left corner. Agent 41 carries the object for the user. This carrying function is represented by a briefcase, which is also carried by the agent. The user can access the objects carried by the agent by selecting Agent Portal 41, which allows the user to enter the agent space corresponding to Agent Portal 41. When the user selects the agent portal 41, the agent notifies the move to the agent space, takes out the briefcase, and opens the briefcase. The CPU then makes a "pass through portal" sound, moving the user to the agent space.

To get an object in the agent's briefcase, the user can pick it up and "drop" it into the agent portal 41. Once in the agent space, users can always access objects dropped on the agent portal 41 from any space. This agent will be described in more detail later in the "Agents" chapter.

[Object Wheel]

You often have to choose from a large number of arbitrary objects. Therefore, the user interface must provide two ways, one is to display a large number of objects and the other is to let the user choose one of the displayed objects.

For example, in the television control space 64 of FIG. 14, the user interface of the present invention must provide a way to display any plurality of television channels for the user to select. Screen space is valuable in both systems. If you have to choose from as many options as possible, the options should be displayed compactly. In the present invention, if there are 500 TV channel objects for the user's selection, not only does it take up a very large screen space to display all the channels at once, but also the visual context of the list (ie why the user Are you looking at this list first?) Is lost. In this way, we need a way to allow users to browse through the available objects.

The user interface of this embodiment introduces an "object wheel" that provides a way for the user to make a selection from a large group of objects. In the object wheel, the rest of the objects in the group are not screened, but a small number of readily accessible objects in the object wheel are displayed on the display screen. To access an object that is not visible on the screen, the user "spins" the object wheel to display the inaccessible object on the display screen.

In FIG. 18, the user has a large number of fruit objects available for selection, but only four of them can be displayed horizontally on the display screen 36 at a time. In the present invention, the "object wheel" 38 is provided with a band of fruit objects so that the user can access all available fruit objects, as shown in the conceptual diagram of FIG. Some objects on the object wheel 38 are readily accessible on the display screen 36. To access an object that is not visible on the display screen 36, the user can rotate the object wheel 38. As the object wheel 38 rotates, new objects appear and others disappear, as shown in Figure 20. To operate the object wheel shown on the display screen 36, the operator uses navigational gestures, namely point, spin, and pickup gestures. These three voyage gestures will be explained in a later chapter.

The object wheel follows the laws of physics so that the user feels as if the real wheel is spinning. For example, the object wheel has mass, which creates centrifugal force. When the user applies more "force" when rotating the wheel, the object wheel rotates at a high speed, and the rotating object wheel gradually stops rotating, so the object wheel can cope with friction. I understand.

The user interface of this example makes a "click" sound while the object wheel is rotating and the object is traversing the screen. The "tick" sound produced by the user interface resembles the sound produced by the claws of a slot machine that applies force to stop and align the wheels. Like a real wheel, the object wheel of this embodiment is always aligned when stopped, and never stops between object choices. You can stop the rotating object wheel at any time by pointing to an object that is moving on the object wheel while the object wheel is rotating.

Object wheels can be arranged vertically or horizontally on the display screen. When arranged vertically, the object wheels can be rotated in either the up or down direction. In addition, when arranged in the horizontal direction, it can be rotated in either the left or right direction.

[User interface navigation gesture]

Navigation of the user interface and operation of objects are performed using a screen instruction device. In the present embodiment, the handheld display device 170 is configured to include a touch screen display 37, and the instruction device to be used is a user's finger. Alternatively, it is possible to realize this user interface by using another type of screen instruction device such as a mouse, a trackball, or a joystick, but in the present embodiment, the case where the user's finger is used as the instruction device of the touch screen is used. It will be described by way of example.

In the embodiment of the present invention, a part of the handheld display device 170 is composed of a touch screen display 37 as shown in FIG. When you place your finger on the display screen, the touch screen display 37 identifies where your finger is. The user selects and manipulates an object on the touch screen using a set of six navigation gestures: point, select, pan, spin, pookup, and drop. .. The navigation gestures used in the user interface will be described individually and in detail below.

[point]

The point is the simplest gesture in the user interface of the present invention. When the user uses a point gesture, the object shown on the display screen is picked up. For example, if the user interface asks the user to select an object from multiple objects displayed on the display screen, the user points to the object and that object is selected.

The user can point to an object by simply pointing at the object displayed on the touch screen. When pointing to an object correctly, the user interface produces a "selectable object" sound. This selectable object sound provides feedback to the user, indicating that the user has pointed to the object correctly.

[Select]

Select gestures are very similar to point gestures. When the operator uses this select gesture, the object calls a method that performs a function or action corresponding to the object. For example, when you select a portal, the portal calls a method that moves the user to the space corresponding to that portal. Also, when a button is selected, the button calls a method that executes the action or function corresponding to that button.

To select an object, first touch the object with your finger and point to the object. If you keep pointing at this object for the specified minimum time, the object is selected. When the user selects an object, the object "informs" and produces a sound for feedback to the user. Then the method in this object is called. If the object is a button, the called method will perform the function corresponding to this button. If the object is a portal, the method moves the user to the space corresponding to the portal.

Alternatively, you can quickly point to an object twice in a row to select it on the display screen. When you point to an object twice quickly and select it, the object does not signal the operation, the user interface only produces the appropriate feedback sound, and the action corresponding to this object is performed. This allows the user to quickly select an object.

[Bread]

Use pan navigation gestures to move through the space. Place your finger on the background image of the space shown on the touch screen display 37 and slide your finger in the desired direction of movement to move the user through the space. The user interface emits a "pan-active" sound that responds by moving the background image of the space and all objects in the space in the opposite direction of the finger movement. As you move the space in the opposite direction, the user interface simulates the movement of the user's point of view in the opposite direction.

10 and 11 show examples of using pan gestures. The operator's finger is placed on the background in Figure 10, and moving this finger to the right on the touch screen moves the background image of the space and all objects in the space to the left, moving the screen to the right. Move to. Figure 11 shows the result of panning to the right, with the house 48 moving from right to left on the display screen.

The background image of each space can be enlarged arbitrarily. If the background image of the space is finite in size, the user interface will give feedback as if it had arrived at the farthest point in the background image. If the user pans all the way to the edge of the background image, the pan gesture does not move the user further in that direction. To inform the user of this fact, the user interface will generate an "edge" sound instead of a "pan active" sound if you try to pan beyond the edge of the background image. With this "edge" sound, the user knows that he is already at the edge of the background image and cannot go any further.

[spin]

The user browses multiple objects on the object wheel using spin gestures. When you spin the object wheel, the objects visible on the object wheel traverse the display screen and disappear at one end of the screen. Then, an object that was previously invisible from the other end appears on the screen.

Place your finger on the object wheel and display it in a direction parallel to the direction of movement of the object wheel. When you cross the screen with your finger, the object wheel spins. Figure 21 shows a spin gesture. In FIG. 21, the user first places his finger on the object wheel (indicated by the star 96). Then move your finger across the object wheel (as indicated by arrow 97). This finger gesture causes the object wheel to spin in the direction opposite to the finger movement direction. If you finger across the display's object wheel at a faster speed, the object wheel will spin at a faster speed. Similar to the numbers that appear on the slot machine wheel, once the object wheel spins, the object on the object wheel appears at one end of the object wheel, traverses the display screen, and disappears from the other end of the object wheel.

[pick up]

Use pickup gestures to pick up objects placed in space or on the object wheel so that you can carry objects throughout the "world" created in the user interface. In this embodiment, various methods are used to pick up an object from a space or an object wheel. Each method will be described later individually. It must be understood here that objects in two situations can be picked up with the same gesture.

Figure 22 shows the gestures used to pick up an object from the object wheel. FIG. 22 shows an object wheel 91 with a large number of fruits for selection. To retrieve one of the fruit objects located on the object wheel 91, first place your finger on the object (as indicated by the star 92) and then in the direction perpendicular to the direction of movement of the object wheel. Move your finger to. Arrow 93 shows how the finger is moved in a direction perpendicular to the direction of movement of the object wheel. The object is stripped from the object wheel so that the user can move it by moving their finger vertically. When an object is stripped from the object wheel, the user interface produces a "tear" sound that signals that the object has been removed from the object wheel. When this object is picked up, it creates a shadow on the background image of the space and moves it with the user's finger.

Figure 23 illustrates the gestures used to pick up an object from a space. In FIG. 23, the space with the objects is drawn in the center. To pick up an object, use your finger to draw a circle around the object, as indicated by arrow 203, and then touch the object with star 201. When an object leaves the space, the user interface makes a tearing sound to let the user know that the object has been picked up. When an object is picked up from space, a ghost image remains where it originally was. The ghost image of the object is drawn in desaturated color, informing the user that the ghost object cannot be selected.

When the user picks up an object, either the object wheel or the space emits an "object-carrying" sound, which indicates that the object is being carried. Users can move objects in or out of space while carrying them, and even drop the objects they bring on top of another object.

[drop]

The last gesture used to interact with the user interface is drop. After picking up the object from the space or object wheel using the pick-up gesture, keep pressing your finger on the screen to carry the object. When you slide your finger across the touch screen display system, the objects you carry move. After moving the carried object to the desired point, the drop gesture is used to drop the object to the desired point. To drop an object, take your finger off the touch screen display. When you drop an object, it falls into the background image of the space as if it were being pulled by gravity, and the shadow of the object disappears.

This drop gesture can be used to perform a number of different functions. For example, if you pick up an object and drop it into a portal where you can copy it in the space corresponding to the portal, you can make a copy of the object with a drop gesture. You can also pick up an object, drop it at the desired location, and use the drop gesture to move the object to a new location. In addition, drop gestures can be used to give a data object containing certain information to another object that uses that information to perform a function. Different uses of drop gestures will be described later.

To make a copy of an object using the drop gesture, first pick up the object and then drop the object onto the portal. When you cross the screen with your finger, the picked up object is carried over the portal. Once the brought object is on the portal, the portal is highlighted and the user is given feedback to signal that the brought object is on the portal. In this embodiment, the portal emits a green light. To drop the carried object into the portal, take your finger off the touch screen display. The object is then displayed to pass through the portal with moderate movement. During the duration of the video, CPU30 produces a "sucked into the portal" sound. When the object disappears completely into the portal, the ghost image of the object, which is drawn in desaturated color in the current space, is displayed again in bright color. This gives the user a visual feedback that a copy of this object was sent through the portal to another space, but the original object still exists in the current space.

Objects can be moved within one space, and can be moved from one space to another. To move an object within a space, first pick up the object using a pickup gesture, move the object to a desired location, and drop the object using a drop gesture. When an object falls, the object's ghost image blends with the falling object in the new location, and one object image appears in bright colors in the new location. This moves the object to a new location and informs the user that no copy was made.

When you move an object to another space through the portal, the object moves to another space. To move an object to another space, first pick it up and then carry it onto the portal. Once the object is brought to the top of the portal, the portal is highlighted for feedback to the user, the object is signaled to be brought to the top of the portal, and the portal glows green. To allow the carried objects to pass through the portal, keep the objects on the portal for a specified minimum time. When the carried object and user move to the space corresponding to the portal, the portal notifies the move. The ghost image of the transported object, which is drawn in desaturated color, also moves to the destination space together with the user and the object. When the transported object is dropped by releasing the finger from the touch screen display system, the transported object and the ghost image are merged into one image and displayed in bright colors. This gives the user feedback that the object has moved to this space but has not been copied.

You can also use this drop gesture to give information to an object. That is, information is given to an object by dropping an object containing data (that is, a data object) onto an object that uses this data.

To give information to an object, first pick up a data object that contains the necessary information. It then brings the data object onto the objects that can use this data. When a data object is brought over an object that can use this information, the object that can use the information is highlighted if the carried object is of the correct class. The highlighting gives the user feedback from the information-available object that the object being carried is of the correct class. To drop a data object into an object for which information is available, simply lift the finger used to carry the data object off the touch screen. The data object then falls into the portal with the appropriate movements and sound effects. When the data object disappears completely among the receiving objects for which information is available, the receiving object performs an appropriate function based on the given data.

For example, a drop gesture can be used to send television program information to a videocassette recorder object that corresponds to a physical videocassette recorder, which is programmed to record the television program. To program a VCR, first use a pickup gesture to pick up a data object containing TV program information. Next, the TV program data object is brought onto the video cassette recorder. The videocassette recorder object then changes its appearance and displays that it is in "programmed videocassette recorder" mode. This allows the videocassette recorder to "understand" that the program information is on the videocassette recorder (ie, when the TV program data object is dropped onto the videocassette recorder, the videocassette recorder records the TV program. The user is informed that it will be programmed to do so.

When a TV program object is dropped into the video cassette recorder, the video cassette recorder object recognizes that the TV program data object contains information available in the video cassette recorder object. The videocassette recorder extracts time and channel information from this TV program data object. This causes the videocassette recorder object to be programmed to record television programs.

[Example of operating tour]

To help you get the most out of the user interface, Figures 9-16 show a "tour" in the sample environment. The sample environment explored on this tour is based on the geographic map structure of the space illustrated in Figures 7 and 8. This tour of the sample environment is not intended to limit the scope of the invention of the present application, but is given as an example to know what is possible.

In Figure 9, we first enter the user interface. The user interface then displays the user in outer space 99. This space is shown in the geographic map structure of the multiple spaces in Figure 7. Initially the user appears in outer space 99, but when the user ceases to use the handheld display, the next user interface begins in the space where the user was last. In outer space 99 of FIG. 9, the earth portal 100 is displayed in bright colors in the black outer space background image on the display screen by the user interface.

As soon as you enter the user interface, a hand 31 will appear on the display screen to inform the user how to navigate within the user interface. The hand 31 that appears on the display screen is the agent's hand. When the agent's hand 31 extends his finger to touch the screen and crosses the screen with this finger, the space and objects on the screen are moved and displayed, and the user interface emits a "panactive" sound. In Figure 9, the agent's hand 31 appears to be traversing the screen. By touching the display screen and traversing the display screen with a finger, the agent's hand 31 explains to the first user how to use the pan gesture to move on the space screen.

After explaining how to use the pan gesture to move around in the space, the agent's hand 31 explains how to use the point and select gestures. The agent actually puts his finger on the Earth Portal 100 to show how to use the point gesture to point to an object. When the agent points to Earth Portal 100, a "selectable object" sound is emitted from the user interface, giving feedback to inform the user that Earth Portal 100 is a selectable object. Next, the agent explains how to use the object select gesture to select an object by continuously pointing to the earth portal 100. When the agent hands 31 to point to the Earth Portal 100 for the specified minimum time, the Earth Portal 100 notifies the portal passage and the CPU generates a "portal passage" sound. When the Earth Portal 100 notifies the user that the portal has passed by rapidly expanding as if the Earth Portal 100 is approaching the user, the screen fades and the Earth surface space corresponding to the Earth Portal 100 as shown in Fig. 10 A new background image showing 32 appears.

In Earth surface space 32, the agent portal 41 "jumps out" into the touch screen 37 and moves to the lower left corner of the display screen. This agent appears in any space except the agent space, so you can access agent 41 at any time. If desired, the agent can be moved from the lower left corner to another location on the display screen. Agents can be moved, so you can change the location of agents in any space.

The wayback portal 45 is displayed in the lower right corner of the display in FIG. The Wayback Portal 45 shows a framed snapshot of Space99 in Figure 9, which has just moved. With the Wayback Portal 45, the user interface allows you to go back to the space displayed inside the Wayback Portal 45. For example, when you want to return to the outer space 99 displayed on the wayback portal 45, you can select the wayback portal 45.

The earth surface space 32 in FIG. 10 is composed of a plurality of buildings such as houses in addition to public buildings such as shops, banks, and schools. Each building in the Earth surface space 32 represents a portal, and you can select this portal to enter the space corresponding to the portal. For example, in the Earth surface space of FIG. 10, the first house 47 and the second house 48 are portals, and the user can select these portals. When the user selects the first house 47 or the second house 48, the user moves to the corresponding house space.

Within the Earth surface space of FIG. 10, the user can move within the space by using pan gestures on the touch screen as described by the agent's hand 31 of FIG. To move, place the user's finger on the screen and then move this finger in the direction in which it was moved. The user interface responds to the movement of the user's finger, producing a "panactive" sound, and moving the background image and object on the screen in the opposite direction of the finger movement to make another object visible.

For example, when the user looking at the Earth surface space 32 in FIG. 10 moves to the right, the user only has to first place his finger on the background of the Earth surface space 32 and then move this finger to the right. The user interface moves the background image of the space and the objects in the space to the left in response to the movement of the user's finger. The Earth surface space 32 after placing the user's finger on the background of FIG. 10 and moving the finger to the right is shown in FIG. The first house 47 shown in FIG. 10 is not visible in FIG. 11 because it is out of sight. The second house 48, shown on the right side in Figure 10, has moved to the left side in Figure 11, and only part of it is visible. As shown in FIG. 11, the bank portal 49, which was not visible in FIG. 10, is coming into view from the right side. Agent 41 and Wayback Portal 45 both remain in the same position in the lower left and lower right corners.

When a portal is selected from the Earth surface space 32 in FIGS. 10 and 11, the selected portal makes a prediction and gives feedback to the user, and the user interface moves the user to the space corresponding to the selected portal. For example, if a user in Earth surface space 32 in FIG. 10 selects the first home portal 47, the user interface will notify the move to the home portal 47 and move the user to the home space corresponding to the home portal 47. .. FIG. 12 shows the home space 50 corresponding to the home portal 47 of FIG.

The house space 50 in FIG. 12 has the outer wall removed so that each room in the house can be seen. The rooms in the house, namely the living room 54, the bedroom 53, the bath 55, the attic 52, and the kitchen 56, are all selectable by the user. As shown in Figure 12, a small snapshot of Earth's surface space 32 is displayed inside the Wayback Portal 45, as it was seen when leaving the previous space. Thus, the wayback portal in Figure 12 not only displays a small snapshot of the Earth's surface space, but also the portion of the Earth's surface space 32 that was visible when the user selected Home Portal 47 in Figure 10. Has been done. To return to the Earth surface space in Figure 10, select Wayback Portal 45 with a snapshot of the Earth surface space.

In FIG. 12, when the living room 54 is selected, the CPU notifies the movement to the living room portal 54 and generates a passing portal sound. Then, the user moves to the living room space corresponding to the living room portal 54 by the user interface. FIG. 13 shows an example of the living room space 60. In the living room 60 of FIG. 13, a television 73, a video cassette recorder 75, and a television program guide book 71 are placed. The TV 73, VCR 75, and TV program guide book 71 can be selected by the user on the portal. The TV 73 and VCR 75 are compatible with real TVs and VCRs, respectively.

When the TV portal 73 displayed in the living room space 60 of FIG. 13 is selected, the TV portal 73 notifies the power operation of the TV, and the graphic user interface corresponding to the real TV is displayed by the user interface. FIG. 14 shows a television control space 64, which is composed of the television graphic user interface of this embodiment. The objects available in this TV space 64 include a green "on" button 61, a red "off" button 62, and multiple TV channel buttons 66-69. The TV channel buttons 66-69 are arranged side by side on the surface of the object wheel 63 so that the object wheel 63 can be spun right or left to see another TV channel button that is not currently displayed on the display screen. Can be.

The "on" button 61, "off" button 62, and TV channel buttons 66-69 available in TV Space 64 are not portals and are not intended to access another space in the space map of Figure 2. However, these "on" buttons 61, "off" buttons 62, and TV channel buttons 66-69 are functional buttons. For example, the green "on" button 61 and the red "off" button 62 correspond to the "TV power on" and "TV power off" functions, respectively. When you select either the on or off button, it notifies the user that the button is pressed and gives feedback to the user that the button has been selected. The handheld display then sends an appropriate command to turn the television on or off. Similarly, if you select any one of the TV channel buttons 66-69, the handheld display will notify you that the TV channel button has been pressed and an appropriate command will be sent from the handheld display to the real TV to select the selected channel. Switch TV to. The process of sending commands to a physical television device is described in detail in the following chapters.

In FIG. 13, when the TV program guide book 71 is selected, the user is moved to the TV program guide space by the user interface, notifying that the TV program guide book 71 will be opened. FIG. 15 shows the book space of the television program guide of this embodiment. In the book space of the TV program guide in FIG. 15, the TV program list is displayed two-dimensionally. The TV channel is displayed vertically in this TV program list, and the time is displayed horizontally. Each TV program is a portal that users can select. Also, when a user carries a TV program list and drops it onto a videocassette portal, the videocassette recorder can be programmed with a handheld display to record the TV program.

The book space in the TV listings in Figure 15 is actually a pair of vertical object wheels connected to each other. FIG. 17 shows a conceptual diagram of the book space in the television program guide. Place the television channel of FIG. 15 on top of the first object wheel 72 of FIG. On the other hand, the TV program list is placed on the second object wheel 85. These two object wheels 72 and 85 are connected so that if you spin one of the object wheels, the other object wheel will spin at the same speed and the TV channel and TV show list will always be side by side. There is.

Each TV program list located on the object wheel 85 of the TV program guide is a portal. When any one of the TV program list portals is selected, the TV program list portal is pre-videoized, and the user moves to the TV program information space by the user interface. FIG. 16 shows a general television program information space. This TV program information space consists of a clock 81, a date 82, and a TV channel 83, and this information space allows the user to know when the TV program is turned on and on which channel the TV program is broadcast. As shown in FIG. 16, the TV program information space also includes articles on TV programs.

[Agent]

An agent is an anthropomorphic character designed to assist the user in various ways. This agent assists the user by helping, giving state information, alerting the user to certain events, and acting like a counselor. Each agent describes each function.

[Help] As already explained in the tour example in Figure 9, the user first enters the user interface. The agent then explains to the user how to use the points and select gestures. In addition, the agent will explain how to use pan gestures to move around in the space. Also, if necessary, you may explain how to use other functions of the user interface.

To further help the user, the agent's briefcase contains a help manual that the user can use at any time. Simply select the Agent Portal to access the help manual. The agent performs the operation of opening the briefcase by pre-animation. The user then selects a help manual from the briefcase.

The state and mode information agent can be used to know the current state (state, referred to herein) or mode of the user interface. Change the agent's outfit, appearance, or props they carry to inform them of their current state or mode.

For example, if the user interface is in diagnostic mode, the clothes the agent is wearing will change to let you know. In Figure 24, as you can see, the agent is dressed as a doctor, symbolizing that the user interface is in diagnostic mode.

Figure 25 shows another example of agent clothing that provides state information. When in the write state, the agent is holding a pencil and the user interface symbolizes that it is in the write state.

[Warning (alert)]

While using the handheld display device of the present invention, an event may occur that requires immediate notification to the user. Such an event (also called an event) is usually called a warning (also called an alert). For example, if a handheld display is connected to an electronic message service and receives a message that the user must be informed, it alerts the user.

The user interface can also use agents to get the user's attention to alert the user. For example, the agent can be waved to get the user's attention. If the warning is very important, you can not only wave the agent's hand, but also make a voice to get the user's attention.

For example, if an event occurs that requires the user to be alerted, the agent will begin waving, as shown in Figure 26. To explain what the reason for the warning is, it is possible to further change the appearance of the agent to provide information on a particular warning, as described in the previous chapter. Specifically, when an emergency message arrives, the agent can be waved by holding a telegram in one hand.

Alerts take precedence over state or mode information because alerts are more important than state or mode information. Therefore, if a significant event occurs while trying to convey state or mode information using the agent's image, the agent's image will change to alert the user and alert the user. Tell. Specifically, if an urgent message arrives while the agent is holding a pencil in write mode, as shown in Figure 25, the agent immediately changes to a waving video, as shown in Figure 26, and the message is sent to the user. Inform.

[Remote device control]

The handheld display is designed to control remote devices. Therefore, the user interface for the handheld display device described in the previous chapter is designed as an ideal graphic user interface for controlling the remote device. For example, to control a device in the user's living room, the user first moves to the living room space 60 of the user interface in FIG. Then select the object that corresponds to the device you want to control. Specifically, when the user controls the television, the television portal 73 corresponding to the actual television is selected. When the user selects TV Portal 73, the TV Portal 73 is pre-videod to display a graphical user interface that controls the actual TV in the user's living room. FIG. 14 is an example of a television control set for television control. The user controls the television using the television control set displayed on the handheld display.

In order for a handheld display to control a remote device, the handheld display must have reliable information about the remote device. That is, the handheld display must be provided with an object-oriented programming type object that corresponds to the remote device. This object is composed of multiple methods, including a method for displaying the object on the display screen of the handheld display device and a method for displaying the graphic user interface for the remote device.

In this embodiment, the required object-oriented programming type objects are automatically transmitted from each compatible remote device to the handheld display device in order to simplify the operation. When an object is transmitted from the remote device to the handheld display, the handheld display calls a method to display the object on the display screen. The user can move this object to any space. When the user selects an object, the selected object is pre-animated and the method that displays the graphic user interface is called. This method displays the graphic user interface of the remote device on the touch screen display of the handheld display. The user can control the remote device by interacting with the graphic user interface of the remote device displayed on the touch screen display.

The handheld display device of the present invention automatically receives a graphic user interface from two types of compatible remote devices. The first type of remote device is an intelligent remote device with a dedicated computer control system. Also, the second type of remote device is a simple remote device that responds only to simple coded commands.

The handheld display interacts with two compatible remote devices in different ways. For communication with the intelligent remote device, peer-to-peer communication with the remote call method for controlling the remote device is used. Moreover, only a simple control code is used to communicate with a simple remote device. However, since it is difficult for the user to distinguish between the intelligent remote device and the simple remote device, the graphic user interface and the interaction with the user are the same for both remote devices. The user does not know the implementation details such as whether the remote method call is made by the user's action or the control code is transmitted via some communication mechanism.

This chapter describes how to transmit information from two types of compatible remote devices to handheld display devices. In addition, a third type of remote device will be described. This third type of remote device consists of an electronic device that can be controlled by a coded infrared signal.

This chapter describes object-oriented programming type devices. Objects of the object-oriented programming type have built-in data and methods that perform their functions when called. For the sake of brevity, the word "program object" is used to represent an object-oriented programming type object. To avoid confusion, the objects displayed on the touch screen display of the user interface described in the user interface above are referred to as "display objects" in this chapter.

[Peer-to-peer communication with remote method calls]

As used herein, a remote device is defined as an "intelligent remote device" if it has a microprocessor, communication system, and sufficient memory to allow the remote device to call methods within a program object. Figure 3 shows the parts in the intelligent remote device in the form of a block diagram. The handheld display communicates peer-to-peer with an intelligent remote device.

The intelligent remote device has a corresponding graphic user interface composed of touch screen control elements such as dials, switches and sliders. The graphic user interface of the intelligent remote device is defined in the user interface program object 303 recorded in the memory 404 of the intelligent remote device. When the intelligent remote device is turned on first, the intelligent remote device automatically transmits the user interface program object 303 to the handheld display. Then, the handheld display device displays the user interface for the intelligent remote device defined by the intelligent program object.

When the user interacts with the graphic user interface of the intelligent remote device as displayed on the handheld display, the handheld display calls a method in the device driver program object 301 in the intelligent remote device to control the remote device. In this way, the software that actually "controls" the intelligent remote device is contained within the intelligent remote device itself.

A method of transmitting the user interface object 303 from the intelligent remote device to the handheld display device will be described with reference to the flowcharts of FIGS. 27 to 30 and 31 to 35. 27 to 30 show concretely how the intelligent remote device 150 and the handheld display device 170 communicate with each other. Further, FIGS. 31 to 34 conceptually illustrate how to communicate with the user interface program object 303 and the handheld display device 170 in the intelligent remote device 150.

In step 101 of FIG. 35, when a new intelligent remote device 150, such as the intelligent thermostat controller of FIG. 27, is first turned on, it is provided with a to export user interface program object via the communication network 160. To communicate. As explained in the hardware chapter, communication over the communication network 160 is high frequency, low power, cellular, infrared with conventional power lines as in the case of the Echelon (Echelon registered trademark) based LON (LON registered trademark) system. This can be done using a signal, an electronic signal, or the like. In step 102, the handheld display 170 receives an interface export message. The handheld display 170 sends a signal back to the new intelligent remote device 150 over the communication network 160, which requests the user interface object 303 to initiate transmission.

Next, as shown in FIG. 27, the intelligent remote device 150 transmits the user interface object 303 to the display device 170 as defined in step 103 of FIG. In FIG. 31, a pointer (indicated by a dotted line) is transmitted from the intelligent remote device 150 to the device driver object 301 in the intelligent remote device 150. The user interface program object 303 defines how the user interface for the intelligent remote device 150 is displayed on the display screen. The user interface program object 303 calls the methods in the device driver program object 301 to handle user interface interactions. The device driver program object 301 contains a method for controlling the actual intelligent remote device 150. In FIG. 32, the display device 170 now obtains a copy of the pointers of the intelligent device user interface program object 313 and the device driver program object 311.

Upon receiving the user interface program object 313, the handheld display device 170 calls a method in the user interface program object 313. This method displays the display objects in the current space as shown in step 104. The intelligent remote device 150 also registers with the remote name server connected to the communication network 160.

When the user selects a display object that corresponds to the new intelligent remote device 150, the handheld display device 170 is the user interface program object 313, as described in step 105 (the object containing the called method is bolded). Call the method in. The method called on the user interface program object 313 displays the graphic user interface of the remote intelligent device on the display screen 171 of the display device 170, as shown in FIG. As mentioned in step 106, the user can interact with the graphic user interface of the remote intelligent device displayed above the display device 170.

The user controls the intelligent remote device 150 by interacting with the graphic user interface displayed above the handheld display device 170. Resolve user interaction Therefore, the user interface program object 313 requests the control method in the device driver program object 301 of the intelligent remote device. To access the control methods in the device driver program object 301 of the intelligent remote device, the handheld display 170 uses the device driver program object pointer 311. This device driver program object pointer 311 calls the method in the actual device driver program object 301. The device driver program object 301 of the intelligent remote device is called when sending a message over the communication network 160, as shown in FIGS. 29 and 33 (the object containing the called method is bold). As described in step 108, the method called in the device driver program object 301 of the intelligent remote device controls the intelligent device 150.

Finally, as shown in FIGS. 30 and 34, step 109 sends some return value or feedback generated by the device driver program object 301 in the intelligent remote device 150 back to the handheld display device 170 via the communication network 160.

When transmitting remote method calls between the handheld display device 170 and the program remote device 150, a robust communication protocol is required in the communication network 160. Both the handheld display 170 requesting a method call and the intelligent remote device 150 executing this method must be able to handle the corrupted communication path. For this reason, both the handheld display device 170 and the intelligent remote device 150 are provided with a dedicated remote procedure call (RPC) manager that handles remote method calls. In FIGS. 31-34, the remote procedure call manager 309 transmits a remote method call request for the handheld display device 170. Meanwhile, the server remote procedure call manager 307 handles incoming remote method calls within the intelligent remote device 150.

Figures 36 and 37 show diagrams used to control a remote procedure call (RPC) manager that handles remote method calls on both the handheld display 170 side and the remote device side. The user interface program object 313 in the handheld display device 170 requests a remote method call to the device driver program object 301 in the remote intelligent device. In FIG. 36, when the user interface object requests a remote method call, the RPC manager of the handheld display moves to the "transmission standby" state 403. Next, the RPC manager of the display device sets an error countdown timer and transmits the packet containing the message together with the method selector and the corresponding data. When the packet transmission is complete, the display device's RPC manager moves to the "waiting for response" state 405. When the appropriate response message is sent back, the method call is complete and the response message is analyzed in state 407.

Remote method calls are not always easy to complete. If the device driver program object 301 in the intelligent remote device 150 needs more time to complete the method, a more time-requesting message is sent back. The RPC manager of the display device responds when it moves to the state 409 where the timeout value increases. If the display device's RPC manager fails to receive a response by the time the error countdown timer expires, the display device's RPC manager transitions to "error recovery" state 411. If the error countdown timer ends less than a predetermined number of times, the RPC manager of the display device returns to the "transmission standby" state 403. When the error countdown timer reaches the predetermined limit, the RPC manager of the display device abandons the remote method call and shifts to the failure state 413. At this time, the user is notified of the communication abnormality.

FIG. 37 illustrates a state machine for the remote procedure call manager 307 in an intelligent remote device that handles remote method calls on the remote device side. The RPC manager 307 of this remote device starts from the "waiting for reception" state 451. When a packet is received, the remote RPC manager analyzes the packet in state 453. If the received packet contains a new message requesting a method call, the remote RPC manager sends the packet to the object executing the method request in state 455. The remote RPC manager then transmits the result and sends it back to standby 451.

If the received packet contains a message that the remote device has already viewed, the intelligent remote device's RPC manager moves to state 457. In this state, the intelligent remote device determines whether the method call request has already been completed or is still executing the request. If the intelligent remote device has already terminated the request, the subsystem retransmits the result of state 459 and sends it back to standby state 451. Alternatively, if the intelligent remote device is still executing its method call, the remote device's RPC manager transitions to state 461. In this state, the remote device's RPC manager requests more time to complete the called method and sends it back to standby state 451.

FIG. 38 shows the sequence of remote method calls (referred to as sequence below) when there is no communication error. In step 501, a user interface object in the handheld display requests a remote method call. Therefore, the user interface object sends a message to the display's RPC manager along with the method selector and arguments. The RPC manager of the display device packs the received message into a packet in step 502 and transmits the packet over the communication network in step 503.

The RPC manager of the intelligent remote device receives the transmitted packet in step 504. In step 505, the remote device's RPC manager opens the packet, receives the message, and sends the packet to the device driver program object in the intelligent remote device. In step 506, the device driver program object calls the method specified by the method selector in the message. The device driver program object generates a return message that is sent back to the remote device's RPC manager. In step 507, the remote device's RPC manager packs the return message into the response buffer. In step 508, the remote device's RPC manager sends out its response buffer.

In step 509, the RPC manager of the display device receives the response buffer, and in step 510, the response buffer is opened and the returned value is retrieved. The RPC manager of the display device sends the return value of the method in the device driver program object to the calling requester user interface program object. Finally, in step 511, the user interface program object analyzes its return value.

As described above, the intelligent remote device can be controlled by calling a method in the device driver program object in the intelligent remote device from a remote location. It can also be seen that the remote device can also call the method in the display device from a remote location. For example, a device driver object in a remote device can call a method in a user interface object in a display device to display a message.

Many remote devices are not sophisticated enough to have internal control methods that can be called from the display device. Therefore, the present embodiment also provides a method of controlling a simple device.

[Simple device control]

In the present specification, in the case of a remote device having no built-in microprocessor or a remote device having a built-in processor that is so primitive that the display device cannot call and control a method in the remote device, such a remote device is referred to as a "simple device". Is defined. Figure 4 shows the parts in a simple remote device in the form of a block diagram. To control a simple device, instead of calling and controlling a method in a simple remote device, a simple predefined control code is transmitted from the handheld display. Therefore, in order to control a simple controller, the handheld display must receive both a user interface program object and a device driver program object from this simple remote device.

An example of receiving both a user interface program object and a device driver program object from a simple remote device will be described with reference to the flowcharts of FIGS. 39 to 42, 43 to 46, and 47. 39 to 42 specifically show how the remote device 155 and the handheld display device 170 of the present invention are communicating with each other. FIGS. 43 to 46 conceptually illustrate how the user interface program object 353 and the device driver program object 351 of the remote external device 150 are transmitted to the handheld display device 170 and used.

In FIG. 39, the thermostat is shown as a simple remote device 155. Both the user interface program object and the device driver program object are transmitted from the simple remote device 155 of FIG. 39 to the handheld display device 170 as specified in step 124 of FIG. As conceptually shown in FIG. 43, the simple device 155 transmits the user interface program object 353 and the device driver program object 351. The transmission of the program object is performed using the communication means 162, which is some means for transferring information to the communication network or the display device 170.

The handheld display device 170 receives a copy of the user interface program object 363 that describes the graphic user interface of a simple device. The handheld display device 170 then calls a method in the user interface program object 363 that displays the object in space on the handheld display device 170. If a display object corresponding to the simple remote device 155 is selected, the handheld display device 170 calls the methods in the user interface program object 363, as shown in steps 125 and 40 of FIG. The method called by the user interface program object 363 displays the graphic user interface corresponding to a simple remote device on the display screen 171 of the display device 170. Then, in step 127, the user can interact with a simple remote device graphic user interface on the display device 170.

Simple device graphics The user interface and user interaction are analyzed by the simple device device driver program object 361 inside the display device 170. The device driver program object 361 resolves the user interaction into a control code that a simple remote device 155 can respond to. As shown in FIG. 45, the device driver program object 361 inside the display device 170 analyzes the user's behavior.

In FIGS. 41 and 45, the handheld display device 170 transmits the control code to a simple remote device 155. In step 130, the simple remote device 155 changes in response to this control coat received. Finally, if the simple remote device 155 and the display device 170 are connected to each other by a communication network, any return value or feedback generated by the simple remote device 155 will be from the simple remote device 155 as shown in FIG. Can be transmitted.

[Control of conventional equipment]

Conventional remote devices not designed for the handheld display device of this embodiment do not have a user interface program object to transmit to the handheld display device. For example, FIG. 5 shows a typical conventional remote device that can only respond to coded command signals such as infrared signals. In this case, the remote device must be used to manually send information to the handheld display. After this, the handheld display uses a generic user interface program object for common remote devices.

For example, in the case of a conventional television controlled by an infrared signal, the television model of the television is input and information about the conventional television is input to the handheld display device. The handheld display accesses a database and brings a list of control codes that can be understood on a conventional television. You then create a television display object to control this traditional television. When the user selects a television display object, the handheld display calls a method that displays the generic television user interface. When the user interacts with this generic television user interface, the display device sends an appropriate infrared control code to control the television.

[Summary of Embodiment]

So far, the intuitive graphic user interface based on the graphic map structure and the system for controlling the remote external electronic device have been described. In the user interface of this embodiment, each space of the geographical map structure is displayed as a space on the touch screen display. Each space has a colored icon called an "object" that can be selected and manipulated by the user. Selecting a certain object, called a portal, moves the user from one space to another. Selecting another object, called a button, performs the action or function corresponding to this object. The handheld display displays a user interface, which can control remote electronic devices. From each remote electronic device, a user interface program object that defines a graphic user interface is transmitted to the display device. When the user selects the icon corresponding to the remote device on the touch screen display of the display device, the graphic user interface corresponding to the remote device is displayed on the handheld display device.

[Summary]

In addition to the inventions described in the claims, the following technical ideas are disclosed in the present specification.

(1) A method of executing a graphic user interface on a computer system consisting of a CPU, memory, and graphic display screen, with steps to generate multiple spaces each composed of graphic background images, and multiple portals (gates). From the graphic icon so that the space where the portal exists and the corresponding target space are connected by the portal, and the step of arranging the space connected to the in the indicated graphic structure (geographical map structure). A corresponding target space is provided for each of the portals, and a step of arranging the plurality of portals inside the space, a graphic background image of the first space provided with at least the first portal, and the first Receives from the user the input consisting of the step of displaying the portals arranged in the space on the graphic display screen and one portal selected from the plurality of portals arranged inside the first space. It has a step of displaying a graphic background image of the target space corresponding to the selected portal on the graphic display screen, and is instructed by selecting the portal in the space. A method characterized in that the user can move between spaces in a graphic structure.

(2) A method of displaying a plurality of objects for which a user selects an object on a display screen corresponding to a computer system, in which a step of arranging a plurality of objects in a circular object list and the circular object are described. It is a step of displaying the first subset of the objects in the object list as the first subset of the graphic objects on the display screen, and the graphic objects displayed on the display screen are arranged along the first axis. However, the first subset display step and the second subset graphic object on the display screen correspond to the second subset of the objects in the circular object list, and the first subset graphic object is on the graphic object. Rotate the circular object list so that when the user slides the instruction device in a direction substantially parallel to the axis of, the first subset of the graphic objects on the display screen replaces the second subset graphic objects on the display screen. It is characterized in that the user can select any one object from the plurality of objects by rotating the circular object list until a desired object appears on the display screen. Method.

(3) The graphic user interface is composed of a plurality of spaces arranged in the designated graphic structure (geographical map structure), each space is composed of a graphic background image, and the space is a plurality of portals. Connected to the indicated graphic structure by, and the portal circulates along the indicated graphic structure in the computer system graphic user interface where the first space and the second space are connected, respectively. In the method of recording the route, the step of displaying the graphic background image corresponding to the first space consisting of at least one portal corresponding to the target space on the display screen and the step of displaying in each space of the plurality of spaces are displayed. A step to generate a wayback portal consisting of a stack of space pointers pointing to the space visited last time, and a step to receive input from the user consisting of one portal selected from the plurality of portals in the first space or the wayback portal. And if the user selects a portal, the user interface pushes a pointer to the space corresponding to the selected portal onto the wayback stack, and the space graphic corresponding to the selected portal. To display the background image on the display screen, or if the user selects the wayback portal, pop the space pointer from the top of the wayback portal and point to it with the space pointer. A method comprising processing the input received from the user so that the space is displayed on the graphic display screen.

(4) Graphic user interface and versatility In a computer system having a video icon, the icon is an anthropomorphic graphic character displayed on a display screen and a means for animating the anthropomorphic graphic character to attract the user's attention. A system characterized in that it is composed of means for transmitting state information by changing the visual appearance of the anthropomorphic character.

(5) The graphic user interface is composed of a plurality of graphic objects displayed on the display screen, and is a touch screen that detects a finger placed on the screen and determines the position of the finger on the screen. The display screen comprises the steps of displaying a graphic object defining an area on the touch screen on the screen in a way that allows the user graphic object to be grabbed within the computer graphic user interface in which it is configured. A step of placing a finger near the graphic object displayed on the touch screen, a step of drawing a circle with the finger around the graphic object displayed on the touch screen, and a step of displaying on the touch screen. It has a step of touching the graphic object and a step of displaying the graphic object at a position on the screen defined by the finger on the touch screen as determined by the touch screen. A method characterized in that the user can select the graphic object by drawing a circle around the graphic object with a finger.

(6) A device that controls at least one remote device, which is connected to a central arithmetic processing unit (CPU), a memory connected to the central arithmetic processing unit, and the central arithmetic processing unit, and is a graphic image. A display screen that displays the above, a communication means that is connected to the central arithmetic processing unit and sends and receives a message via a communication network, and a user interface that is transmitted by a remote device and defines a user interface for the remote device. A means for receiving an object via the communication means, a means for calling a user interface method for displaying a graphic user interface corresponding to the remote device on the display screen from the user interface program object, and a means on the display screen. A means for receiving an input guided by the graphic user interface corresponding to the remote device, and a device driver (drive) method for controlling the remote device, the device driver method corresponding to the remote device. It has means for calling in response to an input from a user, displays the user interface of the remote device on the display screen, and calls the device driver by interacting with the user interface. apparatus.

(7) A method of acquiring a graphic user interface from a remote device and controlling the remote device using a local display device connected to a communication network, wherein the remote device is connected to the communication network, and the above. A step of sending a signal from the remote device that it has a graphic user interface to export when connected to a communication network, a step of requesting the graphic user interface on the local display device, and the graphic user. From the user interface program object, the step of transmitting the graphic user interface program object having the method for displaying the interface to the local display device and the graphic user interface of the remote device to be displayed on the local display device. A method characterized by having a step of calling the method.

(8) A method of copying an object in the graphic user interface, in which the user inputs an input consisting of a step of displaying the first graphic image of the object at the first position on the graphic display screen and a command for copying the object. A step of displaying a ghost image of the object at the first position consisting of a copy of the first graphic image drawn in a blurred color on the display screen, and a copy of the object are arranged. A step of receiving an input consisting of a second position from a user, a step of displaying the first graphic image of the object at the second position, and a ghost image of the object being the first graphic image of the object. A method comprising the step of changing the ghost image of the object at the first position on the display screen so as to be the same, characterized in that two graphic images of the object are present.

(9) A method of moving an object in the graphic user interface, in which the user inputs an input consisting of a step of marking the first graphic image of the object at the first position on the graphic marking screen and a command for moving the object. A step of marking the ghost image of the object at the first position on the marking screen, which consists of a copy of the first graphic image of the object drawn in a blurred color, and a step of marking the object. A step of receiving an input consisting of a second position for arranging the second graphic image from the user, a step of marking the second graphic image of the object at the second position, the ghost image of the object, and the first It has a step of moving the ghost image of the object at the first position to the second position so that the second graphic image of the object at the second position melts, and one of the graphic images of the object. Is currently present in the second position.

(10) Displaying an object with a graphic icon This is a method of sending information to a program object in an object-oriented graphic user interface, in which a step of displaying a graphic image of a data object at a first position on a graphic display screen and the above data. A ghost image of the object at the first position consisting of a step of receiving input from the user consisting of a command to move the object and a copy of the first graphic image of the object drawn in a blurred color is displayed on the display screen. A step of receiving an input from a user consisting of a step of displaying and a second position for locating the data object, the program object is an input receiving step of being placed at the second position, and the program. A method comprising a step of receiving an input from a user, which is a command for supplying the data object to the object, characterized in that two graphic images of the object currently exist.

(11) A graphic user interface is displayed on the display screen, and the graphic user interface is composed of a plurality of spaces each consisting of a graphic background image, and the spaces are configured in a map structure in the graphic user interface. The portal has a corresponding target space and is composed of a graphic icon and a step of displaying a first space showing the current user's position in the map structure on the display screen. The space in which the portal exists and the corresponding target space are connected by the portal, and a plurality of the portals are placed in the first space so that the portal can obtain a place for the user to go. A method comprising a step of displaying and a step of displaying a wayback portal representing an image of the space the user has just left in the first space.

<figref num="1">Overall view of remote control function</figref><figref num="2">Overall view of the hardware used to implement a handheld display</figref><figref num="3">Block diagram of the intelligent remote device to be controlled</figref><figref num="4">Block diagram of a simple remote device to be controlled</figref><figref num="5">Block diagram of a conventional electronic device subject to remote control</figref><figref num="6">Schematic showing the structure of different geographic spaces</figref><figref num="7">Schematic showing a geographic sample map structure of the space used to provide the user interface</figref><figref num="8">Schematic showing a geographic sample map structure of the space used to provide the user interface</figref><figref num="9">Schematic diagram showing the outer space of the user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="10">Figure 1 of the Earth's surface space in the user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="11">Figure 2 of the Earth's surface space in the user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="12">Schematic of the first house space of the user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="13">Schematic diagram showing the living room space in the user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="14">Schematic diagram showing a television space in a user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="15">Schematic diagram showing the space of a TV listing book in a user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="16">Schematic diagram showing the space for one TV show article in a user interface based on the geographic map structure of Figures 7 and 8.</figref><figref num="17">Conceptual diagram of book space in the TV listings in Figure 15</figref><figref num="18">Schematic showing multiple objects not appearing on a single screen display</figref><figref num="19">Conceptual diagram showing multiple objects placed on the object wheel</figref><figref num="20">Conceptual diagram of the object wheel in Figure 19 when the object wheel rotates to show a new object</figref><figref num="21">Schematic showing the "spin gesture" used to rotate the object wheel</figref><figref num="22">Schematic showing the "pickup gesture" used to retrieve an object from the object wheel</figref><figref num="23">Schematic showing the "pickup gesture" used to retrieve an object from space</figref><figref num="24">Schematic showing an agent portal wearing a doctor's uniform to symbolize the diagnostic mode</figref><figref num="25">Schematic showing an agent portal with a pencil to symbolize the write mode</figref><figref num="26">Schematic showing an agent portal trying to alert users to recent events</figref><figref num="27">Schematic diagram showing an intelligent device that transmits a graphic interface to a display device</figref><figref num="28">Schematic diagram showing a display device that displays the graphical interface of an intelligently controllable device</figref><figref num="29">Schematic diagram showing a display device that transmits a user conversation to an intelligent device along with a user interface</figref><figref num="30">Schematic showing an intelligent device that transmits a return value or feedback to a display device</figref><figref num="31">Conceptual diagram of a program object in an intelligent device being transmitted to a display device</figref><figref num="32">Conceptual diagram of the program object of the intelligent device in the display device</figref><figref num="33">Conceptual diagram of a display device that calls a method in an object in a remote intelligent device</figref><figref num="34">Conceptual diagram of a method in an object in a remote intelligent device that returns status</figref><figref num="35">Flowchart of transfer of user interface of intelligent device</figref><figref num="36">State diagram of the RPC manager where the client makes remote method calls</figref><figref num="37">State diagram of the RPC manager where the server handles remote method calls</figref><figref num="38">Flow diagram showing the order of remote method calls</figref><figref num="39">Schematic showing a simple device that transfers a graphic user interface to a display device</figref><figref num="40">Graphical view of a simple controllable device Schematic showing a display device showing a user interface</figref><figref num="41">Schematic diagram showing a display device that transmits a simple control code to a simple device</figref><figref num="42">Schematic showing a simple device that transmits a return value or feedback to a display device</figref><figref num="43">Conceptual diagram of a program object of a simple device being transferred to a display device</figref><figref num="44">Conceptual diagram of a program object of a simple device in a display device</figref><figref num="45">Conceptual diagram of a display device that calls a method in a program object and transfers a control code to a simple device.</figref><figref num="46">Conceptual diagram of a simple remote device that sends back state information</figref><figref num="47">Flowchart of simple device user interface transfer</figref>

Code description

30, 403 CPU 32 Earth surface space 33, 404, 424 Memory 34 I / O unit 35, 401 Communication circuit 37 Touch screen display 41 Agent 42 Other input devices 45 Wayback portal 150 Intelligent remote device 155 Simple remote device 157 Electronic device 160 Communication medium 170 Handheld display 301, 351 Device driver object 303, 353 User interface object 420 Emitter 421, 450 Receiver 425, 455 Device control circuit