Synchronisation of the Force Feedback Behaviour of an Input Device with Image Switching

The present invention relates to an image operation system that displays a pointer in a screen image in a display and variably gives a reactive force to an input operation section of the pointing device according to the displayed location of the pointer.

International patent application <patcit id="pcit0001" dnum="WO9721160A"><text>WO 97/21160 A</text></patcit> discloses an apparatus for providing force feedback to a user operated object such as a joystick in conjunction with a graphical user interface displayed by a host computer. The host computer preferably synchronizes any appropriate visual, audio or other feedback related to the host application with the application of force feedback to the user object.

International patent application <patcit id="pcit0002" dnum="WO9939273A"><text>WO 99/39273 A</text></patcit> relates to the provision of force feedback to a user of a client computer receiving information such as a web page over a network such as the World Wide Web from a server machine. The user experiences physical force feedback through an interface device of the client machine with the force feedback being correlated to the web page objects. The web page may include force feedback information to provide authored force effects. Generic force effects can also be provided which are uniformly applied at the client machine to all web page objects of a particular type.

International patent application <patcit id="pcit0003" dnum="WO0133760A"><text>WO 01/33760 A</text></patcit> discloses a hybrid haptic feedback system in which a haptic feedback interface device in communication with a host computer includes a device microcontroller outputting force values. Some of the host system's processing is used to emulate a haptic feedback device with increased processing power. Haptic commands may instruct an emulation layer emulating the device to delay the output of a force effect by a designated time period or to perform other tasks.

A vehicle navigation system is taken as an example of image operation systems. A user can scroll the map displayed in a display or move a pointer or a button selection cursor in a menu screen by using a remote controller (pointing device) with an input operation section such as a joystick when using this type of vehicle navigation system. When a map scrolls on the display, the map is moved by a predetermined distance in the opposite direction on the instruction signal sent from the remote controller. When selecting a button with a button cursor, the cursor is moved by one button in the signaled direction. When the same signal is sent over a predetermined period of time, the map is scrolled in the same direction successively or the button cursor is moved to the next button successively.

That is, in the vehicle navigation system, a desired operation in a screen image can be achieved when the pointing device such as a remote controller communicates to the vehicle navigation system uni-directionally to send an operational status signal of the pointing device.

On the other hand, for the purpose of operability, a system that has a reactive force on its input operation section according to the display position of a pointer or the like by using a bilateral communication between the pointing device and the vehicle navigation system is becoming close to practical use. That is, a reactive force is applied to the input operation section of the pointing device for the ease of placing the pointer on top of a selection button and the like in the screen image. When the reactive force is introduced, objects in the screen image act according to the Newton's third law of motion, simulating a real world of gravity. For example, the reactive force, when decreasing and then increasing along an X-axis, is integrated as a 'potential' of a drawing force on the pointing device. As a result, the pointing device can comfortably be guided to the bottom of the 'potential' curve, even when arbitrarily operated by an operator. According to this method, the operator can easily place the pointer on top of the area of selection button in the screen image, once the minimum reactive force is placed in the target area.

However, when a bilateral communication like this system is used, the following problem of screen image replacement arises. That is, a reactive force data is formed on a screen image basis to be applied to the input operation section. This reactive force data is outputted from the navigation system to the pointing device according to the screen image to be displayed on a screen. On the other hand, when an image data to generate a screen image is formed in the navigation system and is outputted to the display device, generation of a screen image takes time and a period of generation time varies according to the amount of image data. Therefore, the reactive force being applied to the input operation section and the screen image being displayed does not match and may cause a sense of discomfort to an operator.

The present invention addresses the foregoing shortcomings in image operation systems and improve such systems. That is, this image operation system utilizes a bilateral communication between a pointing device and a system with an attached display to apply a reactive force to an input operation section of the pointing device, thereby minimizing a sense of discomfort felt by an operator.

The image operation system comprises a display portion that has a screen image, a pointing device with an input operation section that instructs the destination of pointer movements on the screen image with a reactive force based on the reactive force data that relates the reactive force with the location of the screen. The reactive force data is so formed and activated that, regardless of the period of time taken by generation of a screen image, the reactive force suits the condition of the screen image being displayed at the time of operation, and more specifically, the reactive force data being used is selected according to the type of screen image to be displayed.

There are a fixed type and an unfixed type of the screen image depending on the context of the operation. Thus the image operation system switches two types of reactive force data corresponding to the situation, that is, a generated one based on the screen image data and a default one.

In terms of the ease of operation, this invention provides an efficient way of directing a pointer in the screen image. The pointing device of the image control system not only has a functionality of indicating the location of the pointer in the screen image, but also has an enter key equivalent function. In addition, the pointing device has a suitable force feedback based on the reactive force data when selecting a button in the screen image, and that extremely facilitates the ease of use of the system.

While the appended claims set forth the features of the present invention with particularity, the invention together with its objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings, in which: <ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">FIG. 1</figref> shows a schematic block diagram of a display operation device according to an embodiment of the present invention;</li><li><figref idref="f0002">FIG. 2</figref> shows a block diagram of the internal structure of the pointing device according to the embodiment;</li><li><figref idref="f0003">FIG. 3</figref> shows a schematic diagram that depicts the movement of a joystick from the neutral position N by X and Y coordinates according to the embodiment;</li><li><figref idref="f0003">FIG. 4A</figref> shows an image diagram taken as an example of destination setting menu;</li><li><figref idref="f0003">FIG. 4B</figref> shows a reactive force characteristics taken along the 4B-4B line;</li><li><figref idref="f0004">FIG.5</figref> shows a functional flow chart of a reactive force data activation procedure in synchronization with the generation of a screen image when a screen image is replaced; and</li><li><figref idref="f0005">FIG. 6</figref> shows a time chart that depicts every step of display switching processes in time series.</li></ul>

As shown in <figref idref="f0001">FIG. 1</figref>, an image operation system according to this embodiment is comprised of a car navigation system (navigation ECU) 1, a display device 3 having a gateway (GW) processor 5, a pointing device 7, and an imaging device 9 having a camera and other devices that take pictures of rear view of a vehicle and the like. Further, the pointing device 7 is connected to the display device 3 and to the navigation ECU 1 via the GW processor 5 by using a dedicated line 11. The pointing device 7 can send an operation signal of its own and can also send the data from the navigation ECU 1. Also, the navigation ECU 1, the display device 3 and the imaging device 9 are all connected with each other through a vehicle LAN 13.

The navigation ECU 1 includes, although not shown, a location detection device, a map data input device, a voice input device, and a controller connected to these devices. Then, the navigation ECU 1 generates a 'vehicle and its surrounding road map' image data that includes an iconized present location of a vehicle. This data is derived from the location detection device and placed on top of the road map data that is formed by the map data input device. The generated image data is then outputted to the display device 3.

The navigation ECU 1 also generates a menu screen image data and outputs it to the display device 3. In this manner, while a menu screen image is outputted and displayed on the display device 3, an input to the menu screen, such as destination facility search, map display condition settings and map scroll condition settings, and the other functionalities of the navigation ECU 1, is accepted by selecting a corresponding button image on the display with the pointer. Upon selection of a menu item, the screen image is replaced with the next one. By iterating selection of a menu item by using the pointer, a desired functionality can be invoked. The navigation ECU 1 generates a various kinds of screen image data corresponding to each functionality and outputs it to the display device 3 in order.

The display device 3 is, for example, composed of a liquid crystal display. It receives screen image data outputted from the navigation ECU 1, and then displays a screen image based on the received screen image data. Also, this display device 3 receives the operation signal from the pointing device 7. The operation signal from the pointing device 7 includes a pointer location signal that indicates the location of the pointer in the screen of the display device 3, and an ON/OFF signal that determines the selection of a menu button and the like. In the display device 3, the display location of a pointer is moved according to the pointer location signal included in the operation signal. Further, the display device 3 receives the image data from the imaging device 9 such as a camera and the like, and the display device 3 generates and displays a screen image based on the image data.

Further, the display device 3 has the GW processor 5, and this GW processor 5 determines the destination of data, for example, determines if the destination of data outputted from the navigation ECU 1 is the display device 3 or the pointing device 7, and transfers the data accordingly as a gateway.

The pointing device 7 has a joystick as an input operation section, and, for example, is installed in the proximity of the console box inside the vehicle. <figref idref="f0002">FIG. 2</figref> shows a block diagram of internal structure of the pointing device 7. As shown in the figure, the pointing device 7 is composed of an X-axis position signal acquisition circuit 70, a Y-axis position signal acquisition circuit 71, A/D converters 72, 73, an X-axis actuator driving circuit 74, a Y-axis actuator driving circuit 75, D/A converters 76, 77, an X-axis reactive force actuator 78, a Y-axis reactive force actuator 79, and joystick 80. This joystick 80 is operable arbitrarily in two dimensional directions represented by an X-axis and a Y-axis.

The X-axis position signal acquisition circuit 70 acquires an analog signal that represents the amount of movement of the joystick 80 from the neutral point N in the X direction. The Y-axis position signal acquisition circuit 71 acquires an analog signal that represents the amount of movement of the joystick 80 from the neutral point N in the Y direction. For example, as shown in <figref idref="f0003">FIG. 3</figref>, an operational movement of joystick 80 is measured and acquired by the combination of displacement Xd in X-axis and displacement Yd in Y-axis.

The A/D converters 72, 73 convert the analog signal that represents the amount of movement in X and Y-axis directions to a digital signal, and outputs it to a controller 90.

The D/A converters 76, 77 receive a digital input signal, that is used to create the reactive force applied to the joystick 80 corresponding to the operation on it, from the controller 90 and convert it to the analog signal. The converted analog signal is outputted and sent to the X-axis actuator driving circuit 74 and the Y-axis actuator driving circuit 75 respectively.

The X-axis actuator driving circuit 74 and the Y-axis actuator driving circuit 75 provide driving power for the X-axis reactive force actuator 78 and the Y-axis reactive force actuator 79 based on the analog signal outputted from the D/A converters 76, 77. The X-axis reactive force actuator 78 and Y-axis reactive force actuator 79 generate power of resistance (reactive force) that forces the joystick 80 to return to the neutral position N corresponding to the provided driving power. A stepping motor is, for example, used as an actuator and the actuator is mechanically connected to the joystick 80.

The controller 90 calculates the pointer position displayed on the display device 3 after a transitional movement based on the positional differences of the joystick 80 on each axis derived from the A/D converters 72, 73. Then, the controller 90 calculates the suitable reactive force at the post-transition position to be applied to the joystick 80 based on the reactive force data that is outputted from the navigation ECU 1 and stored in a reactive force data memory 91 with reference to the screen image being displayed.

When the reactive force to be applied to the joystick 80 is calculated in this manner, the controller 90 outputs a digital signal representing the calculated reactive force to the D/A converters 76, 77. The reactive force data is, then, received by the controller 90 via a communication interface (I/F) 92, and stored in the reactive force data memory 91. This reactive force data memory 91 can store multiple reactive force data at the same time. The controller 90 determines which reactive force data among them to be applied to the joystick 80.

Further, the controller 90 sends the coordinate data of the pointer after a transitional movement and an ON/OFF signal from the enter key at the top, or in the proximity of the top, of the joystick 80 to the display device 3 as an operation signal. Then, the display device 3 moves the pointer to the location according to the coordinate data and transfers the operation signal to the navigation ECU 1. The operation signal is transferred to the navigation ECU 1, because of the following two reasons.

The first reason is, when the enter key is ON, the screen image has to be replaced, and thus an image data for the replaced screen image has to be generated in the navigation ECU 1. The second reason is, when the pointer is placed on one of the selection buttons in the screen image, the screen image data has to be updated so that the selection button under the pointer is highlighted.

The reactive force data is described using <figref idref="f0003">FIG. 4A, 4B</figref> as examples at this point. As shown in <figref idref="f0003">FIG. 4A</figref>, in the destination setting menu screen image, the display device 3 displays destination setting method selection buttons 30 to be chosen from. In this case, the reactive force data is set to draw the pointer 33 into the displayed image area of the selection buttons 30 when the pointer 33 approaches to the selection buttons 30. That is, a drawing reactive force area 35, that covers the selection button and its proximity as shown in <figref idref="f0003">FIG 4A</figref>, is set around each selection button 30. <figref idref="f0003">FIG. 4B</figref> shows changing state of the actual reactive force. In this figure, the reactive force data is described as in one dimensional change taken along the 4B - 4B line instead of two dimensional change in reality for the simplicity of description.

<figref idref="f0003">FIG. 4B</figref> shows the reactive force data setting, that is, when the pointer 33 is outside of the drawing reactive force area 35, the reactive force is at its maximum strength. The closer to the center of the selection button 30 the pointer 33 approaches, the weaker the reactive force becomes. Therefore, as already described in the BACKGROUND OF THE INVENTION section, the joystick 80 is easily operable to guide the pointer 33 approaching from the periphery of the selection button 30 to the center of it.

The characteristics of the present embodiment is described based on the functional flow chart of the image operation system in <figref idref="f0004">FIG. 5</figref>, that is, reactive force data synchronization process that activates the corresponding reactive force data of the screen image in synchronization with the generation of the replaced screen image under the screen image replacement instruction. This process is executed when, for example, as a result of selecting a selection button in a menu screen image, replacement of the present screen image with the search screen image, the search result screen image, or the like is instructed, by the pointing device 7. Also, replacement of the screen images can be instructed by using a touch switch or a mechanical switch that is built into the display device 3 other than the pointing device 7.

Firstly, in step S100, the navigation ECU 1 generates and outputs the image data of the replaced screen image. For example, a fixed format screen image such as a menu screen image can be retrieved and outputted from a formed and stored image data. Further, when a search result screen image is required, a screen image data is generated and outputted after a data search and retrieval with necessary processes such as decompression of the retrieved data.

In step S110, the navigation ECU 1 determines if generation of a screen image to be displayed completes before full reception of the reactive force data by the pointing device 7 because of the smallness of the size of the data. That is, in the step S110, whether generation of a screen image completes earlier than full reception of the reactive force data is determined based on the type and the amount of the screen image. The process of this step S110 starts when the type and the amount of the screen image data to be displayed is revealed, and is carried out in parallel with the processes in step S100 such as retrieval of the screen image data and decompression of the data.

If Yes in step S110, the navigation ECU 1 makes the pointing device 7 use the default reactive force data in step S120. Based on this instruction, the pointing device 7 replaces the type of the reactive force data to be activated from the one corresponding to the pre-replacement screen image to default one in step S130.

The default reactive force data may, in this case, have an even distribution of the reactive force over the entire screen image, or may have a different pattern of reactive force data in order to be applied to a simple screen image that can be generated in a short time. This default reactive force data is stored in the reactive force memory 91 in the pointing device 7 beforehand. Further, the reactive force data corresponding to the pre-replacement screen image may be used as the default data as it is.

In step S140, the navigation ECU 1 generates the reactive force data corresponding to the replaced screen image, and sends it with the screen image type or the amount of the screen image data to the pointing device 7. Moreover, the processes in the steps S110 to S140 are carried out in parallel with the process in step S100. In step S150, the pointing device 7 receives the reactive force data and the screen image type or the amount of the screen image, and the reactive force data is stored in the reactive force data memory 91.

In step S160, the pointing device 7 determines whether generation of a screen image is completed or not based on the received screen image type or the amount of the received image data. If "Yes", the process proceeds to step S170, and the pointing device 7 replaces the default reactive force data with the received reactive force data. On the other hand, if "No" in step S160, the process proceeds to step S180. In step S180, whether the received screen image is a fixed format screen image such as a menu screen or an unfixed format screen image such as a search result display screen image is determined.

Then, if the screen image is determined as a fixed format, the process proceeds to S190 and sets a delay time from the reception of the reactive force data to the activation of the reactive force data depending on the type of the fixed format screen image. The fixed format screen image has a predictable generation period of time, and the delay time is set to be in synchronization with completion of the screen image generation.

If, on the contrary, the screen image is determined as an unfixed format, the process proceeds to the step S200, and the delay time is set based on the amount of the image data. For example, when a search result screen image is displayed after searching for a destination facility, the number of facility that fulfills the search condition varies significantly. Thus generation time of the screen image is not predictable. Under the circumstances, an unfixed format screen image has the delay time that corresponds to the amount of the image data. However, both a fixed format screen image and an unfixed type screen image may have the delay time based on the amount of the image data.

In step S210, the pointing device 7 determines whether the delay time set in step S190 or S200 has passed or not. When elapsed time is considered to reach the set delay time, the process proceeds to step S220 and the reactive force data for the pre-replacement screen image is replaced with the reactive force data that corresponds to the replaced screen image.

The processes described above, when a screen image is replaced, can synchronize the activation time of the reactive force data with completion of the screen image generation, or at least can apply the reactive force based on the default reactive force data in case that the generation of a screen image completes earlier, a sense of discomfort to the operator can be minimized in the pointing device 7 operation.

Next, image replacement processes are described by using a time chart shown in <figref idref="f0005">FIG. 6</figref>. The time chart in <figref idref="f0005">FIG. 6</figref> shows an example of image replacement from a search condition screen image (a screen image A) to a search result display screen image (a screen image B).

As shown in <figref idref="f0005">FIG. 6</figref>, when an image replacement from the screen image A to the screen image B is indicated by the enter key by using the pointing device 7, an internal process of the pointing device 7 is carried out. The internal process ensures the input from the enter key by multiple samplings of the enter key state in a predetermined interval.

If all of the sampled states of the enter key are ON, the enter key operation is positively confirmed as ON. Then, the data signal of enter key ON state and the coordinates of the pointer location are sent to the display device 3, whereupon an internal ECU of the display device 3, for example, receives the data signal from the pointing device 7 in synchronization with the update cycle of screen image generation (refreshing of the screen image). Then, a screen image with the updated pointer location is generated based on the coordinate data of the pointer location in the received data signal. This prepared image data is used at the next update cycle of screen image generation to display a screen image with the updated pointer location, and, as a result, update of the pointer location completes.

The display device 3 sends the data signal from the pointing device 7 with gateway processes to the controller 90 of the navigation ECU 1 in synchronization with the screen image generation of updated pointer location. Then, the navigation ECU 1 retrieves the data signal from the pointing device 7 in synchronization with the above update cycle of screen image generation. An image data in order to generate a screen image with the highlighted selection button being selected by the enter key is formed (SW[software] key highlighting process). This image data at the next update cycle of screen image generation (refreshing of the screen image) generates a screen image on the display device 3, and that completes the screen image with the highlighted selection button.

The above processes are carried out under the instruction of the data signal sent from the pointing device 7 to the display device 3. According to these processes, the pointer will be displayed at the post-transition location in approximately 100 ms after the operation of the joystick 80, and the selected button will be highlighted in several dozen milliseconds after that.

When an image replacement instruction is received by the navigation ECU 1, it searches for a necessary data to generate a replaced screen image and then retrieves the searched data. Upon completion of the searched data retrieval, a screen image to be displayed is identified. Thus a reactive force (denoted as 'R-Force' in <figref idref="f0005">FIG. 6</figref>) data B corresponding to the screen image B is generated. Further, decompression (or expansion) of the retrieved image data is carried out in parallel with generation of the reactive force data B.

When the amount of the data to be displayed is large, the reactive force data B is received by the pointing device 7 before completion of decompression (or expansion) and generation (or preparation) of the image data B as shown in <figref idref="f0005">FIG. 6</figref>. That is, the reactive force data B, after being generated (or prepared), is transferred from the navigation ECU 1 to the display device 3. Then, it is received by the pointing device 7 after a gateway process at the GW processor 5. However, the reception of the reactive force data B might be earlier than the completion of screen image B generation in the display device 3 by couple of hundreds ms.

Therefore, the pointing device 7 uses the reactive force data A corresponding to the screen image A until generation of the screen image B completes even after receiving the reactive force data B corresponding to the screen image B. Then, when elapsed time reaches the delay time of the screen image B generation, the reactive force data A is replaced with the reactive force data B. In this manner, the pointing device 7 activates the reactive force data B in synchronization with generation of the screen image B. In addition, it retains the screen image A by the time of completion of generation of the screen image B. It further retains the reactive force data A corresponding to the screen image A until completion of generation of the screen image B. In this manner, the screen image B and the reactive force data B can be matched seamlessly so that sense of discomfort of an operator can be effectively decreased.

The screen image data and the reactive force data are formed and stored in advance for a fixed format screen image such as a menu screen. Therefore, screen image data and the reactive force data retrieval can be started in parallel upon instruction of the screen image replacement. Thus both of them are outputted when the retrieval completes.

While the invention has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made. In the above embodiment, when a map image is displayed in the navigation ECU 1, the pointer, for example, may be displayed at the center of the map image, and a reactive force may be applied to the joystick 80 of the pointing device 7 during map scroll. In this case, a reactive force data that has a lower reactive force while the pointer is on a road may be formed to guide the pointer to stay within the road, for example, or the reactive force may set to be proportional to the elevation of a map, or the reactive force may reflect the congested condition of a road in the mapped area.

Further, when not only the navigation ECU 1 but also the imaging device such as other camera generates an image data, the reactive force may be adjusted according to an intended rule while the pointer is displayed in the screen image.