Tactile sense providing device, program used for tactile sense providing device and tactile sense providing method

The present invention relates to a device that exhibits a tactile sensation and a program used in the device. More specifically, the present invention relates to a device that presents a tactile sensation based on a touch operation on a touch sensor, a program used in such a device, and a method of presenting a tactile sensation in the device.

In recent years, some mobile terminals such as mobile phones have adopted an input device including a touch sensor such as a touch panel or a touch switch as an input device such as an operation unit or a switch for detecting an operation by a user. In addition to mobile terminals, input devices equipped with touch sensors are widely used in information devices such as calculators and ticket vending machines, home appliances such as microwave ovens, televisions, and lighting equipment, and industrial devices (FA devices). ..

Various types of such touch sensors are known, such as a resistance film type, a capacitance type, and an optical type. However, any type of touch sensor detects contact with a finger or a stylus pen, and the touch sensor itself does not physically displace like a push button switch even if it is touched.

Therefore, even if the operation by contact is recognized by the input device, the operator cannot obtain feedback on the operation. For this reason, the operator does not get a feeling of operation when performing the operation, and repeated operations such as touching the same position many times are likely to occur, which may give stress to the operator.

Therefore, conventionally, a feedback method has been proposed in which the touch sensor vibrates when the touch sensor detects contact, regardless of hearing or sight, to generate vibration at the fingertips of the operator (for example, patent documents). 1 and Patent Document 2).

<p><patcit num="1"><text>Japanese Unexamined Patent Publication No. 2003-288158</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 2008-130055</text></patcit></p>

<p> According to the techniques disclosed in Patent Document 1 and Patent Document 2, when an operation on the touch sensor is detected, vibration can be generated at the fingertip of the operator or the like. Therefore, an operator who performs an operation using such an input device can tactilely recognize that the operation performed on the touch sensor is correctly recognized by the input device.</p><p> However, in order to exhibit a tactile sensation when a contact operation is recognized using such an input device, a process of generating a predetermined vibration according to a predetermined condition is performed in advance by application software (hereinafter, simply "" Must be built into the application). That is, in the above-mentioned input device, if an application developed in advance assuming that it exhibits a tactile sensation is used, a predetermined vibration can be generated according to a predetermined condition. However, even if an application that is not supposed to exhibit a tactile sensation is applied to the above-mentioned input device as it is, it is not possible to appropriately generate a predetermined vibration.</p><p> Therefore, in order to appropriately present the tactile sensation in the input device as described above according to a predetermined condition, a new application is developed or a process for presenting the tactile sensation is added to the existing application. Need to be modified.</p><p> Specifically, when a contact is detected with a touch sensor corresponding to the position of an object such as a key or a button displayed as an image on the display unit of the input device, the input device exhibits a predetermined tactile sensation. Need to add processing. It is also necessary to prevent the input device from exhibiting a tactile sensation even if contact is detected with the touch sensor corresponding to the position where the object such as a key or a button is not displayed on the display unit. By controlling the presence or absence of the presentation of tactile sensation in this way, when the operator's operation is detected by the user interface using the touch sensor and GUI (Graphical User Interface) for input and output, the operator's fingertips and the like are vibrated. It can present a tactile sensation. That is, it is possible to notify by presenting a tactile sensation that the contact is detected by the contact with the object such as the key or the button displayed in the image. On the other hand, even if a contact with a position where there is no object such as a key or a button is detected, feedback can be prevented from occurring.</p><p> When performing such control, the input device is an application that manages the state of the GUI when contact with the touch sensor is detected, that is, the state of an object such as a key or button displayed on the display unit. It is necessary to cooperate.</p><p> As described above, in the conventional input device, even if an application that does not support the presentation of the tactile sensation is applied, the tactile sensation cannot be appropriately presented as it is. Therefore, in order to effectively utilize various application resources developed up to now in the conventional input device without wasting them, each application is modified to present a tactile sensation according to a predetermined condition. Processing must be added. Modifications that add such processing must be performed for each of various applications, and there is a concern that enormous labor and cost will be required. Further, by adding processing to the application in this way, there is a concern that the processing load of the application execution unit that executes the application increases.</p><p> Under these circumstances, it may be a factor that hinders the entry of vendors into the development of applications for input devices that present a tactile sensation, and also a factor that delays the spread of platforms that use touch sensors and GUIs. Can be.</p><p> Therefore, it is expected to reduce the man-hours for developing the application program used for the tactile sensation presenting device and reduce the processing load of the application execution unit built in the tactile sensation presenting device.</p><p> In general, conventionally, a device that generates vibration based on a contact with a touch sensor notifies an application execution unit of information on the position where the contact is detected each time an operator's contact operation is detected, and vibrates. Is configured to determine whether or not it is set to generate. Each time the application execution unit is notified of the information of the position where the contact is detected, it determines whether or not it is necessary to present the tactile sensation corresponding to the contact with respect to the position, and outputs the result of the determination. Then, if it is necessary to present the tactile sensation as a result of the determination, the tactile control unit controls the tactile sensation presenting unit so as to exhibit the tactile sensation instructed by the application execution unit.</p><p> Therefore, every time the display of the display unit that displays an object or the like is updated by changing the conventional configuration as described above, the condition that the application execution unit presents the tactile sensation corresponding to the display (that is, the tactile sensation setting). Is conceivable in a configuration in which is set in the tactile control unit in advance. In the tactile sensation presenting device having such a configuration, when the operator's contact operation is detected, the tactile sensation setting has already been completed, so that the tactile sensation control unit sets the tactile sensation without exchanging information with the application execution unit. It is possible to present a sensation of touch.</p><p> However, even if a configuration such as this tactile presentation device is adopted, it is necessary to modify the source code when using a conventional application. That is, even in the application used for the above-mentioned tactile sensation presenting device, it is necessary to add a process of setting the condition for presenting the tactile sensation according to the display in the tactile sensation control unit. Therefore, from the viewpoint of reducing the development man-hours of the application used for the touch sensor corresponding to the presentation of the tactile sensation, it may not always be sufficient.</p><p> Further, the application to which the process corresponding to the presentation of the tactile sensation is added in this way may become a dedicated application that can be used only by the device equipped with the touch sensor corresponding to the presentation of the tactile sensation. That is, once a process corresponding to the presentation of the tactile sensation is added to the application, the application can no longer be used in a device provided with a touch sensor of a type that does not present the tactile sensation.</p><p> Therefore, if the source code of the application can be used without being modified, it is extremely advantageous from the viewpoint of reducing the man-hours for developing the application. Further, if the same application can be used not only in a device equipped with a touch sensor corresponding to the presentation of tactile sensation but also in a device provided with a touch sensor of a type that does not exhibit tactile sensation, the versatility as an application can be enhanced. .. That is, if it is used for a device equipped with a touch sensor that supports the presentation of tactile sensation, it performs an operation of presenting an appropriate tactile sensation, and if it is used for a device equipped with a touch sensor that does not support the presentation of tactile sensation, other than the presentation of tactile sensation. An application that works normally is very advantageous.</p><p> An object of the present invention made in view of such circumstances is to provide a device that presents a tactile sensation in response to an input to an object by effectively utilizing the resources of an application constituting an existing user interface that does not support the presentation of the tactile sensation. To do. Another object of the present invention is to provide a highly versatile program used for the above-mentioned device. Furthermore, another object of the present invention is to provide a method of presenting a tactile sensation in the above-mentioned device.</p>

<p> The invention of the tactile presentation device according to the first aspect of achieving the above object is With a touch sensor A tactile sensation presenting portion that presents a tactile sensation to a contact object in contact with the touch surface of the touch sensor. Display and Based on the image data supplied from the application, the display unit is controlled to display a predetermined image, and the display unit is controlled to display a predetermined image. Based on the information related to the setting of the tactile sensation added to the data of the image, the tactile sensation presented by the tactile sensation presenting unit is controlled to be set according to the image displayed on the display unit, and the touch sensor displays the display. When a contact at a position corresponding to a predetermined image area displayed on the unit is detected, a control unit that controls the tactile sensation so as to exhibit the tactile sensation set in the image to the contact target, and a control unit. It is characterized by having.</p><p> The invention of the tactile presentation device according to the second aspect is With a touch sensor A tactile sensation presenting portion that presents a tactile sensation to a contact object in contact with the touch surface of the touch sensor. Display and A load detection unit that detects the pressing load on the touch surface of the touch sensor, and Based on the image data supplied from the application, the display unit is controlled to display a predetermined image, and the display unit is controlled to display a predetermined image. Based on the information related to the setting of the tactile sensation added to the data of the image, the tactile sensation presented by the tactile sensation presenting unit is controlled to be set according to the image displayed on the display unit, and the touch sensor controls the display. When the load detecting unit detects a contact at a position corresponding to a predetermined image area displayed on the image, the load detecting unit detects a pressing load satisfying a load reference that presents a tactile sensation set on the image. A control unit that controls the tactile sensation so as to exhibit the tactile sensation set in the image to the contact object, and a control unit that controls the tactile sensation. It is characterized by having.</p><p> Further, the invention of the program relating to the third viewpoint of achieving the above object is With a touch sensor A tactile sensation presenting portion that presents a tactile sensation to a contact object in contact with the touch surface of the touch sensor. Display and A program that constitutes an application that can be used in a tactile presentation device. A process of controlling the display unit to display a predetermined image based on the image data supplied by the application. Based on the information related to the setting of the tactile sensation added to the data of the image, the tactile sensation presented by the tactile sensation presenting unit is controlled to be set according to the image displayed on the display unit, and the touch sensor displays the display. When a contact at a position corresponding to a predetermined image area displayed on the unit is detected, a process of controlling the tactile sensation presenting portion so as to exhibit the tactile sensation set in the image to the contact target, and a process of controlling the tactile sensation presenting portion. Is characterized by executing.</p><p> Furthermore, the invention of the tactile sensation presentation method according to the fourth aspect of achieving the above object is With a touch sensor A tactile sensation presenting portion that presents a tactile sensation to a contact object in contact with the touch surface of the touch sensor. Display and It is a method of presenting a tactile sensation in a tactile sensation presenting device provided with. A step of controlling the display unit to display a predetermined image based on the image data supplied by the application, and A step of controlling to set the tactile sensation presented by the tactile sensation presenting unit according to the image displayed on the display unit based on the information related to the tactile sensation setting added to the image data. When the touch sensor detects a contact at a position corresponding to a predetermined image region displayed on the display unit, the tactile sensation display unit is controlled so as to exhibit the tactile sensation set in the image to the contact target. Steps to do and It is characterized by having.</p>

<p> According to the present invention, it is possible to provide a highly versatile application.</p>

<figref num="1">It is a block diagram which shows the schematic structure of the tactile sensation presenting apparatus which concerns on embodiment of this invention.</figref><figref num="2">It is a figure which shows the example of the mounting structure of the tactile sensation presenting apparatus which concerns on embodiment of this invention.</figref><figref num="3">It is a figure explaining the example of the structure of the display resource including the tactile setting information in the application software which concerns on embodiment of this invention.</figref><figref num="4">It is a figure which shows the example of the correspondence relation between the value which specifies the tactile sensation presented in the tactile sensation setting information shown in FIG. 3 and the content of the tactile sensation presenting operation indicated by the value.</figref><figref num="5">It is a figure which shows the example of the correspondence relationship between the image displayed based on various display resources, and the value of a tactile presentation operation type.</figref><figref num="6">It is a flowchart explaining the operation of the display control part of the tactile sensation presenting apparatus which concerns on embodiment of this invention.</figref><figref num="7">It is a flowchart explaining the operation of the tactile sensation control part of the tactile sensation presenting apparatus which concerns on embodiment of this invention.</figref><figref num="8">It is a figure which showed an example of correspondence between the state which performed the image display based on a display resource, and the value of the tactile presentation operation type assigned to the image.</figref><figref num="9">It is a figure which showed the other example of correspondence between the state which performed the image display based on a display resource, and the value of the tactile presentation operation type assigned to the image.</figref>

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a schematic configuration of a tactile sensation presenting device according to an embodiment of the present invention.

As shown in FIG. 1, the tactile sensation presenting device 1 includes a display control unit 12, a tactile sensation control unit 14, and an application execution unit 20. The display control unit 12, the tactile control unit 14, and the application execution unit 20 are included in the control unit 10. Further, the tactile sensation presenting device 1 includes a display unit 30, a touch sensor 40, a tactile sensation presenting unit 50, a load detecting unit 60, and a storage unit 80.

The storage unit 80 is composed of, for example, a NAND flash memory or the like, and includes an application storage area 82 and a display resource storage area 84 in at least a part of the memory area. The application storage area 82 included in the storage unit 80 can store various applications. In addition, the display resource storage area 84 can store data used for display on the display unit 30 among the data constituting the application. For example, the display resource storage area 84 can store image data such as an image file as a display resource. As will be described later, tactile sensation setting information indicating the type of tactile sensation, presentation conditions, and the like is added to this display resource. Note that these storage areas 82 and 84 represent a distinction between virtual areas for convenience of the following description, and do not necessarily mean a physical distinction, but are displayed in the application storage area 82. A mode in which the resource storage area 84 is included is also conceivable.

The application stored in the application storage area 82 and the display resource stored in the display resource storage area 84 are separate and independent. As will be described later, when the application execution unit 20 executes the processing based on the application, the display resource is read from the display resource storage area 84 based on the source code of the application. Therefore, the source code of the application only has an instruction to read the display resource from the display resource storage area 84, and whether or not the tactile setting information is added to the display resource when the display resource is read. So there is no need to modify the source code of the application. Further, various data such as applications stored in the tactile presentation device 1 need not be stored in the storage unit 80, and for example, communication can be performed by a communication unit (not shown) included in the tactile presentation device 1. It may be stored in a server or the like.

The application execution unit 20 reads a designated one of various applications from the storage unit 80, executes processing based on the application, and controls each functional unit based on the application. The application executed by the application execution unit 20 in the present embodiment will be described later.

The display unit 30 displays a background image or the like, or displays an object such as a push button switch (push type button switch) as an image. This object is an image that suggests to the operator the area to be touched. Further, the push button switch is a button, a key, or the like (hereinafter, simply collectively referred to as "key or the like") used by the operator for input operation. The display unit 30 is configured by using, for example, a liquid crystal display panel (LCD), an organic EL display panel, or the like.

The touch sensor 40 is usually arranged in front of the display unit 30, and the contact (or release) of the object displayed on the display unit 30 by the operator's finger or the like (contact target) is performed by the corresponding touch sensor 40. Detect on the touch surface. Further, the touch sensor 40 detects the position of contact with respect to the touch surface, and notifies the tactile sensation control unit 14 of the detected contact position. The touch sensor 40 is composed of a known type such as a resistance film type, a capacitance type, and an optical type.

The tactile sensation presenting portion 50 is configured by using, for example, a piezoelectric vibrator or the like, and generates vibration according to a predetermined vibration pattern on the touch surface of the touch sensor 40. The tactile sensation presenting unit 50 presents a tactile sensation to a contact object in contact with the touch surface by generating vibration on the touch surface of the touch sensor 40.

The load detection unit 60 detects the pressing load on the touch surface of the touch sensor 40, and is configured by using an element that linearly reacts to the load, such as a strain gauge sensor or a piezoelectric element. Further, the load detection unit 60 notifies the tactile sensation control unit 14 of the detected pressing load on the touch surface of the touch sensor 40.

The display control unit 12 displays a background image, an image of an object, or the like on the display unit 30 based on a display instruction from the application execution unit 20, or changes the display of the image. Control the display. When the application execution unit 20 gives a display instruction to the display control unit 12, the display data of the image to be displayed on the display unit 30 is displayed in the display resource storage area 84 of the storage unit 80. Resources can be read. The display control unit 12 receives the supply of the display data read from the storage unit 80 in this way, and controls the display unit 30 so as to draw a predetermined image. Further, as will be described later, the display control unit 12 includes tactile setting information added to the display resource designated by the application execution unit 20, information on the display position (and display hierarchy) of the display resource, and the like. The tactile sensation is set for the tactile sensation control unit 14 based on the above.

The tactile sensation control unit 14 controls the tactile sensation presenting unit 50 so as to generate a vibration of a predetermined pattern based on a predetermined condition based on the tactile sensation setting by the display control unit 12. For example, when the tactile control unit 14 detects contact at a position corresponding to an image area of a predetermined object or the like displayed on the display unit 30, the touch sensor 40 is in contact with the touch surface of the touch sensor 40. The tactile sensation presenting unit 50 is controlled so as to exhibit the tactile sensation set in the image.

Further, in the tactile sensation control unit 14, when the tactile sensation setting by the display control unit 12 is accompanied by the pressing load condition, the pressing load detected by the load detecting unit 60 satisfies a predetermined criterion for presenting the tactile sensation. Determine if it is. In this case, the tactile sensation control unit 14 causes the tactile sensation presenting unit 14 to generate a predetermined pattern of vibration based on the predetermined conditions when the load detecting unit 60 detects a pressing load satisfying the predetermined criteria for exhibiting the tactile sensation described above. Control 50. For example, in the tactile control unit 14, when the touch sensor 40 detects contact at a position corresponding to a predetermined image area displayed on the display unit 30, the load detection unit 60 sets the tactile sensation in the image. When a pressing load satisfying the load criteria for presenting is detected, the tactile sensation presenting unit 50 is controlled so as to exhibit the tactile sensation set in the image to the contact target. When the tactile sensation control unit 14 controls the tactile sensation presenting unit 50 so as to exhibit the tactile sensation, the tactile sensation control unit 14 appropriately obtains information such as the position information where the contact is detected by the touch sensor 40 and the load detected by the load detection unit 60. Notify the application execution unit 20 at the appropriate timing.

FIG. 2 shows an example of a mounting structure centered on the display unit 30, the touch sensor 40, the tactile sensation presenting unit 50, and the load detecting unit 60 among the tactile sensation presenting devices 1 shown in FIG. A) is a cross-sectional view of the main part, and FIG. 2B is a plan view of the main part. The display unit 30 is stored and held in the housing 71. The touch sensor 40 is held on the display unit 30 via an insulator 72 made of an elastic member. In the tactile sensation presenting device 1 according to the present embodiment, the display unit 30 and the touch sensor 40 have a rectangular shape in a plan view. However, the shape can be made according to various conditions such as the configuration of the touch sensor 40 or the display unit 30 included in the tactile sensation presenting device 1. Further, the tactile display device 1 holds the touch sensor 40 on the display unit 30 via insulators 72 arranged at four corners outside the display area A of the display unit 30 shown by virtual lines in FIG. 2 (B). To do.

Further, the housing 71 is provided with an upper cover 73 so as to cover the surface area of the touch sensor 40 outside the display area of the display unit 30, and is composed of an elastic member between the upper cover 73 and the touch sensor 40. The insulator 74 is arranged. In the touch sensor 40 shown in FIG. 2, the front surface member having the touch surface 40a is made of, for example, a transparent film or glass, and the back surface member is made of glass or acrylic. As the touch sensor 40, when the touch surface 40a is pressed, the pressed portion bends (distorts) by a small amount according to the pressing pressure, or the structure itself bends by a small amount.

On the surface of the touch sensor 40, a strain gauge sensor 62 for detecting the load (pressing pressure) applied to the touch sensor 40 is provided in the vicinity of each side covered with the upper cover 73 by adhesion or the like. Further, on the back surface of the touch sensor 40, a piezoelectric vibrator 52 for vibrating the touch sensor 40 is provided in the vicinity of two opposing sides by adhesion or the like. That is, in the tactile presentation device 1 shown in FIG. 2, the load detection unit 60 shown in FIG. 1 is configured by using four strain gauge sensors 62, and the tactile presentation unit 50 is configured by using two piezoelectric vibrators 52. doing. Then, the touch surface 40a is vibrated by vibrating the touch sensor 40 by the tactile display unit 50. Note that FIG. 2 (B) omits the illustration of the housing 71, the upper cover 73, and the insulator 74 shown in FIG. 2 (A).

Next, information related to the tactile sensation setting (hereinafter, appropriately referred to as "tactile sensation setting information") added to the image data supplied by the application used in the tactile sensation presenting device 1 according to the present invention will be described.

FIG. 3 is a diagram illustrating an example of the configuration of the display resource including the tactile setting information. In the example shown in FIG. 3, an example is shown when the display resource is a BMP (Microsoft Windows (registered trademark) Bitmap Image) file. However, the display resource of the application used in the tactile presentation device 1 according to the present invention is not limited to the BMP file, and may be in another file format such as a PNG file or a JPG file. When other file formats are used, it is preferable to add tactile setting information to the area corresponding to the reserved area of the BMP file described later.

FIG. 3A is a diagram illustrating the configuration of the BMP file format constituting the display resource according to the present invention. As shown in FIG. 3A, the BMP file format is composed of a 14-byte file header, a 12- or 40-byte information header, and image data. The information header is in the Windows® format if it is 40 bytes, and in the OS / 2 format if it is 12 bytes.

FIG. 3B is a diagram illustrating a format structure of a file header that is a part of a BMP file. As shown in Fig. 3 (B), the format of the file header is such that the first 2 bytes indicate the file type, the next 4 bytes indicate the file size, and then 2 bytes each of reserved areas 1 and 2 exist. , The last 4 bytes are the offset from the beginning to the image data. These two-byte reserved areas 1 and 2 are always zero and are not in use. Therefore, in the present invention, this reserved area is utilized as information related to tactile sensation setting, that is, tactile sensation setting information. For example, in the following example, a case where the 2-byte reserved area 1 is used as the tactile setting information will be described.

FIG. 3C is a diagram illustrating a format configuration using the 2-byte reserved area 1 as tactile setting information. In the present invention, as shown in FIG. 3C, for example, the first 1 byte of the 2 bytes constituting the reserved area 1 is used to indicate that the reserved area 1 is used as tactile setting information. Stores the indicated identifier. It can be assumed that any value other than 0x00 or 0xFF can be used for this identifier. Then, in the present invention, for example, the remaining 1 byte out of the 2 bytes constituting the reserved area 1 is used to store the information indicating the tactile presentation operation type. The information indicating the tactile presentation operation type is presented by the operation executed when the touch sensor 40 at the position corresponding to the area of the image displayed based on this display resource, that is, the BMP file, detects contact. , Information indicating the type of tactile sensation.

The above-mentioned tactile setting information needs to be included in the program constituting the application. However, in the present invention, it is sufficient to include this tactile setting information in a part of the file header and add it to the image data. Therefore, in the application according to the present invention, it is not necessary to modify the source code itself, so that the versatility as a program can be enhanced. That is, even when such an application is used in a device equipped with a touch sensor of a type that does not exhibit a tactile sensation, the tactile sensation setting information added to the image data is ignored. Functions can be operated normally.

FIG. 4 shows an example of the correspondence between the value to be stored in the second byte of the reserved area 1 in order to specify the tactile sensation to be presented in the above-mentioned tactile sensation setting information and the content of the tactile sensation presenting operation indicated by the value. It is a figure which shows. In the tactile sensation presenting device 1 according to the present invention, the tactile sensation control unit 14 is set to present the tactile sensation shown in the content of the corresponding tactile sensation presenting operation based on recognizing the value as illustrated in FIG. I do.

As shown in FIG. 4, for example, when the value 0x00 is stored in the tactile presentation operation type described in FIG. 3 (C), the touch sensor 40 at the position corresponding to the image displayed by the image data of the display resource. Even if contact is detected in, it is set so as not to exhibit a tactile sensation. Further, when the value 0x01 is stored in the tactile sensation display operation type, when the touch sensor 40 at the position corresponding to the image displayed by the image data of the display resource detects contact, the normal click tactile sensation is presented. Is set to. The "normal click tactile sensation" means "a reproduction of the tactile sensation of a button being pushed down (clicking) at once" obtained when a mechanical key or button is pressed.

Similarly, when the value 0x02 is stored in the tactile presentation operation type, when a touch is detected by the touch sensor 40 at the position corresponding to the image displayed by the image data of the display resource, a click with a tactile sensation is clicked. It is set to exhibit a tactile sensation. In addition, "click tactile sensation of Gnu" is, for example, "reproduced the tactile sensation that the button gradually sinks (rather than sinking at once)" obtained when pressing a key or button with strong feedback. Means.

Further, when the value 0x11 is stored in the tactile sensation presentation operation type, when the touch sensor 40 at the position corresponding to the image displayed by the image data of the display resource detects contact, it corresponds to the pressing load at that time. It is set to exhibit a normal click tactile sensation. In this way, when the tactile sensation is presented according to the pressing load, when the load detected by the load detecting unit 60 meets the predetermined first stage standard, the tactile sensation of the first stage is set to be exhibited. Will be done. Similarly, in this case, if the load detected by the load detector 60 meets a predetermined second-stage criterion higher than the first-stage criterion, a stronger second-stage normal click than the first-stage. It is set to present a tactile sensation according to a plurality of stages of pressing load, such as presenting a tactile sensation.

Further, when the value 0x21 is stored in the tactile sensation presentation operation type, when the touch sensor 40 at the position corresponding to the image displayed by the image data of the display resource detects the movement of contact, the tactile sensation is smooth. It is set to exhibit a trace tactile sensation. The "smooth touch" is a tactile sensation obtained by controlling the tactile sensation display unit 50 that vibrates the touch sensor to reproduce the sensation obtained when the operator slides a finger or the like on a flat surface with little friction. Means that. In addition, the "rough feel of tracing" is a tactile sensation obtained by controlling the tactile sensation presentation unit 50 that vibrates the touch sensor to reproduce the feel obtained when the operator slides a finger or the like on a flat surface with large friction. Means that.

FIG. 5 is a diagram showing an example of the correspondence between the image displayed on the display unit 30 based on various display resources and the value of the above-mentioned tactile presentation operation type. The left column of FIG. 5 shows an example of an image displayed on the display unit 30 based on the image data of the display resource included in the application that can be used in the tactile display device 1 according to the present invention. Further, the column on the right side of FIG. 5 shows a value indicating a tactile presentation operation type of the tactile setting information included in the file header of the display resource.

For example, the tactile sensation display operation type of an image such as a background image as shown in FIGS. 5 (A) and 5 (B), which should not exhibit a tactile sensation even if contact is detected by the touch sensor 40 at the position corresponding to the portion, is Correspond to the value 0x00 so that the tactile sensation is not presented. On the other hand, for the tactile sensation display operation type of the image that should exhibit the tactile sensation when the touch sensor 40 at the position corresponding to the relevant part, such as the image of the button shown in FIGS. 5 (C), (D) and (E), is detected. Corresponds the values 0x01, 0x11, etc. so that the tactile sensation corresponding to each image is presented.

Further, the tactile sensation display operation type of the image in which the tactile sensation should be presented when the movement of contact is detected in the touch sensor 40 at the position corresponding to the portion such as the image of the slide bar shown in FIG. 5 (F) is, for example, a value. Correspond to 0x22 etc. so that the trace touch feeling etc. is presented. The image shown in FIG. 5 (G) is an image that constitutes a dialog box in the GUI. When an image such as a button is further displayed in this dialog box, the button exhibits a tactile sensation in response to the detection of contact, but does not exhibit a tactile sensation in other parts. The value 0x00 is also associated with the tactile presentation operation type of the image shown in).

Next, in the tactile sensation presenting device 1 according to the present invention, an operation when executing an application including a display resource to which tactile sensation setting information is added will be described. In the present invention, since the tactile setting information is included in the display resource, it is basically updated when the image is first displayed on the display unit 30 or when the image displayed on the display unit 30 is changed. The operation of setting the tactile sensation for the subsequent screen is performed. In the following description, the display control unit 12 and the tactile control unit 14 when the application execution unit 20 gives an instruction to display a predetermined image to the display control unit 12 based on the running application. The operation of is mainly explained.

FIG. 6 is a flowchart illustrating the operation of the display control unit 12. When the operation shown in FIG. 6 starts, the display control unit 12 first determines whether or not the application execution unit 20 has instructed the display control unit 12 to display a predetermined image (step S11). .. When the application execution unit 20 instructs in step S11 to display a predetermined image, the display resource is acquired from the display resource storage area 84 of the storage unit 80 based on the display instruction specified in the application. (Step S12).

After acquiring the display resource from the storage unit 80, the display control unit 12 generates drawing data by plotting the acquired display resource at the display position based on the display instruction (step S13). .. When generating drawing data, information such as a display position for displaying each display resource can be defined in the application, for example, so that the position of each image to be displayed can be determined. Further, even when the display unit 30 performs hierarchical display of each image, it is possible to display an image in a special mode by prescribing information to that effect in the application.

When the drawing data is generated, the display control unit 12 generates tactile setting information based on the file header included in the display resource used in step S13 (step S14). When generating the tactile sensation information, the information such as the position or range for setting the tactile sensation presented by the touch sensor 40 is also included based on the information such as the position (and display hierarchy) for displaying the display resource. be able to.

When the tactile sensation setting information is generated, the display control unit 12 supplies the drawing data generated in step S13 to the display unit 30, and controls the display unit 30 to display an image based on the drawing data (step). S15). As a result, the screen display on the display unit 30 of the tactile presentation device 1 is updated.

When the screen display is updated, the display control unit 12 supplies the tactile sensation setting information generated in step S14 to the tactile sensation control unit 14 (step S16). As a result, the tactile sensation setting can be updated in the tactile sensation control unit 14.

By the operation as described above, when the tactile sensation setting information is included in the display resource, the tactile sensation presenting device 1 can supply the generated tactile sensation setting information to the tactile sensation control unit 14. If the display resource includes tactile sensation setting information, the tactile sensation setting information is not generated in step S14, so that it is not supplied to the tactile sensation control unit 14 in step S16.

FIG. 7 is a flowchart illustrating the operation of the tactile control unit 14. When the operation shown in FIG. 7 starts, the tactile sensation control unit 14 first determines whether or not the tactile sensation setting information has been supplied from the display control unit 12 (step S21). When the tactile sensation setting information is supplied from the display control unit 12 in step S21, the tactile sensation control unit 14 updates the tactile sensation setting information managed by the tactile sensation control unit 12 (step S22). Here, the update of the tactile sensation setting information means that the tactile sensation represented by the tactile sensation presenting operation content corresponding to the value of the tactile sensation presenting operation type included in the tactile sensation setting information for the image displayed based on each display resource is the tactile sensation. Is set in the tactile control unit 14 together with the condition in which is presented. That is, at this time, the tactile sensation control unit 14 presents the tactile sensation presenting unit 50 according to the image displayed on the display unit 30 based on the information related to the tactile sensation setting added to the image data supplied from the application. Control to set the tactile sensation.

When the tactile sensation setting information is updated, the tactile sensation control unit 14 determines whether or not the touch sensor 40 has detected contact with a contact target such as an operator's finger or a stylus (step S23). If the tactile sensation setting information is not supplied from the display control unit 12 in step S21, there is no information to be updated. Therefore, the tactile sensation control unit 14 does not go through step S22, that is, without updating the tactile sensation setting information. Perform the processing after step S23.

If the touch sensor 40 does not detect the contact in step S23, the tactile control unit 14 returns to step S21 to continue the process. On the other hand, when the touch sensor 40 detects the contact of the operator in step S23, the tactile sensation control unit 14 confirms the tactile sensation setting at the contact position (step S24). That is, in step S24, the tactile sensation control unit 14 confirms whether or not the tactile sensation to be presented when the touch sensor 40 detects contact at a position corresponding to the area of the image displayed on the display unit 30 is set. Then, the process proceeds to step S25.

As a result of confirming the tactile sensation setting at the contact position in step S24, if it is determined in step S25 that the presentation of the tactile sensation is not necessary, the tactile sensation control unit 14 returns to step S21 and continues the process.

On the other hand, when it is determined in step S25 that it is necessary to present the tactile sensation, the tactile sensation control unit 14 determines whether or not the tactile sensation set at the position includes the condition of the pressing load (step S26). .. In distinguishing the presence or absence of this condition, for example, the presentation of the tactile sensation having the condition of the pressing load and the presentation of the tactile sensation not having the condition should be distinguished by the value indicating the tactile sensation presentation operation type described in FIGS. Can be done.

In step S26, if the tactile sensation set at the contact position does not include the condition of the pressing load, the tactile sensation is immediately exhibited, so that the tactile sensation control unit 14 proceeds to step S30 and continues the process.

In step S30, the tactile control unit 14 outputs a vibration pattern according to the tactile sensation setting to the tactile sensation presenting unit 50 (step S30). As a result, the tactile sensation presenting unit 50 can present the tactile sensation corresponding to the image corresponding to the position where the contact is detected by the touch sensor 40. That is, when the tactile control unit 14 detects contact at a position corresponding to a predetermined image area displayed on the display unit 30, the tactile control unit 14 refers to the contact target that is in contact with the touch surface of the touch sensor 40. The tactile sensation presenting unit 50 is controlled so as to exhibit the tactile sensation set in the image.

When the tactile sensation is presented in step S30, the tactile sensation control unit 14 notifies the application execution unit 20 of the contact information by the operator (step S31). The application execution unit 20 can execute a predetermined process assigned to the contact in response to the detection of the contact by the operator, using the reception of the notification as a trigger. For example, when the touch sensor 30 at the position corresponding to the image of the play button of the music player displayed on the display unit 30 detects a contact, the application execution unit 20 starts the music playback operation based on the application of the music player. To do. Alternatively, for example, when a touch is detected by the touch sensor 30 at a position corresponding to the image of the keyboard key displayed on the display unit 30, the application execution unit 20 displays the character on the display unit 30 based on the character input application. Start the output operation.

On the other hand, in step S26, when the tactile sensation set at the contact position includes the pressing load condition, the load detecting unit 60 detects the pressing load satisfying a predetermined standard and then presents the tactile sensation, so that the process proceeds to step S27. And continue processing.

In step S27, the tactile control unit 14 determines whether or not the operator's contact with the touch sensor 40 is still continuously detected. If the contact is no longer continued in step S27, it can be determined that the operator's contact operation has not already been performed. Therefore, the tactile control unit 14 assumes that the operator's contact with the touch sensor 40 has been released. Return to S21. That is, when the contact operation of the operator is completed without satisfying the pressing load condition, the tactile sensation control unit 14 does not output the vibration pattern to the tactile sensation presenting unit 50 or notify the application execution unit 20.

On the other hand, if the contact is still continuously detected in step S27, the tactile control unit 14 controls the load detection unit 60 so as to monitor the pressing load on the touch surface of the touch sensor 40 (step S28). After step S28, the tactile control unit 14 determines whether or not the pressing load detected by the load detecting unit 60 satisfies a predetermined criterion while being increased by the pressing of the touch sensor 40 (step S29). The load detection unit 12 detects the load from, for example, the average value of the outputs of the four strain gauge sensors 62. Here, the load satisfying a predetermined standard is set in advance to a value such as 1N (Newton) based on the pressing load when the operator performs a normal pressing operation, and the setting can be changed thereafter. Is preferable. In addition, this predetermined standard is to prevent the operator from accepting an operation when the operator unintentionally touches it lightly as an input, and to give the operator a pressure sensation for a realistic tactile sensation, which will be described later. Therefore, in consideration of the pressing load at the time of pressing input based on the intention of the operator (for example, the average value), do not set an excessively low standard.

If the pressing load does not meet the predetermined criteria in step S29, the tactile control unit 14 returns to step S27 to continue the process. On the other hand, when the pressing load satisfies a predetermined reference in step S29, the tactile control unit 14 performs the processes of steps S30 and S31 described above.

By the process of step S30, when the load detecting unit 60 detects a pressing load satisfying a predetermined standard, the tactile sensation presenting unit 50 gives a tactile sensation according to the image corresponding to the position where the contact is detected by the touch sensor 40. Can be presented. That is, when the tactile sensation control unit 14 detects the contact at the position corresponding to the region of the predetermined image displayed on the display unit 30, the load detection unit 60 sets the tactile sensation on the image. When a pressing load satisfying the load reference for presenting is detected, the tactile sensation presenting unit 50 is controlled so as to exhibit the tactile sensation set in the image to the contact target in contact with the touch surface of the touch sensor 40.

When the process of step S31 is performed through step S29, the tactile control unit 14 presses a predetermined reference detected by the load detection unit 60 when notifying the application execution unit 20 of the contact information by the operator. Information about the load will also be notified. As a result, even when the application execution unit 20 exhibits a tactile sensation in response to a plurality of stages of pressing load, for example, the reception of the notification is used as a trigger to detect the contact by the operator and the contact in response to the pressing load. It is possible to execute a predetermined process assigned to. That is, the application execution unit 20 can execute different operations according to the load in a plurality of stages. For example, like a button that functions as a camera shutter, the autofocus function is activated when the pressing load meets the criteria of the first stage, and then the function of capturing an image is activated when the criteria of the second stage are met. You can assign actions.

The tactile sensation presented by the tactile sensation presenting unit 14 to the pressed object in step S30 can be a realistic tactile sensation such as the above-mentioned click tactile sensation. In order to present a realistic click tactile sensation to the operator, the tactile sensation presenting device 1 stimulates the tactile sensation in a state of stimulating the operator's pressure sensation by performing the following operations. That is, the tactile sensation presenting device 1 stimulates the sense of pressure until the load applied to the touch sensor 40 meets the tactile sensation presenting reference (for example, 1N), and when the load meets the reference, the piezoelectric vibrator 52 is pressed. It is driven by a predetermined drive signal to vibrate the touch surface 40a to stimulate the sense of touch. As a result, the tactile sensation presenting device 1 can present the operator with a click tactile sensation similar to that obtained when a button switch such as a push button switch (push type button switch) is pressed. Therefore, even if the push button switch is drawn on the upper part of the touch sensor, the operator can perform a touch operation on the touch sensor 40 while obtaining a realistic click tactile sensation similar to that when the push button switch is operated. Because you can do it, you will not feel any discomfort. Further, since the contact operation can be performed in conjunction with the consciousness that the touch sensor 11 is "pressed", it is possible to prevent an operation error due to mere pressing.

The drive signal for presenting the click tactile sensation described above, that is, the constant frequency, period (wavelength), waveform, and amplitude that stimulate the tactile sensation can be appropriately set according to the click tactile sensation to be presented. For example, when presenting a click tactile sensation represented by a metal dome switch used in a mobile terminal, for example, the tactile sensation presenting unit 50 is driven by a drive signal for one cycle consisting of a Sin wave having a constant frequency of 170 Hz. The tactile sensation presenting portion 50 is driven by such a drive signal, and the touch surface 40a is vibrated by about 15 μm with a reference pressing load applied. As a result, it is possible to present the operator with a realistic click feeling as if the actual key was clicked.

Next, by the operation described with reference to FIGS. 6 and 7, an image is displayed on the display unit 30 in the tactile sensation presenting device 1, and a tactile sensation to be presented at the position of each image is set as a specific example. Will be described with reference to.

FIG. 8 is a diagram showing the correspondence between the appearance of displaying an image on the display unit 30 based on the display resource described in FIG. 5 and the value of the tactile presentation operation type assigned to the image.

FIG. 8A shows the screen displayed on the display unit 30 based on the fact that the application execution unit 20 instructed the display control unit 12 to display the screen by executing the application with the tactile presentation device 1. It is a figure which shows an example. The example shown in FIG. 8 (A) shows how an application for playing a moving image is executed in the tactile presentation device 1. In the screen shown in FIG. 8 (A), among the display resources described in FIG. 5, the video screen (B), the button objects (C) and (E), are placed on the background image (A). The object (F) on the slide bar is displayed. In the tactile sensation presenting device 1, when displaying these images on the display unit 30, the tactile sensation to be presented according to each image is set by the processing described with reference to FIGS. 6 and 7.

FIG. 8 (B) shows how the tactile sensation presentation operation type set in the tactile sensation control unit 14 is assigned according to each image when the image of FIG. 8 (A) is displayed on the display unit 30. It is a diagram conceptually represented. Note that FIG. 8B is for showing how the tactile sensations to be presented are assigned according to each image, and is not actually displayed on the display unit 30.

In the area where the background image and the moving image are displayed in FIG. 8 (A), a value of 0x00 is set as the tactile display operation type as shown in FIG. 8 (B), so that the touch sensor 40 makes contact with the area. Even if it is detected, the tactile sensation presenting device 1 does not present a tactile sensation. That is, even if the operator performs an operation of touching the touch sensor 40 at a position corresponding to the area where the background image or the moving image is displayed on the display unit 30, no tactile sensation is presented.

On the other hand, in the area where the "play" and "stop" buttons are displayed in FIG. 8 (A), a value of 0x01 is set as the tactile presentation operation type as shown in FIG. 8 (B). Therefore, when the touch sensor 40 detects contact with the region, the tactile sensation presenting device 1 presents a normal click tactile sensation. That is, when the operator touches the touch sensor 40 at the position corresponding to the area where the "play" and "stop" buttons are displayed on the display unit 30, a normal click tactile sensation is presented. To.

Further, in the area where the "return" and "forward" buttons are displayed in FIG. 8 (A), a value of 0x11 is set as the tactile presentation operation type as shown in FIG. 8 (B). Therefore, when the touch sensor 40 detects contact with the region, the tactile sensation presenting device 1 presents a normal click tactile sensation in multiple stages. That is, when the operator makes contact with the touch sensor 40 at the position corresponding to the area where the "return" and "feed" buttons are displayed on the display unit 30, the load is detected stepwise. And a normal click tactile sensation is presented depending on the detected load.

Further, in the area where the slide bar is displayed in FIG. 8 (A), a value of 0x22 is set as the tactile presentation operation type as shown in FIG. 8 (B). Therefore, when the touch sensor 40 detects the movement of contact with respect to the region, the tactile sensation presenting device 1 presents a rough tactile sensation.

FIG. 9A is based on the fact that the application execution unit 20 instructed the display control unit 12 to display the screen by executing the same application as described in FIG. 8A by the tactile presentation device 1. , Is a diagram showing another example of the screen displayed on the display unit 30. The example shown in FIG. 9 (A) shows how the screen display changes based on the operation of the application executed in the state shown in FIG. 8 (A). In the screen shown in FIG. 9 (A), in addition to the state shown in FIG. 8 (A), among the display resources described in FIG. 5, a button object (D) is further placed on the dialog box (G). ) Is displayed. Similar to the case of FIG. 8A, when displaying these images on the display unit 30 in the tactile sensation presenting device 1, the tactile sensation presented according to each image by the processing described in FIGS. 6 and 7. Is set.

In the area where the Yes and No buttons are displayed in FIG. 9 (A), a value of 0x01 is set as the tactile presentation operation type as shown in FIG. 9 (B). Therefore, when the touch sensor 40 detects contact with the region, the tactile sensation presenting device 1 presents a normal click tactile sensation. That is, when the operator makes contact with the touch sensor 40 at the position corresponding to the area where the "Yes" and "No" buttons are displayed on the display unit 30, a normal click tactile sensation is presented. To.

FIG. 9 (B) shows how the tactile sensation presentation operation type set in the tactile sensation control unit 14 is assigned according to each image when the image of FIG. 9 (A) is displayed on the display unit 30. It is a diagram conceptually represented.

As shown in FIG. 9B, in this case, the tactile sensation to be presented is not set except for the area corresponding to the Yes and No buttons. In the state shown in Fig. 9 (A), a dialog box asking the termination of the application is displayed as interrupt processing during normal video playback by executing the application. Therefore, in this case, the tactile sensation presenting device 1 detects contact with the touch sensor 40 at a position other than the area corresponding to the "Yes" and "No" buttons in order to prompt the response to the dialog box with the highest priority. It is set so that it does not show a tactile sensation even if it is done.

As described above, according to the present embodiment, even when the image is superimposed on the display unit 30 to perform the hierarchical display, for example, the tactile sensation to be presented according to the image added to the uppermost display layer is set. Is also possible. In this case, as described above, for example, the tactile setting information added to the display resource displayed in the lower layer can be invalidated.

The present invention is not limited to the above embodiment, and many modifications or changes can be made. For example, in the above-described embodiment, the display control unit 12, the tactile control unit 14, and the application execution unit 20 have been described as separate components. However, the tactile sensation presenting device according to the present invention is not limited to such a configuration, and various configurations can be adopted according to the requirements at the time of design. For example, the control unit may include all or part of the components of the display control unit 12, the tactile control unit 14, and the application execution unit 20.

Further, in the above embodiment, the touch sensor 40 is used to detect the contact of the touch sensor with the touch surface, but when the load sensor (load detection unit) is used to satisfy a predetermined pressing load standard. , It can also be determined that the contact has been made. Such a load detection unit can be configured by using an arbitrary number of strain gauge sensors or the like, similarly to the load detection unit of the above-described embodiment.

Further, in the above-described embodiment, the load detection unit can be configured according to the contact detection method in the touch sensor. For example, in the case of the resistance film method, if the load can be detected from the change in the output signal based on the change in resistance due to the contact area, it can be configured without using the strain gauge sensor. Alternatively, in the case of the capacitance method, even when the load can be detected from the change in the output signal based on the change in capacitance, it can be configured without using the strain gauge sensor.

Further, the tactile sensation presenting portion can be configured by using an arbitrary number of piezoelectric vibrators, or can be configured by providing a transparent piezoelectric element on the entire surface of the touch sensor, or if it can express vibration that presents a tactile sensation, an eccentric motor. Can be configured to rotate once in one cycle of the drive signal. Further, when the load detecting unit and the tactile sensation presenting unit are configured by using the piezoelectric element, the piezoelectric element can be shared to form the load detecting unit and the tactile sensation presenting unit.

In the above-described embodiment, the description has been made assuming a configuration in which the touch sensor is arranged so as to be overlapped on the upper surface of the display unit. It is not essential that the tactile sensation presenting device according to the present invention has such a configuration, and the touch sensor and the display unit may be separated from each other. However, if the touch sensor is arranged so as to be overlapped on the upper surface of the display unit, the operator can easily recognize the correspondence between the displayed image and the presented tactile sensation.

Further, the tactile sensation presenting device according to the present invention has been described on the assumption that the piezoelectric element is driven when the pressing load detected based on the output of the piezoelectric element satisfies a predetermined criterion for presenting the tactile sensation. .. Here, the case where the pressing load detected based on the output of the piezoelectric element satisfies the reference value for exhibiting the tactile sensation may be when the detected pressing load reaches the reference value for exhibiting the tactile sensation. , The detected pressing load may exceed the reference value for exhibiting the tactile sensation, or the reference value for exhibiting the tactile sensation may be detected based on the output of the piezoelectric element.

The program used for the tactile sensation presenting device 1 of the present invention is information for executing the process described in the above-described embodiment, and is stored in the storage unit 80. The control unit 10 of the tactile sensation presenting device 1 of the present invention performs each of the above-described processes based on the program stored in the storage unit 80. The above program does not need to be stored in the storage unit 80, and may be stored in, for example, a server or an external storage medium capable of communicating with a communication unit (not shown) included in the tactile presentation device 1. Good. Further, the program may be stored separately in the storage unit 80 and a server, an external storage medium, or the like.

Further, the display unit 30 and the touch sensor 40 in the description of the present embodiment may be configured by an integrated device by having both the functions of the display unit and the touch sensor on a common substrate. .. As an example of the configuration of a device that integrates both the functions of the display unit and the touch sensor in this way, a plurality of photoelectric conversion elements such as photodiodes are regularly arranged in a matrix-like array of pixel electrodes of the liquid crystal panel. A mixture can be mentioned. While this device displays an image by the liquid crystal panel structure, the tip of the pen that touch-inputs the desired position on the panel surface reflects the light of the backlight for liquid crystal display, and the reflected light is reflected by the peripheral photoelectric conversion element. By receiving light, the touch position can be detected.

The tactile sensation presenting unit 50 may be configured to indirectly vibrate the touch detection unit touch sensor 40 by vibrating the tactile sensation presenting device 1 based on a vibration motor (eccentric motor) or the like. By disposing a piezoelectric element in the sensor 40, the touch sensor 40 may be configured to vibrate directly.

Further, for example, when the load detection unit 60 and the tactile sensation display unit 50 are configured by using the piezoelectric element, the load detection unit and the tactile sensation display unit can be configured by sharing the piezoelectric element. This is because the piezoelectric element generates electric power when pressure is applied and deforms when electric power is applied.

In order for the touch sensor 40 to detect contact, it is not essential that the contact target physically touches the touch sensor 40. For example, when the touch sensor 40 is an optical type, the touch sensor 40 detects the position where the infrared rays on the touch sensor 40 are blocked by a finger, a stylus pen, or the like, so that the contact target does not come into contact with the touch sensor 40. It is unnecessary.

Further, in the present invention, when the pressing load detected by the load detecting unit 60 satisfies a predetermined criterion for being recognized as an input, the tactile sensation presenting unit 50 is driven, but is detected by the load detecting unit 60. When the pressing load that meets the predetermined criteria for being recognized as an input is satisfied, the pressing load detected by the load detecting unit 60 may reach a predetermined value that is recognized as an input. It may be when the pressing load detected by the load detecting unit 60 exceeds a predetermined value recognized as an input, or when a predetermined value recognized as an input is detected by the load detecting unit 60.

1 Tactile presentation device 10 Control unit 12 Display control unit 14 Tactile control unit 20 Application Execution Department 30 Display 40 touch sensor 40a touch surface 50 Tactile presentation 52 Piezoelectric oscillator 60 Load detector 62 Strain gauge sensor 71 housing 72 Insulator 73 Upper cover 74 Insulator 80 memory