Gesture recognition method and touch system incorporating the same

The present invention relates to a touch system in general, and more particularly to a gesture recognition method and a touch system incorporating the gesture recognition method.

Touch systems are well known in the art and typically include a touch screen with a touch surface that is touched using a pointer to generate user input. The contact of the pointer with the touch surface is detected, and the contact of the pointer is used to generate the corresponding output depending on the area of the contact surface with which the contact was made. There are basically two common forms of touch systems available, which can be broadly divided into "active" touch systems and "passive" touch systems.

In an active touch system, the user can generate user input by touching the touch surface with a special pointer that usually requires some form of internal power supply, typically a battery. The special pointer emits signals such as infrared light, visible light, ultrasonic frequency, electromagnetic frequency, etc. that urge the touch surface.

In a passive touch system, the user can generate user input by touching the touch surface with a passive pointer, without the need to use a special pointer to urge the touch surface. The pointer may be a finger, a cylindrical tool, or any suitable object that can be used to contact a given area of interest on the touch surface.

Passive touch systems have the advantage over active touch systems in that any suitable pointing device, including the user's finger, can be used as a pointer in contact with the touch surface. As a result, user input can be easily generated. Moreover, since passive touch systems do not require special active pointers, battery power levels, pointer corruption, theft, or misplacement are not significant issues for the user.

For example, SMART technologies, the assignee of the present invention. US Patent Application No. 09 / 610,148 filed on July 5, 2000, filed on July 5, 2001 and published as WO 02/03316 on January 10, 2002, transferred to Inc.) International application PCT / CA01 / 00980 under the Convention discloses a camera-based touch system with a touch screen that includes a passive touch surface on which computer-generated images are displayed. A rectangular bezel or frame surrounds the touch surface and its multiple corners support multiple digital cameras. The plurality of digital cameras have overlapping fields of view including the touch surface and seeing the entire surface thereof. The digital camera acquires an image of the entire touch surface viewed from different positions and generates image data. The image data acquired by the digital camera is processed by a digital signal processor associated with the digital camera, and it is determined whether or not a pointer exists in the captured image data. When it is confirmed that a pointer exists in the captured image data, the digital signal processor sends a pointer information packet (PIP). Generates packet) and outputs PIP to the master controller. Each PIP contains a header part, a data part, and a checksum. The data part includes a pointer ID field that stores a pointer identifier to enable tracking of a plurality of pointers. The data unit also includes a pointer position parameter that specifies the pointer x position and the pointer tip parameter that specifies the pointer z position. The contact state field stores a value indicating whether or not the pointer is in contact with the touch surface, and enables detection of the pointer floating from the touch surface.

Upon receipt of the PIP, the master controller processes the PIP using triangulation to determine the position of each pointer in the captured image with respect to the touch plane in the (x, y) coordinate system. In this way, since PIP is generated in response to the captured image, the position and movement of the pointer on the touch surface can be tracked. The pointer position data generated by the master controller is sent to a computer running one or more application programs. The computer uses the pointer position data to update the computer-generated image displayed on the touch surface. Therefore, the contact of the pointer with the touch surface and the movement of the pointer on the touch surface can be recorded as characters or graphics or used to control the execution of an application program executed on a computer.

As can be seen from the above description, multiple digital cameras located at the corners of the bezel are used to capture the image data, so the touch system allows multiple pointers to touch and cross the touch surface at any time. You can decide if it worked. This, of course, has extended functionality compared to analog resistance touch systems that can track only a single pointer. The extension is provided by the camera-based touch system described above, but to date this extension has not been fully utilized.

<p> Therefore, an object of the present invention is to provide a novel gesture recognition method and a touch system incorporating the new gesture recognition method.</p>

<p> According to one aspect of the present invention, the step of displaying an image on the touch surface, the contact of the pointer with the touch surface is detected, and the contact of the pointer is examined to recognize the contact of a plurality of pointers representing gestures. A gesture recognition method is provided that comprises a step and a step of updating a displayed image according to the gesture when a contact of a plurality of pointers representing the gesture occurs.</p><p> The contact of a plurality of pointers representing a gesture includes the contact of a plurality of fingers to the touch surface, the contact of a finger to the touch surface and the contact of an object to the touch surface, and the contact of a plurality of objects to the touch surface. ..</p><p> In one aspect, the gesture is a right-click event, in which the first pointer touches on the displayed application and within the threshold distance from the first pointer touch and of the first pointer. Represented by the subsequent second pointer contact that occurs while the contact is maintained. In another aspect, the gesture is a scrolling event, represented by simultaneous pointer contact with the touch surface.</p><p> According to another aspect of the invention, a step of detecting a plurality of pointers close to a touch surface and determining whether the plurality of pointers are used to perform a known gesture, and the said. When a plurality of pointers are used to execute a known gesture, a gesture recognition method having a step of executing a command related to the gesture and a gesture recognition method having the gesture is provided.</p><p> During detection, the pointer touching the touch plane, or the pointer hover near the touch plane above the touch plane, is detected to determine if a known gesture is being performed, especially each. It is preferable to determine if one of the many known gestures associated with different commands is being performed. In a preferred embodiment, the movement of the plurality of pointers with respect to the touch surface and / or the type of pointer determines the gesture being performed.</p><p> According to still another aspect of the present invention, an input detection method in an interactive system capable of detecting the movement of a plurality of pointers in an input area at almost the same time, the step of capturing an image visible over the input area, and the above-mentioned. The step of analyzing the image to detect multiple pointers in the input area, and when multiple pointers are detected, examine the data associated with the multiple pointers and see if that data represents an input gesture. A method is provided that has a step of determining whether or not, and if the data represents an input gesture, a step of executing a command corresponding to the recognized input gesture.</p><p> According to yet another aspect of the present invention, a touch surface touched by at least one pointer, at least one imaging device having a field of view that is nearly visible across the touch surface, and the touch surface are contacted by a plurality of pointers. At that time, it communicates with the at least one imaging device, analyzes the image acquired by the at least one imaging device, determines the position on the touch surface where the pointers are in contact, and the plurality of pointers. To determine if the contact of the plurality of pointers represents a gesture, and if the contact of the plurality of pointers represents a gesture, at least one processor that executes a command related to the gesture. And a touch system with.</p><p> According to yet yet another aspect of the present invention, at least one imaging device having an input region in the field in which one or more pointers are moved to generate user input, and at least one imaging device. And analyzes each image acquired by the at least one imaging device to determine the action of the pointers to the input area and to determine when the actions of the plurality of pointers in the input area represent gestures. An interactive input system is provided that comprises at least one processor that executes a command corresponding to the gesture when the action of the plurality of pointers represents a gesture.</p><p> According to yet another aspect of the invention, a method of simulating a right-click mouse event in an interactive touch system, the first to a touch surface on a displayed application indicating a left-click mouse event. The step of detecting the contact of the pointer and the second step to the touch surface that occurs within the threshold period after the contact of the first pointer and within the threshold distance from the contact of the first pointer. A method is provided that comprises a step of detecting contact with a second pointer and a step of generating a right-click mouse event in response to the touch of the detected second pointer.</p><p> The present invention provides enhanced touch system functionality by detecting gestures represented by contact of multiple pointers to and / or movement on the touch surface and generating corresponding commands. It has the advantage of being able to provide.</p>

Next, embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

Here, with reference to FIG. 1, a camera-based touch system according to the present invention is shown, which is generally specified by reference number 50. The camera-based touch system 50 is similar to that disclosed in PCT application number WO 02/03316, which was assigned to the assignee of the invention, Smart Technology, the contents of which are incorporated herein by reference. There is. As you can see, the Touch System 50 is a digital signal processor (DSP). processor) Includes a touch screen 52 connected to a base master controller 54. The master controller 54 is also connected to the computer 56. The computer 56 executes one or more application programs and sends out a computer-generated image output displayed on the touch screen 52. The coordinate system of the touch system 52 is mapped to the coordinate system of the computer. The floating of the pointer on the touch screen 52 or the contact of the pointer with the touch screen 52, and the movement of the pointer above the touch screen 52 or on the touch screen 52 can be recorded as characters or figures, or performed by the computer 56. The touch screen 52, the master controller 54, and the computer 56 form a closed loop so that they can be used to control the execution of the application program.

Figure 2 shows the touch screen 52 for better understanding. The touch screen 52 of the present embodiment includes a high resolution display device such as the plasma display 58, and the front surface thereof forms the touch surface 60. The touch surface 60 is bordered by an illuminated bezel or frame 62 connected to the display device. The illuminated bezel 62 is of the type disclosed in U.S. Patent Application No. 10 / 354,168, filed January 30, 2003, and is an elongated side frame assembly connected to the sides of the plasma display 58. Contains 64. Each side frame assembly 64 contains a light source (not shown) that projects infrared back illumination over the entire surface of the touch surface 60. The ends of the sideframe assembly 64 are joined by a corner portion 68 that houses the DSP-based CMOS digital camera 70. Each digital camera 70 is mounted within its respective corner portion 68 so that its field of view includes the touch surface 60 and the entire surface of the touch surface 60 can be seen.

During operation, the digital camera 70 acquires an image of the touch surface 60 and generates image data. The acquired image data is processed by a digital signal processor associated with the digital camera 70 to determine if a pointer is present in the captured image. When it is determined that one or more pointers are present in the acquired image data, the digital signal processor of the digital camera 70 generates a pointer information packet (PIP) and digitalizes the PIP. Send to a signal processor (DSP) -based master controller 54. Each PIP contains a header part, a data part, and a checksum. The data part includes a pointer ID field that stores a pointer identifier to enable tracking of a plurality of pointers. The data unit also includes a pointer position parameter that specifies the pointer x position and the pointer tip parameter that specifies the pointer z position. The contact state field stores a value indicating whether or not the pointer is in contact with the touch surface 60, and enables detection of the raised pointer.

Upon receiving the PIP, the master controller 54 processes the PIP using triangulation to determine the position of each pointer in the captured image with respect to the touch plane 60 in the (x, y) coordinate system. In this way, since the PIP is generated in response to the captured image, the position and movement of the pointer above the touch surface 60 can be tracked. The pointers are, for example, the user's finger, a cylindrical tool, a passive or active pen tool or delete tool, or any other suitable object, as the image data is processed to detect the presence of one or more pointers. It can take any suitable form such as. Details of how the image data is acquired by the digital camera 70 and processed by the master controller 54 are based on the Patent Cooperation Treaty filed on July 5, 2001 and published as WO 02/03316 on January 10, 2002. It is described in the international application PCT / CA01 / 00980, the contents of which are incorporated herein by reference. Therefore, the details will not be described further here.

The master controller 54 outputs the generated pointer data to the computer 56 that identifies the position of each pointer with respect to the touch surface as each pointer approaches and / or touches the touch surface 60 and moves on the touch surface 60. The driver loaded into the computer 56 receives the pointer data, examines the pointer data, and determines whether the pointer data corresponding to the known input gesture stored in the gesture library has been generated. In particular, the driver examines the pointer data to detect the presence of multiple pointers in the captured image, and then examines the characteristics of the multiple pointers, such as right-click gestures, scroll gestures, rotation gestures, etc. Determine if a known gesture such as has been performed. When a gesture is being executed, the driver raises a command (event) related to the determined gesture and sends the command to the active application program being executed by computer 56.

Here, with reference to FIGS. 3a to 4f, specific examples of gestures that can be recognized by the touch system and generated actions that are executed in response to the recognized gestures are shown.

<u style="single">Intuitive right-click gesture:</u> Figures 3a through 3d show an intuitive right-click gesture. When the user touches the touch surface 60 on the application displayed on the touch surface with a finger, the driver recognizes the contact as a left-click mouse event and introduces the left-click mouse event into the application. If the user maintains contact with one finger, then contacts the touch surface 60 with the other finger, and the subsequent contact is to the right of and in close proximity to the first contact, the driver will be second. Recognizes the touch surface contact of the right-click gesture and introduces a right-click event into the application. In response to a right-click event, the application opens and displays a drop-down menu (not shown). It will be appreciated that this allows the user to use a hand gesture to activate the right-click action in the same way that was performed when the user used the mouse to activate the right-click action. Figures 3a through 3d show an intuitive right-click gesture performed using two fingers of the same hand, but the right-click gesture can be performed using the fingers of different hands. Will be understood.

<u style="single">Gestures that scroll up / down and left / right:</u> Figures 4a through 4e show up (up) / down (down) and left (left) / right (right) scroll gestures. If the user simultaneously touches the touch surface 60 on the application window displayed on the touch surface with a pair of fingers, and the pair of fingers are placed close to each other in the horizontal direction, the driver will perform the operation. Recognize simultaneous finger touches as scroll gestures and introduce scroll events into your application. The pointer position data sent by the driver to the application in response to the next vertical movement of the finger is interpreted by the application as either a scroll-up (scroll up) or scroll-down (scroll down) command. The application responds to scroll-up or scroll-down commands by moving the information presented in the application window in the direction of vertical movement. The pointer position data sent from the driver to the application in response to the subsequent horizontal movement of the finger is interpreted by the application as a horizontal scroll command. The application responds to the horizontal scroll command by moving the information displayed in the application window to the side corresponding to the horizontal movement. Figures 4a-4f show scrolling gestures performed using two fingers of the same hand, but it is understood that scrolling gestures can be performed using the fingers of different hands. right.

Although not shown, many other gestures can be used to generate commands that are recognized by the driver and control the application running on the computer 56. Here, specific examples of such other gestures will be described.

<u style="single">Page Up / Down Gestures:</u> If the user simultaneously touches the touch surface 60 on the application window displayed on the touch surface with three fingers, and the three fingers are placed close to each other in the horizontal direction, the driver will perform the driver. The simultaneous finger contact is recognized as a page gesture, and a page event is introduced into the application. The pointer position data sent from the driver to the application in response to the subsequent vertical movement of the finger is interpreted by the application as a page up (previous page) or page down (next page) command according to the vertical movement. .. The application responds to page-up or page-down commands by moving the information displayed in the window in the appropriate direction.

<u style="single">Rotating gesture:</u> The user touches the touch surface 60 on the object displayed in the application window with one finger, then touches the touch surface with the other finger, and bows while maintaining contact with the touch surface with one finger. When the other finger is moved, the driver recognizes the bow-shaped movement of the second finger as a rotating gesture. The driver then introduces a rotation command into the application to rotate the object around the point of contact defined by the first finger in the direction of the bow and equal to the path of the arch.

<u style="single">Zoom Gesture:</u> When the user simultaneously touches the touch surface 60 on the application window with a pair of closely spaced fingers, increasing the distance between the fingers almost horizontally, the driver recognizes the finger movement as a zoom out gesture. .. The driver then introduces a zoom out command into the application to increase the amount of information displayed to the application in the application window. When the user simultaneously touches the touch surface 60 on the application window with a pair of distant fingers and moves the fingers almost horizontally toward each other, the driver recognizes the finger movement as a zoom-in gesture. To do. The driver then introduces a zoom-in command into the application to reduce the amount of information presented to the application in the application window.

Alternatively, the zoom out and zoom in commands can be represented by other gestures. For example, when the user touches the touch surface 60 with a grabbing hand holding the fingers bundled together on the application window and spreads the fingers outward, the driver moves the fingers. Is recognized as a zoom-out gesture.

When the user touches the touch surface 60 with a nearly flat hand with spread fingers on the application window, squeezes the hands by grabbing the fingers inward and bundles them together, the driver moves the fingers. Recognize as a zoom-in gesture.

<u style="single">Enlarged gesture:</u> When the user simultaneously touches the touch surface 60 on the application window with a pair of closely spaced fingers and increases the distance between the fingers approximately diagonally, the driver recognizes the finger movement as a window size enlargement gesture. To do. The driver then introduces a window size enlargement command into the application to cause the application to increase the size of the application window. When the user simultaneously touches the touch surface 60 on the application window with a pair of distantly spaced fingers and moves the fingers approximately diagonally toward each other, the driver reduces the window size gesture of the finger movement. Recognize as. The driver then introduces a window size reduction command into the application to cause the application to reduce the size of the application window.

<u style="single">Icon selection and open gestures:</u> When the user simultaneously touches the touch surface 60 on the icon with two closely spaced fingers, the driver recognizes the finger contact as a double-click gesture. The driver then issues an open application command, causing the desktop application running on computer 56 to open the selected application.

<u style="single">Object / Window Move Gesture:</u> When the user moves a pair of closely spaced fingers above the touch plane 60 and above the object displayed in the application window, the driver translates the movement of the floating finger. Recognize as a gesture. The driver then introduces the object translation command into the application to translate the object displayed in the direction in which the finger is moved and by an amount equal to the distance.

When the user moves three closely spaced fingers above the touch plane 60 and above the application window, the driver recognizes the raised finger movement as a window translation gesture. The driver then issues a window translation command to translate the application window to the desktop application running on computer 56 in the direction in which the finger is moved and by an amount equal to that distance.

The above gestures have been described as being recognized in response to multiple finger touches or ascents, but the same gesture can be recognized when other objects are used to perform the gesture. it can. For example, you can use multiple pen tools to perform gestures, or instead, you can use one finger and one pen tool to perform gestures.

Different pointer characteristics can also be used to extend the functionality of recognized gestures. For example, in the case of a scroll gesture, the angle at which the pointer contacts the touch surface 60 can be used to determine the speed at which the displayed information is scrolled. A pointer in contact with the touch surface 60 at a steep angle may represent a slow scrolling speed, whereas a pointer in contact with the touch surface 60 at a shallow angle may represent a high scroll speed.

The type of pointer used by the touch system to make contact with the touch surface 60, as described in co-pending U.S. Patent Application No. 10 / 384,783, filed March 11, 2003. Determine pointer characteristics if and / or as described in co-pending U.S. Patent Application Nos. 10/294, 917 filed November 15, 2002. If possible, and these contents are incorporated here by reference, different functions can be assigned to similar gestures performed using different pointers. For example, in the case of the rotation gesture described above, when the same gesture is performed using the finger and pen tool that draws the first contact with the object in the application window, the driver will touch the finger and move the pen. Is recognized as a pattern fill gesture rather than a rotation gesture. A finger contact followed by a closely spaced pen tool contact represents a circular gesture rather than a scrolling gesture, which is a finger contact followed by a closely spaced eraser. The contact can also represent a page erase gesture. As can be seen from the above description, the ability to discriminate between multiple pointers close to the touch plane 60 provides the functionality provided by performing the same gesture using only individual identifiable pointers. Significantly improve.

The driver was described as examining the pointer data to determine if the pointer data was generated in response to a known gesture, but if the active application running on the computer has the ability to recognize the gesture. Can be understood by those skilled in the art that pointer data can be sent to the active application for gesture recognition.

If desired, the touch surface 60 can be divided into a plurality of areas so that a plurality of users can interact with the touch surface at the same time without causing ambiguity between user inputs. In this case, a plurality of contacts to the touch surface or a plurality of ascents on the touch surface larger than the threshold distance are treated as a plurality of user inputs. Multiple contacts to or ascending to a touch surface within a threshold distance are treated as multiple contacts made by a single user, and whether the multiple contacts represent a gesture. Examined to judge.

Although preferred embodiments of the present invention have been described, those skilled in the art will appreciate that modifications and modifications can be made without departing from the spirit and scope of the invention as set forth in the appended claims.

<figref num="1">It is a schematic diagram of the camera-based touch system based on this invention.</figref><figref num="2">It is a front view of the touch screen which forms a part of the touch system of FIG.</figref><figref num="3a">It is a front perspective view of the touch screen of FIG. 2 showing a right-click input gesture.</figref><figref num="3b">It is a front perspective view of the touch screen of FIG. 2 showing a right-click input gesture.</figref><figref num="3c">It is a front perspective view of the touch screen of FIG. 2 showing a right-click input gesture.</figref><figref num="3d">It is a front perspective view of the touch screen of FIG. 2 showing a right-click input gesture.</figref><figref num="4a">It is a front perspective view of the touch screen of FIG. 2 showing the input gestures of up / down and left / right scrolling.</figref><figref num="4b">It is a front perspective view of the touch screen of FIG. 2 showing the input gestures of up / down and left / right scrolling.</figref><figref num="4c">It is a front perspective view of the touch screen of FIG. 2 showing the input gestures of up / down and left / right scrolling.</figref><figref num="4d">It is a front perspective view of the touch screen of FIG. 2 showing the input gestures of up / down and left / right scrolling.</figref><figref num="4e">It is a front perspective view of the touch screen of FIG. 2 showing the input gestures of up / down and left / right scrolling.</figref><figref num="4f">It is a front perspective view of the touch screen of FIG. 2 showing the input gestures of up / down and left / right scrolling.</figref>