Method for providing feedback responsive to sensing a physical presence proximate to a control of an electronic device

Technical Field The Technical Field The present invention relates to an input device for a computer system. More specifically, the present invention relates to the use of touch-detectable (hereinafter referred to as touch-sensitive) input devices in computer systems.

[0002] This application was filed on September 14, 1998, and was assigned serial number 09 / 152,434 and was assigned to the assignee of this application under the name "Proximity Sensor in a Computer Input Device" by Hinckley et al. This is a partial pending application of a US patent application. This application is incorporated herein by reference. This application was also filed on September 14, 1998, assigned serial number 09 / 152,432, and transferred to the assignee of this application, "A Techinique For Implementing a Two-Handed Desktop Use Interface For a" by Hinckley. It is a partial pending application of a simultaneously pending US patent application named "Computer". This application is incorporated herein by reference. The application was further filed on September 14, 1998, assigned serial number 09 / 152,443 and transferred to the assignee of the application, "Method of Interacting With a Computer" by Hinckley et al. It is a partial pending application of a simultaneously pending US patent application entitled "Using a Proximity Sensor in a Computer Input Device". This application is incorporated herein by reference.

[0003] In addition, this application was filed on November 25, 1998, assigned serial number 09 / 200,325, and transferred to the assignee of this application, "A Technique For Implementing an On-Demand Display Widget" by Hinckley. Through Controlled Fading Initiated By User Contact With a Touch Sensitive Input Device "is a partial pending application of a simultaneously pending US patent application. This application is incorporated herein by reference, but this application was filed on September 14, 1998 and assigned serial number 60 / 100,261, "Toolglass Improvements-On-Demand Tool Sheet; Gesture Through Tool". Claims priority over a US provisional patent application named "Sheets". This application was also filed on November 25, 1998, assigned serial number 09 / 200,321, and transferred to the assignee of this application, "A Technique For Implementing an On-Demand Tool Glass For Use in" by Hinckley. It is a partial pending application of a simultaneously pending US patent application named "a Desktop User Interface". This application is incorporated herein by reference, but this application is referred to as "Toolglass Improvements-On-Demand Tool Sheet; Gesturing Through Tool Sheets" filed on September 14, 1998 and assigned serial number 60 / 100,261. Claims priority over the US provisional patent application of the name.

[0004] This application was filed on April 5, 2000, assigned serial number 09 / 543,723, and was assigned to the assignee of this application, "Method of Interacting With a Computer Using a Proximity Sensor in" by Hinckley et al. Related to a co-pending US patent application named "a Computer Input Device". This application is incorporated herein by reference, but this application is a pending application at US Patent Application No. 09 / 152,443 as set forth above.

Related Technologies Input devices for computer systems such as keyboards, touchpads, trackballs, game controllers and mice include joysticks, touchpads, trackballs, headsets, microphones, buttons, knobs and rockers. Often includes one or more auxiliary controls, such as switches, triggers, sliders, wheels, and biometric sensors. The auxiliary control can be used alone or in combination with the input device and other auxiliary controls.

[0006] Most input devices use transducers or switches to generate input signals. Switches are typically found in keyboard keys, mouse buttons, joysticks, and game controllers. Transducers are located in mice and trackballs and generate electrical signals based on the movement of the ball inside such a device or by optically detecting the movement of the device on a surface. The transducer is also inside the headset, which in this case converts the audio signal into an electrical signal. The touchpad uses a sensor that provides an electrical signal when the user touches the touchpad to generate an input signal, which includes the location of the contact within the touchpad.

[0007] While it is desirable to increase the amount of information that the input device and its auxiliary controls can provide to the computer, the number of transducers and switches that can be added to the input device is a user-specific transducer. Is naturally limited by the ability to memorize all the functions performed by the switch or the switch, and in the real estate available to place the switch and transducer on the input device (eg keyboard or mouse), or at least in a viable form. It is limited by the practical aspect of the area that can be used. The user can determine the function of each auxiliary control, but this ability is also generally a fixed function (static) that the auxiliary control originally has or is close to. You're limited to label), launching each auxiliary control, clicking a button, selecting a menu option, requesting a help file, or reading the user manual. In addition, the functionality of the auxiliary controls may vary from game to game or application. Therefore, the fixed function of an auxiliary control becomes of little value if the function and status of the input device and the auxiliary control vary from application to application. In this regard, even if a user remembers the assignment of buttons and other controls in one game or application, another game or application may assign different functions to the same button or control. Similarly, the status of a function controlled by an auxiliary control can generally only be determined by invoking that control.

[0008] [Problems to be Solved by the Invention] Therefore, the user is provided with the ability to more easily determine the function and status of the auxiliary control regardless of its context (for example, an active application or game). Is needed to do.

To address this need, indicators such as LEDs and small LCDs can be incorporated directly into the input device and its auxiliary controls to provide feedback or status information. However, associating LEDs and LCDs with individual auxiliary controls increases both power consumption and input device costs. In addition, the output form of LEDs and LCDs is very limited and far from the main display (eg computer monitor), so users are always assisted from the screen to determine control functions and status. You have to move your gaze to the control. If the subject of attention is constantly moved in this way, the user must repeatedly grasp the context at that time, which may cause mental fatigue. Therefore, there is a need to provide a more efficient, convenient and / or cost-effective method for determining the function and / or status of an auxiliary control of an input device in a computer system.

[0010] The present invention senses a physical presence and uses visual feedback, auditory feedback (for example, voice or music), or tactile sensation in the display on the screen or the control itself. By providing a touch-sensitive input device with touch-sensitive auxiliary controls that provide feedback (eg, vibration), it overcomes many of the drawbacks found in existing input devices.

[0011] In one aspect of the invention, keys, buttons, knobs, rocker switches, or other input devices such as keyboards, touchpads, trackballs, game controllers, monitors, joysticks, steering wheels, headsets, or mice. Auxiliary controls can be supplemented with sensors that detect contact with the user's hand or its significant proximity. According to another aspect, the sense signal of the auxiliary control causes status, state information, and tool tips (tips) about the control that the user has come into contact with without being activated (eg, pushing, turning, rotating, or otherwise). ), Help text, or other on-screen display of feedback can be provided to the user. Alternatively, the user may be provided with auditory or tactile feedback in place of or in addition to the feedback provided by the on-screen display. Therefore, the user can know the assignment and status of those functions or get other feedback by simply touching various buttons and other controls. Such feedback allows the user to more accurately understand the consequences of the action performed by the button or control if the user subsequently chooses to activate the control. In addition, the user will be able to see the status information quickly and easily.

[0012] Aspects of the present invention display contextual feedback about auxiliary controls on the screen, such as a display widget (eg, a graphical user interface (GUI)), in response to detection of the user's hand. Including providing to. In this way, touch-sensitive controls can be used to predict the context of user behavior. The context of the situation can be determined by the user's preliminary actions such as grabbing, touching, or approaching the control. Knowing the context of the action, the computer can start working and predict the user's will.

[0013] According to another aspect of the invention, the context of the auxiliary control can include, but is not limited to, various applications such as games, utilities, and productivity applications. The context can also be varied for different parts of the application or game.

[0014] In another aspect of the invention, when the GUI is displayed in response to the detection of a physical presence in close proximity to or in contact with an auxiliary control, the user can enter another auxiliary control or input. You can interact with the GUI using the device. The input device includes, but is not limited to, a mouse, a touchpad, or a keypad. For example, when a GUI for volume control is displayed, the user can adjust the volume with the mouse.

[0015] In another aspect of the invention, the input device including the auxiliary control may include a touch sensor that directly detects when a physical entity (eg, the user) touches the device or the control. This can provide feedback display and dismissal in an "on-demand" manner, for example when the user establishes or breaks physical contact between the device and the hand.

[0016] For example, if there is a change in the contact display provided by the device that reflects the user touching the device or control by touching the device with a finger, a tool tip can be displayed. Any change that indicates that the user has released physical contact with the device, such as lifting a finger from the device, can remove the tooltip from the display. In one aspect of the invention, when such a change is detected, a predetermined animation sequence corresponding to the change can be started so as not to distract the user. This sequence takes place over a specified amount of time, where feedback begins to gradually become visible as soon as user contact begins (usually invisible, that is, completely transparent, and finally a given half. Feedback begins to disappear immediately (ie, eventually returns to its original invisible state) as soon as the user's contact with the device is released, for example when the user lifts his or her hand off the device. [0017] In another aspect of the present invention, the toolbar, scroll bar, etc. can be displayed on the display screen only in response to the detection of the physical existence. For example, you can display the toolbar on the screen by touching the auxiliary control without starting it, and you can turn off the toolbar by releasing the contact with the control. In one aspect of the invention, the user has access to an auxiliary control that displays the toolbar on the screen while interfacing with the toolbar using an auxiliary control or input device with one hand, if desired. You can keep in touch. According to this aspect, the clutter on the display can be reduced, and the range of available application screens can be adjusted in a manner that matches and is controlled by the user's actions at the appropriate time when executing the program. Can be expanded. At this time, the user is not perceived as having a significant burden of doing so. This is expected to significantly improve the "user experience".

[0018] Those skilled in the art will appreciate the above and other novel advantages, details, embodiments, features, and purposes of the present invention in the following detailed description, claims, and accompanying drawings of the present invention. Will be clear from. These matters described herein are useful in explaining the present invention.

[Embodiments of the Invention] According to an exemplary embodiment of the invention, the auxiliary control of the input device is such that when a physical entity (eg, a user's hand or stylus) touches the control, it senses it. It can be configured to provide display and deletion of display widgets in an "on-demand" manner. The display widget may be a graphical user interface (GUI) and can contain information such as features (eg tooltips) and / or status information in an "on-demand" fashion. Examples of display widgets include toolbars, toolglass sheets, scrollbars, window frames and window decorations, title bars, floating tool palettes, modal and non-modal dialog boxes, lists or combo boxes, controls, and button settings. , Text input area, etc. are included.

A change in the contact display provided by the input device that reflects the user making physical contact with the auxiliary control, such as touching the control with a finger, can cause the display widget to appear. Changes that indicate that the user has released contact with the auxiliary control, such as lifting a finger from the control, can dismiss the display widget. These detected changes can initiate a given animation sequence corresponding to them so as not to distract the user. This animation takes place over a specified amount of time, where the display widget begins to gradually become visible as soon as the user's contact begins (usually from an invisible state, that is, a completely transparent state, to the end. Becomes a given translucent state), and as soon as the user loses contact with the auxiliary control, such as lifting a finger from the control, the display widget begins to gradually disappear (ie, eventually the original appearance). Return to no state). In addition, depending on the particular touch-sensitive auxiliary control used, it may be preferable to start fading after a short time delay, i.e., a "cooling period". For example, when the user reaches the edge of the touchpad, the user may "re-clutch" the hand (for example, lift the finger for a short time and then reposition it in the center of the pad) to continue acting as a pointer. When this happens, it can be awkward for the display widget to start fading immediately. A short time delay (eg, about 0.5-1.) In combination with continuous finger contact sensing before the start of fading. If you set (0 seconds), you will be able to re-clutch for a short time as described above without changing the screen display at all. Similarly, if the user has not touched the auxiliary control for long enough, a similar time delay in combination with the perceived release of user contact before the start of the fade-in will inadvertently put a small amount of time Touching can prevent the display widget from fading in and out, preventing the user from being annoyed at another point.

[0021] Further, according to the present invention, for displaying and erasing display widgets (eg toolbars) "on demand", contact-induced display and erasure can also be readily used. This is done by sensing contact between the user's preferred hand and another touch-sensitive auxiliary control or input device such as a touch-sensitive mouse. When you reach for and grab an auxiliary control or input device by displaying a display widget in response to contact detection, specifically the detection that the user has grabbed the auxiliary control or input device. It is possible to utilize the user's desire to use the display widget that is potentially suitable for the user's behavior. The display widget disappears when the user releases the auxiliary control or input device, for example when the touch contact is stopped. Displaying and erasing display widgets in this way has the advantage that the additional recognizable burden on the user is negligible.

[0022] FIG. 1 and related description provide a concise description of a suitable computing environment in which the present invention can be practiced. The present invention, at least in part, is described in the general context of computer-executable instructions executed on a personal computer, such as program modules, but it is not essential. Program modules typically include routine programs, objects, components, data structures, etc. that perform specific tasks or perform specific abstract data types. Further, those skilled in the art will appreciate that the present invention can be implemented in other computer system configurations. This includes portable devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers and more. The present invention may also be implemented in a decentralized computing environment in which tasks are performed by remote processing devices linked through a communication network. For distributed computing environments, program modules may reside in either local or remote memory storage.

[0023] With reference to FIG. 1, an exemplary system for carrying out the present invention combines a processing unit (CPU) 21, a system memory 22, and various system components including the system memory 22 into the processing device 21. Includes an exemplary computer system using a conventional personal computer 20 including a system bus 23 to be used. The system bus 23 may be any of several bus structures, including a memory bus or memory controller using any of the various bus architectures, a peripheral bus, and a local bus. The system memory 22 includes a read-only memory (ROM) 24 and a random access memory (RAM) 25. The basic input / output (BIOS) 26 includes a basic routine that helps transfer information between elements in the personal computer 20 at startup or the like, and is stored in the ROM 24. The personal computer 20 further includes a hard disk drive 27 for reading and writing to and from a hard disk (not shown), a magnetic disk drive 28 for reading and writing to and from a removable magnetic disk 29, and a CD. Includes an optical disk drive 30 that reads or writes to and from a removable optical disk 31 such as a ROM or other optical medium. The hard disk drive 27, the magnetic disk drive 28, and the optical disk drive 30 are connected to the system bus 23 by the hard disk drive interface 32, the magnetic disk drive interface 33, and the optical disk drive interface 34, respectively. These drives and their associated computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other personal computer 20 data.

[0024] Although hard disks, removable magnetic disks 29, and removable optical disks 31 are used in the exemplary environments shown herein, those skilled in the art may use magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, and random. It will be appreciated that other types of computer-readable media capable of storing computer-accessible data, such as access memory (RAM) and read-only memory (ROM), can also be used in this exemplary operating environment.

A hard disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25 includes a plurality of programs including an operating system 35, one or more application programs 36, other program modules 37, program data 38, and device driver 60. Modules can be stored. The device driver 60 processes commands and information entered by the user through the input device 43. Input devices include keyboards, mice, game controllers, trackballs, touchpads and the like. These input devices also apply joysticks, gamepads, touchpads, trackballs, keys, headsets, monitors, microphones, buttons, knobs, rocker switches, triggers, sliders, wheels, levers, touch strips, and biometrics. It can also have an auxiliary control such as a sensor. The input device 43 can be coupled to the personal computer 20 by wire or wirelessly.

[0026] According to an exemplary embodiment of the invention, at least one of the input devices as described above may include a touch sensor 40, and an input device such as a mouse may have both a touch sensor 40 and a moving transducer 42. it can. The touch sensor 40 can generate a signal indicating when a physical entity such as a user's hand touches one of the input devices 43 itself or its auxiliary control. The moving transducer 42 can generate a signal indicating when the user moves a part of the input device. The signals generated by the touch sensor 40 and the mobile transducer 42 can be passed along the lead wire connected to the processing device 21 through the serial port interface 46 coupled to the system bus 23, but the connection is a sound card, a parallel port. , Game port or other interface such as Universal Serial Bus (USB).

[0027] A monitor 47 or other type of display device can also be connected to the system bus 23 via an interface such as a video adapter 48. In addition to the monitor 47, the personal computer can typically include other peripheral output devices such as a speaker 45 and a printer (not shown).

[0028] The personal computer 20 can operate in a network environment using a logical connection to one or more remote computers such as the remote computer 49. The remote computer 49 may be another personal computer, portable device, server, router, network PC, peer device or other network node, and Figure 1 shows only the memory storage device 50, but is typically above. It contains many or all of the elements described in connection with the personal computer 20. The logical connections shown in Figure 1 include a local area network (LAN) 51 and a wide area network (WAN) 52. Such networking environments are common in offices, enterprise-scale computer networks, intranets, and the Internet.

[0029] When used in a LAN networking environment, the personal computer 20 connects to the local area network 51 through a network interface or adapter 53. When used in a WAN networking environment, the personal computer 20 typically includes a modem 54 or other means for establishing communication over a wide area network 52 such as the Internet. The modem 54 may be external or internal and is connected to the system bus 23 via the serial port interface 46. In a network environment, the program module described in relation to the personal computer 20 or a part thereof can be stored in a remote memory storage device. It will be appreciated that the network connections shown here are exemplary and other means of establishing communication links between computers can be used. For example, wireless communication links can be set up between one or more parts of a network.

[0030] FIG. 2 is an enlarged block diagram of a part of an embodiment of the input device 43 of FIG. The input device 43 includes an array of four touch sensors 100, 102, 104, and 106. These sensors can accommodate four auxiliary controls on the input device. Each sensor produces an electrical signal through the individual leads 108, 110, 112, and 114, which are connected to analog to digital converters and multiplexers 116. The touch sensors 100, 102, 104, and 106 generate their respective electrical signals based on the actual contact between the user and a portion of the sensor, or based on the significant proximity of the user to the sensor. A touch sensor that relies on contact is called a contact sensor, and a touch sensor that relies on proximity is called a proximity sensor. In the context of this application, touching the touch sensor is when the touch sensor touches it, and in the case of a proximity sensor when the user is sufficiently close to the sensor. It should be understood that according to the present invention, the sensor can be configured such that the degree of contact required for "contact" and "release" is different. For example, in order to detect a "contact" event, contact is required, whereas in order to detect a "contact release" event, it is necessary that the contact is released and the proximity state disappears. Is. Similarly, in a pure proximity sensor embodiment, a very close proximity is required to detect a "contact" event, but the threshold (distance) required to detect a "disengagement" event. Can be higher.

[0031] In some contact sensor embodiments, the touch sensor comprises a conductive film whose capacitance changes upon touch. The sensor also includes a capacitive measurement circuit that produces an electrical signal based on changes in the capacitance of the conductive film. Those skilled in the art will recognize that other contact sensor technologies such as photodiode sensors, piezoelectric material sensors, and capacitive pressure sensors are also available. Any of these sensors can be used in the context of the present invention. In one embodiment of the proximity sensor, the touch sensor uses the reflected light from the LED to detect when the user approaches the sensor. The chip used to drive the LED and sense the reflected light according to this exemplary embodiment can be manufactured by Hamamatsu Corporation in Bridgewater, NJ. In another proximity sensor embodiment, changes in the electric or magnetic field near the input device are used to determine when the user approaches the device.

[0032] In an embodiment of the invention, the touch sensor can provide the same information regardless of where in the touch sensor the user touches the input device or which part of the sensor the user approaches. In this way, these touch sensors separate contact data and position data. FIG. 2 shows an exemplary input device with this type of touch sensor.

[0033] In another embodiment, for example using a touch pad, touch screen, and touch tablet, a given touch sensor is where the user made contact with the touch sensor, or the user made contact with the touch sensor. It is possible to provide position information indicating where the touch sensor is closest to. In such a device, position data cannot be specified without touching the device. Also, the device cannot be touched without specifying the position. Thus, in these devices, contact sensing and position sensing are inextricably linked.

[0034] Referring to FIG. 2, the analog to digital converter and multiplexer 116 converts the analog electrical signals at the wires 108, 110, 112, and 114 into digital values carried by the wire 118. The wire 118 is connected to the microcontroller 120, which controls the multiplexer 116 to selectively monitor the status of the four touch sensors. Microcontroller 120 also receives inputs from various other sensors in the input device. For the sake of brevity, these inputs are collectively shown as input 122. Those skilled in the art will recognize that different input devices and auxiliary controls provide different input signals depending on the type of motion sensor in the input device. Examples of motion sensors include switches that provide signals that represent the motion required to close the switch, microphones that provide signals that represent the movement of air generated by voice signals, mouse balls, trackballs, or mouse wheel movements. It includes an encoder wheel that provides a signal and a resistor wiper that provides an electrical signal that represents the movement of the joystick. Each of these motion sensors functions as an input generator capable of generating input information to be sent to a computer system. This input information can include pressable key states, pressable button states, sound information, or motion information, depending on the particular input generator.

Those skilled in the art will also recognize that the number of input lines connected to the microcontroller 120 depends on the number of sensors in the input device and the configuration of the input device. For example, in the case of a keyboard, the microcontroller uses the input line to determine if any of the auxiliary controls are activated. The microcontroller makes this determination by sequentially testing the state of each auxiliary control on the keyboard using a multiplexer (not shown). The techniques used to detect the input of these auxiliary control states are well known in the technical field of keyboards.

[0036] In a mouse or trackball, the input line 122 includes a line for detecting the switch on and a line for detecting the rotation of the encoder wheel. The switch is located below the mouse or trackball button. The encoder wheel grasps the movement of the mouse ball or trackball. Usually, one encoder wheel grasps the movement in the X direction and the other encoder wheel grasps the movement in the Y direction. In most embodiments, each encoder wheel has an input line to the microcontroller 120 associated with itself. Depending on the mouse, yet another encoder wheel keeps track of the rotation of the wheel at the top of the mouse.

[0037] In some mice, the movement of the mouse in the X and Y directions is grasped by a separate optical microprocessor connected to the microcontroller 120 through the wire 122. The optical microcontroller uses optical data to determine mouse movements. The optical microcontroller converts this optical data into motion values and transmits this to the microcontroller 120 along the input line 122.

[0038] In a game controller such as a game pad, the input line 122 includes a line for detecting the input of a plurality of switches on the game pad and a line for detecting the rotation of the wheel of the game pad. In the case of a joystick, the input line 122 can include a line connected to the resistance wiper of the joystick and the switch of the joystick. In headsets, line 122 includes multiple lines carrying a multi-bit digital value that represents the magnitude of the analog electrical signal generated by the microphone. Usually, an analog-to-digital converter produces this digital value. To reduce the weight of the headset, the analog to digital transducer and microcontroller 120 can be placed on top of a sound board located in the computer. To further reduce the weight of the headset, the multiplexer and A / D converter 116 in Figure 2 can also be implemented on top of the sound board.

[0039] The microcontroller 120 produces an output 124, which is provided for the serial port interface 46 of FIG. Typically, the output 124 is a serial digital value that indicates which motion or touch sensor was activated. For keyboards, this digital value contains a scan code that uniquely identifies the activated key and other auxiliary controls or touch sensors on the keyboard. For mice, this digital value contains a mouse packet, which indicates the current state of each mouse switch and each touch sensor, and the distance the mouse wheel and mouse ball have traveled since the last mouse packet was sent. Describe.

FIG. 3 is a perspective view of the headset 150 of the present invention. The headset 150 includes a microphone 152, a support piece 154, a touch sensor 156, and an output line 158. The support piece 154 is designed to loop around the user's ear and support the headset so that the microphone 152 is in front of the user's mouth.

The output line 158 propagates signals from the microphone 152 and the touch sensor 156. In some embodiments, the headset 150 is connected to a computer system that includes a speech recognition system. In such an embodiment, the speech recognition system is inactive unless the touch sensor 156 indicates that the user is touching the headset 150. Launching the speech recognition system involves loading the speech recognition system into random access memory the first time the user touches the headset 150. It can also include issuing instructions to the speech recognition system in the random access memory so that it can process the input speech signal. In either case, the present invention allows the speech recognition system to process irrelevant speech by activating the speech recognition system only when the headset 150 indicates that the user is touching the headset. The sex is reduced.

FIG. 4A is a perspective view of an embodiment of a mouse 170 according to an exemplary embodiment of the present invention. The mouse 170 includes a palm rest 172, a left button 174, a right button 176, a wheel 178, a side 180, and an output line 182. The two side areas 184 and 186 of the palm rest 172, the left button 174 and the side 180 are coated with separate conductive films. Each of the conductive films is connected to and is part of a separate sensor, such as the sensors 100, 102, 104 and 106 of FIG.

FIG. 4B shows a bottom view of the mouse 170. Mouse 170 includes a trackball 190 placed in a trackball nest 192. When the mouse 170 is moved over the surface by the force applied to the palm rest 172, side 180, left button 174, or right button 176, the trackball 190 rotates in the nest 192. With reference to Figure 4C, a pair of encoder wheels 194 and 196 detect this rotation.

FIG. 4C is a perspective view of a portion of the internal electronic circuit 189 of the mouse 170. The trackball 190 is omitted in FIG. 4C for the sake of clarity. The internal electronic circuit 189 includes encoders 194 and 196, which detect the movement of the two perpendicular trackballs 190. The encoder wheel produces an electrical signal to provide to the microcontroller 200. Microcontroller 200 also receives inputs from switches 202 and 204 located under left button 174 and right button 176, respectively. Switches 202 and 204 indicate when the left button 174 and right button 176 are pressed by the user, respectively. The microcontroller 200 also receives a signal from the switch 201 that indicates when the wheel 178 is pressed, and a signal from the encoder wheel 203 that indicates the rotational movement of the wheel 178. The microcontroller 200 also receives electrical signals from the palm rest 172, left button 174, and four sensors attached to the conductive film on the side areas 184 and 186 of FIG. 4A. In FIG. 4C, these four sensors are collectively referred to as a sensor array 206. The left button 174 and the side areas 184 and 186 may be auxiliary controls that activate when pressed to perform a function.

[0045] As described above, the mouse according to the exemplary embodiment of the present invention can detect when a specific area of the mouse is touched and when a part of the mouse or the whole mouse is moved. it can. Specifically, conductive films on the palm rest 172, left button 174, and side areas 184 and 186 indicate it when the user touches these areas. Note that the sensor associated with the conductive film in Figure 4A produces an electrical signal when the user touches the mouse, without the user moving the mouse or pressing a button. Encoder wheels 194 and 196 generate one type of (a separate) signal when the user moves the mouse, and switches 202, 204, and 201 are separate when the user presses buttons 174 and 176, and wheel 178, respectively. Generates an electrical signal. Thus, the exemplary mouse according to the invention adds functionality without the need for further manual dexterity required to operate its controls.

[0046] In an alternative embodiment of the present invention, instead of the trackball 190 and the encoder wheels 194 and 196, a solid position tracking device that collects an image of the surface on which the mouse has moved and determines a change in the position of the mouse is used. According to this exemplary embodiment, the mouse typically moves by comparing various images with a light source used to illuminate the surface and an optical system used to collect an image of the surface. Includes a processor used to determine if it is moving and, if it is moving, in which direction it is moving. Since the solid-state position tracking device converts motion into an electrical signal, it can be regarded as a sophisticated transducer or motion sensor.

FIGS. 5, 6A-6D, 7A-7B, 8A-8B, 9A-9C, 10A-10H, 11A-11B, 12A-12B, 13A-13D, and 14A-14D are exemplary embodiments of the invention. It shows an alternative configuration of a mouse by morphology. FIG. 5 is a top view of a mouse having a touch sensor only on the palm rest 600. Figures 6A and 6B show embodiments of separate exemplary mice with a palm rest and a sensor on the left side of the mouse, respectively. In FIG. 6A, which is a side view, a single sensor 602 covers both the palm rest and the left side of the mouse. FIG. 6B is also a side view, in which case one sensor covers the palm rest 604 and another sensor covers the left side 606.

[0048] Figures 6C and 6D show separate exemplary mouse embodiments of the invention, each having a sensor on the palm rest and the right side of the mouse. In FIG. 6C, which is a right side view, a single sensor 603 covers both the right side and the palm rest. Figure 6D is also a right side view, in which case one sensor 605 covers the palm rest and another sensor 607 covers the right side.

7A and 7B are side and top views of an exemplary mouse embodiment having a single sensor 608 on the left side of the palm rest and mouse and another sensor 610 on the left button of the mouse, respectively. is there. 8A and 8B are side views and top surfaces of an exemplary mouse embodiment having a single touch sensor 612 on the palm rest and left side of the mouse, a touch sensor 614 on the left button of the mouse, and a touch sensor 616 on the right button of the mouse, respectively. The figure is shown.

[0050] FIGS. 9A-9C show a left side view, a top view, and a right side view of the exemplary mouse 690 of the present invention, respectively. The mouse 690 includes a left side sensor 692, a palm sensor 694, a right side sensor 696, and a button sensor 698. In the mouse 690, the right side sensor 696 and the left side sensor 692 are separate from the palm sensor 694. In another exemplary embodiment of the invention, the three sensors can be formed as a single sensor.

[0051] FIGS. 10A-10H show top views of various exemplary mouse embodiments, showing possible configurations for a touch sensor on the left button of a mouse. These button configurations may be placed alone on the mouse or in combination with other sensors in other parts of the mouse. FIG. 10A shows a single solid-state sensor 618 across the left button. FIG. 10B shows a set of six sensor strips 620, each sensor strip generating a separate electrical signal when touched. FIG. 10C shows two regions 624 and 626 separated by a ridge 628. Areas 624 and 626 both end at the front end 627 of button 622. FIG. 10D shows two regions 634 and 637 separated by a ridge 636, both of which end at the side edge 631 of the button 630. The configuration of buttons 622 and 630 is particularly useful as you walk through the documents as described below. FIG. 10E shows a button configuration of a button 640 with four separate sensor areas formed as rectangles 641, 642, 643, and 644. In some embodiments, the line separating the four sensor regions is formed as a ridge that has a different shape feature than the sensor region. Figure 10F also shows four separate sensors on the button 646. In FIG. 10F, three sensor areas 650, 651, and 652 are at the front end of the button 646, with the remaining sensor 648 covering the rest of the button 646. Figure 10G shows a button 660 with nine sensor areas arranged in a keypad-like layout. FIG. 10H shows a button 670 with an outer circle consisting of eight sensors 672 surrounding the central sensor 674. The configuration of button 670 is particularly useful when manipulating radial menus.

[0052] FIGS. 11A and 11B show exemplary mouse embodiments that include separate sensors on both buttons of the mouse. In Figure 11A, buttons 700 and 702 have sensors, but palm rest 704 does not. In Figure 11B, buttons 706 and 708 and palm rest 710 have separate sensors.

[0053] FIGS. 12A and 12B show exemplary mouse embodiments with multiple sensors along the right side surface of the mouse. FIG. 12A is a right side view, with two sensors 720 and 722 on the right side. In Figure 12B, there are three sensors 724, 726, and 728 on the right side.

[0054] FIGS. 13A-13D show side views of an exemplary mouse embodiment having a plurality of sensors on the left side of the mouse. The mouse of FIG. 13A has two sensors 734 and 736 on the left side surface. The mouse of FIG. 13B has three touch sensors 738, 740, and 742, each separated at regular intervals. The mouse in Figure 13C also has three touch sensors on the left side. However, in FIG. 13C, the touch sensor 744 in the middle, which is located between the sensors 746 and 748, has a raised surface and is formed as a raised portion between the sensors 746 and 748. The raised surface of the sensor 744 provides tactile feedback to the user so that the user can position the thumb without looking at the mouse. FIG. 13D shows a mouse embodiment with multiple strips 752 on the left side of the mouse.

[0055] All exemplary embodiments of FIGS. 12A-12B and 13A-13D can be performed in conjunction with a sensor placed on the palm rest and / or a sensor placed on the left button and / or a sensor placed on the right button. Please note.

[0056] FIGS. 14A-14D are top views of an exemplary mouse embodiment in which the touch sensor is close to the wheel of the mouse. In Figure 14A, the touch sensor is placed directly on the wheel 760. In FIG. 14B, one touch sensor 762 is placed at the front of the wheel 764 and one touch sensor 766 is placed at the rear of the wheel 764. In the embodiment of FIG. 14B, the wheel 764 does not have a touch sensor. In FIG. 14C, one touch sensor 770 is placed at the front of the wheel 768 and one touch sensor 772 is placed at the rear of the wheel 768. In addition, wheel 768 includes a touch sensor. In the embodiment shown in FIG. 14D, the touch sensors are arranged in the wheel 774, the front area 776 in front of the wheel 774, the rear area 778 behind the wheel 774, and the palm rest 780.

[0057] In particular, there are various positions of the touch sensor in FIGS. 5, 6A to 6D, 7A to 7B, 8A to 8B, 9A to 9C, 10A to 10H, 11A to 11B, 12A to 12B, 13A to 13D, and 14A to 14D. Although exemplary embodiments have been described, it should be noted that the sensor may be included in other positions. For example, it is possible to combine some or all of the touch sensors exemplified in one embodiment with some or all of the touch sensors exemplified in another embodiment. In addition, these include the aspects of the mice shown in FIGS. 5, 6A-6D, 7A-7B, 8A-8B, 9A-9C, 10A-10H, 11A-11B, 12A-12B, 13A-13D, and 14A-14D. It should be understood that many of the touch sensor positions, but not limited to, may be a conductive film over an auxiliary control, in which case the control can be pressed to activate and perform a function. In this example, touching a control produces a set of electrical signals, and activating the control produces a second set of electrical signals. In certain exemplary embodiments of the invention, various mice, including their auxiliary controls (eg, buttons, wheels), display a physical entity (eg, a finger) that touches its touch-sensitive surface, ie, a display widget. It is possible to detect the physical presence that represents an explicit user request to display above and / or generate other (eg, voice or tactile) feedback. This feedback can provide the user with information such as status information, control features, and help text. This information may vary from application to application. In some embodiments of the present invention, the position of the display widget on the display screen can be displayed according to the movement of the physical entity on the touch-sensitive surface.

[0058] FIG. 15 is a perspective view of an exemplary trackball 220 of the present invention. The trackball 220 includes a base 222, buttons 224 and 226, and a ball 228. In one embodiment of the invention, the trackball 228 can be coated with a conductive film that contacts three rotating metal wheels (not shown) in the base 222. One of the metal wheels comes in contact with a conductive sheet located at the rear of the wheel and pressed against the wheel by a spring force. The conductive sheet is further connected to a touch sensor, which produces an electrical signal when the user touches the trackball 228. The other two wheels in the base 222 form two perpendicular motion sensors (not shown) that can capture the rotational motion of the trackball 228 in the base 222. The base 222 includes two switches under the buttons 224 and 226, which can generate an electrical signal when the user presses the buttons 224 and 226. Thus, the trackball 220 provides one type of electrical signal based on the user simply touching the ball 228 and another based on the user moving the trackball 228 or pressing a button 224 or 226. Can provide electrical signals. Also, one or more auxiliary controls that can perform a function upon activation may be placed around the trackball 228 on the base 222. The auxiliary control can have a touch-sensitive conductive film on its surface, which can be activated and perform a function in response to a user pressing the control. Therefore, when the control is touched, it produces one set of electrical signals, and when the control is activated, it produces a second set of electrical signals. In an exemplary embodiment of the invention, the trackball and its auxiliary controls (eg, buttons) display a physical entity (eg, a finger) that touches its touch-sensitive surface, i.e., a display widget on the display screen. and / Or other physical presence that represents an explicit user request to generate feedback (eg, voice, tactile) can be detected. Feedback can provide users with information such as status information, control features, and help text. This information may vary from application to application. In some embodiments of the present invention, the position of the display widget on the display screen can be displayed according to the movement of the physical entity on the touch-sensitive surface.

FIG. 16 is a perspective view of an exemplary game controller with a joystick, which includes a base 242, a handle 244, a trigger 246, and buttons 248, 250, 252. In one embodiment of the invention, the trigger 246 can be coated with a conductive film connected to the touch sensor in the base 242. In yet another embodiment, the button 248 can also be coated with a conductive film connected to another touch sensor in the base 242. Triggers 246, and buttons 248, 250, 252 may also be connected to switches that provide a separate electrical signal when the user presses an individual button or trigger. The handle 244 can be connected to a set of transducers that track the relative movement of the handle 244 with respect to the base 242. Thus, the game controller provides one set of electrical signals when the user touches the trigger 246 or button 248, and another electrical signal when the user moves the handle 244, or trigger 246, buttons 248, 250, 252. A set can be provided.

FIG. 17 is a perspective view of another exemplary game controller according to the present invention. FIG. 17 shows a game controller in the form of a game pad 260. The gamepad 260 has side buttons 262 and 264, left hand buttons 266, 268, 270, 272, 274, 276, and right hand buttons 278, 280, 282, 284, 286, 288. Further, the game pad 260 has a start button 290 and a select button 292. In some embodiments of the invention, the side buttons 262 and 264 can each be coated with a conductive film connected to a separate touch sensor in the gamepad 260. The gamepad 260 may also include multiple switches, one for each button on the gamepad. Thus, in some embodiments, the gamepad 260 can provide a set of signals to indicate when the user touches the side buttons 262 and 264, which when the user presses a button on the gamepad 260. A second set of electrical signals can be provided.

[0061] In an exemplary embodiment of the invention, a game controller as shown in FIGS. 16 and 17, including its auxiliary controls (eg, buttons, joysticks, triggers), is a physical entity (eg, a finger) that touches its touch-sensitive surface. ), That is, it is possible to detect the physical presence representing an explicit user request to display the display widget on the display screen and / or to generate other feedback (eg, by voice, tactile). Feedback can provide users with information such as status information, control features, and help text. This information may vary from application to application. In some embodiments of the present invention, the position of the display widget on the display screen can be displayed according to the movement of the physical entity on the touch-sensitive surface.

[0062] FIG. 18A shows a keyboard 300 according to an exemplary embodiment of the present invention. The keyboard 300 has a typical QWERTY array 302 on the left side of the keyboard and a numeric keypad 304 on the right side. The number keypad 304 contains the numbers 0-9, which are arranged in a 3x3 frame. In some embodiments, all nine of these keys can be coated with a conductive film. In another embodiment, the other keys of the keyboard and the auxiliary controls can be covered with a conductive film. The conductive film on each key is connected to and part of a separate touch sensor in the keyboard 300. The fact that each key has a conductive film means that each key can provide two types of signals. One signal can be provided when the user touches without pressing a key, and a second signal can be provided when the user presses a key.

[0063] The touch sensor may be further placed on the keyboard case (casing 301) in the lower part 306 and 307 of the space bar 308, the lower part 309 of the arrow key 310, and the lower part 311 of the keypad 304. it can. The arrow key 310 is typically used by the user to move the cursor while it is displayed. A keyboard 300 is shown above the keys and with touch sensors in parts 306, 307, 309, 311 but in another exemplary embodiment of the invention, only the keys have touch sensors or parts 306. , 307, 309, 311 may have only one touch sensor. In another exemplary embodiment, such touch sensors can be placed on the keyboard 300 in different combinations. Further, some or all of the touch sensors of parts 306, 307, 309, 311 may be proximity sensors. Touch sensors 306, 307, 309, 311 provide a variety of auxiliary controls including, but not limited to, joysticks, gamepads, touchpads, trackballs, buttons, knobs, rocker switches, triggers, sliders, wheels, levers, etc. Can be represented. The proximity sensor can detect when the user's hand approaches the sensor, without the need for the hand to actually touch the sensor. In general, if the user touches the auxiliary control but does not activate it (eg, push, turn, rotate), one type of signal is provided, and when the user activates the auxiliary control, a second signal is provided. To. In an exemplary embodiment of the invention, the keyboard, including its auxiliary controls, displays a physical entity (eg, a finger) that touches its touch-sensitive surface, i.e., a display widget on the display screen, and / or ( It is possible to detect a physical entity that represents an explicit user request to generate other feedback (eg, by voice, tactile). Feedback can provide users with information such as status information, control features, and help text. This information may vary from application to application. In some embodiments of the present invention, the position of the display widget on the display screen can be displayed according to the movement of the physical entity on the touch-sensitive surface.

[0064] FIG. 18B shows a portion of another exemplary keyboard according to the invention, which has auxiliary control inputs including buttons 312A-312G and a touch-sensitive control knob 314. Buttons 312A to 312G each have a touch sensor. FIG. 18C shows yet another exemplary keyboard according to the invention, which includes a touch-sensitive rocker switch 315 and touch-sensitive buttons 312A-312G, respectively.

FIG. 19 is a perspective view of an exemplary touchpad 2000 according to the present invention. This touchpad is currently Synaptics in San Jose, California. You can use the traditional touchpad available from Corporation and you don't need to make any changes. The touchpad 2000 consists of a touch-sensitive surface (tablet) 2100, which, when the user's finger touches the surface and moves over it, provides the two-dimensional position and contact area (surface area) of the finger. Sense. In addition, the touchpad 2000 includes buttons 2220, 2240, 2260, and 2280 located around touch-sensitive surfaces. One or more of the buttons 2220, 2240, 2260, 2280 may be touch sensitive. The output of this touchpad can be routed to a computer system as shown in Figure 1. In embodiments of the invention, the touchpad containing its auxiliary controls displays a physical entity (eg, a finger) that touches its touch-sensitive surface, i.e., a display widget on a display screen, and / or (eg,). It is possible to detect the physical presence that represents an explicit user request to generate other feedback (speech, tactile). Feedback can provide users with information such as status information, control features, and help text. This information may vary from application to application. In some embodiments of the present invention, the position of the display widget on the display screen can be displayed according to the movement of the physical entity on the touch-sensitive surface.

As described above, input device keys, buttons, knobs, rocker switches, or other auxiliary controls include, but are not limited to, the user's hand (eg, finger or palm) or stylus. It can be supplemented with a sensor that detects the contact of a target or its significant proximity. So far, we have provided exemplary input devices including a mouse (Figures 4A-14B), a trackball (Figure 15), a game controller (Figures 16 and 17), a keyboard (Figures 18A-18C), and a touchpad (Figure 19). Shown and explained it. According to an embodiment of the present invention, the sensed signal as described above can be used to provide the user with a screen display, state information, or other feedback of the status associated with the controller touched by the user. ..

[0067] FIG. 20 is a more detailed block diagram of computer 20 useful for explaining a message routing system according to an exemplary embodiment of the present invention. In FIG. 20, the input device 43 provides a serial binary signal to the serial interface 46. The input device 43 may include any of the input devices having a touch sensor and its auxiliary controls described above.

The serial interface 46 converts the serial binary signal from the input device 43 into a parallel multi-bit value and passes it to the device driver 60. In many embodiments of the present invention, the device driver 60 can be implemented as a software routine executed by the CPU 21 of FIG. In such an embodiment, the device driver 60 can be unique to the input device and designed to interact with a particular input device and its auxiliary controls based on a designated protocol. Therefore, when the input device 43 is a mouse, the device driver 60 is a mouse driver designed to receive mouse packets generated by the mouse using the mouse packet protocol. When the input device 43 is a keyboard, the device driver 60 is a keyboard driver designed to receive a keyboard scan code indicating that a key is being pressed or a touch sensor is being touched.

[0069] The device driver 60 converts the multi-bit value into a device message based on the specified protocol and passes it to the operating system 35. This device message represents what kind of event occurred on the input device. For example, if a mouse touch sensor is touched, this message indicates that a particular sensor is being touched. When the hand is released from the touch sensor, another message indicating that the hand is released from the touch sensor is generated by the device driver 60.

The message generated by the device driver 60 is provided to the operating system 35, which controls the routing of this message. According to an exemplary embodiment, device messages are typically sent to focus application 812. A focus application is typically an application that has a window at the top of the display.

[0071] In some exemplary embodiments, operating system 35 maintains a list of message hook procedures registered with that operating system. In such an exemplary embodiment, the operating system 35 passes the device message sequentially to each of the listed message hook procedures before sending it to the focus application 812. Such a message hook procedure is comprehensively shown as the message hook procedure 810 in FIG. Most message hook procedures simply evaluate device messages to determine if any action should be taken. The message hook procedure returns a value to operating system 35 when it evaluates a device message, indicating that the operating system passes the device message to the next procedure in the list. Some message hook procedures have the ability to "eat" device messages, which indicates that the operating system should not pass the device message to other message hook procedures or focus applications. Do it by returning to.

The message hook procedure and focus application initiate various functions described below, particularly with a device message indicating that the touch sensor has been touched.

[0073] For example, FIGS. 21 and 22 show a device message generated based on a signal from an input device according to an exemplary embodiment of the invention, such as mouse 170 or trackball 220, respectively in FIGS. 4A and 15. It is an image of a screen displayed by various applications of the present invention using. FIG. 21 represents an image of screen 320 showing a virtual desktop 322. The virtual desktop 322 contains images of icons 324 and 326, as well as an open window 328. The open window 328 relates to a word processing application known as Microsoft Word® manufactured by Microsoft Corporation in Redmond, Washington.

[0074] In window 328, Caret 330 is placed in the text of the opened document. The position of the cullet 330 can be determined by moving the mouse 170 or the ball 228 of the trackball 220. In FIG. 21, Caret 330 is shown as a vertical line extending between two shorter horizontal lines. Those skilled in the art will recognize that the caret 330 can be in a variety of shapes and will typically appear as an arrow on the desktop 322.

[0075] Tooltip 332 is displayed depending on the position of caret 330 in the text of window 328. Tooltip 332 shows who entered the word below Caret 330.

Window 328 also includes a toolbar 334 containing drawing tools that can be used when drawing pictures in the document of window 328.

[0077] According to an exemplary embodiment of the invention as shown in FIG. 21, caret 330, tooltip 332, and toolbar 334 are assigned to the user to provide tooltips, which is an application of this word processing. It is displayed in window 328 only while touching a part of the input device such as an auxiliary control (for example, a button or a combination of buttons). If the user has not touched the input device, the caret 330, tooltip 332, and toolbar 334 disappear. FIG. 22 shows an image of the display 320 when the user is not touching a portion of the input device. By eliminating the toolbar 334, caret 330, and tooltip 332 when the user is not touching the input device, the present invention reduces the clutter found in window 328 and facilitates the document presented in window 328 by the user. Make it readable.

Those skilled in the art will recognize that the erasure of the caret 330, tooltip 332, and toolbar 334 when the user is not touching the input device can be individually controlled. Therefore, the user can customize the window 328 so that the tooltip 332 and the toolbar 334 disappear when the user releases the input device, but the caret 330 remains visible. In addition, you can control the speed at which items are displayed and redisplayed. Therefore, when the user releases the input device and then touches it, the image can gradually disappear from the display and gradually return to the display. In some exemplary embodiments of the invention, the fade-out period is 0.7-1.0 seconds, the cullet fade-in period is 0 seconds, and the toolbar is displayed instantly so as not to distract the user. The fade-in period of is 0.3 seconds. In certain embodiments, the fade-out time can be a variable that can be set by the user using a defined default period.

[0079] FIGS. 23 to 25 show a series of exemplary display screens including a pull-down menu that can display the result of a keyboard message from the keyboard 300 of FIG. 18A. Specifically, in screen image 350 of FIG. 23, the application is generating an active window 352 on the virtual desktop 354 containing an image of pull-down menu 356. The pull-down menu 356 is associated with a menu heading named "Tools" in the menu bar 358. The pull-down menu 356 is displayed in response to a keyboard message indicating that the user is touching but not pressing one of the keys on the numeric keypad 304 on the keyboard 300.

[0080] In another exemplary embodiment, the user can move the menu bar 358 left and right by using the keys representing the numbers "4" and "6" on the numeric keypad 304. Users can move the menu bar 358 sideways to display different pull-down menus for each menu heading. Specifically, the user causes the application to send a keyboard message by touching the key representing the number "4", and the application displays the menu heading to the left of the current menu heading in header menu 358. To change. Therefore, if the pull-down menu for the menu heading "Tools" is currently displayed in window 352, touching the key representing the number "4" will display the pull-down menu associated with the menu heading "Insert". Similarly, the user can display the pull-down menu of the menu heading to the right of the current menu heading by touching the key representing the number "6" on the numeric keypad 304. Therefore, if the currently displayed pull-down menu is associated with the menu heading "Tools" and the user touches the key representing the number "6", it will be associated with the menu heading "Format" in the header menu 358. The pull-down menu will be displayed. This is shown in FIG. 24, where a pull-down menu 360 for the menu heading "Format" 358 is displayed.

[0081] The user can move up and down in a pull-down menu such as the pull-down menu 360 by touching the keys representing the numbers "2" and "8" on the number keypad 304. As the user navigates through the pull-down menu, different items in the pull-down menu are highlighted. An example of the highlighted item is item 362 in FIG. 25, but in this figure the item "Tabs" in the pull-down window 360 is highlighted as the current item. If the user touches a key representing the number "8" when item 362 is the current item, the application that receives the keyboard message associated with it will high item 364 above item 362 as the current item. Light display. If item 362 is the current item and the user touches the key representing the number "2", item 366 under item 362 is highlighted as the current item.

[0082] FIG. 24 can also be used to describe another embodiment of the present invention. Specifically, the pull-down window 360 can also be activated by placing the caret above the menu heading "Format" and pressing the select button on a pointing device such as the mouse 170 or trackball 220 in Figures 4A and 15, respectively. it can. The user can select an item in the pull-down window 360 by moving the pointing device down in the list of items. As the user moves the input device, the individual items in the list are highlighted.

[0083] In the prior art, the pull-down menu 360 is still displayed even if the caret is placed outside the pull-down menu itself. The only way to get rid of this pull-down menu is to click in the area outside the menu itself. However, according to an exemplary embodiment of the invention, an application that generates a pull-down menu removes the pull-down menu as soon as it receives a mouse message indicating that the user has released the pointing device. This improves user efficiency by reducing the movement the user has to make to close the pull-down window associated with header menu 358.

[0084] FIG. 26 is an image of a display screen including a radial menu 370, displayed by another exemplary embodiment of the present invention. The radial menu 370 contains eight items 371 arranged in a circle around the cancel button 372. The radial menu 370 can be operated, for example, using the keyboard 300 of FIG. 18A or the touch sensor on the mouse button 670 of FIG. 10H.

When using the keyboard 300, the focus application displays a radial menu 370 upon receiving a keyboard message indicating that the user has touched one of the keys on the keypad 304. To highlight a particular item, the user touches a key on the keypad 304 that is spatially associated with that item. For example, to highlight item 373 of the radial menu 370, the user touches the key representing the number "8" located just above the center key representing the number "5". This is because the spatial position of the key "8" with respect to the key "5" is the same as the spatial relationship between the item 373 and the cancel button 372. To select an item, the user presses the key to highlight the item. To turn off the radial menu, the user presses the key "5".

[0086] To operate the radial menu using the touch sensor of the mouse button 670 of FIG. 10H, the user simply touches the touch sensor corresponding to the item of the radial menu. Simply touch the corresponding touch sensor to highlight the item. Pressing button 670 while touching the corresponding touch sensor selects that item. The application determines that two events have occurred based on two separate mouse messages. The first mouse message indicates which touch sensor is currently being touched. The second mouse message indicates that the left button has been pressed.

[0087] According to an exemplary embodiment of the invention, the use of touch-sensitive controls and devices can be transparent to the user. The context of the situation is indicated by the user's pre-stage actions with the control, such as grabbing, touching, or approaching the control device. As shown in Figure 18B, suppose that the current function of the control knob 314 is to control the volume of audio associated with the media (as opposed to controlling the volume of other audio generated by the system). In this example, if the user wants to change the volume, he / she may bring his / her hand closer to the control knob 314 or touch the knob. As shown in FIG. 27, a volume control GUI including the volume status in the form of volume display on the screen can be displayed on the screen before the user turns the knob to activate the control knob 314. The user can then turn the control knob 314 to adjust the volume, or touch it without turning the control knob 314 and use another control such as a key (eg, arrow key), mouse, or other pointing device. You can move the pointer (for example, cullet or arrow) into the GUI to change the volume. If the user touches the control knob 314 while the GUI is displayed and changes the volume using a key or pointing device, the change in volume state is immediately displayed in the GUI or elsewhere on the display screen. be able to. When the user releases the control knob 314, the system knows that the GUI is no longer needed and can turn it off without a perceptible delay. In some embodiments, the GUI is displayed as long as the pointing device continues to interact with the GUI, or as long as the cursor is over the GUI.

[0088] In another exemplary embodiment of the invention, when the user approaches or touches the control knob 314 without invoking it, a display widget such as a tooltip is instantly displayed on the display screen. This allows you to identify the current function of the control knob 314. Tooltips can, for example, indicate one of the following features, but the features are not limited to this. 1) Tuning of various applications including audio and video applications, 2) Volume control of media applications, 3) Volume control of system-generated sounds, and 4) Multiple settings (eg brightness, cropping, color) Control of numerous functions with (etc.). In another exemplary embodiment, when the user approaches a control such as the control knob 314, visual feedback is provided to the actual control knob, either by the LED or LCD alone, or in addition to the display widget on the display screen. Can be done. In yet another exemplary embodiment, auditory or tactile (eg, vibration) feedback can be provided alone or in addition to, or to each other, visual feedback to display screens, input devices and / or controls. Thus, the input device or control can provide direct feedback (eg, audio feedback) with or without the host computer involved or sharing information with the host computer. In yet another exemplary embodiment, one or more parts of the control (eg, top or side), such as the control knob 314, independently detect contact or user proximity, and any part of the control is touched. You can generate your own message to the host computer based on what you are doing.

Another exemplary embodiment of the invention based on the keyboard input device of FIG. 18C can provide a touch-sensitive rocker switch 315. In one context, the rocker switch 315 is similar to using the "Alt" and "Tab" key combinations currently used in operating systems such as Microsoft's Windows® 98 in Redmond, Washington. Allows the user to switch between applications. That is, the rocker switch 315 allows the user to move between running applications. Specifically, when the user touches or approaches the rocker switch 315, an on-screen display showing the icons of each currently running application is displayed and is in the foreground (the order in which the windows overlap), as shown in FIG. The application (at the top of) is highlighted. The user can move between applications to highlight the desired application that he or she wants to bring to the foreground of the display by pressing rocker switch 315. This order can be determined in alphabetical order, or in multiple ways, such as when each application was last time in the foreground of the display. When you release the rocker switch 315, the highlighted application is selected and comes to the front of the display. It should be understood that the touch-sensitive rocker switch 315 can be used in many other applications. This can be done by displaying various items in response to user contact or significant proximity to the switch, highlighting items in response to pressing the switch, and releasing the contact state of the switch. Includes selecting an item from the list by moving away from the switch.

[0090] Tables 1 to 5 below list the various auxiliary controls of an exemplary keyboard input device and the exemplary implementation of contextual responses to their contact. It should be understood that the various controls and reactions can also be applied to other input devices such as game controllers, trackballs, mice and touchpads.

[0091] [Table 1]<img file="JP4138340B2_D0001.tif" />[0092] [Table 2]<img file="JP4138340B2_D0002.tif" />[0093] [Table 3]<img file="JP4138340B2_D0003.tif" />[0094] [Table 4]<img file="JP4138340B2_D0004.tif" />[0095] [Table 5]<img file="JP4138340B2_D0005.tif" />[0096] Table 6 below describes an exemplary technique of contextual response to contact with an exemplary mouse input device according to the present invention.

[0097] [Table 6]<img file="JP4138340B2_D0006.tif" />[0098] In addition to the various visual feedbacks that can be provided to the screen in response to a user's contact with a touch-sensitive control, another possibility is the ability to copy each key combination or other control (eg, "" Standard when set to perform the Ctrl and c keys at the same time and the paste function (for example, the Ctrl and v keys at the same time), as shown in Figures 29A and 29B. Display tooltips. That is, when the user touches the "Ctrl" and "c" keys at the same time, the "Copy" tooltip displayed in Fig. 29A appears on the screen, and when the user touches the "Ctrl" and "v" keys at the same time, The "Paste" tooltip shown in Figure 29B appears on the screen.

[0099] Typically, the on-screen display can be placed near the current cursor or pointer (eg, caret) position regardless of the input device touched by the user. For example, a keyboard tooltip can be displayed next to the cursor. It is useful to pop up the on-screen display in the upper right corner of the current cursor position above. This is a Microsoft-manufactured Windows® operating system in Redmond, Washington, with traditional tooltips and their on-screen display that appear when the user is hovering over an icon. If is heavy (conflict Because there is no (traditional tooltips appear at the bottom right of the pointer or cursor). Tooltips also cause the user to follow the cursor or pointer when moving the cursor or pointer with a pointing device (eg mouse), or keep it stationary at the point where it was first displayed. You can also do it. Keeping the tooltip where it is displayed is easier, more efficient, and more likely to be accepted by the user than moving the tooltip with the cursor. According to another exemplary embodiment of the invention, the display widget is located in the center of the screen, in the center of the then active (focus) application or widget (eg, a text entry box), at the bottom of the screen, or at the top of the system tray icon. Can be displayed.

[0100] FIGS. 30A and 30B show another exemplary display widget (on-screen display), but in this figure, according to an exemplary embodiment of the invention, a user-programmable keyboard hotkey and function " It illustrates the keyboard control features for each application that can be displayed in response to touching the "F" key. If the user simply touches a key and does not activate it, the on-screen display can indicate what the key is and / or what the key does, and the function is not actually performed. Figure 30A shows a GUI of keyboard hotkeys that includes selectable options for users to respond to touching keys, customize keyboard hotkey settings, or view (eg, by mouse or other pointing device). An example of on-screen display is shown. That is, the user can use "Reassign" in this GUI. You can click in the HotKey area to see the current functionality of the hotkey (which may be a combination of keys) and change the assignment of that hotkey's functionality. Figure 30B shows another example of a GUI on-screen display for keyboard hotkeys that includes selectable options for customizing or viewing keyboard hotkey settings in response to a user touching a key. I will provide a. Figure 30B differs from Figure 30A in that it provides the user with more detailed information about what the key is and how it behaves (eg, launching the "My Computer" application). In some embodiments, the touch-activated GUI can receive keyboard focus as soon as the user touches the control associated with it. This allows the user to instantly use the example "Reassign" above. You can hit the "R" key of "HotKey" to activate an alternative function. In another embodiment, the touch-activated GUI does not receive keyboard focus. The user must switch to keyboard focus by clicking on the GUI with the mouse, for example using a combination of the "Alt" and "Tab" keys, or another method for switching to the top level. This latter embodiment prevents the touch-initiated GUI from receiving keyboard input that the user does not intend to receive. Preferred embodiments depend on the particular input device, the touch-sensitive controls of the device, the current application, or the details of the user's preferences.

[0101] In another exemplary embodiment of the invention, a text macro can be assigned to each hotkey, in which case when the hotkey is activated, a block of text is inserted, for example, at a cursor position on the screen. When used in such a context, touching a hotkey will display at least the beginning or other parts of the text macro assigned to that hotkey, if not the full text, as shown in Figure 31. The on-screen window can be automatically resized according to the amount of text assigned to the text macro to display the full text. This context can also be used in conjunction with the scroll control to scroll the text while the user is touching the hotkey to which the text macro is assigned.

[0102] As previously mentioned, feedback according to some According to an exemplary embodiment auditory or tactile of the present invention can be utilized click. Auditory or tactile feedback can be used in conjunction with or separately from visual feedback on the display. In some cases, depending on the running application, the input device used, or the user's preference, visual feedback may not be appropriate and auditory feedback may be desirable.

[0103] According to certain embodiments of the present invention, when a user touches a control, it is possible to generate auditory feedback such as appropriate cue tones and other sounds. The desired audio cue tone can be mapped to a particular function or control. You can adjust the volume, pitch, and timbre to create the appropriate cue that mimics the desired sound, such as voice-generated control notifications. A cue sound can be generated by receiving a parametric sound event request and using a MIDI wavetable synthesizer 662 and arranging it in sequence on a speech generator 640 (eg Creative Labs AWE64 Gold card sound board). (See Figure 6). All specific techniques for setting accurate voice parameters for each voice cue, properly synthesizing that voice cue, and associating the cue with its corresponding control and / or function will be readily apparent to those skilled in the art. It is omitted for the sake of brevity because it is a thing and does not form a part of the present invention.

Auditory feedback for identifying features and other information is particularly useful for gaming applications and products such as MS Game Voice manufactured by Microsoft Corporation in Redmond, Washington. For example, gaming products may include a combination of headset microphones attached to a puck. The pack allows users to selectively talk to multiple combinations of people and teams with which they are playing. Each person is automatically assigned a code (1, 2, 3 ...) and a team (A, B, C ...). Problems can occur when a player must remember which code corresponds to which player and team. If a user wants a reminder about who a button is assigned to, touching a control (eg a number key) will give auditory feedback on the name of the person assigned to that code in their headset. Can be received through.

[0105] In an exemplary embodiment of the invention identified in Tables 1-5, the use of multiple contact areas on an input device can be used to navigate between web pages provided by an internet browser. it can. Examples of input devices that have multiple touch-sensitive areas and are useful for navigating between pages include, among others, the mice in FIGS. 10C, 10D, 12A, 12B, 13A, 13B, and 13C. In the case of FIG. 10C, the function of returning to the page is started when the area 624 is touched and then the area 626 is touched, and the function of advancing the page is started when the area 626 is touched and then the area 624 is touched. In FIG. 10D, touching area 637 and then touching area 634 starts the function to return to the page, and touching area 634 and then touching area 637 starts the function to advance the page. In Figures 12A and 12B, touching areas 722 and 724, respectively, and then touching areas 720 and 728, respectively, initiates the page advance function, touching basins 720 and 728, respectively, and then touching areas 722 and 724, respectively. The function to return to the page is started. In Figures 13A, 13B, and 13C, touching areas 734, 738, and 746, respectively, and then touching areas 736, 742, and 748, respectively, starts the page advance function, and then touching areas 736, 742, 748, respectively. , Touching areas 734, 738, 746 respectively will start the function to return to the page.

It should be noted that a mouse according to an exemplary embodiment of the present invention can be configured to initiate the page-to-page movement function by simply touching one touch sensor instead of touching two touch sensors in succession. Therefore, in FIG. 10C, touching area 624 initiates the page forward function, and touching area 626 initiates the page return function. Similarly, the function of advancing the page can be started by touching the area 734 of FIG. 13A, and the function of returning the page can be started by touching the area 736 of FIG. 13A. In this regard, the touch sensor of the present invention provides the function of the side switch described in US Patent Application No. 09 / 153,148 "INPUT DEVICE WITH FORWARD / BACKWARD CONTROL" filed on September 14, 1998. To do. The inventor of the above invention was obliged to assign the application to the assignee of the present invention.

[0107] FIGS. 32 to 34 show the inter-page movement function performed by using the touch area as described above. Display 460 in FIG. 32 shows an internet browser window 462 displaying the current page 464. The user can display the past page 470 shown in the Internet browser window 472 of FIG. 33 by returning to the Internet page that was displayed before the current page 464. The user can use the combination of touch sensors as described above to proceed to the next page 476 shown in the browser window 478 of the display 480 of FIG. To proceed to the next page 476, the user must at some point return from the next page 476 to the current page 464.

It should be understood that various reactions to touch-sensitive control contact, or significant proximity, can be used in combination. For example, the first time a user touches a button, the function of that button can be displayed. In one implementation, if the user continues to touch the button for longer than the specified amount of time (eg 5 seconds), it is possible to display more detailed status information or make the GUI available to the user. Is. Alternatively, if the same button is continuously touched within a predetermined time, different types of information can be displayed, such as the function of the button being displayed and then the GUI appearing. Tactile or auditory feedback can also be provided after the first contact, and if there is a second contact within a given time or if the contact is detected over a specified time, the same information or more detail as the first. A display of information appears.

[0109] It is possible to cause the on-screen display of the display widget to occur instantaneously in response to the detection of contact or proximity of the input device or its auxiliary controls with the user. Often, the display of the display widget on the screen is slightly so that the display widget does not appear and disappear immediately on the screen, for example when the user activates a control or device to activate a function rather than simply touching it. It may be desirable to delay. By delaying the display of the display widget by a certain amount of time, you are familiar with the operation of active applications and input devices and their auxiliary controls, and immediately launch the device or its controls to activate a function ( You can prevent the user from unnecessarily displaying a display widget that contains tooltip information, for example (such as pressing a button). Some exemplary techniques according to the invention for addressing display control will be described below.

[0110] In one embodiment of the invention, a short time of approximately 0.3 seconds is used. In this example, the user touches a key or other control for less than this short time, but does not activate it to display an on-screen display. If the user activates the control before this short time is reached, there will be no on-screen display in response to the user touching the control.

[0111] According to some embodiments of the present invention, when a control is activated (eg, turning a knob), the state of the controlled function (eg, volume) changes and the user sets the state (eg, volume setting). If you change it, you can display the GUI (for example, for volume setting). On the other hand, if you touch it without starting it for a short period of time, you can display the GUI that shows the existing state (for example, the current volume) of the function, and that state does not change.

[0112] Similarly, according to an embodiment of the present invention, a short time can be used to control the erasure or fade-out of the on-screen display when the user ceases to contact the control. In this example, the on-screen display remains visible for a short period of time, after which the on-screen display disappears or fades out. If the user touches the control again before the short time is reached, the on-screen display remains visible. At a similar time, the display widget disappears (ie disappears or fades out) when it detects user contact with the control, and the display widget reappears or fades in when it loses contact with the control. It should be understood that it can be applied to various embodiments of the present invention.

[0113] According to an exemplary embodiment of the invention, when displaying a display widget that contains tooltips or other information, the display method may be plural, including other information such as controls, timeouts, and mouse movements. Can be based on the criteria of. Figures 35A-35C graphically show three exemplary alternatives for time setting and conditions related to touch-sensitive mice that display display widgets such as tooltips on the display screen. Is.

FIG. 35A shows a time setting diagram of an exemplary embodiment in which a button tooltip is displayed on the display screen after a short timeout (Δt). The user touches the button for a short time, touches the button for a short time before launching, or inadvertently scratches the touch sensor, as shown in the figure where the button touch time is less than the short timeout period (Δt). Even if the timeout is suppressing the button tool. Also, as shown in the figure, if you touch the button longer than the short timeout (Δt), a tooltip will be displayed. This tooltip is displayed until the user releases the contact with the button. In this example, mouse movement has no effect on the display of tooltips.

[0115] According to the time setting diagram of FIG. 35B of the exemplary embodiment of the present invention, the tooltip of the button is not displayed when the user touches (does not activate) the button and moves the mouse. When the user keeps touching the button for a timeout period (Δt) and stops moving the mouse, a hint for that button is displayed. As shown in Figure 35A, the tooltip is displayed until the user releases the contact with the button.

[0116] In the time setting diagram shown in FIG. 35C, when the user touches a button and the mouse is stationary, the user keeps touching the button for a timeout period (Δt), and the tooltip is displayed unless the mouse is moved. To. Similar to Figures 35A and 35B, this tooltip is displayed until the user disengages the button.

Another aspect of the display display scheme for touch-sensitive input controls relates to the reaction of tooltips or other display widgets to the user's activation of a button or other input control. Once the control has been activated, the user may no longer want feedback (about what happens as a result of the activation, etc.). For example, when a user clicks on a button (button down event), the tooltip for that button can be instantly erased (or faded out). Also, if the user continues to touch the button and a button-up event occurs, the button's tooltip may be redisplayed or hidden. The computer system can also display visual feedback that associates the button click with the information displayed in the tooltip before the tooltip disappears. According to an exemplary embodiment as shown in the time setting diagram shown in FIG. 36A, when a button-down event occurs due to a button click, the period Δt elapses after the user releases the button and touches the button again without activating it. The tooltip for that button will not appear again until you do. In other words, after the button is activated, the tooltip for that button is not displayed after the next rising edge of the touch sensor for that button, as shown in Figure 36A. According to another embodiment of the invention represented by the time setting diagram of FIG. 36B, a button if the user maintains contact with the button after a button-up event followed by a time-out Δt. Tooltips are displayed again.

Another embodiment of the present invention relates to simultaneous contact of a single input device with multiple controls. If the user touches the second control while maintaining contact with one control, an on-screen display or other feedback about the second control is generated. Alternatively, a particular function can be defined by a combination of controls, such as a combination of keys. In this example, the tooltip for that combination (for example, a key combination) is displayed when two keys are touched at the same time and are not activated, but there is a tooltip associated with contacting the first key. Is displayed on top of it. When the user releases one of the controls, the on-screen display disappears completely and feedback on the other touched control can be restored.

[0119] In another embodiment of the invention, when an on-screen display in the form of a GUI is displayed or faded in, the GUI may place a cursor or pointer (eg, by mouse movement) within it. , It is displayed even if the user stops contacting the control that displayed the GUI (for example, a touch-sensitive mouse button). The GUI in this example ultimately responds to cursor movements (eg mouse movements) outside of that GUI, invoking another control, or the computer receives any input such as mouse movements. Can disappear after a given inactivity period (eg 5 seconds).

[0120] According to an exemplary embodiment of the invention, the on-screen display can be faded in and faded out using an animated variation. It should be understood that the techniques for controlling the display and erasure of on-screen displays may be used alone or in combination. Hereinafter, an embodiment of the present invention that provides changes by animation will be described in the context of FIG. 37.

FIG. 37 is a simplified block diagram of high-level software and hardware components located within the PC20 for displaying and erasing display widgets such as Tool Glass sheets or toolbars in an on-demand manner. Represents. As shown in FIG. 37, these components include application 36, operating system 35, video adapter 48, display 47, input interface 46, and input device 43. Application program 36 (eg, a word processing or spreadsheet program) can include a GUI 910 in it, which can include a fade-in / fade-out animation process 915 and a bitmap generator 920. The bitmap generator 920 includes a Tool Glass pattern 922, a toolbar pattern 924, and other display widget patterns 926. Other components include the input device driver 60 in operating system 35 and the graphics API (application programming interface) 940.

The contact output provided by the input device 43 can be applied to the input interface 46 as represented by the leads 903 and 907, respectively. This interface can generate a separate signal that reflects whether or not contact is being sensed at the time by the contact sensors of the respective touch-sensitive components (eg, auxiliary controls) of each input device. .. These signals can be sent to the input device driver 60, which forms a component within the operating system 35, as represented by line 905. This device driver interprets the signal generated by the input device and responds by generating the appropriate event. With respect to contact, this event can indicate the state of a particular input device and the corresponding touch sensor, such as whether a hand contact is detected. This event can be passed by operating system 35 to application 36 and eventually to GUI process 910 within that application. During this process, the event is processed by the fade-in / fade-out animation process 915. The animation process is that the state change of each input device has occurred, that is, the device has just started or stopped sensing hand contact, and the display widget (eg Tool Glass or Toolbar) Generates a given animation sequence to fade in or out of the display widget in a controllable manner in response to whether it is currently displayed or not.

Bitmap Generator 920 remembers predetermined patterns 922, 924, and 926, which are typically texture maps, which each display a Tool Glass sheet, a given toolbar, and / or other display widgets. For various display widgets including. Usually, these patterns are stored in the application as separate files. When application program 36 is launched, the application downloads this pattern to graphic API 940 via line 930 during initialization or after any widget changes. The graphics API 940 can send this pattern to a graphics accelerator (not shown) and store it in local storage as a texture map. Alternatively, this download may be managed by a service provided by operating system 35.

During the subsequent display, the accelerator reads the above map from its local store, fills it appropriately into polygons, and draws a pattern on the display 47 that fills the resulting map. Then, when you download this map for a controlled fade-in or fade-out operation, the animation process 915 changes the alpha transparency value as the graphic accelerator draws the pattern that fills the map of the corresponding display widget. For toolbars, the alpha transparency value varies over the entire range of transparency values (ie, between approximately 0α and 1.0α on a linear scale from 0 to 1α, where 0α is completely transparent and 1.0α is completely opaque. Become). For Tool Glass sheets, the alpha transparency value is typically varied from 0α to approximately 0.7α. This means that even with a Tool Glass sheet drawn with maximum opacity, the document objects underneath can still be seen through the sheet (slightly blurry).

[0125] Basically, in the case of a fade-in or fade-out operation, the animation process 915 issues a series of instructions over a predetermined time interval. Specifically, this command is a command for continuously changing the alpha transparency value when displaying a specific display widget for each of the different display frames. Also represented by line 930, this instruction is issued to operating system 35, which in turn passes it to the graphics API 940 and finally to the graphics accelerator.

Fade-in should occur over a relatively short interval of time, such as approximately 0.3-0.5 seconds. However, it is better to take a relatively long time such as about 0.7 to 1.0 seconds to fade out so as not to distract the user. Especially during fade-in, the alpha transparency value is generally changed in a non-linear manner during this time. Experience has shown that we have chosen to utilize traditional throw-in / throw-out techniques commonly used in screen animation. With this technique, opacity initially changes from a virtually transparent state (that is, virtually invisible) to an intermediate value at a slightly slower rate, then at a slightly faster rate. Increases to another intermediate value, then slows down again at higher speeds for maximum opacity (eg 1α for toolbars, Tool In the case of Glass sheet, it reaches 0.7α). This is a nearly "S" -shaped opacity curve that is a function of time. Use this same transparency change to fade out the display widget (but in the opposite direction). It goes without saying that other time-based opacity functions, such as linear functions, can be used to change the opacity during the fade-in and fade-out periods. Ultimately, the particular function you choose (you can also use different functions for fade-out and fade-in; in fact, you can use different functions for different display widgets as described above) if you wish). , Likely to be empirically determined through testing by appropriate users.

[0127] Further, the human eye exhibits higher sensitivity to a particular color, such as a red color tone, than other color tones, such as a blue color tone, at normal luminance. For this reason, fading in display widgets that contain colors like the former can distract the user even at relatively low brightness, especially as such colors become sharper. May be scattered. To avoid this distraction of the user, the display widgets may differ or differ, especially for the colors most sensitive to the human eye, until the final map with the desired color scheme and brightness is displayed. It can be represented by several texture maps with varying color schemes and brightness. In this case, first display the black and white texture map for that widget, then at the appropriate time during the fade-in, a texture map for the same widget with the desired color scheme and increased brightness (or even a different color scheme). (May be), and at the end of the fade-in period, draw a texture map with the final color scheme and brightness. Fade out can be done in a similar way, but in the reverse order. For example, a display widget can fade out to a black and white map using a series of colored texture maps that gradually diminish, and then fade out to a completely transparent black and white map by itself. Alternatively, at the beginning of the fade-out, it turns into a black-and-white map of the widget, from which it can fade out to full transparency.

When not using a graphics accelerator, the graphics API 940 directs the graphics output to the video adapter 48 (specifically, a standard video card in it, not shown), as represented by line 955. Can be provided to. The video adapter 48 then generates the appropriate video signal and adds it to the display 47. In this example, the computer system needs to be able to operate fast enough to perform the appropriate graphics functions that would be provided by the graphics accelerator in the software. In addition, other visual techniques can be used to show and erase display widgets if the fade-in and fade-out graphics features are not supported. This technology includes, for example, a technology that simply translates a widget by sliding or shuffing it from an off-screen position onto the screen, a technology that instantly and completely displays or erases a tool, a widget. A technique for rotating a widget (for example, when the widget is on a 3D surface and rotating it into place) and / or an interface tool or part of a document that gradually expands or contracts. Is included. However, with these technologies, display widgets such as toolbars and Tool Glass sheets are limited to completely opaque display. Both of these techniques can also be used in conjunction with fading with graphic accelerators that support alpha transparency.

The Alpha Transparency feature is supported by a wide range of existing graphics accelerators, but this feature is D3D (a 3D graphics API currently manufactured by Microsoft as a standard component of the WINDOWS® operating system). , OpenGL (currently available in the art), or GDI (historically just a low-level graphics processing layer in 2D. Currently manufactured by Microsoft and as a standard component of the WINDOWS® operating system. With traditional 2D (2D) and 3D (3D) graphics APIs such as (built-in), it can be easily simulated in software in a well-known manner.

[0130] It is conceivable that the display screen displays both the toolbar and the Tool Glass sheet (or a combination of other display widgets) at the same time. In such cases, both tools may fade in or out due to unintended dialogue. To prevent such interaction and the associated user frustration, we have incorporated an appropriate decision process within the scope of the technology into the animation process 915 to fade in or fade in only one of the two display widgets instead of both. Can be faded out. For example, if both tools are visible and the user operates the Touch Mouse, only the Tool Glass fades out and the toolbar remains fully visible. The specific decision process is controlled by specific widgets that can be displayed at the same time, and the need to continue displaying one or more of those widgets is the action being performed at that time, and the relative between the widgets. Based on the application's current context settings, including display priority.

It should be understood that the present invention can activate on-demand operations using other sensing techniques other than contact sensing. Such sensing techniques include, for example, skin electrical reactions, non-contact proximity sensors, pressure sensors, events from touch / proximity sensors on the keyboard, data from GPS (Global Positioning System) receivers (user location). , Or the location of the mobile computer), video data from the camera, and audio data from the microphone.

Although specific embodiments of the present invention have been described and illustrated, it should be understood that the present invention is not limited thereto, as those skilled in the art can make modifications to it. This application is intended for all modifications within the spirit and scope of the underlying invention disclosed and claimed herein.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of a computer system according to an exemplary embodiment of the present invention.

FIG. 2 is a more detailed block diagram of an exemplary embodiment of an input device according to the present invention.

FIG. 3 is a perspective view of a headset according to an exemplary embodiment of the present invention.

4 (A) is a perspective view of a mouse according to an exemplary embodiment of the present invention, (B) is a bottom view of the mouse of (A), and (C) is an exemplary mouse of (A). It is a perspective view of a various circuit boards.

FIG. 5 is a top view of another exemplary embodiment of a mouse according to the present invention.

6 (A) and (B) are left side views of another exemplary embodiment of a mouse according to the invention, and FIGS. 6 (C) and 6 (D) are another exemplary embodiment of a mouse according to the invention. It is a right side view.

7 (A) and 7 (B) are left side views and top views of another exemplary embodiment of a mouse according to the present invention, respectively.

8 (A) and 8 (B) are left side views and top views of another exemplary embodiment of a mouse according to the present invention, respectively.

9 (A)-(C) are a left side view, a top view, and a right side view of another exemplary embodiment of a mouse according to the present invention, in order (respectively).

10 (A) to (H) are top views of various exemplary embodiments of mouse buttons according to the present invention.

11 (A) is a top view of another exemplary embodiment of a mouse according to the present invention, and FIG. 11 (B) is a top view of another exemplary embodiment of a mouse according to the present invention.

12 (A) and 12 (B) are right side views of different exemplary embodiments of mice according to the invention.

13 (A)-(D) are left side views of different exemplary embodiments of the mouse according to the present invention.

14 (A) to 14 (D) are top views of various exemplary embodiments of a mouse showing a touch sensor close to a wheel according to the present invention.

FIG. 15 is a perspective view of an exemplary trackball according to the present invention.

FIG. 16 is a perspective view of an exemplary game controller with a joystick according to the present invention.

FIG. 17 is a perspective view of an exemplary game controller according to the present invention.

FIG. 18 (A) is a perspective view of an exemplary keyboard according to the present invention, and FIGS. (B) and (C) are views showing a part of the exemplary keyboard according to the present invention.

FIG. 19 is a perspective view of an exemplary touchpad according to the present invention.

20 is a more detailed block diagram of the computer of FIG. 1. FIG.

FIG. 21 is a diagram of an exemplary screen display that can be displayed before touching the input device of the present invention.

FIG. 22 is an image of an exemplary screen display after touching the input device of the present invention.

FIG. 23 is an image of an exemplary screen display showing a pull-down menu activated by the present invention.

FIG. 24 is an image of an exemplary screen display showing a second pull-down menu opened by the present invention.

FIG. 25 is an image of an exemplary screen display showing an item selected in a pull-down menu according to the present invention.

FIG. 26 is an image of an exemplary screen display showing a radial menu according to the present invention.

FIG. 27 is a diagram of an exemplary graphical user interface that can be displayed on a display screen according to an exemplary embodiment of the present invention.

FIG. 28 is an exemplary screen display diagram that can be displayed on a display screen according to an exemplary embodiment of the present invention.

FIG. 29 (A) and (B) are diagrams of visual feedback of exemplary tooltips that can be displayed on a display screen according to an exemplary embodiment of the present invention.

30 (A) and 30 (B) are diagrams of exemplary visual feedback that can be displayed on a display screen according to an exemplary embodiment of the present invention.

FIG. 31 is a diagram of exemplary visual feedback that can be displayed on a display screen according to an exemplary embodiment of the present invention.

FIG. 32 is an image of an exemplary screen display showing a web browser that includes the current page.

FIG. 33 is a diagram of an image of an exemplary screen display showing a web browser including previously viewed pages.

FIG. 34 is a diagram of an image of an exemplary screen display showing a web browser including the next page.

35 (A) to 35 (C) are exemplary time setting diagrams for controlling the display of tooltips according to the exemplary embodiment of the present invention.

36A and 36B are exemplary time setting diagrams for controlling the display of tooltips according to an exemplary embodiment of the present invention.

FIG. 37 is a high-level block diagram 900 that sits within PC20 and simplifies the software and hardware components that collectively form an exemplary implementation of the invention.

[Code description] 20 Personal computer 21 Processing device (CPU) 22 System memory 23 System bus 24 Read-only memory 25 Random access memory 26 BIOS27 Hard disk drive 28 Magnetic disk drive 29 Detachable magnetic disk 30 Optical disk drive 31 Optical disk 32 Hard disk drive interface 33 Magnetic Disk Drive Interface 34 Optical Disk Drive Interface 35 Operating System 36 Application Program 37 Program Module 38 Program Data 40 Touch Sensor 42 Mobile Transducer 43 Input Device 45 Speaker 46 Serial Port Interface 47 Monitor 48 Video Adapter 49 Remote Computer 50 Memory Storage 51 LAN 52 WAN53 Adapter 54 Modem 60 Device Driver 100, 102, 104, 106 Touch Sensor 108, 110, 112, 114 Lead 116 A / D Converter and multiplexer 118 Wire 120, 200 Microcontroller 122 Input (input line, line) 124 Output 150 Headset 152 Microphone 154 Support piece 156 Touch sensor 158 Output line 170, 690 Mouse 172, 600, 604, 704, 710, 780 Palm rest 174 Left button 176 Right button 178, 764, 768, 774 Wheel 180 Side 182 Output Line 184, 186 Side Area 189 Electronic Circuit 190, 220, 228 Trackball (Ball) 192 Trackball Nest 194, 196, 203 Encoder Wheel 201, 202, 204 Switch 206 Sensor Arrangement 222 , 242 Base 224, 226, 248, 250, 252, 312A ~ 312G Button 244 Handle 246 Trigger 260 Gamepad 262, 264 Side Button 266, 268, 270, 272, 274, 276 Left Hand Button 278, 280, 282, 284, 286, 288 Right hand button 290 Start button 292 Select button 300 Keyboard 301 Keyboard case 302 QWERTY layout 304 Number Keypad 306, 307, 309, 311 Part 308 Space Bar 310 Arrow Key 314 Control Knob 315 Rocker Switch 320, 350 Screen 322, 354 Virtual Desktop 328, 352 Window (Active Window) 324, 326 Icon 330 Caret 332 Toolbar 334 Toolbar 356, 360 Pull-down menu (pull-down window) 358 Menu bar (header menu) 362, 364, 366, 371, 373 Item 370 Radial menu 372 Cancel button 460, 480 Display 462, 472, 478 Internet browser window 464, 470, 476 pages 602, 603, 605, 607, 608, 610, 612, 614, 616, 624, 626, 634, 637 648, 672, 720, 722, 724, 726, 728, 734, 736, 738, 740, 742 , 744, 746, 748, 762, 766, 770, 772 Sensor (touch sensor, area) 606 Left side 618 Solid sensor 620, 752 Sensor strip (strip) 640 Speech generator 662 MIDI wavetable synthesizer 224, 226, 622, 630, 646, 660, 700, 702, 706, 708, 2220, 2240, 2260, 2280 Button 613 Side end 627 Front end 628, 636 Raised 641, 642, 643, 644, 650, 651, 652 Sensor area 674 Central sensor 692 Left side sensor 694 Palm sensor 696 Right side sensor 698 Button sensor 776 Front area 778 Rear area 810 Message hook procedure 812 Focus application 900 block Figure 903, 905, 907, 930 Line 910 GUI915 Fade In / Fade Out Animation Process 920 Bitmap Generator 922, 924, 926 Pattern 940 Graphic API2000 Touchpad 2100 Surface (Tablet)