Multi-user sensor correlation for computing device interaction

<u>BACKGROUND</u>

0001Many mobile computing devices (e.g., tablets, phones, etc.), as well as other devices such as desktop digitizers, drafting boards, tabletops, e-readers, electronic whiteboards and other large displays, use a pen, pointer, or pen type input device in combination with a digitizer component of the computing device for input purposes. Many of these computing devices have touch-sensitive screens and interact with pen and with bare-handed touch or with the two in combination.

0002<patcit id="pcit0001" dnum="US2006197750A1"><text>US2006/197750 A1</text></patcit> describes hand held devices with multiple touch sensing devices. The touch sensing devices may for example be selected from touch panels, touch screens or touch sensitive housings.

0003<patcit id="pcit0002" dnum="US2012206330A1"><text>US2012/206330 A1</text></patcit> describes a multi-touch orientation sensing input device with a body that is partially enclosed by a multi-touch sensor. An inertia measurement unit that is disposed on the device body measures a tilt angle of the device body with respect to a horizontal surface, as well as a roll angle of the device body along a length-wise axis of the device body.

<u>SUMMARY</u>

0004The invention is defined by the appended independent claims 1, 7, and 8. Additional embodiments of the invention are defined by the dependent claims.

<u>BRIEF DESCRIPTION OF THE DRAWINGS</u>

0005The specific features, aspects, and advantages of the claimed subject matter will become better understood with regard to the following description, appended claims, and accompanying drawings where: <ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">FIG. 1</figref> depicts exemplary naturally-occurring core pen grips and poses.</li><li><figref idref="f0002">FIG. 2</figref> depicts exemplary naturally-occurring single finger extension grips for touch screen manipulation.</li><li><figref idref="f0003">FIG. 3</figref> depicts exemplary naturally-occurring multiple finger extension grips for touch screen manipulation.</li><li><figref idref="f0003">FIG. 4</figref> depicts other types of naturally-occurring pen grips.</li><li><figref idref="f0004">FIG. 5</figref> provides an exemplary system that illustrates program modules for implementing various implementations of the pen and computing device sensor correlation technique, as described herein.</li><li><figref idref="f0005">FIG. 6</figref> provides an exemplary flow diagram of using the pen and computing device sensor correlation technique to provide a correlated touch and sensor pen input mechanism, as described herein.</li><li><figref idref="f0006">FIG. 7</figref> provides an exemplary flow diagram of using the pen and computing device sensor correlation technique to provide metadata based on correlated signals received due to contacts on two or more touch-sensitive devices.</li><li><figref idref="f0007">FIG. 8</figref> provides an exemplary flow diagram of using a pointing device to continue control of an input on a display screen of a touch-sensitive computing device.</li><li><figref idref="f0008">FIG. 9</figref> provides an exemplary flow diagram of passing a touch-sensitive computing device from a primary user to a secondary user.</li><li><figref idref="f0009">FIG. 10</figref> provides an exemplary flow diagram of passing both a touch-sensitive computing device and a touch-sensitive pen from a primary user to a secondary user.</li><li><figref idref="f0010">FIG. 11</figref> provides an exemplary illustration of using the pen and computing device sensor correlation technique to provide a magnifier/loupe tool input mechanism based on the pen being held in a user's preferred hand in a tucked grip, as described herein.</li><li><figref idref="f0010">FIG. 12</figref> provides an exemplary illustration of using the pen and computing device sensor correlation technique to provide a full-canvas pan/zoom input mechanism based on the actions of a user's non-preferred hand, as described herein.</li><li><figref idref="f0011">FIG. 13</figref> provides an exemplary illustration of using the pen and computing device sensor correlation technique to provide a drafting tool input mechanism based on the touch and grip patterns of both the user's preferred hand and the user's non-preferred hand, as described herein.</li><li><figref idref="f0011">FIG. 14</figref> provides an exemplary illustration of using the pen and computing device sensor correlation technique to provide a pen tool input mechanism based on the touch and grip patterns of both the user's preferred hand and the user's non-preferred hand, as described herein.</li><li><figref idref="f0012">FIG. 15</figref> provides an exemplary illustration of using the pen and computing device sensor correlation technique to provide a canvas tool input mechanism based on the touch and grip patterns of both the user's preferred hand and the user's non-preferred hand, as described herein.</li><li><figref idref="f0012">FIG. 16</figref> provides an exemplary illustration of two users passing a touch-sensitive pen between them and initiating context-appropriate capabilities based on the grip patterns of both users on the touch-sensitive pen and the orientation of the pen, as described herein.</li><li><figref idref="f0013">FIG. 17</figref> provides an exemplary illustration of two users passing or sharing a touch-sensitive computing device between them based on the grip patterns of the two users on the touch-sensitive computing device and the orientation of the touch-sensitive computing device, as described herein.</li><li><figref idref="f0014">FIG. 18</figref> is a general system diagram depicting a simplified general-purpose computing device having simplified computing and I/O capabilities, in combination with a touch-sensitive pen having various sensors, power and communications capabilities, for use in implementing various implementations of the pen and computing device sensor correlation technique, as described herein.</li></ul>

<u>DETAILED DESCRIPTION</u>

0006In the following description of the implementations of the claimed subject matter, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific implementations in which the claimed subject matter may be practiced.

1.0 <u>Introduction</u>

0007The following paragraphs provide an introduction to mobile sensing, sensor-augmented pens, grip sensing, and pen + touch input on touch-sensitive computing devices.

1.1 <u>Mobile Sensing on Handheld Computing Devices</u>

0008Tilt, pressure, and proximity sensing on mobile devices enables contextual adaptations such as detecting handedness, portrait/landscape detection, or walking versus stationary usage. Grip sensing allows a mobile device to detect how the user holds it, or to use grasp to automatically engage functions such as placing a call, taking a picture, or watching a video. Implementations of the pen and computing device sensor correlation technique described herein adopt the perspective of sensing natural user behavior, and applying it to single or multiple touch-sensitive pen and touch-sensitive computing device (e.g., tablet) interactions.

0009Multi-touch input and inertial sensors (Inertial Measurement Units (IMU's) with 3-axis gyroscopes, accelerometers, and magnetometers) afford new possibilities for mobile devices to discern user intent based on grasp and motion dynamics. Furthermore, other sensors may track the position of these mobile devices. Implementations of the pen and computing device sensor correlation technique illustrate new techniques that leverage these types of motion sensing, grip sensing, and multi-touch inputs when they are distributed across separate pen and touch-sensitive computing (e.g., tablet) devices.

1.2 <u>Grips and Sensing for Tablets</u>

0010Lightweight computing devices such as tablets afford many new grips, movements, and sensing techniques. Implementations of the pen and computing device sensor correlation technique described herein are the first to implement full grip sensing and motion sensing-on both tablet and pen at the same time-for sensing pen + touch interactions. Note that "grip" may be recognized by the system as a holistic combination of a particular hand-contact pattern that takes into account the 3D orientation or movement of the implement or device as well; that is, no clear line can be drawn between touch-sensitive grip-sensing and inertial motion-sensing, per se, since all these degrees of freedom may be employed by a recognition procedure to classify the currently observed "grip" as accurately as possible. Thus, whenever the term "grip" is used the possible combination of touch with motion or orientation degrees-of-freedom is implied.

1.3 <u>Palm Detection and Unintentional Touch Handling</u>

0011Palm contact can cause significant false-activation problems during pen + touch interaction. For example, some note-taking applications include palm-blocking but appear to rely on application-specific assumptions about how and where the user will write. Some palm-rejection techniques require the user to bring the pen tip on or near the screen before setting the palm down, which requires users to modify their natural movements. Implementations of the pen and computing device sensor correlation technique use sensors to detect when a touch screen contact is associated with the hand holding the pen.

1.4 <u>Sensor-Augmented and Multi-DOF Pen Input</u>

0012Auxiliary tilt, roll, and other pen degrees-of-freedom can be combined to call up menus or trigger mode switches without necessarily disrupting natural use. Implementations of the pen and computing device sensor correlation technique implement capabilities where the user can extend one or more fingers while tucking the pen. The pen and computing device sensor correlation technique implementations can sense these contacts as distinct contexts with separate functions, even if the user holds the pen well away from the screen.

0013Pens can be augmented with motion, grip, and near-surface range sensing. One type of pen uses grip sensing to detect a tripod writing grip, or to invoke different types of brushes. Other systems use an integrated IMU on the pen as a feature to assist grip recognition and sense the orientation of associated computing device/tablet (e.g. for horizontal vs. drafting table use) to help provide appropriate sketching aids. The pen and computing device sensor correlation technique implementations described herein go beyond these efforts by exploring sensed pen grips and motion in combination with pen + touch gestures, and also by extending grip sensing to the tablet itself.

2.0 <u>Natural Pen and Tablet User Behaviors</u>

0014Implementations of the pen and computing device sensor correlation technique described herein use natural pen and touch-sensitive computing device (e.g., tablet) user behaviors to determine the context associated with these behaviors in order to provide users with context-appropriate tools. As such, some common grips that arise during digital pen-and-tablet tasks, and particularly touch screen interactions articulated while the pen is in hand are useful to review and are enumerated below and shown in <figref idref="f0001">FIGs. 1</figref>, <figref idref="f0002">2</figref>, <figref idref="f0003">3 and 4</figref>. A wide variety of behaviors (listed as B1-B11 below) have been observed and were used in designing various implementations of the pen and computing device sensor correlation technique. The following paragraphs focus on behaviors of right-handers; left-handers are known to exhibit a variety of additional grips and accommodations. It should be noted that the behaviors discussed below and shown in <figref idref="f0001">FIGs. 1</figref>, <figref idref="f0002">2</figref>, <figref idref="f0003">3 and 4</figref> are only exemplary in nature and other behaviors are entirely possible.

2.1 <u>Behavior B1. Stowing the Pen While Using Touch.</u>

0015The tendency of users to stow the pen when performing touch gestures on a touch screen of a touch-sensitive computing device such as the tablet is obvious. Users typically only put the pen down when they anticipate they will not need it again for a prolonged time, or if they encounter a task that they feel is too difficult or awkward to perform with pen-in-hand, such as typing a lot of text using the on-screen keyboard.

2.2 <u>Behavior B2. Tuck vs. Palm for Stowing the Pen.</u>

0016There are two distinct grips that users employ to stow the pen. These are a Tuck grip (pen laced between fingers) shown in <figref idref="f0001">FIG. 1</figref> 104 and a Palm grip (with fingers wrapped lightly around the pen barrel) shown in <figref idref="f0001">FIG. 1</figref> 106. Users stow the pen during pauses or to afford touch interactions.

2.3 <u>Behavior B3. Preferred Pen Stowing Grip Depends on Task Context.</u>

0017For users that employ both Tuck grips 104 and Palm grips 106, a Tuck grip affords quick, transient touch interactions, while a Palm grip is primarily used if the user anticipates a longer sequence of touch interactions. Other users only use Tuck grips 104 to stow the pen.

2.4 <u>Behavior B4 Grip vs. Pose.</u>

0018For each grip-that is, each way of holding the pen-a range of poses where the pen orientation is changed occur, often by wrist supination (i.e. turning the palm upward). Human grasping motions with a pen therefore encompass the pattern of hand contact on the barrel, as well as the 3D orientation of the pen. As shown in <figref idref="f0001">FIG. 1</figref>, full palmar supination 112 is observed for the Tuck grip 104 and Palm grip 106, but only half-supination 110 for the Writing grip.

2.5 <u>Behavior B5. Extension Grips for Touch.</u>

0019As shown in <figref idref="f0002">FIGs. 2</figref> and <figref idref="f0003">3</figref>, many Extension Grips exist where users extend one or more fingers while holding the pen to make contact with a touch screen. These were classified broadly as single-finger extension grips (<figref idref="f0002">FIG. 2</figref>, 200) vs. multiple-finger extension grips (<figref idref="f0003">FIG. 3</figref>, 300), which users can articulate from either the Tuck or the Palm grip. (Note that, while not illustrated, three-finger extension grips are also possible from some grips).

2.6 <u>Behavior B6. Variation in Pen Grips.</u>

0020Users exhibit many variations in a tripod grip for writing which leads to variations in users' resulting Tuck, Palm, and Extension grips. For example, one user's style of tucking led her to favor her ring finger for single-touch gestures (see Tuck-Ring Finger Extension Grip (<figref idref="f0002">FIG. 2</figref>, 206)).

2.7 <u>Behavior B7. Consistency in Grips.</u>

0021Each user tends to consistently apply the same pen grips in the same situations. Users also tend to maintain whatever grip requires the least effort, until a perceived barrier in the interaction (such as fatigue or inefficiency) gives them an incentive to shift grips. Users switch grips on a mobile computing device (e.g., tablet) more often when sitting than standing, perhaps because there are few effective ways to hold or re-grip such a device while standing.

2.8 <u>Behavior B8. Touch Screen Avoidance Behaviors.</u>

0022Users often adopt pen grips and hand postures, such as floating the palm above a touch screen while writing, or splaying out their fingers in a crab-like posture, to avoid incidental contact with the screen. Another form of touch screen avoidance is perching the thumb along the outside rim of touch-sensitive computing device (e.g., the tablet bezel), rather than letting it stray too close to the touch screen when picking up the touch-sensitive computing device. These unnatural and potentially fatiguing accomodations reflect a system's inability to distinguish the context of intentional versus unintentional touch.

2.9 <u>Behavior B9. Finger Lift for Activating Pen Controls.</u>

0023It was observed that users only activate a pen barrel button from the Writing grip (<figref idref="f0001">FIG. 1</figref>, 102, 108, 110), and then only with the index finger. Users hold the pen still when tapping the button. The thumb is also potentially available for controls from the Palm-like Thumb Slide grip (<figref idref="f0003">FIG. 4</figref>, 404).

2.10 <u>Behavior B10. External Precision Grip.</u>

0024Users employ an External Precision grip (<figref idref="f0003">FIG. 4</figref>, 402), with the pen held toward the fingertips and perpendicular to the writing surface, for precise pointing at a small target. This provides the possibility to provide contextual enhancements, such as automatically zooming the region of the tablet screen under the pen tip, when this grip is detected.

2.11 <u>Behavior B11. Passing Grip.</u>

0025Passing prehension is observed when participants pass the pen and touch-sensitive computing device (e.g., tablet) to another person. Users tend to hold the device securely, in more of a power grip, and extend it from their body while keeping it level, so that their intent is clear and so that the other person can grab it from the far side.

0026Having described these natural behaviors, the following sections describe how the recognition of all of these grips, touches and motions are used to leverage these behaviors in order to provide context-appropriate tools for carrying out a user's intended actions.

3.0 <u>Introduction to the Pen and Computing Device Sensor Correlation Technique:</u>

0027The pen and computing device sensor correlation technique implementations described herein contribute cross-device synchronous gestures and cross-channel inputs for a touch-sensitive computing device/pen (e.g., tablet-stylus) distributed sensing system to sense the naturally occurring user behaviors and unique contexts that arise for pen + touch interaction. Note, however, that while much of the discussion here focuses on pen/tablet interactions, other pen-like mechanical intermediaries or small wearable devices can enable context-sensing while interacting with tablets using variations of the pen and computing device sensor correlation technique. A small motion-sensing ring worn on the index finger, for example, could sense when the user taps the screen with that finger versus, another digit. Watches, sensors worn on the fingertip or fingernail, bracelets, arm-bands, bandages or wraps, elbow pads, braces, wrist-bands, gloves augmented with sensors, subcutaneous implants, or even e-textile shirt sleeves with embedded sensors, represent other similar examples that would enable and suggest related techniques to individuals skilled in the art. Likewise, other manual tools such as a ruler, compass, scalpel, tweezer, stamp, magnifying glass, lens, keypad, calculator, french curve, shape template, paint-brush, or airbrush could serve as pen-like implements, whether held in the preferred or non-preferred hand, that also enable related techniques.

0028Implementations of the pen and computing device sensor correlation technique described herein employ grip and touch sensing to afford new techniques that leverage how users naturally manipulate these devices. Implementations of the pen and computing device sensor correlation technique can detect whether the user holds the pen in a writing grip or palmed and/or tucked between his fingers in a stowed mode. Furthermore, pen and computing device sensor correlation technique implementations can distinguish bare-handed inputs, such as drag and pinch gestures produced by a user's non-preferred hand, from touch gestures produced by the hand holding the touch-sensitive pen which necessarily imparts a detectable motion signal to the pen. Implementations of the pen and computing device sensor correlation technique can sense which hand grips the touch-sensitive computing device (e.g., tablet), and determine the screen's relative orientation to the pen and use the screen's orientation and touch patterns to prevent accidental screen content rotation. By selectively combining sensor signals from the touch-sensitive pen and the touch-sensitive computing device and using them to complement one another, implementations of the pen and computing device sensor correlation technique can tailor user interaction with them to the context of use, such as, for example, by ignoring unintentional touch inputs while writing, or supporting contextually-appropriate tools such as a magnifier for detailed stroke work that appears when the user pinches with the touch-sensitive pen tucked between his fingers.

0029Implementations of the pen and computing device sensor correlation technique, as described herein, use a touch-sensitive pen enhanced with a power supply (e.g., battery) and multiple sensors (e.g., a sensor pen) to enable a variety of input techniques and commands based on the correlated grip patterns of a user holding the touch-sensitive pen and touch contacts and grips on the touch-sensitive computing device (e.g., a touch-sensitive tablet computer) and associated motions and orientations of these devices. For example, pressure sensors can be used to detect the user's grip patterns on the sensor pen and touch and grip patterns on the touch-sensitive computing device. Implementations of the pen and computing device sensor correlation technique correlate sensor pen grips and touch-sensitive computing device touches and grips to determine the intentions of the user and the context in which the user wishes to use the touch-sensitive pen or the touch-sensitive computing device. This is based on naturally occurring user behaviors such as, for example, whether a user is gripping either device with their preferred hand or their non-preferred hand. The determined user intentions and context of use are then used to generate context-appropriate commands and capabilities for the touch-sensitive computing device and/or the touch-sensitive pen.

0030The term pressure as described herein, as relating to pressure sensors and the like, may refer to various sensor types and configurations. For example, in various cases and implementations, pressure may refer to pen tip pressure exerted on a display. In general, pen tip pressure is typically sensed by some type of pressure transducer inside the pen, but it is also possible to have the pen tip pressure sensing done by the display/digitizer itself in some devices. In addition, the term pressure or pressure sensing or the like may also refer to a separate channel of sensing the grip pressure of the hand (or fingers) contacting an exterior casing or surface of the touch-sensitive pen or touch-sensitive computing device. Various sensing modalities employed by the pen and computing device sensor correlation technique may employ both types of pressure sensing (i.e., pen tip pressure and grip pressure) for initiating various capabilities and commands.

0031Various devices used to enable some of the many implementations of the pen and computing device sensor correlation technique described herein include pens, pointers, pen type input devices, etc., that are often referred to herein as a sensor pen or touch-sensitive pen for purposes of discussion. Further, the sensor pens or touch-sensitive pens described herein can be adapted to incorporate a power supply and various combinations of sensors. For example, there are various possibilities of incorporating power into the pen, such as by inductive coupling, a super capacitor incorporated into the pen that recharges quickly when the pen comes in range or is docked to or placed on/near a computing device, a battery incorporated in the pen, obtaining power via pen tether, or acquiring parasitic power via motions of the pen. The power supply may feature automatic low-power modes when the pen is not moving or not being held. The sensors may inform this decision as well. Various combinations of sensors can include, but are not limited to, inertial sensors, accelerometers, pressure sensors, grip sensors, near-field communication sensors, RFID tags and/or sensors, temperature sensors, microphones, magnetometers, capacitive sensors, gyroscopes, sensors that can track the position of a device, finger print sensors, galvanic skin response sensors, etc., in combination with various wireless communications capabilities for interfacing with various computing devices. Note that any or all of these sensors may be multi-axis or multi-position sensors (e.g., 3-axis accelerometers, gyroscopes, and magnetometers). In addition, in various implementations, the touch-sensitive pens described herein have been further adapted to incorporate memory and/or computing capabilities that allow them to act in combination or cooperation with other computing devices, other touch-sensitive pens, or even as a standalone computing device.

0032Implementations of the pen and computing device sensor correlation technique are adaptable for use with any touch-sensitive computing device having one or more touch-sensitive surfaces or regions (e.g., touch screen, touch sensitive bezel or case, sensors for detection of hover-type inputs, optical touch sensors, etc.). Note that touch-sensitive computing devices include both single- and multi-touch devices. Examples of touch-sensitive computing devices can include, but are not limited to, touch-sensitive display devices connected to a computing device, touch-sensitive phone devices, touch-sensitive media players, touch-sensitive e-readers, notebooks, netbooks, booklets (dual-screen), tablet type computers, or any other device having one or more touch-sensitive surfaces or input modalities. The touch-sensitive region of such computing devices need not be associated with a display, and the location or type of contact-sensitive region (e.g. front of a device on the display, versus back of device without any associated display) may be considered as an input parameter for initiating one or more motion gestures (i.e., user interface actions corresponding to the motion gesture).

0033The term "touch" as used throughout this document will generally refer to physical user contact (e.g., finger, palm, hand, etc.) on touch sensitive displays or other touch sensitive surfaces of a computing device using capacitive sensors or the like. However, some touch technologies incorporate some degree of non-contact sensing, such as the use of highly sensitive self-capacitance detectors to detect the geometry of the fingers, pen, and hand near the display-as well as the pen-tip hover sensing. Arrays of IR sensor-emitter pairs or sensor-in-pixel display elements can also be deployed on pens, tablets, and keyboards for this purpose. Hence touch and grip may incorporate such non-contact signals for a holistic or unified notion of "grip" detection as well.

0034In addition, pen and computing device sensor correlation technique implementations can use a variety of techniques for differentiating between valid and invalid touches received by one or more touch-sensitive surfaces of the touch-sensitive computing device. Examples of valid touches and contacts include user finger touches (including gesture type touches), pen or pen touches or inputs, hover-type inputs, or any combination thereof. With respect to invalid or unintended touches, pen and computing device sensor correlation technique implementations disable or ignore one or more regions or sub-regions of touch-sensitive input surfaces that are expected to receive unintentional contacts, or intentional contacts not intended as inputs, for device or application control purposes. Examples of contacts that may not be intended as inputs include, but are not limited to, a user's palm resting on a touch screen while the user writes on that screen with a pen or holding the computing device by gripping a touch sensitive bezel, etc.

0035The pen and computing device sensor correlation technique implementations provide a number of advantages relating to pen-based user interaction with a touch-sensitive pen and touch-sensitive computing devices, including, but not limited to: <ul id="ul0002" list-style="none" compact="compact"><li>---Novel solutions that sense grip and motion to capture the full context of pen and touch-sensitive computing device (e.g. tablet) use.</li><li>---Using sensors to mitigate unintentional touch (from the palm, or from the thumb when picking up the device), but also to promote intentional touch by a non-preferred hand, or via extension grips to interleave pen and touch inputs.</li><li>---Novel contextually-appropriate tools that combine grip, motion, and touch screen contact, including, for example, distinct tools for bare-handed input, pinch input while tucking the pen, and drafting tools that the user can summon with the non-preferred hand when the pen is poised for writing.</li></ul>

3.0 <u>Exemplary System:</u>

0036The pen and computing device sensor correlation technique implementations operate, in part, by correlating sensor inputs from a touch-sensitive pen and a touch-sensitive computing device to trigger various actions and capabilities with respect to either the touch-sensitive pen or the touch-sensitive computing device or both.

0037<figref idref="f0004">FIG. 5</figref> provides a diagram of an exemplary system 500 that illustrates program modules for implementing various implementations of the pen and computing device sensor correlation technique. More specifically, <figref idref="f0004">FIG. 5</figref> shows a touch-sensitive pen or sensor pen 502 in communication with touch-sensitive computing device 504 via communications link 506. As discussed in further detail herein, the sensor pen 502 can include a variety of sensors. A sensor module 508 in the sensor pen 502 monitors readings of one or more of those sensors, and provides them to a communications module 510 to be sent to the touch-sensitive computing device 504 (or possibly another computing device (not shown) that performs computations and provides inputs to the touch-sensitive computing device 504). It should be noted that in another implementation some (or all) computation may be done directly on the pen before sending (i.e., grip recognition with machine learning), and some data might not always be sent to the touch-sensitive computing device (i.e., if the outcome is used local to the pen). In some instances, the touch-sensitive computing device may send information to the pen instead of, or on top of, the pen sending data to the touch-sensitive computing device.

0038A sensor pen input module 512 receives input from one or more sensors of sensor pen 502 (e.g., inertial, accelerometers, pressure, touch, grip, near-field communication, RFID, temperature, microphones, magnetometers, capacitive sensors, gyroscopes, IR or capacitive proximity sensors, finger print sensors galvanic skin response sensors, etc.) and provides that sensor input to a grip and touch determination module 516. Similarly a computing device touch input module 514 receives input from one or more sensors of the touch-sensitive computing device 504 (e.g., inertial, accelerometers, pressure, touch, grip, near-field communication, RFID, temperature, microphones, magnetometers, capacitive sensors, gyroscopes, IR or capacitive proximity sensors, finger print sensors, galvanic skin response sensors, etc.) and provides that sensor input to a grip and touch determination module 516.

0039The grip and touch determination module 516 determines the grip of a user on the sensor pen 502 based on the contact of a user's hand on touch-sensitive surfaces of the sensor pen (and/or the orientation of the pen-yaw, pitch roll or some subset of that- and/or other information from the sensors). For example, the sensor signals from the user's grip on the pen can be compared to a database 518 of grip patterns in order to determine the grip pattern or patterns of a user gripping the pen. In one implementation a trained classifier is used to classify the sensor signals into grip patterns on the sensor pen 502 based on grip training data. Note that this grip training data may be (1) for a large sample of many users; (2) adapted or trained based only on inputs from the specific user; and (3) some weighted combination of the two. Also, separate databases based on salient dimensions of the input (e.g. left-handed vs. right-handed user, size or type of the device being used, current usage posture of the device, whether on a desk, held-in-hand, resting on the lap, and so forth) may trigger the use of separate databases optimized in whole or in part to each use-case. Similarly, the grip and touch determination module 516 determines the touch of the user on a display of the touch-sensitive computing device 504 based on the signals of contact of the user's fingers or hand on a display of the touch-sensitive computing device (and/or the orientation of the device and/or other information from the sensors). Additionally, the grip and touch determination module 516 can determine if the user is gripping touch-sensitive surfaces of the case of the touch-sensitive computing device. For example, the sensor signals from the user's grip on the case of the touch-sensitive computing device can be compared to a database 518 of grip patterns in order to determine the grip pattern or patterns of the user gripping the device. In one implementation, one or more touch-sensitive sensors report an image of what parts of the case are being touched. Various image processing techniques can be used to interpret the image and deduce the grip. In one implementation, a (multi-touch, capacitive) grip pattern is sensed on the case of the device (e.g., the case incorporates a matrix of capacitive sensors) and motion signals and orientation of the touch-sensitive computing device (and/or pen) are also fed into this determination. In some implementations, if the touch screen has non-contact proximity sensing capability, then sensing proximity at the screen edges of the device can serve as a good proxy for grip sensing on the case. In one implementation a trained classifier is used to classify the sensor signals into grip patterns on the case of the touch-sensitive computing device based on grip training data.

0040The grip and touch determination module 516 correlates the sensor inputs from the sensors of the sensor pen 502 and the touch-sensitive computing device 504 to associate how the user is gripping the pen with how the user is interacting with the screen or the case of the touch-sensitive computing device. This correlated data can be used to determine the user's preferred hand/non-preferred hand of a user touching the sensor pen and/or the touch sensitive computing device. In one implementation, the preferred hand is distinguished from the non-preferred hand due to the pen motion. A bump in the pen motion is measured when a part of the preferred hand comes in contact with the touch screen of the computing device. After the contact, it can also be continuously confirmed that the pen motion correlates to the touch screen motion in order to confirm that the hand holding the pen is being held in the user's preferred hand. In some implementations it is not necessary for the pen to register a bump when it touches the screen (e.g., if the touch to the screen is very subtle or soft) in order to determine the user's preferred hand or non-preferred hand as long as correlating motions are observed.

0041The determined grip and touch patterns, as well as other correlated sensor inputs, can also be input into a context determination module 520. The context determination module 520 determines the user's intent and the context of the actions that the user is intending from the correlated grip patterns, touch patterns and other sensor data. Context examples include, but are not limited to, how many users are interacting with a pen or touch-sensitive computing device, how many devices are being interacted with, whether a user is holding the sensor pen or the computing device in the user's preferred vs. non-preferred hand, individual or relative motions of the pen or the computing device, how the touch-sensitive pen or the touch-sensitive computing device is being gripped or touched, application status, pen orientation, touch-sensitive computing device orientation, relative orientation of the sensor pen to the computing device, trajectories and/or accelerations of the touch-sensitive pen and the touch-sensitive computing device, identity of the user, and so forth. This context data can be sent to a metadata labeling module 524 which can be used to semantically label this data.

0042The touch/grip patterns and other associated sensor readings and any associated context data are input into the command initiation module 522. The command initiation module 522 evaluates the available inputs to trigger one or more commands or capabilities on the touch-sensitive computing device and/or the sensor pen in order to assist in carrying out the user's intent in a more efficient and user friendly manner by triggering or activating one or more commands or motion gestures on the touch-sensitive pen 502 and/or the touch sensitive computing device 504. Examples of various motion gestures triggered or commands (526, 528, 530, 532, 534, 536, 538, and 540) which will be discussed in detail below are activated by the command initiation module 522. Implementations of the pen and computing device sensor correlation technique allow user-defined motion gestures 542 to be defined via a user interface that allows the user to define one or more capabilities or motion gestures using sensor pen grips in combination with touch inputs and/or grips on the touch-sensitive computing device 504. Many of the motion gestures or capabilities 526 through 540 are described in further detail later in this document, with examples of many of these motion gestures and capabilities being illustrated in <figref idref="f0010 f0011 f0012 f0013">FIGs. 11 through FIG. 17</figref>.

0043With respect to hover range, in various implementations, the pen and computing device sensor correlation technique considers distance of the sensor pen 502 above the digitizer of the touch-sensitive computing device 504. While a variety of ranges can be considered, in various tested implementations, three range categories were considered, including: physical contact, within hover range of the digitizer, or beyond range of the digitizer. The activation mechanism for any particular motion gestures may consider these different ranges of the sensor pen, in combination with any other correlated inputs, touches, and/or motions of the computing device.

0044In some implementations the raw sensor readings can be reported or transmitted from the sensor pen 502 to the computing device 504 for evaluation and characterization by the computing device. For example, raw sensor data from inertial sensors within the sensor pen 502 can be reported by the sensor pen to the touch-sensitive computing device 504, with the computing device then determining pen orientation as a function of the data from the inertial sensors. Alternately, in various implementations, the sensor pen 502 uses onboard computational capability to evaluate the input from various sensors. For example, sensor data derived from inertial sensors within the sensor pen 502 can be processed by a computational component of the sensor pen to determine pen orientation, with the orientation of tilt then being reported by the sensor pen to the computing device 504.

0045Any desired combination of reporting of raw sensor data and reporting of processed sensor data to the computing device by the sensor pen 502 can be performed depending upon the computational capabilities of the sensor pen. However, for purposes of explanation, the discussion herein generally refers to reporting of sensor data to the touch-sensitive computing device 504 by the sensor pen 502 for further processing by the computing device to determine various commands, motion gestures or other input scenarios. In some implementations the user's touch on the touch-sensitive screen of the computing device is correlated to a bump to the gyro or accelerometer in the pen in order to determine what the user is doing. In fact, many implementations of the technique use correlations between touch, bump onto a pen and pen grip to determine what the user's intentions are. A few examples of various motion gestures and capabilities enabled by the pen and computing device sensor correlation technique are briefly introduced below.

0046For example, one such input technique, referred to as a "Magnifier" or "Loupe tool" (<figref idref="f0004">FIG. 5</figref>, 530), uses a sensor input from the user's grip on the touch-sensitive pen 502 to discern that the pen is held in a tucked position in the user's hand. Concurrently, the user's touch on the touch-sensitive screen of the computing device is registered to that of the user making a two-finger touch gesture such as a pinching motion relative to the screen. These two sensed inputs are correlated so that the pinching motion with the hand holding the pen brings up the Magnifier/Loupe tool 530. The rationale is that since the user demonstrates the intent to use the pen again soon by the fact the pen is stowed only temporarily and still at the ready, touch interactions with the pen-holding-hand should emphasize capabilities in support of the pen. The Magnifier/Loupe tool 530 is advantageous in that it supports and emphasizes a quick zooming that affects a local area of the content displayed on the screen only, which is especially well suited to detail work with the pen. In some implementations the user's touch on the touch-sensitive screen of the computing device is correlated to a bump to the gyro or accelerometer in the pen in order to determine the user's intent.

0047A related gesture referred to as a "full-canvas pan/zoom" gesture (<figref idref="f0004">FIG. 5</figref>, 532), in one implementation uses a sensed user's two-finger touch gesture such as a pinching motion on the display of the touch-sensitive computing device 504 (e.g., tablet) with the user's non-preferred hand to trigger a standard full canvas zoom of content displayed on the display of the computing device. This functionality is provided when the sensor pen 502 is not at the ready (e.g., when the sensor pen is held in a Tucked grip or a Palmed grip in the user's preferred hand, or the user is not holding the sensor pen at all). The non-preferred hand gesture, then, can be recognized as such due to the lack of corresponding motion in the pen.

0048Another gesture, referred to herein as a "pen tools" gesture (<figref idref="f0004">FIG. 5</figref>, 536), uses sensors of the sensor pen 502 and the touch-sensitive computing device 504 to detect that the user is holding the senor pen in a tucked position (e.g., Tucked grip) in the user's preferred hand and to detect a concurrent contact such as a tap from a finger of the user's preferred hand on the touch-sensitive display of the computing device 504. The correlation of these two actions brings up a menu of pen-specific tools. In some implementations a different palette of pen tool options can be displayed when the pen is held in a fully palmed position in the user's preferred hand.

0049A similar gesture, referred to herein as a "canvas tools" gesture (<figref idref="f0004">FIG. 5</figref>, 538) uses sensors of the pen and on the touch-sensitive computing device to detect when a user is holding the touch-sensitive pen 502 in a non-writing position in the user's preferred hand and to detect a contact such as a finger tap with the user's non-preferred hand on the touch screen of the computing device 504. These correlated concurrent actions cause a menu of canvas tools to be displayed on the display of the computing device. For example, this menu of tools could include undo/redo, cut-copy-paste, new page, search and similar commands.

0050Another gesture, referred to herein as a "drafting tools" gesture (<figref idref="f0004">FIG. 5</figref>, 540) uses sensors of the pen 502 and on the touch-sensitive computing device 504 to detect that the touch-sensitive pen is held in a writing grip in the user's preferred hand and to detect a contact such as a one finger touch to the touch-sensitive display of the computing device 504 with the user's non-preferred hand. The one finger tap with the user's non-preferred bare hand brings up a set of drafting tools when the pen is ready to write. These special tools support the use of the pen. Such tools might be, for example, a compass to draw arcs or an airbrush to color content on the screen.

0051Additionally, an "advanced drafting tools" gesture (<figref idref="f0004">FIG. 5</figref>, 528) uses sensors on the pen 502 and sensors on the touch-sensitive computing device 504 to detect that the sensor pen is held in a writing grip in the user's preferred hand and to detect a two-finger touch gesture such as a pinching motion at the touch-sensitive display of the computing device with the user's non-preferred hand. The pinching motion with the user's non-preferred hand brings up a set of advanced drafting tools when the pen is ready to write. These special tools further support use of the pen that benefits from a second touch. For example, these advanced drafting tools can include a ruler or alignment edge, a French curve or a function to pull a new sheet of paper (with two-finger position, rotation, and/or scaling possibly, depending on the particular advanced drafting tool employed).

0052Other examples of correlated sensor pen motions relative to the computing device include using pen sensors (e.g., accelerometers, pressure sensors, inertial sensors, grip sensors, etc.) to determine when the sensor pen is picked up or put down by the user. By considering the current sensor grip pattern (i.e., tucked in the user's preferred hand, ready to write in the user's preferred hand, put down) correlated with the grip patterns of the user on the computing device (e.g., not held by the user, touching the display with the user's non-preferred hand or non-preferred hand, etc.), appropriate commands can be initiated.

0053Some implementations of the pen and computing device sensor correlation technique use capacitive grip sensing on the back and sides of the case of the touch-sensitive computing device 504 to detect a number of additional contacts.

0054Some implementations of the pen and computing device sensor correlation technique can be used in a multiple user/multiple device mode. For example, in some implementations a grip of a primary user on a touch-sensitive computing device 504 and a grip of a secondary user on the touch-sensitive computing device are sensed and correlated. The grips can be sensed, for example, by touch-sensitive surfaces on the touch-sensitive computing device or the grips can be sensed by determining each user's galvanic skin response and the differences in the galvanic skin response can be used to tell one user from the other. Other sensor data can also be used, such as, for example, accelerometer data, position data, trajectory data and so forth. The grips of the primary and secondary users are evaluated to initiate a command in an application executing on the touch-sensitive computing device 504. The correlated grips and the orientation of the touch-sensitive computing device 504 can be evaluated to determine that the grip of the secondary user represents a handoff of the computing device to the secondary user from the primary user. In this case one or more capabilities of the touch-sensitive computing device may be restricted following the handoff. Alternately, the grip of the secondary user can be determined to be concurrent with the grip of the primary user. In this case, a sharing mode can be entered on the computing device. For example, the secondary user may only be allowed to view and markup only content that is currently displayed on a display of the computing device. There are many instances where the user picks up and holds a pen with both hands, making the pen unavailable.

0055Implementations of the pen and computing device sensor correlation technique use grip to sense with which hand the user is holding the touch-sensitive computing device 504. Implementations of the pen and computing device sensor correlation technique then use this to summon a "Thumb Menu" (<figref idref="f0004">FIG. 5</figref>, 534) at the appropriate side of the touch-sensitive computing device 504, which advantageously allows the user to activate various buttons and menus directly with the thumb. If the user grasps the touch-sensitive computing device with a second hand, implementations of the pen and computing device sensor correlation technique leave the Thumb Menu visible at the side where it first appeared.

0056Similar to the touch-sensitive computing device, in some implementations of the pen and computing device sensor correlation technique a grip of a primary user and the grip of a secondary user on a touch-sensitive pen 502 can be sensed. As discussed previously, other data such as acceleration, position and trajectory data, for example, may also be considered. The grips of the primary and secondary users can then be correlated to initiate a command in an application executing on the touch-sensitive pen 502. For example, the grip of the secondary user can be determined to represent a handoff of the pen 502 to the secondary user from the primary user. In this case data can be transferred from the primary user to the secondary user via the handoff of the pen 502. The secondary user can then download the transferred data to a computing device-for example a different computing device that the stylus was originally used with. Alternately, one or more capabilities of the touch-sensitive pen 502 can be restricted following the handoff.

0057Besides grip patterns on the sensor pen 502 and on the computing device 504, some implementations consider motion of the computing device and the sensor pen. For example, implementations use the motions of the sensor pen 502 and the computing device along with the grip patterns to determine if the pen is held in the user's preferred hand (or non-preferred hand). The user's preferred hand and non-preferred hand can be determined from the correlated grip patterns and associated information. For example, when a motion signal representing a pen bump occurs at the same time as a new contact on the touch-sensitive computing device the grips/touches on both the sensor pen 502 and the touch-sensitive computing device 504 are correlated and specific motion gestures or commands are initiated based on these recognized grip patterns.

0058Further, in various implementations, the pen and computing device sensor correlation technique also advantageously rejects or ignores unwanted or unintended touches by a user. A palm rejection module (<figref idref="f0004">FIG. 5</figref>, 526) can be used for this purpose. In particular, the palm rejection module 526 evaluates any touch to determine whether that touch was intended or it was made by a palm inadvertently resting on the touch-sensitive computing device, and then either accepts that touch as input for further processing, or rejects that touch. In addition, in various implementations, the palm rejection module disables or ignores (i.e., "rejects") user palm touches on or near particular regions of any touch-sensitive surfaces, depending upon the context of that touch. Note that "rejected" touches may still be handled by the pen and computing device sensor correlation technique as an input to know where the palm is planted, but flagged such that unintentional button presses or gestures will not be triggered in the operating system or applications by accident. In some implementations the pen and computing device sensor correlation technique is able to track the palm (for example using an assigned contact identifier) when the contact is moving (as long as the contact is touching.) Furthermore, in some implementations if new contacts are detected within a given radius of the palm contact, they will also be labeled as palm contacts (e.g., knuckles) and ignored. Similar to the palm rejection module 526, a thumb contact rejection module 544 can also be enabled.

4.0 <u>Exemplary Processes</u>

0059An exemplary system for practicing implementations of the pen and computing device sensor correlation technique having been described, the following section discusses exemplary processes for practicing various implementations of the pen and computing device sensor correlation technique.

0060<figref idref="f0005">FIG. 6</figref> depicts an exemplary process 600 for practicing the one implementation of the technique. As shown in block 602, a combination of concurrent sensor inputs from a touch-sensitive pen device and a touch-sensitive computing device are received. These concurrent sensor inputs include one or more sensor inputs based on grip patterns on the touch-sensitive pen and one or more sensor inputs based on touch patterns on the touch-sensitive computing device. In some cases there may be no touch pattern on the pen or the touch-sensitive computing device if the user is not touching or gripping one of these two items. The sensor inputs are used to determine how the touch-sensitive pen is being gripped concurrent with how the touch-sensitive computing device is being touched, as shown in block 604. For example, the grip patterns and touch patterns are evaluated to determine if the user is using the user's preferred hand or the user's non-preferred hand (i.e., in some implementations the dominant/preferred hand is detected based on correlation of the touchdown of the hand on the touch-sensitive computing device with a bump in the accelerometer or the gyro of the touch-sensitive pen, and it is assumed that the pen is held in the preferred hand). A context-appropriate user interface action is initiated based on a combination of the grip pattern on the touch-sensitive pen and the touch pattern on the touch-sensitive computing device, as shown in block 606. For example, the determination of how the pen device is being touched by the user's preferred hand or non-preferred hand can be used to infer the context of use and/or user intent in order to initiate a context-appropriate command or capability such as those described with respect to <figref idref="f0004">FIG. 5</figref>.

0061<figref idref="f0006">FIG. 7</figref> depicts another exemplary implementation 700 for practicing the pen and computing device sensor correlation technique in order to find meta information. As shown in block 702, the signals of contacts on two or more touch-sensitive devices by a single user are correlated. For example, one of the two or more touch-sensitive computing devices can be a sensor pen and one could be a tablet computing device. The context of the contacts on the two or more touch-sensitive devices is determined based on the correlation of the signals, as shown in block 704. The determined context of the contact is labeled as metadata for use in an application, as shown in block 706. For example the context meta data could include, but is not limited to, how many users are interacting with a pen or touch-sensitive computing device, how many devices are being interacted with, whether a user is holding the sensor pen or the computing device in the user's preferred hand, individual or relative motions of the pen or the computing device, how the sensor pen or the touch-sensitive computing device is being gripped or touched, pen orientation, touch-sensitive computing device orientation, relative orientation of the sensor pen to the computing device, and so forth. If finger print sensors are available, fingerprints can be used to determine which user is holding a device, which finger(s) the user is touching the device with, if the user is holding the device with his preferred hand and in which grip, among other things. The finger print sensor could also be used to recognize the user by his or her finger print to establish a preferred mode for the user, for example. The metadata can be further used to initiate a context-appropriate user interface action, as shown in block 708, or it can be used for some other purpose. It should be noted that a similar process could be advantageously used to find and label metadata for more than one user.

0062<figref idref="f0007">FIG. 8</figref> depicts yet another exemplary implementation 800 for practicing the pen and computing device sensor correlation technique. As shown in block 802 when a pointing device comes within a predetermined range of a touch-sensitive surface of a computing device the presence of the computing device is sensed. For example, the pointing device could be a touch-sensitive pen. Control of an input on a display screen of the computing device is started using motions of the pointing device when the pointing device is within the predetermined range, as shown in block 804. Control of the input on the display screen is continued using motions of the pointing device when the pointing device is outside of the predetermined range, as shown in block 806. It should be noted that hover sensing is innate to electromagnetic digitizers, which sense the x,y location of the pointer tip (e.g., pen tip) when the pointer (e.g., pen) is close enough to the screen. The pointer (e.g., pen) will keep reporting its 3D orientation (and motion signals) via a radio link. This can also be used to infer some relative motion, or to continue cursor control. Some implementations of the pen and computing device sensor correlation technique can also use accelerometer or other sensor data to track the position of the computing device.

0063<figref idref="f0008">FIG. 9</figref> depicts another exemplary process 900 for practicing the pen and computing device sensor correlation technique. A grip of a primary user on a touch-sensitive computing device is sensed using sensor data (for example, sensor data shows that the grip is with one hand, the device is flat and acceleration of the device is toward a secondary user who is most likely on the opposite side of the primary user), as shown in block 902. A grip of a secondary user on the touch-sensitive computing device is also sensed using sensor data (for example, sensor data shows a grip of the secondary user on the opposite side of the device as the primary user), as shown in block 904. The grips of the primary and secondary users are correlated to determine the context of the grips (for example whether they are sharing the device or they are passing it the other user) as shown in block 906, and to initiate a command in an application executing on the touch-sensitive computing device, as shown in block 908. As discussed previously the context of the grips could indicate that the touch-sensitive computing device is being passed from the primary user to the secondary user. Alternately, the context of the grips could indicate that the primary user is sharing the touch-sensitive computing device with the secondary user. Depending on the context different commands and capabilities will be enabled on the touch-sensitive computing device. For example, a user's intention to share their device could be confirmed, for example, by using a voice command.

0064<figref idref="f0009">FIG. 10</figref> depicts yet another exemplary process 1000 for practicing the pen and computing device sensor correlation technique. In this implementation both the touch-sensitive computing device and the touch-sensitive pen are passed from a primary user to a secondary user. As shown in block 1002, sensor inputs representing a grip of a primary user on the touch-sensitive computing device and the touch-sensitive pen are sensed. At about the same time, the sensor inputs representing a grip of the secondary user on the touch-sensitive computing device and the touch-sensitive pen are also sensed, as shown in block 1004. The grips of the primary and secondary users on the pen and the touch-sensitive computing device are correlated to determine the context of the grips (block 1006) and to initiate a command in an application executing on the touch-sensitive pen or on the touch-sensitive computing device, as shown in block 1008. For example, the grips can be determined to represent a handoff of the pen and the computing device to the secondary user from the primary user. In this case data can one or more capabilities of the touch-sensitive pen or the touch-sensitive computing device can be restricted following the handoff. Likewise, if the sensors and a digitizer on the touch-sensitive computing device indicate that the same user is employing the same pen on a different digitizer (e.g., on a different computing device), this can carry state information (such as the mode or color/thickness/nib style of the pen) or files and data (such as the current clipboard contents) across to another device. This can be determined by observing that a user is holding one tablet while writing or touching a finger to another, separate tablet, for example. In any of these implementations the determined context can be used to initiate a command in an application executing on the touch-sensitive computing device or on the touch-sensitive pen. Variations are also possible. For example, a single user may send commands or transfer data to many touch-sensitive computing devices using a single touch-sensitive pen. This is advantageous in that it allows a user to transfer data or objects from one device to one or many other devices.

5.0 <u>Details and Exemplary Implementations:</u>

0065An introduction to the pen and computing device sensor correlation technique implementations, as well as an exemplary system and exemplary processes for practicing the technique having been provided, the following paragraphs provide details of various exemplary implementations of the pen and computing device sensor correlation technique. Although many of the details below make reference to using a tablet computer in combination with a pen, it should be understood that this is just one exemplary device used to simplify the description. The exemplary implementations described below could be used with any touch-sensitive computing device (e.g., phone, desktop digitizer, phablet, e-reader, electronic whiteboard, vehicle touch-display and so forth).

0066By evaluating correspondences between touch-screen input (or touch inputs on other surfaces of the computing device) and touch-sensitive pen grips, the pen and computing device sensor correlation technique infers additional information about how the user is touching the screen or other touch-sensitive surface. By correlating signals from touch-sensitive pen grips with signals from bare-handed touch inputs on the computing device as well as motions and orientation of the pen and computing device and other sensor data, the pen and computing device sensor correlation technique enables a variety of context-appropriate tools and motion gestures to aid a user in completing intended tasks which can be inferred from the correlated sensor signals. Note that such correlations may look at correspondences (or non-correspondences, as the case may be) in motion or grip sensor data that occurs before, during, and after contact with the touch screen (for correlations involving the tablet digitizer). Information at or close to the time of contact may be used to determine a preliminary assessment of the type of contact, for example, with the possibility of a more definitive assessment at a later time as further sensor signals and device input events arrive in real-time. In this way, the system can provide immediate, or near-immediate, real-time response and feedback to the user's input events while taking maximum advantage of all the information to determine as correctly as possible the context of the user's actions.

0067For purposes of explanation, the following discussion will refer to a sketching or drawing type application in the context of a tablet-type computing device. However, it should be understood that both the sensor pen and the touch-sensitive computing device(s) are fully capable of interaction and interoperation with any desired application type or operating system type. In other application contexts, such as, for example, active reading or mathematical sketching, different gestures or mappings can be defined. In fact, as noted above, any desired user-definable gestures and concurrent pen-and-touch inputs can be configured for any desired action for any desired application, operating system, or computing device. Further, it should also be understood that voice or speech inputs, eye gaze inputs, and user proximity and body posture inputs (such as provided by a depth camera) can be combined with any of the various input techniques discussed herein above to enable a wide range of hybrid input techniques.

5.1

<b>Grip</b> + <b>Motion Interaction Techniques</b>

0068Exemplary implementations of the pen and computing device sensor correlation technique were employed in the context of a simple sketching application that supports annotation, panning and zooming, and some additional sketching tools. Some of the capabilities of these implementations are discussed in the paragraphs below. Many other implementations and combinations of these capabilities are possible.

5.1.1 <u>Pen Orientation Sensed Relative to a Touch-Sensitive Computing Device</u>

0069In some implementations, inertial sensor fusion allows implementations of the pen and computing device sensor correlation technique to maintain a common reference frame relative to a touch-sensitive computing device such as, for example, a tablet. Some implementations of the technique employ a tablet-relative orientation at the grip-recognition phase, as well as in the interpretation of the pen motion (such as for the airbrush tool, described in more detail later). Some implementations of the technique only can sense the orientation of the pen relative to the touch-sensitive computing device (e.g., tablet). In some implementations, inertial sensing cannot reliabily determine the <i>(x,y)</i> translation or <i>z</i>-altitude of the pen without resort to some other absolute external reference. In other implementations, where an external absolute reference is available, technique implementations may continue to track the absolute (x,y) and altitude (z) of the pen tip even when it is beyond the sensing range of the screen itself, by fusing the intertial sensing capability with the absolute reference. The absolute reference may be an extrinsic source (e.g. optical tracking of markings or IR sources on the pen) or intrinsic sources (e.g. radio signal strength triangulation to determine the approximate distance of the pen radio transciever from other transceivers in the tablet or environment). In general, wireless and radio signal strengths can be used to approximate position which can be enough (when combined with fine-grained inertial sensing) to produce a good estimate of the x,y, z position of a device relative to another--this can be accomplished through the triangulation of multiple sources. Additionally, in some implementations a depth camera may be used to track the pen and/or the touch-sensitive computing device.

5.1.2 <u>Detecting Unintentional Palm Contact</u>

0070Sensing unintentional palm contact on a touch-sensitive surface or screen is a difficult problem because, at the onset of touch, there is often insufficient information to distinguish what type of touch is occuring. A palm can be recognized as a touch with a large contact area, but such contacts typically start small and may take a while to pass a certain size threshold. Also, some unintentional touches (such as contact produced by the knuckles) may never turn into a "large" contact. This strategy therefore necessitates delays (introduces lag) in processing touch events, and still may fail to detect many contacts.

0071To increase stability and avoid fatigue, users naturally rest their hand on the writing surface, but current tablet users are forced to adopt touch screen avoidance behaviors. Simply sensing that the user is holding the pen is not sufficient because people stow the pen while using touch and employ various extension grips to touch the screen. Pen orientation is also insufficient because each grip can be associated with a range of wrist supinations and because users hold the pen in a variety of ways.

0072However, since unintentional touch primarily occurs incident to writing, sensing the Writing grip itself is a powerful cue, particularly because the user typically adopts a writing grip prior to resting his hand on the display. Hence, a highly conservative palm-rejection procedure can simply reject any touch that occurs when the pen is held in the Writing grip. This, however, precludes intentional touches made by the non-preferred hand whenever the pen is held in the writing grip, which eliminates many desirable scenarios (e.g. panning and zooming with the nonpreferred hand), as well as simultaneous pen + touch gestures.

0073When a user plants his hand on the touch-sensitive screen of a computing device (e.g., tablet), it simultaneously induces a corresponding signal on the pen's motion sensors. (The device's motion sensors also pick up some of this motion, but it is damped somewhat by the greater mass of the device.) Nonetheless, motion signals from the pen and/or the touch-sensitive computing device (e.g., tablet) may be used in combination to help infer the type of contact that occurs, and may themselves be correlated with one another, such as to sense that the touch-sensitive computing device (e.g., tablet) is being moved while the pen is docked (attached) to it, or not.

0074When the hand holding the pen contacts the touch-sensitive computing device (e.g., tablet), the pen's motion exhibits a characteristic hard-contact profile similar to that seen with bump, whack, and thump gestures in other contexts. Implementations of the the pen and computing device sensor correlation technique look for a peak in the signal corresponding to the pen's motion sensors (for example, by using the second order finite difference on the three combined axes of t an accelerometer or gyro) that exceeds a minimum threshold within a given window, for example a 10-sample window. It is known exactly when to look for this signal because a palm plant produces a bump in the pen's motion at the same time that the touch screen detects the new contact. Tested implementations of the pen and computing device sensor correlation technique can identify this peak within as little as 5ms, or up to a maximum of 56ms after the touchdown event in the worst case. Although in general the focus here is on palm contact on the screen portion of a device, note that this bump signal can be applied to both front and back-of-device hand contacts, i.e. touch signals produced by the touchscreen, by grip sensors on the casing of the tablet, or both. Hence back/side of device contacts can differentiate various types of hand and pen contact as well.

0075Thus implementations of the pen and computing device sensor correlation technique employ a fairly low threshold for the bump signal, allowing even rather subtle palm contact to be sensed, while also trivially rejecting other motion signals that do not occur coincident to a new touch screen contact. This detection scheme works well for most normal pen interactions during writing.

0076For as long as the detected palm contact persists, implementations of the pen and computing device sensor correlation technique can also flag any new touches as a "palm" if they land within a perscribed radius (for example, a 300 pixel radius) of the initial contact. In some implementations this may incorporate a model of expected hand contact regions, given the user's grip and angle of holding the pen, as well as the relative screen orientation, such as the hand occlusion models , among others. One implementation of the pen and computing device sensor correlation technique provides feedback of the initial palm detection by playing a brief animation of a "radar circle," centered on the palm-down location, that fades as it expands. This advantageously provides non-distracting feedback that confirms to the user that the palm contact was successfully detected. Without this feedback, the user may be left with a nagging uncertainty as to whether or not their palm has triggered an undesired action (such as calling up a menu, or leaving behind an ink trace) that is currently occluded by the hand. Such feedback is optional, however, and may be disabled by the user or by specific applications.

5.1.3 <u>Permitting Intentional Touch</u>

0077Implementations of the pen and computing device sensor correlation technique permit simultaneous intentional touches, even when the palm is resting on the screen. In general, in some implementations, any new touch that occurs away from the palm-which is outside of a predetermined radius of a previously detected palm contact and which is not accompanied by a bump on the pen)--represents a true intentional touch. Some implementations of the pen and computing device sensor correlation technique use the first two additional touches that are not flagged as a palm contact to support a pinch-to-zoom gesture. Palm contact is ignored and does not interfere with pan/zoom, even if the palm moves. Other single or multiple finger or whole hand gestures can also be supported.

0078However, because some implementations of the pen and computing device sensor correlation technique still track the palm-rather than outright 'rejecting' it per se-this approach also can support techniques that use the palm location as an input, such as to help to correctly orient menus or to anchor objects.

5.1.4 <u>The Magnifier Tool vs. Full Canvas Zoom</u>

0079The functionality of the Magnifier/Loupe Tool 1100 shown in <figref idref="f0010">FIG. 11</figref> employs a focus-plus-context magnification technique (known as the "loupe") which is especially well suited to sketching tasks where the user wants to make a few detailed strokes without losing the overall context of the workspace. Implementations of the pen and computing device sensor correlation technique support both the Magnifier and Full-Canvas Zooming by sensing how the user is interacting with the pen and touch-sensitive computing device (e.g., tablet).

0080When the user stows the pen (in the Tuck or Palm grip), implementations of the pen and computing device sensor correlation technique recognize this grip. If the user then brings two fingers into contact with the display, implementations of the pen and computing device sensor correlation technique check the pen for a corresponding "bump" that occurs at approximately the same time as the touch signal. When implementations of the the pen and computing device sensor correlation technique see this combined pen bump + two-finger touch signal, it brings up the Magnifier/Loupe Tool. Note that the two-finger touch signal does not require the fingers to touch at precisely the same time; a short grace period is allowed so that non-simultaneous touches can be correctly recognized as calling up the Loupe. In some implementations a response to a single finger touch is delayed slightly for a second finger touch to arrive. In other implementations a single finger action is started and then canceled (or undone) if a second finger is detected during an allotted time window. Spatial constraints on how close (or far) the two finger contacts must be can also be applied, if desired.

0081If some implementations of the pen and computing device sensor correlation technique see a two-finger touch without any corresponding bump on the stylus, the implementations instead infer that the user made the touch with their other (nonpreferred) hand, which is not holding the pen. In some implementations, this then triggers the standard two-finger pan and zoom interaction to allow Full Canvas Zoom 1200 (rather than the focus-plus-context Magnifier Tool) as shown in <figref idref="f0010">FIG. 12</figref>.

0082In some implementations, the Magnifier/Loupe Tool zooms only the region of the canvas under the circular tool. The Magnifier/Loupe Tool interactively resizes itself according to the spread between the user's two fingers. The user may also touch down a finger on the border of the Magnifier/Loupe Tool to drag it to a new location. A single finger tap, or pen stroke, anywhere outside of the Magnifier/Loupe Tool is dismissed, leaving the canvas undisturbed at its original zoom level.

0083Note that since some implementations of the pen and computing device sensor correlation technique employ a minimum motion threshold to detect the bump signal, if the user touches their fingers down very lightly the pen may not detect a motion signal sufficient to exceed this threshold. Nonetheless, these thresholds of the pen and computing device sensor correlation technique are sufficient to detect the motions produced when users naturally bring their fingers to the screen with the pen is stowed.

5.1.6 <u>The Drafting Tools</u>

0084The Drafting Tools (<figref idref="f0011">FIG. 13</figref>, 1300) capability arose from the observation that users often maintain the Writing grip between bursts of writing activity. For example, during pauses users often rotate the wrist away from the screen, to bring the pen into the Writing-Half Supination pose. Hence, the Writing grip itself represents an interesting context that can be explicitly supported by providing various drafting tools that take into account that the user is holding the pen in a ready-to-write posture.

0085In some implementations of the pen and computing device sensor correlation technique, the user calls up the Drafting Tools menu explicitly, by a single contact such as touching down a single finger of the non-preferred hand (recognized by a single touch without a corresponding bump signal on the pen). If the pen is held in the Writing grip, this brings up a small palette that offers various pen + touch tool modes, including an Airbrush and a Compass and so forth. In some implementations of the the pen and computing device sensor correlation technique the Drafting Tools menu is invoked as soon as the user touches down his finger. In some technique implementations the Airbrush is the initial default mode. In some technique implementations the user can then tap on another tool (such as the Compass) to change modes. In some implemtations, all drafting tools are implemented as spring-loaded modes; that is, the mode is maintained only so long as the user holds down his finger. Note also that the Drafting Tools menu, by default, can activate the most recently used tool (mode) when the user initiates contact. This makes repeated use of the same tool at multiple locations more efficient. Implementations of the Airbrush tool, the Compass tool, as well as a Single-Tap Virtual Pen Barrel button are described in more detail below.

0086Additionally, in some implementations, an "advanced drafting tools" capability can use sensors on the pen and sensors on the touch-sensitive computing device to detect that the sensor pen is held in a writing grip in the user's preferred hand and to detect a two-finger touch gesture such as a pinching motion at the touch-sensitive display of the computing device with the user's non-preferred hand. The pinching motion with the user's non-preferred hand brings up a set of advanced drafting tools when the pen is ready to write. These special tools further support use of the pen that benefits from a second touch. For example, these drafting tools can include a ruler or alignment edge, a French curve or a function to pull a new sheet of paper (with two-finger position and rotation). The user may also move back and forth between the Advanced Drafting Ttools and the standard drafting tools by lifting (or putting back down) one of the two fingers; the tool set shown corresponds to the number of fingers placed on the display.

5.1.6.1 <u>Airbrush Tool</u>

0087One implementation of the Airbrush tool initially shows cursor feedback, as a gray dashed ellipse, of where the airbrush will spray if the user starts the paint flow. The user indicates where the airbrush tool should spray by the position of the (non-preferred hand) finger. This is necessary because (as noted previously) inertial sensing cannot determine the absolute <i>(x,y,z)</i> location of the pen tip above the tablet, only a 3D orientation relative to the tablet. The relative sensing of the pen orientation can be demonstrated by rotating the touch-sensitive computing device (e.g., tablet), rather than the stylus.

0088In one implementation, the user controls the spray ellipse by changing the azimuth and elevation angles of the pen with respect to the touch-sensitive screen/tablet surface. The user can hold the pen well above the screen, making it easy to angle the pen as desired, unlike a previous exploration of an airbrush-like tool which did not employ the pen tilt angles, likely because it is difficult to reorient the pen while also keeping it within the limited ∼15mm proximity sensing range of the tablet. In some implementations, a separate sensing channel (such as a proximity sensor) may be used to determine the altitude of the pen above the screen, which then determines the size (extent) of the spray ellipse.

0089In some implementations, the user turns the paint flow on and off by tapping their index finger on the barrel of the pen, which is sensed as further detailed below. When the user activates the spray, the feedback changes to a bold red dashed ellipse to give clear feedback of the shape of the spray being produced. In one prototype, the highly transparent ellipses are "sprayed" onto the canvas. The user may also adjust the size of the ellipse, or the transparency of the spray, by sliding the finger in contact with the pen barrel as if it were a physical slider control. The motion of the finger contact can be sensed and tracked by the pen's grip sensors.

5.1.6.2 <u>Single-Tap Virtual Pen Barrel Button</u>

0090Implementations of the technique successfully supports single-tap activation of a virtual barrel button by strategically combining some or all of the stylus sensing channels. Implementations of the single-tap virtual pen barrel button described herein use grip sensing data in addition to motion data of the pen.

0091To identify candidate tap events, some implementations of the pen and touch-sensitive computing device sensor correlation technique look for a bump signal on the barrel of the pen from the finger tap at the same time that a new touch contact appears on a capacitance image map created by using capacitance sensors on the barrel of the pen. However, this alone may not be sufficient to filter out false positives produced by re-gripping the pen, because shifting grips can also produce bump signals coincident with new finger contacts. To filter these out, some implementations of the technique rely on the observation that users hold the pen still in a Writing grip to maintain a stable tripod grasp when they lift the index finger to tap on the barrel (per observation B9). This is advantageous because detection of the Writing grip provides a gate to make sure false detections are unlikely to occur. Various implementations of the technique therefore look at the ongoing accelerometer and gyro signals and compute a simple time-decaying motion signal to determine whether a device is moving or held still. Then only the candidate tap events that occur when the pen is not moving are accepted, which effectively filters out any false contacts. In one working implementation, the pen must remain in a new moving (or not moving) state for at least 100ms. Otherwise, the pen barrel tap itself can trigger brief activation of a "moving" signal, which of course would thwart recognition of the barrel tap.

5.1.6.3 <u>Compass Tool</u>

0092The Drafting Tools palette also includes a Compass, which supports a pen + touch mode where the pen is constrained to draw circular arcs centered about the current location of the finger (again of the non-preferred hand).

5.1.7 <u>The Pen Controls</u>

0093As another example, a single-finger contact such as a tap while the pen is stowed brings up a small in-place palette containing the Pen Controls (<figref idref="f0011">FIG. 14</figref>, 1400), allowing the user to change modes, or to modifiy the pen color and stroke thickness, without making a round-trip to a toolbar docked at the edge of the workspace. This example again takes advantage of the bump generated on the pen when the user taps the touch screen from an extension grip, using any single finger to make touch screen contact. The tools appear next to the finger. The user may then interact with the radial menus using either pen or touch, as studies have consistently found that users expect pen and touch to be interchangable for UI controls. Note that in some implementations these or related controls may be activated by either a tap (i.e. a finger-down + finger-up sequence) or by a tap-and-hold (finger-down + maintaining finger contact with the screen). The latter is particularly conducive to spring-loaded modes, which maintained as long as the finger remains in contact with the digitizer.

5.1.8 <u>Canvas Tools</u>

0094The Canvas Tool (<figref idref="f0012">FIG. 15</figref>, 1500) uses sensors of the pen and on the touch-sensitive computing device to detect when a user is holding the sensor pen in a non-writing position in the user's preferred hand and to detect when a single contact such as for example a finger tap with the user's non-preferred hand on the touch screen of the computing device. These correlated concurrent actions cause a menu of canvas tools to be displayed on the display of the computing device. For example, this menu of tools could include undo/redo, cut-copy-paste, new page, search and similar commands. Like the pen tools, in some implementations the canvas tools and related tools can be activated by either a tap or tap-and-hold gesture, depending on the implementation.

5.1.9 <u>Touch-Sensitive Computing Device/Tablet Grip Detection</u>

0095Implementations of the pen and computing device sensor correlation technique use capacitive grip sensing on the back and sides of the touch-sensitive computing device case to detect a number of additional contacts and to determine their contexts.

5.1.9.1 <u>Thumb Menu and Handedness Detection</u>

0096There are many instances where the user picks up and holds a touch-sensitive computing device (e.g., tablet or other similar device) with both hands, making the pen unavailable. Implementations of the pen and computing device sensor correlation technique use grip to sense which hand the user is holding the touch-sensitive computing device with. Implementations of the pen and computing device sensor correlation technique then use this to summon a Thumb Menu (<figref idref="f0004">FIG. 5</figref>, 534) at the appropriate side of the touch-sensitive computing device (e.g., tablet), which allows the user to activate various buttons and menus directly with the thumb. If the user grasps the touch-sensitive computing device (e.g., tablet) with a second hand, implementations of the pen and computing device sensor correlation technique leave the Thumb Menu visible at the side where it first appeared. In other implementations, it may be split (or duplicated) across the two thumbs.

0097If the user is observed grasping the pen while holding the touch-sensitive computing device with one hand, implementations of the pen and computing device sensor correlation technique can immediately infer the user's handedness. The hand holding the pen is inferred to be the preferred hand, and the hand holding the touch-sensistive computing device (e.g., tablet) may be presumed to be not the preferred hand. Some technique implementations can also tell if the pen-holding hand grabs or grips the touch-sensitive computing device (e.g., tablet) by looking for a bump on the pen at the time the user's hand contacts a grip sensor on the touch-sensitive computing device (e.g., tablet).

5.1.9.2 <u>Detecting Unintentional Thumb Contact</u>

0098In some implementations of the pen and computing device sensor correlation technique when the Thumb Menu first appears, it fades in over a short (e.g., 1.5 second) interval, and likewise if the user lets go of the touch-sensitive computing device it fades out after a short time (e.g., 350ms). The purpose of this animation feedback is to present the Thumb Menu in a tentative state, so that if the user's thumb strays onto the touch screen while picking up the tablet, the thumb contact can be ignored or otherwise treated as a likely-unintentional input.

0099Implementations of the pen and computing device sensor correlation technique infer that a thumb represents an unintentional touch if it occurs at the same time as (or soon after) a new hand grip on the corresponding back portion of the tablet case. Some implementations of the pen and computing device sensor correlation technique then detect the thumb as an unintentional contact, and freeze the fade-in of the Thumb Menu if the unintentional thumb contact overlaps it. This feedback indicates to the user that the thumb contact has been recognized, but intercepted to prevent accidental activation of the menu. The user can then intentionally interact with the Thumb Menu, if desired, simply by lifting the thumb and bringing it back down on the menu. The fade-in animation continues as soon as the user lifts his or her thumb. If the user does not place the thumb on the screen when picking up the touch-sensitive computing device, the fade-in also serves as secondary cue that the Thumb Menu is fully ready for use. Since accidental activation mainly tends to occur when the user first grasps the touch-sensitive computing device, after a few seconds elapse it is assumed that any hand contact with the screen was intentional. This therefore illustrates how the detection scheme of implementations of the pen and computing device sensor correlation technique block unintentional touch, while also allowing intentional touches to get through, unlike simple thumb-blocking heuristics which ignore any hand contact near the edge of the screen in certain applications.

5.1.10.3 <u>Handoff: Passing the Pen or the Touch-Sensitive Computing Device to Another User</u>

0100In implementations of the pen and computing device sensor correlation technique passing a touch-sensitive computing device (e.g., a tablet) or a pen to another user is used as a way to offer an alternative, more physical semantic of sharing content with another user. Studies of passing prehension and other user observations indicate that users go through a sequence of specific motions. First they extend the object while holding it approximately level to offer it to the other person, then they maintain their grip until the other person has firmly grasped the object. The person passing the device then lets go, and the other person brings it in closer to their body, often while also orienting the screen to his or her preferred viewing angle. All or parts of this sequence can be sensed to detect passing prehension interactions.

0101For example, <figref idref="f0012">FIG. 16</figref> shows a primary user 1602 passing a pen 1604 to a secondary user 1606. The grip of the primary user on the touch-sensitive pen is sensed (e.g., a baton grip). At the same time, or almost the same time, the grip of the secondary user on the touch-sensitive pen is also sensed. The grips of the primary and secondary users on the pen are correlated to determine the context of the grips and to initiate a command in an application executing on the touch-sensitive pen (and/or computing device in communication with the pen). For example, the grip of the secondary user can be determined to represent a handoff of the pen to the secondary user from the primary user. In this case data can be transferred from the primary user to the secondary user via the handoff of the pen. Alternately, one or more capabilities of the touch-sensitive pen can be restricted following the handoff.

0102Similarly, <figref idref="f0013">FIG. 17</figref> shows a primary user 1702 passing a touch-sensitive computing device 1704 to a secondary user 1706. Some implementations of the pen and computing device sensor correlation technique employ sensors to detect the grips of the two users on the touch-sensitive computing device using the devices's case grip sensing to determine when each user is grasping the device, and the device's orientation to determine if it is level. The detected grips and the orientation of the device (as well as possibly other data) can be used to assign each user a different role with a different set of permissions in using the touch-sensitive computing device. When these conditions are detected in some implementations a special annotation layer peels over the screen, as if a transparency or a sheet of vellum had been dropped over the display. The other user is then free to annotate the content, but not to make digital copies or navigate to other documents or files. This is a very different and much more limited form of sharing than the digital transfer of information supported by other cross-device information transfer techniques. Implementations of the pen and computing device sensor correlation technique do not trigger Tablet Handoff when a single user holds the display up with two hands; during such interactions, users tend to angle the tablet towards themselves, and thus it is not level. In a similar manner, interactions where the device is perfectly flat on a desk can be detected such that users will not unintentionally trigger Tablet Handoff if they happen to touch or hold their device with both hands.

<u>5.2 History Prior to the Correlation with Bump on Sensors.</u>

0103Some implementations of the pen and computing device sensor correlation technique analyze past data of sensors (obtained, for example, by constantly recording sensor data) on the touch-sensitive pen and/or the touch-sensitive computing device. This data can be correlated, for example, with sensor data taken at the time of a bump that is associated with touch down of the pen to confirm or reinforce that the touch was correlated with the movement of the pen. Implementations of the technique that use past correlation data in addition to present correlation data can be more robust than those that do not. For example, there may be an acceleration/decleration pattern associated with the pen a little bit before sensor data associated with a person putting down his or her hand to write that is recognized at the time of the bump detection to confirm that the contact is a palm and not by chance a touch down with the non-dominant hand while the pen is being moved at the same time.

6.0 <u>Exemplary System Hardware:</u>

0104In a prototype implementation, to support the range of context-sensing techniques envisioned, custom hardware was designed to augment the pen and touch-sensitive computing device with inertial sensors and capacitive grip sensing, as well as custom software/firmware to handle simultaneous pen + touch events from the touch screen. The description of the following exemplary prototype hardware and software/firmware is not meant to be limiting but is provided to show how implementations discussed herein could be implemented. Those with ordinary skill in the art will realize that many other implementations are possible.

6.1. <u>Pen Hardware Design</u>

0105In one exemplary working implementation, a flexible capacitive grid consisting of 7x30 sensing elements covers the entire barrel of a pen, which was wrapped in heat-shrink tubing to protect the sensors and to provide a smooth and easy-to-grip cylindrical surface for sensing. The interior of the pen consists of a 3D-printed case that holds a miniature electromagnetic pen, a 4 AAAA battery, and custom circuitry. For inertial sensing a gyroscope as well as an accelerometer/magnetometer module was used. For capacitive sensing a touch controller was employed. The pen streams all data to a host computer using a transceiver operating at 2Mbps. A microcontroller runs the firmware.

0106In this particular implementation, all inertial sensor data is streamed off the pen at 130Hz, and the 7x30 capacitance map at 30Hz. The resulting pen is 19cm long with a 13mm external diameter.

6.2 <u>Touch-Sensitive Computing Device/Tablet Hardware Design</u>

0107The touch-sensitive computing device in this exemplary prototype system is a tablet computer. The tablet case covers the entire back surface and sides of the tablet. The rigid case is constructed from printed circuit boards consisting of 44x26 capacitive sensing elements. There is a small insensitive area (in the middle of the case on the back side) where the integrated circuits are mounted. The case includes the same sensor components as the pen, except there are four touch controllers for different parts of the capacitive grid. The tablet sensor data is streamed via USB, with the tablet's inertial sensors sampled at 100Hz and the tablet's 44x26 capacitance map sampled at 25Hz.

6.3 <u>Simultaneous Pen and Touch from the Touch screen</u>

0108Multi-touch and pen events were handled by intercepting them directly from a Human Interface Device (HID) controller. Using this approach a Samsung Corporation Series 7 Slate was used for the prototype that can report up to 8 touch contacts simultaneously with pen input.

6.4 <u>Software/Firmware Design</u>

0109Some technique implementations aggregate and time-stamp the pen and tablet sensor data on an external PC for processing, and then transmit relevant input events to the tablet for presentation in a user interface. Furthermore, some technique implementations compute grip recognition on the external PC. As such, one implementaton consists of a distributed system with four cooperating components: the touch-sensitive pen, the tablet case, the tablet itself, and the external PC. It should be noted, however, that aggregating and time-stamping the sensor data, as well as grip recognition, can be peformed on the tablet computer thereby obviating the need for the standalone personal computer (PC). The case can be directly integrated into the touch-sensitive computing device. In some implementations, time-stamped data from all the distributed event streams are queued up in synchronized buffers. Some technique implementations then handle events from these buffers up until the latest time-stamp for which all events are available. In other implementations, some events may be dispatched in real-time but then subsequently modified if future event samples arrive from other sources that would alter their interpretation. This can be achieved by delaying feedback, displaying tentative feedback, or undoing speculative actions in end-user applications.

6.5 <u>Inertial Sensor Fusion</u>

0110Some techique implementations combine accelerometer, gyro, and magnetometer sensor inputs using a direction cosine matrix algorithm. This produces stable yaw, pitch, and roll values in an east-north-up Earth coordinate frame. This is used to derive the pen orientation in a consistent reference frame relative to the tablet.

6.6 <u>Pen Grip Classification</u>

0111The prototype implementation recognizes four distinct grips: Writing, Tuck, Palm, and No Grip (for when the pen is not held). Per observed behavior B4 (Grip vs. Pose), the grip recognition considers the pattern of hand contact (capacitive grip sensing) as well as the pose (orientation) of the stylus. The prototype implementation of the system processes the incoming data to extract salient features and then trains a multi-class classifier to extract the pen grips. Technique implementations perform a multi-class classification of the pen grip patterns by using a set of one-vs-all learners, where each learner is a Support Vector Machine (SVM) classifier. The result is a probability distribution over all four grips.

0112Technique implementations select features for grip classification that take into account unique considerations of the pen form-factor. In particular, since the pen is symmetrical (cylindrical) along the axis of the pen barrel, the sensed grips are agnostic to the roll angle of the pen. Similarly, whether the user grips the pen lower or higher on the barrel, or the size of his hand, should not affect the grip classification. Thus some implementations of the system compute a normalized image invariant with respect to both grip height and roll angle. From the raw 7x30 capacitance map, the system fits the non-zero capacitance values into a 7x10 normalized image. The capacitance map is shifted in the y-dimension so that the first row of lit (non-zero capacitance) pixels corresponds to the bottom of the normalized image, and then scale the non-zero capacitance pixels to fit in the 10 rows of the normalized image. The features employed for grip classification therefore include the pen yaw and pitch angles, the normalized grip image, as well as the normalized grip histogram in the x and y dimensions. Technique implementations also include features for the number of lit (non-zero) pixels, and the pixel sum of all 7x30 capacitance values from the raw capacitance map.

6.6.1 <u>Grip Training Dataset Collection</u>

0113In one implementation, nine right-handed participants (4 female), all of whom had prior exposure to pen and tablet use, were used to generate a grip training dataset. Users were led through a script illustrating specific grips and actions to perform in each grip. These included stowing the pen while using touch (per observed behavior B1) from both the Tuck and Palm grips (behavior B2). Different sequences of tasks were also included to capture various common transitions between grips (behavior B3). Users were led through the full range of supination for each grip (behavior B4) which included transitions between Writing and the single-finger and two-finger extension grips (behavior B5), with articulation of direct-manipulation gestures such as tapping, dragging, and pinching. However, no particlar tripod grip to use was specified, but rather users were allowed to hold the pen naturally so that the dataset would capture cross-user variations in Writing grips (per behavior B6). The data collection lasted approximately 15 minutes per user, with a total of 1200 samples for each user, per grip, yielding a total training dataset of 1200 × 3 grips × 9 users = 32400 samples.

6.6.2 <u>Pen Grip Recognition Accuracy</u>

0114A10-fold cross-validation using the collected grip training dataset yielded an overall accuracy of 88% for a user-independent model. A separate check with nine additional right-handed users was conducted, none of whom had contributed data to the training dataset. This yielded user-independent grip recognition accuracy of 93% for the Writing grip, 93% for the Tuck grip, and 77% for the Palm grip. The relatively low recognition rate for the Palm grip appeared to stem from several users' tendency to hold the pen very lightly in this grip, resulting in a somewhat inconsistent pattern of contact sensed by the capacitive grip array. However, the system was still able to distinguish Writing vs. non-writing grips (i.e. Tuck or Palm) with 97% accuracy. Since most interaction techniques do not depend on any distinction between the Tuck versus Palm grips, this user-independent grip model, which works well enough even without collecting training data for newly-encountered users, was used.

7.0 <u>Exemplary Operating Environments:</u>

0115Implementations of the pen and computing device sensor correlation technique described herein are operational within numerous types of general purpose or special purpose computing system environments or configurations. <figref idref="f0014">FIG. 18</figref> illustrates a simplified example of a general-purpose computer system in combination with a pen or pen enhanced with various sensors with which various implementations and elements of the pen and computing device sensor correlation technique, as described herein, may be implemented. It should be noted that any boxes that are represented by broken or dashed lines in <figref idref="f0014">FIG. 18</figref> represent alternate implementations of the simplified computing device and sensor pen, and that any or all of these alternate implementations, as described below, may be used in combination with other alternate implementations that are described throughout this document.

0116For example, <figref idref="f0014">FIG. 18</figref> shows a general system diagram showing a simplified touch-sensitive computing device 1800. In general, such touch-sensitive computing devices 1800 have one or more touch-sensitive surfaces 1805 or regions (e.g., touch screen, touch sensitive bezel or case, sensors for detection of hover-type inputs, optical touch sensors, etc.). Examples of touch-sensitive computing devices 1800 include, but are not limited to, touch-sensitive display devices connected to a computing device, touch-sensitive phone devices, touch-sensitive media players, touch-sensitive e-readers, notebooks, netbooks, booklets (dual-screen), tablet type computers, or any other device having one or more touch-sensitive surfaces or input modalities.

0117To allow a device to implement the pen and computing device sensor correlation technique, the computing device 1800 should have a sufficient computational capability and system memory to enable basic computational operations. In addition, the computing device 1800 may include one or more sensors 1810, including, but not limited to, accelerometers, gyroscopes, magnetometer, finger print detectors, cameras including depth cameras, capacitive sensors, proximity sensors, microphones, multi-spectral sensors, , etc. As illustrated by <figref idref="f0014">FIG. 18</figref>, the computational capability is generally illustrated by one or more processing unit(s) 1825, and may also include one or more GPUs 1815, either or both in communication with system memory 1820. Note that the processing unit(s) 1825 of the computing device 1800 of may be specialized microprocessors, such as a DSP, a VLIW, or other micro-controller, or can be conventional CPUs having one or more processing cores, including specialized GPU-based cores in a multi-core CPU.

0118In addition, the computing device 1800 may also include other components, such as, for example, a communications interface 1830 for receiving communications from sensor pen device 1835. The computing device 1800 may also include one or more conventional computer input devices 1840 or combinations of such devices (e.g., pointing devices, keyboards, audio input devices, voice or speech-based input and control devices, video input devices, haptic input devices, touch input devices, devices for receiving wired or wireless data transmissions, etc.). The computing device 1800 may also include other optional components, such as, for example, one or more conventional computer output devices 1850 (e.g., display device(s) 1855, audio output devices, video output devices, devices for transmitting wired or wireless data transmissions, etc.). Note that typical communications interfaces 1830, input devices 1840, output devices 1850, and storage devices 1860 for general-purpose computers are well known to those skilled in the art, and will not be described in detail herein.

0119The computing device 1800 may also include a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer device 1800 via storage devices 1860 and includes both volatile and nonvolatile media that is either removable 1870 and/or non-removable 1880, for storage of information such as computer-readable or computer-executable instructions, data structures, program modules, or other data. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media refers to tangible computer or machine readable media or storage devices such as DVD's, CD's, floppy disks, tape drives, hard drives, optical drives, solid state memory devices, RAM, ROM, EEPROM, flash memory or other memory technology, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other device which can be used to store the desired information and which can be accessed by one or more computing devices.

0120Storage of information such as computer-readable or computer-executable instructions, data structures, program modules, etc., can also be accomplished by using any of a variety of the aforementioned communication media to encode one or more modulated data signals or carrier waves, or other transport mechanisms or communications protocols, and includes any wired or wireless information delivery mechanism. Note that the terms "modulated data signal" or "carrier wave" generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wired connection carrying one or more modulated data signals, and wireless media such as acoustic, RF, infrared, laser, and other wireless media for transmitting and/or receiving one or more modulated data signals or carrier waves. Combinations of the any of the above should also be included within the scope of communication media.

0121Retention of information such as computer-readable or computer-executable instructions, data structures, program modules, etc., can also be accomplished by using any of a variety of the aforementioned communication media to encode one or more modulated data signals or carrier waves, or other transport mechanisms or communications protocols, and includes any wired or wireless information delivery mechanism. Note that the terms "modulated data signal" or "carrier wave" generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wired connection carrying one or more modulated data signals, and wireless media such as acoustic, RF, infrared, laser, and other wireless media for transmitting and/or receiving one or more modulated data signals or carrier waves. Combinations of the any of the above should also be included within the scope of communication media.

0122Further, software, programs, and/or computer program products embodying the some or all of the various implementations of the pen and computing device sensor correlation technique described herein, or portions thereof, may be stored, received, transmitted, and/or read from any desired combination of computer or machine readable media or storage devices and communication media in the form of computer executable instructions and/or other data structures.

0123Finally, the pen and computing device sensor correlation technique described herein may be further described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The implementations described herein may also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices, or within a cloud of one or more devices, that are linked through one or more communications networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices. Still further, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor.

0124The sensor pen device 1835 illustrated by <figref idref="f0014">FIG. 18</figref> shows a simplified version of a pen or pen augmented with pen sensors 1845, logic 1865, a power source 1875 (e.g., a battery), and basic I/O capabilities 1885. As discussed above, examples of pen sensors 1845 for use with the sensor pen device 1835 include, but are not limited to, inertial sensors, cameras including depth cameras, proximity sensors, finger print sensors, galvanic skin response sensors, accelerometers, pressure sensors, grip sensors, near-field communication sensors, RFID tags and/or sensors, temperature sensors, microphones, magnetometers, capacitive sensors, gyroscopes, etc.

0125In general, the logic 1865 of the sensor pen device 1835 is similar to the computational capabilities of computing device 1800, but is generally less powerful in terms of computational speed, memory, etc. However, the sensor pen device 1835 can be constructed with sufficient logic 1865 such that it can be considered a standalone capable computational device.

0126The power source 1875 of the sensor pen device 1835 is implemented in various form factors, including, but not limited to, replaceable batteries, rechargeable batteries, capacitive energy storage devices, fuel cells, etc. Finally, the I/O 1885 of the sensor pen device 1835 provides conventional wired or wireless communications capabilities that allow the sensor pen device to communicate sensor data and/or other information to the computing device 1800.

0127The foregoing description of the pen and computing device sensor correlation technique has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. Further, it should be noted that any or all of the aforementioned alternate implementations may be used in any combination desired to form additional hybrid implementations of the pen and computing device sensor correlation technique. For example, various devices used to enable some of the many implementations of the pen and computing device sensor correlation technique described herein include pens, pointers, pen type input devices. However, the functionality described herein may be implemented in any desired form factor, e.g., phone, wand, staff, ball racquet, toy sword, etc., for use with various gaming devices, gaming consoles, or other computing devices. Further, the sensor pens described herein are adapted to incorporate a power supply and various combinations of sensors including, but not limited to inertial sensors, cameras including depth cameras, accelerometers, pressure sensors, grip sensors, near-field communication sensors, RFID tags and/or sensors, temperature sensors, microphones, magnetometers, capacitive sensors, gyroscopes, etc., in combination with various wireless communications capabilities for interfacing with various computing devices. Note that any or all of these sensors may be multi-axis or multi-position sensors (e.g., 3-axis accelerometers, gyroscopes, and magnetometers). In addition, in various implementations, the sensor pens described herein have been further adapted to incorporate memory and/or computing capabilities that allow the sensor pens to act in combination or cooperation with other computing devices, other sensor pens, or even as a standalone computing device.

0128While the pen and computing device sensor correlation technique senses actual touch to a sensor pen and a touch-sensitive computing device, it may also be employed with virtual touch inputs. Virtual touch inputs relative to projected displays, electronic whiteboards, or other surfaces or objects are treated by the pen and computing device sensor correlation technique in the same manner as actual touch inputs on a touch-sensitive surface. Such virtual touch inputs are detected using conventional techniques such as, for example, using cameras or other imaging technologies to track user finger movement relative to a projected image, relative to text on an electronic whiteboard, relative to physical objects, etc.

0129In addition, it should be understood that the pen and computing device sensor correlation technique is operable with a wide variety of touch and flex-sensitive materials for determining or sensing touch or pressure. For example, one touch-sensing technology adapted for use by the pen and computing device sensor correlation technique determines touch or pressure by evaluating a light source relative to some definite deformation of a touched surface to sense contact. Also, it should be noted that sensor pens, as discussed herein, may include multiple types of touch and/or pressure sensing substrates. For example, sensor pens may be both touch-sensitive and/or pressure sensitive using any combination of sensors, such as, for example, capacitive sensors, pressure sensors, flex- or deformation-based sensors, depth sensors, etc.

0130It is intended that the scope of the disclosure be limited not by this detailed description, but rather by the claims appended hereto. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.