Area selection for hand held devices with display

CROSS-REFERENCE TO RELATED APPLICATIONS

0001This application claims the benefit of provisional patent application Ser. No. 61/470,444, filed 2011 Mar. 31 by the present inventor, which is incorporated by reference.

STATEMENTS REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

0002Not applicable.

REFERENCE TO A MICROFICHE APPENDIX

0003Not applicable

BACKGROUND OF THE INVENTION

00041. Field of the Invention

0005The present invention generally relates to hand held devices with display, and more particularly to the process of selecting a desired area, a marker position, or multiple objects from the contents view associated with the display of the hand held devices.

00062. Description of the Related Art

0007In this specification, I refer to the Area Selection operation as the common user activity performed on information processing devices with visual displays for the purpose of defining and selecting a portion of the contents of a displayed file, or for the purpose of selecting multiple objects represented by icons on the display. The contents of the displayed file may be graphical, text, media, or any other type of data that may be displayed on the device's display.

0008Area selection within the contents of a displayed file is typically associated with many user interface functions, including Cut and Paste, Drag and Drop, Copy, Highlight, Zoom in, and Delete. Both the Cut and Paste and Copy operations are used to select a portion of the display and copy it into another place of the same display or via the common clipboard onto other active or inactive applications of the device. The Cut and Paste operation causes the originally selected area to be deleted while the Copy operation preserves the originally selected area. The area selection operation within a graphical file is typically selected within a bounding rectangle whose two corners are specified by the user. For text documents, the area selection is a block selection operation, where the selected block is defined between two user selected endpoints placed at two character positions within the text.

0009For some applications, the area selection operation highlights a portion of the display which is then used as an input for some processing (e.g. speech synthesis, graphical processing, statistical analysis, video processing, etc.). Area selection can be also used to select multiple objects that are not part of a single file, where the individual graphic objects are represented by icons spread across the display.

0010Desktop systems typically use a pointer device like a mouse or a joystick to select the cut and paste area. Other common techniques include touch screen and voice control selections. When selecting a block of text one can often use pre-assigned keyboard commands.

0011Hand held devices with a small physical display often must show a virtual stored or a computed contents view that is larger than the screen view of the physical display. Since only a portion of the contents display (also called “virtual display”) can be shown at any given time within the screen view, area selection on hand held devices poses more of a challenge than desktop area selection. This is particularly the case when the desired selected area from the virtual display is stretching beyond the small screen view.

0012Today's most popular user interface in hand held devices is the touch screen display. The touch screen display enables the user to create single-touch and multi-touch gestures (also called “touch commands”) to navigate (or “scroll”) the display as well as to activate numerous functions and links. There are two main limitations for the touch screen display area selection operation: the setting of the area corners, and the placement accuracy due to the relatively wide finger tip.

0013When setting area corners for a selected area by touch gestures, one encounters the problem that the touch gesture may inadvertently navigate the screen (or follow a link) instead of placing the corner. Alternatively, touch gestures intended for view navigation may be confused for corner selection during the process. This problem is currently solved by training the user to perform precise and relatively complex touch gestures that attempt to distinguish between navigation commands and corner placement commands. This further poses a major disadvantage for most users who must spend the time to gain expertise in the precise handling of their device touch interface.

0014U.S. Pat. No. 7,479,948 by Kim et al. describes a method for area selection using multi-touch commands where the user touches simultaneously with several fingers to define a selected area. These unique multi-touch commands limit confusion with view navigation commands, but they are cumbersome and require extensive user training This approach seems to be limited for a selected area that is small enough to be fully enclosed within the screen view of the display. The complexity of using touch commands for area selection is further illustrated in US patent application 2009/0189862 by Viberg, where the operation of moving a word is facilitated into a complex four touch operation.

0015Another approach that utilizes complex touch gestures is illustrated in the article “Bezel Swipe: Conflict-Free Scrolling and Multiple Selection on Mobile Touch Screen Devices” by V. Roth and T. Turner, In CHI 2009, Apr. 4-9, 2009, Boston, Mass., USA. Bezel Swipe requires an initial gesture that starts with the bezel, a touch insensitive frame around the boundary of the display. From that point, the user touches the screen and moves the finger to select the desired area, ending the selection process by lifting the finger. Solutions like the Bezel Swipe and the patents mentioned above are particularly cumbersome when the desired selected area or objects span beyond the boundaries of the display. Often selection errors are inadvertently made and the user must re-do the selection process.

0016Touch based area selection of the prior art also face the problem of inaccurate corner points positioning due to the wide contact area between the user's finger and the screen. Stylus devices with sharp tips have been well known to provide accurate positioning of selection points. US patent application 2010/0262906 by Li attempts to solve the problem of distinguishing between area selection commands and view navigation commands. It proposes a special stylus that has a built in key that transmits a special instruction to the device to perform a selection and copy command at the area touched by the stylus. US patent application 2008/0309621 by Aggarwal et al. teaches the use of a proximity based stylus which can interact with the device screen without necessitating that the stylus makes physical contact with the display. The area selection process is started by making a physical contact between the stylus and the display at one corner of the desired selected area. The user then hovers the stylus slightly over the display to navigate to the other corner of the selected area. The two preceding patent applications are disadvantaged by the need of a special active stylus, and they do not perform well when the selected area is much larger than the size of the screen.

0017U.S. Pat. No. 7,834,847 by Boillot et al. offers a touch-less control of the screen of a mobile device using a sensing system for detecting special movement of the user's fingers in the space above the display. The patent teaches the use of special finger gestures to initiate area selection and cut and paste operations. This solution requires a complex and expensive system for detecting the touch-less finger gestures and it burdens the user with the need of extensive gesture training, which is still prone to errors.

0018Area selection in hand held devices can be made also by a joystick or special keyboard, as illustrated in US patent application 2006/0270394 by Chin, which uses a multi-stage hardware button to activate special functions like cut and paste. The need of activating different positions of the button creates cumbersome user interface as the button needs continuously be switched from selection mode to view navigation mode.

0019The view navigation system of a mobile device may utilize a set of rotation and movement sensors (like a tri-axis accelerometer, gyroscope, tilt sensor, camera tilt detector, or magnetic sensor). An early tilt and movement based view navigation system is disclosed in my U.S. Pat. Nos. 6,466,198 and 6,933,923 which have been commercialized under the trade name RotoView. This system is well adapted to navigate the device's screen view across an arbitrarily large contents view and it provides coarse and fine modes of navigation. At fine mode navigation, relatively large orientation changes cause only small view navigation changes. Conversely, at coarse navigation mode, relatively small orientation changes cause large view navigation changes. Later examples include U.S. Pat. No. 7,667,686 by Suh which shows how a selected area from a virtual display may be dragged and dropped. However, the '686 patent completely ignores the problem of area selection which is central to the present invention.

0020Therefore, it would be desirable to provide methods and systems that can perform area selection on hand held devices with display without the need of sophisticated stylus devices, proximity detectors, or special buttons. Furthermore, it should not require extensive user training and it should be accurate and error free when selecting areas that are either smaller or larger than the display size.

BRIEF SUMMARY OF THE INVENTION

0021With these problems in mind, the present invention seeks to provide intuitive, convenient, and precise area selection techniques for hand held devices with a small display.

0022In one embodiment of the present invention, a hand held device with touch screen display uses a combination of both touch screen gestures and tilt and movement based view navigation modes. For normal operation, view navigation can be made by various touch gestures or by tilt and movement based view navigation. During the area selection operation, the device reserves the touch commands only for the selection of the corner points of the selected area. Once the first corner is selected, the device uses tilt and movement view navigation exclusively to reach the general area of the second corner. Once the area of the second corner is reached, the user completes the area selection by touching the desired second corner. This guarantees that corner selection touch gestures may not be wrongly interpreted as view navigation commands.

0023If the contents view displays text only, the area selection is essentially enclosed between two endpoints along the text. The present invention simplifies the tilt and movement based view navigation to correlate the three dimensional tilt and movement gestures into a linear up/down move along the text and setting the endpoints for the selected text at words boundaries.

0024In yet another embodiment of the present invention, a special touch gesture provides both initiation of the area selection operation as well as the actual selection of the first corner of the selected area.

0025The present invention also offers marker repositioning techniques to allow precise adjustment of the corner locations that are placed by the relatively inaccurate touch commands that use the relatively wide finger tip. These techniques can be used to reposition any marker set by a touch command.

0026Another embodiment of the present invention offers a method for boundary adjustment of a user selected area to reduce the affect of unwanted truncation of contents. Such a contents aware method offers the user an automatic boundary adjustment choice at the end of the area selection process to eliminate the need to repeat the entire process.

0027These and other objects, advantages, and features shall hereinafter appear, and for the purpose of illustrations, but not for limitation, exemplary embodiments of the present invention are described in the following detailed description and illustrated in the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

0028The drawings are not necessarily drawn to scale, as the emphasis is to illustrate the principles and operation of the invention. In the drawings, like reference numerals designate corresponding elements, and closely related figures have the same number but different alphabetic suffixes.

0029<figref idref="DRAWINGS">FIG. 1</figref> shows an example of contents view with a defined selected area.

0030<figref idref="DRAWINGS">FIG. 2A</figref> to <figref idref="DRAWINGS">FIG. 2D</figref> detail the process of marking the selected area from the contents view shown in <figref idref="DRAWINGS">FIG. 1</figref> in accordance with one embodiment of the present invention.

0031<figref idref="DRAWINGS">FIG. 3</figref> illustrates the block diagram of the embodiment of a hand held device with touch screen display incorporating the present invention.

0032<figref idref="DRAWINGS">FIG. 4A</figref> outlines the software flow diagram for the embodiment of the invention for selecting an area from a general contents view.

0033<figref idref="DRAWINGS">FIG. 4B</figref> outlines the software flow diagram for another embodiment of the invention for selecting an area from a general contents view.

0034<figref idref="DRAWINGS">FIG. 5</figref> shows the process of selecting a block of text with another embodiment of the present invention.

0035<figref idref="DRAWINGS">FIG. 6</figref> outlines the software flow diagram for the process of selecting a block of text in the embodiment of the invention shown in <figref idref="DRAWINGS">FIG. 5</figref>.

0036<figref idref="DRAWINGS">FIG. 7A</figref> to <figref idref="DRAWINGS">FIG. 7C</figref> show the use of auto-displaced corner points to allow precise corner repositioning of the selected area.

0037<figref idref="DRAWINGS">FIG. 8</figref> outlines the software flow diagram for corner repositioning of the selected area using tilt and movement based view navigation set at fine mode in another embodiment of the present invention.

0038<figref idref="DRAWINGS">FIG. 9A</figref> and <figref idref="DRAWINGS">FIG. 9B</figref> show another embodiment of the present invention that performs contents aware boundary adjustment of the selected area.

0039<figref idref="DRAWINGS">FIG. 10</figref> shows the software flow diagram for the automatic boundary adjustment of the selected area in the embodiment of the invention shown in <figref idref="DRAWINGS">FIG. 9</figref>.

0040<figref idref="DRAWINGS">FIG. 11</figref> shows the software flow diagram for the extension of the corner repositioning technique of <figref idref="DRAWINGS">FIG. 8</figref> to a general marker repositioning on a mobile touch screen display.

DETAILED DESCRIPTION OF THE INVENTION

0041Hand held devices have typically small screens and often need to show information contents that are larger than the size of their displays. They employ a virtual display (also called “contents view”) which is stored in the device memory, while a part of the virtual display is shown in the physical display (“screen view”). In many systems, the virtual display may be dynamically downloaded to the device (e.g. from the internet or externally connected devices) so that at various times only a part of the virtual display is actually stored in the device.

0042<figref idref="DRAWINGS">FIG. 1</figref> shows an example of a virtual display <b>20</b> which contains several graphic items <b>22</b>, <b>24</b> and <b>26</b>. In a typical area selection operation, the user must define a selected area <b>30</b> by depicting two opposite corners <b>32</b> and <b>34</b> of a rectangular boundary. Two opposite corners define a unique rectangular boundary, provided the base of the rectangle is parallel to the bottom line of the display. Traditionally, such rectangular boundaries are used in most area selection operations in computer systems. Therefore, throughout this specification and the appended claims, it is assumed that any pair of selected area corners are used as opposite corners for a rectangular selected area boundary whereby the base of the boundary is parallel to the bottom of the display. In general, other geometrical shapes may be used as boundaries for unique area selection operations. Such non-rectangular shapes also require a set of defining points, so the teaching of this invention can be trivially extend for non-rectangular boundaries. In the example of <figref idref="DRAWINGS">FIG. 1</figref>, the selected area <b>30</b> captures only the graphic item <b>24</b> which includes the astronaut and the flag.

0043<figref idref="DRAWINGS">FIG. 2A-FIG</figref>. <b>2</b>D illustrate the process of marking the selected area <b>30</b> on the virtual display <b>20</b> in the example of <figref idref="DRAWINGS">FIG. 1</figref> on a hand held device <b>40</b> that incorporates one embodiment of the present invention. The hand held devices of the present invention are capable to respond to user's touch gestures as well as to perform tilt and movement based view navigation. Touch gestures (also called “touch commands” in this specification and the appended claims) are detected by the touch screen display <b>42</b> which is responsive to the touch of one finger (single-touch) or multiple fingers touch (multi-touch) on the screen. The touch commands can perform view navigation (e.g. display scrolling), as well as many other specific control commands. In the present invention, all touch commands are partitioned into two sets. The first set includes all the view navigation touch commands, and the second set includes all the other touch commands that do not affect view navigation. In this specification I refer to the first set of view navigation touch commands as “TOUCH NAV”. TOUCH NAV commands may include scrolling by flicks, swipes, touch and drag, and other commands. They also include all touch commands that activate links embedded in the screen view, since the activation of the embedded links can change the current view.

0044The present invention also incorporates tilt and movement based view navigation, like the system disclosed in my U.S. Pat. Nos. 6,466,198 and 6,933,923 which have been commercialized under the trade name RotoView. Tilt and movement based view navigation essentially translates the user's three-dimensional tilt and movements of the hand held device <b>40</b> into scrolling commands along two generally perpendicular axes placed on the surface of the display. Tilt and movement gestures can also be used to move a cursor on the screen. Optional button <b>44</b>, voice commands, joystick, keyboard, camera based visual gesture recognition system, and other user interface means may be incorporated on the hand held device <b>40</b>.

0045In <figref idref="DRAWINGS">FIG. 2A</figref>, the user can employ any view navigation method available on the device <b>40</b> (e.g. TOUCH NAV, tilt and movement based view navigation, or joystick/keyboard scrolling) to navigate the screen view <b>42</b> to arrive at the general area of the first corner point <b>32</b> of the desired selected area <b>30</b> (defined in <figref idref="DRAWINGS">FIG. 1</figref>). The user activates the area selection process by a variety of means that may include a specific touch (or multi-touch) gesture, a voice command, a keyboard command, a visual gesture that may be detected by camera or proximity sensors, or a movement gesture (e.g. device shake). The display may respond with some marker or other indicator to show that the system entered the area selection mode. During the area selection mode, all TOUCH NAV commands must be suspended, leaving only the tilt and movement based view navigation active. This eliminates the problem of misinterpreted touches that may be confused as TOUCH NAV commands instead of corner selection commands. The user then touches the first corner point <b>32</b> of the desired selected area with her finger <b>46</b> in order to select it. In other embodiments of the present invention, the user's selection command may be a touch gesture which also defines the first area corner <b>32</b>, as it will be described in <figref idref="DRAWINGS">FIG. 4A</figref>. The accuracy of the corner placement can be increased by employing the corner repositioning method that will be described below.

0046<figref idref="DRAWINGS">FIG. 2B</figref> shows how tilt movement based view navigation is exclusively used for changing the temporary selection boundary <b>52</b>. As the user tilts or moves the device <b>40</b> in three-dimensional space, the system translates these orientation changes or movements into scrolling commands along two generally perpendicular rotation axes. Borrowing from avionics terminology, we say that axis <b>60</b> is set along the roll axis of the device <b>40</b>. In this example we show one technique which merely ignores any rotation changes along the axis perpendicular to the plane of the screen view and use only pitch and roll data. Various other techniques to translate absolute tilt changes and movements in real three dimensional space onto the two dimensions of the screen view are known in the art, and they can be employed with the present invention. The device uses first rotation axis <b>60</b> (along the roll axis of the device <b>40</b>) to translate device tilt changes and lateral movements along arrow <b>64</b> into rightwards horizontal scrolling of the screen view <b>42</b> relative to the virtual display <b>20</b>. Similarly, the second rotation axis <b>62</b> is set along the pitch axis of the device <b>40</b> and is used to translate device tilt changes and lateral movements along arrow <b>66</b> into downwards vertical scrolling. Arrow <b>65</b> represents horizontal lateral movement that may be used to scroll the screen view to the right. Similarly, Arrow <b>67</b> represents vertical lateral movement that may be used to scroll the screen view down. While the device is manipulated by the user, the first corner of the temporary selection boundary <b>52</b> remains anchored to the first corner point <b>32</b> on the virtual display <b>20</b>. The second corner <b>54</b> of the temporary selection boundary <b>52</b> propagates at around the screen view center. The temporary second corner <b>54</b> can be rigidly fixed at the screen view center or may be dynamically “pulled” (with some small time delay) towards the center while the screen view navigates the virtual display. At the stage shown in <figref idref="DRAWINGS">FIG. 2B</figref>, only a small section of the desired selected area is now enclosed within the temporary selection boundary <b>52</b>.

0047At <figref idref="DRAWINGS">FIG. 2C</figref>, the temporary second corner <b>54</b> has been brought close to the desired second corner location <b>34</b>. Once the user sees the desired corner location <b>34</b> within the screen view, she touches the location <b>34</b> to complete the area selection process, as shown in <figref idref="DRAWINGS">FIG. 2D</figref>. Since all TOUCH NAV commands are suspended, any touch sensed by the touch screen display is safely interpreted as a corner selection command. Once the system receives the touch command at location <b>34</b>, the temporary second corner position <b>54</b> at the center of the display flips to location <b>34</b>. This creates the desired selected area within the rectangular boundary <b>30</b>, and the system exits the area selection mode. This in turn reactivates the TOUCH NAV commands, allowing the user to perform touch screen based view navigation (swipe, flicks, etc.). The selected area is now available to the calling application program (cut and paste, move, copy, zoom in, etc.). To increase user friendliness of the system, the final selected area <b>30</b> may be drawn differently on the screen (color wise, style wise) compared to the temporarily boundary <b>52</b>. The corner markers <b>32</b> and <b>34</b> may be removed from the final selected area at the end of the area selection process.

0048<figref idref="DRAWINGS">FIG. 3</figref> discloses an embodiment of a hand held device with a touch screen display incorporating the area selection methods of the present invention. The processor <b>100</b> provides the processing and control means required by the system, and comprises at least one microprocessor or micro-controller. The processor <b>100</b> uses the memory subsystem <b>102</b> for retaining the executable program, the data and the display information. A display interface module <b>104</b> controls the touch screen display <b>106</b> which provides the screen view <b>42</b> to the user. The display interface module <b>104</b> is controlled by the processor <b>100</b> and further interfaces with the memory subsystem <b>102</b> for accessing the virtual display and creating the screen view <b>42</b>. The display interface module may include local graphic memory resources. The display interface module <b>104</b> also provides the processor <b>100</b> with touch screen gestures made by the human operator (“user”) of the hand held device. Such touch screen gestures may be made by one or more fingers.

0049A tilt and movement sensor <b>108</b> interfaces with the processor to provide ballistic data relating to the movements and rotations (tilt changes) made by the user of the device. The ballistic data can be used by the micro-controller to navigate the screen view <b>42</b> over the virtual display <b>20</b>. The ballistic data can also be used for cursor movement control. Typically, the tilt and movement sensor <b>108</b> comprises a set of accelerometers and/or gyroscopes with signal conversion for providing tilt and movement information to the processor <b>100</b>. A 6-degree-of-freedom sensor, which comprises a combination of a 3-axis accelerometer and 3-axis gyroscope can be used to distinguish between rotational and movement data and provide more precise view navigation. It should be pointed out that tilt and movement based navigation can be implemented with only accelerometers or with only gyroscopes. Other tilt and movement sensors may be mechanical, magnetic, or may be based on a device mounted camera associated with vision analysis to determine movements and rotations.

0050The processor <b>100</b> can optionally access additional user interface resources such as a voice command interface <b>110</b> and a keyboard/joystick interface <b>114</b>. Another interface resource may be a visual gesture interface <b>116</b>, which detects a remote predefine visual gesture (comprising predefined movements of the hand, the fingers or the entire body) using a camera or other capture devices. It should be apparent to a person familiar in the art that many variants of the block elements comprising the block diagram of <figref idref="DRAWINGS">FIG. 3</figref> can be made, and that various components may be integrated together into a single VLSI chip.

0051<figref idref="DRAWINGS">FIG. 4A</figref> illustrates the software flow diagram of one embodiment of the present invention that performs the area selection process shown in <figref idref="DRAWINGS">FIG. 2</figref>. The process connects to the regular operating system flow at the beginning step <b>200</b> by a parent application that needs area selection. It first resets the selection mode to indicate normal operation mode at step <b>210</b>. At steps <b>216</b> and <b>220</b>, the user navigates the virtual display to select the first area corner <b>32</b>. As shown in step <b>216</b>, the user can use any view navigation method available at the device during normal operation mode, including touch screen view navigation (TOUCH NAV) and tilt and movement based view navigation (TILT/MOV NAV). Step <b>216</b> also represents all other non related device operations, including all sub-processes of the parent application. At step <b>220</b> the system checks if a predefined touch gesture to enter the area selection operation has been detected. For example, such a predefined touch gesture may be an ‘x’ shape finger movement on the display where the ‘x’ center is at the desired location for the first corner of the selected area. If step <b>220</b> does not detect a selection gesture, the regular operation of the device continues along step <b>216</b>.

0052If step <b>220</b> detects a selection touch gesture, the area selection mode is activated at step <b>224</b>, which may optionally activate a selection indicator or marker on the display, alerting the user that the device is in area selection mode. At step <b>230</b> the system converts the gesture defined touch location (e.g., the center point in an ‘x’ shape touch gesture) as the first corner <b>32</b> of the selected area at the exact touch location on the portion of said virtual display currently shown on said touch screen display. Once the first area corner <b>32</b> is selected, step <b>232</b> suspends the set of the TOUCH NAV commands, allowing the tilt and movement based view navigation to work during the following selection of the second corner of the selected area. The suspension of the TOUCH NAV commands is crucial to insure that any kind of touch detection in the following steps will be interpreted solely in the correct context of the area selection process. Step <b>234</b> offers an optional corner repositioning that can achieve more precise positioning of the area corner. The optional corner repositioning is described in greater detail below. Optional joystick or keyboard based view navigation may be also allowed to work along with the tilt and movement based view navigation during the area selection process.

0053The sub-process <b>238</b> is used to select the second corner <b>34</b> of the selected area. The system processes the tilt and movement based view navigation at step <b>240</b>. At step <b>244</b>, a temporary selected area boundary <b>52</b> is drawn from the first corner <b>32</b> onto a temporary corner <b>54</b> at the general center of the screen view <b>42</b> as it scrolls the virtual display <b>20</b> in response to the tilt and movement based view navigation. At step <b>250</b> the system checks for any touch command. If a touch command is not detected, the process continues along steps <b>240</b> and <b>244</b>. If a touch command is detected, the touch location is used as the second corner <b>34</b> of the selected area at step <b>254</b>. Step <b>256</b> offers the optional corner repositioning sub-process that achieves more precise positioning of the final selected area's corner. The final selected area <b>30</b> is drawn on the virtual display <b>20</b>. At step <b>258</b> the selection mode is deactivated and the set of TOUCH NAV commands is reactivated. Finally, the system provides the selected area information to the calling application as the process ends at step <b>260</b>.

0054<figref idref="DRAWINGS">FIG. 4B</figref> illustrates the software flow diagram of another embodiment of the present invention to perform area selection. The process connects to regular operating system flow at the beginning step <b>270</b> by a parent application that uses the area selection operation. It first resets the area selection mode at step <b>272</b> to indicate normal operation mode. The user can employ any view navigation method available at the device during normal operation mode. At step <b>276</b>, the system continuously monitors for an area selection command which may be initiated by several sources. Such an area selection command can be initiated by a touch or movement gesture, by a voice command, by a keyboard of switch button press, by a predefined visual gesture, or by any other common user interface means. It can also be initiated by the parent application itself in response to its program flow. Step <b>276</b> also represents all other device operations, including all sub-processes of the parent application that may need the area selection operation. At step <b>280</b> the system determines if an area selection command has been detected. If a selection command is not detected, the regular operation of the device continues along step <b>276</b>.

0055If step <b>280</b> detects a selection command, the selection mode is activated at step <b>282</b> and the set of TOUCH NAV commands is suspended as explained earlier. The system now executes steps <b>286</b> and <b>290</b> to determine the location of the first corner <b>32</b> of the selected area. At step <b>286</b>, the system scrolls the display by tilt and movement based view navigation to reach the desired virtual display area to place the first corner point. Step <b>286</b> may optionally activate a blinking marker or an enlarged crosshair marker on the display's center, alerting the user that the device has entered into the selection mode and a selection of the first corner <b>32</b> is needed. At step <b>290</b> the system checks if a touch was detected. If a touch is not detected, the user continues to navigate for the location of the first corner <b>32</b> at Step <b>286</b>.

0056If step <b>290</b> detects a touch, the system uses the touch location to place the first corner <b>32</b> at step <b>292</b>. Step <b>294</b> offers the optional corner repositioning sub-process that achieves more precise positioning of the selected corner <b>32</b>. The sub-process <b>238</b> of <figref idref="DRAWINGS">FIG. 4A</figref> is now performed at step <b>296</b> in order to complete the area selection and provide the calling application with the selected area at the end step <b>298</b>.

0057The area selection techniques described above are based on a rectangular boundary that is defined by two opposite corners with a base parallel to the bottom of the display. It should be clear that the teaching of the present invention can be easily extended for area selection that uses other geometrical shapes. In the case of polygon-like shapes that use more than two corners, the extension of the present invention requires orderly repetition of step <b>238</b> and <b>296</b> in <figref idref="DRAWINGS">FIGS. 4A and 4B</figref> to set all the corners.

0058It appears that for a small area selection which is fully visible within the screen view, one may perform the processes in <figref idref="DRAWINGS">FIGS. 4A and 4B</figref> with minimal or even no use of the tilt and movement based view navigation (steps <b>240</b> and <b>286</b>). It should be noted that even in this case, the suspension of the TOUCH NAV commands during the area selection is a key feature of the present invention to avoid the unintended activation of touch command that will inadvertently change the view. Also, it is quite common that users perform a zoom in operation prior to the selection of small areas in order to increase the selection accuracy. This zoom in creates a virtual display that is larger than the screen view and requires the tilt and movement based view navigation.

0059Common applications like word processors require area selection from a virtual display that may contain only text. Some of these applications may have a virtual display <b>20</b> with text lines widths which are larger than the width of the screen view <b>42</b>. In such cases the selection of a text block can be made similar to the embodiments of the present invention shown in <figref idref="DRAWINGS">FIG. 2</figref> and the associated flow charts in <figref idref="DRAWINGS">FIG. 4</figref>. However, most text applications limit the width of the virtual display to fit the text lines within the screen view so that the user does not need to scroll left and right.

0060<figref idref="DRAWINGS">FIG. 5</figref> illustrates another embodiment of the present invention for selecting a block of text from a virtual display <b>20</b> that includes lines of text that are fully enclosed within the width of the screen view <b>42</b>. Although text is spread on a two dimensional area, it is essentially arranged linearly along a single list of characters and spaces which is divided into multiple text lines. As a result, text block selection is defined by two endpoints (e.g. block-start point <b>70</b> and block-end point <b>72</b>) along the list of the characters of the text.

0061The user initiated the text block selection process by a touch gesture at point <b>70</b>, when the desired section of the text area was shown in the screen view <b>42</b>. The touch gesture may be shaped as virtual letter ‘x’ and the first endpoint <b>70</b> may be selected as the nearest inter words space to the gesture's ‘x’ center location. The system enters text selection mode where the set of TOUCH NAV commands is suspended and the user can use the tilt and movement based view navigation to scroll the display. As the user scrolls the display downwards, a temporary endpoint <b>72</b> is placed at or near the center of the screen view <b>42</b>, and the text block <b>74</b> from the starting endpoint <b>70</b> to the temporary endpoint <b>72</b> is highlighted. Once the desired second endpoint of the selection block <b>78</b> appears anywhere on the screen view, the user touches this endpoint's location, and completes the text block selection process.

0062Since the virtual display <b>20</b> is adjusted to fit the width of the screen view <b>42</b>, there is no need for horizontal navigation of the temporary endpoint <b>72</b>. Therefore, it is possible to map the two axes view navigation obtained from the tilt and movement sensor into a single axis corresponding along the character list of the text. For a left to right language like English, both roll rotation <b>64</b> to the right and pitch rotation down <b>66</b> (or movements to the right <b>65</b> and down <b>67</b>) are translated into a downwards text scrolling. Roll rotation to the left and pitch rotation up are similarly translated into an upwards text scrolling. For a right to left language like Hebrew, both roll rotation <b>64</b> to the left and pitch rotation <b>66</b> down are translated into downwards text scrolling. Roll rotation to the right and pitch rotation up are similarly translated into an upwards text scrolling. The tilt and movement based view navigation of the present invention is particularly useful when the length of the text block is longer than the height of the screen view <b>42</b>.

0063<figref idref="DRAWINGS">FIG. 6</figref> illustrates the software flow diagram used to compute the text block selection of the system shown in <figref idref="DRAWINGS">FIG. 5</figref>. The process connects to regular operating system flow at the beginning step <b>300</b> by a parent application that uses area selection operation. It first resets the text selection mode to indicate normal operation mode at step <b>310</b>. During normal operation mode <b>316</b>, the user can use any view navigation method available at the device, including TOUCH NAV and tilt and movement based view navigation. Step <b>316</b> also represents all other device operations, including all sub-processes of the parent application that may need area selection. At step <b>320</b> the system checks if a selection touch gesture has been detected. For example, such a touch gesture may be an ‘x’ shape finger movement on the display where the ‘x’ center is at the desired first endpoint of the selected block. If no touch gesture is detected, the regular operation of the device continues along step <b>316</b>.

0064If step <b>320</b> detects a selection gesture, the text selection mode is activated at step <b>324</b>, which may optionally activate a selection indicator or marker on the display, alerting the user that the device is in a text selection mode. The set of TOUCH NAV commands is suspended at step <b>324</b> as explained earlier. At step <b>328</b> the system uses the finger touch location (e.g., the center point in an ‘x’ shape touch gesture) as the first endpoint <b>70</b> of the text block selection. The system may set the block endpoint at the inter words space nearest to the gesture location.

0065The system now executes steps <b>340</b>, <b>344</b>, <b>354</b>, <b>358</b>, <b>362</b> and <b>366</b> to allow the user to select the second endpoint for the selected block. Steps <b>340</b> and <b>344</b> detect the user tilt and movement based view navigation commands and steps <b>354</b> and <b>358</b> respond to these commands by scrolling the text up or down. Assuming the text language is English, if at step <b>340</b> the system detects a tilt and movement up or to the left, it scrolls the text list of characters up at step <b>354</b>. If at step <b>344</b> the system detects a tilt and movement down or to the right, it scrolls the text list of characters down at step <b>358</b>. After each scrolling action, step <b>362</b> sets the temporary endpoint <b>72</b> generally towards the screen view center and the block of text <b>74</b> between endpoints <b>70</b> and <b>72</b> is highlighted.

0066At step <b>366</b> the system checks for a touch command. If a touch command is not detected, the scrolling process described in the previous paragraph is repeated. Once a touch is detected, the finger touch location is used as the second endpoint <b>78</b> of the selected block at step <b>370</b>. The system may set the endpoint <b>78</b> at the inter words space nearest to the finger touch location. The final text block selection is highlighted on the virtual display. At step <b>374</b> the text selection mode is deactivated, and the set of TOUCH NAV commands is reactivated. The system provides the selected text block information to the calling process as the process ends at step <b>380</b>.

0067Referring back to <figref idref="DRAWINGS">FIGS. 2A and 2C</figref>, one can appreciate that the user's finger <b>46</b> has a substantial size relative to the size of the screen view <b>42</b>. Therefore, setting the selected area's corner points <b>32</b> and <b>34</b> by finger touches is not very accurate. <figref idref="DRAWINGS">FIG. 7A</figref> approximates this inherent inaccuracy with an uncertainty area <b>80</b> occurring when the user aims to touch a desired point <b>82</b> on the screen view <b>42</b> of the hand held device <b>40</b>. The uncertainty area <b>80</b> of the finger touch is significantly larger than the uncertainty associated with stylus pointing due to the sharp tip of the stylus. The following embodiments of the present invention offer several corner repositioning techniques that achieve more precise placement of the selected area's corners. The corner repositioning operations are automatically initiated only when the user touches the screen for the actual selection of either the first or second corner points, at steps <b>234</b> and <b>256</b> of <figref idref="DRAWINGS">FIG. 4A</figref> or at step <b>294</b> of <figref idref="DRAWINGS">FIG. 4B</figref>. The corner repositioning operation is not activated when the user performs other touch commands that are not associated with corner placement.

0068<figref idref="DRAWINGS">FIG. 7B</figref> illustrates an auto-displacement that positions the actual corner point <b>84</b> above the actual touch point <b>82</b>, at a distance sufficient to avoid visual obstruction by the finger <b>46</b>. When the user first touches the screen at the finger contact point <b>82</b> in order to select a corner point, the system enters a corner repositioning mode which remains in effect as long as the user continues to touch the screen. The actual corner point <b>84</b> is preferably marked by an increased crosshair cursor (which may be optionally blinking) during the corner repositioning mode to alert the user that the repositioning mode is on, and to enable better repositioning. The movement of the touching finger <b>46</b> is translated to the corner point <b>84</b>, so that any vertical <b>86</b> and horizontal <b>88</b> movements of the finger cause corresponding vertical <b>87</b> and horizontal <b>89</b> corner point movements. The direction of the finger movement is translated to a same direction of the displaced corner movement. However, it is possible to achieve higher repositioning accuracy if the length of the movement of the finger is translated into a proportionally smaller length of movement of the displaced area corner. This causes relatively large finger movements to make fine movements of the corner, hence the increased placement accuracy.

0069When the user reaches the exact corner point position, she lifts her finger <b>46</b> from the screen <b>42</b>, as shown in <figref idref="DRAWINGS">FIG. 7C</figref>. This terminates the corner repositioning mode and the cursor <b>84</b> is converted into the final fixed corner point <b>90</b>, at the exact desired place. If the corner cursor was replaced by a crosshair cursor during the corner repositioning, it is returned to the normal size and shape.

0070In another embodiment of the present invention, the user can perform corner repositioning using tilt and movement based cursor control set at a fine navigation mode, as illustrated in <figref idref="DRAWINGS">FIG. 8</figref>. The corner repositioning operations are optionally made at steps <b>234</b> and <b>256</b> of <figref idref="DRAWINGS">FIG. 4A</figref> or at step <b>294</b> of <figref idref="DRAWINGS">FIG. 4B</figref> following an initial, relatively inaccurate corner placement by a finger touch. Referring back to <figref idref="DRAWINGS">FIG. 8</figref>, the corner repositioning process begins at step <b>400</b> with the currently selected corner. At step <b>402</b> the corner repositioning mode is activated, and at step <b>404</b> the corner's cursor is replaced with an enlarged crosshair marker at its initial, inaccurate position. Optionally, the enlarged crosshair marker may be set to blink during the corner repositioning mode. This style change in the corner marker provides clear feedback to the user indicating that corner repositioning is on. The enlarged crosshair marker further facilitates more accurate repositioning.

0071A corner repositioning elapsed timer may optionally be started at step <b>406</b>. Step <b>408</b> activates the tilt and movement based cursor control to move the crosshair marker. The tilt and movement based cursor control is set to fine response mode which translates relatively large tilt and movements of the hand into small movements of the crosshair cursor. The system performs the corner repositioning via the loop of steps <b>410</b>, <b>412</b> and <b>414</b>. At step <b>410</b>, the system continuously uses tilt and movement based cursor control set at a fine navigation mode to move the crosshair. Fine navigation mode causes relatively large movements and tilt changes to make fine movements of the crosshair, hence the increased placement accuracy. Corner repositioning mode can be terminated by a touch command, detected at step <b>412</b>, or at the expiration of the optional timer at step <b>414</b>.

0072Once the corner point is placed at the exact desired location, the user touches the screen at the vicinity of the corner point to end the corner repositioning mode by step <b>412</b>. It should be noted that the exact location of the touch that ends the corner repositioning mode does not change the crosshair marker position. The position of the crosshair marker is fixed and replaced by the final corner at step <b>416</b>, and the corner repositioning mode is reset at step <b>418</b>. This completes the repositioning process at step <b>420</b>.

0073Another embodiment of the present invention provides automatic boundary adjustment for the area selection to reduce the effect of unwanted truncation of the contents within the selected area. This contents aware area boundary adjustment helps to avoid the need to repeat the area selection process. This embodiment of the present invention is applicable to any computerized system with any type of display where area selection operation is performed.

0074<figref idref="DRAWINGS">FIG. 9A</figref> illustrates a crowded virtual display <b>20</b> that includes three graphical objects <b>24</b>, <b>25</b>, and <b>26</b> at relatively close proximity, assuming that the user wishes to select an area that will contain the astronaut object <b>24</b>. The selected area <b>30</b> performed by the user in <figref idref="DRAWINGS">FIG. 9A</figref> seems to miss some parts of object <b>24</b>, including portion of the left hand <b>92</b>, portion of both feet <b>94</b>, and part of the top <b>95</b>. Once the area selection <b>30</b> is completed with the selection of the second corner point <b>34</b>, the system automatically determines the truncated portions <b>92</b>, <b>94</b> and <b>95</b> of the astronaut object <b>24</b>. It also detects that a top portion <b>98</b> of object <b>26</b> and a small corner <b>96</b> of the flag <b>25</b> have also been truncated.

0075<figref idref="DRAWINGS">FIG. 9B</figref> shows how the program automatically attempts to adjust the boundary of the selected area <b>30</b> with a modified selection area <b>31</b> that will properly enclose the main object <b>24</b>. Using the shapes list analysis described below, the program identify all the truncated shapes. Following a connectivity analysis, the program determines that the truncated shapes <b>96</b> and <b>98</b> are not connected to the main object at the center of the original selected area <b>30</b>. The program then continuously increases the height and/or the width of the area selection <b>30</b> until the resulting modified boundary <b>31</b> encloses all the truncated shapes that deemed connected to the main object. It should be noted that certain truncated shapes may be too large and the system will not be able to enclose them within the modified boundary. If a contents aware correction is not possible, the program aborts with the original selected area <b>30</b>. If a correction is possible, the system displays both the original selected area <b>30</b> and the modified boundary <b>31</b> as illustrated in <figref idref="DRAWINGS">FIG. 9B</figref>. The system may need to zoom out the virtual display <b>20</b> if the boundaries <b>30</b> and <b>31</b> go beyond the screen at the current zoom level. The system then prompts the user to accept or reject the modified boundary <b>31</b>.

0076<figref idref="DRAWINGS">FIG. 10</figref> illustrates the software flow diagram of the program used to perform the automatic contents aware boundary adjustment like the process shown in <figref idref="DRAWINGS">FIG. 9</figref>. The program starts at step <b>440</b> following the user completion of an area selection operation as the program is presented with an input area boundary from the selected area. This input area boundary is stored in step <b>442</b> as the initial value for the modified boundary, and a recognizable shapes list used for the subsequent decomposition and analysis is emptied. Steps <b>444</b>, <b>445</b>, <b>446</b>, <b>450</b>, <b>454</b>, <b>458</b>, and <b>460</b> perform the contents awareness analysis of all objects found within the current boundary.

0077In step <b>444</b>, the contents of the input area boundary and its immediate surrounding area are decomposed into recognizable shapes and put into the shapes list. These recognizable shapes include primitive geometric shapes as well as more complex shapes. Complex implementations may utilize advanced expert systems techniques known to the art which provide learning capabilities and dynamically expanding the database of recognizable shapes. Such dynamic update methods may add unrecognized shapes remaining after the decomposition process, possibly following a connectivity analysis to determine that the unrecognized shape/s create a unique aggregation of a new shape.

0078If the decomposition process at step <b>444</b> fails, the system is adapted to abort the automatic correction program at step <b>445</b>. A failure of the decomposition process occurs if there are no recognizable shapes detected within the input boundary or if there are too many recognizable shapes above a certain overflow limit. A copy of the complete shapes list is retained at step <b>446</b> for subsequent connectivity analysis. Every shape in the recognizable shapes list is analyzed in step <b>450</b> to determine if it is truncated by the input area boundary. Each shape that is not truncated is removed from the shapes list. Step <b>454</b> checks if the shapes list is empty. If the list is empty, there is no need to adjust the boundary since there are no recognized truncated shapes, and the program ends at step <b>480</b>.

0079If step <b>454</b> finds that the shapes list is not empty, the program runs a connectivity analysis of each truncated shape in the recognizable shapes list at step <b>458</b>. Here the program uses the copy of the full recognizable shapes list made at step <b>446</b> to determine if the truncated shape is connected to any other shapes within the input area boundary. Truncated shapes that are not connected (like shapes <b>96</b> and <b>98</b> in <figref idref="DRAWINGS">FIG. 9A</figref>) are removed from the shapes list. If the recognizable shapes list is empty at step <b>460</b>, the program terminates to step <b>480</b>.

0080If the recognizable shapes list is not empty, the program proceeds to adjust the modified boundary along steps <b>464</b>, and <b>465</b>. At step <b>464</b>, the program removes a connected truncated shape from the shapes list and attempts to increase the modified boundary until it encloses the truncated shape. If the currently increased boundary does not reach the end of the virtual display or does not exceed a preset limit, the currently increased boundary replaces the last modified boundary. Otherwise, the last modified boundary is restored and the process continues, recognizing that the just removed shaped will remain truncated. This may result in a partial correction which still achieves the objective to reduce the number of truncated shapes. Step <b>465</b> causes step <b>464</b> to repeat until the recognizable shapes list becomes empty, so that step <b>464</b> may continuously increase the modified boundary to enclose as many truncated and connected shapes as possible.

0081When the recognizable shapes list is finally empty, step <b>466</b> compares the modified boundary and the input area boundary. If the modified boundary remains the same as the input area boundary, the process aborts. If the modified boundary has changed, step <b>470</b> displays the larger modified boundary <b>31</b> together with the originally selected area <b>30</b> in <figref idref="DRAWINGS">FIG. 9B</figref>, and it prompts the user to accept the adjustment of the selected area. If the user rejects the modified boundary at step <b>474</b>, the program ends at step <b>480</b> without adjustment of the selected area. If the user approves the modified boundary, the program replaces the area selection with the modified boundary at step <b>476</b> and ends at step <b>480</b>.

0082The corner repositioning method described above can be extended for use with any marker placed inaccurately on a hand held device with a touch screen display due to the inherent thickness of the finger tip. <figref idref="DRAWINGS">FIG. 11</figref> illustrates the flow diagram <b>500</b> of the program used to perform such marker repositioning. The program starts at step <b>504</b> when a user places a marker on the touch screen display. The process allows a certain period of time following the marker placement for the user to issue a marker repositioning command. The repositioning command can be selected from all available user interface commands, including a movement gesture, a predefined touch gesture, a voice command, a keyboard command, or a predefined visual gesture. The period of time to issue a repositioning command can be defined by activating a reposition command timer at step <b>506</b>. The system monitors for the user marker repositioning command in step <b>508</b>. If the user command is detected at step <b>510</b>, the system proceeds with the corner repositioning process of <figref idref="DRAWINGS">FIG. 8</figref> (where all references to “corner” are replaced by the placed marker at step <b>504</b>). The marker is changed to a large crosshair to assist the positioning and alert the user that the repositioning process is on.

0083If the repositioning command timer expires, the program quits without performing the repositioning. Alternatively, the period of time during which the system waits for the repositioning command may be terminated by a user touch command, detected at step <b>514</b>. If this alternative approach is taken, the touch command to terminate the period must be different than any touch gesture that may be used for the reposition command. If the marker repositioning command is not a touch gesture, then any touch command detected at step <b>514</b> will quit the program without performing the marker repositioning. A combination of both timer expiration and touch termination command can work well with the present invention.

0084The description above contains many specifications, and for purpose of illustration, has been described with references to specific embodiments. However, the foregoing embodiments are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Therefore, these illustrative discussions should not be construed as limiting the scope of the invention but as merely providing embodiments that better explain the principle of the invention and its practical applications, so that a person skilled in the art can best utilize the invention with various modifications as required for a particular use. It is therefore intended that the following appended claims be interpreted as including all such modifications, alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.