Searching for commands and other elements of a user interface

CROSS-REFERENCES TO RELATED APPLICATIONS

0001This application is a continuation of pending U.S. patent application Ser. No. 11/496,006, filed Jul. 28, 2006 which is a continuation-in-part of pending U.S. patent application Ser. No. 10/985,630, filed Nov. 10, 2004 each of which is incorporated by reference in its entirety. The present application is related to pending U.S. patent application Ser. No. 10/431,424 filed May 6, 2003, the disclosure of which is incorporated herein by reference.

FIELD OF THE INVENTION

0002The present invention relates generally to search functionality for a user interface, and more particularly to techniques for searching for commands within menus and other interface elements of a user interface.

BACKGROUND OF THE INVENTION

0003As software applications and operating systems become increasingly complex, the number of commands, options, and dialog boxes grows. As a result, users (particularly novice users) have a difficult time finding a command or element they are looking for. A user may know the general nature of the command he or she is looking for, but may not know which menu or dialog box contains that command. Searching among various menus (including hierarchical menus) and/or consulting printed or electronic documentation can be frustrating and time-consuming. Customization of user interfaces can make the task of finding a desired command more difficult, as a particular command may be found in different locations in different variations of the software product.

0004In particular, when a new version of a software application or operating system is released, it is common for commands to be moved from one place to another; users who are used to the previous version of the software application or operating system may be unfamiliar with the new locations of commands.

0005Once a user has become familiar with the organizational scheme for the software application, he or she can more easily find a particular command when it is needed. Accordingly, it would be beneficial to have a mechanism for training the user as to the locations of various commands, without forcing the user to consult manuals or other documentation. It would further be beneficial to provide a mechanism that makes it unnecessary for users to memorize the location of commands.

0006In this context, what is needed is a system and method for assisting a user in locating and accessing commands such as those within a menu structure, even when the user is unfamiliar with the organizational scheme for the software application. What is further needed is a system and method for training the user so that, over time, he or she becomes more familiar with the locations of various elements of a user interface, including for example commands within a menu structure. What is further needed is a system and method for providing a convenient way for both novice and advanced users to navigate a user interface, and for providing a mechanism that avoids the need for users to memorize the location of commands in a user interface. What is further needed is a system and method for integrating search functionality for user interface elements with other search functionality of a software application and/or operating system.

SUMMARY OF THE INVENTION

0007The present invention provides a search capability to allow a user to easily locate commands within menus, dialog boxes, or windows of a software application or operating system. In one embodiment, a user enters a search term (or a portion thereof) in a search field. The user is presented with a results menu containing a list of matches. The user selects the desired item from the results menu, and the corresponding command or other user interface element is displayed in its context within a menu, dialog box, window, or the like.

0008In one aspect, as the user enters the search term (or as he or she starts typing the search term, so that only a portion thereof has been entered), a list of results is presented. The user can navigate among the results by clicking on directional keys on a keyboard. The user can also click or hover over various results. In one aspect, as the user selects various results, corresponding user interface elements are brought into view in context, and (optionally) highlighted, so that the user can better understand the navigational structure of the user interface and the arrangement of commands therein. The user can initiate a command by activating (for example by clicking on) a displayed result corresponding to the command.

0009In one aspect, even before the user selects among displayed results, certain menu headings within the menu structure are highlighted automatically. Specifically, those headings that represent menus containing items that match the search term (such as menu commands containing the term, or menu commands that activate dialog boxes containing the term), are highlighted. These results may be updated dynamically as the user continues to enter the search criteria. In one aspect, other user interface elements may be highlighted as the user is entering the search term. For example, icons or buttons on toolbars may be highlighted when they correspond to commands or dialog boxes that contain text matching the characters that the user has so far entered.

0010In one aspect, different types of highlighting are used depending on the degree of certainty as to which menu or other user interface element is likely to contain the correct match. When there is relative uncertainty as to which menu, dialog box, command, or other element is being sought, menu titles or other user interface elements corresponding to commands containing the search term are highlighted using a first type of highlighting. When sufficient search term input has been provided that yields greater certainty as to which item is being sought, a second, more prominent type of highlighting is used.

0011By highlighting icons and items in this manner, the present invention provides a training mechanism that allows a user to learn, over time, where to find various menu commands and/or other user interface elements. In one embodiment, this training mechanism is implemented in a manner that is relatively innocuous, in that it does not interfere with the user's normal operation of the system. The present invention provides a mechanism that avoids the need for users to memorize the location of commands in a user interface

0012In one aspect, the results menu is updated automatically as the user types in the search field. Thus, the user need not hit Enter, or any other key, to activate the search functionality of the present invention.

0013The functionality of the present invention is not limited to menu commands. As described in more detail below, the present invention can be implemented in any context where it is desirable to dynamically locate and/or highlight user interface items in response to a search term, while the search term is being entered.

0014In one aspect, the present invention searches for alternative terms as well as literal matches. Common misspellings, descriptive text, and/or alternative terminologies are associated with controls, so that if the user enters a search term that is misspelled or that uses alternative terminology, a match is still indicated.

0015In one aspect, the functionality of the present invention is integrating with other search functionality. Thus, the user can enter a query term and see results that include user interface elements, documents, messages, and/or other results that match the query term.

BRIEF DESCRIPTION OF THE DRAWINGS

0016The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention.

0017<figref idref="DRAWINGS">FIG. 1</figref> is a screen shot depicting an example of a menu bar including an icon for initiating a search function according to one embodiment of the present invention.

0018<figref idref="DRAWINGS">FIG. 2</figref> is a screen shot depicting an example of a query field for entering a search query according to one embodiment of the present invention.

0019<figref idref="DRAWINGS">FIG. 3</figref> is a screen shot depicting an example of a menu including a results list for a search query according to one embodiment of the present invention.

0020<figref idref="DRAWINGS">FIGS. 4A</figref>, <b>4</b>B, and <b>4</b>C are screen shots depicting a technique for highlighting a menu command corresponding to a selected result in the results list according to one embodiment of the present invention.

0021<figref idref="DRAWINGS">FIGS. 5A and 5B</figref> are screen shots depicting a technique for highlighting an on-screen button corresponding to a selected result in the results list according to one embodiment of the present invention.

0022<figref idref="DRAWINGS">FIG. 5C</figref> is an additional screen shots depicting a technique for highlighting a menu command corresponding to a selected result in the results list according to one embodiment of the present invention.

0023<figref idref="DRAWINGS">FIG. 6</figref> is a block diagram depicting a functional architecture for implementing the invention according to one embodiment.

0024<figref idref="DRAWINGS">FIG. 7</figref> is a screen shot depicting an example of a search box for initiating a search function according to one embodiment of the present invention.

0025<figref idref="DRAWINGS">FIG. 8</figref> is a screen shot depicting an example of a search box including a results list for a search query according to one embodiment of the present invention.

0026<figref idref="DRAWINGS">FIG. 9</figref> is a screen shot depicting a technique for highlighting a menu command corresponding to a selected result in the results list according to one embodiment of the present invention.

0027<figref idref="DRAWINGS">FIG. 10</figref> is a block diagram depicting a software architecture for implementing the invention according to one embodiment.

0028<figref idref="DRAWINGS">FIG. 11</figref> is a flow diagram depicting a method for practicing the present invention according to one embodiment.

0029<figref idref="DRAWINGS">FIG. 12</figref> is a screen shot depicting an alternative form of highlight for a menu command according to one embodiment.

0030<figref idref="DRAWINGS">FIG. 13</figref> is a screen shot showing an install dialog box according to one embodiment.

0031<figref idref="DRAWINGS">FIG. 14</figref> is a screen shot showing an example of results menu presenting search results that include user interface elements as well as other locations where query text was found.

0032<figref idref="DRAWINGS">FIG. 15</figref> is a block diagram depicting an architecture for practicing the present invention according to one embodiment.

0033<figref idref="DRAWINGS">FIG. 16A</figref> is a screen shot depicting an example of results menu presenting search results in response to user entry of a search term according to one embodiment.

0034<figref idref="DRAWINGS">FIGS. 16B to 16E</figref> are screen shots depicting examples of user interface elements being displayed in response to user highlighting of an item in results menu.

0035<figref idref="DRAWINGS">FIG. 17A</figref> depicts an overview of classes and relationships for implementing the present invention according to one embodiment.

0036<figref idref="DRAWINGS">FIG. 17B</figref> depicts an event trace depicting the flow of control that takes place when a user enters text in a field according to one embodiment.

0037<figref idref="DRAWINGS">FIG. 17C</figref> depicts a class hierarchy <b>1730</b> for GRLs, according to one embodiment.

0038<figref idref="DRAWINGS">FIG. 17D</figref> depicts a SCTInvocation class and a SCTGRLResolver class.

0039<figref idref="DRAWINGS">FIG. 17E</figref> depicts an example of a process of resolving a window to its content view based on a windowDidBecomeVisible notification.

0040<figref idref="DRAWINGS">FIG. 17F</figref> depicts an example where a “Show Ruler” checkbox is displayed in response to user selection of a “Show Ruler” result.

0041One skilled in the art will recognize that these Figures are merely examples of the operation of the invention according to one embodiment, and that other user interface arrangements and modes of operation can be used without departing from the essential characteristics of the invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

0042The present invention is now described more fully with reference to the accompanying Figures, in which several embodiments of the invention are shown. The present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather these embodiments are provided so that this disclosure will be complete and will fully convey the invention to those skilled in the art.

0043For illustrative purposes, the invention is described in connection with a mechanism for searching for menu commands within a user interface of a software application or operating system. Various specific details are set forth herein and in the Figures, to aid in understanding the present invention. However, such specific details are intended to be illustrative, and are not intended to restrict in any way the scope of the present invention as claimed herein. In particular, one skilled in the art will recognize that the invention can be used to assist the user in locating other types of user interface elements, including for example dialog boxes and elements thereof, toolbar icons, and the like. In other words, in the following description, menu commands are merely an example of items to be located. In addition, the particular screen layouts, appearance, and terminology as depicted and described herein, are intended to be illustrative and exemplary, and in no way limit the scope of the invention as claimed.

0044Referring now to <figref idref="DRAWINGS">FIG. 15</figref>, there is shown a block diagram depicting an architecture for practicing the present invention according to one embodiment. In one embodiment, the present invention is implemented in a conventional personal computer system <b>1502</b>, such as an iMac, Power Mac, or Power-Book, (available from Apple Computer, Inc. of Cupertino, Calif.), running an operating system such as MacOS X (also available from Apple Computer, Inc.). Computer <b>1502</b> includes a processor <b>1504</b>, memory <b>1505</b>, local storage <b>1506</b>, one or more input devices <b>1514</b> (such as keyboard, mouse, and the like), and one or more output devices <b>1515</b> (such as display screen). In one embodiment, a graphics module <b>1513</b> is included for generating and rendering the graphic presentations for displaying user interface screens and elements according to the techniques described herein. User <b>1501</b> provides input via input device <b>1514</b> and receives output via output device <b>1515</b>.

0045It will be recognized that the invention may be implemented on other devices as well, such as handhelds, personal digital assistants (PDAs), mobile telephones, consumer electronics devices, embedded devices, and the like.

0046The invention may be embodied in software that runs on the personal computer. The invention may be included as add-on software, or it may form part of the operating system itself, or it may be a feature of an application that is bundled with the computer system or sold separately. In one embodiment, the user can choose whether to install the feature. Referring now to FIG. <b>13</b>, there is shown an example of a dialog box for installing the feature that implements the present invention. The user can click OK or Cancel to select whether or not to install the feature.

0047The various features of the invention as described herein include output presented on a display screen that is connected to the personal computer. In addition, the invention makes use of input provided to the computer system via input devices such as a keyboard, mouse, touchpad, or the like. Such hardware components, including their operation and interactions with one another and with a central processing unit of the personal computer, are well known in the art of computer systems and therefore are not depicted here. Several Figures are presented as screen shots depicting examples of the user interface as it might appear on a display screen or other output device.

0000Method

0048Referring now to <figref idref="DRAWINGS">FIG. 11</figref>, there is shown a flow diagram depicting a method for practicing the present invention according to one embodiment.

0049Referring also to <figref idref="DRAWINGS">FIG. 1</figref>, there is shown a screen shot depicting an example of a menu bar <b>100</b> including a number of menu headings <b>101</b> and an icon <b>102</b> for initiating a search function according to one embodiment of the present invention. In the example of <figref idref="DRAWINGS">FIG. 1</figref>, icon <b>102</b> is located within menu bar <b>100</b>, although one skilled in the art will recognize that icon <b>102</b> can be located anywhere within the display area of the screen. In one embodiment, icon <b>102</b> is consistently shown as the rightmost item within menu bar <b>100</b> in every application. Alternatively, the search functionality described herein can be activated by a keyboard command, voice command, touch command, or by any other mechanism that may or may not correspond to an on-screen icon such as <b>102</b>.

0050The present invention makes it easier for a user to find a particular menu command or other user interface element when he or she does not know where the command is located within the user interface. The user clicks <b>1102</b> on icon <b>102</b> to cause a search field <b>201</b> to be displayed <b>1103</b>, as shown in <figref idref="DRAWINGS">FIG. 2</figref>.

0051The system of the present invention receives <b>1104</b> query text <b>301</b> forming at least part of a search query entered by the user in search field <b>201</b>. Referring now to <figref idref="DRAWINGS">FIG. 3</figref>, as the user enters query text <b>301</b>, results <b>304</b> are displayed <b>1105</b>. In one embodiment, menu <b>302</b> appears, showing a list <b>303</b> of search results <b>304</b> corresponding to commands that at least partially match the entered text. If the user enters <b>1106</b> additional text, the method returns to steps <b>1104</b> and <b>1105</b> to display new results for the updated query text <b>301</b>.

0052In the example shown in <figref idref="DRAWINGS">FIG. 3</figref>, the user has entered “filter”. Menu <b>302</b> includes search results <b>304</b> that contain the text string “filter”. For some of the listed search results <b>304</b>, a path to the corresponding command is shown, to help the user understand the menu structure so that he or she can better locate the command in the future without necessarily performing another search. For example, for a Video Filters command within the Effects menu, menu <b>302</b> includes the path “Effects->Video Filters”. This tells the user that the Video Filters command can be accessed by opening the Effects menu.

0053In one embodiment, as the user enters query text <b>301</b>, menu headings <b>101</b> that contain commands matching the entered text <b>301</b> are highlighted using a distinctive glowing or spotlight effect. This effect may be the same as the spotlight effect described herein and applied to matching commands <b>504</b>, or it may be a different type of highlight, or any other mechanism for directing attention to particular commands <b>504</b> and/or UI elements. For example, an arrow or other type of pointer can be displayed. In some contexts a tactile mechanism (e.g. vibration) can be provided.

0054Referring now to <figref idref="DRAWINGS">FIG. 14</figref>, there is shown an embodiment where menu <b>302</b> presents search results <b>304</b> that include user interface elements as well as other locations where the query text <b>301</b> was found. The UI element results can be integrated with other results or they can be listed in a separate section of menu <b>302</b> or list <b>303</b> (or in a separate menu <b>302</b> or list <b>303</b>). In addition, certain types of results can be highlighted or displayed differently if desired.

0055For example, in <figref idref="DRAWINGS">FIG. 14</figref> the query text <b>301</b> is the word “mailbox”. Result <b>304</b>D is a “top hit”, indicating that it has been determined to be the most likely search result of interest to the user. Results <b>304</b>E represent documents that contain query text <b>301</b>. Results <b>304</b>F represent user interface elements that contain query text <b>301</b>. Results <b>304</b>G represent folders whose titles contain query text <b>301</b>. Results <b>304</b>H represent mail messages containing query text <b>301</b>. In one embodiment, the user can select which types of results are displayed and how they are displayed.

0056Accordingly, the functionality of the present invention can be implemented in or added to any search functionality for searching documents and/or other items, without requiring the user to access a separate menu or command to perform UI element searches.

0057In one embodiment, as will be described in more detail below in connection with <figref idref="DRAWINGS">FIG. 6</figref>, a search table <b>601</b> is maintained internally, and can be updated periodically if desired. In table <b>601</b>, each command <b>504</b> is associated with one or more text strings and/or other metadata, including for example a title for command <b>504</b>, other text that appears within a dialog box or other element associated with menu command <b>504</b>, and/or alternative and descriptive text strings for command <b>504</b>. Thus, search results list <b>303</b> in menu <b>302</b> can include not only names of commands <b>504</b>, but also descriptive text and/or alternative text for which there is a match with query text <b>301</b>. In one embodiment, search table <b>601</b> is constructed in advance by automatically grabbing all text that appears in and/or is associated with each command <b>504</b> (possibly omitting commonly used words such as prepositions, articles, and the like). In another embodiment, search table <b>601</b> is manually constructed by a developer, author, or other individual.

0058Accordingly, in one embodiment, results menu <b>302</b> includes some results <b>304</b> that do not themselves contain the query text <b>301</b> entered by the user, but that are associated with descriptive text and/or alternative text that does contain that query text <b>301</b>. For example, referring now briefly to <figref idref="DRAWINGS">FIG. 5C</figref>, Include Original Attachment is associated with alternative text that specifies that it is used in the context of replying to an email message. In other words, users may be searching for an Include Original Attachment option, without realizing that the appropriate command <b>504</b> in which the option appears is actually the Reply command <b>504</b>. Thus, when the user enters “reply” as query text <b>301</b>, “Include Original Attachment” appears as one of the search results <b>304</b> listed in menu <b>302</b>. Such alternative text can include common misspellings, as well as alternative terminology that may be used in other operating systems, or that may be commonly used by users or other individuals when referring to a particular feature or function.

0059Thus, search results as presented in menu <b>302</b> include not only those results <b>304</b> whose names match query text <b>301</b>, but also includes results <b>304</b> corresponding to commands <b>504</b> that have matching descriptive or alternative terms. In one embodiment, results <b>304</b> that are semantically related to query text <b>301</b> are also included.

0060In one embodiment, menu <b>302</b> only includes results <b>304</b> corresponding to commands <b>504</b> that apply to installed features. If a particular hardware or software function is not installed on the user's computer, then any results <b>304</b> corresponding to commands <b>504</b> that apply to that feature are omitted from menu <b>302</b>. In one embodiment, the exclusion of such results <b>304</b> is facilitated by maintaining, in table <b>601</b>, keys that indicate the hardware and/or software configurations to which each entry in table <b>601</b> applies. If the hardware and/or software configuration for a particular entry in table <b>601</b> is not present in the system, the entry is excluded from searches. In one embodiment, if the user enters query text <b>301</b> that refers to an uninstalled feature, he or she is notified that the feature is not present (for example, by presenting a dialog box, or by placing explanatory text within menu <b>302</b>). In one embodiment, the user is given an opportunity to click on a link or button to install the feature.

0061For example, if the user searches for BlueTooth in a system that does not have BlueTooth, any results <b>304</b> relating to BlueTooth-related commands <b>504</b> are omitted from the search results. A dialog box or other explanatory text is presented to explain to the user that the system does not have BlueTooth.

0062In one embodiment, when displaying a menu item representing an alternative terminology match, the alternative terminology is displayed alongside or instead of the menu item, so as to clarify the reason for the appearance of the item on menu <b>302</b>. In another embodiment, only the alternative terminology appears as the menu item. In yet another embodiment, only the primary terminology appears, with no reference to the alternative terminology. In yet another embodiment, only the primary terminology appears, but it is displayed using some distinctive visual characteristic (such as a different color), or is accompanied by some indicator to clarify that it is an alternative text match.

0063Referring again to <figref idref="DRAWINGS">FIG. 11</figref>, in one embodiment, the user can indicate an interest <b>1107</b> in one of the listed results <b>304</b>, for example by clicking on it, hovering a mouse cursor over it, and/or selecting it via keyboard up/down arrow keys. In one embodiment, as the user selects <b>1107</b> items in this manner, corresponding user interface elements are brought into view and highlighted <b>1108</b>. For example, referring now to <figref idref="DRAWINGS">FIGS. 5A and 5B</figref>, there are shown examples of a distinctive spotlight effect that can be used to highlight a user interface element <b>402</b> corresponding to a selected result <b>304</b>A or <b>304</b>D in menu <b>302</b>. The spotlight effect is achieved by slightly darkening most of the display, except for a circular or oval-shaped area <b>401</b> immediately surrounding the element <b>402</b> to be highlighted. Thus, the circular area <b>401</b> appears slightly brighter than does the remainder of the screen, so that the overall effect is that of a spotlight shining on the highlighted element <b>402</b>. The effect is reinforced by a gradual transition of brightness from the central area for the highlighted element <b>402</b> to the darker region. In alternative embodiments, the spotlight effect can be accomplished by brightening the area surrounding the highlighted element <b>402</b> and leaving the remainder of the screen at the same brightness as before. In yet other embodiments, other types of highlighting may be used.

0064In one embodiment, the user interface elements are highlighted automatically, so that the user need not select or otherwise indicate an interest in one of the listed results <b>304</b> in order to activate such highlights. For example, user interface elements <b>402</b> for all listed results <b>304</b> can simultaneously be brought into view and highlighted. Alternatively, user interface elements <b>402</b> for some subset of the listed results <b>304</b> can be brought into view and highlighted. For example, the N most likely matches can be shown and highlighted, with N being determined based on display limitations (because of the possible difficulty in displaying many menus and/or user interface elements <b>402</b> simultaneously) or other factors. Sequential highlighting can be used instead of simultaneous highlighting, with each of a number of user interface elements <b>402</b> (each corresponding to a listed result <b>304</b>) being highlighted in turn.

0065In one embodiment, the menu (or element) that contains a result element <b>402</b> is highlighted first. Then, when the user selects or hovers over the containing element, the next step on the path to the result element <b>402</b> is highlighted. In this manner, the user is shown the path to follow to reach the result element <b>402</b>, in a step-by-step manner. For example, searching for “Font Book” might cause “Finder” to be highlighted. When the user selects Finder, the “Services” menu is highlighted. When the user selects “Services”, the Font Book menu item is highlighted.

0066“Highlighting,” as used herein, is not limited to the specific spotlight effect described and illustrated. Rather, the term “highlighting” is intended to refer to any form of effect by which some user interface elements <b>402</b> or items are distinguished from others. Examples of such effects include changing the size, shape, color, contrast, and/or other visual features of the user interface element <b>402</b> or of the surrounding area. In other embodiments, highlighting refers to displaying one or more items in a specific area of the screen or window, or rearranging some items within a display area to indicate that they are highlighted. In yet other embodiments, highlighting is accomplished using nonvisual means; for example, highlighted items can produce a distinctive sound when pointed to by a cursor, or they can have a distinctive tactile characteristic. Other alternatives will be apparent to one skilled in the art.

0067One example of an alternative form of highlighting is shown in <figref idref="DRAWINGS">FIG. 12</figref>, where an oval <b>1201</b> (or other shape) is drawn around the highlighted menu command <b>504</b> or other element. The oval <b>1201</b> (or other shape) can be presented in a distinctive color, if desired.

0068In one embodiment, the presentation of menu <b>302</b> takes place automatically as the user types query text <b>301</b> in search field <b>201</b>. Thus, the user need not hit an Enter key or otherwise manually activate the search command. The list <b>303</b> of results <b>304</b> in menu <b>302</b> is updated automatically as the user enters additional characters in field <b>201</b>, or as the user hits the backspace key to delete characters. In general, typing additional characters lengthens query text <b>301</b> string, thus narrowing the search and causing menu <b>302</b> to include fewer items; conversely, deleting characters shortens query text <b>301</b> string, thus broadening the search and causing menu <b>302</b> to include more items. In one embodiment, processing is performed as each character is input, and feedback is provided substantially instantaneously upon receipt of the input, for example as described in U.S. Pat. No. 4,704,703 for “Dynamic Input Processing System.”

0069In an alternative embodiment, searches are performed in response to a certain trigger event. For example, a search may be initiated or updated whenever the user pauses while typing query text <b>301</b>; a predetermined minimum pause length can be established. Alternatively, a search may be initiated or updated when the user requests it, by hitting an Enter key or search button, or issuing a verbal command, or the like.

0070In one embodiment, when a user interface element <b>401</b> is brought into view and highlighted <b>1108</b> in response to user selection of a result <b>304</b>, surrounding user interface elements are also displayed so as to provide context. For example, a menu, window, or dialog box containing the element <b>401</b> can be shown, so that the user can see where the element <b>401</b> is positioned in relation to other user interface elements. Referring again to <figref idref="DRAWINGS">FIG. 5A</figref>, for example, Outbox window <b>403</b> is shown, so that the user can see the context of reply button <b>402</b> within the overall user interface.

0071Referring also to <figref idref="DRAWINGS">FIGS. 16A to 16E</figref>, there are shown screen shots depicting an example of results menu <b>302</b> presenting search results, and examples of user interface elements being displayed in response to user highlighting of an item in results menu <b>302</b>.

0072<figref idref="DRAWINGS">FIG. 16A</figref> depicts results menu <b>302</b> presented in response to user entry of query term <b>301</b> in field <b>201</b>. Menu <b>302</b> includes list <b>303</b> of results <b>304</b> as described above. Results <b>304</b> can be highlighted by the user to cause a user interface element to be displayed along with relevant context. For example, the user can use arrow keys on a keyboard, or an onscreen cursor, to highlight any result <b>304</b> and cause the appropriate user interface context element(s) to appear on the screen.

0073In <figref idref="DRAWINGS">FIG. 16B</figref>, the user has highlighted “Preferences>New Document” result <b>304</b><i>i</i>. Preferences window <b>1601</b> is automatically displayed, and New Document item <b>1603</b> within window <b>1601</b> is shown. In this embodiment, the displayed item <b>1603</b> is highlighted in a different manner than described in connection with <figref idref="DRAWINGS">FIG. 5A</figref>. Here, arrow <b>1602</b> is shown pointing to item <b>1603</b>. In one embodiment, arrow <b>1602</b> is presented in a distinctive color and is animated when first presented, so as to emphasize its importance and to draw attention to item <b>1603</b>.

0074In <figref idref="DRAWINGS">FIG. 16C</figref>, the user has highlighted “Preferences>Show Properties . . . ” result <b>304</b><i>j</i>. The corresponding user interface element <b>1604</b> is shown. Since element <b>1604</b> is presented within New document pane of Preferences pane <b>1601</b>, New document item <b>1603</b> is activated so as to display the appropriate pane for showing element <b>1604</b>. It can be seen, therefore, that when appropriate to display initially hidden elements, the method appropriate user interface elements are activated to cause the hidden elements to be displayed.

0075In <figref idref="DRAWINGS">FIG. 16D</figref>, the user has highlighted “Font>Document Color” result <b>304</b><i>k</i>. Font dialog box <b>1605</b> is shown, with arrow <b>1602</b> displayed and pointing to document color icon <b>1606</b>.

0076In <figref idref="DRAWINGS">FIG. 16E</figref>, the user has highlighted “Document Properties>Author(s)” result <b>304</b><i>m</i>. Document Properties dialog box <b>1607</b> is shown, with arrow <b>1602</b> displayed and pointing to author field <b>1608</b>.

0077In one embodiment, the system of the present invention maintains a hierarchical representation of relationships among menus, windows, dialog boxes, and other user interface elements, so that the appropriate set of user interface elements can be displayed in order to reach the user element represented by the user-highlighted result item <b>304</b>. Additional information describing the schema for the hierarchical representation is provided below.

0078In one embodiment, the user can activate <b>1109</b> the highlighted element <b>401</b> or any other displayed element. For example, in <figref idref="DRAWINGS">FIG. 5A</figref>, the user can click on and activate any button or icon in window <b>403</b>, and is not limited to activating the highlighted element <b>401</b>. In one embodiment, the user can activate the highlighted element <b>401</b> by clicking on it, or by clicking on the corresponding result <b>304</b>A in menu <b>302</b>, or by hitting Enter (or some other key) while result <b>304</b>A is selected or highlighted. In response to user activation <b>1109</b> of a selected result <b>304</b> in menu <b>302</b>, or of the corresponding element <b>401</b>, the system of the present invention initiates <b>1110</b> the corresponding command and dismisses <b>1111</b> menu <b>302</b> and search field <b>201</b>. In one embodiment, when the user activates result <b>304</b> from menu <b>302</b>, the corresponding element or command momentarily blinks, so as to further reinforce the connection between results <b>304</b> and their corresponding elements or commands.

0079In one embodiment, an icon (not shown) is displayed adjacent to one or more results <b>304</b> in menu <b>302</b>. Clicking on the icon causes the corresponding command or UI element to be brought into view, in a similar manner to that described above (for example, displaying the menu and/or window containing the command or user interface element).

0080In another embodiment, whenever one or more elements <b>401</b> are highlighted, non-highlighted user interface elements are temporarily disabled, although the user can click on a highlighted element <b>401</b> to activate it.

0081Referring now to <figref idref="DRAWINGS">FIG. 4A</figref>, there is shown an example where the selected search result <b>304</b>B in menu <b>302</b> corresponds to a menu command <b>504</b>B in menu <b>552</b>. Accordingly, menu <b>552</b> is brought into view, and command <b>504</b>B is highlighted using a spotlight effect.

0082Referring now to <figref idref="DRAWINGS">FIG. 4B</figref>, there is shown an example where the selected search result <b>304</b>F in menu <b>302</b> corresponds to a menu command <b>504</b>F in menu <b>552</b>A, which is a submenu of menu <b>552</b>B, which is in turn a submenu of menu <b>552</b>C. Accordingly, all three menus <b>552</b>A, <b>552</b>B, and <b>552</b>C are brought into view to show the user the path to command <b>504</b>F. Also, the commands <b>554</b>B, <b>554</b>C that are used to activate the respective submenus <b>552</b>B, <b>552</b>A are highlighted using standard inverse-video highlighting techniques.

0083Similarly, <figref idref="DRAWINGS">FIG. 4C</figref> shows another example where the selected search result <b>304</b>G in menu <b>302</b> corresponds to a menu command <b>504</b>G in menu <b>552</b>D, which is a submenu of menu <b>552</b>E, which is in turn a submenu of menu <b>552</b>F. Again, all three menus <b>552</b>D, <b>552</b>E, and <b>552</b>F are brought into view to show the user the path to command <b>504</b>G. Also, the commands <b>554</b>F, <b>554</b>E that are used to activate the respective submenus <b>552</b>E, <b>552</b>D are highlighted using standard inverse-video highlighting techniques.

0084Similarly, <figref idref="DRAWINGS">FIG. 5C</figref> shows another example where the selected search result <b>304</b>E in menu <b>302</b> corresponds to a menu command <b>504</b>E in menu <b>555</b>, which is a submenu of menu <b>553</b>. Here, both menus <b>553</b>, <b>555</b> are brought into view to show the user the path to command <b>504</b>E. Command <b>554</b> that is used to activate the submenu <b>555</b> is highlighted using standard inverse-video highlighting techniques.

0085The user can navigate within menu <b>302</b>, for example by clicking up/down arrows on a keyboard, or by moving an on-screen cursor so that it hovers over a result <b>304</b>. In one embodiment, the currently selected result <b>304</b> in menu <b>302</b> is highlighted using standard inverse-video highlighting techniques. One skilled in the art will recognize that any other distinctive visual technique can be used to indicate a highlighted or selected result <b>304</b> in menu <b>302</b>.

0086In one embodiment, as described above, a glowing or spotlight (or other type of highlight) is applied to headings <b>101</b> for menus containing matching commands <b>504</b>. Hovering over one of the highlighted menu headings <b>101</b> causes the corresponding menu to be brought into view, with matching command(s) <b>504</b> within that menu highlighted using the spotlight effect. In another embodiment, such a display takes place in response to the user activating the menu by clicking on heading <b>101</b>.

0087The present invention provides a mechanism for allowing a user to quickly locate a menu command or other user interface element he or she is interested in. In addition, the invention serves to train the user as to which menus, dialog boxes, or windows contain which commands. By highlighting various commands and user interface elements as the user enters query text, and by providing additional distinctive highlighting in response to navigation within the menu containing search results, the present invention provides the user with a quick mechanism for learning how to find commands and other user interface elements.

0088In one embodiment, the present invention also provides quick access to help functionality. For example, next to each search result <b>304</b> an icon can be displayed; clicking on the icon causes help text to appear that is relevant to the command identified by the search result <b>304</b>. Alternatively, the user can enter a keystroke or key combination while a search result <b>304</b> is highlighted to cause help text to be displayed for the command identified by the highlighted search result <b>304</b>.

0089Referring now to <figref idref="DRAWINGS">FIG. 7</figref>, there is shown an alternative embodiment where text field <b>201</b> is presented within menu search window <b>701</b> that is not connected to the menu bar <b>100</b>. Window <b>701</b> may be moved around the screen, resized, maximized, minimized, and/or dismissed according to the user's wishes. In one embodiment, the user may drag menu <b>302</b> (shown in <figref idref="DRAWINGS">FIG. 3</figref>) off of menu bar <b>100</b> to automatically convert menu <b>302</b> into menu search window <b>701</b>, and may drag menu search window <b>701</b> back onto menu bar <b>100</b> to convert it back to menu <b>302</b>. Referring now to <figref idref="DRAWINGS">FIG. 8</figref>, there is shown an example of menu search window <b>701</b> including query text <b>301</b> and results <b>304</b>. In one embodiment, the user interacts with results <b>304</b> the same way, regardless of whether they are presented in menu search window <b>701</b> or in menu <b>302</b>. <figref idref="DRAWINGS">FIG. 9</figref> depicts an example of a command <b>504</b> being automatically brought into view in response to the user having selected a result <b>304</b>C in menu search window <b>701</b>.

0090The present invention provides functionality for locating user interface elements that are currently visible on the screen, as well as user interface elements that are not currently visible on the screen. For example, the user can enter a search query for a command within a print dialog box; the results including the command will be shown whether or not the print dialog box is currently on the screen.

0000Functional Architecture

0091Referring now to <figref idref="DRAWINGS">FIG. 6</figref>, there is shown a block diagram depicting a functional architecture for implementing the invention according to one embodiment. As described above, the various components of <figref idref="DRAWINGS">FIG. 6</figref> may be implemented in software. System <b>600</b> includes various menus <b>500</b> (and/or other user interface elements). Each menu <b>500</b> includes any number of commands <b>504</b>. Search table <b>601</b> maintains search terms and their relationships to menus <b>500</b> and commands <b>504</b>; as described above, search table <b>601</b> may include names for commands <b>504</b> as well as alternative and/or descriptive text. Search table <b>601</b> may be automatically constructed and updated from menus <b>500</b> and commands <b>504</b>, or it can be manually constructed and updated. User interface <b>602</b> accepts input from input device <b>605</b> and outputs results via display <b>603</b>. User interface <b>602</b> includes search field <b>201</b> for accepting user entry of query text <b>301</b>, as described above. User interface <b>602</b> also includes menu <b>302</b> that presents search results <b>304</b> as described above. Query engine <b>604</b> is a software component that takes query text input <b>301</b> from search field <b>201</b>, compares it with entries in search table <b>601</b>, and generates search results <b>304</b> to be displayed in menu <b>302</b>. Menu <b>302</b> is a component of user interface <b>602</b> that is output via display device <b>603</b>. When menus <b>500</b> and/or commands <b>504</b> are to be displayed, user interface <b>602</b> handles the display and interaction with these elements as well.

0092As will be apparent to one skilled in the art, the invention is not limited to the particular application set forth in the above description, which is intended to be exemplary. For example, the invention can be used for searching for commands within windows, dialog boxes, folders, subfolders, taskbars, toolbars, or any other user interface element. The invention can be used for searching for user interface elements other than commands. The invention can also be used for searching for commands that are represented as icons or other non-textual elements.

0000Software Implementation

0093In the following description of a software implementation, the term “GRL” (Graphic Resource Locator) is used. A GRL is a persistent representation of how to reach a particular UI Element within the UI, containing enough information to display the item and/or make it perform its action. In one embodiment a GRL is a pair (path, data). The GRL path is a description on how to get to a UI element, and data can be any kind of information about the element (such as the title of the element, bounding box, description, or the like).

0094A GRL stores attributes that can be later used for searching. A UI item reachable if it can be made visible by sending a sequence of events to the application. Some reachable UI items are reachable only with user entering or selecting user-specific data. In one embodiment, the system of the present invention is able to display UI items that are reachable without the need for user-specific data.

0095An example of a text-based representation of a GRL is: path=<menu: “TextEdit”><window: “Preferences . . . ”><tab: “New Document”><view: 3><GUI state: “Plain Text”>data=“Plain Text”

0096Referring now to <figref idref="DRAWINGS">FIG. 10</figref>, there is shown a block diagram depicting a software architecture for implementing the invention according to one embodiment.

0097In one embodiment, the present invention is implemented as a stand-alone application. In other embodiments, it can be implemented as an embedded feature of a software application or operating system, or as a plug-in or widget

0098In one embodiment, the invention is implemented using three functional software modules: indexer module <b>1001</b>, search module <b>1002</b>, and display module <b>1003</b>.

0099Indexer module <b>1001</b> is responsible for computing an internal representation <b>1004</b> of an application for search purposes. In one embodiment, internal representation <b>1004</b> is a tree architecture. The root of the tree represents the application (it is actually wrapper for the AXUIElementRef of the current application processor identifier (PID). Each node in the tree describes a user interface element. Each node contains the following information:

0100a) information about the UI element, including its role, title, local graphical position within a parent window, and in general any attributes that are relevant to the search functionality;

0101b) pointers to maintain the tree representation; and

0102c) the element icon which is used when the element string is displayed in menu <b>302</b>.

0103Search module <b>1002</b> is responsible for retrieving elements in the application that match query text <b>301</b>. It checks whether query text <b>301</b> matches any nodes in the tree or other internal representation <b>1004</b>. The elements that match are added to results menu <b>302</b>. Search module <b>1002</b> is also responsible for ranking results <b>304</b> according to a given heuristics (e.g. most recently used, most used, extent of the match, or the like). Search module <b>1002</b> is also responsible for building results menu <b>302</b>. Results menu <b>302</b> is empty when text field <b>201</b> contains no query text <b>301</b>.

0104Display module <b>1003</b> is responsible for displaying the text search field <b>201</b> and for implementing the animation of the highlighted commands <b>504</b> and other user interface elements. In one embodiment, text search field <b>201</b> is implemented as a standalone window <b>701</b> as shown in <figref idref="DRAWINGS">FIG. 7</figref>; in another embodiment it is attached to menu bar <b>100</b> as shown in <figref idref="DRAWINGS">FIG. 2</figref>.

0105In one embodiment, display module <b>1003</b> includes animation code for displaying various commands <b>504</b> and user interface elements in response to user selection of results <b>304</b> within menu <b>302</b>. Displaying a user interface element may involve bringing a window (or any other user interface element) into view, if it is hidden or inactive when the corresponding result <b>304</b> is selected in menu <b>302</b>. In other cases, displaying a user interface element may involve opening a menu <b>554</b> and/or submenu <b>552</b>, as depicted in <figref idref="DRAWINGS">FIG. 5</figref>.

0106In addition, display module <b>1003</b> includes animation code for implementing a spotlight effect to highlight the selected user interface element, as described above. In one embodiment, the graphic effects are achieved using the OpenGL API, available from Apple Computer, Inc. of Cupertino, Calif. In another embodiment, the graphic effects are achieved using CoreGraphics, a framework available from Apple Computer, Inc.

0107In one embodiment, display module <b>1003</b> sends display information to user interface <b>602</b> so that the appropriate display can be shown to the user.

0108In one embodiment, an additional module (not shown) is responsible for performing the action associated with a selected result <b>304</b> when the user hits the Enter or Return key.

0109In one embodiment, indexer module <b>1001</b> generates a dynamic index by storing the UI elements that become visible at run time (as a result of a user action). Dynamic indexing stores information about various UI elements as they become visible at run time. Whenever a new UI element becomes visible on the screen, a node is added to the tree that forms internal representation <b>1004</b>. Nodes are not removed when a UI element is dismissed from the screen. In this fashion, the construction of internal representation <b>1004</b> of the UI elements takes place as a learning process over time; the result is a tree-based organizational scheme that can be used to quickly identify relationships among UI elements.

0110In one embodiment, in order to avoid presenting results <b>304</b> that are no longer valid based on the current state of the user interface, the UI element for each potential results <b>304</b> is validated before menu <b>302</b> is presented. Validation is performed by determining whether or not the UI element can be displayed given the current state of the application and/or operating system. In one embodiment, results <b>304</b> that correspond to currently disabled or unavailable commands or elements are shown in grayed-out fashion or using some other distinctive visual technique within menu <b>302</b>; in another embodiment, they are shown at the bottom of the list of results; in another embodiment, they are not shown at all.

0111In one embodiment, the system of the present invention makes its internal representation <b>1004</b> available for use by other applications and/or by other instances of the system, for example in a publicly available location identified by a URL. In this manner, one user's system can take advantage of internal representation <b>1004</b> generated by another user's system, and can even update internal representation <b>1004</b> as appropriate. In some cases, various applications and/or user computers can share the same internal representation <b>1004</b>.

0112In another embodiment, the system of the present invention uses static indexing. In static indexing, the UI structure of the application and/or operating system is scanned in advance using a software tool. The software tool activates every available UI element so that it can be added as a node in internal representation <b>1004</b>. The system of the present invention can thereby generate internal representation <b>1004</b> without waiting for all of the various UI elements to be brought into view during the normal course. In one embodiment, internal representation <b>1004</b> can be shipped along with the software product, so that users of the software product can immediately take advantage of the features provided by the present invention. Internal representation <b>1004</b> can be updated as appropriate whenever a software application or operating system is updated or reconfigured. A description of a software tool for scanning an application, referred to as an “App Analyzer”, is provided below.

0113In one embodiment, an internal representation <b>1004</b> is generated and stored for each localized version of a software application or operating system. In another embodiment, internal representation <b>1004</b> is generated and stored for one canonical version of the software application, such as for example the U.S. version, and other versions are handled via on-the-fly scanning of localized strings that differ from the U.S. version. Thus, a hybrid approach can be realized wherein a static index is used for canonical versions and a dynamic index is used for other versions.

0114In one embodiment, the index is language-dependent. For a multi-lingual operating system environment, an index is computed and stored for each language. Depending on which language is used for the current instance of the application or operating system, the appropriate index is used.

0115In another embodiment, the user interface elements are not indexed. Rather, an application programming interface (API) is provided that allows applications to register their UI elements and structure with the system of the present invention.

0116In one embodiment, the present invention is implemented using a Model View Controller design pattern. Referring now to <figref idref="DRAWINGS">FIG. 17A</figref>, there is shown an overview <b>1700</b> of classes and relationships for implementing the present invention according to one embodiment.

0117SCTGRLIndex <b>1701</b> is the model for the hierarchical representation depicting relationships among user interface elements. It contains a collection of Graphical Resource Locators (GRLs).

0118NSArray <b>1705</b> contains the data according to the model.

0119SCTSearchManager <b>1702</b> is the primary controller according to one embodiment. It coordinates user interactions and queries on the model. It also handles wiring for various app level notifications.

0120NSView <b>1703</b> and NSTableView <b>1704</b> specify the appearance of the search and results interface. In one embodiment, NSView <b>1703</b> and NSTableView <b>1704</b> use AppKit view classes placed inside a Carbon Help Menu. SCTSearchManager <b>1702</b> is used as the data source and delegate as appropriate. A specialized class SCTAnimationController <b>1706</b> handles the openGL animation to showcase the UI element to the user, for example by dimming the screen and drawing a spotlight around the displayed UI element. In the context of this description, “showcase” means to display a UI element.

0121Referring now to <figref idref="DRAWINGS">FIG. 17B</figref>, there is shown an event trace <b>1710</b> depicting the flow of control that takes place when a user enters text in field <b>201</b>.

0122The user enters <b>1713</b> query <b>301</b> in field <b>201</b>. SCTSearchManager <b>1702</b> is activated as a result of text being changed <b>1714</b> in field <b>201</b>. SCTSearchManager <b>1702</b> initiates search <b>1716</b> on index <b>1701</b>, generating search results <b>1711</b>. NSTableView <b>1704</b> is informed that an update to its display is needed via reloadData message <b>1717</b>.

0123The user selects <b>1718</b> one of the displayed results <b>304</b>. SCTSearchManager <b>1702</b> is informed that the selection changed via selectionChanged message <b>1719</b>. SCTSearchManager <b>1702</b> causes the appropriate UI element to be displayed by sending a showcase message <b>1720</b> to SCTAnimationController <b>1706</b> that references the appropriate GRL <b>1712</b>.

0124In one embodiment, Graphical Resource Locators (GRLs) are represented according to a hierarchy. Referring now to <figref idref="DRAWINGS">FIG. 17C</figref>, there is shown a class hierarchy <b>1730</b> for GRLs, starting with class SCTGRL <b>1731</b>.

0125In the example shown in <figref idref="DRAWINGS">FIG. 17C</figref>, GRLs are subclassed to represent particular kinds of UI elements such as Views, Windows, Menu Items, Accessibility Attributes, and the like. Classification can also take place between directly reachable UI elements (those that are accessible programmatically when the application is launched and has a default document window open) and indirect UI elements (those that are only visible after performing one or more actions in the user interface). Examples of directly reachable UI elements include: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0126">Menu Items reachable via the Menu Bar.</li><li id="ul0002-0002" num="0127">Buttons/Checkboxes/Text fields/etc. on the main window.</li></ul></li></ul>

0128An examples of indirect UI elements might be a check box buried inside a preferences window; the preferences menu would have to be invoked in order to show the preferences window and thereby display the check box.

0129Classification can also take place between statically reachable UI elements (those that are always reachable) and data-dependent UI elements (those that are reachable only when the application is in specific states, and/or require the entry of specific data). Examples of statically reachable UI elements include: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0130">Items in the File Menu.</li><li id="ul0004-0002" num="0131">Controls in the preferences panel.</li></ul></li></ul>

0132Examples of data-dependent UI elements include: <ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0000"><ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0133">Table inspector (visible only after inserting a table into text).</li><li id="ul0006-0002" num="0134">Login Options tab (visible only after clicking on a lock icon in an Accounts pane of system preferences and entering a password).</li></ul></li></ul>

0135In one embodiment, data-dependent UI elements are not indexed or searched on by the system of the present invention.

0136Members of the SCTMenuItemGRL class <b>1732</b> include paths of strings that identify the path to the menu item of interest.

0137Members of the SCTWindowGRL class <b>1733</b> include a key to recognize the type of window it is (usually the title+the class) and a trigger GRL. The trigger GRL is the action that causes the window to appear.

0138Members of the SCTViewGRL class <b>1734</b> are views that are contained within windows or other containers. They contain the GRL for their container and a path of subview indices that locate the specific view. View GRLs are subclassed for various specialized types such as Buttons (SCTButtonGRL class <b>1735</b>) and TextFields (SCTTextFieldGRL class <b>1736</b>).

0139Members of the SCTHelpItemGRL class (not shown) contain a Uniform Resource Locator (URL) that points into a help system.

0140Members of the SCTApplicationGRL class <b>1737</b> point to another application. This may be used, for example, to locate other indices that may be linked to the application.

0141Members of the SCTAccessibilityGRL class <b>1738</b> provide access to the accessibility hierarchy.

0142Referring now to <figref idref="DRAWINGS">FIG. 17D</figref>, there is shown a SCTInvocation class <b>1741</b> and a SCTGRLResolver class <b>1742</b>. In some cases, an indirect UI element can be displayed only by triggering a series of events in order. For example suppose a GRL is provided for a “Show Ruler” checkbox in a preferences panel. If the preferences panel is not visible it must first be displayed before the checkbox can be showcased. The SCTInvocation class <b>1741</b> and a SCTGRLResolver class <b>1742</b> accomplish this task.

0143The SCTInvocation class <b>1741</b> is a wrapper to store a target and a selector. When a client calls on it, it can invoke the selector on the target passing in the argument passed into perform. It is able to do this either directly or it can be wired as the target of a notification in which case it passes in the notification object as the argument to perform.

0144The SCTGRLResolver class <b>1742</b> is a subclass of the SCTInvocation class <b>1741</b> that contains a reference to a GRL. It functions the same way as an Invocation except that whatever is passed into the perform method is resolved (changed) by the GRL. For example, a window GRL will resolve a window to its content view. A view GRL will resolve a root view in a view hierarchy to the specific sub-view (such as the OK button in a dialog box) that the GRL refers to.

0145Referring now to <figref idref="DRAWINGS">FIG. 17E</figref>, there is shown an example of the process <b>1751</b> of resolving a window to its content view based on a windowDidBecomeVisible notification. The notification passes in a window which is passed to resolveResource on the GRL. This returns the contentView, which is in turn passed to a call of myMethod on myTarget.

0146<figref idref="DRAWINGS">FIG. 17F</figref> shows another example <b>1761</b> where a “Show Ruler” checkbox is displayed in response to user selection of a “Show Ruler” result.

0000App Analyzer

0147In one embodiment, the system of the present invention searches the application for all reachable UI Items to build an index. A software tool known as App Analyzer scans the UI structure of an application and/or operating system in advance so that it can be indexed appropriately. App Analyzer operates in a controlled environment to walk through the application and/or operating system and expose UI elements therein.

0148App Analyzer creates a collection of GRLs that represent UI elements in an application. This collection forms the index that will be searched when a query is entered. The App Analyzer tool starts by enumerating all the visible GUI elements as GRLs. For example, when the App Analyzer analyzes an application, the initial collection contains the menu items in the menu bar and the document window for the application.

0149Then, the tool executes the action for each GRL in the current collection, and it observes any changes that happen as a result. Examples include: a new window shows up, a window is resized, or a window title changes. In other words, the tool uses a heuristic to determine that new UI elements have been revealed.

0150The GRLs for the new GUI elements (subindex) are added to the collection and the process continues until all GRLs are explored and no new UI elements are found.

0151The index generated by the App Analyzer is then read back into memory when needed, and is used by the system of the present invention to search for UI elements to be showcased.

0152The above description is set forth in the context of a mechanism for locating user interface elements for a software application. However, one skilled in the art will recognize that the invention can also be implemented as a mechanism for locating user interface elements for an operating system, web-based application, applet, desk accessory, widget, gadget, consumer electronic device, or the like. The invention can also be implemented across more than one such entity; for example, a search query can yield results for an operating system as well as for all installed applications.

0153In the above description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention. It will be apparent, however, to one skilled in the art that the invention can be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the invention.

0154Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.

0155Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

0156It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

0157The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. It will be recognized that the invention may be implemented any type of electronic device, such as handhelds, personal digital assistants (PDAs), mobile telephones, consumer electronics devices, embedded devices, and the like.

0158The algorithms and modules presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatuses to perform the method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the modules, features, attributes, methodologies, and other aspects of the invention can be implemented as software, hardware, firmware or any combination of the three. Of course, wherever a component of the present invention is implemented as software, the component can be implemented as a standalone program, as part of a larger program, as a plurality of separate programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future to those of skill in the art of computer programming. Additionally, the present invention is in no way limited to implementation in any specific operating system or environment.

0159It will be understood by those skilled in the relevant art that the above-described implementations are merely exemplary, and many changes can be made without departing from the true spirit and scope of the present invention. Therefore, it is intended by the appended claims to cover all such changes and modifications that come within the true spirit and scope of this invention.