Facilitating interaction between users and their environments using a headset having input mechanisms.

(54) Title: FACILITATION OF INTERACTION BETWEEN USERS AND THEIR ENVIRONMENTS USING A HEADPHONE WITH INPUT MECHANISMS.

(54) Title: FACILITATING INTERACTION BETWEEN USERS AND THEIR ENVIRONMENTS USING A HEADSET HAVING INPUT MECHANISMS.

(57) Summary

Described here is a headset for presenting audio information to the user as the user interacts with a space, for example, as when a user navigates through a path within a space. The handset may include a set of input mechanisms to receive commands from the user. The commands, in turn, invoke respective space interaction related functions that will be performed by the space interaction module (SI). The headset can operate with or without a separate user computing device.

(57) Abstract

A headset is described herein for presenting audio Information to the user as the user interacts with a space, eg, as when a user navigates over a route within a space. The headset may inelude a set of input mechanisms for receiving commands from the user. The commands, in turn, invoke respective space-interaction-related functions to be performed by a space interaction (SI) module. The headset may operate with or without a separate user computing device.

FACILITATION OF INTERACTION BETWEEN USERS AND THEIR ENVIRONMENTS USING A HEADPHONE WITH INPUT MECHANISMS

BACKGROUND

A user can make use of various conventional mechanisms in the generation and execution of travel plans and / or in exploring their environment in a more spontaneous and unlimited way. For example, a user can use a route selection application to generate a route for use in traveling between two locations. The same application can then guide the user as the user moves along the route through a series of directions. The directions can correspond to spoken and / or displayed messages, such as a message that verbally instructs the user to make a turn within a set distance. However, there is considerable room for improvement in these conventional mechanisms.

BRIEF DESCRIPTION OF THE INVENTION

Various tools are described here to help the user in their interaction with physical and / or virtual spaces. Such assistance, for example, can facilitate user navigation within physical virtual spaces and / or spaces. As used herein, navigation refers to a purposeful movement of the user through a space according to a pre-established plan, and / or a movement that may reflect spontaneous decisions that do not necessarily conform to any predetermined plan. The tools can also help a user's exploration within a space at a given moment, for example, which corresponds to a process by which the user orients himself in the space.

As a general principle, the tools are designed to provide timely and highly informative assistance to the user in a simple way, thus enriching the user experience of the physical and virtual spaces through which the user moves, but without distracting. unduly the user of the actual task of interacting with the spaces. In view of these characteristics, the tools can be used successfully even by users with impaired vision and / or other handicaps. However, the tools are general-purpose in nature, and therefore can provide non-obstructive user-condescending guidance to any user in performing any task in which the user interacts within their environment.

The tools have several aspects that contribute to the performance outlined above. In accordance with one aspect, a headset is described herein for presenting audio information to the user as the user interacts with space. The headset can also include a set of input mechanisms to receive commands from the user. The commands, in turn, invoke respective functions related to space interaction to be performed by a space interaction module (SI).

For example, the commands can instruct the SI module to: stop delivering audio information; repeat an article of last installment of audio information; repeating a set of articles from last delivery of audio information (for example, when rewinding); start a scan mode; start an orientation mode; perform a more information function (to request additional information); start or stop a listening mode (in which speech recognition is on and off, respectively), and so on. In browse mode, the space interaction module provides information on a set of items of interest that are associated with a subspace to which the user's attention is currently directed. In orientation mode, the space interaction module provides information about a set of items of interest that are associated with an entire space around the user, at the current time.

In one implementation, the headset may include its own orientation, movement, and / or position-determining mechanism (s) (eg, a compass). In another implementation, the headset makes use of a separate orientation, movement, and / or position-determining mechanism (s) provided by a separate user device (eg, a smartphone).

In one implementation, a system implements the SI module using processing resources associated with the handset alone. In another implementation, the system implements the SI module using processing resources associated with the separate user computing device alone. In other implementations, the functions performed by the SI module may be shared by either the handset, and / or the user computing device, and / or one or more remote processing resources (eg, cloud-implemented services).

The above features contribute to the above objective of allowing the user to move safely and efficiently through their environment. For example, the features provide a convenient way through which the user can activate various operational modes (for example, by interacting with the headset's user input mechanisms) without unduly distracting the user as the user moves through. environment, for example, without the user having to access and interact with a separate user device.

The above approach can manifest itself in different types of systems, devices, components, methods, computer-readable storage media, data structures, graphical user interface displays, articles of manufacture, and so on.

This Brief Description is provided to present a selection of concepts in a simplified form; these concepts are further described in the Detailed Description below. This Brief Description is not intended to identify the main characteristics or essential characteristics of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 shows an overview of a system to help a user interact with a physical and / or virtual space.

Figure 2 shows an example in which the user uses the system of Figure 1 to take a trip that is defined by a plurality of route points. In other cases, the user can use the system to interact with an environment in a more open and spontaneous way.

Figure 3 shows an overview of the compute functionality that can be used to implement the system in Figure 1.

Figure 4 shows an implementation of a space interaction module, which is a component of the compute functionality of Figure 3.

Figure 5 shows an implementation of an application interface module, which is a component of the space interaction module of Figure 4.

Figure 6 shows an example of a headset, portable user computing device (user device for brevity) and another user computing device; These devices can implement one or more aspects of the system of Figure 1.

Figure 7 shows one way to implement the headset and user device of Figure 6.

Figure 8 shows another way to implement the headset and user device (optional) of Figure 6.

Figure 9 shows a scenario in which a user interacts with the user device of Figure 6 within a vehicle.

Figure 10 is a flow chart illustrating one mode of operation of the equipment shown in Figure 6.

Figure 11 shows different workspaces that can be presented by the application interface module of Figures 4 and 5, for example, in a user interface presentation provided by a display output mechanism of a user device. .

Figure 12 shows a gesture that can be used to transition from a first workspace to a second workspace.

Figure 13 shows a default hub menu that can be displayed in a home workspace, according to one implementation.

Figures 14-16 show respective context menus that can be displayed in the home workspace.

Figure 17 shows a menu identifying a plurality of functions that can be invoked; that menu can be presented in a main workspace.

Figure 18 shows a menu that a user can use to configure various parameters; that menu can be presented in a settings workspace.

Figure 19 shows a menu that a user can use to access information that is relevant to the user, 10 as they travel or interact with an environment; that menu can be presented in an information workspace.

Figure 20 shows a menu that a user can use to access information regarding nearby items of interest (lOls); That menu can be presented in a 15 workspace near me.

Figure 21 shows an example of a transient menu that can be displayed in the home workspace.

Figure 22 shows an example of an overlay menu that, in this particular case, is presented in the settings workspace 20

Figure 23 shows the gestures that a user can use to obtain the audio information about their current context.

Figure 24 shows a gesture that a user can use to activate any menu in any workspace.

Figure 25 shows a gesture that a user can use to navigate through a collection of menu items on any menu.

Figure 26 shows a return gesture that a user can use to navigate from a current menu to some other previous menu.

Figure 27 shows an alternative organization of workspaces, compared to the implementation of Figure 11.

Figure 28 shows tab information in a home workspace of Figure 27, corresponding to a state in which a user has not yet created any tabs.

Figure 29 shows a tab menu in the home workspace of Figure 27, corresponding to a state in which a user has now created a collection of tabs.

Figure 30 shows an alternative way of organizing menu items within a menu, compared to the implementation of Figure 25.

Figures 31 and 32 show an alternative way of scrolling through menu items within a menu and then selecting a menu item, compared to the implementation of Figure 25.

Figure 33 shows a return gesture performed at the periphery of a menu and an action that is performed in response to this gesture.

Figure 34 shows a gesture that is performed by drawing a circle shape on a menu, and a return action that is performed in response to the gesture.

Figure 35 shows a gesture that is performed by drawing a half circle on a menu, and a return action that is performed in response to the gesture.

Figures 36 and 37 show different gestures that can be used to increase or decrease, respectively, a level of verbosity provided by the system of Figure 1 and a level of contextual information provided by the system.

Figure 38 is a flow chart summarizing one mode of operation of the application interface module of Figures 4 and 5.

Figure 39 shows the use of three-dimensional audio information to create a perception of sound emanating from a particular location within space.

Figure 40 shows the use of three-dimensional audio information to create a perception of sound moving through a series of locations within space.

Figure 41 shows one way in which a path guide module can use three-dimensional sounds (eg, a periodic rhythm of sound) to guide the user in the desired direction.

Figure 42 shows one way in which a scanning module can use three-dimensional sounds to identify the locations of 10s that are associated with a current focus of interest to the user.

Figure 43 shows one way in which an orientation module can use three-dimensional sounds to identify 10s that are associated with an entire space around the user.

Figure 44 is a flow chart describing the use of three-dimensional sounds to help the user interact with a space.

Figure 45 is a flow chart describing one way in which the path guide module (Figures 4 and 41) can use three-dimensional sounds to guide the user along a desired path.

Figure 46 is a flowchart describing one way in which the space interaction module can use three-dimensional sounds to identify lOls, for example, in an automated way (for example, while the user travels a route or otherwise moves in space), in a scan mode, or in an orientation mode.

Figure 47 shows an environment that has a plurality of beacons. In an illustrative application, beacons produce non-overlapping ranges where their respective signals can be detected by a user moving through the environment or otherwise not interacting with the environment.

Figures 48-50 show other environments, each with a greater number of beacons compared to the example in Figure 47.

Figure 51 is a flow chart showing one way of operation of a beacon-based guidance module, within the context of the types of environments of Figures 47-50.

Figure 52 is a flow chart that provides more details about one way in which the beacon-based guidance module can determine the current location of the user within an environment.

Figure 53 shows illustrative computational functionality that can be used to implement any aspect of the features shown in the previous drawings.

The same numbers are used throughout the description and figures to refer to similar components and features. Serial numbers 100 refer to features originally found in Figure 1, serial numbers 200 refer to features originally found in Figure 2, serial numbers 300 refer to features originally found in Figure 3, and so on successively.

DETAILED DESCRIPTION

This description is organized as follows. Section A provides an overview of a system to help a user interact with a real and / or virtual space. Section B describes different types of handsets and portable user computing devices that can be used in the Section A system. Section C describes an illustrative user interface experience that can be provided by the Section A system. Section D describes the functionality to generate and use three-dimensional sounds, and other (non-three-dimensional) types of sounds. Section E describes beacon-based guidance functionality to help the user navigate through an environment that is populated with beacons having, in an illustrative implementation, non-overlapping ranges. And Section F describes illustrative compute functionality that can be used to implement any aspect of the features described in the previous sections.

As a preliminary matter, some of the figures describe concepts in the context of one or more structural components, referred to as functionality, modules, features, elements, etc. The various components shown in the Figures can be implemented in any way by any physical and tangible mechanism, for example, through software running on computer equipment, hardware (for example, chip-implemented logic functionality), etc. ., and / or any combination thereof. In one case, the illustrated separation of the various components in the figures into different units may reflect the use of different corresponding tangible and physical components in an actual implementation. Alternatively, or in addition to, any individual component illustrated in the figures may be implemented by a plurality of actual physical components. Alternatively, or in addition, the representation of two or more separate components in the figures may reflect the different functions performed by a single real physical component. Figure 53, which is described in turn, provides additional details on the illustrative physical implementation of the functions shown in the figures.

Other figures describe the concepts in flowchart form. In this way, certain operations are described as different building blocks performed in a certain order. Such implementations are illustrative and not limiting. Some blocks described here can be grouped and performed in a single operation, some blocks can be separated into plural component blocks, and some blocks can be performed in an order that differs from that illustrated here (including a parallel way to perform the blocks). The blocks shown in the flowcharts can be implemented in any way by any physical and tangible mechanism, for example, by software running on computer equipment, hardware (for example, chip-implemented logic functionality), etc. , and / or any combination thereof.

In terms of terminology, the phrase configured to encompasses any way in which any tangible and physical functionality can be constructed to perform an identified operation. The functionality can be configured to perform an operation using, for example, software running on computer equipment, hardware (for example, chip-implemented logic functionality), etc., and / or any combination thereof.

The term logic encompasses any physical and tangible functionality to perform a task. For example, each operation illustrated in the flow chart corresponds to a logic component to perform that operation. An operation can be performed by, for example, software running on computer equipment, hardware (eg, chip-implemented logic functionality), etc., and / or any combination thereof. When implemented by computer equipment, a logic component represents an electrical component that is a physical part of the computer system, however implemented.

The following explanation may identify one or more features as optional. This type of statement should not be construed as an exhaustive indication of features that may be considered optional; that is, other features can be considered as optional, although it is not explicitly defined in the text. Furthermore, any description of a single entity is not intended to preclude the use of the plural of such entities; likewise, a description of plural entities is not intended to preclude the use of a single entity. Furthermore, while the description may explain certain features as alternative ways of performing identified functions or implementing identified mechanisms, the features can also be combined in any combination. Finally, the terms exemplary or illustrative refer to one implementation among the many possible implementations.

A. System Overview

Figure 1 shows an overview of a system 102 to assist a user 104 to interact with a physical and / or virtual space. A physical space can correspond, for example, to an interior space defined by the interior of one or more buildings, or to an exterior space that exists outside the interior of buildings, or to a combination of interior and exterior spaces. A virtual space can correspond, for example, to a domain that is filled with virtual objects of any type or types. In some scenarios, a virtual space can be superimposed on a physical space through which the user can physically move. In those cases, some virtual objects in virtual space can be assigned corresponding real positions in physical space (which will be clarified below).

Figure 1 broadly presents illustrative components that may be used in system 102. Components are optional in the sense that a particular implementation of system 102 may omit one or more of the illustrated components. Additionally, or alternatively, a particular implementation of system 102 may include additional components not shown in Figure 1.

During their exploration of the space (s), user 104 can interact with system 102 via a user computing device 106 of any type (referred to simply as a user device below for short) and a headset 108, or just the 108 earphone alone. User device 106 can correspond to any type of portable computing device having any form factor. For example, user device 106 may correspond to a smartphone, a tablet-type computing device, a portable computing device, a netbook-type computing device, a media consuming device (such as a mobile device). book reader type computing device or music playback device), a portable gaming device, a portable device (such as glasses, glasses, etc.), and so on. In other cases (not shown), a user 104 can carry and use two or more user computing devices, such as a smartphone in combination with a tablet-type computing device.

Headphone 108 can also correspond to any type of device for providing audio information to user 104. In one case, for example, headphone 108 can provide audio information using conventional speakers placed on, or in proximity to, one or more from the user's ears 104. Otherwise, the headset 108 may provide audio information using a bone conduction technique. In a bone conduction technique, the earpiece 108 passes information to the user's eardrums through vibrations imparted to the bones in the user's head. An earphone that uses bone conduction cannot block the ear canals of the user 104 and thus allows the user 104 to hear other (outside) sounds produced by the spaces through which he or she navigates; This result may be desirable, in some cases, to increase the security with which users interact with their physical environments, particularly in the case of users with visual impairments and in the case of users who are not familiar with their environment.

In accordance with a general use mode, user 104 can interact with user device 106 to receive information about their interaction with a space primarily in visual form. Additionally, or alternatively, user device 106 may provide information in non-visual ways, such as by producing haptic feedback signals (eg, vibration-based signals). User 104 may primarily interact with headset 108 to receive audio information about their interaction with a space. That audio information can include spoken information, other sounds, etc. System 102 may also receive instructions from either user device 106 or handset 108, or both.

User 104 can optionally also interact with system 102 through one or more traditionally stationary computing devices 110, such as a desktop computing device, a game console device, a cable tv box device, and so on. For example, user 104 may interact with system 102 using the other user computing device 110 prior to a trip to create travel information that defines a particular route through a space. 104 The user can then upload the trip information to the user device 106 and / or the headset 108. After the trip, the user 104 can re-interact with the system 102 through the other user computing device 110. For example, system 102 may download information about a completed trip to the other user computing device 110, allowing user 104 to review information about that trip at any time, for any purpose.

In one case, all of the functions associated with the system 102 are performed by the processing functionality provided by the components identified above, namely, the user device 106, the handset 108, and the other user computing device 110. In another case, one or more remote processing resources 112 may implement at least some aspects of the processing performed by the system 102, such as those aspects that are particularly computationally intensive in nature. For example, remote processing facilities 112 may include functionality for creating new trips, modifying existing trips, interpreting spoken user instructions, and so on. In one case, the remote processing resources 112 may correspond to one or more server computing devices, one or more data stores, and so on.

System 102 may use any combination of communication conduits 114 to couple the components together described above. For example, user device 106 can interact with headset 108 through any wireless communication mechanism (eg, BLUETOOTH communication), and / or any other wired communication mechanism (eg, via a wireless connection). USB, etc.). User device 106 and handset 108 can communicate with remote components of system 102 through a cellular connection, a Wi-Fi connection, a cable connection, etc., or any combination thereof.

Additionally, although not shown, user device 106 can interact with any of the remote position determination systems in order to determine the position of user device 106. Headset 108 can perform the same function. The remote positioning mechanisms can correspond to any satellite-based positioning system (eg, a GPS system), terrestrial communication towers, and so on. Furthermore, the user device 106 and / or the handset 108 can also interact with local beacons through any communication mechanism (for example, through BLUETOOTH communication, Wi-Fi communication

Fi, etc.).

The remainder of this section (Section A) provides an overview of the functionality provided by the system 102. Later sections provide additional details on the individual components of the system.

In general, a user can utilize the system 102 as a way to enrich the user experience when the user moves through any type of space, familiar or unfamiliar. In one scenario, for example, the user can use the system 102 as a guide in conducting a planned trip from an origin location to a destination location, subject to a desired schedule. In other cases, the user may use system 102 to offer assistance in exploring a space in a more open and spontaneous manner, for example, without a previously planned and / or scheduled trip. For example, a user may use system 102 to offer assistance such as him or her wandering around an unfamiliar city, or past meanders displaying a fair or museum; the user can participate in this activity without a fixed itinerary, and can modify their travel course and objectives in a free and spontaneous way. In another case, the user can use the system 102 as a way to animate a familiar route through familiar surroundings, and so on. In another case, a user may use the system in a hybrid form of operation, for example, where some aspects of the user interaction follow a prepared plan and other aspects are more open and spontaneous in nature.

Referring to Figure 2, this figure shows an example in which a user uses the system 102 of Figure 1 to take a previously planned trip. The illustrative user experience in conducting this journey will be described below, as a way to introduce the reader to the types of functions that the system 102 can perform. Note that this illustrative experience is presented in the spirit of illustration, not limitation; As noted above, the system 102 can be applied in a wide variety of other contexts in which a user interacts with their environment, for any purpose.

In the non-limiting case of Figure 2, the user (a man named John) is assumed to have created a trip before embarking on the trip (although, again, this does not have to be the case). For example, suppose that the user has created a route to take him from his residence in London to a doctor's appointment in another part of the city. The user may have created a trip for this task because he is not familiar with the part of the city through which he will travel. Or the user may have one or more disadvantages presenting various mobility related problems. For example, the user can have any form (and degree) of vision impairment. In this context, the user may have created the trip to help him navigate the route, although he may be familiar with the route. Otherwise, some other entity than the user may have created the trip on behalf of the user. In these cases, the planned trip is defined by the trip information, which describes all aspects of the trip.

Figure 2 specifically depicts the planned route 202 of the trip as a solid line. In some cases, the user may use the system 102, to generally adhere to the planned route 202. In other cases, the user may deviate from the planned route 202 for some reason, for example, because he encounters an obstacle along the route. the planned route 202, or he makes a spontaneous decision to change the course of his trip for whatever reason. For example, in this merely illustrative case, the user has left the planned route 202 in order to visit a store 204 along the way, to buy some item (eg, a sandwich, etc.). Figure 2 depicts the user's actual route 206 as a dashed line.

The planned route 202, in the case illustrated in Figure 2 is defined by a series of transition points (Wi, w₂, w₃, w₄ and w₅) or stations, referred to as reference points in this document. For example, the reference point w₁ may correspond to the starting point of the user's journey, while the reference point w₅ it may correspond to the user's travel destination. The reference points w₂ and w₃ they can correspond to two intersections; at each intersection, the user goes from a first street to a second street. The reference point w₄ it can correspond to any station in which a user can change his mode of transport. For example, the user can travel to the waypoint w₄ walking. The reference point w₄ it may correspond to a transport station where the user waits for the arrival of a transport 208, which arrives at a predetermined time. The user can then continue to the waypoint w₅ in transportation 208. In other cases, the mode of transportation may correspond to a train, bus, tram, subway, or any private mode of transportation (for example, private car, bicycle, etc.).

Therefore, the planned route can be conceived as having a series of segments (yes, s₂, s₃ ys₄). The example in Figure 2 is a simplified case. In other cases, a trip can include many more waypoints and associated segments. And that trip can encompass any combination of modes of transportation. However, in other cases a journey may be less complex than the one shown in Figure 2, for example, including only a starting and ending landmark. And, to repeat, in other cases, the trip cannot be defined in advance.

A context, as the term is used here, generally refers to the circumstance that a user faces at any given time while traveling or interacting in some way with the environment. The circumstance, in turn, is governed by at least the characteristics of the environment with which the user can interact, the current time of day (and the day of the week), etc., and the goals that the user faces in the current moment, etc. For example, at a time t₁₍ user is traversing segment s₁ in an attempt to travel from waypoint w, to waypoint w₂. The context Ci of the user's journey is therefore defined, at least in part, by the user's current location along the planned journey and the user's effort to reach the waypoint w₂ over the planned segment s₂. The context of the user at other times will differ depending on the environment the user is currently facing, along with the respective local goals of the users at that time.

With the foregoing preliminary description of planned route 202 and actual route 206, now consider an illustrative user experience as the user travels from waypoint Wi to waypoint w₅. Assume, in one example, that the user wears at least the user device 106 and wears the headset 108 (shown in Figure 1). User device 106 is referred to below as a smartphone for simplicity of reference to this device, although user device 106 may include any type of device mentioned above with reference to Figure 1. In other cases, the user can navigate only with the use of the headset 108 (ie, eliminating the use of the smartphone).

As a general principle, system 102 exposes relevant information to the user at appropriate junctions along the path of the user, the user making their journey, or otherwise interacting with space. Examples of information that can be automatically presented to the user acoustically, visually, and / or haptically are referred to here as Articles of Interest (lOls). To perform this function, the system 102 determines the current context of the user at each moment in time. For example, system 102 detects the location and orientation (and, optionally, movement) of the user at each particular moment. System 102 may use any technology or combination of technologies to accomplish this task, examples of which are presented below in the context of the description of Figure 3. System 102 then identifies 10s that are relevant to the current context of the user. In some cases, for example, the system 102 may determine that an IOI is relevant to the user's current context because the user is within a certain distance of a physical object (or region) that is associated with the IOI, where that distance is defined and stored in advance. System 102 then provides information about those 10s to the user. As will be described in greater detail below, the user can also manually explore future contexts that he wants (or may) be confronted with, at later times in the journey, allowing the user to prepare for those situations.

System 102 may provide 10s based on information drawn from various sources. For example, system 102 can determine road locations, natural features, etc. from publicly available map information (such as map information provided by the BING map service provided by MICROSOFT Corporation in Redmond, Washington). System 102 can determine the locations of public and private entities (eg, stores, public buildings, etc.) from any published directory information. System 102 can determine the occurrence of relevant events from various published sources, such as public transportation schedule information, public safety information. System 102 may determine user-related information 10 from various services, such as one or more online social networking applications, one or more calendar applications, etc., and so on.

Each IOI refers to a particular topic or focus of expertise. 10s can be classified in different ways, along different explanatory dimensions. For example, first-class lOls directly map to physical objects or physical events in space through which the user is moving or otherwise interacting. For example, the IOI of this type may correspond to a warehouse that is in the vicinity of the user, or an open culvert that is in front of the user, or the next waypoint that the user is within a prescribed distance of arrival, etc. The lOls of a second class of lOls do not necessarily have a direct physical counterpart in the environment. For example, such an IOI may correspond to an advertisement that is played when the user approaches a bus stop. The ad may be associated with a space around the bus stop, but is not otherwise a description of the bus stop per se.

Another virtual 101 can correspond to a newspaper headline that attracts the user's attention as a coffee stand approaches. System 102 may present information regarding that IOI to the user based on the premise that a user may wish to consume that IOI while having a cup of coffee. Another IOI of virtual type can correspond to a weather report that is delivered to the user as the user leaves a metro station. System 102 may present information regarding that IOI to the user based on the premise that the user may wish to prepare for the weather that he is about to face when exiting the station. Another virtual-type IOI may correspond to a message retrieved from a social network application as the user approaches his personal residence. System 102 may present information regarding that IOI to the user based on the premise that the user may wish to catch up on all messages sent by family members or friends, before entering their home. Many other scenarios are envisaged in which the user receives some type of information (and / or the opportunity to interact with functionality) that is related to the current circumstance of the user, but could not serve to describe a real object or event in the vicinity immediate user.

Different lOls (either the first real class or the second virtual class described above) can also be classified based on their functions that they serve in space exploration. For example, LOLs can be classified along this dimension as Warning LOLs, Travel LOLs, Contextual LOLs, Available Information LOLs, etc. (although those categories are not, strictly speaking, mutually exclusive). The system 102 sends information about warning lOls to alert the user of events that may affect the safety of users during the journey (such as an indication that there is an open sewer in front of the user). The warning messages are sharp and to the point. The system 102 sends information about trip 10s to alert the user to events that may affect the progress of the user's trip (such as an indication that a next waypoint is approaching, or that a bus will be late). The system 102 sends information regarding contextual items to the user to alert the user of objects and events in the user's vicinity (either real or virtual) that may be of interest to the user (such as an indication that a coffee shop is facing the user on their journey). System 102 alerts the user to the existence of 10s of available information, without automatically delivering them. A user can then choose to receive this information in an on-demand form, or ignore it. For example, a user can access available information by touching an appropriate function button on headset 108 or through an information menu provided by user device 106, or using the equivalent voice command.

The system can deliver information about each IOI in any way. For example, for at least some 10s, the system can provide the IOI by sending a telltale sound, followed by a spoken message. Preview sound allows the user to tune in to hear the spoken announcement, that is, by drawing attention to the spoken announcement. The sound also alerts the user as to the type of information to be followed, so the user is in a better position to decide whether he or she is going to devote their attention to it. For example, users may choose to pay more attention to travel information, but pay less attention to contextual information (for example, regarding a warehouse in the user's vicinity, or a promotional offer pertaining to the user's current circumstance. ).

At any point in the journey, the user can make a request to hear the information about one or more trip lOls (and / or other types of lOls) that were recently read aloud. This request can be made in various ways which are described below. If a warning has been read aloud in the last 30 seconds, then the system 102 will respond to the action by the user by repeating the warning as well, followed by repeating the trip information. In some cases, the user may request that the system 102 repeat the information, but this information is no longer available. The system 102 can reproduce a sound suitable to alert the user of this situation, following a spoken message, the information is not available at the moment, or the like.

As another general feature, the system 102 may automatically present information about some 10s during hours when the user does not expect to be overwhelmed with other sensory information from the environment. For example, the system 102 may refrain from sending user messages regarding the planned route 202 until the user reaches a certain quiet zone along the planned route, in which the user can consume safely and enjoyably. information. The system 102 can store, in advance, the locations of the regions that are considered quiet zones.

Now consider more closely the particular experience of the user while navigating the space shown in Figure 2. Assume that, at time t ^ the user is trying to reach the second benchmark w₂. After detecting the user's location (and header), the system 102 can send the user directions (visually and / or audibly) that help the user reach the second set point w₂. In addition, the system 102 can send the user information regarding the second reference point w₂ automatically when the user is within a prescribed distance from the second waypoint w₂. This will allow the user to make adequate preparations for any changes assuming that the second reference point W₂ can carry. The information cited above can be formulated primarily as travel information.

In certain situations, the user's actual journey may deviate from the intended journey, for example with respect to the route he is actually taking and / or the time of the user's journey through the space. To deal with those situations, the system 102 may automatically (or on a demand basis) determine whether the user's current situation will affect any remaining portion of the user's journey. For example, the user's system 102 may update the user's estimated time of arrival at a bus station and then determine whether the user will continue to arrive in time to catch a previously identified bus or transportation. If the user's current situation impacts the user's journey in this way, the system 102 can automatically re-generate information that is used to help the user navigate within the space. That information can be expressed as a new set of 10s. For example, system 102 may advise the user to take a subsequent bus or transportation (and may automatically make reservations, send appropriate notifications, and / or make other arrangements). Furthermore, as noted above, the user can explore the updated information in any way, for example, by expressly requesting information regarding a reference point to be found in the future, etc.

Figure 2 generally also indicates that, at time ti, the system 102 automatically informs the user of the existence of various topics of interest (lOls) of a contextual nature (ie, in the terminology set forth above, contextual lOls). For example, an illustrative contextual IOI might correspond to a warehouse, restaurant, government office, natural landmark, and so on. In one case, the system 102 may present a contextual IOI for the user's consideration only if it is within a certain distance of the user's present time and the user has not already passed.

The user can customize the behavior of the system 102 in any mode (with respect to contextual lOs and / or any other type of IOI). For example, the user can specify the types of messages that they want to receive along their route, for example, indicating that they would like to receive information about a first type of store, but not a second type of store. The user can also specify the moment when he would like to receive the information. For example, the user can specify the maximum distance (from his current location) that should be considered when notified of the existence of contextual lOls. The user can make these adjustments before starting the journey. In addition, the user can dynamically change the type and amount of information provided by the system during the course of the trip, for example, by checking the amount of information that is automatically provided by the system. The user can choose to reduce the amount of information because he considers it unnecessary or distracting at a certain point along the journey.

System 102 may present information about contextual items in any application specific form. For example, in one case, the system 102 may advertise a collection of contextual lOls in the order in which they appear in front of the user, from left to right or from right to left (for example, essentially forming an angular sweep, with the user placed at the origin of the sweep), or front to back (in terms of distance from the user), or back to front, etc. As noted above, in one implementation, the system 102 may precede each announcement with a telltale sound that alerts the user that information regarding a contextual IOI is to follow. System 102 may then describe the contextual IOI, for example, by providing audio information that announces, Jane's Cafe, 45.72 meters. System 102 may alternatively provide different preliminary sounds associated with different types of establishments, such as by providing a first type of preliminary sound for restaurants, and a second type of sound for bus stops, etc.

In some cases, there may be a large number of contextual lOls within the user's vicinity at a current time. To reduce the amount of information provided to the user, the system 102 may consolidate this information into one or more summary messages, such as posting, Restaurants, generally at 30.48 meters. The same is true for any type of IOI. In general, the system 102 can determine whether a group of individual 10s to be delivered to the user at a certain time have at least one common characteristic. System 102 may then provide a summary message that advertises the group of 10s as a block or set, rather than individually. In the specific example noted above, the system 102 determines that a group of contextual 10s are located in the vicinity of the user and that these 10s belong to the same type of establishment. In another example, system 102 may provide the summary warning IOI, numerous potholes in the next 91.4 meters.

System 102 may use other rules to intelligently serve advertisements. For example, some lOls are only relevant if the user is in close proximity to these lOls. For example, a user may be interested in a park bench only if the user is within a few meters of this object. Therefore, the system 102 can announce the presence of these types of objects only when the user is relatively close to these objects. But here again, the user can customize the behavior of the system in this regard.

In accordance with another illustrative feature, the system 102 may use three-dimensional sounds to announce the presence of some types of lOls, such as some types of contextual lOls. A three-dimensional sound refers to a sound that the user perceives as coming from a particular location (or locations) within a physical space. In reality, however, the audio information is presented to the user through earpiece 108 and has no physical origin within the environment. As described below in Section D, the system 102 can achieve the above result by using Header Related Transfer Functions (HRTF), along with different types of wideband audio sounds. In other implementations, the system 102 may use other techniques and technologies to create a three-dimensional audio effect (instead of HRTF or in addition to HRTF), such as Ambiophonics, Ambisonics, wave field synthesis, etc.

For example, in the context of Ct (at time ti), the system 102 may send a series of directional advertisements to the user that identify at least three contextual lOls (lOh, IOI₂, IOI₃). System 102, for example, may announce the presence of contextual IOh by sending a preliminary sound to the user that the user perceives as originating from a particular location in the space in which the entity associated with the contextual IOI is physically located. That is, if the contextual IOI corresponds to Jane's Cafeteria, the message could direct the user's attention to the actual location of Jane's Cafeteria. The system 102 may also provide the description of this establishment by a three-dimensional sound, such as announcing Jane's Cafe, 45.72 meters in a directional manner, that is, using three-dimensional spoken information.

Additionally, or alternatively, the system 102 may send the user information about one or more 10s of available information. As explained above, the system 102 can perform this operation by sending a message that generally alerts the user to the existence of information that may refer to the user's current context, but without immediately announcing the particular substance of that information. The user can then manually request that the system 102 receive the information, for example, by performing a more information instruction. Or the user can ignore the information.

All of the above information delivery features contribute to the goal of delivering useful information to the user when the user walks through a space, without overloading the user with too much information.

As another feature, at any time along the user's route, the system 102 can compare the user's actual direction of travel with a desired direction of travel. System 102 can determine the actual address of the user using any of the mechanisms described below in relation to Figure 3. System 102 can determine the desired address by determining where the user is expected to be headed by the moment. System 102 can then determine deviation information that determines the extent to which the user's actual direction deviates from the desired direction. System 102 may send user information attempting to guide the user along a desired path.

For example, consider the user at time t₂. At this point the user is trying to get to the waypoint w₃. Assume that the user's actual direction is represented by arrow 210, and the user's desired direction is represented by arrow 212. System 102 will send user instructions that attempt to direct the user away from their current errant direction, towards the direction desired.

In one implementation, the system 102 achieves the above objective using a three-dimensional heartbeat or other type of periodic sound. Three-dimensional heartbeat sound is any type of repetitive sound (such as a clicking sound) that is perceived by the user as coming from a particular location (or locations) in physical space. In the case of Figure 2, at time t₂, the system 102 will emit a three-dimensional heartbeat sound that appears to come from the user's left along their direction of travel. This will indicate to the user that: (a) he is heading in a non-optimal direction; and (b) that it should turn slightly to the left to obtain a more desirable trajectory.

System 102 can modulate the beat sound to achieve other effects. For example, system 102 may change the pitch and / or periodicity and / or volume (and / or some other aspect) of the heartbeat sound depending on the degree to which the user is currently directed in a non-optimal direction. The user will interpret the heartbeat sound as an indication of the extent to which he is heading in the wrong direction.

Until now, the description has primarily emphasized the ability of the system 102 to automatically provide the information to the user at appropriate junctions along the user's path. In this mode of operation, the user's encounter with the environment can constitute the events that trigger the delivery of information, for example, in the form of the different types of lOls previously described. In addition, at any time, the user can manually interact with either the smartphone or the headset 108 to manually scan their environment and to obtain information on any of the above described 10s. For example, in a first form of interaction, the user can tap on a touch-sensitive surface of the smartphone at any time during the journey. In response to a single touch, the system 102 will announce higher level information regarding the current context of the user. For example, at time h, the system can respond to a single touch announcing that the user is directed to the reference point w₂, which can be considered as a travel IOI. In response to a double tap, the system 102 will provide more detailed information about the current context. In response to a triple tap, the system 102 will provide instructions that allow the user to interact with the smartphone, for example, in order to obtain additional information about the context and to invoke various functions related to the current context.

As another feature, at any time, the user can interact with the smartphone to activate one or more menus that are relevant to the user at any given time. For example, at time b, the user can perform a touch and hold gesture on the surface of the smartphone. In response, the system 102 may activate a menu associated with the current context of the user. The user can then interact with the smartphone to explore the immediately presented menu or navigate to any other menu.

More specifically, as will be explained in detail in Section C that follows, the system 102 may represent a collection of menus that are accessible through a set of workspaces. Each workspace is in a fixed position relationship with the other workspace. The user can access the desired information and / or functionality by navigating to a suitable workspace and associated menu.

According to one feature, the user can perform all the above screen interaction tasks with one hand, for example using the thumb of the hand holding the smartphone. System 102 may also provide audible and / or haptic feedback signals as the user interacts with the touch-sensitive surface of the smartphone. Collectively, all of these features reduce the degree to which the user needs to divert their attention from the environment while interacting with the system 102. For example, the user can keep his eyes on the road he is walking on while interact with the smartphone. A person with visual impairment can also successfully interact with the system 102 due to the non-visual characteristics outlined above.

As described in Subsection C.3, the user can also manually interact with the system 102 through voice instructions. Additionally, or alternatively, the user may manually interact with the system 102 through input mechanisms provided by the handset 108 (as described in Section B). System 102 may provide other mechanisms by which the user can manually interact with system 102.

The user can also invoke special modes to be used in exploring their immediate environment. For example, in response to activation of a scan mode, system 102 may determine the current focus of the user's attention, which may correspond to the direction in which the user is presumed to be looking at the current time (which, in turn, it may be determined on the basis of one or more orientation determination mechanisms). System 102 can then determine the contextual 10s that are encompassed by (or otherwise associated with) a subspace formed around the user's attention direction. The system 102 can then read these contextual lOls by announcing these contextual lOls using three-dimensional sounds, and / or by sending visual messages for presentation on the user's smartphone, etc.

Some of these contextual lOls may refer to real objects in the environments with respective real positions within the subspace. These lOls serve primarily to identify or mark the locations of these physical entities in physical space. Other contextual lOls can be virtual in nature, since they belong to the subspace (given the current context of the user), but they cannot describe the objects directly in that subspace. In other words, these other contextual lOls convey information and / or an experience that is related to the current context of the user, beyond the identification of the locations of physical entities.

To cite an example, the contextual IOI of the virtual variety may correspond to a message, Remember our fallen troops as the user approaches a military barracks. That message is related to the barracks, but cannot be accurately estimated to simply mark the location of the barracks. The intention of this message is to create a mental association to enrich the user experience when approaching the barracks. Alternatively, the contextual IOI in that circumstance may correspond to a song or a fragment of an inspirational speech or a personal message that has a special meaning for the user (as previously specified and uploaded to the system 102 by the user).

The same is true for other types of lOls (except contextual lOls). That is, some lOls serve primarily as marking labels, while other lOls are intended to stimulate cognitive associations, memories, emotions, etc. The last group of lOls is referred to as virtual here, in the sense that they belong to a realm of associations, experiences, meanings, etc., which is not a surface-level transcription of events and objects in the environment. Such 10s could alternatively be referred to as inference, suggestive, relational, etc.

According to another feature, in response to activation of an orientation mode, system 102 can perform a full 360 scan around the user to identify all items of interest that are associated with a prescribed distance from the user. The system 102 can also perform this 360 degree scan for successive levels in the vertical dimension, for example, to determine the stores at different levels of a shopping complex or the like. The user can customize the behavior of the browse mode and the orientation mode in any way described above, for example, by changing the types of lOls identified, the dimensions of the space in which the presence of lOls is searched, and so on. . In addition, the user can interact with the system 102 to govern how the system 102 reads the 10s.

Now assume that, at time t₃, the user spontaneously decides to deviate from the planned route 202 to visit the warehouse 204, for example, to buy a sandwich. When in store, the context (c₃) of the user at this time belongs to a warehouse environment. Therefore, the system 102 can perform the same functions as previously described, but now in the context of the interior environment of the warehouse. For example, the system 102 may automatically determine contextual lOls as the user traverses an island of warehouse 204 and announce contextual lOls to the user. For example, after approaching the dairy section of warehouse 204, the user may receive a message that reads, milk, cheese, and yogurt, 6.09 meters ahead. The system 102 can send more detailed information progressively as the user approaches the products that interest him. Again, some of the contextual lOls may have a less direct correspondence with physical objects in the warehouse, for example, as in a message delivered in the soup section, which alerts the user to the presence of high levels of sodium in many soups.

Also the user can manually interact with the system 102 within the warehouse environment in any way. For example, the user can manually browse the different menus associated with the different products. The user can also use the smartphone to perform various transactions in the warehouse environment, such as buying an item, researching an item, etc.

In some implementations, system 102 may determine the user's location within warehouse 204 by determining if the user is within range of a set of beacons, which have been placed (in advance) at different locations within warehouse 204. As will be described in Section E, beacons may have non-overlapping ranges.

Upon exiting warehouse 204, system 102 can recalculate the user's journey to bring the user back to planned route 202. For example, system 102 can provide the user with instructions that allow the user to get to waypoint w₄, associated with a transportation point. Upon reaching that set point, the system 102 may then provide information that is relevant to the user at the time, such as by announcing the expected time of arrival of the user's transport 208. That information can be delivered as one or more trip logs.

The system 102 can continue to provide services to the user while he is traveling on the transport 208. For example, the system 102 can notify the user of the expected time of arrival at the final destination, that is, the waypoint w₅ (the user's medical office). System 102 may also provide other messages that may be of use to the user, depending on the nature of the public (or private) transportation that the user is traveling on. For example, when traveling on a bus and approaching a final destination, the system 102 may alert the user to the existence of a high curb that it expects to encounter when exiting the bus. In addition, the system 102 may, with the user's permission, alert the bus driver that a person requiring assistance will get off at an upcoming bus stop.

In short, throughout the user's journey, the user can receive a large amount of information in audible form, for example, in the form of spoken messages, other sounds (three-dimensional sounds and non-three-dimensional sounds), etc. System 102 may use various techniques to manage the presentation of this information, some of which have already been mentioned above (such as the ability to redial or dial on the amount of information that is delivered). This feature allows the user to receive the most relevant information in a timely manner, without burdening the user with too much information.

For example, the system 102 may reproduce sounds during a journey subject to different standards to address the situation where the delivery of one sound potentially interferes with the delivery of another sound. In accordance with an illustrative rule, the system 102 will reproduce the heartbeat sound in a continuous loop to guide the user in a desired direction, for example, while walking.

The system 102, however, can temporarily disable this sound (or reduce the volume of this sound compared to a normal state of this sound) when any other sound is being played. This allows the user to hear the other sounds without interference from the beat sound, which is considered a low priority sound.

In accordance with another illustrative rule, the system 102 may unconditionally reproduce sounds that represent events related to the interface, such as scroll gestures that change the menu or context that is presented to the user. To avoid overloading the user with too much audio information, these types of sounds can be short and distinct. A user can control the playback of these types of signals, at least to some extent, by temporarily suspending their interaction with system 102 (because no interaction signal will occur if the user is not actively interacting with system 102).

In accordance with additional illustrative rules, the system 102 may prioritize navigation sounds by assigning the highest priority level to warning sounds (for example, for warning lOls), the next highest priority level for travel information ( eg for travel lOls) and the next level above contextual information (eg for any kind of contextual lOls). In some cases, the system 102 will delay the delivery of information, for example, because the most important information is being reproduced. Furthermore, in some cases, a delayed message will no longer be relevant by the time the system 102 is able to present it (for example, because the user has moved to a new context in which the information is no longer relevant); if so, system 102 may refrain from presenting that information.

In conclusion, the scenario described above is also useful to highlight some of the advantageous technical effects of the system 102. Generally, the system 102 allows any user to receive guidance that serves different but related purposes. First, the system 102 attempts to expose the user to the most useful information at any given time along their journey, thereby empowering the user to more effectively navigate within their environment, or to achieve other goals. Second, the system 102 enriches the user's experience of the environment beyond providing navigational assistance, allowing the user to learn new information about the environment that may not be immediately apparent to the user, without the use of the system 102; In this sense, the system 102 allows the user to metaphorically penetrate below the surface of the environment to understand previously hidden aspects and connections that refer to the environment 102. Third, system 102 attempts to provide this useful information to the user in a manner that minimizes distractions placed on the user. The third objective is useful to provide a more pleasant and useful experience for the user, for example, allowing the user to keep the primary focus on their interaction with the real world, and not on the tools that he or she is using to interact with it. real world. Put negatively, the third objective tries to reduce the level of anxiety that can occur when asking the user to interact with a cumbersome and complex tool, in which the user would have to devote a great deal of attention. The third objective also allows the user to quickly and efficiently access the desired information in a secure manner, without feeling overwhelmed at any given moment with too much information.

A number of technical characteristics contribute to the objectives outlined above, particularly with respect to the third objective. Features include, but are not limited to: a) the use of a one-handed interaction experience; b) the use of a condescending menu structure with the user adopting gestures that can be performed on a touch-sensitive surface in a location-agnostic manner (described below); c) the use of a simple, easy-to-learn workspace structure that provides access to a safe home workspace (described below); d) the use of multiple mechanisms to enter commands (eg, via headset 108, user device 106, voice recognition, etc.); e) the use of audio information and / or haptic signals to transmit information without unduly disturbing the user's focus on the journey; f) the use of three-dimensional and non-three-dimensional sounds to help guide the user in a desired direction without inundating users with complex instructions, or alerting the user to the location of the lOls, and so on.

The aforementioned advantages apply to any user of the system 102. The system 102 can also be used successfully by people with any type of condition that hinders their ability to travel. These users can include users with partial or total vision loss, users with cognitive or psychological disabilities, users with mobility-related disabilities, and so on. For these users, the system 102 acts as a virtual guide dog, assisting the user at each stage of their journey in a safe way or, otherwise, assisting the user in interacting with their environment, whatever the user's objective. at the time. For these users, in addition to the general benefits outlined above, system 102 also allows the user to access information and guidance that would not otherwise be available to them, thus potentially improving the mobility, confidence, and overall quality of life of the user. these users.

Moving now to Figure 3, this figure shows a high level overview of computing functionality that can be used to implement the system 102 of Figure 1. The functionality of Figure 3 is shown in device agnostic form. In a real implementation, the functionality can be assigned, for example, to any of the components introduced in Figure 1, or any combination of these components. For example, Figure 3 shows that the functionality includes a space interaction module (SI) 302. The SI module 302 performs all (or most) of the functions described in relation to the scenario in Figure 2. Some parts of the module SI 302 can be implemented by the user device 106, while other parts of the SI 302 module can be implemented by processing the components located in the handset 108. In addition, or alternatively, some parts of the SI module 302 can be performed by the remote processing resources 112.

The SI module 302 can receive input information from any combination of input mechanisms 304 and can provide its output information for display to any output mechanisms 306. For example, the 10 input mechanisms 304 may include one or more orientation determining mechanisms 308 and / or one or more movement determining mechanisms 310 and / or one or more position determining mechanisms 312, and so on.

The orientation determination mechanism (s) 308 determines the orientation of the device incorporating these mechanisms 308.

For example, if it is housed by the user device 106, the orientation determination mechanism (s) 308 determines the three-dimensional orientation of this user device 106. If it is housed by the earpiece 108, the orientation determination mechanism (s) 20 orientation determines the three-dimensional orientation of the handset 108.

More generally said, the orientation determining mechanism (s) 308 can determine the direction the user is pointing their smartphone or turning their head (over which the earpiece 108 is placed). Movement determining mechanism (s) 310 determines the nature and degree of movement of the device incorporating this mechanism (s) 310. The position determination mechanism (s) 312 determines the absolute and / or relative position of the device incorporating this mechanism (s) 312.

The mechanisms (308, 310, 312) can be implemented using any combination of sensors, including but not limited to: magnetometers, accelerometers, gyroscopic sensors, gravity-based sensors, torque sensors, strain gauges, bending sensors, optical encoder mechanisms, and so on. In addition, some of the mechanisms (308, 310, 312) can receive signals from external systems or sources. For example, mechanisms (308, 310, 312) may include a sensor for determining the position of a device based on signals received from a satellite-based navigation system (eg, a Global Positioning System (GPS)). In addition, or alternatively, the mechanisms (308, 310, 312) may include functionality to determine the position of a device by triangulation and / or other processing based on signals received from a plurality of external sources, such as signals received from multiple radio towers and / or localized directional antennas, etc. Additionally, or alternatively, the mechanisms (308, 310, 312) may include functionality to determine the position of a device using dead reckoning technique. In addition, or alternatively, the mechanisms (308, 310, 312) may include functionality to determine the position of a device when processing information from local beacons (eg, Wi-Fi and / or BLUETOOTH beacons, etc.), and so on.

The input mechanisms 304 may also include any combination of manual input mechanisms 314. These mechanisms may include any of the following: key input mechanisms, touch-sensitive input mechanisms (eg, a touch-sensitive screen 314 '), joysticks, microphones (for example, to receive voice instructions), video cameras and / or depth cameras (for example, to receive free space gestures), and so on. For example, in the case of Figure 1, user device 106 may use a touch-sensitive display screen as its primary form of interaction with the user. For example, without limitation, such a touch sensitive display screen may incorporate a capacitive touch screen mechanism that determines when a user touches and / or hovers over the screen. User device 106 can also include a camera, microphone, etc. Headphone 108 may include a microphone (for receiving voice instructions) along with one or more dedicated input mechanisms, for example, as implemented as buttons on the side of handset 108 (described in greater detail in the next section).

The output mechanisms 306 may incorporate one or more audio output mechanisms 316, one or more display output mechanisms 318, one or more haptic output mechanisms 320, and so on. For example, the audio output mechanism (s)

316 it can correspond to conventional speakers of any type.

Additionally, or alternatively, the audio output mechanism (s) 316 may incorporate bone conduction audio devices (eg, as provided by the earpiece 108), such as 5 bone conduction transducers produced by AFTERSHOKZ,

LLC, of ​​Syracuse, New York. The display output mechanism (s) 318 may correspond, for example, to a type of LCD screen (eg, as provided by user device 106). Haptic output mechanism (s) 320 may correspond, for example, to a vibration-producing mechanism (eg, as provided by user device 106 and / or earpiece 108, etc.). A vibration producing mechanism can achieve a vibration effect by using a rotating off-balance weight, and / or by some other mechanism (s).

The SI module 302 can also interface with remote functionality 322 which can be considered external to the SI module 302 per se. For example, the SI module 302 may interact with a search engine 324 for the purpose of performing a search. For example, search engine 324 may correspond to search engine BING provided by

MICROSOFT Corporation in Redmond, Washington. Additionally, or alternatively, the SI module 302 can interact with a trip calculation engine 326 for the purpose of generating a new trip and / or modifying an existing trip. Additionally, or alternatively, the SI module 302 may interact with a speech processing engine 328 to interpret spoken instructions made by the user. For example, the speech processing engine 328 may correspond to the CORTANA system provided by MICROSOFT Corporation of Redmond, Washington. In other cases, one or more aspects of remote functionality 322 may be incorporated into the SI module 302 as native resources.

In addition, the SI module 302 can interact with any external information provided in one or more data stores 330. For example, the external information can provide publicly accessible map information, transportation schedule information, alert information, business directory information. and personal information, social network information, calendar information, etc., and so on. In some cases, the SI module 302 may interact with external sources (eg, external web sites) through application programming interfaces (APIs) provided by those external sources.

Referring now to Figure 4, this figure shows an implementation of the SI module 302, which was previously presented. From a high-level point of view, the SI module 302 can include (or can be conceptualized as including) a plurality of sub-components that perform different respective functions. Furthermore, some sub-components can make use of the results generated by other sub-components. An application interface module (AIM) 402 allows the user to interact with any of the sub-components. For example, the application interface module 402 can provide menu functionality that exposes various functions provided by the subcomponents.

In general, referring to the subcomponents from top to bottom, the SI module 302 may include various data stores to store the information that can be used by the SI module 302 in the performance of its functions. For example, a data store 404 may store the information that defines one or more trips. For example, the trip information can describe the landmarks on the trip, and / or any other information related to the trip. A data store 406 can store search results provided by a search engine; These results can be produced, after the direction of the user, during the course of the trip or a more general interaction of the user with a space. A data store 408 may store a history of tabs created by the user in the course of the user's interaction with the SI module 302. A tab generally corresponds to a bookmark for a menu or other information and / or functionality and / or option item. A user can create a tab when he or she visits that menu or other item or information and / or functionality and / or option; In one implementation, the data store 408 does not initially contain tabs when a user embarks on a journey and has not yet begun to interact with the system 102. System 102 may use any way to represent a collection of tabs, for example, as a list of tabs, a radial menu of tabs, and so on.

The SI module 302 can also provide various support modules 410 that perform any type of support services. For example, a configuration module can allow a user to assign a value to any parameter that affects the operation of system 102. A data store 412 can store all of these configurations. Support modules 410 may also include a portal for interacting with an external search engine to provide search results. Or the support modules 410 may include a natively provided search engine.

A sound generation module 414 performs various operations related to sound generation. For example, the sound generation module 414 can reproduce particular sounds when various triggering circumstances are encountered. Some triggering circumstances correspond to actions performed by the user when interacting with the application interface module 402. Other triggering circumstances correspond to changes in the state of the system 102 that are not directly caused by the user's interaction with the system 102. Other triggering circumstances correspond to events that occur during the course of a trip (or, more generally, the interaction with a space), and so on. A data storage 416 can store files that, when played, produce the desired sounds. Subsection C.2 (below) provides additional information about different types of sounds that can be generated, and the circumstances under which these sounds are reproduced.

Some sounds are non-three-dimensional or non-spatial in nature. In addition, the sound generation module 414 can produce three-dimensional audio information. As described above, audio information is three-dimensional, in the sense that a user will perceive this information as coming from one or more locations within a three-dimensional physical or virtual environment. The sound generation module 414 can create these sounds by transforming the original sound information using one or more Header Transfer Functions (HRTF).

Also, a haptic signal generation module 418 can produce different types of haptic feedback experiences under different triggering circumstances. In one case, haptic signal generation module 418 produces signals that produce vibration signals, for example, delivered to user device 106, headset 108, and / or some other device.

A path guide module 420 uses the sound generation module 414 to generate the above-described three-dimensional periodic sound (eg, beat). The purpose of this periodic sound is to guide the user in a particular direction. Path guidance module 420 produces this effect by determining the user's actual and current header, the desired header, and the difference between the current and desired headers (corresponding to offset information). Next, the path guide module 420 makes use of the sound generation module 414 to produce a suitable looping sound that is perceived by the user as coming from a particular direction. That is, the sound generating module 414 produces the loop sound based on the deflection information fed to it by the path guide module 420. The user can respond to this sound by moving in the desired direction. In another case, the heartbeat sound can be perceived as it travels through a series of locations in physical space. The user can interpret this experience as an instruction to move in the direction that the sound travels.

A beacon-based guidance module 422 provides assistance to the user in navigating within an interior and / or exterior space by detecting signals that are emitted from a collection of beacons having, in an illustrative implementation, respective non-overlapping ranges. . Section E provides additional information about the operation of the beacon-based guidance module 422. Beacon-based guidance module 422 may query beacon information provided in data storage 424. Beacon information may describe the codes associated with beacons that have been placed in the environment and their respective locations within the environment.

A relevant information determining module 426 (RID) performs the general function of determining the relevant information to present to the user at any given time. In the context of the description of Figure 2, the RID module 426 determines the various types of information items (10s) that affect the current context of the user. To accomplish this task, the RID module 426 receives various contextual inputs that define the context of the user at the current time. These contextual entries can describe the user's current location, the user's current header, the user's current goals, and so on. Contextual input can also describe the environment itself, such as objects and events associated with the environment, both physical and virtual. Any of these entries can be extracted from map information, directory information, social network information, calendar information, etc. Contextual inputs can also describe environmental factors that affect user interaction with space, such as information about public transportation, weather information, etc., as obtained from any source (s).

The RID module 426 operates by determining whether it is appropriate to notify the user of any information at a given time, based on contextual inputs at the current time, and based on various rules (provided in a data store 428). The behavior of the RID module 426 may also be defined by one or more of the parameters set by the user and stored in data storage 412. For example, the RID module 426 can determine if there are any contextual 10s in proximity to the user at the current time, based on a user-defined depth range. If such contextual lOls exist, the RID module 426 may interact with the sound generation module 414 and / or menu functionality provided by the application interface module 402 to notify the user of these contextual lOls.

A scanning module 430 and an orientation module 432 perform respective services that can be invoked in a manner on demand by the user. As described with reference to the scenario in Figure 2, the scanning module 430 determines any contextual lOls that are associated with a subspace that is in front of the user (which, in turn, can be determined by the position of the user's head , along with a configuration parameter that defines the depth of the investigation, and a configuration parameter that describes the span of the search space). To accomplish this task, the scanner module 430 makes use of the services of the RID module 426. The scanner module 430 then notifies the user of the contextual lOls using three-dimensional sounds, displayed messages, and so on. Orientation module 432 performs a similar task to scanning module 430. But instead of investigating 10s associated with a subspace that is projected in front of the user, the orientation module 432 can scan all of the three-dimensional space that exists around the user at the present time.

Figure 5 shows an implementation of the application interface module 402 (AIM). As noted above, the application interface module 402 generally provides an interface that allows the user to interact with the various sub-components described above, with reference to Figure 4.

Application interface module 402 may include various components that interpret user input, for example, to determine the nature of an instruction the user is making. For example, a gesture interpretation module 502 may determine a gesture that the user has made when interacting with the touch screen of user device 106, or a free space gesture, etc. The gesture interpretation module 502 can accomplish this task by comparing the marks or touches or appearance of float (or other behavior) performed by the user with a data store that identifies patterns associated with known gestures. If the user's behavior matches a pattern associated with a particular gesture with a certain matching degree of confidence, then the gesture interpretation module 502 can conclude that the user has performed that gesture.

A voice interpretation module 504 can interpret instructions spoken by the user, for example, that can be received through a microphone on the user device 106 and / or the earpiece 108. In one case, the voice interpretation module 504 may correspond to a portal for the remote speech processing engine 328 (eg, the CORTANA system). In another case, the voice interpretation module 504 can correspond to any type of native functionality for interpreting spoken expressions. In any case, the agent performing the speech recognition can use any technology to carry out this task, such as technology based on the Hidden Markov Model, technology based on neural networks, etc.

A headset button interpreter module 506 interprets how the user interacts with the input mechanisms of headset 108 (described below). For example, in some cases, a set of buttons can perform different functions depending on how the user interacts with them, for example, depending on whether the user touches a button but does not press it, or depending on whether the user user taps and releases the button one or more times, or depending on whether the user presses and holds the button, etc. The headset button interpretation module 506 delineates user behavior for a particular instruction.

An action take module 508 may invoke an action based on the interpretations provided by the interpretation modules described above (502, 504, 506). For example, in response to the interpretation, the action take module 508 may: invoke a menu; close a menu; transition between workspaces (described later); perform a function; save a configuration; submit an information article, and so on.

B. Headset and User Device Options to Facilitate Interaction Between Users and Their Environments

Figure 6 shows additional details about the headset 108 to the user device 106 and the other user computing device 110 introduced earlier, in the context of Figure 1. The characteristics of these devices are presented here in the spirit of illustration. , and not limitation.

Referring first to earpiece 108, this device can provide a frame 602 made of plastic and / or metal and / or any other material. Frame 602 may be flexible and may be attached to the user's head through tension on the frame that pushes inwardly laterally against the user's head, and / or by some other attachment mechanism. Headphone 108 includes transducers (604, 606) that transmit vibrations to the bones of the wearer's head; The bones in the user's head then transfer these vibrations to the user's eardrums, where they are perceived as audio information. The use of a bone conduction type headset prevents the headset 108 from obstructing the user's ear canals, thus allowing the user to respond correctly to other sounds in the environment. However, alternatively, earpiece 108 may include conventional speakers that are placed on or near the ears of the user.

Headphone 108 may also optionally include a set of input mechanisms 608 anywhere on its frame. The user can interact with the input mechanisms 608 with one or more fingers of his hand 610 (or hands). Alternatively, a separate device (not shown) may provide the input mechanisms 608 and that other separate device may communicate with the handset 108 via wireless communication (eg, BLUETOOTH communication) and / or wired communication. Figure 6 shows that the entry mechanisms 608 include three buttons, but, more generally, the entry mechanisms 608 can include any number of mechanisms and these entry mechanisms can be placed on the frame in any way. In addition, the input mechanisms 608 may include other types of input mechanisms in addition to buttons, such as knob or wheel mechanisms, sliders, etc. Alternatively, or in addition, the earpiece 108 may incorporate one or more touch sensitive surfaces. For example, different regions of the earpiece 108 may incorporate different touch-sensitive mechanisms, and those regions may be associated with different respective functions.

The handset 108 may include processing mechanisms that perform various tasks (described below). The handset 108 may include a compartment 612 that houses those processing mechanisms. For example, in one case, compartment 612 is located at the rear of headset 108. However, the processing mechanisms can be physically located at any location (or locations) on headset 108. The processing mechanisms themselves may include one or more processing devices of any type, memory, etc., and / or dedicated logic components, for example, one or more application-specific integrated circuits (ASIO), etc.

Input mechanisms 608 may initiate any of the operations when activated. For example, without limitation, the user may use input mechanism 608 to instruct SI module 302 to invoke scan mode or orientation mode (described above). Additionally, or alternatively, after listening to a summary of a given topic (eg, the name of a contextual IOI), the user can use input mechanisms 608 to instruct the SI module 302 to provide additional information on the identified topic; This instruction can be viewed as a more information instruction.

Additionally, or alternatively, the user may use input mechanisms 608 to instruct the SI module 302 to activate a listening mode, or to stop the listening mode. In listening mode, voice interpretation module 504 processes the user's voice to determine if the user has spoken an instruction.

Additionally, or alternatively, the user may use input mechanisms 608 to instruct SI module 302 to repeat the most recent audio message that has been provided. Additionally, or alternatively, the user may use input mechanisms 608 to request the SI module 302 to repeat a set of previously delivered audio messages, for example, starting with the most recent message and moving backward in time, message by message. , for a predetermined number of previous messages. Such an instruction can be referred to as a rewind instruction.

Alternatively, or in addition, the user can utilize input mechanisms 608 to turn the three-dimensional heartbeat sound on or off, and so on. Other implementations may use input mechanisms 608 to issue other instructions and / or skip one or more of the instructions discussed above.

The above functions, or some subset of them, can be assigned to any number of respective buttons and / or other input mechanisms in any way. In addition, in some implementations, the system 102 may include a customization module that allows the user to define the mapping between the input mechanisms and the operations that the mechanisms invoke. Furthermore, as described above, the same button can also perform two or more functions, depending on the way in which the user interacts with it. In one implementation, the SI module 302 can announce the function performed by a button when the user touches it, but does not press it.

According to another feature, the user can deactivate a function that is currently in the active state by pressing the same button that was used to activate it. For example, the user can stop the SI 302 module from announcing the information by pressing the same button that was used to request that the SI 302 module deliver this information, etc. Alternatively or in addition, the handset 108 may incorporate a dedicated button that stops whatever function is being performed.

The user device 106 can correspond to any of the types of portable devices described above, such as a smartphone or a tablet-type computing device. As described above, a user can interact with user device 106 using a single hand 614 (and / or optionally with two hands). The other user computing device 110 may correspond to any type of traditionally immobile computing device, such as a workstation computing device, a game console, a cable tv box device, etc.

One or more communication paths 616 may couple the user device 106 with the headset 108. Such a communication path, for example, may correspond to a BLUETOOTH communication path, a hard-wire communication path (eg, a USB communication), etc. One or more communication paths 618 may couple the other user computing device 110 to the user device 106 and / or the handset 108. As described above, one of the reasons a user may wish to establish a connection between the other user computing device 110 and the user device 106 and / or the headset 108 is to upload information to these devices. The communication paths 618 may be implemented in any way, for example, through any type of wireless and / or wired communication path described above.

Other implementations of the system 102 may use different types of headsets, for example, compared to the particular type of headset 108 shown in Figure 6. For example, in another implementation, a headset may incorporate some of the features identified above (including 608 input mechanisms) along with a head-mounted display device (HMD) of any type, for example , such as that physically implemented as lenses (eg glasses), helmets, etc. System 102 can display any type of information using the head-mounted display device. For example, system 102 may display any of the menus and other information described in the next section (Section C) through the head-mounted display device, or some subset thereof. In another case, the system 102 can display computer generated information that is mixed with the information associated with the real world with which the user interacts, thus offering an augmented reality experience. For example, system 102 may display descriptive labels in positional proximity to associated objects and events within the user's field of view. In another case, the system 102 may display directional cues that help the user to move in a recommended direction, and so on. In addition, system 102 can modify the type of information that is displayed to accommodate any visual impairment that may affect a particular user, for example, by displaying expanded and simplified information for visually impaired users.

System 102 can achieve an augmented reality experience using any technology, for example, by using an optical mixer that displays computer-generated information about the user's direct visual perception of the real environment (for example, using partially reflective mirrors or similar). In another implementation, the system 102 may use a video camera to capture the actual environment, along with an optical mixer that mixes video information from the video camera with computer generated information. In both cases, the system 102 can determine the user's presumed field of view using one or more devices that detect the user's location within the environment, one or more devices that detect the position and orientation of the user's head, and / or one or more devices that detect the user's gaze direction, etc.

Otherwise, another type of portable device, in addition to a headset, or in addition to a headset, can perform any of the functions listed above. For example, the wearable device may correspond to a device mounted on the wrist, an article of clothing, etc.

To simplify the explanation, the following description will assume that the user interacts with the headset 108 of the basic type shown in Figure 6, although, to repeat, the headset 108 may incorporate any number of ancillary functions mentioned above (such as a display mounted on head). Also, another type of portable device can be used in place of a headset, or in addition to a headset.

Figure 7 shows one way to implement the system 102 of Figure 1. In this implementation, the system 102 makes use of a user computing device 702 (a user device for short) and a headset 704. More specifically, of According to a function assignment, user device 702 implements most of the functionality of system 102. That is, user device 702 includes SI module 302 and a collection of input / output mechanisms 706, including any orientation, movement, and / or position determination mechanisms 708, any touch-sensitive input mechanisms 710, any mechanisms haptic output 712, any display output mechanisms 714, and so on. Examples of these mechanisms are provided above in connection with the description of Figure 3. User device 702 also includes a power source 716 (eg, a battery) and one or more communication mechanisms 718 for communicating with handset 704 .

On the other hand, the headset 704 includes only one or more audio output mechanisms 720 (such as a bone conduction audio mechanism), a power source 722 (such as a battery), and one or more communication mechanisms 724 for communicate with user device 702. Additionally, the handset 704 may include any of the types of input mechanisms 726 described above with reference to Figure 6 (eg, corresponding to the input mechanisms 608 of Figure 6). Communication mechanisms 724 can transmit instructions, invoked by input mechanisms 726, to user device 702. User device 702, in turn, can send the audio information to handset 704 for display by audio output mechanism (s) 720.

Figure 8 shows another way to implement system 102.

Here, the system 102 includes a headset 802 that has more processing power compared to the headset 704 in Figure 7. In fact, in one implementation, the headset 802 can perform all aspects of system 102 space interaction. , without the use of any separate user device 106. In another implementation, headset 802 can continue to interact with a separate user device 106 (not shown in Figure 8). That separate user device 106 may include all of the components of the user device 702 of Figure 7, or any subset of these.

More specifically, the handset 802 in the case of Figure 8 may include a handset-side SI module 804 that performs a subset of the functions described above, with reference to Figure 4. In addition, the handset 802 may include any mechanisms for orientation, movement and / or position determination 806. These mechanisms are generally referred to hereinafter as head sensing mechanisms, because they determine the physical posture or movement of the user's head, which may, in turn, reflect the direction of the user's focus of attention. In addition, the handset 802 may include any type of input mechanisms 808 described above with reference to Figure 6. Additionally, handset 802 may include a power source 810 and one or more communication mechanisms 812 to interact with user device 106 (if system 102 makes use of a user device 106 in this implementation, which it does not need). In addition, the earpiece 802 includes any type (s) of audio output mechanism (s) 814 described above.

In one mode of operation, the system 102 of Figure 8 may make use of the head sensing mechanisms provided in the handset 802 when they are provided and functioning properly. Head sensing mechanisms provided by headset 802 may be preferable to counterpart sensing mechanisms provided by user device 106 because head sensing mechanisms can more accurately reflect orientation, movement, and / or position. user. In addition, the use of head sensing mechanisms eliminates the need for the user to interact with user device 106 to record its orientation, movement, and / or position.

In some cases, the headset 108 may optionally send the information generated by the head sensing mechanisms to the user device 106. The user device 106 can use this information to perform processing and then send the results of its processing back to the user device. headphone 802. Headphone 802 can then transmit sounds to the wearer's ears through a bone conduction technique that transmits the results. In another case, the hearing aid-side SI module 804 can perform this processing without the need to send the information provided by the head detection mechanisms to the user device 106. In another case, the ear-side SI module 804 can perform some operations natively, and relies on user device 106 to perform other operations. For example, the earpiece-side SI module 804 can rely on the user device 106 to perform computationally intensive operations (such as calculating three-dimensional sounds, etc.) because these operations can be performed more efficiently on the user device. 106 compared to headphone 108.

Otherwise, the system 102 may make use of the orientation determination mechanism (s), movement determination mechanism (s) and / or position determination mechanism (s) provided by the user device 106 when the counterpart components they are not working properly on the 802 handset.

Otherwise, system 102 may make use of sensor readings provided by both handset 802 and user device 106, using, for example, sensor readings from a device to identify glaring errors in sensor readings. from the other device and / or to average the two different versions of the sensor readings, and so on.

In any of the implementations of Figures 7 and 8, at least some of the functions performed by the headsets (eg 704, 802) and / or user devices (eg 702, 106), may alternatively be, or in addition, performed by the remote processing resources 112 of Figure 1 (for example, corresponding to cloud computing resources).

Figure 9 shows a scenario in which a user 902 interacts with a user device 904 within a vehicle. Although not specifically illustrated in Figure 9, the user 902 may also make use of a bone conduction headset that transmits audio information to the user without occluding the user's ears. More generally, Figure 9 is an example of the more general point that the functionality described above can be applied to additional use scenarios than the cases described above.

In the case of Figure 9, user 902 has mounted user device 904, using a mount 906, over the dashboard region of a vehicle. User 902 can be the driver of the vehicle (as shown in Figure 9) or a passenger. A power cord 908 can supply power to user device 904 from a power outlet provided by the vehicle.

Figure 10 shows a process 1004 that describes one mode of operation of the equipment shown in Figure 6. At block 1004, the system 102 receives an instruction as a result of user action of at least one headphone input mechanism ( for example, based on user activation of one of the earphone input mechanisms 608 shown in Figure 6). At block 1006, system 102 performs a function based on the instructions provided at block 1004, to provide an output result. Block 1006 can be performed by the handset

108 and / or user device 106. In block 1008, handset 108 applies the output result to deliver the audio information, which helps the user navigate through a route, within a space, or, more generally, it interacts with space.

In summary, the above features contribute to the above goal of allowing the user to move safely and efficiently through their environment. For example, the features provide a convenient way by which the user can activate various operational modes (for example, by interacting with the user input mechanisms 608 of the headset 108) without unduly distracting the user as the user moves through. environment, for example, without the user having to access and interact with a separate user device.

C. Illustrative User Interface Functionality to Facilitate Interaction Between Users and Their Environments

This section provides illustrative details about the operation of the application interface module 402 of Figures 4 and 5. To repeat, the application interface module 402 allows the user to interact with the various components of the SI module 302. For ease of reference Repeated to Application Interface Module 402, this section will refer to this component in abbreviated form such as AIM 402. The AIM 402 can refer to a discrete component, or to a set of functions performed by two or more components in an actual implementation of the system 102.

The user interface experience, in turn, has different components or aspects. Subsection C.1, below, 5 provides details about an illustrative way in which the AIM 402 allows a user to interact with the SI 302 module through a user interface display, for example, as provided by user device 106. Subsection C.2 provides details about an illustrative manner in which the AIM 402 can provide various sounds and haptic feedback signals to the user. Subsection C.3 provides details about an illustrative way in which the AIM 402 allows the user to interact with the SI module 302 through spoken instructions.

Generally note that the following explanation describes the

AIM 402 in the context of a journey taken by a user through space, for example, as in the example in Figure 2. But the AIM 402 provides a similar service in other scenarios where a user interacts with a space, including the case at 20 where the user travels the space without a prepared route for the purpose of exploring the space in an ad hoc manner. As an additional note, the user interface features described below are also general-purpose in nature, and therefore may be applied in other contexts that are not necessarily related to a user's interaction with a space.

As another note, the description sets out many gestures interpreted by the AIM 402 when describing actions related to the gesture performed by a user, along with corresponding operations taken by the AIM 402 in response to actions related to the gesture. As more fully described above with reference to Figure 5, the AIM 402 performs this task in each case by: a) detecting input information describing a gesture performed by the user, for example, when the user makes a revealing gesture, etc.; b) comparing the input information with stored patterns associated with known gestures, to determine the special gesture that the user has invoked; and c) executing the operations associated with the detected gesture. The explicit citation of these individual operations is omitted in many cases below for the sake of brevity.

C.1. Visual Experience: Interaction with Workspaces and Menus

In one aspect, the AIM 402 organizes a master workspace into a collection of smaller workspaces. Workspaces have positional relationships with respect to each other. This structure is beneficial because it allows the user to develop a mental image of the organization of the application, much like what he or she might be familiar with the zones of a physical space through repeated encounters with its zones. This feature, in turn, allows the user to efficiently access the information and / or the desired functionality.

For example, Figure 11 shows an example where the AIM 402 organizes a master workspace into five smaller workspaces. A home workspace 1102 serves as the central point in user interaction while traveling. For example, as described below, home workspace 1102 presents information about the user's current context, at each particular point in the journey, or any other interaction with a space. If the user has not yet started the journey, the home workspace 1002 may present a default hub page.

A main workspace 1104 is located to the left of home workspace 1102, and a workspace near me 1106 is located to the right of home workspace 1102. An information workspace 1108 is located in the upper part of home workspace 1102, while a configuration workspace 1110 is located at the bottom of home workspace 1102. The respective roles of these workspaces will be described in more detail below.

The position of each workspace described above is stated in the spirit of illustration, and not limitation. Other implementations may provide other workspace locations. Also, other implementations may vary the number of workspaces that are provided by the AIM.

402. For example, another implementation may provide four other workspaces by providing workspaces that are diagonally positioned relative to the home workspace 1102.

Figure 12 shows one way in which a user can navigate from home workspace 1102 to main workspace 1104. In this non-limiting case, the user executes a right turn gesture on home workspace 1102 to navigate to the main workspace 1104. Although not shown, the user can perform a left turn gesture (or other right turn gesture) to move from the main workspace 1104 back to the home workspace 1102.

Similarly, the user can execute a left turn gesture in the home workspace 1102 to navigate to the workspace near me 1106. The user can execute a turn down gesture in the home workspace 1102 to navigate to workspace 1108, and a flip up gesture in home workspace 1102 to navigate to configuration workspace 1110. The user can perform any of these gestures, for example, by using the thumb of the hand holding the user device, for example, by placing the thumb on the touch-sensitive surface of the user device 106 and moving in a desired direction. In a rolling motion, the user contacts the touch-sensitive surface with one or more fingers, moves the finger (s) a distance across the surface, and then removes the finger (s) from the surface , all in relatively rapid succession (as if moving a physical object across a flat surface).

In one implementation, in order to navigate to any peripheral region, the user is expected to first navigate to workspace 1102. However, in another implementation, the AIM 402 may allow the user to navigate from one peripheral region to another without first move back to domestic workspace 1102.

More generally, in all the examples in this section, a dashed circle on the surface of a screen represents the point where the user contacts the surface with a digit. In some cases, the screen surface will show two different sized dashed circles. The larger of the two circles represents the location where the user applies their digit, while the smaller of the two circles represents the location where the user has withdrawn their digit. In other cases (not shown), a user may perform at least one gesture that involves simultaneously touching the touch-sensitive surface in two or more locations (eg, to perform a squash-type gesture). In other cases (not shown), the user can perform at least one gesture by floating above the surface of the device without actually touching it. In other cases (not shown), the user can perform a free space gesture, which can be detected by a video camera and / or a depth camera, etc.

As a final introductory note, this section describes particular gestures, menu structures, and menu items, all in spirit and illustrative, not limitation. Other implementations may vary any aspect of these user interface features. For example, in another implementation (not illustrated), the AIM 402 could allow the user to transition between workspaces with a touch-type gesture, or by drawing a particular shape on the screen surface, etc.

Figures 13-16 illustrate the type of information that can be displayed in home workspace 1102. As mentioned earlier, home workspace 1102 serves as the primary focus of the user's attention as they navigate through a path or interact with space. In a predetermined state, as shown in Figure 13, the home workspace 1102 may display a hub page. The particular nature of the menu structure and the menu items within that structure will be described below.

In the course of the user's journey, the home workspace 1102 may present information about the user's current context (although other information may also be displayed in the home workspace 1102 during the journey, at the user's request). For example, in the case of Figure 14, it is assumed that the user is traveling on a train. The home workspace 1102 may present a menu that focuses on the user experience on the train. In the case of Figure 15, it is assumed that the user is already walking on a particular street. By default, home workspace 1102 will now display information about the user experience on the street. For example, home workspace 1102 may display a map showing the current location of user 1502 on the street. Alternatively, the user can interact with the AIM 402 to produce the type of menu shown in Figure 14, but where that menu will now contain menu items that refer to the user's experience while walking down the street. By virtue of the above behavior, the AIM 402 covers the information that is most relevant to the user in their current context in the environment, and presents that information within the home workspace 1102; in this way, the user can easily find and consume the most relevant information without having to hunt it down, for example without having to navigate through a complicated menu structure. This feature also reduces distractions placed on the user, thereby contributing to the safety of the system 102.

However, the user can interact with the AIM 402 to activate information, for display in the home workspace 1102, that does not pertain to the user's immediate environments. For example, the user can activate a tab menu, for example by activating a tab menu option, which is described below. In response, the AIM 402 will present the tab menu shown in Figure 16. The tab menu displays a collection of open tabs, corresponding to previously open menus. Some of these tabbed menus may correspond to information about previous trip segments that the user has already completed. The user can activate any menu item related to the tab. In response, the AIM 402 may display information regarding a previous travel step in the home workspace 1102, for example, in the form of a menu or some other format. The menu structure may also allow the user to examine future travel steps, after the user's request, that have not yet been found on the way.

Additionally, even after embarking on a journey, the user can instruct the AIM 402 to return to the hub menu shown in Figure 13, for example, so that the user can access the information and / or functionality specified in that menu. default hub.

For clarification, in the example above, the current context refers to a physical location or segment in the global user journey. In other cases, the home workspace 1102 may present context information about the user's exploration of a virtual space. For example, the user can navigate within a hierarchy of products offered by a warehouse. In that scenario, the home workspace 1102 may present a menu that belongs to a group of products, an individual product, and so on.

Generally speaking, the above-described menu types shown in Figures 13-16 correspond to primary menus. The AIM 402 displays these types of menus in the home workspace 1102. In contrast, the AIM 402 displays submenus in the other workspaces that are on the periphery of the home workspace 1102. Figures 17-20 show Illustrative submenus that can be displayed in these peripheral workspaces.

For example, Figure 17 shows a main menu for presentation in the main workspace 1104. The main menu identifies actions that are commonly used when interacting with the different aspects of the SI 302 module. Figure 18 shows a menu settings for presentation in the 1110 settings workspace. The settings menu allows the user to change various parameters that affect the operation of the SI 302 module. Figure 19 shows an information menu for display in the information workspace 1108. The information menu presents convenient access to system information, such as remaining battery life, signal strength. The information menu also provides a convenient portal for notifications and other useful warning information and travel information, such as real-time updates on public transportation services.

Figure 20 provides a near me menu for presentation in the near me workspace 1106. The near me menu del presents information about contextual articles of interest (lOls) that are very close to the user at the present moment (or associated with the current context of the user), as identified by the RID 426 module in Figure 4. The user can use the settings menu to specify a maximum distance that determines what constitutes a near contextual IOI. Each contextual IOI is identified in the menu near me by its name, its category (such as the Food and Drink category), and its distance from the user. A user can activate an item in this menu, causing the SI module 302 to provide audio information about this contextual IOI. That audio information can be formulated as three-dimensional sound information, such that, at the time of delivery, it appears to originate from the same direction that the entity associated with the contextual IOI can be found in physical space.

Figure 21 shows an example of a menu of transients that may be displayed in the home workspace 1102. For example, a user can activate this menu of transients by first navigating to an information menu shown in Figure 19, and which is presented in the information workspace 1108. The user can then activate the notifications menu item in the menu. In response, the AIM 402 can display the transients menu shown in Figure 21.

Figure 22, by contrast, shows an example of an overlay menu. In this particular example, the user can activate the overlay menu by first advancing to the settings menu shown in Figure 18, as presented in the settings workspace 1110. The user can then activate the menu item contextual awareness in that menu. The AIM 402 responds to user action by displaying the overlay menu shown in Figure 22. Unlike the transient menu shown in Figure 21, the overlay menu shown in Figure 22 is displayed above the menu of configurations shown in Figure 18 in the configurations workspace 1110, rather than in the home workspace 1102. Overlay menus and transient menus can also exhibit different behaviors in response to the execution of a back command, as described below.

In the particular case of Figure 22, the overlay menu allows the user to change the level of contextual awareness provided by the SI 302 module. For example, through this menu, the user can set the parameter value that determines the amount of information that is sent to the user. In one case, the user can choose to receive all the information that is potentially relevant to his current context. Otherwise, the user may choose to receive only warnings and other items of information that are considered of great importance.

Figure 23 shows one way in which the user can instruct the SI module 302 to present information about their current context, at any time in the user's journey. According to a non-limiting case, the user can use a digit (eg, a thumb) to make a single touch gesture on the touch-sensitive surface of the user device 106, at any location on the surface. In response to the detection of this gesture, the SI module 302 can present, as a spoken message, high-level information about the current context of the user. For example, the SI module 302 may announce the title of the context menu that would be appropriate for this context. For example, in the example of Figure 14, the SI module 302 could announce, Transport E: London Paddington.

Alternatively, the user can do a double tab gesture. In response to the detection of this gesture, the SI module 302 can present more detailed information about the current context of the user, again as a spoken message. For example, in a double tap, the SI 302 module can announce menu items in the menu shown in Figure 14.

Then assume that the user performs a triple tap gesture. In response to the detection of this gesture, the SI module 302 can announce instructions that inform the user how to interact with the type of menu shown in Figure 14, if that menu, in fact, corresponds to the current context of the user. For example, the instructions can inform the user how to activate the menu, how to navigate within the menu, how to select menu items within the menu, and so on.

Moving on to Figure 24, this figure shows how a user can activate a menu and then how the user can subsequently interact with the menu. In a non-limiting case, the user can activate the menu shown in Figure 24 by making a touch and hold gesture at any point on the surface of the touch screen surface of the user device 106. For example, as shown in Figure 24, a dotted line circle 2402 indicates the location where the user has touched and held their thumb on the surface of the touch-sensitive surface.

In response to the detection of this gesture, the AIM 402 displays menu 2404. The menu is centered on the point (corresponding to a screen location) on the surface where the user touches the touch surface (corresponding to a touch location ). The user may find this feature useful as it eliminates the need to hunt down a particular item in a UI presentation in order to activate menu 2404. Instead, the user can perform the touch and hold gesture anywhere on the surface. This feature, in turn, reduces distractions placed on the user in using system 102. But not all locations will result in successful activation of menu 2404. For example, if the user touches very close to the top or bottom of the surface, the AIM 402 can optionally present an error message to the user (for example, a spoken message), asking the user to repeat their touch gesture and keep closer to the middle of the surface.

The menus illustrated so far have a particular uniform menu structure. Menu 2404 shown in Figure 24 has the same structure, which will be described below, except that this structure is set forth in the spirit of illustration, not limitation. Other implementations may adopt other menu structures and behaviors. For example, this subsection will close with an example of an implementation that adopts a different menu structure compared to menu 2404 in Figure 24, and that exhibits different menu behavior. In addition, the menus described here present their menu items in the form of linear lists; but other menus can be used (eg radial or segment menus, etc.) that present your menu items in other ways.

The menu structure shown in Figure 24 has two groupings of menu items, separated by a bookmark item 2406. More specifically, a first grouping 2408 of the menu items presents menu items that are particularly relevant to the current context of the user or the task in hand. A second grouping 2410 of menu items presents menu items that are relevant to various types of menus that can be displayed in different workspaces, and therefore can be viewed as global or general menu items. For example, the command stop listening, which is a menu item in the second grouping 2410, is relevant across different menus and workspaces, while the menu item item A, which is a menu item in the first grouping 2408, may be chosen because it is particularly relevant to whatever task the user is attempting to perform interacting with menu 2404.

The second grouping 2410, however, can omit certain global choices that are not relevant to a particular menu. For example, if the user is already viewing a tab menu, the second grouping associated with that menu can skip a menu option that allows the user to access the tab menu (because the user is already viewing that menu).

Different menu items (either in the first cluster 2408 or the second cluster 2410) invoke different operations when selected. For example, a first type of menu item can invoke an action when selected. For example, the menu item start listening, when invoked, instructs the SI module 302 to start listening for user voice commands. A second type of menu item may present information when it is invoked, such as information about battery status, etc. A third type of menu item may present an opportunity for the user to change the value of a property, when that menu item is invoked. For example, in some cases, user activation of this type of menu item may activate an overlay menu. The user can then interact with the overlay menu to change the value of a property under consideration. In another case, user activation of this type of menu item can directly change the property value, for example, by changing the setting from an on state to an off state, or vice versa. Still other types of menu items and associated invocation behaviors are possible.

Marker 2406 displays the message free to discard. This message informs the user that he or she can release his or her finger from the surface of the touch-sensitive surface without selecting any menu items, assuming, that is, that the user releases their finger while it was positioned on the marker item 2406, and not some other menu item. In response to that gesture, the AIM 402 can display the 2412 menu in its original disabled state. Bookmark item 2414 in that menu 2412 carries the touch and hold message, which invites the user to perform a touch and hold gesture to reactivate menu 2404 in its active state.

Referring to Figure 25, assume that instead of disabling the menu 2404, the user decides to scroll up through the items in the first grouping 2408. The user can perform this action by moving their finger in the upward direction. , while keeping your finger on the touch-sensitive surface. In response, the AIM 402 may produce menu status 2502 shown in Figure 25. In particular, the AIM 402 that has responded to the user's gesture can move the menu items in the downward direction, causing a first menu item 2504 to be highlighted, instead of the bookmark item 2406. To select this menu item 2504, the user can release his finger from menu item 2504. Alternatively, the user can deactivate the menu by moving back to the bookmark item 2406 and then releasing their finger while hovering over that item. Or the user can scroll down in the opposite direction of the second grouping 2410 of menu items. The user may release their finger or while positioned on one of the menu items in the second cluster 2410 to select that menu item. A user may find the above user interface behavior beneficial because he or she can interact with user device 106 with one hand in a fluid and seamless manner, with minimal distractions placed on the user as he or she moves. through the environment. For example, in this Implementation, the user is not required to find and select a series of command buttons or menu items; to do so, the user is required to pay special attention to the user device 106.

In the specific example of Figure 25, it is assumed that the user scrolls to the top of the first grouping 2408 of items to select a menu item of more items 2506 and then releases their finger on this menu item 2506. In response, the AIM 402 displays menu 2602 shown in Figure 26. This menu provides another first grouping 2604 of menu items that represents a continuation of the first grouping 2408 shown in the previous menu; that is, while the first cluster 2408 in Figure 25 displays menu items 1, 2, and 3, the first cluster 2604 in Figure 26 displays menu items 4, 5, and 6. More generally, the menu structure can represent a complete list of menu items such as a series of small linked lists. Two linked lists are shown in the example in Figures 25 and 26, but the entire list can be made up of any number of linked lists.

The user can return to the menu state 2502 shown in Figure 25 by scrolling to the previous page menu item 2606 and then releasing their finger from this item. Alternatively, to execute this instruction back, as shown in Figure 26, the user can move their finger to the periphery of the touch-sensitive surface (for example, as indicated by the dotted line circle 2608) and then , turns inward toward the center of the surface. The user can perform the same operation by turning inward from the opposite edge of the surface.

Although not shown in Figure 26, menu 2606 may also give the user the option (via an appropriate menu item) to return to the first page of a series of cascading menu lists, for example, instead of successively moving through the linked lists by issuing a series of return instructions. Additionally, although not shown in the drawings, each cluster may include an end-of-list marker item, which designates the end of a cluster. The user can deactivate the menu by moving to this item and removing their finger while hovering over that item.

More generally, the user can execute the type of return gesture shown in Figure 26 in different types of menus, which can produce different types of actions. For example, a return action in a root menu (primary) presented in the domestic workspace 1102 can cause the user to exit the application associated with the SI 302 module, after confirmation by the user, which is what he or she wants to do. A return action on a transient menu that is displayed in the home workspace 1102 can cause the AIM 402 to display any context menu that was presented in the home workspace 1102. A return action on an overlay menu can result in sometimes displaying an underlying child menu in a child workspace. For example, a return action on an overlay menu related to particular settings (for example, as shown in Figure 22) may result in the display of an underlying configuration menu (for example, as shown in Figure 18 ) in the Settings workspace 1110.

Figure 27 shows an alternative workspace organization, compared to the implementation of Figure 11, corresponding to a second implementation of the menu functionality provided by the AIM 402. In summary, the workspace structure Figure 27 includes five workspaces (2702, 2704, 2706, 2708, 2710) that serve as the same basic functions as those shown in Figure 11. For example, the home workspace 2702 continues to serve as the central focus of the user's attention throughout a trip. The home workspace 2702 also continues to display menus and other information associated with the current context encountered in the user's journey (or other interaction with a space). But workspace 2702 no longer presents a concentrate menu as a default. Rather, the home workspace is dedicated to displaying tab-related information by default.

More specifically, the home workspace 2702 displays the default tabs information shown in Figure 28 when there are no active tabs. The default tab information can provide guidance on how the user can start creating tabs. In contrast, the home workspace 2702 can display the tab menu shown in Figure 29 when there are active tabs. By default, the tab menu item that has the first position in the list (that is, tab No. 1) corresponds to a current context.

Figure 29 also shows that the second implementation offers a different menu structure compared to the menu structure described so far. For example, the second implementation no longer organizes the menu into two groups of menu items, separated by a menu marker. Rather, the second implementation presents a single list of menu items that are determined to be relevant to the user's current focus of interest. If the user wishes to access the type of global menu items that were previously presented in the second grouping, the user can navigate to the main workspace 2704 to access those items.

Figure 30 presents additional information about a possible organization of menu items in a menu item list. Here, the menu items represent search result items that are produced in response to conducting a search for stores that are close to the user.

The user can activate any menu, such as the menu shown in Figure 30, by performing the touch and hold gesture described above with respect to the first implementation (as illustrated in Figures 23 and 24).

Upon menu activation, in the second implementation, the AIM 402 presents the first entry in the list near the middle of the user device screen surface. The user can then move up through the list by making one or more pan gestures in an upward direction, or move down through the list by making one or more pan gestures in the downward direction. When the user reaches the end of the list in either direction, the list repeats. That is, when the user advances past the last item in the list, the user will find, as a next entry, the first item. When the user advances past the first item in the list, the user will find the last item. Because of this menu display strategy, the second implementation can dispense with the use of multi-page lists, as is used in the first implementation. That is, the user navigates through a single master list using one or more of the pan gestures. A user makes a pan gesture by placing one or more fingers in contact with the touch-sensitive surface and dragging those fingers in a desired panning direction; the movement here is slower compared to a turning gesture.

Figure 31 provides additional information on the behavior described above. Here the user makes a pan gesture by dragging the displayed menu in the downward direction. But unlike the case of the first implementation, the user does not need to keep their finger on the surface when navigating through the list. For example, the user can perform multiple pan gestures; Between each panning gesture, the user can remove their finger from the surface. The second implementation will not interpret the removal of the user's finger from the surface as an instruction to select any menu item that will be highlighted at that time. The second implementation can deactivate a menu after a specified amount of time of inactivity by the user, or when the user gestures back.

As shown in Figure 32, a user can select a highlighted item in the list by making a brief touch-drag gesture to the periphery of the surface. The AIM 402 will provide audible and / or haptic feedback when it has interpreted the user's gesture as a request to select the highlighted item. The AIM 402 will then select the item when the user removes their finger from the surface of the user device 106. Alternatively, instead of removing their finger, the user can drag their finger back in the opposite direction to cancel the selection operation, where the AIM 402 will reconfirm (eg, providing an appropriate signal or signals).

Figure 33 shows a back gesture performed at the periphery of a menu and the response to this gesture. More specifically, the user can now perform a backward gesture by pulling to the right or left from the corresponding edge for a short distance that even the AIM 402 indicates that it has understood the user's gesture (for example, by providing a signal or appropriate signs). At that time, the user can release their finger to perform an action. Or the user can move their finger in the opposite direction until the AIM 402 interprets the user's action as a request to revoke the previous action. In the example in Figure 33, the above action causes the AIM to

100

402 move from a search results menu to a tabbed menu.

Figure 34 shows a gesture that a user can perform when drawing a circle shape 3402 in a menu, for example, when drawing a circle in a search results menu (for example) that is presented in the home workspace 2702 In response to this gesture, the AIM 402 will return to the tab menu. Although not displayed, the user can later perform the same circle gesture in the tab menu. In response, the AIM 402 will return to the search results menu (or whatever page the user originated from, that is, any page where the user drew the original circle).

Figure 35 shows a gesture that the user can perform when drawing a semi-circle 3502 in a menu, for example, when drawing a semi-circle in a search results menu (for example) that is presented in the home workspace 2702 In response, the AIM 402 will return the focus to what was presented in the current context. In the example in Figure 35, the AIM 402 returns the information associated with a current step in the journey (as shown on the right side of Figure 35). If the user is not currently on a trip, the current context can correspond to any menu that was last opened. The user can reverse this transition by drawing a semicircle on the page shown on the right side of Figure 35.

Figures 36 and 37 show different gestures that can be

101 used to increase or decrease, respectively, a level of verbosity provided by the system of Figure 1 and a level of contextual information provided by the system. For example, as indicated in Figure 36, the user can draw a right arrow shape 3602 in any menu (here, a tab menu) to increase the verbosity level of the SI 302 module. As indicated in Figure 37, the user can draw the left arrow shape 3702 in any menu to decrease the verbosity level of the SI 302 module. Verbosity level refers to the brevity in which the SI 302 module offers its audio information, for example, ranging from brief to detailed.

As also indicated in Figure 37, the user may draw an up arrow shape 3704 or a down arrow shape 3706 in any menu to increase or decrease, respectively, a level of information that is provided to the user. For example, the user can reduce the amount of contextual information by refraining from submitting the user information about one or more of the less critical categories of information articles (such as omitting the contextual information, but sending the alert information and travel information ).

The second implementation can also execute other new gestures (compared to the first implementation), although it is not specifically illustrated in the figures. For example, the user can perform a touch and hold gesture on a first menu

102 to go from that menu to the action menu (where actions refer to actions that are relevant to the context associated with the first menu). The user can perform the same gesture in the action menu to return to the original menu (first). The touch and hold gesture in this case involves a longer hold action than the touch and hold gesture that is generally used to activate any menu.

As another new gesture, the user can perform a tilt gesture, while in the menu items of a menu. The AIM 402 will interpret this action as a request to fast-forward through menu items, either in an up or down direction, depending on the direction of the turn. As another new gesture, the user can pan gesture on a menu item in a menu to advance to the next menu item in the list. The user can perform multiple pan skins to successively advance through the list, one menu item at a time.

In either of the first or second implementations (corresponding to Figures 11 and 27), the user can perform a gesture to place the user device 106 in a pocket mode. When user device 106 detects that gesture, it will ignore any subsequent gestures that the user may perform, with the exception of a gesture that has the effect of canceling pocket mode. As the name suggests, the pocket mode is useful in those cases where the user wants to save the device from

103 user 106 in a pocket or some other compartment (such as a purse, bag, etc.). When active, the pocket mode prevents accidental touch contact and / or movements by the user as (incorrectly) interpreted as meaningful gestures. The gestures used to invoke and revoke the mode may correspond, for example, telltale sweep gestures etc.

C.2. Haptic Sounds and Signals

The AIM 402 can present various sounds by making use of the sound generation module 414. The AIM 402 can also present various haptic feedback experiences (eg, vibration signals) by using the haptic signal generation module 418. The AIM 402 can produce such vibratory sounds and signals in response to different types of triggering events. For example, the AIM 402 can present these haptic sounds and signals in response to certain changes in state, in response to certain interface actions taken by the user (such as navigating between menu items), in response to certain events in a trip, and so on.

The following list describes representative sounds (and associated haptic signals) and the illustrative circumstances in which they are invoked.

Load / busy. The AIM 402 can repeatedly play a loading / busy sound while the SI 302 module is in a state where the user cannot interact with it. Sound

104 guarantees the user that the SI 302 module is working, but is currently busy performing an action. For example, this sound may resemble a ping pong ball bouncing up and down; the frequency of the bounce may increase as processing nears completion and may end with a flourish-type sound.

Transition To Zone Menu. This sound indicates that a sub-menu, displayed in one of the dedicated sub-workspace zones, has been moved to focus and is now displayed in place of the context menu in the home workspace. This sound can also convey the direction in which the user has made the corresponding gesture. For example, the sound may resemble a hiss, like a gust of wind in a respective direction. Also, in one case, the AIM 402 can express this sound as a three-dimensional sound.

Transition From Zone Menu. This sound indicates that the current context menu has been moved back into focus and is now presented in place of a secondary menu in one of the dedicated workspace zones. This sound can also convey the direction in which the user has made the corresponding gesture. For example, the sound may resemble a hiss that is the counterpart of the Transition A Zone Menu hiss, but moving in the opposite direction of the Transition A Zone Menu sound. Again, the AIM 402 can optionally express this sound as a three-dimensional sound.

105

Menu activation. This sound indicates that a menu is now active and can be manipulated. This sound can be described as a sound fade ending with a popping type sound. Additionally, the AIM 402 may present a brief vibration signal that confirms that the user intends to cause a change in state. Furthermore, the AIM 402 may invoke a verbal signal that announces, for example, release to reject menu. That signal informs the user that the menu is active and that the user can deactivate the menu by releasing their finger from the marker item.

Deactivation menu. This sound indicates the menu has been deactivated and therefore can no longer be manipulated without reactivating it. The sound can be described as a fade type sound.

Gesture not supported. The AIM 402 can play this short sound indicating that a gesture was recognized but is otherwise invalid (for example, because it is currently not supported). This sound may resemble a thud, followed by a soft two-tone access denied style notification. The AIM 402 can play a more specific Do Not Return sound to indicate that a return gesture has been performed, but it cannot be performed, for example, because it is not possible to go further back.

Change in Menu Item Selection. This sound indicates that the user has moved to a menu item in a menu. The pitch of this sound depends on the position of the menu item in

106 the menu, for example, where the pitch can increase as the user moves up the list and decreases as the user moves down the list. Also, in one implementation, the AIM 402 can display different scales when traversing the first grouping of menu items (associated with context-specific items), compared to the second grouping of menu items (associated with global items). . In both cases, the user can perceive the sounds that are produced in a similar way to the movement of a hand on the keys of a piano; but each sound in this case can be shorter than the pitch of a piano, and similar to the popping of a bubble.

Additionally, the AIM 402 may present a vibration signal as it traverses each menu item. The user may experience the vibration signals as similar to the movement of a hand over an uneven surface, with each dent representing a different menu item. A user can consciously or unconsciously count dents to quickly get a general idea of ​​their position within the list. Additionally, the AIM 402 may present an audio message after advancing to a menu item that informs the user that the menu item is in focus and will be selected when the user's finger is released. For example, the AIM 402 can advertise the menu item, giving its number in the list and then announcing a description of the item.

107

Confirm Menu Selection. This short sound indicates when an action has caused a change in state. It can be implemented as a short, precise sound with a flourish at the end. For example, the sound may resemble a beep, followed by a chirping sound journey that fades toward its end. The AIM 402 can also play a short vibration signal confirming the selection action. Additionally, the AIM 402 can play a verbal signal confirming that the topic has been selected.

Show Dialog Menu. This short sound indicates that an overlay menu is now displayed on top of other content. This sound may resemble a fade-up type sound. The AIM 402 can also provide an accompanying short vibration signal, further confirming that a change in state has taken place. In addition, the AIM 402 can reproduce a speech signal stating that the overlay menu has now been presented.

Close Dialog Menu. This short sound indicates that a dialog (for example, an overlay menu) has been closed and that focus has returned to a previous context. The sound may resemble a fade-down type sound.

Context switch. This short and unmistakable sound indicates that the current context menu state has moved to present new content, for example, because the user has advanced to a new travel step, etc. The sound may resemble a beep followed by a subtle lathe noise.

108

The sounds described above are representative, and not exhaustive, of the wide variety of sounds that the AIM 402 can provide. The following additional sounds, for example, can be activated after certain actions, performed by the user, while interacting with the AIM 402: a) a For Actions sound to indicate that a menu of actions is present, based on the selection of a menu item made from a previous menu; b) an Actions sound to indicate that an active menu is present in response to returning from an action menu; c) 10 an Actions Unavailable sound to indicate that an action instruction present for a given menu has been acknowledged, but that there are no actions associated with the given menu; d) a Selectable Item sound indicating that a menu item has been correctly marked for selection 15 upon releasing the user's finger; e) an Item Not Selectable sound to indicate that an item that was previously marked for selection-after-release has been correctly deselected and will therefore no longer be selected when the user's finger is released; f) an Item Not Selectable sound to indicate that a gesture to mark an item for selection has been recognized, but the gesture is not applicable to the item under consideration; g) Selection-But-No Change sound to indicate a selection that has been made, but that no change in focus is appropriate; h) a Subsequent Success sound to indicate 25 that a previous gesture was recognized and the previous action has been

109 invoked; i) a Switch-To-Tabs sound to indicate that a switch-to-tabs gesture has been correctly recognized and that the tab menu is now displayed; j) a Switch-From-Tabs sound to indicate that a switch-from-tabs gesture has been recognized and the previous menu, which was displayed before switching to the tab menu, has been restored; k) a Start-To-Listen sound to indicate that the SI module 302 is now listening for voice commands from a user; i) a Cancel-Listen sound to indicate that the voice recognition mode has been canceled; m) a Processing-Complete sound to indicate that the application has finished receiving and processing a voice input; n) an ActionTak sound to indicate that an action has been taken as a result of voice-based input, and so on. In addition, the AIM 402 can also present any type of haptic signals that will accompany any of the sounds described above.

The following illustrative sounds may be triggered after certain events that may occur in the course of user interaction with the headset 108: a) a Button-Hit sound (played prior to performing an action) to indicate that a touch has been recognized on a headset button; b) a Push-Button sound (played before any action is performed) to indicate that a push gesture has been recognized on a headset button; c) a Hold-Button sound (played before performing an action) indicating that a hold gesture has been

110 recognized on a headset button; d) a CancelAction sound indicating that any previous action, invoked from a headset button, has been canceled, for example, to stop a request by announcing contextual lOls as part of the targeting action, and so on. In addition, the AIM 402 can also present any type of haptic signals that will accompany any of the sounds described above.

The following illustrative sounds can be activated after certain events that may occur during the course of a trip: a) a Trip-Started sound to indicate that a trip has already started and that navigation is now in progress; b) a Mode-Switch sound indicating that the user has changed their mode of transportation (eg, from walking to the train, etc.); c) a Heartbeat sound to directionally indicate the next point along the journey that the user is attempting to reach as part of navigation; d) a Warning sound to indicate that warning information is to be read aloud, played to increase user awareness and give the user time to tune in to that information; e) a Waypoint Reached sound to indicate that the user has reached a travel waypoint and that the navigation information is to be read aloud, played back to increase user awareness and give the user time to tune in to that point. information; f) a Landmark Approach sound to indicate that the user is approaching a landmark and that the landmark information is

111 navigation is to be read aloud, reproduced to increase user awareness and give the user time to tune in to that information; g) a Refresh-Trip sound that is played to indicate that information regarding a change in the current trip is about to be read aloud, played to increase user awareness and give the user time to tune in to that information; h) a contextual information sound indicating that contextual information is to be read aloud; i) an Information-Additional sound indicating that the additional information is to be read aloud, and so on. In addition, the AIM 402 can also present any type of haptic signals that will accompany any of the sounds described above.

Generally, the aforementioned sounds and haptic signals further further the goal of providing useful information to the user as the user interacts with his environment, without unduly distracting the user. For example, the haptic sounds and signals that the user hears while navigating a menu allow the user to interact with the menu without diverting their attention from the environment.

C.3. Voice Recognition Mode

System 102 supports a voice recognition mode in which the user can issue commands to system 102 through spoken instructions, for example, in addition to manually interacting with headset 108 and / or user device 106. OR

112 system 102 may use voice recognition mode as the only way the user interacts with system 102. To recognize the user's voice commands, system 102 may rely on pre-existing voice recognition technology ( for example, the CORTANA system provided by MICROSOFT Corporation) and / or native speech recognition technology. Figure 3 illustrates speech recognition technology as the 328 speech processing engine.

As mentioned above, the user can invoke the voice recognition mode through an appropriate command issued through the headset 108 and / or the menus of the user device 106. In other cases, the user device 106 may already be interacting with a speech recognition system (eg CORTANA) as part of its default mode of operation, but not in the context of performing navigation. Here, the user can interact with the SI 302 module in speech recognition mode by issuing a spoken command that is preceded by the name of the navigation application, for example, saying the command, Soundscape, starts mode. to explore. If the user has already activated the speech recognition mode in the context of the navigation application, the user can simply say, Start browsing mode, or the like.

The following list identifies the illustrative operations that can be started in voice mode: a) The user can open a

113 existing trip (saved); b) the user can activate or reactivate the orientation mode; c) the user can activate or deactivate the scan mode; d) the user can request more information on a particular topic; e) the user can request the termination of all (or some) of the spoken messages; f) the user can make various parameter settings; g) the user can save travel information, and so on.

The illustrative commands can correspond to: a) Create a destination route [x], leaving a time [y], using only public transport [z]; Find the closest coffee shop; c) Take me to a train station; d) What is time ?; e) Increase volume; f) Remove restaurants from articles of interest, etc.

The speech processing engine 328 can sometimes encounter a situation where it understands the user's command, but determines that the command omits one or more items of necessary information. In response, the speech processing engine 328 may ask the user to provide the missing items of information.

In other cases, the speech processing engine 328 may encounter a situation where it does not understand the user's command. In those cases, the speech processing engine 328 may ask the user to rephrase the command. If this is not satisfactory, the speech processing engine 328 may present proposed interpretations of the user's pronunciation for the

114 user (for example, based on keyword detection in the user's command) and then prompting the user if any of the interpretations are correct. If this is not successful, the speech processing engine 328 may invite the user to enter an equivalent command through the handset 108 and / or the user device 106.

Figure 38 shows a process 3802 that summarizes one way of operation of the AIM 402 of Figures 4 and 5. In block 3804, the AIM 402 detects a first gesture, performed by the user, corresponding to an instruction to activate a menu in an associated workspace. The associated workspace corresponds to one of a plurality of workspaces and the associated workspace has a certain spatial relationship with respect to other workspaces in the plurality of workspaces. The first gesture may, for example, correspond to the above described touch and hold gesture shown with reference to Figure 24. At block 3206, the AIM 402 activates the menu in response to the first gesture.

At block 3808, the AIM 402 detects a second gesture, performed by a user, corresponding to an instruction to advance to a particular menu item, among a collection of menu items on the menu. The second gesture, for example, can correspond to any of the types of scrolling or panning gestures described above (for example, with reference to Figures 25 and 31). At block 3810, the AIM 402 advances to the

115 particular menu in response to the second gesture.

At block 3812, the AIM 402 detects a third gesture, made by a user, corresponding to an instruction to select a particular menu item. The third gesture, for example, may correspond to the type of release gesture shown in Figure 25 or the type of pull-to-the-side gesture shown in Figure 32. At block 3814, the AIM 402 performs an operation. in response to the third gesture. The operation can correspond to the invocation of an action, the configuration of a parameter, the presentation of the information, etc.

In summary, the above features allow the user to move safely and efficiently through their environment. For example, workspaces provide an easy way for the user to overlay the information most relevant to the user as the user moves through the environment. Additionally, some of the features that allow the user to interact with a user interface presentation without directing visual attention to that presentation, and / or without having to perform complex and cumbersome gestures that divert the user's attention from the physical task of the interaction with the environment. For example, the user can perform some of the gestures with one hand without looking at the user interface presentation.

116

D. Facilitation of Interaction between Users and their Environments Using Sounds

As described in Introductory Section A, the sound generation module 414 (Figure 4) can generate non-three-dimensional (non-spatial) sounds and three-dimensional (spatial) sounds. A three-dimensional sound is a sound that is perceived by the user as coming from at least one location in physical space, although it has no real origin in physical space. Specifically, Figure 39 shows the use of three-dimensional audio information to create a perception of sound emanating from a particular location within space. Figure 40 shows the use of three-dimensional audio information to create a perception of sound moving through a series of locations in space.

In one implementation, the sound generation module 414 can produce a three-dimensional sound by using a library of Head Related Transfer Functions (HRTFs), for example, as provided in data storage 416. An HRTF models anatomical features of the face of a person who has a specific person in the way the person perceives the origin of the sound in their environment. To produce three-dimensional sound, the sound generation module 414 can choose an HRTF that is appropriate for a particular person and for a certain location of sound in space (relative to the location of the person). The 414 sound generation module

117 You can then apply that HRTF to modify a non-three-dimensional sound (eg flat or non-spatial sound), to produce the three-dimensional sound, eg by convolving the non-spatial sound with the chosen HRTF.

More specifically, in one case, the sound generation module 414 may receive input information describing: (a) the non-three-dimensional sound to be reproduced; (b) the location in space in which the sound is perceived as originating; and (c) (optionally) the identity of the user. The input information may describe non-three-dimensional sound by providing reference information identifying the audio information in data storage 416, and / or it may provide the same audio information. For example, non-three-dimensional sound can correspond to a telling tone and / or a spoken message. Next, for each of the user's ears, the sound generation module 414 can: (a) identify an HRTF, associated with the location for the particular user under consideration; and (b) applying the HRTF to the non-three-dimensional sound to produce the three-dimensional sound. The input information that is provided to the sound generation module 414 can originate from one or more of the other modules of the SI module 302 described below, for example, corresponding to the path guide module 420, the module of determining relevant information (RID) 426, scanning module 430, orientation module 432, etc.

118

The sound generation module 414 performs the aforementioned functions with respect to a library of HRTF provided in a data store 416. In one implementation, the SI module 302 can store HRTFs in the data store 416 that have been prepared for particular individuals (and that, therefore, take into account the particular anatomical characteristics of these people). For example, HRTFs, for each person, can be obtained by measuring the physical characteristics of each person's face.

That task, in turn, can be accomplished manually (for example, by using physical distance measurement tools), by using a depth camera (for example, using the KINECT system provided by MICROSOFT Corporation of Redmond, Washington), and so on. More specifically, the set of HRTF that is generated, for a particular person, includes an HRTF for each location of the sound in space (with respect to the position of the user's head) and for each of the two ears of the user. user. Different implementations may rely on different individuals to create these HRTFs (eg, acoustic engineers, system administrators, end users, etc.).

In a second implementation, the sound generation module 414 can store separate groups of HRTFs that work well for different groups of respective people. System 102 may then invite an end user to choose the

119 HRTF set that is perceived as producing the most realistic three-dimensional sounds. For example, a user may make this selection at an initial stage, requesting the system 102 to produce the same three-dimensional sound (s) using different HRTFs; the user can choose the sound (s) (and the corresponding HRTF) that produce the most desirable result. In a third implementation, the sound generation module 414 can store a unique set of HRTFs that have proven suitable for a large population of users, even though these HRTFs are not customized for any person or group of people. Other techniques can be used to generate and / or select suitable HRTFs.

In one implementation, the sound generation module 414 generates three-dimensional sounds that transmit broadband audio information. Wideband audio information is audio information that includes a wide distribution of audio frequencies, for example, in one implementation, in the range of 300 Hz to 3.4 kHz. Additionally, the various three-dimensional sounds are expressive in nature to further enhance their directionality. For example, one of these three-dimensional sounds may resemble a gust of wind blowing from left to right or right to left. Other properties of audio information (in addition to its HRTF-based directionality) can contribute to its expressiveness, such as by providing variations in volume, pitch, reverb,

120 frequency of repetitions (for repetition of sounds), etc. All of these factors contribute to a realistic perception that sound is coming from a particular location (or locations) in physical space.

As also described in Section A, different modules can make use of three-dimensional sounds for different purposes and in different modes of operation. These modules include Path Guidance Module 420, RID Module 426, Scan Module 420, Orientation Module 432, etc. To simplify the description, each module is sometimes described as producing three-dimensional sounds. In reality, each module produces the sound by generating the previously described input information, which is fed to the sound generating module 414; the sound generation module 414 then uses the input information to produce the actual three-dimensional sounds.

Referring to Figure 41, first consider the operation of the path guide module 420. As already described, the path guide module 420 determines the current direction in which the user is heading. The trajectory guidance module 420 can make this evaluation based on any information provided by the system orientation determining mechanism (s), motion determining mechanism (s), position determining mechanism (s), and so on. In one case, for example, the

121 trajectory 420 can determine the direction in which the user's head is pointed (for example, based on an orientation determination mechanism in the handset 108) and then use this information as a proxy for the direction in which it is pointed. presumes that the user is pointing. Alternatively, or in addition, path guide module 420 may determine a series of locations the user has just traversed. The trajectory guide module 420 can use this information to project a direction that appears as if the user is pointed. Alternatively, or in addition, the trajectory guide module 420 may determine the direction in which the user is deliberately directing his user device 106.

Path guide module 420 can also determine the user's desired direction. The path guide module 420 can determine the desired direction based on at least the user's current location and the location of the next waypoint. The trajectory guide module 420 may also take the map information into account when determining the desired direction. For example, the map information may reveal that the user's path is restricted in various ways, for example, by the course of a road, obstacles of any kind, etc. The trajectory guidance module 420 can use all this information to determine the direction in which the user should be heading to a last place, put the user on course to reach the next waypoint, in the most efficient way.

122 possible.

Based on the actual direction and the desired direction, the trajectory guide module 420 can then determine the degree to which the user can deviate from the desired direction, for example, to provide deviation information. The path guide module 420 can then use the deviation information to generate a three-dimensional sound that will have the effect of directing the user in a desired direction, for example, by inputting the deviation information to the sound generation module 414 as part. input information.

To clarify the above description, consider the example in Figure 41. Here, the user is at current location 4102. The next waypoint (w) is located at target location 4104. User's current address (actual) is denoted as address 4106. The user's desired address 4108 corresponds to the vector that connects the user's current location 4102 to the target location 4104, although, as noted above, this should not be the case in all situations (for example, due to an obstruction blocking a direct route from current location 4102 to target destination 4104). The offset information may correspond to the angular difference between the current address 4106 and the desired address 4108.

In response to the above determinations, the path guide module 420 can produce a three-dimensional sound (using the sound generation module 414) that appears

123 come from location 4110. In other words, the user is currently directed to the left of the desired direction 4108. The user will hear the directional signal coming from the right. The user will perceive that signal as pushing the user to veer to the right to correct the direction of his trajectory. If the user begins to move in a direction that is too far to the right of the desired direction 4108, the path guide module 420 can produce a three-dimensional sound (using the sound generation module 414) to the left of the user. The user can continue in, essentially follow or chase the direction of the perceived three-dimensional sound. When the user heads in the right direction, the three-dimensional sound will be perceived as emanating directly in front of the user; the user then goes directly to that sound.

As noted above, the path guide module 420 may use the sound generation module 414 to create a three-dimensional sound that is periodic in nature, eg, corresponding to a beat sound. For example, the heartbeat sound may correspond to a repetitive single-tone or two-tone or n-tone repetitive sound, and so on. Additionally, or alternatively, the three-dimensional sound may appear to move in the direction that the user is pushed to move, for example, left to right or right to left, etc.

In addition, the path guide module 420 can

124 providing input information to the sound generation module 414 that has the effect of varying one or more audio characteristics of the beat sound, depending on the extent to which the user's current address 4106 deviates from the desired address 4108. For example, the trajectory guide module 420 (in conjunction with the sound generation module 414) can generate a heartbeat sound having a first tone and / or a first loop frequency when the current deviation of the user from the ideal trajectory is within a first threshold. The trajectory guide module 420 (along with the sound generation module 414) can generate a beat sound having a second tone and / or a second loop frequency when the user's current deviation is outside the first threshold, but within of a second threshold, and so on. The trajectory guidance module 420 can classify the user deviation into any number of such deviation categories, each associated with a particular heartbeat sound.

In the particular example of Figure 41, the path guide module 420 defines a first range 4112 of deviations from the desired direction 4108 and a second range 4114 of deviations from the desired direction 4108, the second range 4114 being larger than the first rank 4112. The current address 4106 falls within the second range 4114, and thus the path guide module 420 (in conjunction with the sound generation module 414) will generate a beat sound that is appropriate for this range 4114.

In actual practice, path guide module 420 is useful

125 in many circumstances, but it can be perceived as more useful in two circumstances. In the first case, a user can come to an intersection where he or she turns in a particular direction, among a fixed set of available options. Or the user may come to a fork in the road where he is expected to choose one of the possible paths. The trajectory guide module 420 (in conjunction with the sound generation module 414) can provide unambiguous guidance to the user by reproducing a three-dimensional heartbeat sound that the user perceives as resolving his navigation options. For example, when at an intersection, the user will hear the beat sound by driving to the left or right (for example). When at a fork in the road, the user will perceive the heartbeat sound as leading him down the correct fork in the road.

In a second scenario, the user moves through a larger space in which he or she is given more latitude to gradually deviate from his or her course (compared to the case where the user is walking on a highway or route well defined). Here, the trajectory guide module 420 can give the user incremental pushes in the manner illustrated in Figure 41 to keep the user on a proper heading.

In another usage scenario, the Relevant Information Determination (RID) module 426 may determine, at each point along a user path and for each current context

126 corresponding, the location of relevant articles of interest (lOls), such as contextual lOls. The RID module 426 can then use the sound generation module 414 to generate a three-dimensional sound for each IOI. The user will perceive that sound as coming from the actual location of the corresponding physical entity in physical space, that is, assuming that the IOI corresponds to a physical entity that has a discrete physical location in space. For example, the RID module 426 (in conjunction with the sound generation module 414) can generate the following audio information for a particular IOI that has a physical counterpart entity in close proximity to the user: [3D Sound], Coffee Shop from Jane, restaurant, 9.14 meters ahead. In other words, in this case, the audio information includes a three-dimensional sound, followed shortly thereafter by a spoken message. The user will perceive the preliminary three-dimensional sound as coming from a physical location in Jane's cafeteria. The spoken message can also be presented as a three-dimensional sound (although it is not necessary). The spoken message could also include more (or fewer) information fields, or it could be omitted entirely.

In other cases, an IOI may have a general origin in space, rather than a precision location. For example, the IOI may correspond to an advertisement that is associated with a general area which, in turn, corresponds to a bus stop. The 25 RID 426 module (together with the sound generation module

127

414) can generate a sound for that IOI that is perceived by the user as coming from the general area of ​​the bus stop, or from the center of that area, etc. This type of IOI can be considered virtual insofar as it is not intended to describe the bus stop, but functions as an audio attribute of the bus stop.

In another implementation, the audio information may also include a speech signal that specifies the directional location of the IOI, relative to the user. That directional signal can be expressed broadly (for example, specifying that Jane's Cafeteria is ahead of the user and to the right), or more closely (for example, specifying that Jane's Cafeteria is at 10 o'clock, relative to the user's current address). Additionally, or alternatively, the RID module 426 (along with the sound generation module 414) can provide different three-dimensional sounds for different respective categories of lOls, for example, by reproducing a first type of restaurant sound and a second type of restaurant sound. sound for toilets, etc.

Both the scan module 430 and the orientation module 432 make use of the previously described behavior of the RID module 426, for example, in the scan mode and the orientation mode, respectively. As described in Section A, the user can enter a command to expressly turn these modes on and off, for example, through

128 manual interaction with headset 108 and / or user device 106 and / or through voice instruction.

More specifically, Figure 42 shows illustrative behavior of the scan module 430, operating in the scan mode. In this mode, once the scan mode is activated by the user, the scan module 430 determines the current location of the user and the current focus of the user's interest. For example, scanning module 430 may use any position determination mechanism (s) on earpiece 108 and / or 10 on user device 106 to determine the user's current location (eg, using a sensor GPS, etc.). Scanning module 430 may use any orientation mechanism (s) on headset 108 and / or user device 106 to determine the direction in which user 15 appears to be their body orientation. For example, the orientation mechanism (s) in the earpiece 108 can determine the direction in which the user's head is pointing, which can be used as a proxy indication of the user's focus of interest. Alternatively, or in addition, the system 102 may instruct the user 20 to direct the user device 106 in the direction that coincides with its focus of interest; in that case, the orientation mechanism (s) of the user device 106 will correctly reflect the user's intended focus of interest. Other techniques can be used to assess the user's focus of interest.

Next, the scanner 430 defines a space

129 search of interest. In one case, the scanning module 430 may define the search space as a volume in any way that is centered on the presumed focus of interest of the user. For example, the volume may correspond to a wedge-shaped volume that originates from the current location of the user and is bisected by the presumed focus of interest of the user. System-defined and / or user-defined parameter settings can specify volume depth, volume angular range, volume width, and so on.

The scan module 430 can then use the RID module 416 to search the volume for the presence of LOLs of interest, such as, but not limited to, contextual LOLs. The RID 416 module can accomplish this task by querying one or more data stores that provide information about the lOls, along with any parameter settings that define the particular interests of the user (for example, the user could have indicated that he or she is, by default, interested in restaurant information, to the exclusion of cafeteria information). In some cases, an IOI may be associated with the search space because it has a counterpart physical entity that is physically located in the search space. In other cases, an IOI may be associated with the search space due to some other nexus established in another way. For example, an administrator (or his or her end user) can manually specify, in advance, that the area

130 near the exit of the metro station is associated with a weather report IOI, and so on.

Finally, the scanning module 430 can generate three-dimensional audio messages that announce the presence of the 10s that have been identified. The scanner module 430 can generate these messages in any way described above by making use of the sound generation module 414. For example, as described above, the audio information can correspond to an introductory sound, followed by a spoken message. announced by the IOI. The Scanner Module 430 (along with the Sound Generator Module 414) can read these 10s in an order, such as clockwise through volume, or counterclockwise, and / or counterclockwise. or increasing or decreasing the relative distance from the user's location, etc.

Figure 42 clarifies the mode of operation described above. Here, the user is currently at a current location 4202. The user's current attention direction is indicated by the dashed line 4204. The search volume is defined by contour 4206, which is bisected by the user's attention direction. That search volume defines an angular fringe and can also be a certain height (not shown), such as 30 feet. In general, the search volume resembles a wedge. The scan module 430 uses the RID module 426 to find three contextual lOls in the search volume. The 430 Scan Module (along with the Scan Module

131 sound generation 414) summons these contextual lOls by three-dimensional sounds that appear to originate from the locations of the physical entities associated with the contextual lOls.

In general, the user can interact with the scanning module 430 by rotating their direction of attention, pivoting around their current location, in stages. At each stage, the user can wait to hear the lOls that lie within (or are associated with) the search volume thus defined by his presumed current focus attention, before moving to a new focus of attention. For example, in one case, the swath spanned by a direction of interest to the user is 45 degrees. The user could take a complete inventory of lOls around it by turning northeast, southeast, southwest, and northwest, in succession. In each orientation, the user will hear the lOls that are included in that quadrant of interest.

According to another illustrative feature, the user can select an IOI in any way, after listening to it. For example, the user may issue voice instructions Take me there or more information, after hearing about Jane's Cafeteria. Or the user can select the IOI by turning their focus to the perceived location of the IOI (eg, based on the perceived location of the three-dimensional sound announcing this IOI). The user can perform this task, for example, by turning their head or body or user device 106 directly towards the

132 perceived location of the IOI. Or the user can select the IOI through a suitable menu provided by the user device 106. Or the user can make an appropriate selection through a headset button after hearing the IOI being announced, and so on. In response to the selection of the IOI, however, the SI module 302 can present additional information about the IOI, or provide instructions on how to get to the IOI, etc. The SI module 302 can also confirm the user's selection by increasing the volume of the three-dimensional sound announcing the presence of the chosen IOI or, otherwise, making the sound more prominent in the user experience.

Finally, the orientation module 432 operates by determining the current location of the user in the manner described above. Next, a three-dimensional volume is defined around the user. For example, the three-dimensional volume can correspond to a cylinder or sphere or box or rectangle (for example), with the user positioned in the center of the volume. The targeting module 432 then uses the RID module 426 to identify the set of 10s that exist within the volume, if any. The orientation module 432 then uses the sound generation module 414 to read the 10s in any order.

For example, in Figure 43, the targeting module 432 defines a three-dimensional volume that is shaped like a cylinder, with the user positioned at its center at a current location 4302. The targeting module 432 uses the RID module 426 to

133 identify the lOls that lie in the cylinder or that are otherwise associated with the space defined by the cylinder. The orientation module 432 then reads the 10s in any order. For example, suppose that a user visits a shopping center that has three floors. Suppose the user is standing in the outdoor atrium on the first floor of the mall. The orientation module 432 can read the lOls it finds on a floor by floor base, for example, floor z₁₍ floor z₂ and then floor z₃. The orientation module 432 can announce the lOls on each floor in any order, such as a clockwise order, a counterclockwise order and so on. The user can then optionally select any IOI in any of the ways described above.

As a closing comment, the previous explanation establishes the use of three-dimensional sounds to announce the presence of contextual objects, such as restaurants, etc. But the SI module 302 can use three-dimensional sounds to announce the presence of any kind of lOls, such as warning lOls and travel lOls. For example, the SI 302 module can generate a warning about a pothole in the road that appears to come from the location of the pothole. As another clarification, the SI module 302 can also provide many types of lOls in a flat or non-spatial form. For example, the SI 302 module may produce another warning IOI that has no directivity to generally notify the user that the road they are currently traveling on is

134 slippery due to rain.

Furthermore, the above description is implicated in the use of sound that the user perceives as originating from locations in physical space. Additionally, or alternatively, the haptic signal generation module 418 may generate vibration signals that convey directional information. For example, the earpiece 108 may include two or more vibration mechanisms, for example, a first vibration mechanism on the left side of its frame 602 and a second vibration mechanism on the right side of its frame 602. The generation module haptic signal 418 can activate either the first or the second vibration mechanism to provide the user with instructions to turn left or right, respectively. This type of instruction can include additional gradations by including additional vibration mechanisms. The same effect can be achieved by activating different vibration mechanisms on user device 106, for example by activating a left-side vibration mechanism to provide a signal to turn left, and a right-side vibration mechanism. as an indication to the right. The vibration mechanisms can be attached to other devices or parts of the user's body.

Figure 44 shows a process 4402 that describes a use of three-dimensional sounds to guide the user in navigation along a desired route. At block 4404, the SI module 302 determines a current location of a user along a route,

135 within space. At block 4406, the SI module 302 generates a three-dimensional sound. Three-dimensional sound creates a perception, by the user, that three-dimensional sound emanates from at least one specific location within space. At block 4408, the SI module 302 supplies the three-dimensional sound to the user. The three-dimensional sound helps the user to navigate along the route.

Figure 45 shows a process 4502 that describes one way that the path guide module 420 can use the three-dimensional sounds to guide the user along a desired path. At block 4504, path guide module 420 determines a current direction in which the user is heading. At block 4506, path guide module 420 identifies a desired direction in which the user expects to be directed. At block 4508, the path guide module 420 determines a difference between the current direction and the desired direction, to provide offset information. At block 4510, path guide module 420 uses sound generation module 414 to generate a periodic three-dimensional sound (eg, a heartbeat sound), based on the deflection information. The three-dimensional sound directs the user in the desired direction. Three-dimensional sound can have one or more properties (such as pitch) that depend on the extent to which the user has deviated from the desired direction.

Figure 46 shows a 4602 process that describes one way the SI 302 module can use three-dimensional sounds.

136 to identify and advertise lOls. At block 4604, the SI module 302 determines a set of items of interest (10s), each item of interest corresponding to an entity or event or piece of information that is relevant to the user in a current context. At block 4606, the SI module 302 generates a three-dimensional sound for each item of interest. The user will perceive the sound as coming from the same physical location (or locations) which, in turn, is associated with the item of interest, for example, by perceiving a sound associated with a warehouse as coming from the warehouse location .

In one case, the SI module 302 can apply the process 4602 as a background service while the user travels a route. For example, the RID module 426 can alert the user to the existence of lOls when the user draws close enough to the locations associated with these lOls, governed by any distance-based parameter settings.

In another scenario, as indicated in block 4608, the scanning module 430 may apply the aforementioned general process 4602 through the RID module 426 to identify a set of 10s that are associated with a subspace to which a user's attention is directed. current, or supposedly currently directed. In another scenario, as indicated in block 4610, the orientation module 432 may apply the aforementioned general process 4602 via the RID module 416 to identify a set of 10s that are associated with an entire space around the user in the

137 current moment, without reference to the user's current focus of interest (because the user is now interested in the entire volume of space around him or her at that moment). The user can invoke and suspend the reading of the scanning module 430 or the orientation module 432 by issuing appropriate instructions, for example, through the earpiece 108 or the user device 106.

As a final topic, the previous examples were based on the assumption that the position of the virtual sound-producing entity (for example, the virtual sound source) in space is stable with respect to the location of the listener, at least in the course of three-dimensional sound supply. This may be a valid assumption in many cases, but it may not be true for all situations.

For example, consider the following scenarios. In a first case, the user can hear a three-dimensional sound that describes (or otherwise refers to) a fixed-position entity, as the user passes the entity in a vehicle of any type (for example, in a train ). For example, the user may listen to a message describing the Seattle Space Needle as the user is traveling as a passenger in a car, in any direction relative to the Space Needle. In a second case, you may hear a three-dimensional sound that describes (or otherwise refers to) a fixed-position entity, while the user stands still, but nevertheless moves his head, in such a way that the position of the user's left and right ears

138 it is changing relative to the location of the fixed position entity. In a third case, a user can hear a three-dimensional sound that describes (or otherwise refers to) a moving entity, relative to a fixed position of the listener. For example, the user may listen to a message announcing the arrival of an aircraft moving down a runway, while the user remains in a fixed position at the airport terminal. In a fourth case, both the IOI location and the user may be in motion during the delivery of the audio information.

Also note that an IOI does not always need to correspond to a physical entity, such as a Space Needle or a moving aircraft. For example, a virtual IOI in motion may correspond to a virtual billboard that delivers a spatial message that the user perceives as going down the street towards the entrance of a nightclub, prompting the user to enter that establishment. Furthermore, everything that is discussed here regarding the sounds associated with lOls applies equally to other sounds, such as the sound of periodic beats.

To cope with any of the above situations, the SI 302 module can dynamically update the three-dimensional sound information that is being produced in the course of its delivery, to reflect, in each instance of time, the relative position between the user's head. and the IOI (for example, the location of the Space Needle or the movement of aircraft, in the examples cited

139 previously). The SI 302 module can perform this task in different ways. In a non-limiting aspect, the SI module 302 can perform the following operations in an iterative process through the delivery of the audio information: (a) determine the position of the user's head in relation to the IOI, to provide position information relative; (b) selecting an HRTF (per ear) based on the relative position of information; (c) convolving any audio information to be delivered at the current time by the selected HRTF, to produce a three-dimensional sound that is suitable based on the relative position information.

To cite a specific example, consider a message that takes 5 seconds to deliver to the user. The SI module 302 can determine, for each second of that delivery, the relative position between the user's head and the IOI. The SI module 302 can use that information to produce a three-dimensional sound for every second of message delivery that is based on the relative position between the user's head and the IOI at that time instance.

More generally, the aforementioned iterative processing can be performed at different update rates. The update rate can depend on the speed at which the relative position between the listener and the IOI changes. That is, the SI module 302 can update HRTFs at a relatively fast rate for large changes relative to relative position information, as

140 a slower pace for slower changes. The update rate can also take into account the amount of processing resources that are available in the system 102 to update the three-dimensional sound information.

The SI 302 module can also apply various techniques that are designed to speed up previous dynamics processing (which can put a large processing load on your resources). For example, in some cases, the system 102 can precompute a dynamic sound for at least one predicted trajectory; each path defines some type of relative motion between a listener and the IOI, due to: (a) motion of the listener in space; or (b) the movement of the IOI in space; or (c) the movement of both the listener and the IOI. For example, suppose that a train passes an IOI, generally at the same speed each day. In this case, it is possible to pre-calculate a three-dimensional sound (per ear) that is predicted in a dynamically changing HRTF. This three-dimensional sound takes into account a supposed progression of the user's head as the train moves along the tracks, and can be based on the assumption that the user's head has a certain fixed orientation as the user moves along. along the tracks. The SI module 302 can launch that three-dimensional sound when it detects that a listener position reaches a predetermined trigger location relative to the IOI, such as a location along the tracks that is a predetermined distance from the IOI.

141

This form of operation can be expanded to take into account the different scenarios. For example, the system 102 can pre-calculate different dynamic three-dimensional sounds to take into account for different train speeds, different fixed head orientations (for example, indicating whether the user is searching in a straight line during message delivery, or looking out the window, etc.), and so on.

The above situation can also be extended to a more general setting where the movement between the user and an IOI can be decomposed into a number of possibilities, allowing pre-calculation of the respective dynamic three-dimensional sounds for these possibilities. The SI module 302 can then initiate any pre-calculated sound when it determines that the user's current context matches one of the predetermined possibilities.

In another case, the SI module 302 may dynamically perform processing to predict user movement relative to the location of an IOI, for example, based on the current address of the user. The SI module 302 can then dynamically precompute a three-dimensional sound based on a projected user path relative to the IOI.

Other techniques can be used to speed up the calculation of dynamically changing three-dimensional sounds. In some implementations, for example, the SI module 302 can trace over the enhanced processing capabilities of the

142 remote processing 112, for example, by making use of parallel processing performed by those resources 112.

In summary, the above features contribute to the aforementioned goal of allowing the user to move efficiently, safely, and fun through their environment. For example, features provide the user with different categories of information as the user walks through the environment, using different presentation modes. The functionality presents this information to the user in an easy-to-use manner that avoids overwhelming the user with too much information at any given time. The use of three-dimensional sounds further enhances the user's ability to understand the link between the information being provided and objects, regions, and events in the environment.

E. Use of Beacons to Assist Users in Interaction with their Environments

The following section provides additional information about the operation of the beacon-based guidance module 422. To repeat, the beacon-based guidance module 422 provides a strategy for guiding the user through a space making use of beacons that have defined ranges. respective. The space can correspond to an interior space, for example, a space defined by the interior of one or more buildings or structures of any type. Alternatively, or in addition, the space can correspond to an outdoor space.

143

Beacons can emit electromagnetic radiation of any kind, such as radio waves, infrared waves, etc. In other cases, beacons can emit sound waves (for example, in the ultrasound range). Also, in some cases, beacons can generate signals according to any protocol or combination of protocols, such as the BLUETOOTH protocol, the Wi-Fi protocol, etc. For example, without limitation, beacons may correspond to BLUETOOTH Low Energy Beacons (BLEs). Additionally, a beacon can have a range of any desired depth, to better suit the target environment in which it is deployed. In some indoor configurations, for example, beacons are chosen with relatively short ranges, for example ranges of one or more meters. In some cases, a declared beacon range could be based on the implicit or explicitly stated assumption that its signals are going to be detected by a certain type of user device (such as a particular smartphone) or a particular class of user devices ( such as a particular class of smartphones), which have known and stable signal reception characteristics.

In another case, each beacon is a passive device (such as a passive RFID) that can be interrogated by user device 106 (or some other interrogation device) when the device is within a set distance to the beacon. That prescribed distance corresponds to the range of the device. However, for ease of explanation, the remaining description will assume that each

144 beacon actively emits a signal.

In an application, each beacon has a code that defines its identity. The beacon may constantly or periodically (or on a demand basis) emit signals announcing its code (and / or any other application-specific information), which can be detected by receiving devices within range of the beacon. For example, consider a BLE beacon that has a range of approximately one meter. A receiving device that lies within that range can detect the beacon signal and read its particular code.

In the example of Figure 47, an indoor (and / or outdoor) environment is characterized by a collection of corridors and obstacles (eg, obstacle 1, obstacle 2, and obstacle 3). This scenario involves pre-filling the environment with a collection of beacons (for example, b-ι, b_2l... b₇), for example, placing the beacons at each intersection of two or more corridors. The outward-most dashed circle surrounding each beacon represents the range of the beacon. As noted above, a receiving device that is within range of a beacon can correctly detect the beacon; otherwise, the receiving device will not be aware of the beacon. Note that, in a single illustrative implementation, the beacon ranges do not overlap. This feature is advantageous because it removes any ambiguity as to the user's location at any given time. In other words, the user's device cannot simultaneously detect two or

145 more beacons, because the ranges of these beacons do not overlap, and the user cannot exist simultaneously in two places at the same time. In some environments, the non-overlapping nature of the beacon ranges can be ensured by also taking into account the nature of the user device (or the class of devices) that will be receiving the signals emitted by the beacons.

As a first preliminary step, the locations of all beacons in the environment of Figure 47 can be loaded into data store 424. Data store 424 may also store the codes associated with those beacons.

As a second preliminary step, the system 102 can use any route planning tool to generate a route through the environment of Figure 47. The route planning tool can apply any algorithm to accomplish this task. For example, the routing planning tool may express the search space defined by corridors and obstacles as a graph that has a plurality of nodes and links. That is, the links represent the corridors and the nodes represent the corridor intersections. The routing planning tool can then use any algorithm (such as the well-known Dijkstra algorithm) to find the most efficient route through the graph. Once the route is defined, the beacon-based guidance module 422 can identify a subset of beacons (hereinafter referred to as route-specific beacons) that will be traversed.

146 by the user traveling on the route and store the identities of those beacons. In the case of Figure 47, the user is expected to find all beacons, but this is generally not the case (clarified below).

Now suppose that the user embarks on the route using the system 102 described above as a guide. System 102 may provide real-time guidance to the user in the manner described above via headset 108 and / or user device 106. In this case, however, system 102 uses a different technique to determine current location. of the user at all times, compared to the techniques described above.

More specifically, beacon-based guidance module 422 constantly scans the environment to determine if the user is in range of any beacon. If so, beacon-based guidance module 422 identifies the user's current location as the current location of the beacon that has been detected. In other words, beacon-based guidance module 422 knows, a priori, the code associated with the detected beacon and its location in the environment. If the user's hearing aid 108 or a user device 106 detects the presence of that beacon (based on the code transmitted by the detected signal), then the beacon-based guidance module 422 will be able to hypothesize that the user has the same location than the beacon detected.

At this time, the beacon-based guidance module 422 can make use of the services of the trajectory guidance module.

147

420 to direct the user in the desired direction by means of a three-dimensional sound and / or based on other guidance information (such as a non-three-dimensional sound, presented instructions, etc.). For example, suppose that the user's user device 106 detects that it is in the beacon range b₄. The trajectory guidance module 420 will determine, based on the predetermined travel information, that the next waypoint along the user's journey corresponds to beacon b₅. The path guide module 420 can then generate a three-dimensional sound that the user will perceive as coming from the right side of the user, which serves to direct the user to the next waypoint (e.g. beacon b₅). The user will interpret this signal as an instruction that he or she should turn right. The user will continue in this direction until he or she finds another beacon (for example, b₅), at which time the user's address can be updated.

In some cases, a user may deviate in an unexpected way from a planned trajectory in such a way that he or she falls out of range of the beacon, that he or she was scheduled for the next encounter. Figure 48 represents one way to approach this situation. The strategy is to increase the number of beacons along the user's intended route, which has the effect of increasing the frequency at which the user's current position is evaluated, and therefore decreasing the chances that the user may go off course (for example, to the extent that he or she

148 falls outside the range of a beacon he or she expects to find). Note that Figure 48 shows only those beacons that the user expects to travel on the planned route. But the environment can include additional beacons (not shown) that the user does not expect to pass through.

The opportunity for a user to go wrong walks is not great in Figures 47 and 48, for example, because the user's choices are significantly restricted by obstacles. However, a user can get confused and take a wrong turn, causing him to abandon the planned route. Or the user may deliberately decide to deviate from the planned route. For example, at the intersection associated with beacon b₉ in Figure 48, the user can take a left turn, instead of a right turn. Therefore, finally the user can find an off-route beacon (not shown), which he does not expect to find.

Beacon-based guidance module 422 can address this situation in different ways. In one case, beacon-based guidance module 422 may inform the user that guidance cannot be provided to the user, because the user appears to have wandered off the path and it is no longer possible to determine the heading and the user's intent. The beacon-based guidance module 422 can also query the user if he or she intends to proceed with the original route defined by the environment.

Otherwise, if enough information can be obtained about the current location, heading, and user intent, the module

149 beacon-based guidance guide 422 can direct the user back to the planned route. For example, beacon-based guidance module 422 can determine the direction that the user appears to be currently heading (even if it appears to be wrong), by forming a trajectory based on a set of the user's most recently known positions. Beacon-based guidance module 422 may use such header information, along with information about the user's current position (if known), to direct the user to the planned route. Or the beacon-based guidance module 422 may query the user to determine if he or she intends to search for the planned route, or select another route.

In either case, beacon-based guidance module 422 can also rely on other evidence of the user's current location and departure (in addition to information provided by the BLE beacons), when that information is available. For example, beacon-based guidance module 422 can collect that information from GPS sensor sources, dead reckoning techniques, and so on. In other cases, it is assumed that at least some of these additional sources are not available or are not reliable.

Figure 49 depicts another situation in which the space over which the user walks is more open ended, compared to the examples in Figures 47 and 48. The space is pre-filled with a collection of short-range beacons. For example, the space can be filled with a regular array of those beacons, if it is

150 allowed by the physical characteristics of the space. Beacons have ranges that do not overlap.

Again, a route planning tool can generate a 4902 route through space based on any objective input and use any route planning algorithm for this purpose. Beacon-based guidance module 422 then determines the beacons that the user expects to travel as he or she travels the planned route 4902. These beacons are represented as solid black beacon symbols.

During actual travel of route 4902, beacon-based guidance module 422 performs the same function as previously described. That is, when beacon-based guidance module 422 determines that the user has entered the range of a beacon along the expected route (such as beacon b₂₂) then accepts the location of that beacon as the user's current location. It then recalculates the user's trajectory and updates the 3D heartbeat sound (and / or other guidance information) to guide the user to the next beacon (for example, beacon b₃₃).

The risk of the user going off the road in the case of Figure 49 is greater than in the case of Figure 48, because the user is given a greater degree of freedom in which he can err. Figure 50 describes one way to address this situation. Here, the beacon-based guidance module 422 defines the beacons (represented as solid black beacon symbols) that the user may encounter along a planned route 5002 in a more general way, for

151 For example, also encompassing the surrounding beacons that lie on both sides of the most optimal route 5002. The beacon-based guidance module 422 can continue to provide guidance to the user if he or she travels in the range of these surrounding beacons, under the assuming the user is still trying to adjust to planned route 5002, but drifted slightly off course. Beacon-based guidance module 422 can only generate an error condition when the user wanders beyond the limits associated with surrounding beacons.

In another application, beacon-based guidance module 422 may form a postulate as to the user's desired destination based on beacons that the user has already encountered along their path, thus far. For example, beacon-based guidance module 422 may form a path based on the beacon locations found thus far. Beacon-based guidance module 422 can determine a probable intermediate or final destination to which the path points, for example, by extending the path along its current direction. Beacon-based guidance module 422 may then ask the user if he or she wants to search for a path to the putative destination. If so, then the beacon-based guidance module 422 can subsequently provide the same type of navigation assistance described above that assists the user in reaching the identified destination.

In any of the cases described, the guide module based on

152 beacon 422 may also take into account historical information about the user's previous travel habits and / or historical information about the travel habits of others (with respect to a specified environment). That information, when available, can provide further evidence of the user's intention to reach a desired destination.

Figure 51 shows a process 5102 describing one way of operation of beacon-based guidance module 422, for example, within the context of the types of environments of Figures 47-50. At block 5104, beacon-based guidance module 422 receives a particular beacon signal from a sensor (s) of a computing device operating at a current location within an environment. The computing device may correspond to the user device 108 or the handset 108, etc. As described above, the environment is filled with a plurality of beacons having, in an illustrative implementation, respective non-overlapping ranges. Furthermore, as a preliminary step, a route planning module may have defined a desired route, which is described by trip information. Or the desired route can be generated dynamically as the user traverses the environment, based on assumptions made regarding the intention of the intermediary or the user's final destination. In any case, that desired route traverses the ranges associated with a specific set of beacon routes, out of the total set of beacons in the environment.

153

At block 5106, beacon-based guidance module 422 determines, based on the particular beacon signal, whether the user is within range of one of the specific route beacons; this operation returns current location information when the user is within range. At block 5108, beacon-based guidance module 422 determines the next waypoint that the user expects to reach, based on the predetermined travel information, to provide next waypoint information. In some cases, the next waypoint may correspond to the next beacon along the user's predetermined journey. At block 5110, beacon-based guidance module 422 then determines address information based on current location information and next waypoint information. Direction information reflects a direction in which users are advised to reach the next landmark. At block 5112, beacon-based guidance module 422 generates audio information (and / or other guidance information) based on the address information. At block 5114, beacon-based guidance module 422 provides the audio information to the user, eg, as a three-dimensional heartbeat sound. The audio information helps the user to reach the next benchmark. The beacon-based guidance module 422 can perform the aforementioned functions, with the help of the trajectory guidance module 420 and the sound generation module 414.

154

Figure 52 shows a process 5202 that provides more details about one way that beacon-based guidance module 422 can determine the current location of the user within an environment. At block 5204, beacon-based guidance module 422 identifies a particular beacon code associated with a particular beacon signal that has been received. At block 5206, beacon-based guidance module 422 identifies, based on the particular beacon code, a beacon that is associated with the particular beacon code. At block 5208, beacon-based guidance module 422 identifies the location of the particular beacon based on stored information (in data storage 424), which identifies the beacon codes and respective beacon locations in the environment.

In summary, the above features contribute to the aforementioned goal of allowing the user to move safely and efficiently through their environment, especially in those situations where the user cannot rely on the other modes to determine their location (for example , based on the use of a satellite-based navigation system). Furthermore, the use of non-overlapping beacon ranges (in accordance with an illustrative implementation) provides an efficient mechanism to disambiguate the location of the user, since the user cannot simultaneously exist within the ranges of two or more beacons at the same time.

In the previous description of a first implementation,

155 assumes that user device 106 (and / or handset 108) receives, at any given time, a signal transmitted by zero beacons or a single beacon, but not a plurality of beacons. In other implementations, the above feature can be relaxed in different ways.

For example, in a second implementation, beacon-based guidance module 422 will conclude that user device 106 and / or headset 108 is within range of a particular beacon if it receives a signal from that beacon that has a signal strength that is above a certain threshold. But unlike the first implementation, the beacon-based guidance module 422 can also simultaneously receive the weakest signals from one or more other beacons in the environment, where the strength of each of those signals is below the threshold. preset. In this scenario, the environment is filled with beacons that have positions such that, at any given time, beacon-based guidance module 422 will receive: (1) no signal that has a signal strength above the threshold; or (2) just a signal that has a signal strength that is above the threshold. In practice, the second implementation works in the same way as the first and offers the same benefits, for example, by providing a binary mechanism to disambiguate the user's location at any given time, assuming the user is within the range of one of the beacons. The beacons in the second implementation can therefore be considered as

156 functionally or effectively non-overlapping due to the above behavior. And consequently, any reference to non-overlapping as used herein is understood to encompass both the case where the beacons have ranges that literally do not overlap, as well as the case where the ranges can be considered non-overlapping because the user device 106 and / or handset 108 at most can receive a signal from a beacon that has a signal strength above the preset threshold.

In a third implementation, beacon-based guidance module 422 can receive, at any given time, signals from any number of beacons having any arbitrary signal strengths. The set of signals (and intensities) at a given location defines the signal profile information for that location. In a preliminary operation, beacon-based guidance module 422 can store signal profile information for each navigable location in the environment, for example, constituting information about the signals and their respective intensities at that location. Collectively, the stored information constitutes a profile map of the environment. During navigation, beacon-based guidance module 422 can determine the signals it is currently receiving at a given location to provide current signal profile information. The beacon-based guidance module 422 can then use the current signal profile information as a key to find the location that has the signal profile information.

157 closest match. That location defines the likely location of the user at any given time. In some environments, the intensities of the signals emitted by beacons and / or the ability to detect those signals may vary over time for various environmental specific reasons. Beacon-based guidance module 422 can address this issue by comparing standard versions of the signal profile information. Note that the positions of the beacons in the third implementation need not meet the non-overlapping constraints associated with the first or second implementations described above.

F. Representative Computing Functionality

Figure 53 shows the compute functionality 5302 that can be used to implement any aspect of the system 102 stated above. For example, the type of computing functionality 5302 shown in Figure 53 can be used to implement any of the user device 106, any of the remote processing resources 112, the processing equipment used by the handset 108, the computing device separate user 110, and so on. In all cases, the compute functionality 5302 represents one or more physical and tangible processing mechanisms.

The compute functionality 5302 may include one or more processing devices 5304, such as one or more central processing units (CPUs), and / or one or more processing units.

158 graphics processing (GPU), and so on.

The compute functionality 5302 can also include storage resources 5306 to store any type of information, such as code, settings, data, etc. Without limitation, for example, storage resources 5306 can include any RAM of any type (s), ROM of any type (s), flash drives, hard drives, optical drives, and so on. More generally, any storage resource can use any technology for storing information. Additionally, any storage resource can provide retention of volatile or non-volatile information. Additionally, any storage resource can represent a fixed or removable component of 5302 compute functionality. The compute functionality 5302 can perform any of the functions described above when the processing devices 5304 perform instructions stored on any storage resource or combination of the storage resources.

In terms of terminology, any of the storage resources 5306, or any combination of the storage resources 5306, can be considered as a computer-readable medium. In many cases, a computer-readable medium represents some kind of tangible, physical entity. The computer-readable medium term also encompasses, for example, propagated signals, for example, transmitted or received through a

159 physical conduit and / or air or other wireless medium, etc. However, the specific terms computer-readable storage medium and computer-readable medium device expressly exclude propagated signals per se, although including all other forms of computer-readable media.

The compute functionality 5302 also includes one or more drive mechanisms 5308 for interacting with any storage resource, such as a hard disk drive mechanism, an optical disk drive mechanism, and so on.

The 5302 compute functionality also includes a 5310 input / output module to receive different inputs (through 5312 input devices) and to provide various outputs (through 5314 output devices). Illustrative input devices include a keyboard device, a mouse input device, a touch-sensitive input device, a digitizing keyboard, one or more video cameras, one or more depth cameras, a gesture recognition mechanism. free space, one or more microphones, a voice recognition mechanism, any motion detection mechanism (for example, accelerometers, gyros, etc.), and so on. A particular output mechanism may include a display device 5316 and an associated graphical user interface (GUI) 5318. Other output devices include a

160 printer, a model generation mechanism, a touch output mechanism, a file mechanism (for storing output information), and so on. The compute functionality 5302 may also include one or more network interfaces 5 5320 for exchanging data with other devices through one or more communication conduits 5322. One or more common communication conductors 5324 communicatively couple the above components together. described.

Communication conduit (s) 5322 can be implemented in any form, for example, via a local area network, wide area network (eg, Internet), point-to-point connections, etc., or any combination of the same. Communication conduit (s) 5322 can include any combination of wired links, wireless links, routers, gateway functionality, name servers etc. governed by any protocol or combination of protocols.

Alternatively, or in addition, any of the functions described in the previous sections can be carried out, at least in part, by one or more hardware logic components.

For example, without limitation, the 5302 compute functionality can be implemented using one or more of the following: Field Programmable Gate Arrays (FPGAs); Application Specific Integrated Circuits (ASIC); Application Specific Standard Products (SPs); System on a chip (SOC) systems; Devices

Complex Programmable Logics (CPLDs), etc.

161

In conclusion, the following summarizes the respective aspects of the system 102 described above.

According to a first aspect, a system is described for helping a user to interact with a space. The system includes a headset to present audio information to the user as the user interacts with the space. The headset, in turn, has at least one set of input mechanisms to receive commands from the user, the command invoking functions related to the respective space interaction to be performed by means of a space interaction module.

According to a second aspect, the headset uses a bone conduction mechanism to deliver the audio information to the user.

According to a third aspect, at least one command instructs the space interaction module to initiate a listening mode. In listening mode, the space interaction module receives instructions spoken by the user.

According to a fourth aspect, at least one command instructs the space interaction module to stop delivering the audio information.

According to a fifth aspect, at least one command instructs the space interaction module to repeat one or more articles of audio information delivered earlier.

According to a sixth aspect, at least one command instructs the space interaction module to initiate a scan mode. In

162 In browse mode, the space interaction module provides information on a set of items of interest that are associated with a subspace to which the user's attention is currently directed.

According to a seventh aspect, at least one command instructs the space interaction module to initiate an orientation mode. In orientation mode, the space interaction module provides information about a set of items of interest that are associated with an entire space around the user, at the current time.

According to an eighth aspect, at least one command instructs the space interaction module to perform a more information function. The space interaction module performs the function of more information by providing additional information on a topic associated with a previously submitted article of interest.

According to a ninth aspect, at least one input mechanism can invoke a plurality of functions, depending on the way a user interacts with the input mechanism.

According to a tenth aspect, the space interaction module is implemented by the handset.

According to an eleventh aspect, the headset further includes one or more of the following: a headset orientation determining mechanism for determining headset orientation, to provide headset orientation information;

163 an earphone movement determining mechanism for determining a movement of the earphone, to provide information about the movement of the earphone; and / or an earpiece position determining mechanism for determining an earpiece position, to provide earpiece position information.

According to a twelfth aspect, the system also comprises at least one user computing device. The user computing device provides a screen output mechanism for displaying information regarding the interaction, by the user, with space. The user computing device implements at least part of the functionality associated with the space interaction module. Furthermore, the headset includes a first communication mechanism for communicating with the user computing device and the user computing device includes a second communication mechanism for communicating with the headset.

In accordance with a thirteenth aspect, the user computing device is configured to perform processing based on earpiece orientation information and / or earpiece movement information and / or earpiece position information to generate a result. and to transmit the result to the handset.

According to a fourteenth aspect, the user counting device further includes: an orientation determination mechanism for determining an orientation of the user counting device.

164 user counting, to provide device orientation information; a device movement determining mechanism for determining a movement of the user computing device, to provide information on the movement of the device; and / or a device position determination mechanism for determining a position of the user computing device, to provide information on the position of the device.

In accordance with a fifteenth aspect, the headset is configured to perform functions based on device orientation information and / or device movement information and / or device position information, as supplied by the computing device of user, when counterpart information is not available from the handset.

According to the sixteenth aspect, at least the user's headset and / or computing device is configured to receive information and / or use the functionality provided by an external computing device.

According to the seventeenth aspect, a method, implemented by a system, is described to help a user interact with a space. The method includes: receiving an instruction based on the actuation, by the user, of at least one input mechanism that is coupled to a headset; perform a function based on the instruction, to provide a result; and based on the output results, provide information

165 audio, through the headset, which helps the user to interact with the space.

According to an eighteenth aspect, the operation to perform the function involves the performance of the function using a space interaction module that is implemented, at least in part, by the handset.

According to a nineteenth aspect, the operation to perform the function involves the performance of the function using a space interaction module that is implemented, at least in part, by a separate user computing device that is communicatively coupled to the handset.

In accordance with a twentieth aspect, a headphone is described that includes: an audio output mechanism for the presentation of audio information that assists a user as the user interacts with a space; and a set of input mechanisms for receiving commands from the user, the commands invoking respective space interaction related functions that are performed by a space interaction module. At least one command instructs the Space Interaction module to start a scan mode, and at least one command instructs the Space Interaction module to start an orientation mode. In browse mode, the space interaction module provides information about a set of items of interest that are associated with a subspace to which the user's attention is currently directed. In orientation mode, the

166 Space interaction provides information on a set of articles of interest that are associated with an entire space around the user, at the current moment.

A twenty-first aspect corresponds to any combination (eg, any permutation or subset) of the first to twenty-first aspects mentioned above.

According to a twenty-second aspect, one or more computing devices (and / or one or more headsets) are provided to implement any of the first to twenty-first aspects.

According to a twenty-third aspect, a system is provided to implement any of the first to twenty-first aspects.

According to a twenty-fourth aspect, one or more computer-readable storage media is provided that includes logic that is configured to implement any of the first to twenty-first aspects.

According to a twenty-fifth aspect, one or more means are provided for implementing any of the first to the twenty-first aspects.

In addition, to conclude, the functionality described in this document may employ various mechanisms to ensure that user data is handled in a way that conforms to applicable laws, social norms, and user expectations and preferences. For example, the functionality may allow a user to expressly choose (and then not to expressly choose)

167 of the functionality provisions. The functionality may also provide adequate security mechanisms to ensure the privacy of user data (such as data sanitization mechanisms, cryptic encryption mechanisms, password protection mechanisms, etc.).

Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are described as illustrative ways to implement the claims.

168