Contextual user interface.

(54) Title: CONTEXTUAL USER INTERFACE. (54) Title: CONTEXTUAL USER INTERFACE.

(57) Summary

The embodiments of the present invention analyze a context in which a user interacts with a computer interface and automatically optimizes the interface for the context. The controller or control mode that the user selects for the interaction can define the context, in part. Examples of control modes include gestures, audio control, use of companion devices, and use of dedicated control devices, such as game controllers and remote controls. Different input devices are designed for different tasks. However, a user will frequently attempt to perform a task using a control input that is not adapted for the task. The embodiments of the present invention change the characteristics of the user interface to facilitate the user completing a desired task using the input device of the user's choice.

(57) Abstract

Embodiments of the present invention analyze a context in which a user interacts with a Computer interface and automatically optimizes the interface for the context. The controller or control mode the user selects for interaction may define the context, in part. Examples of control modes inelude gesturing, audio control, use of companion devices, and use of dedicated control devices, such as game controllers and remote Controls. The different input devices are designed for different tasks. Nevertheless, a user will frequently attempt to perform a task using a control input that is not adapted for the task. Embodiments of the present invention change the characteristics of the user interface to make it easier for the user to complete an intended task using the input device of the user's choice.

Institute

Mexican Property

Industrial _SE_ tÉCWTAtUA Oí ÉCÍJNOMIA

<img file="MX340692B_D0001.tif" />

<img file="MX340692B_D0002.tif" />

PATENT TITLE NO. 340692

Owner (s): MICROSOFT TECHNOLOGY LICENSING, LLC

Address: One Microsoft Way, Redmond, Washington, 98052, USA

Name: CONTEXTUAL USER INTERFACE

Classification: IC.8: A63F13 / 20; G06F3 / 01; G06F3 / 033; G06F3 / 041; G06F3 / 048; G06F9 / 44;

H04N21 / 4722

Inventor (s): SHIRAZ CUPALA; MARIO ESPOSITO; CYRUS KANGA; KATHLEEN PATRICIA

MULCAHY; MATTHEW ALAN MEDAK; NEIL WARREN BLACK; ALEXANDER D. TUDOR; ERIC CHRISTOPHER VOREIS; WILLIAM F. SCHIEFELBEIN;

ZHANGWEI XU; KENNETH ALAN LOBB; RONALD OSWIN PESSNER, JR .; JASON ROBERT TUCK; HEATHER GRANTHAM BRODIE; SZYMON PIOTR STACHNIAK; JOHN DARREN ELLIOTT; CLINTON CHI-WEN WOON; RUSSELL P. SONGCO; HOWARD B. NAGER; JOSHUA C ZANA; MARC VICTOR HADO AH; ALEXANDER JAMES SOULE

REQUEST

Number:

MX / a / 2014/014621

International filing date: May 24, 2013

PRIORITY

Country:

US

US

Validity: Twenty years

Date:

June 2012 December 21, 2012

61/654,63

13/725,92

Expiration Date: May 24, 2033

The reference patent is granted based on articles 1, 2, section 7 ñ 'racdón lll, and 59 of the Industrial Property Law.

Pursuant to article 23 of the Industrial Property Law, a twenty-year non-extendable term shall be present.

-Counted from the filing date of the international application i will be subject to the payment of the fee to keep the rights in force i;

«Who subscribes the pri Industrial Property

01/26/2004, 06/16/20 'I resented

Iterií

005,2!

ite title.-it does so based on the provisions of articles 6 fractions lll and 7 ° bis 2 of the Official River Law of the Federaetoo (D.OF.) 06/27/1991, amended on 08/02 / 1994, 10/25/1996, 26 / i2 / l9d7, 05/17/1999, 01/25/2006, 06/05/2009, 06/01/2010, 06/18/2010, 06/28/2010 , 01/27/2012 and 04/09/2012); articles 1, 3 fraction V Clause a), 4 and 12 “fractions I and lll of the Regulation of the Mexican Institute of Industrial Property (DOF 14/12/1999, amended on 07/01/2002, 07/15/19) 2004, 07/28/2004 and 09/07/2007); Articles 1, 3, 4, 5, section V, Section a), 16 sections I and III and 30 of the Organic Statute of the Mexican Institute of Industrial Property (DOF) 12/27/1999, amended on 10/10/2002, 07/29/2004, 08/04/2004 and 09/13/2007); 1, 3 and 5 subsection a) of the Agreement that delegates powers to the Deputy Directors General, Coordinator, Divisional Directors, Holders of the Regional Offices, Divisional Deputy Directors, Departmental Coordinators and other subordinates of the Mexican Institute of Industrial Property. (DOF 12/15/1999, amended on 02/04/2000, 07/29/2004, 08/04/2004 and 09/13/2007).

<img file="MX340692B_D0003.tif" />

Arenal No. 550, Step 1.

Coi. FueO or Santa María Tepepan Xochimiicc CP. '5020

Mexico City

Tol. (55) 53 3-4 0 ”00 ww.impi qob.mx

Issue Date: July 20, 2016

THE DIVISIONAL DIRECTOR OF PATENTS

<img file="MX340692B_D0004.tif" />

NAHANNY CANAL REYES

<img file="MX340692B_D0005.tif" />

<img file="MX340692B_D0006.tif" />

<img file="MX340692B_D0007.tif" />

MX / 2016/56995 ¿/ο / v ¡i

IMPI

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX340692B_D0008.tif" />

CONTEXTUAL USER INTERFACE

User interfaces for televisions, game consoles, media players, personal computers, and other devices have become increasingly complex. Devices have more and more applications and programs through which users can access an ever-expanding amount of content. Additionally, users have a variety of mechanisms to interact with interfaces. An interface adapted for one form of interaction may not be suitable or optimized for a second kind of interaction.

BRIEF DESCRIPTION OF THE INVENTION

This Brief Description is provided to introduce a selection of concepts in a simplified form which is further described below in the detailed description. This Brief Description is not intended to identify key characteristics or essential characteristics of the claimed topic, nor is it intended to be used in isolation as an aid in determining the scope of the claimed topic.

The embodiments of the present invention analyze a context in which a user interacts with a computer interface and automatically optimizes the interface for the context. The controller or control mode that the user selects for

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL interaction can define the context, in part. Examples of control modes include gesticulation, audio control, use of companion devices, and use of dedicated control devices, such as game controllers and remote controllers. Different input devices are designed for different tasks. However, a user will frequently attempt to perform a task using a control input that is not adapted for the task. The embodiments of the present invention change the characteristics of the user interface to facilitate the user completing a desired task using the input device of the user's choice.

In addition to adapting the interface to the chosen interaction mechanism, the interfaces can be further optimized based on ambient conditions. Environmental conditions include the location of a person in relation to other people and objects in the room as well as the content presented at the one or more interfaces. For example, an interactive menu could be presented in part of a screen that is not being blocked by a person or object standing between the screen and the user who is trying to interact with the content on the screen. That way, if the bottom right corner of the screen was blocked by a person, then help information would be presented on a different part of the screen. Similarly, if a user has a phone or other object in their right hand, then a gesture-based menu can be optimized for use with a left hand. By

<img file="MX340692B_D0009.tif" />

For example, menu objects can be moved to the left side of the screen so that the user does not have to stretch their body to visually select interface objects.

The embodiments of the present invention can use various cues to evaluate the present context. Illustrative signals include color image data collected from a camera, 3D image data collected from a depth camera or other 3D capable acquisition device, audio signals, and status information from one or more computing devices and controllers in the closeness.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention are described in detail below with reference to the attached figures, where:

Figure 1 is a block diagram of an illustrative computing environment suitable for implementing modalities of the Invention;

Figure 2 is a diagram illustrating a variety of means of communication between game consoles, game services, and companion devices, in accordance with an embodiment of the present invention;

Figure 3 is a diagram showing a suitable computing environment to use an interaction context to optimize a user interface, according to a modality of

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0010.tif" />

the present invention;

Figure 4 is a diagram showing a user interacting with the interface optimized to use gestures, in accordance with an embodiment of the present invention;

Figure 5 is a diagram showing a user interacting with an interface that is automatically optimized for use with a companion device that is recognized by the game console, in accordance with one embodiment of the present invention.

Figure 6 is a diagram showing a user interacting with an optimized help interface for use with a game controller, in accordance with an embodiment of the present invention;

Figures 7 is a diagram showing a user interacting with an optimized help interface for use with an audio control, in accordance with an embodiment of the present invention;

FIG. 8 is a diagram showing a user interacting with an optimized help interface for use with a companion device, in accordance with an embodiment of the present invention;

Figure θ is a diagram showing a user interacting with an interface optimized for left-handed gestures, in accordance with an embodiment of the present invention;

Figure 10 is a diagram showing a user interacting with an interface optimized for right-handed gestures,

IMPI Mexican institute OF INDUSTRIAL PROPERTY

<img file="MX340692B_D0011.tif" />

in accordance with an embodiment of the present invention; ......... Figure 11 is a diagram showing a user interacting with an optimized interface as a result of a user-driven object, according to a embodiment of the present invention;

Figure 12 is a diagram illustrating using the identification of an object or clothing within the environment to generate a corresponding interface, in accordance with an embodiment of the present invention;

Figure 13 is a flow chart showing a method of displaying an interface that is optimized for a present context, in accordance with an embodiment of the present invention;

Figure 14 is a flow chart showing a method for displaying an infern that is optimized for a present context, in accordance with an embodiment of the present invention; and

Figure 15 is a flow chart showing a method of displaying an interface that is optimized for a present context, in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

The subject matter of the embodiments of the invention is described specifically herein to satisfy regulatory requirements. However, the same description is not intended to limit the scope of this patent. Rather, the inventors have contemplated

IMPI Mexican institute OF THE INDUSTRIAL

<img file="MX340692B_D0012.tif" />

that the claimed topic may also be cepiesentadu in — other ……

ways, to include different steps or combinations of steps similar to those described in this document, in conjunction with other present or future technologies. Furthermore, although the terms step and / or block as used here may be used here to connote different elements of methods employed, the terms should not be construed as implying any particular order between two or more of several steps described herein unless and except when the order of individual steps is explicitly described.

The embodiments of the present invention analyze a context in which a user interacts to interact with a computer interface and automatically optimize the interface for the context. User interactions with a user interface may include giving different degrees of attention to the interface, ranging from ignoring the interface to full attention. The controller or control mode that the user selects for interaction can define the context, in part. Examples of control modes, including gesticulation, audio control, use of companion devices, and use of dedicated control devices such as game controllers and remote controls, different input devices are designed for different tasks. However, a user will frequently attempt to perform a task using a control input that is not adapted for the task. The modalities of the present invention will change the characteristics of the interface of

MEXICAN INSTITUTE Dt THE PROPERTY

INDUSTRIAL user to facilitate the user to complete a desired task using the input device of the user's choice.

In addition to adapting the interface to the chosen interaction mechanism, the interfaces can be further optimized based on environmental conditions. Environmental conditions include the location of a person in relation to another person and objects in the room as well as the content presented at the one or more interfaces. For example, an interactive menu could be presented in part of a screen that is not being blocked by an object or person standing between the object and the user who is trying to interact with the content on the screen. That way, the lower right corner of the screen was locked by one person, so help information would be presented on a different part of the screen. Similarly, if a user has a phone or other object in his right hand, then he can optimize his gesture-based menu for use with his left hand. For example, menu objects can be moved to the left side of the screen so that the user does not have to stretch her body to virtually select interface objects.

The embodiments of the present invention can use various signals to evaluate the present context. Illustrative signals include color image data collected from a camera, 3D image data collected from a depth camera or other 3D capable acquisition device, audio signals, and status information from one or more imaging devices.

IMPI

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX340692B_D0013.tif" />

computing and drivers nearby.

Having briefly described an overview of embodiments of the invention, an illustrative operating environment suitable for use in implementing embodiments of the invention is described below.

Illustrative Operating Environment

Referring to the drawings in general, and initially to Figure 1 in particular, an illustrative operating environment for implementing embodiments of the invention is generally shown and designated as computing device 100. Computing device 100 is only an example of an environment It is not intended to suggest any limitations as to the scope of use or functionality of the invention. The computing device 100 should also not be construed as having any dependency or requirement that relates to any or combination of illustrated components.

The invention may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program components, which are executed by a computer or other machine, such as a personal data assistant or other portable device. . Generally, program components, including routines, programs, objects, components, data structures, and the like, refer to code that performs tasks.

<img file="MX340692B_D0014.tif" />

, or implements particular abstract data types. The embodiments of the invention can be practiced in a variety of system configurations, including portable devices, consumer electronics, general-purpose computers, specialty computing devices, etc. The embodiments of the invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

With continued reference to Figure 1, computing device 100 includes a common conductor 110 that directly or indirectly couples the following devices: memory 112, one or more processors 114, one or more display components 116, input / output ports ( l / O) 118, l / O components 120, illustrative power supply 122, radio 124, and sensor 126. Common conductor 110 represents what may be one or more common conductors (such as a common address conductor, common data conductor, or combination thereof). Although the various blocks in Figure 1 are shown with lines for clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would be more precisely gray and fuzzy. For example, a display component such as a display device can be considered to be an I / O component 120. Also, processors have memory. The inventors here acknowledge that such is the nature of the technique, and

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0015.tif" />

reiterate that the diagram in Figure 1 is merely illustrative of an illustrative computing device that can be used in connection with one or more embodiments of the invention. No distinction is made between such categories as workstation, server, laptop, portable device, etc., since all are contemplated within the scope of Figure 1 referring to a computer or computing device.

Computing device 100 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computing device 100 and includes both volatile and nonvolatile media, removable and nonremovable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and nonremovable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules, or other data.

Computer storage media includes RÁWT, ROM, EEPROM, flash memory or other memory technology, CDROMs, digital versatile discs (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other devices. Mexican iNSTmrro storage

OF THE PROPERTY

INDUSTRIAL magnetic. Computer storage media does not comprise a propagated data signal.

The communication media typically represent computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any means of information delivery. The term modulated data signal means a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. By way of example, and not limitation, the communication media includes wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above must also be included within the scope of computer readable media.

Memory 112 includes computer storage media in the form of volatile and / or nonvolatile memory. Memory 112 can be removable, non-removable, or a combination thereof. Illustrative memory includes solid state memory, hard drives, optical disc drives, etc. Computing device 100 includes one or more processors 114 that read data from various entities such as common conductor 110, memory 112, or l / O components 120. Presentation component (s) 116 present data indications to a user or other device. The

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0016.tif" />

Illustrative display components 116 include a display device, horn, printing component, vibrating component, etc. The l / O ports 118 allow the computing device 100 to be logically coupled to other devices including l / O components 120, some of which can be incorporated. Illustrative l / O 120 components include a microphone, joystick, game pad, scanner, hard / soft button, touch screen, etc.

Radio transmissions 124 are received radio communications. The computing device 100 can be a wireless terminal adapted for received communications and media over various wireless networks. Computing device 100 can communicate via wireless protocols, such as code division multiple access (CDMA), global mobile system (GSM), or time division multiple access (TDMA), as well as others, to communicate with other devices (not shown in Figure 1). Radio communications can be a short-range connection, a long-range connection, or a combination of a short-range and long-range wireless telecommunications connection. When referring to short and “targa” connection types, it is not intended to refer to the spatial relationship between two devices. Instead, we are generally referring to short-range and long-range as different categories, or types, of connections (i.e., a primary connection and a secondary connection). A short connection

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0017.tif" />

Range can include a WI-FI® connection to a device (eg, mobile access point) that provides access to a wireless communication network, such as a WLAN collection that uses the 802.1 protocol. 1. A Bluetooth connection to another wireless device. computation is the second example of a short-range connection. A long-range connection can include a connection using one or more of CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

Sensor 126 observes device and environmental characteristics and conditions. Illustrative sensors comprise accelerometers, gyroscopes, GPS / Location, proximity sensors, light sensors, and vibration sensors.

Multiple Device User Experience

Turning now to Figure 2, a networked operating environment 200 is shown comprising multiple computing devices that can provide a company experience, in accordance with embodiments of the present invention. Environment 200 includes primary device 210, companion devices 252, 254, 256, and 258, a wireless router 250, a base station 260, a network 230, and a companion experience server 220. These devices are simply non-restrictive and are not intended to be limiting.

The primary device 210 may be a game console, media console, or other suitable title-presenting computing device. Titles can be games, movies,

IMPI

MEXICAN INSTITUTE PE THE PROPERTY

INDUSTRIAL

<img file="MX340692B_D0018.tif" />

applications, music, videos, television shows, and other media content. The game console can be coupled to a presentation, such as a television.

Companion devices 252, 254, 256, and 258 are computing devices. A companion device, as used in this application, is a personal computing device that provides a second presentation. Examples include laptops, PCs, netbooks, smart phones, electronic readers, PDAs, and tablets. A companion experience allows the user to interact with content related to media playback on a primary device 210 through interfaces on the companion device. The company experience can allow the user to control the presentation of media through an interface on the company device.

Company experience server 220 facilitates company experiences by providing company content, company applications, registering and authenticating company devices, facilitating communications between primary devices and company devices, and performing other tasks. The company experience server can be accessed through a wide area network, such as the Internet.

Companion devices 252, 254, 256, and 258 can communicate directly with primary device 210 via Bluetooth or through a wired connection. Companion devices could also communicate over a network

IMPI Mexican Institute Dt LA PROWLDAD INDUSTRIAL

<img file="MX340692B_D0019.tif" />

local wireless generated by the 2bu Wireless carrier. These connections must be direct or they could be routed through the company experience server. Companion devices could also communicate with the primary device through a data service provided by base station 260. Base station 260 could route communications to the primary device through any communication to Internet 230 that the primary device is using. Base station 260 could also be in direct communication with the primary device, if the primary device is using the same data service.

Context Specific Infer Processor

Turning now to Figure 3, a context-specific interface processor 300 is shown with components suitable for detecting an interaction context and optimizing a user interface, in accordance with one embodiment of the present invention. Processor 300 includes a context determination component

310, a context data store 312, a user data store 314, a person recognition component 316, an object recognition component 3t &amp;, a device recognition component Ú2Ü, and an interface activation component 322. The context-specific interface processor 300 is merely an example of a suitable architecture and is not intended to suggest any limitation as to the scope of use or functionality herein.

<img file="MX340692B_D0020.tif" />

IMPI invention. The context specific interface processor 300 should also not be construed as having any dependency or requirement related to any single component or combination of components illustrated therein. The context specific interface processor 300 can reside on a single computing device or be distributed across multiple devices including multiple devices connected in a home area network, local area network, or wide area network. Parts of individual components may reside in part on a client and in part on a server or otherwise be distributed across multiple devices.

Context determination component 310 determines the present context of user interaction. Context comprises user actions and user interactions with devices, other people, and content that is currently presented on one or more devices. Context determination component 310 can use context templates to identify and label a context. A context template can describe context features and recommend interface features that are optimized for the context. Context characteristics are used to identify the present context. The characteristics of inferred recomerrdatfas are those that are optimized or recommended for the context.

Context determination component 310 may be involved in constructing or optimizing context templates based on observed user actions while using the system.

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

User responses to optimized interfaces can be tracked and used to change context templates. For example, when a user does not interact with features added to an optimized interface, those features could be removed from the template. If users frequently navigate from a first interface feature to a second feature in a second interface, then the first and second interface features can be combined into a new optimized interface.

In one embodiment, data from user interactions with an interface can be communicated to server-based machine analysis and learning that combines information from multiple users to create and modify templates. The templates could then be associated with similarly situated people and used to recognize interactions that indicate a user's intention to interact with an interface in a particular way. For example, context templates could be associated with men 30-35 years of age who use a game pad with their right hand to navigate a media selection interface. The user characteristic could be observed, or collected from account data, a corresponding template retrieved, and the interface optimized according to the template. Context templates can be stored on the server and accessed on demand or periodically communicated to a computing device.

For example, a Mexican Property INSTMrrc gesture input context template

INDUSTRIAL has gesture input as a feature. Gesture input occurs when a user moves with their body, or other body parts, in a gesture that is recognized by an input device, such as a depth camera. For example, a user may perform a thumbs-up gesture that aims to turn up the volume on the television. In either case, the context template can Include context features that include the user who is not holding a control device, or the user who does not appear to interact with a control device being held, in addition to making recognizable gestures. In response, the optimized user interface features may include larger clickable objects. Therefore, the user interface may zoom or the selectable objects may be enlarged. In another embodiment, the selection area around objects increases. Each context has different characteristics and optimized interface aspects. Some contexts may share features with other contexts and optimized interface features.

The input mode is only a context feature. For example, there could be a whole class of contexts based on inaccurate gesture right-handed gestures, left-handed gestures, obstructed gestures, and multi-person gestures. Context determination component 310 can use input mode as a starting point for selecting a context class. As more information is received, the context can be updated and the Mexican msrrrvro DE LA PROTIEDAD INDUSTRIAL interface adjusted accordingly. The interface can be dynamically updated as confidence increases or decreases in a contextual scenario. In general, a lower confidence in a particular context may result in a more generic interface presentation that will not be optimized for a particular context but is adapted for use in different contexts.

The use of a specified device entry is another kind of context. Dedicated devices include remote controls and game controllers and tend to be highly specialized and less flexible. Features used to determine that a dedicated device is being used include the user picking up a dedicated device and using the dedicated device. This is in contrast to the device moving away or just relocating. In addition to using a camera to determine if the user appears to be using the device with a control mechanism, the signals received from the dedicated device can confirm what is observed by the image data. In this case, the interface can be adapted or optimized for use with the selected dedicated device. The context determination component 310 can work with the recognition component of device 320 to identify the particular dedicated device.

Company interactions are another kind of context. Company interaction involves the use of a company device, such as a smartphone or tablet. Company devices can have a touch screen that allows the user

ΓΝίΤΓΠΤΤΟ MEXICAN Μ THE PROPERTY

INDUSTRIAL make fine or granular selections and view additional content. Companion devices can also have a hard or soft keyboard that allows the user to enter text. Companion device interaction features include picking up a companion device. A companion device can be identified by its appearance and by signal data received by the device. For example, data packets communicated from a companion device during the setup of a communication session may include a URL that identifies the device and device characteristics.

In one embodiment, the person recognition component 316 is used to associate individual devices with people. For example, multiple people can be in the room with multiple devices. The 316 person recognition component can help determine who is doing what with your device. For example, a first person may be associated with a device that is placed on the table being ignored. In this case, the user interface would not want to push relevant content to this person companion device or move to an interface context that is optimized for control by a companion device. On the other hand, if a second person is viewing their companion device, content supplemental to the device can be powered to enhance or enhance the second person's viewing experience or ability to control an interface if desired.

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0021.tif" />

The embodiments of the present invention attempt to anticipate that the user will want to use a companion device as part of the interface interaction. For example, a user who is trying to navigate a web browser interface using gestures can pick up a companion device right after selecting a text box. The embodiments of the present invention can automatically open the soft keyboard on the companion device to allow the user to type using the companion device. This optimization could occur without the user doing anything else differently when picking up the companion device or, if they are already holding it, moving it in a way that indicates they are preparing to use it. The context in this case also takes into account the state of the browser application. In this case, the context includes an active text box. If the user is not holding the companion device, the soft keyboard would not open, and a gesture keyboard might open instead in the main presentation.

An audio context is another kind of context where an interface can be optimized. For example, it can be difficult to differentiate between people speaking in the room and a person who wants to start using audro commands to control an interface. In one mode, the audio interface is turned on upon detecting that a user is directly interacting with the interface using voice commands. The user's attention to the interface can be determined by image data confirming that the

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0022.tif" />

User is viewing the interface, in contrast to talking to other people. The presence of other people in the room can also help determine the context. For example, people may be less likely to speak to themselves, and any words spoken could turn on the audio recognition system when there is only one person in the room. In contrast, with multiple people in the room it may be undesirable to turn on the audio recognition system without a more explicit user command.

In addition to optimizing the visual appearance of the interface, you can introduce yourself to a completely new interface when determining a context. For example, if a user is interacting with a web browser using gestures and then picking up a game controller, a contextual determination can be made that the user wants to play a game instead of interacting with the web browser using the game controller. In this case, instead of optimizing the web browser for use with a game controller, a games home page or other game related interface could be presented. In one mode, the user is automatically taken to the last game played.

The context determining component 310 may use one or more machine learning algorithms to recognize contextual situations and associate them with an appropriate context template. In particular, the context determination component 310 can enter various user characteristics stored in

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0023.tif" />

user data storage 314 to help identify the context that is appropriate. In this way, the user's past behavior and preferences can be incorporated into the context determination process.

Context data store 312 stores context characteristics and additional information. Context characteristics can be determined or stored as context templates. In one embodiment, context usage for a user is stored in context data store 312. In other words, the most commonly used contexts can be evaluated for a particular user and used in the future to remove ambiguity from a context determination. Context data storage

312 it can reside on a single computing device or be spread across multiple devices including multiple devices connected on a home area network, local area network, or wide area network. The data associated with the context data store 312 may reside in part on a client and in part on a server or otherwise be distributed across multiple devices.

31S object recognition component uses image data to recognize inanimate objects in the room. This is in contrast to people who are recognized using the person recognition component 316. The object recognition component 318 can recognize game objects or other objects from the Mexican PROPERTY institute.

INWTnUAh particular interest in determining a present context. For example, the object recognition component 318 can recognize a game object such as a tennis racket or wizard hat. By recognizing a tennis racket, a tennis game can be immediately opened in a mode where the user can play using the racket. In the magician hat example, the embodiments of the present invention can select the user's favorite game that includes a magician and select the character based on the hat. In games with different characters, different prominent pieces of clothing or game paraphernalia can be used as a clue to automatically open the game or select a particular character.

In another embodiment, the object recognition component 318 recognizes various objects with which the user is interacting and sets an appropriate context. For example, a user drinking a cola with the user's right hand may wish to gesture with the left hand and not leave the cola behind. In this case, the gesture interface can be optimized for use with the left hand.

A device recognition component 320 recognizes control input devices, including dedicated devices and companion devices. Different devices are recognized using image data, similar to those used by the object recognition component 318, but can also use signal data received from the

<img file="MX340692B_D0024.tif" />

company and dedicated devices. Signal data can be used in combination with camera data to identify a particular device within the environment. For example, several game controllers may look alike. However, using camera data showing user interaction, such as push buttons or moving the joystick, the particular controller will then be identified by correlating the image and control data. In one embodiment, the person recognition component 316 is used to associate a particular person or user with a particular device. In this way, players could be brought to their last progress point, or game states, simply by lifting the controller.

With companion devices, login data and other user characteristics can also be evaluated to associate the device with a particular person or player with a source on a primary device. Additionally, like the game controller, observed interactions with a device can be synchronized with control data received from the device to place the device within the environment and with a particular person.

The interface activation component 322 activates an appropriate interface for a context determined by the context determination component 310. In one embodiment, the interface activation component 322 communicates with the active application, where appropriate, and provides new settings

<img file="MX340692B_D0025.tif" />

INDUSTRIAL interface that must be used in context. The interface configurations can be elaborated from a context template.

Contextual Interfaces

Figures 4 and 5 illustrate an embodiment of the invention that automatically optimizes interface-based detection of a change in the method of user interaction with content. The user interaction method is an example of context. Different interfaces can be optimized depending on the input mechanism used by the user to interact with the interface. For example, an optimized gesture interface, as provided by a depth camera, includes large buttons or other large interface objects. In contrast, a touchscreen, joystick, push buttons, or other controller, which provides more granular control, may be able to display more information and smaller Interactive objects.

Figure 4 illustrates a user 405 who is sitting in his chair interacting with an interface 415 displayed on the display device 400. The input mechanism is a gesture infer with another performance area captured by the camera 410. The user the 405 gestures with his hand, feet, whole body, or a combination to manipulate interface objects, such as boxes E, F, G, and H.

Figure 5 shows a similar scene except that the user

IMPI iwítitvto Mexican OF PROPERTY

INDUSTRIAL

<img file="MX340692B_D0026.tif" />

405 has collected a game controller 510. The embodiments of the present invention are not limited to a game controller. The user could pick up a companion device such as a tablet or smartphone. As you can see, the new interface 515 provides additional options that are compatible with the game controller 510. For example, the user can scroll horizontally by pushing the buttons on controller A or B. Also, the E, F, G, and H boxes are smaller because they can be easier to select with the controller, allowing the user to manipulate a selector or cursor with a joystick or up and down arrows. down.

The modalities of this invention involve two steps. The first step is to detect that the user has changed their interface interaction method. For example, when the user changes gestures to a controller, the active context can be updated. The second step is to update the user interface to incorporate the new input method. Several different detection methods can be used. For example, a camera can be used to actively track a user and see what input method they are currently using. In another example, a depth camera can be used to identify and track a user input or interaction method. A vision based system can be used to detect specific markers on the input device to determine if it is a

MEXICAN INSTITUTE PE THE PROPERTY

INDUSTRIAL tablet, smartphone, television remote control, or game controller.

The system will also determine the type of device the user is holding. An algorithm, for example a machine learning depth shape detector, can be used to determine the type of device supported by the user. Potentially, a series of unique markers or internal sensors are entered, or placed on top of the device in order for the system to better distinguish the device. Other markers could be added to help the system effectively distinguish between two identical devices, such as two game controllers. Different devices may generate different gyroscope or accelerometer output due to the number and location of the sensors within an input device. This output can form a pattern.

A reading from internal sensors, such as a gyroscope, within a device picked up by a user can be corrected for a library of known patterns and used to determine the exact external input device held by the user.

Detection could also be carried out by evaluating device identification information within the radto signals transmitted by the device. Opening a communication session between a controller, or other device, and the computing device that controls the user interface may also include information that identifies the device. For example, as previously described, a companion device can wsmwo MEXICANO DELA PROPIEDAD

INDUSTRIAL connect to a game console. After this connection, the game console and companion device are able to communicate. This connection can be opened even if the user is not actively using the companion device. In this way, the depth camera could detect that the user has traveled far from a gesture interface when the user picks up a companion device.

The camera could be a broad determination that the user has picked up a device that matches one or more companion devices that have open communication sessions with the user. The camera could then correlate inputs received from the device with detected movements. For example, the camera might determine that the user was touching the touchscreen on the companion device at the same time that the touchscreen input was received through the open connection. At this point, a final determination could be made that the user has transited the entry experience to the specific carrier device. After making the determination, the user interface could transition to one optimized for interaction with the companion device.

Regardless of the exact method used to detect the companion device, the next step is to change the user interface. Once the system has determined that the user is participating with a new input device, the Ul is notified and updated accordingly to reflect the new input. By

MEXICAN PROPERTY MWHWTO

INDUSTRIAL example, when a user is watching a movie on a device that supports gesture controls and voice controls, he can use his voice and hand gestures to navigate the user interface. If you get bored watching the movie and decide to pick up the controller to play something instead, the context switch is detected. Once the system detects that they have picked up the controller, a controller-navigable user interface appears, allowing them to start a set of controller-based games.

In another example, a user is playing a game that is primarily controlled by a controller, but has a secondary device, such as a tablet, that presents additional information. While interacting with the title, the user picks up the tablet, and the title automatically changes to a presentation that enhances the tablet experience, such as a map, or an item management screen. This example shows that the user interface can transit in a way that is both optimized in terms of size or available control as well as generating a totally different interface with different content based on a task that is likely to be performed when the user The user collects the evo with tr arad or .Thus, the UI can be optimized based on the input device capacity and tasks probably performed with the device. The user's driver and device usage history can be analyzed to determine the optimal user interface.

IMPI Mexican institute OF INDUSTRIAL PROPERTY

<img file="MX340692B_D0027.tif" />

Figures 6, 7, and 8 illustrate help interfaces that are optimized to interact with different inputs. As mentioned, inputs are characteristics of a context for which interfaces are optimized.

Figure 6 depicts a helper interface that is optimized for a game controller 510, in accordance with an embodiment of the present invention. As can be seen, a user 405 is trying to interact with interface 615 using the game controller 510. The active input mechanism can be evaluated through a combination of image data collected by camera 410 and from data from signal received from game controller 510. As you can see, interface 615 includes a help message 630 that mentions pressing A for more information. A describes a level with a button on the game controller 510. The user can access a different menu or interface by pressing A in the particular context. The general media presentation shown in interface 615 may not change from context to context. For example, if the car shown is a commercial or part of a movie then its appearance may remain the same from context to context. Help overlay and. Other interactive features of the 615 interface may change with context.

Turning now to Figures 7, an optimized help interface for audio control is shown, in accordance with an embodiment of the present invention. The characteristics of the Figure

IMPI

Mexican Institute of Industrial Property

<img file="MX340692B_D0028.tif" />

they are similar to those previously described with reference to Figure 6. However, note that user 405 is no longer holding a controller. Although not shown, the user may be saying commands to indicate that an audible infer should be activated. Item 715 includes a help interface 730 that mentions saying 'more' for more information. In an optimized audio mode, various audio commands can be suggested that are available to the user based on the task that the user appears to be completing.

Turning now to Figure 8, a helper or overlay interface is shown that is optimized for use with a companion device, in accordance with one embodiment of the present invention. Interface 815 is similar to the interface previously described with reference to Figures 6 and 7 but includes a help overlay that mentions, Select 'car' icon on the tablet for more information. This message refers to an icon when it is currently displaying on the user's 810 companion device. This help overlay takes into account not only what type of device the user is trying to use to control the interface, but also the content currently presented on the device. T_a «contextual interfaces can be built by exchanging status information between applications running on the device of company 810 and in the device generating interface 815.

Camera 810 can be used to determine that the user

IMPI ttlSTITUTO MEXICANO DE LA PBOPrCDAI)

INDUSTRIAL

<img file="MX340692B_D0029.tif" />

You are currently holding the 810 companion device and want to use it as a control mechanism before the interface is changed to be optimized for use with the 810 companion device. For example, if the 810 companion device was placed on the table, the context It could be different than when it is held by a user.

Turning now to Figure 9, a contextual interface is shown that is optimized based on a person's location within a gaming environment, in accordance with one embodiment of the present invention. Figure 9 includes players 924 and 926 playing a video game that is displayed on display device 900. Display device 900 may be a television. Each player is associated with their own split screen interface. Player 922 is associated with split screen interface 912 and player 924 is associated with interface 914.

Camera 910 can be a depth camera, or other device capable of generating data that can be used to recognize people and objects. Arrows indicate that the 912 and 914 split screen interfaces can change places if players

922 and 924 change places. This transition can be carried out --- automatically by determining that the context has changed. In this case, the changing context is the changing location of the two players. In this way, players are always exposed to a direct view of the split screen interface with which

IMPI

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX340692B_D0030.tif" />

they are interacting regardless of their movements.

In addition to transiting different interface locations during the game, the initial settings can be optimized before playing by doing person recognition and by understanding that the person is associated with which user file in the game. Although not shown, 922 and 924 players may be using game controllers or companion devices to interact with the video game. In another mode, they are using gestures for audio control to play the video game. However, its relative location can be identified and the interface optimized accordingly.

Turning now to Figure 10 and Figure 11, optimized user interfaces for right or left handed are shown, in accordance with one embodiment of the present invention. Figure 10 shows a person 1040 observing the content displayed on the display device 1000. The content includes a primary display 1020 and a supplemental content window 1030. Supplemental content window 1030 includes clickable object 1032, clickable object 1034, and clickable object 1036. The icons and buttons are examples of clickable objects. The 1040 camera is capable of capturing data that is used to identify people and objects. Additionally, information can be used to determine a person's movements, such as those associated with gestures. In this case, the interface, which has the selectable buttons to the «SITTUTO MEXICANO E LA PKOWEDAD

INDUSTRIAL left, optimized for use with left-handed gestures. As can be seen, player 940 is raising his left hand 1042 to make a gesture.

The user's right-handed or left-handed preference can be evaluated in a number of different ways. In one modality, the user's history is performed to assess their right or left-handed preference. Users who are known to be right or left handed, or who use their right or left hand more frequently when gesturing, will automatically be presented with an interface optimized for their preferred hand.

In addition to previous use, the current user situation can be evaluated to determine which hand the user is likely to use. In one embodiment, a user is considered unlikely to use a hand that is currently positioned to be used differently. For example, if the hand is the right hand holding a glass, a phone, or is in a pocket, then the assumption can be made that the user wants to use the left hand and a left-handed optimized interface is presented. The context template associated with right or left handed users can look at a number of factors, to determine which hand the user is likely to use.

Now switching to Figure 11, the same interface shown in Figure 10 is shown, except that it is now optimized for right-handed use. As can be seen, user 1040 has his right hand 1044 raised and is using it to interact with the

IMPI

MEXICAN INSTITUTE OF THE TEOHEtlAD

INDUSTRIAL

<img file="MX340692B_D0031.tif" />

presentation. In one example, in a right-handed interface or zTTrdá Sé presents an anticipation of user gestures interfering with another person. For example, if it appears from the image data that a user would have to place their left hand in front of another person's load to make a gesture with their right hand, then a left-handed interface may be presented to encourage the user to use her left hand. In another embodiment, if it appears that the user's right hand is blocked from view of the 1010 camera by another person or object standing in front of their right hand, then again a left-handed interface would be presented to encourage the use of the left hand and avoid the occlusion problem created by the interfering object. In this way, contextual determination can try to avoid a problem by creating an interface that is optimized to avoid problems detected in the environmental context.

Turning now to Figure 12, the identification of an object or clothing within the environment is illustrated to generate a corresponding interface, in accordance with an embodiment of the present invention. In Figure 12, user 1220 holds a tennis racket 1222 in front of display device 1200 and camera 1210. Camera 1210 captures image data that is used to recognize tennis racket 1222 and perhaps person 1220. In response, a tennis game is automatically displayed on interface 1212. This prevents the user who needs to navigate through various interfaces from playing a virtual tennis game, which

IMPI

MEXICAN INSTITUTE

DE IA KCT1EDAD INDUSTRIAL

<img file="MX340692B_D0032.tif" />

may be the most likely reason that the user is displayed in front of the interface with the 1122 tennis racket. Alternatively, the tennis racket could be used as a track that the 1220 person wants to watch tennis. User observation history and preferences can be used to remove ambiguity from the context.

In another embodiment, articles of clothing are recognized and used to establish context and present an appropriate interface. In one embodiment, a team shirt is recognized and used to automatically select a shirt of a similar color or even exactly the same team within a video game. For example, you can start a video game that has NFL teams with the user playing with their team indicated on their shirt. In a multiplayer environment, you could assign multiple people to teams corresponding to their shirts. When an exact match is not possible, similar colors can be automatically assigned to the computer you are controlling.

Depending on other contextual factors, such as an ongoing sporting event involving the team identified from clothing, the presentation of that sporting event may

<img file="MX340692B_D0033.tif" />

automatically in the user interface. For example, if a soccer game involving the team indicated on the person's shirt is currently available, then it can be displayed. If the game is in progress, but is not currently available due to the user's cable package, for example,

IMPI

MEXICAN INSTITUTE OE THE PROPERTY

INDUSTRIAL

<img file="MX340692B_D0034.tif" />

A mechanism for viewing the game may be presented, such as subscribing to a particular cable package to the user and automatically configured. Alternatively, a web page displaying a game update can be automatically presented.

The removal of ambiguity from playing the video game against viewing game contexts may depend on whether there is a sporting event available involving the relevant team. In this way, contextual factors may include more than just what is happening in the present environment, but may include evaluation of various contents that may be of interest to the user available to the user at a particular time. When multiple contexts are available, a previous user observation history can be considered to help remove ambiguity from the selection. Alternative interfaces can also be presented to the user. For example, the user might be asked if they want to see their team playing the current competition or playing a video game.

In another embodiment, clothing associated with various video games can be used to automatically choose characters or various available games. For example, a person wearing a wizard hat can automatically be assigned to the wizard character when they start a game involving a wizard. In a slight variation, in response to using the magician's hat, a game involving a magician can be opened without selecting a particular character. When multiple games are available involving

IMPI

MEXICAN LA PROREDAD WITTVTO

INDUSTRIAL

<img file="MX340692B_D0035.tif" />

A magician can be selected and presented in a very recently played game involving a magician.

Methods for Generating Contextual Interfaces

Turning now to Figure 13, a method of presenting an interface that is optimized for the present context is shown, in accordance with an embodiment of the present invention. The method 1300 interface may be a computer interface displayed on a television, monitor, or other display device. The interface can be generated by a game console, media player, personal computer, or other similar device.

In step 1310, a present context for a user interface is determined by analyzing image data illustrating an environment near the user interface. In one embodiment, the environment near the user interface is the volume of space illustrated in the image data. In another embodiment, the environment extends behind the area captured by the image data and is demarcated by physical boundaries such as those in a room or residence such as a house or apartment. Although these other areas may not be in view of a camera that captures image data, they may receive signals from d + s in the environment ----— as well as audible commands from users who are not in view of a camera. camera. In one embodiment, the image data is three-dimensional image data, such as that captured by a depth camera. Image data can be

IMPI • MEXICAN AGE fSTTTVTO

INDUSTRIAL

<img file="MX340692B_D0036.tif" />

described as a cloud of depth. Three-dimensional data allows the size, shape, and movement of objects within the data to be detected. The depth cloud can be used to recognize gestures made by a user. A gesture can be a hand movement, body movement, facial expression, or other action recognized by a gesture interpretation component to control an interface.

At step 1320, a user interface is generated that is optimized through a present context. The user interface is sent for presentation to the user. Various ways to optimize a user interface have been previously described. For example, the user interface can be optimized for use with a particular control mode. Within the various control modes, the task that the user is seeking to complete through the interface can be anticipated and the user interface updated accordingly. The desired task is part of the context derived from the evaluation of the environment. User interactions with objects in the environment can also be used to determine context.

Now changing Figure 14, a method 1400 is shown for presenting a .inte.rphase „ctue.es optimized to the present context, according to an embodiment of the present invention. In step 1410, a present context for a user interface is determined by analyzing actions of a person who is close to the user interface. Actions comprise use of

IMPI

WJTTTUTO MEXICAN PROPERTY

INDUSTRIAL

<img file="MX340692B_D0037.tif" />

control with the interface. Illustrative modes of control include using gestures, using a companion device, using a dedicated device, and audible control. The present context may be more granular than a particular mode of control. For example, within a gesture control mode, the context may determine that right-handed or left-handed gestures are preferable.

At step 1420, a user interface is generated that is optimized to the present context. The user interface is sent for presentation to the user. Methods for optimizing the user interface have been previously described in view of the present context.

Turning now to Figure 15, a method 1500 is shown for displaying an interface that is optimized to the present context, in accordance with an embodiment of the present invention. In step 1510, a present context for a user interface is determined by analyzing image data illustrating an environment near the user interface. Present contexts and the use of image data to determine the present context have been previously described with reference to Figure 13 and elsewhere.

At step 1520, a present user interface is generated that is optimized to the present context. The user interface

<img file="MX340692B_D0038.tif" />

i user.

At step 1530, a new context for the user interface is determined by analyzing image data illustrating the environment near the user interface. The image data illustrates changes to the environment that result in the new context. In

ΤΟυΤΟ MEXICANO ”OR £ INDUSTRIAL PROPERTY In step 1540, the user interface is automatically optimized for the new context.

Embodiments of the invention have been described to be illustrative rather than restrictive. It will now be understood that certain features and their combinations are useful and can be used without reference to other features and sub-combinations. This is contemplated by and is within the scope of the claims.

IMPI msTrruTu mexican INDUSTRIAL PROPERTY

<img file="MX340692B_D0039.tif" />