Systems and methods for providing audio to a user based on gaze input

ES 2 633 016 T3

DESCRIPTION

Systems and methods for providing audio to a user based on gaze input.

Background of the invention

The present invention relates, in general, to systems and methods for human-computer interaction using gaze direction information and, in particular, to systems and methods for use in connection with graphical user interfaces, games or other intensive applications of human-computer interaction.

The interactions between users and computers take many forms and have evolved greatly over the years. Typical interactions are based on mechanical type inputs by the user, to the computer. For example, a user typically operates a computer through the use of a keyboard and mouse. The keyboard and mouse depend heavily on a user's fine motor reflexes and capabilities. Take a modern computer game as an example, success may depend on the ability to move, quickly and precisely, a mouse in response to actions displayed on a screen. The user's ability in the game is therefore based on their level of coordination.

Recently, new entry modalities have gained popularity. For example, input devices based on touch and gestures are in some way in popular use. A touch-based input device relies on a user physically contacting a surface, typically a display, to interact with a computer. A gesture-based input device relies on a user performing a gesture with a body part, for example waving or waving a hand in the air. An optical system that somehow comprises a camera and a light source captures this gesture.

All traditional input modes share a common problem, they depend on the ability of a user to precisely direct a part of the body to a particular point of interest on a computer display. This presents the possibility of error between a user's intended POI on the display and the actual POI provided to the computer through the traditional input mode.

A true representation of a point of interest of a user can be obtained by following the direction of the user's gaze. Eye or gaze tracking technology is not new, many examples are provided on the market, such as the technology sold by Tobii Technology (www.tobii.com) and described in US Pat. US number 7,572,008.

The use of gaze direction information for simple interactions with a computer has also been previously described, an example is US patent. No. 6,204,828 which describes a system for moving a cursor on a display in response to a user's gaze direction.

Previous entry systems using gaze directions have focused on basic entry techniques, where a manual entry method is replaced by a gaze component. There is a problem with how to fully utilize the benefits of gaze direction information in human-computer interaction, such that gaze direction information not only replaces a portion of a mechanical input method but complements and enhances the ability of a user to interact with a computer.

Documents EP 1 854 516 A1 and US 6,675,075 B1 respectively describe a method according to the preamble of claim 1, a system according to the preamble of claim 14 and a non-transient machine-readable medium according to the preamble of claim 16.

Brief description of the invention

In one embodiment, there is provided a method of offering audio to a user according to claim 1. The method may include determining, with an eye tracking device, a point of gaze of a user on a display. The method may also include making, with a computer system, an audio device produce audio for the user, where the content of the audio may be based at least in part on the user's point of view on the display.

In another embodiment, a system is provided for delivering audio to a user according to claim 14. The system may include an eye tracking device for at least determining a user's gaze point on a display. The system may also include a computer system for at least causing an audio device to produce audio for the user, where the content of the audio may be based at least in part on the user's point of view on the display.

In another embodiment, a non-transient machine-readable medium is provided with instructions stored there to deliver audio to a user according to claim 16. The instructions can be executed by at least one processor to at least determine a point of gaze of a user in a visualization. Instruction can

ES 2 633 016 T3 also be executable to make an audio device produce audio for the user, where the content of the audio can be based at least in part on the user's point of view on the display.

Brief description of the drawings

The present invention is described in conjunction with the attached figures:

Figure 1 is a block diagram of a method of the invention for issuing commands at a gaze location;

Figure 2 is a block diagram of a method of the invention for providing audio to a user based on gaze input;

Figure 3 is a block diagram of a method of the invention for displaying a map that is interactive according to a location of a user's gaze;

Figure 4 is an exemplary command and menu system of the invention using gaze input to make selections;

Figures 5A & 5B are schematic diagrams of two methods for providing depth of field visual effects to an observing user;

Figures 6A & 6B show an exemplary result of an embodiment of the invention that modifies a display according to the position of the user's head;

Figure 7 is a block diagram of a method of the invention for changing the behavior of program elements based on gaze input; and Figure 8 is a block diagram of an exemplary computer system that may be used in at least some portion of the apparatus or systems of the present invention, or that may implement at least some portion of the methods herein. invention.

In the accompanying figures, similar components and / or features may have the same reference numeral label. Furthermore, several components of the same type can be distinguished by having the reference label followed by a letter that distinguishes between similar components and / or characteristics. If only the first reference label is used in the specification, the description is applicable to any of the components and / or similar features that have the same first reference label regardless of the letter suffix.

Detailed description of the invention

The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the description. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. One skilled in the art will understand that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope of the invention as set forth in the appended claims. For example, not every detail of each embodiment described herein will be present in all versions of that embodiment, or details of one embodiment described herein may be present in any version of other embodiments described herein.

Specific details are provided in the following description to provide a thorough understanding of the embodiments. However, one of ordinary skill in the art will understand that the embodiments can be practiced without such specific details. For example, circuits, systems, networks, processes, and other elements of the invention may be shown as components in block diagram form so as not to obscure embodiments in unnecessary detail. In other instances, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

Also, it is noted that individual embodiments can be described as a process that is illustrated as a flow chart, a data flow chart, a structure chart, or a block diagram. Although a flowchart can describe functions as a sequential process, many of the functions can be performed in parallel or at the same time. Also, the order of functions can be arranged differently. A process can end when its functions are complete, but it could have additional stages not described or included in a figure. Also, not all functions in a particularly described process may occur in all embodiments, or there may be additional functions not described in specific relation to the particular process. A process can correspond to a method, function, procedure, subroutine, subprogram,

ES 2 633 016 T3 etc. When a process corresponds to a function, its completion corresponds to a return from the function to the calling function or main function.

Furthermore, embodiments of the invention can be implemented, at least in part, either manually or automatically. Manual or automated deployments can be executed, or at least assisted, through the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination of these. When implemented in software, signature, middleware, or microcode, the program code or code segments to carry out the necessary tasks can be stored on a machine-readable medium. One or more processors can carry out the necessary tasks.

Command issue at gaze location

According to one embodiment of the present invention, a method for human-computer interaction is provided whereby a computer displays items on a display. The user can issue commands to the computer that modify the items on the display. In order to determine which items will be modified, a gaze detection unit detects the user's point of interest on the display. The user then issues commands to the computer so that items located at or near the point of interest are modified. Both the visual and underlying aspects of the articles can be modified. For example, the appearance of an article can be modified, or the characteristics of the article, and how it interacts with the user and / or carries out other tasks in the context of the particular program can be modified (e.g., the speed at which moves, the actions it can take, etc.).

The user can issue commands to the computer using any form of input device, including traditional keyboard and mouse, voice commands, and / or gaze detection unit. In some embodiments, combinations of said inputs can be combined and interpreted to mean different commands than those that occur when an individual input is received by a single device.

To demonstrate such an embodiment, the example of a computer game is provided. A computer game displays many items on one display, for example a first person shooter (FPS) style computer game will often realistically show a war and war conditions as eg terrain, moving vehicles, soldiers, and the like. The user interacts with the game by giving the computer commands such as where to move a character, where to shoot, when to jump, when to run, when to crawl, etc.

A gaze detection unit determines the user's point of interest on a display and, according to an embodiment of the present invention, the user enters a command to control a character or other component in a game. The command modifies items on the display at the location of the user's POI as determined by the gaze detection unit. For example, in the case of an FPS-style computer game, a user can press a mouse button or game controller button to fire a weapon. By registering the user's POI, when the user provides the command to fire the weapon, the command output takes place at the POI location. In this way, the user can look at the intended target of the simulated gun shot and maintain a higher level of control over game accuracy, when compared to traditional computer game controls.

As another example, the user's POI can be used to indicate which object in a visualization to perform an action on. Take, for example, a computer game in which a character lies on the ground and takes aim through a sniper rifle. Traditionally, if an enemy character approaches, the user would fire the sniper rifle after issuing a fire command to the computer program. By using the gaze information, however, the user can look to the left or right of the screen, which may mean that the user wants to use a different weapon such as a handgun or the like. In this way, the user can control the character to leave the sniper rifle on the ground, engage the enemy with an alternate weapon, and then return to the sniper rifle to continue advancing through the game.

Accordingly, referring to Figure 1, in a method 100 provided by an embodiment of the invention, the point of gaze of a user in a display can be determined at block 110. In some embodiments, an additional input such as, For example, a keystroke, controller input, or voice command, may also be received at block 120. At block 130, it can be determined on which displayed object or other program element is intended to act in correlation with the gaze point and / or additional input. At block 140, it can be determined which specific action correlates to the additional gaze point and / or input. At block 150, the action can be taken on the displayed object or other program item.

Directional audio

ES 2 633 016 T3

In another embodiment of the present invention, a method is provided for influencing the audio that an operator of a computing device hears.

According to the present embodiment, a computing device, a gaze detection device and a computer program executed by the computing device are provided. As with any embodiment described herein, the gaze detection device may be integrated with the computing device or it may be separate but connected to the computing device through wired or wireless means. Furthermore, the gaze detection device can be worn by the user or observe the user from another separate position.

The gaze detection device determines the location on a screen at which the user is looking, namely the point of gaze. The computer program uses the point of gaze to carry out an action that emits (or, in other words, reproduces and / or produces) sound from the computing device or manipulates the sound already emitted from the computer.

For example, in the case of a computer game that has multiple characters (avatars, eg, representations of people or the like) on a screen with which each character is simulated speaking, the gaze direction can be used to determine which voice of the characters will be emitted by the computing device.

The voice can be emitted by more than one character at a time and only the volume of the voice altered according to the location of the gaze, or only the voice of one character can be emitted according to the location of the gaze.

For example, in the case of a computer game where a user's visual field is simulated on a visualization, detecting a gaze location on a character on the left side of the visual field can result in a character's uttered voice. or characters on the right side of the visual field that is reduced in volume, while the voice of the character on the left side of the visual field is kept at a normal or increased volume.

Such alteration of the volume can occur almost instantaneously, so that while the user contemplates different locations in a visualization, the volume of the voice of different characters increases and decreases according to the proximity of the user's gaze to the character. Alternatively, altering the volume can occur non-instantaneously to provide smooth audio output to a user and accommodate rapid gaze movement. For example, the audio can only change based on the user's gaze direction if the user fixes their attention on a location for a predetermined time threshold.

Of course, the voice is simply an example of audio that can be emitted by a computing device. Music, sound effects and the like can also be broadcast. In fact, any sound that a computing device can produce can be used successfully with the present invention.

The present invention can also use audio channels, for example left and right channels can be used in a stereo audio system to output different volumes and / or different sounds depending on the location of the gaze of the user. For example, if the gaze direction is determined to be in the center of a visualization, all sounds intended to be transmitted to the left of the center of the visualization can be output through the left audio channel of the computing device and t All sounds that are intended to be transmitted to the right of the display can be output through the right audio channel of the computing device. If the gaze direction is determined to be to the left or right of the center of the display, the left or right audio channel can be used, as appropriate, in isolation. Sounds transmitted by ambient sound systems (eg, 5.1 or 6.1 channel audio systems) can be manipulated in similar ways depending on the direction of the gaze detected. Also, different audio wavelengths can be separated and output from the same or different speakers depending on the position of the gaze direction.

Any form of sound emitting device is suitable for use with the present embodiment of the present invention, for example, headphones, speakers, or the like. Audio devices can be mono, stereo, or multichannel devices (eg, ambient sound). In some embodiments, the audio device, while stereophonic only, can produce simulated multichannel sound).

Any form of sound or audio is suitable for use with the present embodiment of the present invention, eg, voice, sound effects, music, etc. Sound can be provided by a computer program directly or by a third party such as, for example, a human user running a version of it or a similar computer program in another location and sounding, such as voice of the user transmitted through the internet or another network or communication protocol.

Accordingly, referring to Figure 2, a method 200 for providing audio to a user is shown. At block 210, the gaze point of a user on a display can be determined. In block 220, a

ES 2 633 016 T3 audio device can produce audio according to at least in part the user's gaze point on the display, wherein the content of the audio depends on the user's gaze point.

The manner in which the content is determined may vary according to the embodiment of the invention. At block 221, the delivered audio content may be associated with a particular area of ​​the display in which the user's point of view is located, where the particular area is a sub-portion of a total area of ​​the display. In some embodiments, this may mean that only the audio associated with the particular area is produced. In other embodiments, the audio associated with the particular volume may be presented at a volume and other audio associated with other portions of the display may be produced at a lower volume.

In an alternative embodiment, at block 222, it can be determined how many channels of sound, real or simulated, the audio device produces. Such information can be used to deliver audio through the multichannel audio device in a number of ways.

At block 223, one or more sound channels associated with a particular area of ​​the display on which the gaze rests can be determined and produced. In some embodiments, some channels may not have any audio content produced when the point of gaze is associated with sub-portions of the display corresponding to another channel. In other embodiments, different channels can be produced at different volumes depending on where the point of view rests.

At block 224, the content delivered by the audio device may vary in frequency depending on the location of the gaze point. Which channels produce which frequencies can also vary depending on your point of view.

At block 225, a virtual distance and direction can be determined from each of multiple virtual sound sources to the user's gaze point on the display. Virtual distance and direction can be measured along the X, Y and Z axes of the visualization and / or virtual world. The audio content may then include each virtual sound source that is produced in a manner based at least in part on the virtual distance and direction from the virtual sound source to the point of view of the user.

As an example, let's consider an interactive game where the user is interacting within a simulated war environment. Many audio-producing items can be displayed on the screen, such as characters firing weapons, tanks firing weapons, explosions, characters speaking or yelling, and airplanes flying. According to embodiments of the present invention, when a user places his gaze on a point on the display, a virtual distance is determined between that point and other audio-producing items on the display. According to this virtual distance, the audio levels are provided to the user, for example an explosion next to the point in the display can be provided at a relatively high volume, while an airplane that is flying at a certain distance from the point in the display can supply at a relatively low volume. According to the present invention it is possible to emit multiple of said audio sources at different dynamic volume levels according to their virtual distance from a point on the display (eg, the point of gaze).

Map control

In another embodiment of the present invention, a method is provided for manipulating an object on a display representing a topographic view of a large area or the like. In other words, a map displayed on a visualization can be manipulated based on gaze direction information.

A computer program operates on (ie, is executed by) a computing device, the computer program displays on a display device connected to the computing device a map representing an area. Portions of the map can be enlarged or minimized in order to see the portions in greater or less detail, this is commonly known as zooming in or out of the map.

The location of the portion of the map to zoom in or out is determined by a user. Typically, such determination is made by positioning a focusing device such as a mouse on a portion of the map and then offering a command to zoom in or out on the map. The command can be provided by moving a scroll wheel on a mouse, pressing a key on a keyboard, or performing a gesture on a touch screen. In some instances, the step of positioning a focusing device is not required and just one command is sufficient to zoom in or out on the map.

According to an embodiment of the present invention, the step of positioning a focusing device can be provided by gaze information from a gaze detection device. By determining the location on a visualization that the user is looking at (gaze direction), the map can zoom in or out on that location.

ES 2 633 016 T3

Referring to FIG. 3, an exemplary method 300 of the present embodiment of the present invention is shown. At block 310, a map is displayed on a display device. At block 320, the location of a user's gaze on the map and / or display device is determined.

In block 330, a command is received from the user. The command can be provided by way of known input means including, but not limited to, a keyboard, mouse, scroll wheel, gestures on a touch screen, gestures in 3d space, virtual reality headset, voice, or the like.

At block 340, the action to be taken can be determined, based on at least one of the gaze location and / or command. For example, if the gaze location is anywhere but an edge region of the map, then the map can be zoomed in or out from the gaze location, possibly depending on the command received (e.g., by reducing the zoom when a voice command is received to that effect or when a keystroke of a subtraction sign is received via a keyboard; and zoom in when a voice command is received to that effect or when a keystroke of a plus sign is received via a keyboard).

If the gaze location is in an edge region of the map, or outside the map edge, then the map can be planned in the direction of the gaze location relative to the center of the map. For example, if your gaze location is to the left of the map and you receive a command to pan (for example, by a voice command to that effect or a keystroke such as, for example, p) , then a pan left action should be interpreted. Regardless of whether the map will be zoomed in or panned, at block 350 the map is re-visualized in the new configuration.

Furthermore, when a map according to the present invention is enlarged, it is advantageous to automatically pan the map by a small amount to reorient the enlarged view of the map in order to avoid distortion of the intended zoom area. In other words, if a user intends to zoom in on a point three-quarters of the way along and three-quarters of the way up a map, zooming in slightly will no longer be three-quarters of the way up and three-quarters of the way across the map. Therefore, as the map zooms in, you need to pan slightly to reorient the point so that it remains substantially in its original position relative to the rest of the map.

Look Click / Pink Command

In another embodiment of the present invention, a method is provided for manipulating navigation between selection choices on an on-screen display.

According to the present embodiment, a menu comprising multiple choices is displayed on a display by a computer program. A user can navigate between articles using their gaze direction, optionally in combination with another input device.

The menu can be arranged in any format, such as a traditional multi-choice item menu, all arranged at the same time on the screen. Or, in addition, a smaller number of items can be displayed, and by at least partially activating one item, more menu items can be displayed.

In an example of the present invention, a computer program displays an icon on the display. By determining that a gaze direction is on or in close proximity to the icon, additional icons are displayed near the initial icon and the gaze direction can be moved to these new icons in order to select them for activation.

In a further enhancement, an input device can be used to aid in the selection of items on the display. For example, a key on a keyboard can be pressed and held to allow selection of items within the menu. While the key is pressed, the items in the menu change appearance when the gaze direction is on the item, helping the user to know when an item is being selected. After releasing the pressed key, the item that currently has the gaze direction on it will be activated and a computer program will carry out the action associated with said icon.

Consider, by way of example, an on-screen menu in a computer game or the like. Computer games typically require quick reflexes and responses, therefore it is advantageous to provide a method for quickly selecting an item from multiple options. Such a method will display an initial icon in the center of the display, by activating an input device such as a keyboard or mouse or game controller, the computer program is notified that the menu will be accessed by the information from the gaze. After placing the gaze direction on or in close proximity to an icon, more icons appear next to the initial icon. These new icons can then be accessed by looking at the icons. After activating an input device, the selected icon in response to a detected gaze direction becomes the activated icon and the computer program performs a predetermined function. In this way, a representation can be accessed

ES 2 633 016 T3 pink type of an on-screen menu by looking at information, a pink type representation is a representation where the icons expand outward from a small initial subset of icons.

Figure 4 demonstrates a display of such embodiment 400. Item 410 may initially be the only item present in a display and represent the desire to initiate a menu command. When a wearer's point of view rests on element 410, it may be rendered shaded 410a or otherwise distinguished. Upon receiving an additional command from an input device, element 410 can be expanded to have additional sub-elements 420. Sub-items 420 may represent sub-menus of possible commands.

When the wearer's point of gaze rests on a specific sub-element 420, it may be rendered shaded 420a or otherwise distinguished. Upon receiving an additional command from an input device, element 420 can be expanded to have additional sub-sub-elements 430. Sub-sub-elements 430 can represent possible commands.

When the user's gaze rests on a specific sub-sub-element 430, it can be rendered shaded 430a or distinguished in another way. Upon receiving an additional command from an input device, the gaze on sub-sub-element 430 can finally be selected for execution.

After execution, the sub-sub-item can be displayed in isolation only and / or additional graphical indicators such as a 440 check mark can be displayed. Any number of levels and sub-levels of menus and selections is possible and Figure 4 it is for example only. In addition, menu level options may disappear from the display once they are selected and acted upon, depending on the embodiment.

An input device can be any known form of input device such as, for example, a keyboard, mouse, voice input, gestures, either contact or non-contact, virtual reality headset, and the like. Depth of field graphical implementation

In another embodiment of the present invention, gaze information is used to dynamically tailor a depth of field representation in an interactive entertainment program.

Gaze information provided by a gaze detection device or the like is used by a computer program to dynamically alter displayed objects on a display. In the case of the computer program that is an interactive entertainment program such as a computer game or the like, the objects displayed on the display may be arranged to represent a view of a real life scenario such as , a street scene, landscape, interior of a building or the like. Often in such a computer game, a user controls a character and navigates the character through a simulated world, with the view displayed on the display representing a view of the world seen through the eyes of the character.

Since a visualization is a two-dimensional plane, graphical techniques are used to provide a simulation of a three-dimensional space. One such graphic technique is to vary the depth of field of various objects on the display. For example, where the game wants to indicate to the user that it is centered on an object close to the character, objects that are intended to be away from the character may be blurred and / or their colors, sizes, or other rendering characteristics may be altered, giving the user the impression that there is depth to the scene displayed on the visualization.

According to the present invention, the rendering of objects on the screen is altered according to the presence or absence of the direction of the gaze on said objects. By way of example, the invention works as follows:

• items displayed on a display by a computer program;

• gaze direction information received by the computer program; and • the rendering of an article, or articles, on the display is altered when gaze direction information indicates that a user is looking at said article or articles.

Rendering is intended to include any form of display alteration of a graphic object in a display, including alteration of quality of texture, size, color or shape, anti-lapping, and the like.

The present invention is best suited when used in combination with lazy rendering, whereby sections of a displayed screen are rendered together, as opposed to a system whereby the entire scene is rendered together.

ES 2 633 016 T3

In use, the present embodiment of the present invention allows a computer game to dynamically alter the depth of field that is shown to a user in a display, such that the user focuses their gaze on an area in which the visualization, the displayed scene is altered to accurately represent the scene as if the user were viewing the scene in real-life three-dimensional space.

The altered item on the display can be an individual object such as a building, tree, car, etc. or it can be a portion of the display such as a quarter, eighth, and so on. of the displayed scene. It is totally dependent on the rendering system of the computer program and the scene being viewed, to determine the best way to alter the depth of field to better represent the scene in order to accurately reflect real life.

Examples of possible implementations are shown in Figures 5A & 5B. In Figure 5A, the point of view

510a is located in the upper right corner of display 520a. In some embodiments, a portion 530a of display 520a may be graphically displayed differently than the rest of display 520a. The size and shape of the portion 530a may vary depending on the embodiment. Furthermore, different areas of the display 520a in addition to the two portions shown in Figure 5A can be displayed in different ways. For example, the methods used to display portions of the display 520a may vary as the distance from the gaze point 510a increases. Additional methods of viewing different regions of the 520a display according to gaze input are described in US Patent Application No. 14 / 197,171, filed on March 14, 2014, and entitled GAZE AND SACCADE BASED GRAPHICAL MANIPULATION ', the full description of which is incorporated herein by reference, for all purposes, as if fully established herein.

In Figure 5B, gaze point 510b is located in the lower left corner of display 520b. In some embodiments, the different horizontal layers 530, 540 of display 520b can be displayed in different ways in order to simulate depth of field. While layer 530b can be reproduced in one way, the reproduction of layers 540b can change, potentially to a greater degree, the further away they are from layer 530b.

One method of modifying the depth of field of a displayed object on a display, as will be understood by one of ordinary skill in the art, is as follows.

• Simulate a beam of light that is projected from a character;

• determine which objects the beam will connect to in the visualization;

· Depending on whether the objects are opaque or transparent, send more than one ray;

• measure the distance between the character and a connected object;

• the connected object becomes the center of the user in order to render the object, or area of ​​the screen containing the object, differently from other objects.

As a further improvement of the present invention, the scene can be altered so that an area of ​​the screen at which the user is looking is enlarged or enlarged. In this way, a user can enlarge a particular portion of a scene on a display, simply by looking at that portion.

Parallax

In a further embodiment of the present invention, the eye tracking information is used to determine the position, angle and / or orientation of a user's head.

According to the present embodiment, a computer program uses the information derived from a gaze detection device to determine the position, angle, orientation and / or size of a user's head, hereinafter called head information. Such information can then be used to alter a graphical display as shown in the example of Figures 6A & 6B. In Figures 6A & 6B, while the user 610 tilts his head to the left, the display 620 can be modified to reflect such head movement. More complicated implementations are also possible as described below.

Information about the head can be used by a computer program to alter the display and behavior of objects in a display. By way of example, when a computer program is in the form of an interactive entertainment program such as a computer game, the head information can be used to control a displayed character on the display. For example, in a form of computer game where a character is positioned behind a wall, the head of the character on display can be made to mimic the movement of the head of a user in real life. This

ES 2 633 016 T3 way, for example, a user can tilt their head slightly as if looking around a corner of the wall and the game can accept the information about the detected head and react to it by making the character in the game look around the corner identically.

In another example of the present embodiment of the present invention, in the case where the computer program is a fight simulation computer game where the control of an airplane, spaceship or the like is simulated, the information on the head is You can use it to realistically portray on display a character turning his head to change his field of view to encompass different portions of a cockpit or the like.

In another example of the present embodiment of the present invention, in the case where the computer program is a role-playing computer game or a first-person shooter type computer game where a user controls a character so that the visual field of the characters is displayed on the screen to present the user with a view as if the user were present in the game world in a first-person setting, the information on the head can be used to provide commands to areas of the game world or scene silently. In other words, consider a first person style computer game that represents a war situation based on a brigade, in this situation the enemy characters can be altered by the noise made by a user's character. By configuring the game to accept input commands based on head information, the user's character can be made to silently command and / or respond to other characters in the game, whether controlled by humans or by the computer, simply by moving his head. For example, a head tilt can be used to confirm an order.

In some embodiments, information about the head can be determined from or based on gaze information provided by a gaze detection device. Gaze information may include location, size and orientation of the eyes, size, location and orientation of the pupil, size, location and orientation of the cornea, and the like. The gaze information is then examined and a calculation of the position, orientation, angle and size of a user's head can be carried out based on the information corresponding to their eyes. For example, if a user's head is tilted, then the user's eyes will be tilted at a similar angle.

In addition, the gaze detection device can be adapted to provide information regarding other features on a user's face to provide head tracking capability. Such information can include the location and orientation of key features such as nose, mouth, ears, hair, and the like. The gaze detection device can capture images of the user's head using an image sensor located in the gaze detection device.

It is preferable that a computer program is specifically notified by an input device that the information will be provided on the head, so that the computer program can use the information.

In some examples, additional head positioning aid can be provided, such as by infrared light reflected off the head of a user and detected by an image sensor. The image sensor can be the same sensor used for gaze detection or it can be an additional sensor. Infrared light can be reflected off custom material or patterns placed on or near a user's head in order to track the position and orientation of the user's head.

Alternatively, a separate head tracking device can be used to determine head information and such head information can be used by a computer program alone or in combination with gaze information from a gaze detection device. . Appropriate head tracking devices may rely solely on images captured by a camera such as a dedicated webcam or camera, and / or may also require additional hardware such as a head mountable item.

Gaze information as input to interactive episodes (eg, artificially intelligent characters)

In another embodiment of the present invention, gaze information can be used by a computer program to dynamically adjust the behavior of computer program elements, eg interactive episodes such as artificially intelligent characters. (AI).

In the case of a computer game that has interactive computer-controlled episodes and / or artificially intelligent characters that engage with the user-controlled character, the gaze information can be used to carry out the behavior of those elements with the in order to make its behavior appear more realistic to the user.

For the purposes of the present description, Artificial Intelligence (AI) and the like are intended to represent characters, episodes, or other items / objects within an interactive game or video game. Said articles respond to user actions or other elements within the game or program to provide a

ES 2 633 016 T3 representation or impression that they intelligently determine said response. In other words, characters, episodes, or articles / objects may or may not comprise dedicated artificial intelligence, but they may also function in an essentially binary fashion - altering their behavior between two (or more) states depending on the presence or absence of gaze information. .

For example, consider a computer game whereby a user is pointing a gun at a building that comprises multiple windows. AI characters can be made to appear in the window, eg, for the purpose of shooting the user's character, in response to detecting that the user's gaze is in or near the window. Traditionally, these characters would appear randomly or due to a predefined pattern in the programming of the computer program. However, by using the gaze information, it is possible for the computer program to determine the frequency with which a user observes different windows in the building and to ensure that AI characters appear or do not appear in those windows. This will increase or decrease the difficulty of the computer game depending on the preferences set for the computer game, eg, by the developer and / or user.

As a further example, consider a computer game in which a user interacts with characters who intend to act as humanly as possible. An example of a game genre in which this occurs are the games known as Role Games. In the present example, the gaze information can be used by the computer program to determine exactly where a user is looking and allow the AI ​​characters to interact with the user in a realistic manner. For example, if the gaze information indicates that a user is looking at a particular area on the screen, an AI character can make an observation about it or interact with that area.

As a further example, gaze information may indicate that a user's attention is or is not on the game. The game can then alter its behavior, or the behavior of game elements, depending on the user's attention. For example, if it is determined by gaze information that a user does not look at a visualization for five minutes, the game may be stopped at least temporarily and / or the characters in the visualization may alter their behavior or carry out a command or specific functions, such as speaking.

Accordingly, in one embodiment, a method for changing the behavior of the elements of the computer program can be provided, which basically consists of two stages. First, by using an eye tracking device, a gaze point of a user can be determined. A computer system can then cause an interactive episode controlled by the computer system to alter its behavior based at least in part on the user's gaze.

In some embodiments, causing the interactive episode and / or AI character to alter its behavior may include causing the episode or AI character to begin or continue interaction with the user based at least in part on the user's point of view that is located in a particular graphical representation, perhaps of the AI ​​character, in a visualization. In some embodiments, the interaction may be altered according to at least in part the point of view of the user who is located at a particular portion of said graphical representation. For example, a user who is looking at a weapon held by an AI character can have the AI ​​character point or use the weapon in some way.

In some embodiments, the behavior of the interactive episode and / or character AI may be altered according to at least in part the point of view of the user who is located in a portion of a display in addition to the aforementioned graphical representation. For example, the user looking at a thing, creature, or other character on the display can have an AI character or other program item interact with that thing, creature, or other character, as well as with the user.

In some embodiments, altering the behavior of an interactive episode and / or AI character may include causing a program element and / or the AI ​​character to move to or from a portion of a display close to the user's gaze point. In still other embodiments, an interactive episode and / or the behavior of an AI character can be altered at least in part when the user's gaze is not located on a display, which may or may not currently have a graphical representation. of a particular program item and / or AI character.

Accordingly, in one embodiment of the invention, as shown in Figure 7, a method 700 that determines a user's gaze point may be provided at block 710. At block 720, additional inputs may be received from others. input devices in addition to an eye tracking device.

At block 730, it can be determined whether the user's gaze is located on a certain AI character, object, or other area that correlates to a certain predetermined action that an AI character or other program item will take. If a correlation exists, at block 740, the appropriate predetermined action is taken. How I know

ES 2 633 016 T3 described above, other factors such as, for example, secondary inputs and / or other settings, may also inform what predetermined action will be taken.

Gaze triggers and the like

In a further embodiment, triggers for interaction can be incorporated into a scene displayed on a display. Such triggers can only be activated by the gaze of a user located on the triggers. Triggers can be visible or invisible to the user. When a user looks at a trigger, a computer program can cause a specific action to occur.

For example, in a computer game that shows animals on a visualization, the animals can run away when a user looks at them. In an example where the computer program is a horror-type game, enemies that intend to attack the user's character can only move towards the user when the user is not looking at the character.

In a further embodiment, commands can be issued to characters in a multi-character game by looking at the characters. This is particularly beneficial in games that contain a large number of characters, whereby commands must be issued to specific individual characters or groups of characters.

In a further embodiment, the windows or other subdivisions of a display can be dimmed or brightened depending on whether a user is looking at the windows or other subdivisions. For example, if a user is looking at a particular window, this increases the brightness compared to the rest of a visualization until the user looks at another window.

With reference to any or all of the embodiments described herein, a further improvement of the present invention is that information related to a user's interaction with the computer can be stored for analysis and display by the user. For example, the computer can determine the frequency, location, and outcome of each gaze or head-based interaction. This can be presented to the user so that the user can analyze their game or the game of another user. Such display can be in a data-driven format such as a table, or it may be presented in a visual format such as a recording or live display of a location of the user's gaze while the user interacts with the computer. For example, where the user provides a gaze or head input that is used by the intelligence of a game, the computer may record the type of input provided and the resulting output of the game.

For the purposes of this report, any reference to the terms computer, computing device or the like is intended to mean an electronic device that requires the input of a user, this includes, but is not limited to: personal computers, tablets, laptops, mobile phones, portable gaming devices, home entertainment systems, game consoles and other video game systems, televisions, microwaves, refrigerators, etc.

For the purposes of this specification, any reference to the terms eye tracking, gaze direction, gaze point, point of interest, point of interest information or the like is intended to refer to the information and / or detection of information that indicates a user's attention to a point of interest on an object, such as a visualization. A point of interest can be determined by a known point of interest input device such as, for example, a traditional eye tracking device that uses at least one image sensor, at least one light source, and a processing device to determining a point of interest from light reflections in a user's eyes. The point of interest, however, can be determined by other input devices such as gesture recognition systems, virtual reality headsets, face detection systems, conventional keyboard and mouse, handheld controller, etc.

For the purposes of this specification, any reference to the terms gaze detection, gaze information, gaze direction, eye tracking, point of interest, point of consideration, or the like is intended to mean the information and / or detection information that defines the position in which a user is looking, usually in a visualization or the like.

For the purposes of the present specification, any reference to the terms gaze detection device, eye tracker or the like is intended to mean a device by which the gaze direction of a user is determined. Typically, said devices comprise at least one image sensor and at least one illuminator. The at least one illuminator can emit infrared or near-infrared light that is reflected from the cornea of ​​a user's eye, and the image sensor captures an image comprising the cornea and a reflection. A computing unit determines the direction of the user's gaze from the position of the reflex on the cornea. The computing unit can be integrated with the gaze detection device on a printed circuit board or the like, or it can be integrated into the programming of a larger computing unit. Eye tracking devices

ES 2 633 016 T3 suitable for the present invention are known and will be readily understood by one of ordinary skill in the art, for example Tobii Technology AB of Sweden (www.tobii.com) provides suitable devices.

It is intended that any device that indicates the direction of a user's gaze can be used with the present invention, regardless of the technology comprised in said device, including the presence of lighting. Furthermore, the gaze direction information can be provided by a unit whose primary purpose is not gaze detection, such as a camera in a mobile phone.

For the purposes of this specification, any reference to the terms computer game, video game, interactive entertainment program, computer program, or the like is intended to mean a set of instructions executable by a computing device to provide or manipulate items displayed on a display. .

For the purposes of this specification, any reference to the terms computer, computing device, computing system, or the like is intended to mean any device capable of executing programmed instructions. This includes commonly known items such as, for example, desktop computers, laptops, netbooks, ultrabooks, tablets, mobile phones, portable game consoles, and other video game systems, such as those that are connected to a television for video purposes. entertainment and the like.

The computing devices or systems described herein are not limited to a particular hardware architecture or configuration. A computing device can include any appropriate arrangement of components that provides a result conditional on one or more inputs. Suitable computing devices include multi-function microprocessor-based computer systems that access stored software that programs or configures the computer system from a general-purpose computing apparatus to a specialized computing apparatus that implements one or more embodiments of the present object.

Embodiments of the present invention are suitable for both remote gaze detection devices and wearable gaze detection devices. Virtual Reality headsets such as Oculus Rift, wearable displays such as Google Glass and the like all have the ability to provide interactive episodes to a user. By providing the gaze and / or head tracking capability in a wearable device, the present invention can be utilized. In addition, the wearable device can provide the head or gaze information required for the present invention to work directly. As used in the present application, wearable devices or wearable displays and the like are intended to include any article that is adapted for a user to wear, such as glasses, helmets or the like.

By way of example and not limitation, Figure 8 is a block diagram illustrating an exemplary computing device 802 for implementing certain embodiments. Computing device 802 may include a processor 804 that communicatively couples to memory 806 and executes computer-executable program instructions and / or accesses information stored in memory 806. Processor 804 may comprise a microprocessor, an application-specific integrated circuit (ASIC), a state machine, or other processing device. Processor 804 can include any of a number of computer processing devices, including one. Such a processor may include or may be in communication with a computer-readable medium that stores instructions that, when executed by processor 804, cause the processor to perform the steps described herein.

The computing device 802 may also include a bus 808. The bus 808 may communicatively couple one or more components of the computing system 802. The computing device 802 may also include and / or be communicatively coupled to a number of external or internal devices, such as input or output devices. For example, computing device 802 is shown with input / output (I / O) interface 810, display device 812, input devices 814, and output devices 815.

Non-limiting examples of a display device 812 include a display integrated into computing device 802, an external monitor and coupled to the computing system, and so on. Non-restrictive examples of input devices 814 include gaze detection devices, touch screens, touch buttons, external mouse devices, microphones, and / or other devices mentioned herein, etc. A non-restrictive example of an 815 output device is a speaker. In some embodiments, the display device 812, input devices 814, and output devices 815 may be separate devices. In other embodiments, the display device 812 and at least some of the input devices 814 can be integrated into the same device. For example, a display device 812 may be a display and an input device 814 may be one or more components that provide eye-tracking and / or touch screen functions for the display device, such as emitters to broadcast. light and / or cameras to capture the image of a user's eyes and / or a touch area, etc. The screen, the components of the device 13

ES 2 633 016 T3 input and any component of the output device can be integrated within the same housing or in other integrated configurations.

The computing device 802 may modify, access, or otherwise use electronic content. Electronic content may reside on any appropriate non-transitory computer-readable medium and run on any suitable processor. In one embodiment, electronic content may reside in memory 806 on computing device 802. In another embodiment, electronic content can be accessed by computer system 802 from a remote content provider over a data network.

Memory 806 may include any appropriate non-transient computer-readable medium. A computer-readable medium may include, but is not limited to, an electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable instructions or other program code. Other examples include, but are not limited to, a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, ASIC, a configured processor, optical storage, magnetic tape or other magnetic storage or any other medium from the which a computer processor can read instructions. The instructions may comprise processor-specific instructions generated by a compiler and / or interpreter from code written in any appropriate computer programming language including, for example, C, C ++, C #, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript.

A graphics module 816 stored in memory 806 may configure processor 804 to prepare electronic content for rendering on a graphical interface and / or rendering of electronic content on the graphical interface. In some embodiments, graphics module 816 may be a standalone application run by processor 804. In other embodiments, graphics module 816 may be a software module included in or accessible by a separate application run by processor 804 that is configured to modify, access, or otherwise use electronic content.

It should be understood that the different methods described herein for interacting with and controlling computer devices and computer programs may be implemented in the form of computer-readable instructions or other program code, which may have several different and alternative functional arrangements. , processing flows, method steps, etc. Any appropriate programming, scripting, or other language or language combinations may be used to implement the teachings contained herein in the software to be used in the programming or configuration of a computing device.

Unless specifically stated otherwise, it is appreciated that throughout this specification, descriptions using terms such as, for example, processing, computing, calculating, determining and identifying or the like refer to actions or processes of a computing device. The use of adapted (s) to or configured (s) for herein has the sense of an open and inclusive language that does not exclude devices adapted to or configured to carry out additional tasks or steps. Furthermore, the use of based on (s) is intended to be open and inclusive, in the sense that a process, step, calculation or other action based on one or more described conditions or values ​​may, in practice, be based on conditions or additional values ​​beyond those established. Headings, lists, and numberings included herein are for ease of explanation only and are not intended to be restrictive.

The invention has now been described in detail for the purposes of clarity and understanding. However, it will be appreciated that certain changes and modifications can be made within the scope of the appended claims.