Systems and methods of eye tracking control on mobile device
Summary by NHIP
Eye tracking control on mobile devices
An eye tracking device synchronizes internal light emission with camera capture for a depth of field between 40 cm and 80 cm. The system filters light through an infrared pass filter to allow only 800-900 nm wavelengths into a camera matching the light emission field of view.
Claim Score by NHIP
Abstract
Methods and systems to facilitate eye tracking control on mobile devices are provided. An image of a portion of a user is received at an eye tracking device, where the image includes reflections caused by light emitted on the user from one or more light sources located within the eye tracking device. One or more eye features associated with an eye of the user is detected using the reflections. Point of regard information is determined using the one or more eye features, where the point of regard information indicates a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken. The point of regard information is sent to an application capable of performing a subsequent operation using the point of regard information.

Term
7.4 yearsleft in the term
Expires 10 February 2034.
- Priority
- Filed
- Granted
- Today
- Expires
18 claims: 4 independent, 14 dependent
- 1Broadest claimClaim Score 34, narrow(NHIP)A method comprising:controlling, at an eye tracking device, one or more light sources located within the eye tracking device to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of field approximately in the range 40 cm to 80 cm, wherein the controlling further includes filtering the light from the light sources using an infrared pass filter so that only light in the 800-900 nm range enters the camera, and wherein the camera has a field of view that matches the field of emission of the one or more light sources;receiving, at the eye tracking device, an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources located within the eye tracking device;detecting one or more eye features associated with an eye of the user using the reflections;determining point of regard information using the one or more eye features, the point of regard information indicating a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken;andsending the point of regard information to an application capable of performing a subsequent operation using the point of regard information.
- 7A non-transitory machine-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:controlling one or more light sources located within an eye tracking device to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of field approximately in the range 40 cm to 80 cm, wherein the controlling further includes filtering the light from the light sources using an infrared pass filter so that only light in the 800-900 nm range enters the camera, and wherein the camera has a field of view that matches the field of emission of the one or more light sources;receiving an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources located within an eye tracking device;detecting one or more eye features associated with an eye of the user using the reflections;determining point of regard information using the one or more eye features, the point of regard information indicating a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken;andsending the point of regard information to an application capable of performing a subsequent operation using the point of regard information.
- 10An eye tracking device comprising:one or more light sources configured to emit light, wherein the one or more sources further comprise an infrared pass at filters that to be within an 800-900 nm range;andone or more hardware processors in communication with the one or more light sources, the one or more hardware processors configured to: control the one or more light sources to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of field approximately in the range 40 cm to 80 cm, wherein the camera has a field of view that matches the field of emission of the one or more light sources;receive an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources;detect one or more eye features associated with an eye of the user including detecting the one or more eye features using the reflections;determine point of regard information using the one or more eye features, the point of regard information indicating a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken;andsend the point of regard information to an application capable of performing a subsequent operation using the point of regard information.
- 16A system comprising:a mobile computing device;andan eye tracking device coupled to and in communication with the mobile computing device, the eye tracking device being configured to: control one or more light sources located within the eve tracking device to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of held approximately in the range 30 cm to 60 cm, wherein to control the one or more light sources the eve tracking device is further configured to filter the light from the one or more light sources using an infrared pass filter so that only the 800-900 nm range enters the camera and wherein the camera has a field of view that matches the field of emission of the one or more light sources;receive an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources located within the eye tracking device;detect one or more eye features associated with an eye of the user using the reflections;determine point of regard information using the one or more eye features, the point of regard information indicating a location on a display of the mobile computing device at which the user was looking when the image of the portion of the user was taken;andsend the point of regard information to an application on the mobile computing device capable of performing a subsequent operation using the point of regard information.
Independent claims4
54 paragraphs in 4 sections, as filed
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims a priority benefit of U.S. Provisional Application No. 61/730,407, filed Nov. 27, 2012, entitled “Systems and Methods of Eye Tracking Control on Mobile Device,” which is incorporated herein by reference in its entirety.
TECHNICAL FIELD
The present disclosure generally relates to mobile devices and, more specifically, to systems and methods for facilitating eye tracking control on mobile devices.
BRIEF DESCRIPTION OF THE DRAWINGS
Some embodiments are illustrated by way of example and not of limitation in the figures of the accompanying drawings.
<figref idref="DRAWINGS">FIG. 1</figref> is a device diagram of an example mobile device coupled to a docking device capable of facilitating eye tracking control, according to some embodiments;
<figref idref="DRAWINGS">FIG. 2</figref> is a device diagram of another example of a mobile device coupled to a docking device capable of facilitating eye tracking control, according to some embodiments;
<figref idref="DRAWINGS">FIGS. 3A-3C</figref> are device diagrams of example mobile devices capable of facilitating eye tracking control, according to some embodiments;
<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram of an example software architecture for facilitating eye tracking control, according to some embodiments;
<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of an example flow of data used to facilitate eye tracking control, according to some embodiments;
<figref idref="DRAWINGS">FIG. 6</figref> is a flowchart of an example method for facilitate eye tracking control, according to some embodiments; and
<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram of a machine in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed, according to some embodiments.
DETAILED DESCRIPTION
Example systems and methods to facilitate eye tracking control on mobile devices are described. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of example embodiments. It will be evident, however, to one skilled in the art that the present technology may be practiced without these specific details.
A user of a mobile device may interact with and control objects and applications displayed on the mobile device through the user's eye movement. An image of the user's eyes and/or face, captured by a front-facing camera on or coupled to the mobile device, may be analyzed using computer-vision algorithms, such as, for example, eye tracking algorithms and gaze detection algorithms. For example, the captured images may be processed to extract information relating to one or more features of the user's eyes and/or face. The mobile device may then use the extracted information to determine the location of the user's eyes and estimate the location on the display at which the user is looking For example, the mobile device may be able to estimate at which icon on the display the user is looking The estimation of where the user is looking may be used to direct one or more objects, applications, and the like to perform a particular operation. For example, the user may direct and control the movement of an object on the screen depending on where the user is looking on the display of the mobile device, including controlling scrolling functions, the movement of objects in a virtual game, and the like.
<figref idref="DRAWINGS">FIG. 1</figref> is a device diagram <b>100</b> of an example mobile device <b>102</b> coupled to a docking device <b>104</b> capable of facilitating eye tracking control. The mobile device <b>102</b> may be any type of mobile device, including, but not limited to, a smart phone, a personal digital assistant (PDA), a mobile phone, a computing tablet, an electronic reader, and the like. During eye tracking control, the mobile device <b>102</b> may be used by the user by holding the mobile device <b>102</b> with one hand, both hands, or while the mobile device <b>102</b> is on a stand.
A docking device <b>104</b> may be coupled to the mobile device <b>102</b> in any manner, such as through a USB port on the mobile device <b>102</b>, micro USB port on the mobile device <b>102</b>, and the like. While the docking device <b>104</b> of <figref idref="DRAWINGS">FIG. 1</figref> is depicted at the bottom of the mobile device <b>102</b>, one of ordinary skill in the art will appreciate that the docking device <b>104</b> may be located at any suitable location relative to the mobile device <b>102</b>. The docking device <b>104</b> may include a camera module <b>108</b> and one or more light-emitting diodes (LEDs) <b>106</b>. For explanatory purposes, LEDs <b>106</b> are depicted and described throughout the disclosure. However, one of ordinary skill in the art will appreciate that any appropriate light-emitting source may be used (e.g., infrared laser).
The docking device <b>104</b> may include any number of infrared LEDs <b>106</b> that may be placed in a suitable location in any manner within the docking device <b>104</b> (e.g., tilted at an angle such that it points toward the user's face). In some embodiments, the docking device <b>104</b> may have either three or six LEDs <b>106</b> (e.g., for batteries with voltage=5V). However, any number of LEDs and any arrangement may be used with or without voltage divider circuitry. In a specific embodiment, the one or more LEDs <b>106</b> may have any one or more of the following features or characteristics: 1 Watt or 3 Watts emitting at 850 nm, emitting light at approximately 30-40 degrees field of emission, placed in pairs (e.g., to bring the voltage level to an appropriate voltage (e.g., from 3.7V to 1.85V) based on the current the LEDs need), blinks in the same frequency as the camera records, and the like.
In some embodiments, the docking device <b>104</b> may also include a suitable type of infrared pass filter (e.g., active, mechanical, high-pass, band-pass, etc.). In some embodiments, a high-pass filter that blocks light below 800 nm and allows light above 800 nm is used. In some embodiments, the infrared pass filter may only allow light between 800-900 nm to enter the one or more cameras of the camera module <b>108</b>.
The camera module <b>108</b> may include one or more front-facing cameras placed in any suitable location in any manner within the docking device <b>104</b> (e.g., tilted at an angle such that it points toward the user's face) and may be used to capture images of the user's eyes and/or face. The one or more cameras may be placed at an appropriate distance from the LEDs to optimize the proper capture of the infrared light. In some embodiments, a camera on the mobile device <b>102</b> is used in combination with camera module <b>108</b> in stereo mode. The camera module <b>108</b> may include any one or more of the following: a black and white (e.g., monochrome) or color (e.g., RGB) CMOS sensor, running at an appropriate frame per second rate (e.g., minimum high-definition at 30 frames per second), a lens without an infrared block filter and with an appropriate field of view (e.g., approximately 35 degrees) and depth of field (e.g., approximately 40-80 cm), and the like. The one or more cameras in the camera module <b>108</b> may be positioned such that the one or more cameras are tilted upward (e.g., toward a user's face).
The images captured by the camera may need to be rotated. The eye tracking software can use sensors on the mobile device <b>102</b> (e.g., accelerometer, magnetometer, etc.) to detect the orientation of the mobile device <b>102</b> and rotate the image accordingly so that it can be properly processed.
The LEDs <b>106</b> emit light that is focused and centered toward the eyes of the user. The infrared light from the LEDs <b>106</b> is reflected in the pupil and on the cornea of the user and recorded by the cameras in the camera module <b>108</b>. The LEDs <b>106</b> may be synchronized with the one or more cameras so that the LEDs <b>106</b> are on only when the one or more cameras are grabbing an image. In some embodiments, to improve the image quality, the visible light below 800 nm is filtered out using an infrared pass filter. The field of view and depth of view of the lenses of the one or more cameras in the camera module <b>108</b> may allow the user to move around, thereby accommodating for head pose variance of the user. The eye tracking control software may analyze the images taken by the camera module <b>108</b> to provide x,y coordinates of where the user is looking on the display of the mobile device <b>102</b>. The x,y coordinates may be used for any number of applications (e.g., scrolling, moving objects, selecting icons, playing games, etc.).
The LEDs <b>106</b> and the camera module <b>108</b> may be turned on and/or off in any manner, such as by utilizing an external slider, an on-off dedicated button on the side or on the back of either the mobile device <b>102</b> or the docking device <b>104</b>, controlled by an application or a digital button on the screen, controlled by movement or shaking of the mobile device <b>102</b> and/or the docking device <b>104</b>, controlled by voice commands, on-screen capacitive buttons, touch pad(s), bio-signals (e.g., EMG, EEG, etc.) and the like. As such, in some embodiments, the eye tracking components may consume power only while the LEDs and the camera are turned on (e.g., when the user is using the eye tracking features).
In some embodiments, the eye tracking features are optimized when the camera is located at the bottom of the mobile device <b>102</b> (e.g., with respect to the perspective of the user). The user may rotate the mobile device <b>102</b> coupled to the docking device <b>104</b> to properly orient the camera module <b>108</b> such that it is located at the bottom of the mobile device <b>102</b>. In some embodiments, using the accelerometer and/or magnetometer of the mobile device <b>102</b>, the LEDs, the pass filter, and/or the camera may be turned on and/or off depending on the orientation of the mobile device <b>102</b> and the docking device <b>104</b> (e.g., turn off the LEDs and the camera when the mobile device <b>102</b> and the docking device <b>104</b> are rotated such that the camera module <b>108</b> is located at the top of the mobile device <b>102</b> with respect to the perspective of the user).
The LEDs and the camera may be turned off when the user's face is not recognized for a predetermined amount of time (e.g., 5-10 seconds) and may turn on again when the user's face is detected and recognized.
<figref idref="DRAWINGS">FIG. 2</figref> is a device diagram <b>200</b> of another example of a mobile device <b>202</b> coupled to a docking device <b>204</b> capable of facilitating eye tracking control. The example shown in <figref idref="DRAWINGS">FIG. 2</figref> may operate similarly to the example shown in <figref idref="DRAWINGS">FIG. 1</figref> and may incorporate any one or combination of features described for <figref idref="DRAWINGS">FIG. 1</figref>. However, <figref idref="DRAWINGS">FIG. 2</figref> shows that the docking device <b>204</b> may be integrated with LEDs <b>206</b>, and the camera module <b>208</b> of the mobile device <b>202</b> may be used (instead of the camera module being integrated with the docking device <b>204</b>). In some embodiments which couple the mobile device <b>202</b> with the docking device <b>204</b> using a USB, a micro-USB port, or a proprietary port, the configuration depicted in <figref idref="DRAWINGS">FIG. 2</figref> may allow for faster transfer of images from the camera since the camera of the mobile device <b>202</b> is used to capture the images. The front-facing camera for eye tracking control may be utilized while simultaneously utilizing one or more back-facing cameras.
<figref idref="DRAWINGS">FIGS. 3A-3C</figref> are device diagrams of example mobile devices capable of facilitating eye tracking control. The exampled shown in <figref idref="DRAWINGS">FIGS. 3A-3C</figref> may operate similarly to the example shown in <figref idref="DRAWINGS">FIG. 1</figref> and may incorporate any one or combination of features described for <figref idref="DRAWINGS">FIG. 1</figref>. However, the LEDs and camera modules are integrated into the mobile device (instead of being part of a docking device). <figref idref="DRAWINGS">FIGS. 3A-3C</figref> depict mobile devices <b>300</b>, <b>310</b>, <b>320</b>, respectively, with LEDs <b>302</b>, <b>312</b>, <b>322</b> and camera modules <b>304</b>, <b>314</b>, <b>324</b> integrated into the mobile devices <b>300</b>, <b>310</b>, and <b>320</b> in different example configurations (with respect to the user's perspective).
The LEDs and the camera modules on the mobile devices may be located in any one of a number of configurations on the mobile devices. <figref idref="DRAWINGS">FIG. 3A</figref> shows the LEDs <b>302</b> and the camera module <b>304</b> being located at the bottom of the mobile device <b>300</b>. <figref idref="DRAWINGS">FIG. 3B</figref> shows the LEDs <b>312</b> being located on one side of the mobile device <b>310</b> while the camera module <b>314</b> is located on the opposite side of the mobile device <b>310</b>. <figref idref="DRAWINGS">FIG. 3C</figref> shows the LEDs <b>302</b> and the camera module <b>304</b> being located on the same side of the mobile device <b>300</b>.
<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram of an example software architecture for facilitating eye tracking control. Any one or more of the components of the software architecture may run on either a control processing unit (CPU) of the mobile device or on a combination of a CPU and a graphics processing unit (GPU) of the mobile device. In some embodiments, any one or more of the components of the software architecture may run on a dedicated chip. The software may run as a background process (e.g. as part of the operating system (OS), in a web browser, etc.) and may provide an application programming interface (API) that other applications can access. The API may fire an event or use some other similar mechanism to send the information of where the user is looking on the screen to other applications.
The software architecture may be divided into different layers. The bottom layer would correspond to the hardware (e.g. the camera(s), the infrared illumination, etc.). A camera layer may be in charge of communicating with the camera(s) in order to perform camera operations such as, for example, starting the camera, grabbing images, controlling the camera properties, and the like. This layer may also synchronize the one or more cameras and the infrared emitters so that the lights are turned on when there is an image being captured and off the rest of the time (e.g. strobing).
The camera layer may deliver images to the eye tracking layer. In the eye tracking layer, images may be processed to find features like face location, eye region location, pupil center, pupil size, location of the corneal reflections, eye corners, iris center, iris size, and the like. These features are used in the gaze estimation stage, which may be in charge of calculating the point of regard of the user, which may be the location on the display where the user is looking The gaze estimation stage may also calculate the optical and visual axes of the user's eyes.
The API layer may be used for communication between the eye tracking layer and applications that use eye gaze information (e.g., OS API, games that employ eye gaze information, etc.). The API may send data calculated by the eye tracking layer, such as coordinates of the point of regard, three-dimensional (3-D) location of the user's eyes, pupil size, and the like. The API may also accept commands from an application to the eye tracking layer (e.g., to start and/or stop the eye tracking engine, query for specific information, etc.). An application may connect to the eye tracker's API and use eye gaze information for any suitable purpose (e.g., control an app or a game, record eye data for visual behavior studies, etc.).
<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of an example flow of data used to facilitate eye tracking control. The one or more cameras and the infrared LED illumination may capture an image of the user and use the captured data to detect eye features (e.g., location of eye(s), pupils, corneal reflections, etc.). Using the detected eye features, the gaze estimation module may estimate the user's point of regard, which may then be used to control aspects of an application.
A calibration process may be conducted the first time the user uses the eye tracking functionality in order to calculate personal parameters (e.g., vertical and horizontal offset between optical and visual axes). These personal parameters and the information of the face and eyes are then employed to estimate where the user is looking on the screen through a gaze estimation algorithm.
<figref idref="DRAWINGS">FIG. 6</figref> is a flowchart of an example method <b>600</b> for facilitate eye tracking control. The method <b>600</b> may be performed using the mobile device, cameras, and LEDs in any configuration.
In operation <b>602</b>, an image of a portion of a user is received at the mobile device. The image includes reflections (e.g., corneal reflections) caused by light emitted on the user.
In operation <b>604</b>, eye features (e.g., pupil location, size, corneal reflection location, eye corners, iris location, etc.) of the user are detected using the reflections received.
In operation <b>606</b>, a point of regard is determined for the user using the eye features detected. Optical and/or visual axes may also be determined. The determination of the point of regard may account for the location of the one or more camera and the LEDs with respect to the screen.
In operation <b>608</b>, the point of regard information is sent to an application capable of using the point of regard information in a subsequent operation.
Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A hardware module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.
In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.
Accordingly, the term “hardware module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where the hardware modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.
Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple of such hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation, and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).
The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules.
Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment or as a server farm), while in other embodiments the processors may be distributed across a number of locations.
The one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., Application Program Interfaces (APIs)).
Example embodiments may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Example embodiments may be implemented using a computer program product, e.g., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable medium for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers.
A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
In example embodiments, operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations can also be performed by, and apparatus of example embodiments may be implemented as, special purpose logic circuitry (e.g., a FPGA or an ASIC).
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In embodiments deploying a programmable computing system, it will be appreciated that that both hardware and software architectures require consideration. Specifically, it will be appreciated that the choice of whether to implement certain functionality in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or a combination of permanently and temporarily configured hardware may be a design choice. Below are set out hardware (e.g., machine) and software architectures that may be deployed, in various example embodiments.
<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram of a machine in the example form of a computer system <b>700</b> within which instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
Example computer system <b>700</b> includes a processor <b>702</b> (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory <b>704</b>, and a static memory <b>706</b>, which communicate with each other via a bus <b>708</b>. Computer system <b>700</b> may further include a video display device <b>710</b> (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). Computer system <b>700</b> also includes an alphanumeric input device <b>712</b> (e.g., a keyboard), a user interface (UI) navigation device <b>714</b> (e.g., a mouse or touch sensitive display), a disk drive unit <b>716</b>, a signal generation device <b>718</b> (e.g., a speaker) and a network interface device <b>720</b>.
Disk drive unit <b>716</b> includes a machine-readable medium <b>722</b> on which is stored one or more sets of instructions and data structures (e.g., software) <b>724</b> embodying or utilized by any one or more of the methodologies or functions described herein. Instructions <b>724</b> may also reside, completely or at least partially, within main memory <b>704</b>, within static memory <b>706</b>, and/or within processor <b>702</b> during execution thereof by computer system <b>700</b>, main memory <b>704</b> and processor <b>702</b> also constituting machine-readable media.
While machine-readable medium <b>722</b> is shown in an example embodiment to be a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions or data structures. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing, encoding or carrying instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present technology, or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, including by way of example semiconductor memory devices, e.g., Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
Instructions <b>724</b> may further be transmitted or received over a communications network <b>726</b> using a transmission medium. Instructions <b>724</b> may be transmitted using network interface device <b>720</b> and any one of a number of well-known transfer protocols (e.g., HTTP). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), the Internet, mobile telephone networks, Plain Old Telephone (POTS) networks, and wireless data networks (e.g., WiFi and WiMAX networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such software.
Although an embodiment has been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the technology. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof, show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments illustrated are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. This Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.
Such embodiments of the inventive subject matter may be referred to herein, individually and/or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed. Thus, although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.
Contents4
10 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10
Every citation, both waysCites: the store holds 233 of 234
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US10996753B1 | Cited by | United States of America | Search report |
| US11525714B2 | Cited by | United States of America | Search report |
| WO0152722A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| EP0816983A2 | Cites | European Patent Office (EPO) | Applicant |
| CN100589752C | Cites | China | Applicant |
| CN101282680A | Cites | China | Applicant |
| EP1114608A1 | Cites | European Patent Office (EPO) | Applicant |
| EP1336372A2 | Cites | European Patent Office (EPO) | Applicant |
| EP1679577A1 | Cites | European Patent Office (EPO) | Applicant |
| DE19731301A1 | Cites | Germany | Applicant |
| JP2000137792A | Cites | Japan | Applicant |
| US2002036617A1 | Cites | United States of America | Applicant |
| US2002070965A1 | Cites | United States of America | Applicant |
| US2002070966A1 | Cites | United States of America | Applicant |
| US2002070968A1 | Cites | United States of America | Applicant |
| US2002075384A1 | Cites | United States of America | Search report |
| US2002105482A1 | Cites | United States of America | Applicant |
| US2002180801A1 | Cites | United States of America | Applicant |
| US2003038754A1 | Cites | United States of America | Applicant |
| US2003123027A1 | Cites | United States of America | Applicant |
| US2003179314A1 | Cites | United States of America | Applicant |
| US2004001100A1 | Cites | United States of America | Applicant |
| US2004005083A1 | Cites | United States of America | Applicant |
| WO2004045399A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2004061831A1 | Cites | United States of America | Search report |
| US2004073827A1 | Cites | United States of America | Applicant |
| WO2004090581A2 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2004213463A1 | Cites | United States of America | Applicant |
| WO2005046465A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2005047629A1 | Cites | United States of America | Search report |
| US2005050686A1 | Cites | United States of America | Applicant |
| US2005100191A1 | Cites | United States of America | Applicant |
| US2005110887A1 | Cites | United States of America | Applicant |
| US2005119642A1 | Cites | United States of America | Applicant |
| US2005199783A1 | Cites | United States of America | Applicant |
| US2005243054A1 | Cites | United States of America | Applicant |
| US2005289363A1 | Cites | United States of America | Applicant |
| WO2006016366A2 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| WO2006020408A2 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2006047386A1 | Cites | United States of America | Applicant |
| US2006093998A1 | Cites | United States of America | Applicant |
| US2006120599A1 | Cites | United States of America | Applicant |
| US2006238707A1 | Cites | United States of America | Applicant |
| US2006239670A1 | Cites | United States of America | Applicant |
| WO2007076479A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| WO2007085682A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| WO2007101690A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2007122140A1 | Cites | United States of America | Applicant |
| US2007159599A1 | Cites | United States of America | Applicant |
| US2007164990A1 | Cites | United States of America | Applicant |
| US2007230933A1 | Cites | United States of America | Applicant |
| US2007282506A1 | Cites | United States of America | Applicant |
| KR20080106218A | Cites | Republic of Korea | Applicant |
| WO2008056274A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| WO2008066460A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2008079687A1 | Cites | United States of America | Applicant |
| US2008122792A1 | Cites | United States of America | Applicant |
| US2008284980A1 | Cites | United States of America | Applicant |
| US2008297614A1 | Cites | United States of America | Search report |
| WO2009022924A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2009027336A1 | Cites | United States of America | Applicant |
| WO2009027773A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2009067680A1 | Cites | United States of America | Applicant |
| US2009073128A1 | Cites | United States of America | Applicant |
| US2009073142A1 | Cites | United States of America | Applicant |
| WO2009101238A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2009103048A1 | Cites | United States of America | Applicant |
| US2009125849A1 | Cites | United States of America | Applicant |
| US2009179853A1 | Cites | United States of America | Applicant |
| US2009251407A1 | Cites | United States of America | Applicant |
| US2009263012A1 | Cites | United States of America | Applicant |
| US2009273562A1 | Cites | United States of America | Applicant |
| US2010066975A1 | Cites | United States of America | Applicant |
| US2010077421A1 | Cites | United States of America | Search report |
| US2010079508A1 | Cites | United States of America | Applicant |
| WO2010127714A2 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2010195881A1 | Cites | United States of America | Applicant |
| US2010205667A1 | Cites | United States of America | Applicant |
| US2010262281A1 | Cites | United States of America | Applicant |
| US2010272363A1 | Cites | United States of America | Applicant |
| KR20110124246A | Cites | Republic of Korea | Applicant |
| US2011013007A1 | Cites | United States of America | Applicant |
| US2011029918A1 | Cites | United States of America | Applicant |
| US2011051239A1 | Cites | United States of America | Applicant |
| JP2011081807A | Cites | Japan | Applicant |
| WO2011089199A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| JP2011115606A | Cites | Japan | Applicant |
| US2011304706A1 | Cites | United States of America | Applicant |
| US2012105490A1 | Cites | United States of America | Search report |
| US2012272179A1 | Cites | United States of America | Applicant |
| US2012319972A1 | Cites | United States of America | Applicant |
| US2012327101A1 | Cites | United States of America | Applicant |
| WO2013060826A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2013076885A1 | Cites | United States of America | Applicant |
| WO2013102551A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2013106681A1 | Cites | United States of America | Applicant |
| US2013107214A1 | Cites | United States of America | Applicant |
| EP2037352A2 | Cites | European Patent Office (EPO) | Applicant |
| EP2581034A1 | Cites | European Patent Office (EPO) | Applicant |
| DE4122752A1 | Cites | Germany | Applicant |
6 priority claims, no other members on record
Priority claims6
| Document | Office | Kind | Date |
|---|---|---|---|
| 201261730407 | United States of America | P | |
| 201261730407 | United States of America | P | |
| 201314089609 | United States of America | A | |
| 61730407 | – | – | – |
| US201261730407P | – | – | – |
| US201314089609 | – | – | – |
73 transactions on the USPTO file
Allowed after 2 non-final rejections, 1 final rejection and 1 RCE.
- Non-final rejections
- 2
- Final rejections
- 1
- RCEs
- 1
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Post Issue Communication - Certificate of CorrectionN423 | N423 | |
| Recordation of Patent Grant MailedPGM/ | PGM/ | |
| Patent Issue Date Used in PTA CalculationAllowedPTAC | PTAC | |
| Email NotificationEML_NTR | EML_NTR | |
| Issue Notification MailedAllowedWPIR | WPIR | |
| Dispatch to FDCD1935 | D1935 | |
| Application Is Considered Ready for IssuePILS | PILS | |
| Issue Fee Payment VerifiedN084 | N084 | |
| Entity status set to undiscounted (initial default setting or status change)BIG. | BIG. | |
| Issue Fee Payment ReceivedIFEE | IFEE | |
| Email NotificationEML_NTR | EML_NTR | |
| Email NotificationEML_NTR | EML_NTR | |
| Filing Receipt - CorrectedFLRCPT.C | FLRCPT.C | |
| Change in Power of Attorney (May Include Associate POA)PA.. | PA.. | |
| Correspondence Address ChangeC.AD | C.AD | |
| Miscellaneous Incoming LetterLET. | LET. | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Notice of AllowanceAllowedMN/=. | MN/=. | |
| Notice of Allowance Data Verification CompletedAllowedN/=. | N/=. | |
| Reasons for AllowanceEX.R | EX.R | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Non-Final ActionA... | A... | |
| Request for Extension of Time - GrantedXT/G | XT/G | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Disposal for a RCE / CPA / R129AbandonedABN9 | ABN9 | |
| Request for Continued Examination (RCE)RCEX | RCEX | |
| Workflow - Request for RCE - BeginBRCE | BRCE | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Final Rejection (PTOL - 326)Final rejectionMCTFR | MCTFR | |
| Final RejectionFinal rejectionCTFR | CTFR | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Email NotificationEML_NTR | EML_NTR | |
| Email NotificationEML_NTR | EML_NTR | |
| Filing Receipt - ReplacementFLRCPT.R | FLRCPT.R | |
| Change in Power of Attorney (May Include Associate POA)PA.. | PA.. | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Response after Non-Final ActionA... | A... | |
| Request for Extension of Time - GrantedXT/G | XT/G | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Application ready for PDX access by participating foreign officesCCRDY | CCRDY | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Oath or Declaration Filed (Including Supplemental)C602 | C602 | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Email NotificationEML_NTR | EML_NTR | |
| PG-Pub Issue NotificationPG-ISSUE | PG-ISSUE | |
| Reference capture on IDSRCAP | RCAP | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| FITF set to NO - revise initial settingFTFI | FTFI | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Email NotificationEML_NTR | EML_NTR | |
| Application Is Now CompleteCOMP | COMP | |
| Filing ReceiptFLRCPT.O | FLRCPT.O | |
| Sent to Classification ContractorPGPC | PGPC | |
| Applicant Has Filed a Verified Statement of Small Entity Status in Compliance with 37 CFR 1.27SMAL | SMAL | |
| Cleared by OIPE CSRL194 | L194 | |
| Applicants have given acceptable permission for participating foreignAPPERMS | APPERMS | |
| IFW Scan & PACR Auto Security ReviewSCAN | SCAN | |
| Entity status set to undiscounted (initial default setting or status change)BIG. | BIG. | |
| Initial Exam Team nnIEXX | IEXX |
11 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Maintenance fee paymentMAFP | MAFP | |
| AssignmentAS | AS | |
| Maintenance fee paymentMAFP | MAFP | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| Certificate of correctionCC | CC | |
| Information on status: patent grantGrantedPATENTED CASESTCF | STCF | |
| AssignmentAS | AS | |
| AssignmentAS | AS | |
| AssignmentAS | AS |
Numbers
- Publication
- 09612656
- Publication, DOCDB
- 9612656
- Publication, EPODOC
- US9612656
- Application
- 14089609
- Application, DOCDB
- 201314089609
- Application, EPODOC
- US201314089609
Titles
- English
- Systems and methods of eye tracking control on mobile device
Classification
- CPC, 9
- G06F3/013
- G02B5/208
- G06K9/00604
- G06V40/19
- G06K9/0061
- G06V40/193
- G06T7/74
- G06T2207/10048
- G06T2207/30201
- IPC, 1
- G06F3 01
- USPC, 1
- 001001000