MX348608B

Calibration of virtual reality systems.

Abstract

A virtual reality (VR) console receives slow calibration data from an image processing device, and fast calibration data from an inertial measurement unit in a virtual reality (VR) viewer that includes a front rigid body and a body rigid rear. The slow calibration data includes an image where so The locators are visible on the rear rigid body. An observed position is determined from the slow calibration data, and a predicted position is determined from the fast calibration data. If a difference between the observed position and the predicted position is greater than a threshold value, the predicted position is adjusted by a time offset until the difference is less than the threshold value. Temporal offset is removed by recalibrating the rear rigid body with the front rigid body once the locators on both the front rigid body and the rear rigid body are visible in an image in the slow calibration data.

MX348608B, drawing sheet 1 — Sheet 1 of 57

Term

8.3 yearsleft in the term

Expires 6 January 2035.

19 claims: 3 independent, 16 dependent

1
REIVINDICACIONES 1. Un sistema para calibración de sistemas de realidad virtual que comprende:un visor de realidad virtual (VR) que incluye una pluralidad de localizadores y una unidad de medición inercial (IMU) para la producción de datos de calibración rápida que comprenden una o más posiciones estimadas intermedias de un punto de referencia en el visor de realidad virtual (VR), estando cada posición estimada intermedia separada de una posición estimada intermedia subsiguiente por un valor de tiempo de posición;un dispositivo de procesamiento de imágenes para la producción de datos de calibración lenta, que incluyen una serie de imágenes que muestran porciones de localizadores observados de la pluralidad de localizadores, en el visor de realidad virtual (VR), estando cada imagen separada de una imagen subsiguiente en la serie por un valor de tiempo de imagen que es mayor que el valor de tiempo de posición;una consola de realidad virtual (VR) que comprende: un procesador, y una memoria acoplada al procesador y que provoca que el procesador rastree el visor de realidad virtual (VR) utilizando los datos de calibración lenta y los datos de calibración rápida, la memoria almacena módulos, los módulos que comprenden: un módulo de estimación para: identificar localizadores de modelo, cada uno correspondiente a un localizador en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando un modelo de visor almacenado asociado con el visor de realidad virtual (VR);generar las posiciones estimadas de uno o más de los localizadores en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando el modelo de visor;un módulo de ajuste de parámetros para: ajustar uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes, sea menor que un valor umbral;generar las posiciones calibradas del punto de referencia basándose, cuando menos en parte, en las posiciones estimadas ajustadas de uno ó más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta, una posición calibrada asociada con una imagen a partir de los datos de calibración lenta;determinar una o más posiciones predichas del punto de referencia basándose, cuando menos en parte, en las IMPIAS «ππνη mjuuciw «5«ájr3í> EMU MiMlWXUJ posiciones calibradas del punto de referencia, una popretffSI’ra asociada con un tiempo entre imágenes subsigΟΊΰ11tus d paflh dctuj datos de calibración lenta;y ajustar uno o más de los parámetros de calibración, de tal manera que la posiciones estimadas intermedias del punto de referencia estén dentro de una distancia umbral de las posiciones predichas determinadas del punto de referencia.
2
El sistema de acuerdo con la reivindicación 1, en donde la unidad de medición inercial (IMU) incluye un giroscopio de tres ejes para medir la velocidad angular.
3
El sistema de acuerdo con la reivindicación 1, en donde la unidad de medición inercial (IMU) incluye un acelerómetro de tres ejes.
4
El sistema de acuerdo con la reivindicación 1, en donde la unidad de medición inercial (IMU) incluye un magnetómetro de tres ejes.
5
El sistema de acuerdo con la reivindicación 1, en donde la pluralidad de localizadores están dispuestos en un patrón en el visor de realidad virtual (VR) que no es coplanar.
6
El sistema de acuerdo con la reivindicación 1, en donde la pluralidad de localizadores son diodos emisores de luz (LEDs).
7
El sistema de acuerdo con la reivindicación 6, en donde los LEDs son modulados para mantener uno de dos o más niveles de brillantez predeterminados durante un intervalo de tiempo.
8
El sistema de acuerdo con la reivindicación 7, en donde la modulación de los LEDs es seleccionada de un grupo que consisTé de:modulación de amplitud, modulación de frecuencia y cualquier combinación de los mismos. 5
9
El sistema de acuerdo con la reivindicación 6, en donde los LEDs emiten luz dentro de una banda específica seleccionada de un grupo que consiste de una banda visible y una banda infrarroja.
10
El sistema de acuerdo con la reivindicación 7, en donde la pluralidad de localizadores emiten en la banda infrarroja y una 10 superficie exterior del visor de realidad virtual (VR) es transparente en la banda infrarroja, pero opaca en la banda visible.
11
El sistema de acuerdo con la reivindicación 1, en donde el visor de realidad virtual (VR) incluye un cuerpo rígido frontal y un cuerpo rígido trasero, cada uno tiene uno o más localizadores, y el 15 cuerpo rígido frontal incluye la unidad de medición inercial (IMU).
12
El sistema de acuerdo con la reivindicación 11, en donde el cuerpo rígido frontal está acoplado de manera no rígida al cuerpo rígido trasero medíante una banda elástica.
13
El sistema de acuerdo con la reivindicación 11, en donde 20 la memoria además provoca que el procesador:determine una posición observada del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando los datos de calibración lenta;determine una posición predicha del cuerpo rígido trasero para 25 el valor de tiempo de imagen particular utilizando los datos de calibración rápida y un vector de posición que describe un desfasamiento calibrado entre el cuerpo rígido frontal y el cuerpo rígido trasero;determine que una diferencia entre la posición observada y la posición predicha es mayor que un valor de umbral;en respuesta a la determinación de que una diferencia entre la posición observada del cuerpo rígido trasero y la posición predicha del cuerpo rígido trasero es mayor que un valor umbral, ajuste el vector de posición por un valor de desfasamiento, de tal manera que la diferencia entre la posición observada y la posición predicha es menor que el valor umbral;y determine una posición predicha subsiguiente del cuerpo rígido trasero para una imagen a partir de la serie de imágenes asociadas con un valor de tiempo de imagen subsiguiente que se presenta después del valor de tiempo de imagen basándose en los datos de calibración rápida y el vector de posición ajustado.
14
Un sistema para calibración de sistemas de realidad virtual que comprende:un visor de realidad virtual (VR) que incluye una pluralidad de localizadores y una unidad de medición ¡nercial (IMU) para producir los datos de calibración rápida que comprenden una o más posiciones estimadas intermedias de un punto de referencia en el visor de realidad virtual (VR), cada posición estimada intermedia separada a partir de una posición estimada intermedia por un valor de tiempo de posición, y la unidad de medición ¡nercial (IMU) se acopla a un sensor de posición;un dispositivo de procesamiento de i datos de calibración lenta, que incluyen una serie de imágenes que muestran porciones de localizadores observados de la pluralidad de localizadores en el visor de realidad virtual (VR), cada imagen separada de una imagen subsiguiente en las series por un valor de tiempo de imagen que es mayor que el valor de tiempo de posición;y una consola de realidad virtual (VR) que comprende: un procesador, y una memoria acoplada al procesador que provoca que el procesador rastree el visor de realidad virtual (VR) y calibre el visor de realidad virtual (VR) utilizando los datos de calibración lenta y los datos de calibración rápida, la memoria almacena módulos, los módulos que comprenden: un módulo de estimación para: identificar localizadores de modelo, cada uno correspondiente a un localizador en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando un modelo de visor almacenado asociado con el visor de realidad virtual (VR);y generar las posiciones estimadas de uno o más de los localizadores en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando el modelo de visor;y un módulo de ajuste de parámetros para: ajustar uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una las posiciones estimadas ajustadas de uno o más r igs...__ localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes, sea menor que un valor umbral;generar posiciones calibradas del punto de referencia basándose, cuando menos en parte, en las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta, una posición calibrada asociada con una imagen a partir de los datos de calibración lenta;determinar una o más posiciones predichas del punto de referencia basándose, cuando menos en parte, en las posiciones calibradas del punto de referencia, una posición predicha asociada con un tiempo entre imágenes subsiguientes a partir de los datos de calibración lenta;y ajustar uno o más de los parámetros de calibración, de tal manera que la posiciones estimadas intermedias del punto de referencia estén dentro de una distancia umbral de las posiciones predichas determinadas del punto de referencia.
15
El sistema de acuerdo con la reivindicación 14, en donde la pluralidad de localizadores incluye un localizador que es un LED que es modulado para mantener uno de dos o más niveles de brillantez predeterminados durante un intervalo de tiempo. 100 IMPI OíMnortEDAD IMXJSTliAL de acuerdo con la reivindicación 15, en
16
El sistema la modulación del LED es seleccionada de un grupo que consisté^TS~—— modulación de amplitud, modulación de frecuencia y una combinación de los mismos.
17
El sistema de acuerdo con la reivindicación 15, en donde el LED emite luz dentro de una banda específica seleccionada de un grupo que consiste de una banda visible y una banda infrarroja.
18
El sistema de acuerdo con la reivindicación 17, en donde el LED emite en la banda infrarroja y una superficie exterior del visor de realidad virtual (VR) es transparente en la banda infrarroja, pero opaca en la banda visible.
19
El sistema de acuerdo con la reivindicación 14, en donde el visor de realidad virtual (VR) incluye un cuerpo rígido frontal y un cuerpo rígido trasero, cada uno tiene uno o más localizadores, y el cuerpo rígido frontal incluye la unidad de medición inercial (IMU). 101 IMPI eenwro MIXKAXO osla nzjrtHM» ítWSTFIAL

Independent claims19

409 paragraphs in 31 sections, as filed

(54) Title: CALIBRATION OF VIRTUAL REALITY SYSTEMS.

(57) Summary

A virtual reality (VR) console receives slow calibration data from an image processing device, and fast calibration data from an commercial measurement unit in a virtual reality (VR) viewer that includes a front rigid body and a rear rigid body. The slow calibration data includes an image where only the locators are visible on the rear rigid body. An observed position is determined from the slow calibration data, and a predicted position is determined from the fast calibration data. If a difference between the observed position and the predicted position is greater than a threshold value, the predicted position is adjusted by a time offset until the difference is less than the threshold value. Temporal offset is removed by recalibrating the rear rigid body with the front rigid body once the locators on both the front rigid body and the rear rigid body are visible in an image in the slow calibration data.

(57) Abstract

A virtual reality (VR) consolé receives slow calibration data from an imaging device and fast calibration data from an ¡nertlal measurement unlt on a VR headset ¡ncludlng a front and a rear rigid body. The slow calibration data includes an image where only the locators on the rear rigid body are visible. An observed position is determined from the slow calibration data and a predicted position is determined from the fast calibration data. If a difference between the observed position and the predicted position is greater than a threshold value, the predicted position is adjusted by a temporary offset until the difference is less than the threshold value. The temporary offset is removed by re-calibrating the rear rigid body to the front rigid body once locators on both the front and rear rigid body are visible in an image in the slow calibration data.

PATENT TITLE No. 348608

Holders), eyes VR, Inc.

Address: 1601 Willow Road, Menlo Park, California, 94025, USA

Name: CALIBRATION OF VIRTUAL REALITY SYSTEMS

Classification: CIP: G06T19 / 00; G06T15 / 20; G02B27 / 01; G06F1 / 00; G06T7 / 00

CPC: G06T15 / 205; G02B27 / 017; G02B2027 / 014; G02B2027 / 0138;

G02B2027 / 0178; G02B2027 / 0187; G06T7 / 0018; G06T7 / 0046; G06T19 / 006;

G06T2207 / 30204

Inventor (s): DOV KATZ; JONATHAN SHINE; ROBIN MILLER; MAKSYM KATSEV; NEIL KONZEN;

STEVE LAVALLE; MICHAEL ANTONOV

REQUEST

Number: International Presentation Date:

MX / a / 2016/008908 January 6, 2015

PRIORITY* *

Country:

Date:

January 2014 December 5, 2014

Number:

61/923,895 62/088,088

Validity: Twenty years

Expiration Date: January 6, 2035

Issue Date: June 21, 2017 ......

The reference patent is granted based on articles 1or, 2or fraction: V, 6or fraction and 59 of the Industrial Property Law.

In accordance with article 23 of the Industrial Property Law, this patent is valid for twenty years, non-extendable from the date of filing the international application and will be subject to payment of the fee to keep the rights in force.

Whoever signs this title does so based on the provisions of tos-articles 6 ”sections III and 7 bis 2 of the Industrial Property Law (Official Gazette of the Federation (0.0 F.) 06/27/1991, amended on 08/02/1994, 10/25/1996, 12/26/1997, 05/17/1999, 01/26/2004, 06/16/2005, 01/25/2006, 05/06/2009, 01/06/2010, 06/18/2010, 06/28/2010, 01/27/2012 and 04/09/2012). items 1or, 3 “fraction V part a), 4or and 12th sections I and III of the Regulations of the Mexican Institute of Industrial Property (DO.F. 12/14/1999, amended on 07/01/2002, 07/15/2004. 07/28/2004 and 09/07 / 2007); items 1or, 3°, 4or, 5or fraction V part a). 1β sections I and III and 30 of the Organic Statute of the Mexican Institute of Industrial Property (DOF 12/27/1999, amended on 10/10/2002, 07/29/2004, 08/04/2004 and 13W / 2007); 1®, 3 ° and 5 'subsection a) of the Agreement that delegates powers to the Deputy General Directors, Coordinator, Divisional Directors, * Heads of the Regional Offices Divisional Subdirectors, Departmental Coordinators and other subordinates of the Mexican Institute of Industrial Property. - (& O.F. 12/15/1999, amended on 02/04/2000, 07/29/2004, 08/04/2004 and 09/13/2007).

This document is signed with an advanced electronic signature (FIEL), based on articles 7 BIS 2 of the Industrial Property Law; 3 of its Regulations, and 1 section III, 2 section V, 26 BIS and 26 TER of the Agreement establishing the guidelines for the use of the Payment and Electronic Services Portal (PASE) of the Mexican Institute of Industrial Property, in the procedures indicated.

THE DIVISIONAL DIRECTOR OF PATENTS

NAHANNY CANAL REYES

Original string:

NAHANNY MARISOL CANAL REYES | 00001000000403252793 | Administration Service

Tax | 1695 || MX / 2017/51032 | MX / a / 2016/008908 | PCT patent title | 1027 | RGZ | Page (s) 2 | / + gK + xMIJQsqFLrDgmxuqZdKRoO =

Digital stamp:

lnqxnWEq + ONIca33E956oMYdzHlsBColDOUHVf2hkhTFMJ + eBOqOLI9K4hLsNq6l362EcCL + WWuN8v¡4pjZXytMnZ3

MdcZgvvlyHhgdLLYzE5DPMhkfDOQCmRc7bNpE4dwjrUjW OpwGJk4pdUOStl41leiyojDUS + ++ + 8msqAZQWo hwUZHD43q3qEcu1eE uvJShezZQcZZD4NkNYV6IYq4BbHuPI82Oo0tc4GRDsD8UOGJsU9oUx55dL7JrpqWRLNPnNpFXy3sFLOt yZp02R8wEB6TCFQjq1oyPPrdWJ / XKgXu7fK2OeAuj8r6rCfPETYuhDFMtaedx2dKnf rapQ ++ == * Additional information on the back

Arenal No 550. Floor 1, Pueblo Santa Mana Tepepan, Xochimilco. 16020.

Mexico City <55) 53340700 v / wvv gob mx / impí

ungodly. wmmiouixkuno Λ

MuraoftuHA, «WJUSTUM. * 1 ^, .. in *

CALIBRATION OF VIRTUAL REALITY SYSTEMS

BACKGROUND OF THE INVENTION

The present description refers generally to calibration systems, and more specifically refers to the calibration of virtual reality systems.

Motion tracking is an old problem that has been addressed on numerous devices. Examples include global positioning systems, aircraft radar systems, robotic systems, and home entertainment systems. In the latter case, existing devices track the movement of game controllers or people interacting with the game.

Several important characteristics determine the types of sensing and computing hardware that are most appropriate: 1) the size of the rigid body, 2) its volume of space over which the rigid body can move, 3) limits on the maximum rates of speed and acceleration of the body, 4) the predictability of the rigid body.

In the motion tracking system, there is typically a requirement to track the movement of devices that are in contact with the human body while in a relatively small space, typically while sitting indoors, but this is not necessary. . In a more particular way, the intention is to track the movement of

* JWIV ungodly »Mexican 0

M LA NontdAO the head while the user directs a screenNrK ^ Yada'errra head, for the purposes of current reality and increased loyalty. This results in extremely strict requirements on the operation of the system. For example, for virtual reality, position and orientation errors cause a bad experience because the user does not feel truly immersed. Additionally, there may be a mismatch between the signals provided to the brain by the human vestibular system and the signals provided to the brain by the human vision system during screen viewing.

This example use case involves limitations on the volume of space over which movement can occur. This also limits speed and accelerations to those induced by human movement; however, it is also crucial that the movement is generally unpredictable. Furthermore, it is difficult to model the physics that governs motion due to the complexity of the human body's motion and its interaction with other rigid bodies.

Virtual reality (VR) devices include components for determining the position and movement of a viewer worn by a user. These components must be calibrated at different times, initially due to manufacturing tolerances and later due to normal use of the system. Operation of an improperly calibrated virtual reality (VR) device may result in improper position tracking or

IMPI ftemvro MttlCANO OBVAMOHS) AD of the movement of the viewer, which causes a disofiaffo'na between the movement of the user and the media delivered to the user and the average user of the viewer. Furthermore, one or more of the components that determine the position and movement of the viewfinder can lose calibration over time or with use. For example, changes in temperature or vibrations can cause a viewfinder shake imaging camera to lose calibration.

BRIEF DESCRIPTION OF THE INVENTION

A virtual reality (VR) viewer of a virtual reality (VR) system includes a front rigid body and a rear rigid body, which are coupled to each other in a non-rigid manner. For example, the front rigid body is attached to the rear rigid body by an elastic band, so the virtual reality (VR) system continues to detect the movement of an entity wearing the virtual reality (VR) viewer when the rigid body head-on is directed away from an image processing device included in the virtual reality (VR) system. Both the front rigid body and the rear rigid body include locators for tracking the position of the virtual reality (VR) viewer. A locator is an object located at a specific position in the virtual reality (VR) viewer in relation to one or more components, such as another locator, of the virtual reality (VR) viewer and relative to a landmark in the virtual reality (VR) viewer. Because the relationship between the bodies

4 MAID. IKUI.UIO MEXICAN

THAT noFtlTY

The front and rear rigid body is not necessarily fixed, the virtual reality (VR) system may lose track of the position of the front rigid body in relation to the rear rigid body, causing the virtual reality (VR) system to revert to calibrate to re-acquire virtual reality (VR) viewer tracking. In some modalities, the virtual reality (VR) console determines the subsequent predicted positions of the rear rigid body during the times after the particular image time value of the image in the slow calibration data including only the rear rigid body locators using the calibration data and the position vector adjusted by the offset value until re-calibration between the front rigid body and the rigid body can occur rear.

The virtual reality (VR) system re-calibrates itself when tracking of the front rigid body position or rear rigid body position is lost. For example, the virtual reality (VR) system determines when to recalibrate based on a measured difference between the estimated positions of the locators on the rear body and the intermediate estimated positions of a reference point on the front rigid body determined by a inertia measurement unit! (IMU) within the first rigid body based on data from one or more position sensors (e.g. accelerometers, gyros) included in the first rigid body. An intermediate estimated position of the reference point is a position determined by5 • ungodly WTIIUTD MKOCANO LífegSj 7Λ

Pt la monteAt) O «s» ¿2a \ J industrial from the rapid calibration data and may be associated with a time associated with an image, or with a time between the times associated with an image and a subsequent image from slow calibration data.

The components of a virtual reality (VR) system are calibrated to keep track of a virtual reality (VR) viewer associated with the virtual reality (VR) system. The virtual reality (VR) system uses slow calibration data received from an image processing device and fast calibration data received from an inertial measurement unit (IMU) embedded in the virtual reality (VR) viewer for imaging. calibration. In some embodiments, the components of the virtual reality (VR) system can be calibrated by initially applying one or more default parameters to the components. Based on the default parameters, the virtual reality (VR) system tracks the movement of the virtual reality (VR) viewer by identifying the positions associated with one or more locators included in the virtual reality (VR) viewer. A locator is an object located at a specific position in the virtual reality (VR) viewer relative to one or more components, such as another locator, of the virtual reality (VR) viewer, and relative to a landmark in the virtual reality (VR) viewer. In some modalities, the virtual reality (VR) viewer includes two rigid bodies that are coupled in a non-rigid way to each other, with locators included in each one.

INSTITUTO MEXICANO OS THE INDUSTRIAL PROPERTY of rigid bodies for tracking the position and orientation of the user's head. The virtual reality (VR) system adjusts one or more calibration parameters until the differences between an estimated position of one or more locators differs from an observed position of the one or more locators by less than a threshold value.

In some embodiments, the virtual reality (VR) system includes a virtual reality (VR) console that receives slow calibration data that includes a series of images showing a portion of a plurality of locators on the virtual reality (VR) viewer. ) from an image processing device. Each image is separated from a subsequent image in the series by an image time value.

Additionally, the virtual reality (VR) console receives quick calibration data comprising one or more intermediate positions of the reference point in the virtual reality (VR) viewer of the inertial measurement unit (IMU) included in the image viewer. virtual reality (VR). An estimated intermediate position of the reference point is a position determined from the quick calibration data and may be associated with a time associated with an image, or with a time between the times associated with an image and a subsequent image from slow calibration data. The unit of measure inertia! (IMU) determines the estimated intermediate positions of the reference point based on data from one or more position sensors (for

IMPI t <FHTUTD MMICANO DI U rtO ^ ITY example, accelerometers, gyroscopes) included in the virtual viewer (VR). Each estimated position interm ^ Tria '”δ'6 SSTpaTS 0Θ line1 subsequent intermediate estimated position by a position time value that is less than the image time value.

In some modalities, a virtual reality (VR) console included in the virtual reality (VR) system receives the slow calibration data that includes a series of images from the virtual reality (VR) viewer taken at image time values from an image processing device. At least one image from the series of images includes only the locators on the rear rigid body, and is associated with a particular image time value. Additionally, the virtual reality (VR) console receives the rapid calibration data from the inertial measurement unit (IMU) comprising one or more intermediate estimated positions of a reference point of the front rigid body of the virtual reality viewer. (VR), determined from one or more position sensors included in the front rigid body of the virtual reality (VR) viewer. The virtual reality (VR) console determines an observed position of the rear rigid body for the particular picture time value using the slow calibration data, and determines a predicted position of the rear rigid body for the picture time value associated with the image including locators from only the rear rigid body of the virtual reality (VR) viewer using the quick calibration data as well as a position vector describing a

MAID?

fNSTnUTO KtolKANO

THE PROPERTY Cs

INDUSTRIAL '* »¡¡^« ^ 5®' calibrated offset between the front rigid body and the rear rigid body.

The virtual reality (VR) console determines a difference between the observed position of the rear rigid body and the predicted position of the rear rigid body. If the difference is greater than a threshold value, the virtual reality (VR) console adjusts the predicted position of the rear rigid body by a time offset value, so that the difference between the observed position of the rear rigid body and the predicted position of the rear rigid body is less than the threshold value. In some modalities, the virtual reality (VR) console determines the subsequent predicted positions of the rear rigid body for the times after the particular image time value of the image in the slow calibration data that includes only the locators from the rear rigid body using the data of fast calibration and the position vector adjusted by the offset value until re-calibration between the front rigid body and the rigid body can occur rear.

Recalibration uses at least one image from the slow calibration data that has a time after the particular image time that includes only the locators on the rear rigid body, and that includes at least one front threshold number of the localizers observed on the front rigid body, and a rear threshold number of locators observed on the rear rigid body. Console

IMPI mStnUTO MEXICAN> S

DI LA PROFIFDAD Λ · Μ & ί <g INDV ^ TJUAL. **** virtual reality (VR) identifies the model locators corresponding to the observed locators ..... 8 paftih (TeTas ”images of the slow calibration data, using a virtual reality viewer model (VR ). For example, the virtual reality (VR) console extracts the locator information from the images in the slow calibration data, where the locator information describes the positions of the locators observed in the virtual reality viewer ( VR) relative to each other in a given image. In at least one of the images from the slow calibration data, the virtual reality (VR) console identifies the model locators that correspond to the locators observed on both the front rigid body and the rear rigid body. The virtual reality (VR) console compares the locator information with a viewer model to identify the model locators that correspond to the observed locators.

Based on the information from the locators, the virtual reality (VR) console generates the estimated positions for the observed locators using the viewer model. For example, the virtual reality (VR) console uses the viewer model and the information that identifies the positions of the observed locators to determine a projection matrix for the translation of the ideal positions (described by the viewer model) to the positions on the image plane (described by the images of the observed locators) of the scanning device

ungodly

ΙΜΤΠντΟΜΚΚΛΝΟ

OHAFKOFItCAn industrial image processing.

The virtual reality (VR) console uses the projection matrix to estimate the positions of the observed locators. The virtual reality (VR) console uses the projection matrix to estimate the positions of the observed locations, and adjusts one or more calibration parameters to adjust one or more of the estimated positions of the observed locators on the front rigid body to a relative distance between the adjusted estimated positions of the observed localizers on the front rigid body and their corresponding positions determined by the observed locations of the viewer model is then a threshold value. In a similar manner, the virtual reality (VR) console determines the estimated positions of the observed locators on the rear rigid body, and adjusts the estimated positions of the observed locations on the rear rigid body as described above. Based on the adjusted estimated positions of the observed locators on the first rigid body, the virtual reality (VR) console determines the calibrated positions of the first rigid body reference point for one or more images from the slow calibration data.

The virtual reality (VR) console determines a position of the rear rigid body relative to the reference point of the first rigid body. For example, the virtual reality (VR) console identifies a rear landmark on the rear rigid body using the adjusted estimated positions of the Mexican iwrnvro locators fb-íí ·· ce la monedad

INDUSTRIAL ^ * ¿5 “/ · observed on rear rigid body. The virtual reality (VR) console then identifies a position of the rear landmark in relation to the landmark on the front rigid body. In alternate modes, the virtual reality (VR) console identifies the position of each observed locator on the rear rigid body relative to the reference point on the front rigid body. The virtual reality (VR) console further adjusts one or more calibration parameters, such that the intermediate estimated positions of the reference point on the front rigid body and / or the rear reference point are within a threshold value of the predicted positions of the reference point on the front rigid body and / or the point rear reference point determined from the calibrated position of the reference point on the front rigid body and / or the calibrated position of the rear reference point (for example, by curve fitting) from the slow calibration data.

The modalities of a virtual reality (VR) system provide high precision tracking by using only a single camera, as opposed to many, and high precision is achieved through a high plurality of light-emitting diodes (LEDs ) well separated on the surface of the body. The modulation approach allows each light emitting diode (LED) to be identified in a unique way, resulting in a system that is more robust to occlusion and interference from nearby light sources. Additionally, the modulation approach, in conjunction with digital hardware, Mexican twrmrro Di LA FXOnFDAlJ INDUSTRIAL reduces power consumption because the light-emitting diodes (LEDs) are powered only when the camera shutter is open.

The virtual reality (VR) system solves the problem of tracking the screen mounted on the head or other objects, such as game controllers, with a level of operation that is suitable for virtual and augmented reality, while at the same time dramatically reduces the economic and logical requirements compared to existing technologies.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a block diagram of a system environment where a virtual reality console operates, according to one embodiment.

Figure 2A is a schematic of a virtual reality viewer, according to one embodiment.

Figure 2B is a schematic of a virtual reality viewer that includes a front rigid body and a rear rigid body, according to one embodiment.

Figure 3 is a block diagram of a tracking module of a virtual reality console, according to one embodiment.

Figure 4 is a flow diagram of a process for calibration of a virtual reality system, according to one embodiment.

-> · μ «ιι iw, ftcrs :::»

Ungodly mSTITVTO 1000 Lí ICAN

OF THE INDUSTRIAL rROMEDAD fi.

Figure 5 is a flow diagram of a process for re-calibration between two rigid bodies in a virtual reality viewer included in a virtual reality system, according to one modality.

Figure 6 is a flow diagram of a process for maintaining a positional relationship between two rigid bodies in a virtual reality viewer included in a virtual reality system, according to one embodiment.

Figure 7 is an example graph illustrating a series of calibrated positions of a virtual reality viewer, according to one embodiment.

Figure 8 is an example high-level diagram of a position tracking system.

Figure 9 illustrates an example of the observed bright spots and predicted projections.

Figure 10 illustrates an example of posture optimization.

The figures illustrate the embodiments of the present description for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits obtained, of the disclosure described herein. document.

14 IMPI fMTTTVTO MUICAnq

DtUMOHEDAD rNtxjsTKiAi

DETAILED DESCRIPTION OF THE INVENTION

System architecture

Figure 1 is a block diagram of one embodiment of a virtual reality (VR) system environment 100 in which a virtual reality (VR) console 110 operates. The system environment 100 shown in Figure 1 comprises a viewer virtual reality (VR) device 105, an image processing device 135, and a virtual reality (VR) input interface 140, which are each coupled to virtual reality (VR) console 110.

While Figure 1 shows an example of the system 100 that includes a virtual reality (VR) viewer 105, an image processing device 135, and a virtual reality (VR) input interface 140, in other embodiments, You can include any number of these components in system 100. For example, there may be multiple virtual reality (VR) viewers 105 each with an associated virtual reality (VR) input interface 140 and monitored by one or more image processing devices 135, where each virtual reality viewer ( VR) 105, the virtual reality (VR) input interface 140, and the image processing devices 135 communicate with the virtual reality (VR) console 110. In alternative configurations, different and / or additional components can be included in the environment of the system 100.

Virtual reality (VR) viewer 105 is a head-mounted display that presents media to a user. Examples of15 WICKED

IWTTTVTO MEXICANO J. '

DBLA MIOHEDAD OarJ5 & iv 'NDUSTXEAL The media presented by the virtual reality (VR) viewer includes one or more images, video, audio, or some combination thereof. In some embodiments, the audio is presented by means of an external device (for example, speakers and / or headphones) that receives audio information from the virtual reality (VR) viewer 105, the virtual reality (VR) console 110, or both, and the audio data is presented based on the audio information. Exemplary embodiments of virtual reality (VR) viewer 105 are further described in conjunction with Figures 2A and 2B.

In various embodiments, the virtual reality (VR) viewer 105 can comprise one or more rigid bodies, which can be rigidly or non-rigidly coupled to each other. A rigid coupling between the rigid bodies causes the coupled rigid bodies to act as a single rigid entity. In contrast, a non-rigid coupling between the rigid bodies allows the rigid bodies to move relative to each other. One embodiment of the virtual reality (VR) viewer 105 that includes two rigid bodies that are non-rigidly coupled to each other is further described below in connection with FIG. 2B.

The virtual reality (VR) viewer 105 includes an electronic display 115, one or more locators 120, one or more position sensors 125, and an inertial measurement unit (IMU) 130. The electronic display 115 displays the images to the user of the device. based on data received from virtual reality (VR) console 110. In various embodiments, electronic display 115 can

«IMPI ® * n * TUTO MEXICANO WU MOMEDAD INDUSTRIAL comprise a single electronic screen or -multiple electronic screens (for example, one screen for each eye of a user). Examples of electronic display 115 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode (AMOLED) display, some other display, or some combination of them. Additionally, the electronic display 115 may be associated with one or more optical components that correct one or more types of optical errors (eg, field curvature, astigmatism, barrel distortion, cushion distortion, chromatic aberration, etc.). In some embodiments, the media provided to the electronic display 115 for display to the user is pre-distorted to aid in correcting one or more types of optical errors. Additionally, the optical components can increase a field of view of the displayed media through amplification or through another suitable method. For example, the field of view of the displayed media is such that the displayed media is displayed using almost all (eg, 110 degrees diagonally), and in some cases all, of the user's fields of view.

Locators 120 are objects located at specific positions on virtual reality (VR) viewer 105 relative to each other, and relative to a specific reference point on virtual reality (VR) viewer 105. A locator 120 may be a light-emitting diode (LED), a corner reflector cube, a

IMPI • WNJIU MUICANO O »U INDUSTRIAL MOFITY of light that contrasts with a

virtual reality (VR) 105, or the modalities where the reflective marker, a type of source environment where the viewer of some combination of them operates. When locators 120 are active (i.e., a light-emitting diode (LED) or other type of light-emitting device), locators 120 can emit light in the visible band (approximately 380 nanometers to 750 nanometers), in the infrared (IR) band (approximately 750 nanometers to 1 millimeter), in the ultraviolet band (10 nanometers to 380 nanometers), in some other part of the electromagnetic spectrum, or in some combination thereof.

In some embodiments, the locators are below an outer surface of the virtual reality (VR) viewer 105, which is transparent to the wavelengths of light emitted or reflected by the locators 120 or is thin enough not to substantially attenuate the wavelengths of light emitted or reflected by locators 120. Additionally, in some embodiments, the outer surface or other portions of the virtual reality (VR) viewer 105 are opaque in the visible range. Accordingly, locators 120 can emit light in the infrared (IR) band below an external surface that is transparent in the infrared (IR) band, but opaque in the visible band.

The Commercial Measurement Unit (IMU) 130 is an electronic device that generates rapid calibration data based on the

NC I ungodly iwrnvro msxica

MunepirDxn Α ^ ζ ^ Γζ, »

INDUSTUlAL measurement signals received from one or more of the 125 position sensors. A 125 geneis position sensor1'one OR TTOS' signals1measurement in response to movement of the virtual reality (VR) viewer 105. Examples of position sensors 125 include: one or more accelerometers, one or more gyros, one or more magnetometers, or any other suitable type of sensor, or some combination thereof. The position sensors 125 can be located external to the inertial measurement unit (IMU) 130, internal to the inertial measurement unit (IMU) 130, or some combination thereof.

Based on the one or more measurement signals from the one or more position sensors 125, the inertial measurement unit (IMU) 130 generates quick calibration data indicating an estimated position of the virtual reality (VR) viewer 105 relative to a virtual reality (VR) viewer home position 105. For example, 125 position sensors include multiple accelerometers to measure translational motion (forward / backward, up / down, left / right), and multiple gyros to measure rotational motion (eg, pitch, yaw, wobble). . In some embodiments, the inertial measurement unit (IMU) 130 rapidly samples the measurement signals and calculates the estimated position of the virtual reality (VR) viewer 105 from the sampled data. For example, Inertial Measurement Unit (IMU) 130 integrates the measurement signals received from accelerometers over time to estimate a velocity vector, and integrates the

ΙΜΡΙ »Μ Mexican ownrvro

Say the rBoritOAu Ow «rs.-i — ζκ;> IXMJSTXlAl velocity vector over time to determine an estimated position of a reference point (by BJSTTTpnj 'la Position” ”” estimated intermediate) in the virtual reality viewer ( VR) 105. Alternatively, the inertial measurement unit (IMU) 130 provides the sampled measurement signals to the virtual reality (VR) console 110, which determines the quick calibration data. The reference point is a point that can be used to describe the position of the virtual reality (VR) viewer 105. Although the reference point can generally be defined as a point in space, however, in practice, the point of Reference is defined as a point within the virtual reality (VR) viewer 105 (eg, the center of the inertial measurement unit (IMU) 130).

Inertial Measurement Unit (IMU) 130 receives one or more calibration parameters from virtual reality (VR) console 110. As further discussed below, the one or more calibration parameters are used to keep track of the viewfinder of virtual reality (VR) 105. Based on a received calibration parameter (for example, Inertial Measurement Unit (IMU) parameters), Inertial Measurement Unit (IMU) 130 can adjust its operation (for example, sample rate change, etc. ). In some embodiments, as further described below, certain calibration parameters cause the inertial measurement unit (IMU) 130 to compensate for an estimated position of the virtual reality (VR) viewer 105 to correct for positional errors that may occur when only certain parts of the virtual reality (VR) viewer 105 are visible to the image processing device 135. In some embodiments, certain calibration parameters cause the Inertial Measurement Unit (IMU) 130 to update an initial reference point position such that it corresponds to the next calibrated reference point position. Updating the home position of the reference point as the next calibrated reference point position helps reduce the accumulated error associated with the determined estimated position. The accumulated error, also known as drift error, causes the estimated position of the reference point to drift away from the actual position of the reference point over time.

Image processing device 135 generates slow calibration data in accordance with calibration parameters received from virtual reality (VR) console 110. Slow calibration data includes one or more images showing the observed positions of locators 120 that are detectable by the image processing device 135. Image processing device 135 may include one or more cameras, one or more video cameras, any other device capable of capturing images, including one or more of locators 120, or some combination thereof. In addition, the image processing device 135 may include one or more filters (for example, used to increase the ratio of the

signal to noise). Image processing device 135 is configured to detect light emitted or reflected from locators 120 in a field of view of image processing device 135. In embodiments where locators 120 include passive elements (for example, a retroreflector), image processing device 135 may include a light source that illuminates some or all of locators 120, which retro-reflect light to the rear. light source in image processing device 135. Slow calibration data is communicated from image processing device 135 to virtual reality (VR) console 110. Image processing device 135 receives one or more calibration parameters from virtual reality (VR) console 110, and can adjust one or more image processing parameters (eg, focal length, focus, frame rate, scale sensitivity, sensor temperature, shutter speed, aperture, etc.) based on the calibration parameters.

Virtual reality (VR) input interface 140 is a device that allows a user to send action requests to virtual reality (VR) console 110. An action request is a request to perform a particular action. For example, an action request can be to start or end an application or to perform a particular action within the application. The virtual reality (VR) input interface 140 can include one or> »—w

IMPI rwfrnvro MEXICANO IX U PROPERTY plus input devices. Examples of input devices include: a keyboard, a mouse, a contiUIU'UUI 06 JtTé ^ oST'OR any other device suitable for receiving action requests and communicating action requests received to the console. virtual reality (VR) 110. An action request received by the virtual reality (VR) input interface 140 communicates with the virtual reality (VR) console 110, which performs an action corresponding to the action request. In some embodiments, the virtual reality (VR) input interface 140 can provide haptic feedback to the user in accordance with instructions received from the virtual reality (VR) console 110. For example, haptic feedback is provided when an action request is received, or the virtual reality (VR) console 110 communicates instructions to the virtual reality (VR) input interface 140 causing the virtual reality (VR) input interface ) 140 generates haptic feedback when the virtual reality (VR) console 110 performs an action.

Virtual reality (VR) console 110 provides means to virtual reality (VR) viewer 105 for presentation to the user in accordance with information received from one or more of: image processing device 135, virtual reality viewer (VR) 105, and the virtual reality (VR) input interface 140. In the example shown in Figure 1, the virtual reality (VR) console 110 includes a media store 145, a tracking module 150, and a virtual reality (VR) engine 155. Some

IMPT ^

Μηπντο MEXICAN

DELAHOPIFDAD CVw INDUSTRIAL modalities of the virtual reality (VR) console 110 have different modules from those described in relation to figure 1. In a similar way, the functions described further below can be distributed among the components of the virtual reality console (VR) 110 in a different way than what is described here.

The application store 145 stores one or more applications to be executed by the virtual reality (VR) console 110. An application is a group of instructions, which when executed by a processor, generates the means for presentation to the user. Media generated by an application may be presented in response to input received from the user by movement of the virtual reality (VR) viewer 105 or the virtual reality (VR) interface device 140. Examples of the applications include ; game applications, conference applications, video player application, or other suitable applications.

Tracking module 150 calibrates the environment of system 100 using one or more calibration parameters. As further described in connection with Figures 3-5, the tracking module 150 may adjust one or more calibration parameters to reduce error in determining the position of the virtual reality (VR) viewer 105. For example, the tracking module 150 adjusts the focus of the image processing device 135 to obtain a more precise position for the locators. 24 ΙΜΡΙΟ ^ uownrro mukanu ζ1

DC THE MOMUMD C'.2- ^ 4.¿ ¡J *

INBMSTBIAL observed in the virtual reality viewer (VR) 105. Furthermore, the calibration performed by the tracking module 150 also counts for the information received from the inertial measurement unit (IMU) 130. Additionally, as discussed in greater detail below in relation to Figures 4 and 5, if that tracking of virtual reality (VR) viewer 105 is lost (for example, image processing device 135 loses line of sight of at least a threshold number of locators 120), the tracking module 140 re-calibrates some or all of the environments of the system 100. As used herein, "loss of tracking" can generally refer to a loss of calibration of the image processing device 135 or the inertial measurement unit (IMU) 130, a loss of relative positions of one or more bodies rigid in the virtual reality (VR) viewer 105, a loss of position of the virtual reality (VR) viewer 105 relative to the image processing device 135, or some combination thereof.

The re-calibration of the environment of the system 100 is generally transparent to the user. In some embodiments, the tracking module 150 may ask the user to move the virtual reality (VR) viewer 105 to an orientation in which one or more sides of the virtual reality (VR) viewer 105 are visible to the processing device. Images 135. For example, the tracking module 150 prompts the user to look up, look down, look left, look right, or look in another specified direction. 25 IMPIf ^ nwrnvro MEXICAN 'i

MY MONEY <J

XDUST21AL * í such that one or more sides of the virtual reality (VR) viewer ——..................

105 are visible to the image processing device 135. Once a threshold number of locators 120 on the virtual reality (VR) viewer 105 are projected by the image processing device 135, the tracking module 150 re-establishes the calibration. In some embodiments, the tracking module 150 may continuously calibrate the environment of the system 100 or it may calibrate the environment of the system 100 at periodic intervals to maintain accurate tracking of the virtual reality (VR) viewer 105.

Tracking module 150 can calibrate an environment of system 100 that includes a virtual reality (VR) viewer 105 comprising one or more rigid bodies (eg, see Figures 2A and 2B). Additionally, as further described below in conjunction with the figures. 3 and 5, the calibration may count for a virtual reality (VR) viewer 105, including two rigid bodies that are coupled in a non-rigid way (eg, coupled to each other by means of an elastic band). The two rigid bodies can be a front rigid body that includes the inertial measurement unit (IMU) 130 that is placed in front of the user's eyes, and a rear rigid body that is placed at the back of the user's head. This configuration of the front rigid body and rear rigid body allows a user to rotate 360 degrees relative to the image processing device 135. However, because the relationship between the front rigid body and the rear rigid body is not necessarily fixed, the environment of the system 100

MAID

INSTITUTO MEXCANO ~ f '. · - ·' · *, <5 ** DI LA FROHEDAD (Ά '. Y / INDUSTRIAL may lose the calibration of the position of the front rigid body in relation to the rear rigid body. More so, as discussed in detail below with respect to FIG. 6, in some embodiments, if tracking between multiple rigid bodies is lost in virtual reality (VR) viewer 105, tracking module 150 may offset the position of a rigid body. until re-calibration can occur. In these cases, in some embodiments, the tracking module 150 may determine an offset value to the intermediate estimated position of the virtual reality (VR) viewer 105 and provide it to the inertial measurement unit (IMU) 130 as a parameter of calibration.

Alternatively, the tracking module 150 may adjust a position vector that describes the relative position of the front rigid body to the rear rigid body by the offset value. In some embodiments, the tracking module 150 determines when to recalibrate based on a measured difference between the movement indicated by the locators 120 on the rear rigid body, and the movement predicted from the quick calibration data received from the tracking unit. Inertial Measurement (IMU) 130. The tracking module 150 recalibrates using the slow calibration data that includes one or more images including the front rigid body locators 120 and the rear rigid body locators.

Additionally, the tracking module 150 tracks the movements of the virtual reality (VR) viewer 105 using the calibration data.

MAID

MEXICAN INSTITUTE

DE LA PRÜHEDAP from the image processing device * ^^. ConTo ^ is further described below as a set of how the tracking module 150 determines the positions of a reference point of the virtual reality (VR) viewer 105 using the localizers observed from the slow calibration data and a model of the viewfinder. virtual reality (VR) 105. The tracking module 150 also determines the positions of a reference point of the virtual reality (VR) viewer 105 using the position information from the quick calibration data. Additionally, in some embodiments, the tracking module 150 may use portions of the fast calibration data, the slow calibration data, or some combination thereof, to predict a future location of the display 105. Tracking module 150 provides the estimated or predicted future position of virtual reality (VR) viewer 105 to virtual reality (VR) engine 155.

The virtual reality (VR) engine 155 runs the applications within the system environment and receives position information, acceleration information, speed information, predicted future positions, or some combination thereof, from the virtual reality (VR) viewer 105 from tracking module 150. Based on the information received, the virtual reality (VR) engine 155 determines the means to be provided to the virtual reality (VR) viewer 105 for presentation to the user. For example, if the information received indicates that the user has looked to the left, the virtual reality (VR) engine 155 generates the media

Ungodly rwrrruTO m exican

D »THE« VmlQVí.O í * INDUSTRIAL OPTION for virtual reality (VR) viewers 105 that reflect the user's movement in a virtual environment. Additionally, the virtual reality (VR) engine 155 performs an action within an application that runs on the virtual reality (VR) console 110 in response to an action request received from the virtual reality (VR) input interface 140. and provides feedback to the user that the action was performed. The feedback provided can be visual or audible feedback through the virtual reality (VR) viewer 105 or haptic feedback through the virtual reality (VR) input interface 140.

FIG. 2A is a line diagram of one embodiment of a virtual reality viewer . The virtual reality (VR) viewer 200 is a modality of the virtual reality (VR) viewer 105 and includes a front rigid body 205 and a band 210. The front rigid body 205 includes the electronic screen 115 (not shown), the unit Inner Measurement Sensor (IMU) 130, the one or more position sensors 125, and the locators 120. In the embodiment shown in Figure 2A, the position sensors 125 are located within the inertial measurement unit (IMU) 130, and neither the position sensors 125 nor the inertial measurement unit (IMU) 130 are visible to the user. .

Locators 120 are located at fixed positions on front rigid body 205 relative to each other, and relative to a reference point 215. In the example of Figure 2A, reference point 215 is in the center of the inertia measuring unit! (IMU) 130. Each of the locators 120

Mexican ungodly rwrmvro <.

DE U CURRENCY indcttmal ^ mQííLJS3* emits light that can be detected by image processing device 135. Locators 120, or portions of locators 120, are located on a front side 220A, on a top side 220B, on a bottom side 220C, on a side right 220D, and on a left side 220E of the front rigid body 205 in the example of FIG. 2A.

Figure 2B is a line diagram of one embodiment of a virtual reality (VR) viewer 225 that includes a front rigid body 205 and a rear rigid body 230. The virtual reality (VR) viewer 225 shown in Figure 2B , is an embodiment of the virtual reality (VR) viewer 105 wherein the front rigid body 205 and the rear rigid body 230 are coupled to each other through the band 210. The band 210 is not rigid (for example, it is elastic), such that the front rigid body 205 does not rigidly couple to the rear rigid body 210. Consequently, the rear rigid body 230 can be moved relative to the rigid body. front 205 and specifically can be moved relative to reference point 215. As further discussed below in conjunction with Figures 3 and 5, the rear rigid body 230 allows the virtual reality (VR) console 110 to keep track of the virtual reality (VR) viewer 105, even if the front rigid body 205 is not visible to image processing device 135. Locators 120 on rear rigid body 230 are in fixed positions relative to each other, and at

IMPI Mexican tNJTrrvro OtUíROPIEtMO noumiAi.

relation to reference point 215 on the front rigid body

205. In the example of Figure 2B, one or more locators 120, or portions of locators 120, on the rear rigid body 230 are located on a front side 235A, on an upper side 235B, on a lower side 235C, on a lower side. right 235D, and on a left side 235E of the rear rigid body 230.

Figure 3 is a block diagram of one mode of the tracking module 150 included in the virtual reality (VR) console 110. Some modes of the tracking module 150 have different modules than those described herein. In a similar way, the functionality described in relation to Figure 3 can be distributed among the components in a different way than that described herein. In the example of Figure 3, the tracking module 150 includes a tracking database 310, an initialization module 320, an estimation module 330, a parameter setting module 340, and a control module 350.

Tracking database 310 stores information used by tracking module 150 to track one or more virtual reality (VR) headsets 105. For example, tracking database 310 stores one or more models of viewers, one or more values of the calibration parameters, or any other information suitable for tracking a virtual reality (VR) headset 105. As reference previously with respect to FIG. 1, a viewer model describes the ideal positions of each of the locators 120 with respect to the

IMPIOS i Wni uto Mexicano

DC THE HLOPIETY

INDUSTRIAL 2 others and with reference point 215. Each locator 120 is associated with a corresponding model locator in the viewer model; therefore, a model locator corresponding to a locator 120 describes an ideal position of the locator 120 according to the viewer model. Additionally, the viewer pattern may include information describing changes in pattern positions of locators 120 or reference point 215 as a function of different calibration parameters. In some embodiments, the viewfinder pattern may describe pattern positions of the locators 120 on a rear rigid body 230 with respect to each other, the pattern positions of a rear reference point describing a position of the rear rigid body 230, the default positions of the rear reference point relative to a reference point 215 on the front rigid body 205, the default positions of the pattern locations of locators 120 on rear rigid body 230 relative to reference point 215, or some combination thereof.

Calibration parameters are parameters that can be adjusted to affect the calibration of the 105 Viewer. Example calibration parameters include image processing parameters, Inertial Measurement Unit (IMU) parameters, or some combination of the above. themselves. Image processing parameters and Inertial Measurement Unit (IMU) parameters can be included in the imaging parameters.

IMPI® nUTTTVTO MEXICANO .....

nreusTMAL of frame rate processing, shutter calibration. Examples of the image parameters include: focal length, focus, ISO sensitivity scale, camera orientation speed, source activation (in the modes where the image processing device 135 uses a source to illuminate the reflective locators 120), the phase shift of a sensor of projection relative to the center of a lens of image processing device 135, lens distortion parameters, sensor temperature, or any other parameter used by the image processing device 135 to produce the slow calibration data. Inertial Measurement Unit (IMU) parameters are the parameters that control the collection of quick calibration data. Examples of Inertial Measurement Unit (IMU) parameters include: a sampling rate of one or more of the measurement signals from the position sensors 125, an output rate of the fast calibration data, other suitable parameters used by the inertial measurement unit (IMU) 130 to generate the measurement data quick calibration, commands to feed power to the inertial measurement unit (IMU) 130 a on or off, commands to update the home position to the current reference point position, offset information (for example, offset to position information), or any other suitable information.

Initialization module 320 initializes the system environment

100 using the information from the trace database 310, such as the calibration parameters retrieved from the trace database 310. In modes where the environment of the system 100 was not previously calibrated, the default calibration parameters are retrieved from trace database 310. If the environment of system 100 was previously calibrated, the adjusted calibration parameters can be retrieved from trace database 310. Initialization module 320 provides the retrieved calibration parameters to Inertial Measurement Unit (IMU) 130 and / or image processing device 130.

Estimation module 330 receives slow calibration data and / or fast calibration data from virtual reality viewer (VR) 105 and / or from inertial measurement unit (IMU) 130. Slow calibration data is received from the image processing device 135 at a slow data rate (eg, 20 Hz). In contrast, fast calibration data is received from the Inertial Measurement Unit (IMU) 130 at a data rate (eg, 200 Hz or more) that is significantly faster than the data rate at which it is used. receive slow calibration data. Accordingly, the fast calibration data can be used to determine the position information of the virtual reality (VR) viewer 105 among the images of the virtual reality (VR) viewer 105 included in the slow calibration data.

By using a viewer model from tracking database 310 and slow calibration data from image processing device 135, estimation module 330 identifies model locators corresponding to one or more locators in the viewer. virtual reality (VR) devices 135 identified from the images captured by the image processing device 135. The estimation module 330 extracts the locator information from the images in the slow calibration data, the locator information that describes the positions of the observed locators 120 with respect to each other in a given image. For a given image, the locator information describes the relative positions between locators 120 observed in the image. For example, if an image shows the observed locators A, B, and C, the locator information includes the data that describes the relative distances between A and B, A and C, and B and C. As described above, the model Viewer includes one or more model positions for the locators in the virtual reality (VR) viewer 105. The estimation model 330 compares the relative positions of the observed locators 120 with the relative positions of the model locators in order to determine the correspondences between the observed locators 120 in the virtual reality (VR) viewer 105 and the model locators. from the viewfinder model. In modalities where calibration is occurring for a 225 virtual reality (VR) viewer, including μ ΙΜ ΙΜI nwrnvrt! MEXICAN

OF THE ΡΕΟΗξΓΆΓ · -ATTTjtXjfiD

INDUSTRIAL multiple rigid bodies, the ΐηηοϋταΗη ^ ο da model corresponding to the localizers observed in both the front rigid body 205 and the rear rigid body 230 are identified from at least one of the images of the slow calibration data.

In addition, based on the viewer model and information describing the model locators and the observed locators 120, the estimation module 330 generates the estimated positions for the observed locators 120. The estimation module 330 determines a projection matrix based on the viewer model and information describing the model locators and the observed locators 120. The projection matrix is a mathematical construction that translates the ideal positions of the locators 120, described by the viewfinder model, to the positions in an image plane, described by the images of the observed locators 120, of the image processing device. images 135. Accordingly, the estimation module 330 estimates the positions of the observed locators 120 using the projection matrix and the positions of the model locators described in the viewer model. One or more calibration parameters may be applied to the projection matrix, such that adjustments to one or more of the calibration parameters modify the estimated positions of the observed locators 120.

The estimation module 330 also extracts the information from Mexican fNsTmrro 'l. . ..... ... . . . «4J * 9ÍS» P intermediate position, speed information TOt3 ^ echw¿ = na> intermediate acceleration information, or nlgmr combination dm In? __________ themselves, from the quick calibration data. As fast calibration data is received more frequently than slow calibration data, the information extracted from the fast calibration data allows estimation module 330 to determine position information, velocity information, or the acceleration information during the time periods between images from the slow calibration data. The information of an intermediate estimated position (eg, an intermediate estimated position) describes a position of the reference point 215 at a time associated with an image, or at a time between times associated with an image and a subsequent image from slow calibration data. Intermediate velocity information describes a velocity vector associated with reference point 215 in a time between a time associated with one image and a time associated with a subsequent image from the slow calibration data. The intermediate acceleration information describes an acceleration vector associated with the set point 215 in a time between a time associated with one image and a time associated with a subsequent image from the slow calibration data. In some embodiments, the estimation module 330 is configured to obtain the intermediate estimated position information using the intermediate acceleration information or from the

IMPI 0¾) rtisTfrvf · Mexican x. ,. , DI LA PHOPfFDAO C * ea¿ © ¿íZ · 'information of the intermediate speed. The module of ealPffra ^ ordnSSO ^J provides the intermediate position to the mookrio 1 do adjust of--, parameters 340.

The parameter adjustment module 340 adjusts one or more calibration parameters to adjust the estimated positions until the relative distances between the adjusted estimated positions of the observed locators 120 and the positions of their corresponding model locators are less than a threshold value. If a relative distance between an estimated position of an observed locator 120 and a position of its corresponding model locator is equal to or greater than a threshold value (for example, 1 millimeter), the parameter adjustment module 340 adjusts one or more parameters calibration parameters (for example, image processing parameters) until the relative distance is less than the threshold value. For example, the parameter adjustment module 340 modifies a calibration parameter, while other calibration parameters are held fixed in order to determine a value for the calibration parameter being modified, resulting in less than a threshold distance. between the estimated position of an observed locator 120 and a position of its corresponding model locator. The parameter adjustment module 340 can then set the calibration parameter to the determined value and repeat the process of modifying the values for the individual calibration parameters while maintaining other calibration parameters at values.

IMPI irthtuto mujoano D £ La riOfUUAD INDVSTMA4.

constant until the relative distances between the adjusted estimated positions of at least a threshold number of observed locators 120 and the positions of their corresponding model locators are less than the threshold value. Using the adjusted estimated positions of the observed locators 120, the parameter adjustment module 340 generates the calibrated positions of the reference point 215 for one or more frames of the slow calibration data.

In the modalities where the virtual reality (VR) viewer 105 includes two rigid bodies (for example, the virtual reality (VR) viewer 225), the parameter adjustment module 340 determines a position of the rear rigid body 230 in relation with the reference point 215 on the front rigid body 205. In some embodiments, the parameter setting module 340 identifies a rear reference point on the rear rigid body 230 using the locators 120 observed on the rear rigid body 230 and their corresponding model locators. The parameter setting module 340 then identifies a position of the rear reference point relative to the reference point 215 on the front rigid body 205. Alternatively, virtual reality (VR) console 110 identifies the position of each observed locator 120 on rear rigid body 230 relative to reference point 215 on front rigid body 205. In some embodiments, the parameter setting 340 generates the calibrated positions of reference point 215 in

ungodly

IMIIRflU MEXICAN 1 ·

Or "IA noNEDA"?

response to the determination that a number '^ ffHUal ^^ - locators have been projected (observed locators) on one and ------ more sides of each rigid body 205, 230, or that a threshold number of locators have been projected projected (observed locators) on all sides of each rigid body 205, 230. For example, the threshold number of projected locators on one side of a rigid body 205, 230 is greater than or equal to zero. If the threshold number of locators is not projected, the parameter setting module 340 may ask the user through the virtual reality (VR) viewer 105 or through another suitable component, to orient the virtual reality (VR) viewer 105 in a specific direction relative to image processing device 135, or continue to move virtual reality (VR) viewer 105 until the threshold number of locators is projected.

The parameter adjustment module 340 also determines a prediction function that predicts the positions of set point 215 and adjusts one or more calibration parameters until the intermediate estimated positions of set point 215 from the quick calibration data are within a threshold value of the predicted positions of reference point 215. For example, the prediction function is generated by fitting a curve to the series of calibrated positions. The parameter adjustment module 340 then adjusts one or more calibration parameters until a distance between the intermediate estimated positions of the reference point 215 and the predicted positions

IMPIAS rHSHTVHj MMICANO of reference point 215 is less than an example, the parameter setting module ρ ·· ^ Ηα increase |to sampling rate of the commercial measurement unit (IMU) 140 until the distance between the intermediate estimated positions of the reference point 215 and the predicted positions of the reference point 215 is 1 millimeter or less. In other embodiments, the parameter adjustment module 340 adjusts one or more calibration parameters, such that the distances between each intermediate estimated position and a calibrated position (eg, CPi) of the reference point 215 associated with the image, are less than a distance value between the calibrated position (for example, CPi) of the reference point 215 associated with the image and the calibrated position of the reference point 215 associated with the subsequent image (eg CP2).

In some embodiments, the parameter setting module 340 updates the initial position of the inertial measurement unit (IMU) 130 to be the next calibrated position of the set point 215. As discussed above in relation to Figure 1 and below in relation to Figure 6, the Inertial Measurement Unit (IMU) 130 collects the quick calibration data in relation to reference point 215 positions previously determined by the unit. Inertial Measurement (IMU) 130. Consequently, the drift error increases the longer the inertial measurement unit (IMU) 130 collects the data without updating the home position to a calibrated position. The

41 IMPI οστττντο mrxicano d »la rmun

ΓΝΤΧΛΤΙΙΙΑΙ.

Parameter adjustment module 340 compares the intermediate estimated positions with an update threshold value. If one or more of the intermediate estimated positions exceed the update threshold value, the parameter setting module 340 communicates an instruction to the inertial measurement unit (IMU) 130 to update the home position as the position associated with the next calibrated position. . Alternatively, after determining a calibrated position, the parameter adjustment module 340 instructs the inertial measurement unit (IMU) 130 to update the home position to the determined calibrated position. The parameter setting module 340 stores the adjusted calibration parameter values in the tracking database 310 and may also provide the adjusted calibration parameters to other components in the virtual reality (VR) console 110.

The supervision module 350 monitors the environment of the system 100 to determine loss of calibration. In various embodiments, the monitoring module 350 monitors the relative distances between the adjusted estimated positions of the observed locators 120 and the positions of their corresponding model locators. If a relative distance between an adjusted estimated position of an observed locator and a position of its corresponding model locator is less than a threshold value (for example, 1 millimeter), the monitoring module

350 provides the calibrated position of reference point 215 determined from the positions of locators 120

MAID

ΜΠΤΤυΤΟ MEXICANO ¡M LA rFOPIkDAl INDUSTRIAL observed to the virtual reality (VR) engine 155. Conversely, if the relative distance between an observed locator and its corresponding model locator is greater than the threshold value (for example, 1 millimeter), the monitoring module 350 determines that the calibration was lost and prompts the parameter setting module 340 to recalibrate the environment of the system 100.

The relative distances determined by the parameter setting module 340 between the intermediate estimated positions and their corresponding predicted positions are monitored. If a distance between a predicted position and its corresponding intermediate estimated position is less than a threshold value (eg, 1 millimeter), the monitoring module 350 provides the intermediate estimated position to the virtual reality (VR) engine 155. In some embodiments, the monitoring module 350 may also provide the intermediate speed information or the intermediate acceleration information extracted from the quick calibration data to the virtual reality (VR) engine 155. Conversely, if the distance between the predicted position and its corresponding intermediate estimated position is greater than the threshold value, the monitoring module 350 determines that the calibration was lost and causes the environment of the system 100 to re-establish the calibration.

In some cases, the locators 120 on the rear rigid body 230 are visible only to the image processing device 135. When only the 43 Mexican IMPI twmvro

M LA PSOMEDaíi CVaJJiíLj. '. ., · RuDomiAi locators 120 on rear rigid body 230 for image processing device 135, in some embodiments, if a difference between the estimated position of rear rigid body 230 (for example, generated from locators 120 observed on the rear rigid body 230) and a predicted position of the rear rigid body 230 (for example, generated using the quick calibration data) is greater than a threshold value, the monitoring module 350 determines that the calibration has been lost and causes the environment of the system 100 to re-establish the calibration. Furthermore, if the difference between the estimated position of the rear rigid body 230 and the predicted position of the rear rigid body 230 is greater than the threshold value, the virtual reality (VR) console 110 adjusts the predicted position of the rear rigid body 230 by a time offset value, such that the difference between the estimated position of the rear rigid body 230 and the predicted position of the rear rigid body 230 is less than the threshold value. The monitoring module 350 can then use the time offset value (or subsequently generated time offset values) to more accurately predict the position of the rear rigid body 230 until re-calibration can occur between the front rigid body 205. and the rear rigid body 230. Alternatively, if a difference between the estimated positions of the locators 120 on the rear rigid body 230 and the positions of their corresponding model locators, relative to the reference point 215,

If the “οπίΟΑΓ OamariAjtjr INDUmiAl is greater than a threshold value, the monitoring module 350 determines that the calibration has been lost and causes the system environment 100 to reset the calibration. In some embodiments, when the slow calibration data includes an image that includes a threshold number of locators on the front rigid body 205 and a threshold number of locators on the rear rigid body 230, the tracking module 150 begins re-calibration.

Additionally, in some embodiments, once tracking is lost, the monitoring module 350 automatically prompts the user to adjust the virtual reality (VR) viewer 105, so that the locators are visible on both the front rigid body 205 as in the rear rigid body 230.

Calibration of virtual reality systems

Figure 4 is a flow diagram of one embodiment of a process for calibrating a virtual reality (VR) system, such as the environment of system 100 described above in connection with Figure 1. In other embodiments, the method includes different, additional, or fewer steps than those represented in figure 4. Additionally, in some embodiments, the steps described in relation to figure 4 can be performed in different orders.

Virtual reality (VR) console 110 initializes 410 the system environment using one or more calibration parameters. For example, virtual reality (VR) console 110 retrieves one or more calibration parameters associated with the virtual reality headset.

45 MAID

Μρπντο MEXICAN

MLA ηοΠΕΟΛΟ! NOOFT »IAL (VR) 105 from the tracking database 310. In some modalities, the virtual reality (VR) console 110 retrieves the adjusted calibration parameter values from the database tracking 310 if the image processing device 135 or the inertial measurement unit (IMU) 130 were previously calibrated for a particular virtual reality (VR) viewer 105. If the image processing device 135 or the inertial measurement unit (IMU) 130 were not previously calibrated for the virtual reality (VR) viewer 105, the virtual reality (VR) console 110 restores the default calibration parameters to from tracking database 310. Virtual reality (VR) console 110 provides the calibration parameters to inertial measurement unit (IMU) 130 or image processing device 135.

The virtual reality (VR) console 110 receives (420) the slow calibration data from the image processing device 135, and the fast calibration data from the inertial measurement unit (IMU) 130. The slow calibration data includes a series of images including one or more of locators 120 in virtual reality (VR) viewer 105. A locator 120 included in an image from the slow calibration data is referred to herein as an observed locator. The quick calibration data may include one or more intermediate estimated positions of the reference point 215 (eg, a center of the inertial measurement unit (IMU) 130). In other modalities,

IMPL · (MRIIUTO MBXlCANt) 'i' MUAFRCTIEDAD industrial

Ο. or "Quick Calibration" data includes the intermediate Hp acplpranion information and / or the intermediate speed information from which the virtual reality (VR) console 110 determines one or more intermediate estimated positions of the setpoint 215.

Based, at least in part, on the slow calibration data and a viewer model, the virtual reality (VR) console 110 identifies (430) the model locators, which are the locators in the viewer model. The virtual reality (VR) console 110 extracts the locator information describing the positions of the observed locators 120 relative to each other in the form of slow calibration data, and compares the locator information with a model. retrieved from trace database 310 to identify (430) model locators that correspond to observed locators. Model locators are components of the viewer model, such that identification (430) of a model locator associated with an observed locator allows virtual reality (VR) console 110 to subsequently compare a position of the observed locator with the ideal position, from the viewer model of the model locator associated with the observed locator.

Using the viewer model, virtual reality (VR) console 110 generates (440) the estimated positions of one or more of the observed locators 120. The scope model describes the ideal positioning between locators 120 and the point of

IMPI t> s the v

Noum'At reference 215. In various modalities, the virtual reality (VR) console 110 uses the viewer model and the information from the locators to determine a projection matrix for the translation of the ideal positions in the viewer model to the Positions in an image plane of image processing device 135. Virtual reality (VR) console 110 uses the projection matrix to estimate the positions of the observed locators. Accordingly, the estimated position of an observed locator 120 identifies an ideal position of the observed locator 120 in the image plane of the images from the slow calibration data.

Based, at least in part, on the relative distances between the estimated positions of the one or more observed locators 120 and the positions of the model locators corresponding to the one or more observed locators 120, the virtual reality (VR) console 110 adjusts (450) one or more calibration parameters that adjust the estimated positions of the one or more locators 120, such that a relative distance between the estimated positions of the observed locators 120 and the positions of their corresponding model locators from the viewer model is less than a threshold value (eg, 1 millimeter). Adjusting the calibration parameters affects the projection matrix (for example, the change in focal length, etc.), such that changing one or more calibration parameters can affect the estimated positions

of the 120 localizers observed. If the distances between the estimated positions of the observed locators 120 and the positions of their corresponding model locators are equal to or greater than the threshold value, in one mode, the virtual reality (VR) console 110 adjusts (450) a calibration parameter , while keeping the other calibration parameters fixed to determine a value for the calibration parameter being adjusted that results in a distance between the estimated position of an observed locator 120 and the position of its corresponding model locator being less than the threshold value. The calibration parameter can then be set to the determined value, while another calibration parameter is modified such that the distance between an estimated position of an additional locator 120 and an additional position of a model locator corresponding to the additional locator is less than the threshold value. Various calibration parameters can be adjusted (450) as described above such that the relative distances between the adjusted estimated positions of at least a threshold number of observed locators 120 and the positions of their corresponding model locators are less than the threshold value. If the distances between the estimated positions of at least a threshold number of the observed locators 120 and the positions of their corresponding model locators are less than the threshold value, the calibration parameters are not adjusted (450).

49 IMPIOUS?

Mexican rxsTmrro

Dt THE CURRENCY

INDUCTUAt

Virtual reality (VR) console 110 determines (460) whether a threshold number of observed locators 120 are on each side of front rigid body 205 (i.e., front side 220A, top side 220B, bottom side 220C, the right side 220D, and the left side 220E). If the threshold number of observed locators 120 are associated with each side, the virtual reality (VR) console 110 generates (470) the calibrated positions of the reference point 215 for one or more frames of the slow calibration data using the estimated positions. of the observed locators. In modalities where the virtual reality (VR) viewer 105 includes multiple rigid bodies, the virtual reality (VR) console 110 generates the calibrated positions of the reference point 215 that respond to the determination that a threshold number of locators are projected (observed locators) on one or more sides of each rigid body 205, 230, or that respond to the determination that a threshold number of locators are projected (observed locators) on all sides of each rigid body 205, 230. If the threshold number of observed locators 120 are not associated with each side, the virtual reality (VR) console 110 can communicate a message to the user through the virtual reality (VR) viewer 105 or other component, to reposition the virtual reality (VR) viewer 105 such that slow calibration data can be captured including locators on one or more sides of the virtual reality (VR) viewer 150.

50 WICKED

INSTITUTO MEXICANO nor the μοπεοαο Cw ^ rawCí

IN GUSTELA l

The virtual reality (VR) console 110 adjusts (480) one or more calibration parameters until the intermediate estimated positions of the virtual reality (VR) viewer 105 received from the quick calibration data are within a threshold distance of the predicted positions for virtual reality (VR) viewer 105 or landmark 215, wherein the predicted positions are determined from the calibrated positions of the reference point 215 associated with various images from the slow calibration data. In some embodiments, virtual reality (VR) console 110 determines a predicted position of set point 215 by generating (eg, by curve fitting) a prediction function using calibrated set point positions. reference 215 associated with different images from the slow calibration data. The virtual reality (VR) console 110 adjusts one or more of the calibration parameters until the distances between the intermediate estimated positions of the reference point 215 and the predicted positions of the reference point 215 are less than a threshold distance. For example, virtual reality (VR) console 110 may increase the sample rate of inertial measurement unit (IMU) 130 until the distances between the intermediate estimated positions of the landmark 215 and the predicted positions of the landmark 215 are all 1 millimeter or less, or until the distances between at least a threshold number of intermediate estimated positions of the point of

WICKED

INSTITUTO MIXICANO reference 215 and the predicted positions of the punfo ^ c »iArefb * ^^ 215 are less than 1 millimeter. In other npntjahdarie *.| a virtual reality (VR )_console 110 determines a predicted position of reference point 215 as a position between a calibrated position of reference point 215 associated with an image from the slow calibration data and a calibrated position of reference point 215 associated with a subsequent image from the slow calibration data. The virtual reality (VR) console 110 then adjusts (480) one or more calibration parameters such that the distances between each intermediate estimated position and a calibrated position (eg, CPi) of the reference point 215 associated with the image , are less than a distance between the calibrated position (for example CPi) of the reference point 215 associated with the image and the calibrated position of the reference point 215 associated with the subsequent image (for example, CP2). Additionally, the virtual reality (VR) console 110 can update the initial position of the inertial measurement unit (IMU) 130 to be the calibrated position of the reference point 215.

In some embodiments, the virtual reality (VR) console 110 stores the adjusted calibration parameter values in the tracking database 310 or provides the adjusted calibration parameter values to other components of the virtual reality console ( VR) 110. Adjusted calibration values may reduce calibration times for

subsequent operations of the

INDUSTRIAL 100 system environment, improving the user experience.

Virtual reality (VR) console 110 monitors (490) the environment of system 100 to determine loss of calibration.

For example, virtual reality (VR) console 110 monitors the relative distances between the adjusted estimated positions of the observed locators 120 and the positions of their corresponding model locators. If a relative distance between an adjusted estimated position of an observed locator 120 and a position of its corresponding model locator is less than a threshold value (for example, 1 millimeter), the virtual reality (VR) console 110 provides the calibrated position. to the virtual reality (VR) engine 155. Conversely, if the relative distance between an estimated position of an observed locator and a position of its corresponding model locator is greater than (or equals or exceeds) the threshold value (for example, 1 millimeter), the display console virtual reality (VR) 110 determines calibration lost, receives (420) slow calibration data and fast calibration data, and performs the functions identified above to recalibrate the environment of the system 110.

Additionally, the virtual reality (VR) console 110 monitors (490) the relative distances between the intermediate estimated positions of the reference point 215 and the predicted positions of the reference point 215. For example, if a distance between a curve of predicted positions from benchmark 215 im '

IMPI t »rrrrvTO mxxican M LA PtOH LOAD and an estimated intermediate position of the point of reference.1· 2ΐΓ- & 5 ^ less than a threshold distance (for example, 1 millimeter), - virtual reality (VR) console 110 provides the estimated intermediate position to the virtual reality (VR) engine 155. In some modalities, the reality console Virtual reality (VR) 110 can also provide the intermediate speed information or the intermediate acceleration information extracted from the quick calibration data to the virtual reality (VR) engine 155. Conversely, if the distance between the predicted position of the landmark 215 and an intermediate estimated position of the landmark 215 is greater than, or equal to or exceeds, the threshold distance, the virtual reality (VR) console 110 determines After the calibration was lost, it receives (420) the slow calibration data and the fast calibration data, and performs the functions identified above to recalibrate the environment of the system 100.

In some embodiments, the inertial measurement unit (IMU) 130 and the image processing device 135 may be calibrated simultaneously. To simultaneously calibrate the inertial measurement unit (IMU) 130 and the image processing device 135, the virtual reality (VR) console 110 estimates the positions of the reference point 215 for a series of images using the estimated positions of the locators observed.

Additionally, the virtual reality (VR) console 110 uses the quick calibration data, including the estimated positions.

IMPI wsTrruro Mexican

QE LA rxOHEPAn INtX'JTlUAL benchmark 215 at the particular time values that correspond to the images from the slow calibration data when the inertial measurement unit (IMU) 130 and the image processing device are calibrated

135. When the calibration parameters are adjusted simultaneously from the inertial measurement unit (IMU) 130 and the image processing device 135, the virtual reality (VR) console 110: (1) adjusts the estimated positions of the locators observed in such a way that the relative distance between the adjusted estimated positions of the observed locators and the positions of their corresponding model locators is less than a threshold value; and (2) adjusts the estimated positions for the reference point such that the relative distance between the estimated positions for the reference point at the particular time values corresponding to the images from the slow calibration data and the positions from a model reference point determined from the model locators, is less than the threshold value.

Figure 5 is a flowchart illustrating one embodiment of a process for re-calibrating two rigid bodies of a virtual reality viewer 225 included in the environment of the system 100. In other embodiments, the procedure includes additional, different steps, or in fewer numbers than those represented in figure 5. Additionally, in some embodiments, the steps described in relation to figure 5 are

MAID

INSTITUTO MEJICANO for the INDUÍTJUAL raortroAD can be carried out in different orders.

The virtual reality (VR) console 110 receives 7 ^ $ TT ^ s ”slow calibration data, including images showing a front threshold number of locators 120 on a front rigid body 205 and a rear threshold number (for example, at least one) of the locators 120 on a rear rigid body 230 of the virtual reality (VR) viewer 225. A locator 120 included in an image from the slow calibration data is referred to herein as an observed locator. As described above in connection with Figures 2 to 4, virtual reality (VR) console 110 receives (150) slow calibration data from image processing device 135, and fast calibration data from unit Inertial Measurement (IMU) 130. The quick calibration data may also include the intermediate acceleration information and / or the intermediate speed information from which the virtual reality (VR) console 110 determines one or more intermediate estimated positions of the reference point 215 of the virtual reality (VR) viewer 225.

Based at least in part on the slow calibration data and a viewer model, virtual reality (VR) console 110 identifies (520) model locators, which are locators in the viewer model. Virtual reality (VR) console 110 extracts locator information describing the positions of observed locators 120 relative to each other from slow calibration data, and compares the

INDUSTRIAL locator information with a model of TlSof retrieved from tracking database 310 to identify (520) model locators that correspond to observed locators 120. In at least one of the images, model locators corresponding to the locators observed on both the front rigid body 205 and the rear rigid body 230 of the virtual reality (VR) viewer 225 are identified. Model locators are components of the viewer model, such that identification (520) of a model locator associated with an observed locator allows the virtual reality (VR) console 110 to subsequently compare a position of the observed locator with the observed locator. ideal position, from the viewer model of the model locator associated with the observed locator.

Using the viewer model, virtual reality (VR) console 110 generates (530) the estimated positions of one or more of the observed locators 120. The scope model describes the ideal positioning between locators 120 and reference point 215. In various modalities, the virtual reality (VR) console 110 uses the viewer model and locator information to determine a projection matrix for translation from ideal positions in the viewer model to positions in an image plane. image processing device 135. Virtual reality (VR) console 110 uses projection matrix to estimate locator positions

120 observed.

Therefore,

IMPI rNyrmrr · Mexican Dt LA HtOFIEDAD nousnuAL

the estimated position of an observed locator 120 identifies an ideal position of the observed locator 120 in the image plane of the images from the slow calibration data.

Based at least in part, on the relative distances between the estimated positions of the one or more observed locators 120 and the positions of the model locators corresponding to the one or more observed locators 120, the virtual reality (VR) console 110 adjusts ( 540) the relative distance between the estimated positions of the observed locators on the front rigid body 205 and the positions of their corresponding model locators, by less than a threshold value (for example, 1 millimeter). Adjusting the calibration parameters affects the projection matrix (eg, the change in focal length, etc.), such that changing one or more calibration parameters can affect the estimated positions of the locators 120 observed. If the distances between the estimated positions of the observed locators 120 and the positions of their corresponding model locators are equal to or greater than the threshold value, in one mode, the virtual reality (VR) console 110 adjusts (540) a parameter of calibration, while keeping other calibration parameters fixed to determine a value for the calibration parameter being adjusted, which results in a distance between the estimated position of an observed locator 120 and the position of

ΪΜΡΙ oerm / το μεχκλνο WL * ntINDUSTRIAL HONEY its corresponding model locator is less than the threshold value. The adjustment (540) of the calibration parameters is further described above in connection with Figure 4. If the distances between the estimated positions of at least a threshold number of the observed locators 120 and the positions of their corresponding model locators are less than the threshold value, the calibration parameters are not adjusted (540).

After adjusting (540) the calibration parameters such that at least a threshold number of the relative distances between the estimated positions of the observed locators and the positions of their corresponding model locators are less than the threshold value, The virtual reality (VR) console 110 generates (550) the calibrated positions of the reference point 215 associated with one or more images from the slow calibration data using the adjusted estimated positions of the observed locators 120. In some embodiments, the virtual reality (VR) console 110 generates the calibrated positions of reference point 215 in response to determining that a threshold number of locators are projected (observed locators) on one or more sides of each rigid body 205 , 230, or by determining that a threshold number of locators are projected (observed locators) on all sides of each rigid body 205, 230. If the threshold number of locators is not projected (on one side of a

IMPI rWÍHTUTO MBOCANe DE UA INDUTTJUAL PROPERTY rigid body 205, 230, or on all sides of each rigid body 205, 230), the virtual reality (VR) console 110 can ask the user, through the virtual reality viewer (VR ) 105 or by means of another suitable component, which orients the virtual reality (VR) viewer 105 in a specific direction relative to the image processing device 135, or continue moving the virtual reality (VR) viewer 105 until the threshold number of locators is projected.

The virtual reality (VR) console 110 also determines (560) a position of the rear rigid body 230 relative to the reference point 215. In some embodiments, the virtual reality (VR) console 110 identifies a rear reference point in rear rigid body 230 using observed locators 120 and their corresponding model locators. The virtual reality (VR) console 110 then identifies the position of the rear reference point in relation to the reference point 215 on the front rigid body 205, such that the rear reference point is positioned relative to the reference point. reference 215 by a position vector. Alternatively, virtual reality (VR) console 110 identifies the position of each observed locator on rear rigid body 230 relative to reference point 215, such that the positions of each observed locator on the rigid body rear 230 are positioned relative to reference point 215 by their own position vector.

Virtual reality (VR) console 110 adjusts (570) one or more calibration parameters such that the intermediate estimated positions of reference point 215 are within a threshold distance of the predicted positions of reference point 215. The adjustment of the calibration parameters such that the intermediate estimated positions of the reference point 215 are within a threshold value of the predicted positions of the reference point, is further described above in connection with FIG. 4. After adjusting (570) one or more calibration parameters, the virtual reality (VR) console 110 monitors (580) to determine the loss of calibration of the environment of the system 100, as described above in connection with FIG. 4.

When monitoring (580) for loss of calibration, virtual reality (VR) console 110 uses the images from the slow calibration data, which may include the observed positions of locators 120 on front rigid body 205 , on the rear rigid body 230, or some combination thereof. In some embodiments, the threshold value between a position of an observed locator 120 and a position of its corresponding model locator may vary as a function of the rigid body in which the observed locator 120 is located. For example, the threshold value may be 1 millimeter for the 120 locators observed on the front rigid body 205, and 2 millimeters for the 120 locators observed on the body.

rear rigid 230.

Additionally, in some scenarios, the image processing device 135 is not able to see the locators 120 on the front rigid body 205, but is able to see the locators on the rear rigid body 230. In these scenarios, the tracking is monitored employing the process described below with respect to Figure 6.

When the slow calibration data includes an image that includes a threshold number of locators on the front rigid body 205 and a threshold number of locators on the rear rigid body 230, the steps described above in connection with FIG. 5 are repeated to reestablish the system environment calibration 100. In some embodiments, when tracking is lost, the virtual reality (VR) console 110 automatically prompts the user to adjust the virtual reality (VR) viewer 105, such that the locators on both the front rigid body 205 and rear rigid body 230, are visible to image processing device 135. The request presented to the user may provide the user with specific instructions for positioning the virtual reality (VR) viewer 105 such that the locators on both the front rigid body 205 and the rear rigid body 230 are visible to the user. image processing device 135.

Figure 6 is a flow chart illustrating one embodiment of a process for maintaining a mate.

I— τ>

MAID

BOTTUTO MEXICANO Pt LAMUNNUfeO ΙΝΟΙΉΙΙΙΑΙ between two rigid bodies of a virtual reality viewer 225 included in the environment of the system 100. In other modalities, the procedure includes different, additional, or fewer steps than those represented by figure 6. Additionally, In some embodiments, the steps described in relation to Figure 6 can be performed in different orders.

The virtual reality (VR) console 110 receives (610) the slow calibration data from the image processing device 135. The slow calibration data includes a series of images that includes an image associated with an image time value, and having only the locators 120 seen on the rear rigid body 230 visible to the imaging device 135. An image time value is a value of time when the image was captured by the image processing device 135. Additionally, the virtual reality (VR) console 110 receives 620, from the inertial measurement unit (IMU) 130, the quick calibration data that includes the intermediate estimated positions of a reference point 215 for a series of time values including the time value of the image.

Based on the slow calibration data, the virtual reality (VR) console 110 determines (630) an observed position of the rear rigid body 230 at the time value of the image. To determine (620) the observed position of the rear rigid body 230, the virtual reality (VR) console 110 extracts the information from the

IMPI • βΠΤνΤΌ MWCAHtt '& SLíí CULAHOnWAn locators that describes the positions of the localizers ΐ2ΐΓ observed in the rear rigid body 230 υηοδ'δΠ felaoion δδϊΠδΤ others, from the slow calibration data, and compares the information of the locators with a model of viewer retrieved from trace database 310, to identify pattern locators corresponding to observed locators 120. After identifying the model locators, the virtual reality (VR) console 110 determines the observed locators 120 corresponding to each model locator, and determines a rear reference point for the rear rigid body 230 using the positions of the observed locators 120. . In some embodiments, the observed position of the rear rigid body 230 is the position of the rear reference point. In alternative embodiments, the observed position of the rear rigid body 230 may be the observed position of one or more of the observed locators.

The virtual reality (VR) console 110 determines (640) a predicted position of the rear rigid body 230 at the picture time value using the quick calibration data and a position vector. The position vector describes a calibrated offset between the front rigid body 205 and the rear rigid body 230. For example, the position vector describes a calibrated offset between the reference point 215 associated with the front rigid body 205 and a rear reference point associated with the rear rigid body 230. Additionally, in some embodiments, the θ4 vector.

IMPI® namvro MíXKAHO

HUWHlymp ________. . . ·,. . .rrWSTMAL - position may include one or more sub-vectors each describing the relative calibrated offset between reference point 215 and different locators on rear rigid body 230.

From the quick calibration data, the virtual reality (VR) console 110 determines an intermediate estimated position of the reference point 215 on the front rigid body 205. In some embodiments, the virtual reality (VR) console 110 determines the predicted position of the rear rigid body 230 as a position relative to the position of the reference point 215 based on the position vector. For example, the position vector identifies a relative positioning of a rear reference point on the rear rigid body 230 to the reference point 215. Alternatively, the position vector identifies the relative position of one or more locators 120 ( including observed locators) on rear rigid body 230 relative to reference point 215.

Virtual reality (VR) console 110 determines (650) whether a difference between the observed position and the predicted position is greater than a threshold value (eg, 1 millimeter). If the difference is less than the threshold value, tracking of the virtual reality (VR) viewer 225 is maintained, and the slow calibration data is received (610), and the process proceeds as described above. However, if the difference between the observed position of the rear rigid body 230 exceeds the threshold value, the virtual reality (VR) console 110 determines that the tracking of the viewfinder was lost.

IMPT ^

MEXICAN INSTITUTE. M LA nOUFMO virtual reality (VR) 105, and adjusts (660) the position pRS * tñWa pomTrT lag value. The date offset value miid dg such that the difference between the observed position of the rear rigid body 230 and the predicted position of the rear rigid body 230 is less than the threshold value. For example, virtual reality (VR) console 110 uses the position vector modified by the offset value to more accurately determine the position of the rear rigid body 230 from the quick calibration data. Alternatively, the virtual reality (VR) console 110 communicates an instruction to the inertial measurement unit (IMU) 130 to compensate for intermediate estimated positions based on the offset value without modifying the position vector.

Based on the quick calibration data and the adjusted vector, the virtual reality (VR) console 110 determines (670) the subsequent predicted positions of the rear rigid body until re-calibration occurs (eg, as described above with respect to figure 5). In some embodiments, when tracking is lost, the virtual reality (VR) console 110 prompts the user to adjust the virtual reality (VR) viewer 105, such that the locators on both the front rigid body 205 and the rear rigid body 230 are visible to image processing device 135. The message presented to the user may provide the user with specific instructions to position the virtual reality (VR) viewer 105 such that the ίΗίττηπη μ mica nq, ιΏ ce i * nopiepaP. . = ___.____ x- x ...... ......... ÍNixjyntixL -Ή .-- locators on both the front rigid body 205 and the rear rigid body 230 are visible to the OS Image Processing 135 device to facilitate the re-calibration described in detail above with reference to Figure 5.

Figure 7 illustrates an example graph 700 illustrating a series of calibrated positions of a virtual reality viewer 105. In Figure 7, the vertical axis represents position, and the horizontal axis represents time. Graph 700 includes a series of calibrated positions 710A-C of a virtual reality (VR) viewer 105 reference point, at times Ti, T2 and T3, respectively. Chart 700 also includes a series of estimated intermediate positions 715A-D and 720A-H of the reference point. The 710A-C calibrated positions are generated using the slow calibration data from an image processing device 135, and the 715A-D and 720A-H intermediate estimated positions are generated using the fast calibration data from the inertial measurement unit. (IMU) 130 embedded in a virtual reality (VR) viewer 105. Note that the relative time scales of the 710A-C calibrated positions and the 715A-D intermediate estimated positions are different, and that the 715A-D and 720A-H intermediate estimated positions are determined more frequently than the 710A calibrated positions. -C.

Graph 700 shows a predicted position curve 725 described by a prediction function that describes the predicted position of the reference point. The prediction function is generated

by fitting a curve to calibrated positions 710A-C, and by determining a function that describes the fitted curve. Any suitable method can be used to determine the position function from the 710A-C calibrated positions.

In the example of FIG. 7, the intermediate estimated positions 715A-D are the initial intermediate estimated positions determined using the quick calibration data prior to adjustment of the calibration parameters. The intermediate estimated position 715A is relatively close to the predicted position curve in Figure 7, but as time progresses, the intermediate estimated positions move further away from the predicted position curve 725, the intermediate estimated position 715D being of Figure 7 is the furthest from the predicted position curve 725. The difference between the predicted position curve and the intermediate estimated position can be attributed to a combination of actual user movements, drift error, as well as additional factors. As discussed above, because the inertial measurement unit (IMU) 130 determines an intermediate estimated position relative to a previously determined position, the error compounds result in a larger deviation between the predicted position curve 725 and the estimated intermediate positions 715 over time. To account for drift error, the virtual reality (VR) console 110 can update an initial position of the drive unit.

IMPL twrrnvroMEncANa i

OF U 'OFIETY' inertial measurement (IMU)

130 as the subsequent calibration position. The inertial measurement unit (IMU) 130 then generates a quick calibration with respect to the updated home position and the intermediate estimated positions determined after the home position. In this mode, the virtual reality (VR) console 110 updates the starting point as the calibrated position 710B.

Another way to reduce the error associated with estimated intermediate positions is by increasing the frequency that estimated intermediate positions are determined. In the example of Figure 7, the virtual reality (VR) console 110 determines the intermediate estimated positions 720A-H at twice the frequency of the intermediate estimated positions 715A-D, resulting in a smaller difference between the intermediate positions. intermediate estimated positions 720A-H and the predicted position curve 725. Additional configuration information

A rigid body tracking system is introduced for tracking the unpredictable movement of rigid bodies. This rigid body tracking system can be useful for virtual reality, augmented reality, games, education, training, and therapy.

One embodiment of the rigid body tracking system is to follow the movement of a head-mounted display that is attached to the wearer's head. In this case, the light-emitting diodes (LEDs) are attached to the surface of the screen. A unit of

IMPI wstttvtc Mexican Dt LA HLOHEDA ».... rnoumui inertial measurement is mounted inside the screen and can include a sensor that measures angular velocity, such as one or more gyroscopes. It may additionally include one or more accelerometers and one or more magnetometers. A separate camera is placed in a fixed location, facing the user. In some modalities, the rigid body tracking system is the virtual reality (VR) system 100.

In another embodiment, the rigid body being tracked can be held in the user's hand. For example, this could allow the position and orientation of the hand to be maintained in a virtual reality experience. In another embodiment, the camera can track multiple rigid bodies, including more than one viewfinder and more than one object held by the additional hand.

The sensors that make up the inertial measurement unit provide sufficient measurements to estimate the orientation of a rigid body. These sensors include a gyroscope, an accelerometer, and possibly a magnetometer. If the rigid body is in view of a stationary camera and at an appropriate distance, then additional orientation information can be inferred. Additionally, the position of the rigid body is inferred. The tracking method combines measurements from all detection sources, including one or more cameras, in such a way that the position and orientation of the rigid body are accurately maintained. The placement, number, and modulation of light-emitting diodes (LEDs) on the surface of the rigid body are

HE0

'or IMPI Mexican errwro LA WtOfUDAIi '«NDUSnUAL crucial to the rigid body tracking system. These factorst. . . _ ______ together lead to reliable and highly accurate tracking of the rigid body.

Note that the various features of the present invention can be practiced alone or in combination. These and other features of the present invention will be described in greater detail later in the Detailed Description of the Invention and in conjunction with Figures 8 to 10.

The Rigid Body Tracking System works by merging sensor data from two sources: an inertial measurement unit inside the rigid body and a stationary camera that looks into the body. The most common embodiment is that the rigid body is a head-mounted display that attaches to a human head. In this case, the position and orientation of the head must be tracked with low latency and jitter. The information provided by the sources is complementary in that the inertial measurement unit provides the most powerful limitations on head orientation, and the camera provides the most powerful limitations on head position. Together, they provide the inputs to the filters that estimate the position and orientation of the head.

The most important component of the inertial measurement input is a high-frequency, three-axis gyroscope, which measures the angular velocity of the rigid body. After numerical integration of the sensor data that is accumulated through the

time, the current orientation is estimated relative to the initial orientation. To account for dead reckoning errors that accumulate over time, an accelerometer or other sensor can be used to estimate the direction below to compensate for tilt errors. A magnetometer can possibly be included in the inertial measurement to compensate for errors in the estimated orientation with respect to rotations around the vertical axis (parallel to gravity). This is particularly useful when the rigid body is not in the camera's field of view; otherwise, the camera may alternatively provide this correction.

The camera captures frames at a standard rate of

Hz, although both higher and lower speeds may offer advantages in other modes. The resolution is 640 x 480 (standard VGA), and other resolutions can also be used. The images are sent over a serial communication link (USB) to the central processor. There is also a serial link between the camera and the virtual reality (VR) viewer. This is used to command the camera to open and close the shutter for short, precise time intervals. A typical exposure time is 0.1 milliseconds. The lens can be any diopter, including narrow vision, wide angle, or fish eye. The camera may or may not have an infrared (IR) filter on its lens.

Imagine a user wearing a virtual reality (VR) headset

and is about to be magically transported ^ ias4ÍA »ww *» 4 * -reart to a virtual world. Ideally, the movements of the user's head in the real world should be perfectly reflected in the virtual world so that the user's brain is completely fooled. When images of the virtual world are presented to the user's eyes, they must correspond exactly to what the user expects to see based on the user's sense of immersion. A critical component to achieving this is head tracking, which involves collecting data from sensors and processing it to determine how a user's head is moving.

A user's head and virtual reality (VR) viewer together can be thought of as a rigid body that moves through 3D space. A rigid body has up to six degrees of freedom (DOF), which means that six independent parameters are needed to fully specify its position and orientation in space. Using these six parameters, we can calculate the coordinates of any point on the body. (It took a long time to find out in the history of physics, most of the credit being given to Euler and Lagrange, in the late 18th century). Although there are many ways to choose these parameters, they are generally done in such a way that three correspond to orientation (the way the body rotates) and three correspond to position (where the body is placed). These can be referred to as the orientation parameters of

yaw, pitch, and wobble. In previous methods, only these three parameters were scanned, resulting in the 3-degree-of-freedom (3-DOF) scan. Using these, a virtual reality (VR) system would know which way its head is pointing, but unfortunately it has to guess where the user's eyes might be located.

Using all three parameters, a 3D rotation is applied to the base of the head model, and the eyes are moved to a reasonable, but not necessarily correct, location. The first problem is the measurement of the head model: How big is the head of a user? A bigger problem is that there is nothing to prevent a user from moving the base of the head. For example, when a user looks forward and moves from side to side, their head turns very little, but the position changes a lot. Alternatively, if a user leans forward at their hips, while keeping their neck rigid, the base of their head will travel a lot. Failure to take these types of head base movements into account causes vestibular mismatch, which can contribute to simulator dizziness. The problem is that the sensors in a user's inner ear do not agree with what their eyes are seeing. The inner ear detects movement, but the eyes do not see it.

The rigid body tracking system tracks all six parameters directly from the sensor data, rather than having to guess the three additional position parameters. East

IMPI ^ g »mrrvK> mmican

M LA ntOMLPAD is 6-degree-of-freedom (6-DOF) tracking, which allows the rigid body tracking system to estimate both the direction a user is in and the position of the eyes while move your head around. This assumes that some fixed, person-specific quantities have been measured, such as interpupillary distance (IPD). In addition to a reduction in vestibular mismatch, the level of immersion is incredible! A user can move their head back and forth to judge depth. While standing on a pirate's board, a user can bend over to gaze out at the gruesome waters. The possibilities are endless!

VISION-BASED POSITION TRACKING

Position tracking is not a new problem in virtual reality (VR). There have been many attempts using a variety of technologies: single or multiple cameras, structured light, time of flight, and magnetic sensors to name a few. Solutions vary based on cost, power consumption, accuracy, computational requirements, and environmental limitations. To this end, some modalities describe a position tracking solution that is based on a low-cost, low-resolution single camera.

Camera-based position tracking, also known as posture estimation, is a major problem for many robotics and computer vision applications. The relevant literature is huge. However, most of this literature

• πτυτο has nothing to do with precision, resources'rmrrpe + eeieuaLae ^ JjU— latency. When navigating a car in the desert, a position estimate that deviates by a few centimeters is quite good, and there is no need to update the position very frequently. In contrast, in the context of virtual reality (VR), tracking must be very accurate, with low latency, and cannot divert computing resources away from rendered content.

In some modalities, the position tracking system is made up of three components: a set of markers built into the virtual reality (VR) viewer (for example, the virtual reality (VR) viewer 105), a measurement unit, nercial (IMU) and an external camera. Figure 8 shows a high-level flow diagram of a rigid body tracking system.

The rigid body tracking system acquires images from the virtual reality (VR) viewer. Through image processing, the rigid body tracking system extracts the image position of the markers. Each marker is an infrared (IR) light-emitting diode (LED). Based on the image positions of the markers, and on the known 3D model of the virtual reality (VR) viewer, the rigid body tracking system calculates a 6D posture. The algorithm takes advantage of the information available from the Inner Measurement Unit (IMU) (gyroscope, magnetometer and accelerometer). After merging the vision data and the commercial measurement unit (MIU), we determined the

MAID

Mexican Henrurc monroAP _____ optimal viewer posture. The result is provided by the application for representation. We now return to an inalioij ι.ιΧ, ΟΓΠΐΤίΤίυ of the individual components of the algorithm.

The rigid body tracking system identifies each of the visible markers (light-emitting diodes (LEDs)). In each captured image, the rigid body tracking system detects several bright spots. Each point corresponds to a light-emitting diode (LED). The rigid body tracking system identifies which of the light emitting diodes (LEDs) physically produces each point. Once this is done, the rigid body tracking system associates a known 3D model of the light-emitting diodes (LEDs) in the virtual reality (VR) viewer with their observed image projections. With this association, and given a calibrated camera, the rigid body scan can estimate the posture that best explains the observed pattern of points.

As a result, the identification problem is difficult. In order to identify individual light-emitting diodes (LEDs), the rigid body tracking system extracts some unique information about each one. There are many possible ways to achieve just that. For example, the rigid body tracking system could use a variety of visible light light-emitting diodes (LEDs) and could distinguish them based on color. Another interesting approach is to incorporate geometric information in the form of lines or other shapes. Based on the way the points are arranged, the rigid body tracking system

77 DREAM> WU IUH> MEXICAN

OF THE HIOFIEDAO

INDUSTMAL can determine your individual identifications. And also the spatial arrangement, such as allowing only coplanar markers to simplify their identification.

Most of these ideas are computationally complex, are not strong for occlusion and noise, or place strict requirements on the geometry and appearance of the virtual reality (VR) viewer. In some embodiments, infrared (IR) light-emitting diodes (LEDs) are used in a general 3D configuration. Light Emitting Diodes (LEDs) are indistinguishable based on appearance. They all look like bright spots. Additionally, there is also no special geometric arrangement that can simplify its identification. Bright spots are also very abundant in images; noise, other light sources, and our light-emitting diodes (LEDs) all look the same. And finally, because the light-emitting diodes (LEDs) are scattered throughout the virtual reality (VR) viewer, only some of the light-emitting diodes (LEDs) are visible in any given image, and some diodes Light emitters (LEDs) can be obstructed by the user's hands.

Light-emitting diodes (LEDs) are built into the virtual reality (VR) viewer. In some embodiments, a solution is based on modulating the brightness of light-emitting diodes (LEDs). Over time, each light-emitting diode (LED) displays a unique light pattern. Because light-emitting diodes

ΙΜΡΙ®>

INTTTnjTnMWiCASL r> »the homedai> (LEDs) operate in the near infrared (IR) spectrum, soWVhv i sttrfe? for the human eye. However, in the luxury (IR) mirror, the »light-emitting diodes (LEDs) are very bright and have a wide field of illumination (approximately 120 degrees). The camera is designed to be sensitive to light in the near infrared (IR) frequency, and has a wide field of view lens. These properties - bright light-emitting diodes (LEDs) and wide field of view - allow for a large scan volume.

A vision algorithm detects individual light-emitting diodes (LEDs) and tracks them across multiple frames. It then analyzes the light pattern displayed to determine the unique identification of each light-emitting diode (LED). Because the rigid body tracking system decodes each light emitting diode (LED) individually, this approach is strong for noise and occlusion. However, it makes use of several frames before decoding is achieved. It also means that, in any given frame, there may be some new (not yet decoded) light-emitting diodes (LEDs), and some decoded light-emitting diodes (LEDs) may be out of sight. The reconstruction algorithm has to be robust against false positives.

Note that the camera has a very fast shutter (less than a millisecond long) and the light-emitting diodes (LEDs) are synchronized so that they only light when the shutter is open. As a result, the rigid body tracking system can greatly reduce the amount of ambient light collected pnr_ Ja ñamara, save energy, minimize blurring during rapid head movement, and most importantly, maintain the known synchronization between the camera and the virtual reality (VR) viewer. This can be important when the rigid body tracking system performs sensor fusion between the inertial motion unit (IMU) and vision measurements.

RECONSTRUCTION OF POSTURE

Once the identification of each light-emitting diode (LED) is known, and given a known 3D model of light-emitting diodes (LEDs), the rigid-body tracking system is ready to solve a classic projective geometry problem. - which is the 6D posture (orientation and 3D position) of the virtual reality (VR) viewer that best explains the observed projection of light-emitting diodes (LEDs).

The rigid body tracking system has two types of posture reconstruction modes: bootstrapping and incremental.

The first variant, bootstrapping, occurs when the rigid body tracking system first perceives the images from the light-emitting diodes (LEDs). The rigid body tracking system previously solved the identification problem, which means that you can have a guess about the identifications of some of the bright spots in the frame. But the rigid body tracking system must have

ΓΗΓΓΤΤυΤΟ MkJCKANO i

..... '□ E INDUSTRIAL PTOFTETY Caution: some of the identifications may be wrong, some of the points may not have been identified yet, and some of the points may be due to other light sources in the environment. During bootstrapping, the rigid body tracking system has to resolve all these conflicts quickly and calculate the posture of the virtual reality (VR) viewer.

Bootstrapping is a computationally very efficient method for generating posture hypotheses based on decoded light-emitting diodes (LEDs). The rigid-body tracking system sorts through these hypotheses using a variant of random sampling and consensus (RanSAC) to find the subset of light-emitting diodes (LEDs) that were correctly identified.

In order to understand what these posture hypotheses pose, a triangle ABC projected onto a plane triangle in the image is taken. A posture hypothesis works backwards; it tells the rigid body tracking system what is the position and orientation of the triangle that the image explains. In the case of 3 points A, B and C, there cannot be more than one position with the same projection. It is important to note that this posture ambiguity disappears when the points are not coplanar, and decreases with more than 3 points.

The last part of the bootstrapping is to refine the reconstructed posture by including as many light-emitting diodes (LEDs) as possible. The tracking system

MAID

IHSTTTUTO MULLALO HEARD THE MOHEDAL? INDUSTRIAL

rigid body does this by projecting our model

3D on the image using the calculated posture. The rigid body tracking system then matches the expected light-emitting diode (LED) positions for the observations. This allows the rigid body tracking system to dramatically increase the number of identified light-emitting diodes (LEDs) and refine our calculated posture as a result. The coincidence of the positions of the expected light-emitting diode (LED) image with the observations turns out to be the key to solving the second version of the posture reconstruction problem: the incremental problem.

Figure 9 illustrates the observed bright spots (the largest spots) and the predicted projections (the smallest spots). Matching the pairs of the predicted and observed light-emitting diodes (LEDs) can be challenging in some cases.

It's worth noting that matching the expected and observed light-emitting diode (LED) positions can be quite difficult, particularly with fast head movements and because all light-emitting diodes (LEDs) have the same appearance.

The second variant of the posture estimation problem is much easier. Fortunately, it's the one that the rigid body tracking system has to solve most of the time! When the rigid body tracking system is on a (ΜΤΥΛΛΌ MEXICAN

DXLAAkíWlEDAO -.:7J

INeuSTlUL incremental mode, the 'rigid body tracking system can use the virtual reality (VR) viewer posture from the previous frame as a starting point for calculating your posture in the current frame. How does it work? In a word: the prediction. During incremental mode, the rigid body tracking system merges information from vision and the inertial measurement unit (IMU) to calculate where the virtual reality (VR) viewer is expected to be. The rigid-body tracking system then matches the prediction with the observations, and makes the necessary corrections.

Incremental estimation has been applied to many different problems. For example, many optimization algorithms use an iterative approach in which each step gets a little closer to the solution. Another example is the problem where measurements are provided in real time (online). A wide variety of filters have been proposed, the most popular of which are those of the Bayesian filter family. This family includes the Kalman Filter, the Extended Kalman Filter, the Sequential Monte Carlo filters, and others. Real-time posture estimation is also often approached using a Bayesian filter.

The rigid body tracking system uses a custom filter that merges the information and can make predictions. The starting point is the position and orientation calculated in the table above. It's a good starting point, but you can certainly move a bit between camera frames (16.6 ms @ 60 Hz).

If the rigid body tracking system goes the pnctnra c ^ hra i?and----- last frames, some prediction can be made based on the speeds. Now we are a little closer. Fortunately, the rigid body tracking system also has an inertial measurement unit (IMU). This means that between the previous and current image, the rigid body tracking system actually has approximately 16 measurements of angular velocity and linear acceleration. Our custom filter merges all of this information, and provides history-based posture estimates, which are based on positions, velocities, and accelerations. The end result is quite close to the actual position in the current frame.

The rigid body tracking system uses predicted posture to project light-emitting diodes (LEDs) onto the image plane. The rigid-body tracking system then matches the predicted light-emitting diode (LED) image coordinates with the observations. If the prediction is correct, the pairings are perfect. And finally, the rigid body tracking system calculates the necessary correction to the estimated posture, resulting in a perfect match.

Figure 10 illustrates the posture optimization step. A horizontal 2D plane is shown, where the optimal solution is the global minimum. The rigid body tracking system begins with a predicted posture based on history, speed, and

I λί p

INSTITUTO MSXICaMO 'of the r * ortttun INDUSTRIAL acceleration. The solution is improved iWnthOmcnt? nHiiTanriQ gradient descent optimization, until the rigid body tracking system finds the optimal posture. Incremental posture estimation is computationally efficient, and it is the mode that the rigid body tracking system expects to be in most of the time. Once bootstrapping is successful, and as long as the virtual reality (VR) viewer is visible to the camera, the system can continually go through this process of: predicting posture - matching observations refining posture.

The three main considerations of a rigid body tracking system are solidity to occlusion, efficient computation, and accuracy. The method is inherently robust to occlusion. During identification, the rigid body tracking system can recognize individual light-emitting diodes (LEDs). One condition is to see a few light-emitting diodes (LEDs) across several frames. Posture reconstruction is also robust to occlusion. The rigid body tracking system is capable of verifying a prediction and refining the posture estimate of the virtual reality (VR) viewer by viewing a few light-emitting diodes (LEDs) per frame.

The precision requirements of head tracking in virtual reality (VR) are high. The rigid body tracking system provides the necessary stability (<0.1 millimeter and <0.1 degrees) through careful design of the placement of the

light-emitting diodes (LEDs) in the virtual reality (VR) viewer, and

reliable synchronization between the light-emitting diodes (LEDs), the inertial measurement unit (IMU) and the camera. Fusing all the information together is what ultimately allows for accurate prediction and consistent tracking.

An interesting property of position tracking is grounding. For example, the rigid body tracking system can now give meaning to looking ahead. It simply means looking towards the camera. In contrast to gyros, accelerometers and magnetometers, vision does not drift. Therefore, with position tracking, the rigid body tracking system has a reliable and simple solution to drift correction.

Position tracking opens up some interesting questions. The scale is a good example. Would a user expect a 1 to 1 correlation between movement in the real world and in the virtual world? Probably yes, because any mismatch can disrupt the wearer's vestibular system. But then again, there are some use cases where speed control based on head displacement would be useful - the further the user is from the center, the faster the user moves. Another interesting question is how to motivate the user to stay in the camera's field of view.

Another example is when a user leaves the field of view, the rigid body tracking system loses tracking of the · μι η ι π ·

DMA CURRENCY iN 3VST »t AL position, but still has orientation tracking thanks to inertial measurement unit (IMU). Every time the rigid body tracking system re-acquires position tracking, it goes back from 3D to 6D. What should the rigid body tracking system do after re-entering 6D? Two possible solutions are instant adjustment to the correct position as soon as vision is regained, and slowly interpolating to the correct position, but both solutions are disturbing in different ways.

Areas of interest

A rigid body tracking system that achieves high precision and robustness to occlusion due to a high plurality of modulated light emitting diodes (LEDs) arranged on the surface of the body, where the additional components of the system are: at least one stationary digital camera, which is external to the body being tracked; an inertial measurement unit, which is rigidly attached to the body being tracked; and the digital hardware that receives and sends the information between the components of the commercial metering unit, the camera, and the main CPU.

The rigid body tracking system, where the inertial measurement contains a three-axis gyroscope to measure angular velocity.

The rigid body tracking system, where the inertial measurement contains a three-axis accelerometer.

87 IMPI i »ΧτηντοΜίχΐίΛΝΟ

M THE MUG

INDUSTRIAL

The rigid body tracking system, where the inertial measurement can contain a three-axis magnetometer.

The rigid body tracking system, where the surface light-emitting diodes (LEDs) are arranged in a careful pattern that is not coplanar and provides sufficient spacing between neighboring light-emitting diodes (LEDs).

The rigid body tracking system, where the light emitting diodes (LEDs) are modulated in such a way that they can maintain one of two or more previously determined levels of infrared brightness for a desired time interval.

The rigid body tracking system, where the modulation of the light emitting diodes (LEDs) is controlled by the digital hardware component.

The rigid body tracking system, where the modulation of the light emitting diodes (LEDs) can be in amplitude, frequency, some combination, or by other means of signal coding.

The rigid body tracking system, where the light-emitting diodes (LEDs) can be in the visible light spectrum or in the infrared spectrum.

The rigid body tracking system, where the digital hardware component generates and records timestamps of less than one millisecond, for the times when the camera shutter is open or the inertial measurement unit provides a new measurement.

—— «Μ—» · MU— Μι <ΜΡΙ ^^

BWTTTUTO MEXICANO U

Dt THE CURRENCY JNXHJSTWAL · "

A method for rigid body tracking, in conjunction with a head-mounted display, I bought `` TOISITE 61 METHOD 06 '' head tracking: updating estimates of body orientation from velocity measurements. angular high frequency (at least 1,000 Hz) obtained by a gyroscope in the inertial measurement unit; updating the estimated body position from the images containing a uniquely identified subset of the light-emitting diodes (LEDs); improve computational efficiency and estimation precision by firmly integrating measurements from both the inertial measurement unit and one or more cameras; and providing estimates of the position and orientation of the body, with the additional ability of predicting future positions and orientations.

The method as mentioned above, where accelerometer measurements are used to compensate for dead reckoning errors of the tilt orientation.

The method as mentioned above, where the camera images, and possibly the magnetometer measurements, are used to compensate for dead reckoning errors from the yaw orientation.

The method as mentioned above, wherein low-level image processing is performed to extract the locations of the centers of the light-emitting diodes (LEDs) in the

ΒβΤΓΠΛΌ MEXICAN DE LA MbrHHUD industrial image with sub-pixel precision.

The method as mentioned above, wherein the modulation levels of the light-emitting diodes (LEDs) provide a digitally encoded identifier on the consecutive camera images, thus solving the problem of the identification of the light-emitting diodes (LED).

The method as mentioned above, wherein the position and orientation estimates from the images are calculated incrementally from frame to frame.

The method as mentioned above, wherein gyroscope and accelerometer measurements are used during the time interval between consecutive shutter openings.

The method as mentioned above, wherein the precise prediction estimates of the position and orientation are made by combining the estimate from the above frame with the accumulated measurements from the gyroscope and accelerometer.

The method as mentioned above, wherein a method iteratively disturbs the predictive estimate until the new position and orientation estimates optimize the error, which is the mismatch between the light-emitting diode (LED) center locations. ) expected in the image and their measured locations in the image.

Compendium

The above description of the description modalities is 90 ungodly rwrmromukaw

I HEARD THE FROUSDAD industrial Sta has submitted for illustration purposes; It is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Those skilled in the relevant art may appreciate that many modifications and variations are possible in light of the above description.

Some parts of this description describe the modalities of the disclosure in terms of algorithms and symbolic representations of the operations on the information. These descriptions and algorithmic representations are commonly used by experts in data processing techniques to convey the substance of their work effectively to other experts in this field. It is understood that these operations, although they are functionally, computationally or logically described, are implemented by computer programs or by equivalent electrical circuits, microcode, or the like. Additionally, it has also proved convenient, at times, to refer to these operation configurations as modules, without losing generality. The operations described and their associated modules can be incorporated into software, firmware, hardware, or any combination thereof.

Any of the steps, operations or processes described in this document can be carried out or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product that

91 IMPI • W1JVTO «OtieiNO k Oí LA PtWtW)

Sj »INDUSTRY * comprises a computer program code that contains a computer-readable medium, which can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

Modalities of the disclosure may also refer to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and / or may comprise a general purpose computing device selectively activated or reconfigured by a computer program stored in the computer. This computer program can be stored in a non-transitory, tangible, computer-readable storage medium, or any type of suitable means to store electronic instructions, which can be coupled to a bus bar of a computer system. Additionally, any computing system referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing power.

The modalities of the disclosure may also refer to a product that is produced by a computational process described in this document. This product can understand the information resulting from a computing process, where the information is stored in a non-transitory, tangible, computer-readable storage medium, and can include

ΊΜΡΙ ^^^

Βτττντο mixicaho εζ-λΉ '' A

M LA MOR t DAI> &

• MDUSTR1AL any form of a computer program product or other data combination that is described in this document.

Finally, the language used in the specification has been selected primarily to facilitate reading and instructional purposes, and may not have been selected to delimit or circumscribe the subject matter of the invention. Accordingly, the scope of the disclosure is not intended to be limited by this detailed description, but rather by any claims arising from an application based on this. Accordingly, the description of the modalities is intended to be illustrative, but not limiting, of the scope of the disclosure, which is set forth in the following claims.

MAID

ΜβΠΤνΓ *? MJtKlCAfO d i * M- mtno p ¿y

MOUSTWAL ^ <Λί5

Contents31

33 members in 11 offices

This recordMember, by publication datePriority claim1 more office is in the list below

Priority claims11

Priority claims
Document	Office	Kind	Date
61923895	United States of America	–	2014-01-06
201461923895	United States of America	P	2014-01-06
62088085	United States of America	–	2014-12-05
62088088	United States of America	–	2014-12-05
201462088088	United States of America	P	2014-12-05
201462088085	United States of America	P	2014-12-05
14589755	United States of America	–	2015-01-05
14589774	United States of America	–	2015-01-05
201514589774	United States of America	A	2015-01-05
201514589755	United States of America	A	2015-01-05
2015010344	United States of America	W	2015-01-06

Members33

Family members, oldest first
Document	Office	Kind
CA2930773A1	Canada	A1	2015-07-09
US2015193949A1	United States of America	A1	2015-07-09
US2015193983A1	United States of America	A1	2015-07-09
WO2015103621A1	World Intellectual Property Organization (WIPO)	A1	2015-07-09
EP3017591A1	European Patent Office (EPO)	A1	2016-05-11
AU2015203932A1	Australia	A1	2016-06-09
CN105850113A	China	A	2016-08-10
KR20160105796A	Republic of Korea	A	2016-09-07
MX2016008908A	Mexico	A	2016-09-16
US9524580B2	United States of America	B2	2016-12-20
JP2017503418A	Japan	A	2017-01-26
US2017053454A1	United States of America	A1	2017-02-23
US9600925B2	United States of America	B2	2017-03-21
JP6110573B2	Japan	B2	2017-04-05
AU2015203932B2	Australia	B2	2017-04-06
US2017147066A1	United States of America	A1	2017-05-25
MX348608BThis record	Mexico	B	2017-06-21
CA2930773C	Canada	C	2017-07-11
KR20170086707A	Republic of Korea	A	2017-07-26
KR101762297B1	Republic of Korea	B1	2017-07-28
BR112016013563A2	Brazil	A2	2017-08-08
JP2017142813A	Japan	A	2017-08-17
IL245813A	Israel	A	2017-08-31
EP3017591A4	European Patent Office (EPO)	A4	2017-09-27
US9779540B2	United States of America	B2	2017-10-03
CN105850113B	China	B	2018-02-13
CN108107592A	China	A	2018-06-01
US10001834B2	United States of America	B2	2018-06-19
JP6381711B2	Japan	B2	2018-08-29
EP3017591B1	European Patent Office (EPO)	B1	2018-12-12
CN108107592B	China	B	2019-11-05
BR112016013563A8	Brazil	A8	2020-05-19
KR102121994B1	Republic of Korea	B1	2020-06-11

Calibration of virtual reality systems.

Abstract

Term

CPC: G06T15 / 205; G02B27 / 017; G02B2027 / 014; G02B2027 / 0138;

G02B2027 / 0178; G02B2027 / 0187; G06T7 / 0018; G06T7 / 0046; G06T19 / 006;

G06T2207 / 30204

STEVE LAVALLE; MICHAEL ANTONOV

REQUEST

THE DIVISIONAL DIRECTOR OF PATENTS

NAHANNY CANAL REYES

CALIBRATION OF VIRTUAL REALITY SYSTEMS

BACKGROUND OF THE INVENTION

BRIEF DESCRIPTION OF THE INVENTION

BRIEF DESCRIPTION OF THE DRAWINGS

DETAILED DESCRIPTION OF THE INVENTION

IWTTTVTO MEXICANO J. '

MAID

MAID

MEXICAN INSTITUTE

DC THE HLOPIETY

MAID

WICKED

MAID

MAID

HE0

VISION-BASED POSITION TRACKING

MAID

OF THE HIOFIEDAO

RECONSTRUCTION OF POSTURE

MAID

M THE MUG

INDUSTRIAL

BWTTTUTO MEXICANO U

MAID

Priority claims11

Members33

Events

Numbers

Titles2

Classification