MX2016008908A

Calibration of virtual reality systems.

Abstract

A virtual reality (VR) console receives slow calibration data from an imaging device and fast calibration data from an inertial measurement unit on a VR headset including a front and a rear rigid body. The slow calibration data includes an image where only the locators on the rear rigid body are visible. An observed position is determined from the slow calibration data and a predicted position is determined from the fast calibration data. If a difference between the observed position and the predicted position is greater than a threshold value, the predicted position is adjusted by a temporary offset until the difference is less than the threshold value. The temporary offset is removed by re-calibrating the rear rigid body to the front rigid body once locators on both the front and rear rigid body are visible in an image in the slow calibration data.

MX2016008908A, drawing sheet 1 — Sheet 1 of 10

Term

8.3 yearsleft in the term

Expires 6 January 2035.

40 claims: 6 independent, 34 dependent

1
REIVINDICACIONES 1. Un método que comprende:recibir los datos de calibración lenta desde un dispositivo de procesamiento de imágenes, incluyendo los datos de calibración lenta una serie de imágenes de un visor de realidad virtual (VR) que incluye un cuerpo rígido frontal y un cuerpo rígido trasero, en donde cada uno tiene uno o más localizadores, incluyendo la serie de imágenes una imagen con solamente los localizadores del cuerpo rígido trasero visibles y asociados con un valor de tiempo de la imagen;recibir, desde una unidad de medición inercial (IMU) dentro del cuerpo rígido frontal, los datos de calibración rápida que comprenden las posiciones estimadas intermedias de un punto de referencia del cuerpo rígido frontal;determinar una posición observada del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando los datos de calibración lenta;determinar una posición predicha del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando los datos de calibración rápida y un vector de posición que describe un desfasamiento calibrado entre el cuerpo rígido frontal y el cuerpo rígido trasero;en respuesta a la determinación de que una diferencia entre la posición observada del cuerpo rígido trasero y la posición predicha del cuerpo rígido trasero es mayor que un valor umbral, ajustar el vector de posición por un valor de desfasamiento, de tal manera que la diferencia entre la posición observada y la posición predicha es menor que el valor umbral;y determinar una posición predicha subsiguiente del cuerpo rígido trasero para una imagen a partir de la serie de imágenes asociadas con un valor de tiempo de imagen subsiguiente que se presenta después del valor de tiempo de imagen basándose en los datos de calibración rápida y el vector de posición ajustado.
2
El método de acuerdo con la reivindicación 1, el cual comprende además:en respuesta a la determinación de que la diferencia entre la posición observada del cuerpo rígido trasero y la posición predicha del cuerpo rígido trasero es mayor que el valor umbral, volver a calibrar la posición relativa del cuerpo rígido trasero con el cuerpo rígido frontal para eliminar el desfasamiento de las estimaciones de posición predichas del cuerpo rígido trasero basándose, cuando menos en parte, en una imagen a partir de la serie de imágenes en los datos de calibración lenta, que incluye un número umbral frontal de localizadores del cuerpo rígido frontal, e incluye un número umbral trasero de localizadores en el cuerpo rígido trasero.
3
El método de acuerdo con la reivindicación 2, en donde la re-calibración de la posición relativa del cuerpo rígido trasero con el cuerpo rígido frontal para eliminar el desfasamiento temporal de las estimaciones de posición predichas del cuerpo rígido trasero comprende además:identificar los localizadores de modelo que corresponden a los localizadores incluidos en por lo menos una imagen basándose en un modelo de visor asociado con el visor de realidad virtual (VR);generar las posiciones estimadas para cada uno de los uno o más localizadores incluidos en cuando menos una imagen utilizando el modelo de visor;ajustar uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de los localizadores en el cuerpo rígido frontal incluidos en cuando menos una imagen y las posiciones asociadas con sus localizadores de modelo correspondientes, sea menor que un valor umbral;generar las posiciones calibradas del punto de referencia basándose, cuando menos en parte, en las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR), e incluyendo en la por lo menos una imagen a partir de los datos de calibración lenta, una posición calibrada asociada con una imagen a partir de los datos de calibración lenta;determinar una posición del cuerpo rígido trasero en relación con el punto de referencia basándose, cuando menos en parte, en las posiciones estimadas de uno o más localizadores incluidos en cuando menos una imagen a partir de los datos de calibración lenta e incluidos en el cuerpo rígido trasero;determinar una o más posiciones predichas del punto de referencia basándose, cuando menos en parte, en las posiciones calibradas del punto de referencia;una posición predicha asociada con un tiempo entre las imágenes subsiguientes a partir de los datos de calibración lenta;y ajustar uno o más de los parámetros de calibración, de tal manera que las posiciones estimadas intermedias del punto de referencia estén dentro de una distancia umbral de las posiciones predichas determinadas del punto de referencia.
4
El método de acuerdo con la reivindicación 3, en donde la determinación de la posición del cuerpo rígido trasero en relación con el punto de referencia comprende:determinar un punto de referencia trasero basándose en las posiciones estimadas de uno o más localizadores incluidos en cuando menos una imagen a partir de los datos de calibración lenta e incluidos en el cuerpo rígido trasero;y determinar una posición del punto de referencia trasero en relación con el punto de referencia.
5
El método de acuerdo con la reivindicación 3, en donde la determinación de la posición del cuerpo rígido trasero en relación con el punto de referencia comprende además:determinar las posiciones de los uno o más localizadores incluidos en cuando menos una imagen a partir de los datos de calibración lenta e incluidos en el cuerpo rígido trasero en relación con el punto de referencia basándose, cuando menos en parte, en las posiciones estimadas de los uno o más localizadores incluidos en cuando menos una imagen a partir de los datos de calibración lenta e incluidos en el cuerpo rígido trasero.
6
El método de acuerdo con la reivindicación 3, el cual comprende además:determinar que los datos de calibración lenta no incluyen cuando menos una imagen que incluya un localizador del cuerpo rígido frontal;estimar las posiciones de uno o más localizadores incluidos en cuando menos una imagen a partir de los datos de calibración lenta e incluidos en el cuerpo rígido trasero en relación con una posición estimada intermedia del punto de referencia, a partir de los datos de calibración rápida;y determinar que una diferencia entre las posiciones de los localizadores de modelo correspondientes a los uno o más localizadores incluidos en cuando menos una imagen a partir de los datos de calibración lenta, basándose en un modelo de visor asociado con las posiciones del visor de realidad virtual (VR) y las posiciones estimadas de los uno o más localizadores incluidos en por lo menos una imagen a partir de los datos de calibración lenta e incluidos en el cuerpo rígido trasero, supera el valor umbral.
7
El método de acuerdo con la reivindicación 3, en donde el ajuste de uno o más parámetros de calibración para ajustar las posiciones estimadas de tal manera que la distancia relativa entre las posiciones estimadas ajustadas de los localizadores en el cuerpo rígido frontal y e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones asociadas con sus localizadores de modelo correspondientes, sea menor que el valor umbral, comprende:ajustar un parámetro de calibración para la unidad de medición inercial (IMU);y ajustar las posiciones de uno o más localizadores de modelo basándose en parte en el ajuste del parámetro de calibración.
8
El método de acuerdo con la reivindicación 7, en donde el ajuste del parámetro de calibración para la unidad de medición inercial (IMU) comprende:establecer una posición inicial de la unidad de medición inercial (IMU) hasta una posición calibrada del punto de referencia asociado con una imagen a partir de los datos de calibración lenta en un tiempo después de la inicialización de la unidad de medición inercial (IMU), de tal manera que la unidad de medición inercial (IMU) determine los datos de calibración rápida en relación con la posición calibrada del punto de referencia.
9
El método de acuerdo con la reivindicación 7, en donde el parámetro de calibración ajustado se selecciona a partir de un grupo que consiste en:una frecuencia de muestreo de una o más de las señales de medición producidas por un acelerómetro, una frecuencia de muestreo de una o más de las señales de medición producidas por un giroscopio, una velocidad de salida de los datos de calibración rápida, un comando para proporcionar energía a la unidad de medición inercial (IMU), un comando para actualizar una posición inicial hasta una posición calibrada del punto de referencia, y cualquier combinación de los mismos.
10
El método de acuerdo con la reivindicación 3, en donde la generación de las posiciones estimadas para cada uno de los uno o más localizadores incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando el modelo de visor, comprende:utilizar una matriz de proyección para traducir las posiciones de los localizadores de modelo hasta las posiciones estimadas de los uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta.
11
El método de acuerdo con la reivindicación 10, en donde el ajuste de uno o más parámetros de calibración para ajustar las posiciones estimadas de tal manera que la distancia relativa entre las posiciones estimadas ajustadas de los localizadores en el cuerpo rígido frontal e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones asociadas con sus localizadores de modelo correspondientes sea menor que el valor umbral, comprende:ajustar un parámetro de calibración que ajusta la traducción de las posiciones de los localizadores de modelo hasta 100 las posiciones estimadas de uno o más de los localizadores en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta, mediante la matriz de proyección.
12
El método de acuerdo con la reivindicación 11, en donde el parámetro de calibración se selecciona a partir de un grupo que consiste en:longitud focal, enfoque, velocidad de cuadros, escala de sensibilidad ISO, velocidad de obturación, abertura, orientación de la cámara, desfasamiento de un sensor de la proyección con respecto al centro del lente, parámetros de distorsión del lente, temperatura del sensor, o alguna combinación de los mismos.
13
El método de acuerdo con la reivindicación 3, en donde el ajuste de uno o más de los parámetros de calibración de tal manera que las posiciones estimadas intermedias del punto de referencia estén dentro de la distancia umbral de las posiciones predichas determinadas del punto de referencia, comprende:determinar una función de predicción que predice la posición del punto de referencia en un tiempo particular utilizando las posiciones calibradas;determinar las distancias entre una o más de las posiciones estimadas intermedias del punto de referencia y las posiciones predichas del punto de referencia determinadas utilizando la función de predicción;y ajustar un parámetro de calibración de tal manera que las distancias determinadas sean menores que la distancia umbral. 101
14
Un método que comprende:recibir los datos de calibración lenta desde un dispositivo de procesamiento de imágenes, incluyendo los datos de calibración lenta una imagen de una porción de un visor de realidad virtual (VR) en un valor de tiempo de la imagen, y el visor de realidad virtual (VR) incluye un cuerpo rígido frontal y un cuerpo rígido trasero, en donde cada uno tiene uno o más localizadores, y solamente los localizadores en el cuerpo rígido trasero son visibles en la imagen;recibir, desde una unidad de medición inercial (IMU) dentro del cuerpo rígido frontal, los datos de calibración rápida que comprenden una posición estimada intermedia de un punto de referencia del cuerpo rígido frontal en el valor de tiempo de la imagen;determinar una posición observada del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando los datos de calibración lenta;determinar una posición predicha del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando los datos de calibración rápida y un vector de posición que describe un desfasamiento calibrado entre el cuerpo rígido frontal y el cuerpo rígido trasero;y en respuesta a la determinación de que una diferencia entre la posición observada del cuerpo rígido trasero y la posición predicha del cuerpo rígido trasero es mayor que un valor umbral, ajustar la posición predicha del cuerpo rígido trasero por un valor de 102 desfasamiento, de tal manera que la diferencia entre la posición observada y la posición predicha sea menor que el valor umbral.
15
El método de acuerdo con la reivindicación 14, en donde el ajuste de la posición predicha del cuerpo rígido trasero por un valor de desfasamiento, de tal manera que la diferencia entre la posición observada y la posición predicha sea menor que el valor umbral, comprende:ajustar el vector de posición por el valor de desfasamiento.
16
El método de acuerdo con la reivindicación 14, en donde el ajuste de la posición predicha del cuerpo rígido trasero por un valor de desfasamiento, de tal manera que la diferencia entre la posición observada y la posición predicha sea menor que el valor umbral, comprende:instruir a la unidad de medición inercial (IMU) para desfasar una o más posiciones estimadas intermedias del punto de referencia por el valor de desfasamiento.
17
El método de acuerdo con la reivindicación 14, en donde la determinación de una posición predicha del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando los datos de calibración rápida y un vector de posición que describe un desfasamiento calibrado entre el cuerpo rígido frontal y el cuerpo rígido trasero, comprende:identificar la posición predicha del cuerpo rígido trasero como una posición de un punto de referencia trasero en el cuerpo 103 rígido trasero en relación con la posición estimada intermedia del punto de referencia por el vector de posición.
18
El método de acuerdo con la reivindicación 14, el cual comprende además:recibir los datos de calibración lenta adicionales desde el dispositivo de procesamiento de imágenes, incluyendo los datos de calibración lenta una imagen subsiguiente en un valor de tiempo de imagen subsiguiente, y solamente son visibles los localizadores en el cuerpo rígido trasero en la imagen subsiguiente;recibir desde la unidad de medición inercial (IMU), datos de calibración rápida adicionales que comprenden una posición estimada intermedia subsiguiente del punto de referencia del cuerpo rígido frontal en el valor de tiempo de imagen subsiguiente;determinar una posición observada subsiguiente del cuerpo rígido trasero para el valor de tiempo de imagen subsiguiente utilizando los datos de calibración lenta;y determinar una posición predicha subsiguiente del cuerpo rígido trasero para el valor de tiempo de imagen subsiguiente utilizando los datos de calibración rápida y el vector de posición ajustado.
19
El método de acuerdo con la reivindicación 14, el cual comprende además:volver a calibrar el cuerpo rígido frontal con el cuerpo rígido trasero una vez que los datos de calibración lenta incluyan una imagen que comprenda los localizadores en el cuerpo rígido frontal y 104 en el cuerpo rígido trasero.
20
Un sistema que comprende:un visor de realidad virtual (VR) que incluye un cuerpo rígido frontal y un cuerpo rígido trasero, incluyendo el cuerpo rígido frontal una unidad de medición inercial (IMU) configurada para producir los datos de calibración rápida que comprenden las posiciones estimadas intermedias de un punto de referencia dentro del cuerpo rígido frontal;un dispositivo de procesamiento de imágenes configurado para producir los datos de calibración lenta, el cual incluye una serie de imágenes del visor de realidad virtual (VR), y en una imagen a partir de la serie de imágenes, los localizadores en el cuerpo rígido trasero son los únicos localizadores visibles en la imagen, y la imagen se asocia con un valor de tiempo de imagen particular, y una consola de realidad virtual (VR) que comprende: un procesador;y una memoria acoplada al procesador y que incluye instrucciones que, cuando son ejecutadas por el procesador, hacen que el procesador: reciba los datos de calibración lenta desde el dispositivo de procesamiento de imágenes, reciba los datos de calibración rápida desde la unidad de medición inercial (IMU), determine una posición observada del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando 105 los datos de calibración lenta, determine una posición predicha del cuerpo rígido trasero para el valor de tiempo de imagen particular utilizando los datos de calibración rápida y un vector de posición que describe un desfasamiento calibrado entre el cuerpo rígido frontal y el cuerpo rígido trasero;determine que una diferencia entre la posición observada y la posición predicha es mayor que un valor umbral;en respuesta a una determinación de que una diferencia entre la posición observada del cuerpo rígido trasero y la posición predicha del cuerpo rígido trasero es mayor que un valor umbral, ajustar el vector de posición por un valor de desfasamiento, de tal manera que la diferencia entre la posición observada y la posición predicha sea menor que el valor umbral;y determine una posición predicha subsiguiente del cuerpo rígido trasero para una imagen a partir de la serie de imágenes asociadas con un valor de tiempo de imagen subsiguiente que se produzca después del valor de tiempo de imagen, basándose en los datos de calibración rápida y en el vector de posición ajustado.
21
Un método que comprende:recibir, desde un dispositivo de procesamiento de imágenes, los datos de calibración lenta que incluyen una serie de imágenes que muestran porciones de uno o más localizadores en un visor de realidad virtual (VR), estando cada imagen separada de una 106 imagen subsiguiente en la serie por un valor de tiempo de la imagen;recibir, a partir de una unidad de medición inercial (IMU) en el visor de realidad virtual (VR), los datos de calibración rápida que comprenden una o más posiciones estimadas intermedias de un punto de referencia en el visor de realidad virtual (VR);identificar los localizadores de modelo en donde cada uno corresponde a un localizador en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando un modelo de visor almacenado asociado con el visor de realidad virtual (VR);generar las posiciones estimadas de uno o más de los localizadores en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando el modelo de visor;ajustar uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de uno o más de los localizadores en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes, sea menor que un valor umbral;generar las posiciones calibradas del punto de referencia basándose, cuando menos en parte, en las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los 107 datos de calibración lenta, y una posición calibrada asociada con una imagen a partir de los datos de calibración lenta;determinar una o más posiciones predichas del punto de referencia basándose, cuando menos en parte, en las posiciones calibradas del punto de referencia, y una posición predicha asociada con un tiempo entre las imágenes subsiguientes de los datos de calibración lenta;y ajustar uno o más de los parámetros de calibración, de tal manera que las posiciones estimadas intermedias del punto de referencia estén dentro de una distancia umbral de las posiciones predichas determinadas del punto de referencia.
22
El método de acuerdo con la reivindicación 21, en donde la generación de las posiciones calibradas del punto de referencia basándose, cuando menos en parte, en las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta, comprende:determinar si los datos de calibración lenta incluyen imágenes que tengan cuando menos un número umbral de localizadores en una pluralidad de lados del visor de realidad virtual (VR);y comunicar un mensaje al visor de realidad virtual (VR) para su presentación, incluyendo el mensaje una o más instrucciones para cambiar la posición del visor de realidad virtual (VR) en relación con el dispositivo de procesamiento de imágenes. 108
23
El método de acuerdo con la reivindicación 22, en donde los lados del visor de realidad virtual (VR) se seleccionan a partir de un grupo que consiste en:un lado superior, un lado inferior, un lado derecho, un lado izquierdo, un lado frontal, y cualquier combinación de los mismos.
24
El método de acuerdo con la reivindicación 21, en donde el ajuste de uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes sean menores que un valor umbral, comprende:ajustar un parámetro de calibración para la unidad de medición inercial (IMU);y ajustar las posiciones de uno o más localizadores de modelo basándose en parte en el ajuste del parámetro de calibración.
25
El método de acuerdo con la reivindicación 24, en donde el ajuste del parámetro de calibración para la unidad de medición inercial (IMU) comprende:establecer una posición inicial de la unidad de medición inercial (IMU) con una posición calibrada del punto de referencia asociado con una imagen a partir de los datos de calibración lenta en un tiempo después de la inicialización de la unidad de medición 109 inercial (IMU), de tal manera que la unidad de medición inercial (IMU) determine los datos de calibración rápida en relación con la posición calibrada del punto de referencia.
26
El método de acuerdo con la reivindicación 24, en donde el parámetro de calibración para la unidad de medición inercial (IMU) se selecciona a partir de un grupo que consiste en:una frecuencia de muestreo de una o más de las señales de medición producidas por un acelerómetro, una frecuencia de muestreo de una o más de las señales de medición producidas por un giroscopio, una velocidad de salida de los datos de calibración rápida, un comando para proporcionar energía a la unidad de medición inercial (IMU), un comando para actualizar una posición inicial con una posición calibrada del punto de referencia, y cualquier combinación de los mismos.
27
El método de acuerdo con la reivindicación 21, en donde la generación de las posiciones estimadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando el modelo de visor, comprende:utilizar una matriz de proyección para traducir las posiciones de los localizadores de modelo hasta las posiciones estimadas de los uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta.
28
El método de acuerdo con la reivindicación 27, en donde 110 el ajuste de uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes sea menor que un valor umbral, comprende:ajustar un parámetro de calibración que ajusta la traducción de las posiciones de los localizadores de modelo hasta las posiciones estimadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta mediante la matriz de proyección.
29
El método de acuerdo con la reivindicación 28, en donde el parámetro de calibración se selecciona a partir de un grupo que consiste en:distancia focal, enfoque, velocidad de cuadros, escala de sensibilidad ISO, velocidad de obturación, abertura, orientación de la cámara, desfasamiento de un sensor de la proyección con respecto al centro del lente, parámetros de distorsión del lente, temperatura del sensor, o alguna combinación de los mismos.
30
El método de acuerdo con la reivindicación 21, en donde el ajuste de uno o más de los parámetros de calibración, de tal manera que las posiciones estimadas intermedias del punto de referencia estén dentro de una distancia umbral de las posiciones predichas determinadas del punto de referencia, comprende:111 determinar una función de predicción que predice la posición del punto de referencia en un tiempo particular utilizando las posiciones calibradas;determinar las distancias entre una o más de las posiciones estimadas intermedias del punto de referencia y las posiciones predichas del punto de referencia determinadas utilizando la función de predicción;y ajustar un parámetro de calibración, de tal manera que las distancias determinadas sean menores que la distancia umbral.
31
Un método que comprende:recibir, desde un dispositivo de procesamiento de imágenes, los datos de calibración lenta, incluyendo una serie de imágenes en valores de tiempo particulares que muestran porciones de uno o más localizadores observados a partir de una pluralidad de localizadores en un visor de realidad virtual (VR);recibir, desde una unidad de medición inercial (IMU) en el visor de realidad virtual (VR), los datos de calibración rápida que comprenden la información sobre la posición de un punto de referencia en el visor de realidad virtual (VR) en los valores de tiempo particulares;extraer la información de los localizadores a partir de las imágenes en los datos de calibración lenta, la información de los localizadores extraída a partir de una imagen que describe las posiciones de los localizadores observados unos en relación con los otros en la imagen;112 comparar la información de los localizadores con un modelo de visor para identificar los localizadores de modelo que corresponden a cada uno de los uno o más localizadores observados;generar las posiciones estimadas de los localizadores observados utilizando el modelo de visor;estimar las posiciones del punto de referencia para la serie de imágenes utilizando las posiciones estimadas de los localizadores observados;ajustar uno o más parámetros de calibración para, de una manera simultánea: ajustar las posiciones estimadas de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de los localizadores observados y las posiciones de sus localizadores de modelo correspondientes sea menor que un valor umbral;y ajustar las posiciones estimadas del punto de referencia de tal manera que una distancia relativa entre las posiciones estimadas para el punto de referencia y las posiciones de un punto de referencia del modelo, determinadas a partir de los localizadores de modelo, sea menor que el valor umbral.
32
El método de acuerdo con la reivindicación 31, en donde la generación de las posiciones estimadas de los localizadores observados utilizando el modelo de visor, comprende:utilizar una matriz de proyección para traducir las posiciones de los localizadores de modelo hasta las posiciones 113 estimadas de los uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta.
33
El método de acuerdo con la reivindicación 32, en donde el ajuste de uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de los localizadores observados y las posiciones de sus localizadores de modelo correspondientes, sea menor que un valor umbral, comprende:ajustar un parámetro de calibración que ajusta la traducción de las posiciones de los localizadores de modelo hasta las posiciones estimadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta mediante la matriz de proyección.
34
El método de acuerdo con la reivindicación 31, en donde un parámetro de calibración se selecciona a partir de un grupo que consiste en:una frecuencia de muestreo de una o más de las señales de medición producidas por un acelerómetro, una frecuencia de muestreo de una o más de las señales de medición producidas por un giroscopio, una velocidad de salida de los datos de calibración rápida, un comando para proporcionar energía a la unidad de medición ¡nercial (IMU), un comando para actualizar una posición inicial hasta una posición calibrada del punto de referencia, y cualquier combinación de los mismos. 114
35
Un sistema que comprende:un visor de realidad virtual (VR) que incluye una pluralidad de localizadores y una unidad de medición inercial (IMU) configurada para la producción de los datos de calibración rápida que comprenden una o más posiciones estimadas intermedias de un punto de referencia en el visor de realidad virtual (VR), estando cada posición estimada intermedia separada de una posición estimada intermedia subsiguiente por un valor de tiempo de posición;un dispositivo de procesamiento de imágenes configurado para la producción de los datos de calibración lenta, que incluye una serie de imágenes que muestran porciones de los localizadores observados de la pluralidad de localizadores, en el visor de realidad virtual (VR), estando cada imagen separada de una imagen subsiguiente en la serie por un valor de tiempo de imagen que es mayor que el valor de tiempo de posición;una consola de realidad virtual (VR) que comprende: un procesador;y una memoria acoplada al procesador y que incluye instrucciones que, cuando son ejecutadas por el procesador, hacen que el procesador: reciba los datos de calibración lenta desde el dispositivo de procesamiento de imágenes que recibe los datos de calibración rápida desde el visor de realidad virtual (VR);identifique los localizadores de modelo, cada uno correspondiente a un localizador en el visor de realidad virtual 115 (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando un modelo de visor almacenado asociado con el visor de realidad virtual (VR);genere las posiciones estimadas de uno o más de los localizadores en el visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta utilizando el modelo de visor;ajuste uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes, sea menor que un valor umbral;genere las posiciones calibradas del punto de referencia basándose, cuando menos en parte, en las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta, y una posición calibrada asociada con una imagen a partir de los datos de calibración lenta;determine una o más posiciones predichas del punto de referencia basándose, cuando menos en parte, en las posiciones calibradas del punto de referencia, y una posición predicha asociada con un tiempo entre las imágenes subsiguientes a partir de los datos de calibración lenta;y 116 ajuste uno o más de los parámetros de calibración, de tal manera que la posiciones estimadas intermedias del punto de referencia estén dentro de una distancia umbral de las posiciones predichas determinadas del punto de referencia.
36
El sistema de acuerdo con la reivindicación 35, en donde un localizador comprende diodos emisores de luz infrarroja, y el dispositivo de procesamiento de imágenes está configurado para detectar la luz infrarroja.
37
El sistema de acuerdo con la reivindicación 34, en donde el ajuste de uno o más parámetros de calibración para ajustar las posiciones estimadas, de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes sea menor que un valor umbral, comprende:ajustar un parámetro de calibración para la unidad de medición inercial (IMU);y ajustar las posiciones de uno o más localizadores de modelo basándose en parte en el ajuste del parámetro de calibración.
38
El sistema de acuerdo con la reivindicación 37, en donde el ajuste del parámetro de calibración para la unidad de medición inercial (IMU) comprende:establecer una posición inicial de la unidad de medición 117 inercial (IMU) hasta una posición calibrada del punto de referencia asociado con una imagen a partir de los datos de calibración lenta en un tiempo después de la inicialización de la unidad de medición inercial (IMU), de tal manera que la unidad de medición inercial (IMU) determina los datos de calibración rápida en relación con la posición calibrada del punto de referencia.
39
El sistema de acuerdo con la reivindicación 35, en donde el ajuste de uno o más parámetros de calibración para ajustar las posiciones estimadas de tal manera que una distancia relativa entre las posiciones estimadas ajustadas de uno o más de los localizadores del visor de realidad virtual (VR) e incluidos en cuando menos una imagen a partir de los datos de calibración lenta y las posiciones de sus localizadores de modelo correspondientes sea menor que un valor umbral, comprende:ajustar un parámetro de calibración para el dispositivo de procesamiento de imágenes;y ajustar las posiciones de uno o más localizadores de modelo basándose en parte en el ajuste del parámetro de calibración.
40
El sistema de acuerdo con la reivindicación 34, en donde el ajuste de uno o más de los parámetros de calibración de tal manera que las posiciones estimadas intermedias del punto de referencia estén dentro de la distancia umbral de las posiciones predichas determinadas del punto de referencia, comprende:determinar una función de predicción que predice la 118 posición del punto de referencia en un tiempo particular utilizando las posiciones calibradas;determinar las distancias entre una o más de las posiciones estimadas intermedias del punto de referencia y las posiciones predichas del punto de referencia determinadas utilizando la función de predicción;y ajustar un parámetro de calibración de tal manera que las distancias determinadas sean inferiores a la distancia umbral. 119

Independent claims40

184 paragraphs in 6 sections, as filed

(54) Title: CALIBRATION OF VIRTUAL REALITY SYSTEMS.

(57) Summary

A virtual reality (VR) console receives slow calibration data from an image processing device, and fast calibration data from an inertial measurement unit in a virtual reality (VR) viewer that includes a rigid front body and a rear rigid body. Slow calibration data includes an image where only locators are visible on the rigid rear body. An observed position is determined from the slow calibration data, and a predicted position is determined from the rapid calibration data. If a difference between the observed position and the predicted position is greater than a threshold value, the predicted position is adjusted by a time shift until the difference is less than the threshold value. The time lag is removed by recalibrating the rear rigid body with the front rigid body once the locators are visible on both the front rigid body and the rear rigid body in one image in the slow calibration data.

(57) Abstract

A virtual reality (VR) consolé receives slow calibration data from an imaging device and fast calibration data from an inertial measurement unit on a VR headset including a front and a rear rigid body. The slow calibration data includes an image where only the locators on the rear rigid body are visible. An observed position is determined from the slow calibration data and a predicted position is determined from the fast calibration data. If a difference between the observed position and the predicted position is greater than a threshold valué, the predicted position is adjusted by a temporary offset until the difference is less than the threshold valué. The temporary offset is removed by re-calibrating the rear rigid body to the front rigid body once locators on both the front and rear rigid body are visible in an image in the slow calibration data.

CALIBRATION OF VIRTUAL REALITY SYSTEMS

BACKGROUND OF THE INVENTION

The present description refers generally to calibration systems, and more specifically refers to the calibration of virtual reality systems.

Tracking motion is an old problem that has been addressed on numerous devices. Examples include global positioning systems, aircraft radar systems, robotic systems, and home entertainment systems. In the latter case, existing devices track the movement of game controllers or people interacting with the game.

Several important characteristics determine the types of detection and computing hardware that are most appropriate: 1) the size of the rigid body, 2) its volume of space over which the rigid body can move, 3) the limits on the maximum rates of body speed and acceleration, 4) the predictability of the rigid body.

In the motion tracking system, there is typically a requirement to track the movement of devices that keep in contact with the human body while in a relatively small space, typically while sitting indoors, but this is not necessary. . In a more particular way, the intention is to track head movement while the user directs a head-mounted display, for the purposes of virtual reality and augmented reality. This results in extremely strict requirements on the operation of the system. For example, for virtual reality, position and orientation errors cause a bad experience because the user does not feel truly immersed. Furthermore, there may be a mismatch between signals provided to the brain by the human vestibular system and signals provided to the brain by the human vision system during screen viewing.

This example use case involves limitations on the volume of space over which movement can occur. This further limits speed and accelerations to those induced by human movement; however, it is also crucial that the movement is generally unpredictable. Furthermore, it is difficult to model the physics that governs movement due to the complexity of the movement of the human body and its interaction with other rigid bodies.

Virtual reality (VR) devices include components to determine the position and movement of a viewer worn by a user. These components must be calibrated at different times, initially due to manufacturing tolerances and, subsequently, due to normal use of the system. Operation of an incorrectly calibrated virtual reality (VR) device may result in improper tracking of the position or movement of the viewfinder, causing dissonance between the user's movement and the media presented to the user through the viewfinder. Furthermore, one or more of the components that determine the position and movement of the viewfinder may lose calibration over time or with use. For example, temperature changes or vibrations can cause a viewfinder motion imaging camera to lose calibration.

BRIEF DESCRIPTION OF THE INVENTION

A virtual reality (VR) viewer of a virtual reality (VR) system includes a rigid front body and a rigid rear body, both of which are coupled together in a non-rigid manner. For example, the rigid front body is attached to the rigid rear body by an elastic band, so the virtual reality (VR) system continues to detect the movement of an entity wearing the virtual reality (VR) viewer when the rigid body Front is directed away from an image processing device included in the virtual reality (VR) system. Both the rigid front body and the rigid rear body include locators for tracking the position of the virtual reality (VR) viewer. A locator is an object located at a specific position in the virtual reality (VR) viewer in relation to one or more components, such as another locator, of the virtual reality (VR) viewer and in relation to a reference point in the virtual reality viewer (VR). Because the relationship between the front and rear rigid bodies is not necessarily fixed, the virtual reality (VR) system may lose track of the position of the front rigid body relative to the rear rigid body, causing the reality system Virtual (VR) is recalibrated to re-acquire virtual reality viewer (VR) tracking. In some modalities, the virtual reality (VR) console determines the subsequent predicted positions of the rear rigid body during the times after the particular image time value of the image in the slow calibration data including only the rear rigid body locators using the calibration data fast and the position vector adjusted by the offset value until re-calibration can occur between the rigid front body and the rigid body rear.

The virtual reality (VR) system recalibrates itself when tracking the position of the front rigid body or the position of the rear rigid body is lost. For example, the virtual reality (VR) system determines when to re-calibrate based on a measured difference between the estimated positions of the locators on the rear body and the estimated intermediate positions of a reference point on the front rigid body determined by a inertial measurement unit (IMU) within the first rigid body based on data from one or more position sensors (eg, accelerometers, gyros) included in the first rigid body. An intermediate estimated reference point position is a position determined from the quick calibration data and may be associated with a time associated with an image, or with a time between times associated with an image and a subsequent image from slow calibration data.

The components of a virtual reality (VR) system are calibrated to keep track of a virtual reality (VR) viewer associated with the virtual reality (VR) system. The virtual reality (VR) system uses the slow calibration data received from an image processing device and the fast calibration data received from an inertial measurement unit (IMU) included in the virtual reality (VR) viewer to calibration. In some embodiments, the components of the virtual reality (VR) system can be calibrated by initially applying one or more default parameters to the components. Based on the default parameters, the virtual reality (VR) system tracks the movement of the virtual reality (VR) viewer by identifying the positions associated with one or more locators included in the virtual reality (VR) viewer. A locator is an object located at a specific position in the virtual reality (VR) viewer in relation to one or more components, such as another locator, of the virtual reality (VR) viewer, and in relation to a reference point in the virtual reality (VR) viewer. In some modalities, the virtual reality (VR) viewer includes two rigid bodies that are coupled in a non-rigid way to each other, with locators included in each of the rigid bodies to track the position and orientation of the user's head . The virtual reality (VR) system adjusts one or more calibration parameters until the differences between an estimated position of one or more locators differs from an observed position of the one or more locators by less than a threshold value.

In some embodiments, the virtual reality (VR) system includes a virtual reality (VR) console that receives slow calibration data that includes a series of images showing a portion of a plurality of locators in the virtual reality (VR) viewer ) from an image processing device. Each image is separated from a subsequent image in the series by an image time value.

Additionally, the virtual reality (VR) console receives quick calibration data comprising one or more reference point intermediate positions in the inertial measurement unit (IMU) virtual reality (VR) display included in the reality display. virtual (VR). An intermediate estimated reference point position is a position determined from the quick calibration data and may be associated with a time associated with an image, or with a time between times associated with an image and a subsequent image from slow calibration data. The inertial measurement unit (IMU) determines the estimated intermediate positions of the reference point based on the data from one or more position sensors (for example, accelerometers, gyroscopes) included in the virtual reality (VR) viewer. Each intermediate estimated position is separated from a subsequent intermediate estimated position by a position time value that is less than the image time value.

In some embodiments, a virtual reality (VR) console included in the virtual reality (VR) system receives slow calibration data including a series of images from the virtual reality (VR) viewer taken at image time values from an image processing device. At least one image from the series of images includes only the locators on the rear rigid body, and is associated with a particular image time value. Additionally, the virtual reality (VR) console receives the quick calibration data from the inertial measurement unit (IMU) that comprises one or more intermediate estimated positions of a reference point of the rigid front body of the virtual reality viewer ( VR), determined from one or more position sensors included in the rigid front body of the virtual reality (VR) viewer. The virtual reality (VR) console determines an observed position of the rear rigid body for the particular image time value using the slow calibration data, and determines a predicted position of the rear rigid body for the image time value associated with the image including locators from only the rear rigid body of the virtual reality (VR) viewer using the quick calibration data, as well as a position vector describing a calibrated offset between the rigid front body and the rigid rear body.

The virtual reality (VR) console determines a difference between the observed position of the rigid rear body and the predicted position of the rigid rear body. If the difference is greater than a threshold value, the virtual reality (VR) console adjusts the predicted position of the rear rigid body by a time offset value, such that the difference between the observed position of the rear rigid body and the predicted position rear rigid body is less than the threshold value. In some modalities, the virtual reality console (VR) determines the subsequent predicted positions of the rear rigid body for the times after the particular image time value of the image in the slow calibration data including only the locators from the rear rigid body using the data Quick calibration and position vector adjusted by the offset value until re-calibration can occur between the rigid front body and the rigid body rear.

Re-calibration uses at least one image from the slow calibration data that has a time after the particular image time that includes only the locators on the rigid rear body, and that includes at least a front threshold number of the locators observed on the rigid front body, and a rear threshold number of the locators observed on the rigid rear body. The virtual reality (VR) console identifies the model locators corresponding to the observed locators from the images of the slow calibration data, using a virtual reality (VR) viewer model. For example, the virtual reality console (VR) extracts the locator information from the images in the slow calibration data, where the locator information describes the locator positions observed in the virtual reality viewer ( VR) with respect to each other in a given image. In at least one of the images from the slow calibration data, the virtual reality (VR) console identifies the model locators that correspond to the locators observed in both the front rigid body and the rear rigid body. The virtual reality (VR) console compares the locator information with a viewer model to identify the model locators that correspond to the observed locators.

Based on the locator information, the virtual reality (VR) console generates the estimated positions for the observed locators using the viewer model. For example, the virtual reality (VR) console uses the viewfinder model and information that identifies the positions of the observed locators to determine a projection matrix for the translation of the ideal positions (described by the viewfinder model) up to positions on the image plane (described by the images of the observed locators) of the image processing device. The virtual reality (VR) console uses the projection matrix to estimate the positions of the observed locators. The virtual reality (VR) console uses the projection matrix to estimate the positions of the observed locations, and adjusts one or more calibration parameters to adjust one or more of the estimated locators positions observed on the front rigid body up to a relative distance between the adjusted locators estimated positions on the front rigid body and their corresponding positions determined by The observed locations of the viewer model is then a threshold value. In a similar way, the virtual reality (VR) console determines the estimated positions of the observed locators on the rear rigid body, and adjusts the estimated positions of the observed locations on the rear rigid body as described above. Based on the adjusted estimated positions of the locators observed on the first rigid body, the virtual reality (VR) console determines the calibrated positions of the first rigid body reference point for one or more images from the slow calibration data.

The virtual reality (VR) console determines a position of the rear rigid body relative to the reference point of the first rigid body. For example, the virtual reality (VR) console identifies a rear reference point on the rear rigid body using the adjusted estimated positions of the locators observed on the rear rigid body. The virtual reality (VR) console then identifies a position of the rear reference point relative to the reference point on the rigid front body. In alternative modalities, the virtual reality (VR) console identifies the position of each locator observed on the rear rigid body relative to the reference point on the front rigid body. The virtual reality (VR) console additionally adjusts one or more calibration parameters, such that the estimated intermediate positions of the reference point on the front rigid body and / or the rear reference point are within a threshold value of the predicted positions of the reference point on the front rigid body and / or the point rear reference point determined from the calibrated position of the reference point on the front rigid body and / or the calibrated position of the rear reference point (for example, by adjusting the curve) from the slow calibration data.

The modalities of a virtual reality (VR) system provide high-precision tracking using only a single camera, as opposed to many, and high precision is achieved through a high plurality of light-emitting diodes (LEDs). ) well separated on the body surface. The modulation approach allows each light emitting diode (LED) to be uniquely identified, resulting in a system that is more robust to occlusion and interference from nearby light sources. Additionally, the modulation approach, in conjunction with digital hardware, reduces power consumption because light emitting diodes (LEDs) are powered only when the camera shutter is open.

The virtual reality (VR) system solves the problem of tracking the head-mounted screen or other objects, such as game controllers, with a level of operation that is suitable for virtual reality and augmented, while at At the same time, it dramatically reduces the economic and logical requirements compared to existing technologies.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a block diagram of a system environment where a virtual reality console operates, according to one embodiment.

Figure 2A is a schematic of a virtual reality viewer, according to one embodiment.

Figure 2B is a schematic of a virtual reality viewer including a rigid front body and a rigid rear body, in accordance with one embodiment.

Figure 3 is a block diagram of a virtual reality console tracking module, according to one embodiment.

Figure 4 is a flow diagram of a process for the calibration of a virtual reality system, according to one embodiment.

Figure 5 is a flow chart of a process for reestablishing calibration between two rigid bodies in a virtual reality viewer included in a virtual reality system, according to one embodiment.

Figure 6 is a flow chart of a process for maintaining a mating relationship between two rigid bodies in a virtual reality viewer included in a virtual reality system, in accordance with one embodiment.

Figure 7 is an example graph illustrating a series of calibrated positions of a virtual reality viewer, according to one embodiment.

Figure 8 is an example high-level diagram of a position tracking system.

Figure 9 illustrates an example of observed bright spots and predicted projections.

Figure 10 illustrates an example of posture optimization.

The figures illustrate the embodiments of the present description for illustration purposes only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein can be employed without departing from the principles, or benefits obtained, of the disclosure described herein. document.

DETAILED DESCRIPTION OF THE INVENTION System architecture

FIG. 1 is a block diagram of one embodiment of a virtual reality (VR) system environment 100 where a virtual reality (VR) console 110 operates. The system environment 100 shown in FIG. 1 comprises a viewer virtual reality (VR) console 105, an image processing device 135, and a virtual reality (VR) input interface 140, which are each coupled to the virtual reality (VR) console 110.

While FIG. 1 shows an example of system 100 that includes a virtual reality (VR) viewer 105, an image processing device 135, and a virtual reality (VR) input interface 140, in other embodiments, it can be include any number of these components in system 100. For example, there may be multiple virtual reality (VR) viewers 105 each with an associated virtual reality (VR) input interface 140 and monitored by one or more image processing devices 135, where each virtual reality viewer ( VR) 105, virtual reality (VR) input interface 140, and image processing devices 135, communicate with virtual reality (VR) console 110. In alternative configurations, different and / or additional components may be included in the environment of system 100.

The Virtual Reality (VR) Viewer 105 is a head-mounted display that features media for a user. The media examples presented by the virtual reality (VR) viewer include one or more images, video, audio, or some combination thereof. In some embodiments, the audio is presented by an external device (eg, speakers and / or headphones) that receives audio information from the virtual reality (VR) viewer 105, the virtual reality (VR) console 110, or both, and the audio data is presented based on the audio information. Example embodiments of the virtual reality (VR) viewer 105 are further described in conjunction with Figures 2A and 2B.

In various embodiments, the virtual reality (VR) viewer 105 may comprise one or more rigid bodies, which may be rigidly or non-rigidly coupled to each other. A rigid coupling between the rigid bodies causes the coupled rigid bodies to act as a single rigid entity. Conversely, a non-rigid coupling between the rigid bodies allows the rigid bodies to move relative to one another. An embodiment of the virtual reality (VR) viewer 105 including two rigid bodies that are non-rigidly coupled to each other is described further below in connection with FIG. 2B.

Virtual reality (VR) viewer 105 includes an electronic display 115, one or more locators 120, one or more position sensors 125, and an inertial measurement unit (IMU) 130. The electronic display 115 displays the images to the user of according to the data received from the virtual reality console (VR) 110. In various embodiments, the electronic display 115 may comprise a single electronic display or multiple electronic displays (eg, one display for each eye of a user). Examples of the electronic display 115 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED) display, some other display, or some combination of them. Additionally, the electronic display 115 may be associated with one or more optical components that correct one or more types of optical errors (eg, field curvature, astigmatism, barrel distortion, cushion distortion, chromatic aberration, etc.). In some embodiments, the means provided to the electronic display 115 for presentation to the user is previously distorted to aid in the correction of one or more types of optical errors. Additionally, the optical components can increase a field of view of the media displayed through amplification or through another suitable method. For example, the field of view of the displayed media is such that the displayed media is rendered using almost all (eg, 110 degrees diagonally), and in some cases all, of the user's fields of view.

Locators 120 are objects located at specific positions in virtual reality (VR) viewer 105 relative to each other, and relative to a specific reference point in virtual reality (VR) viewer 105. A locator 120 can be a light emitting diode (LED), a corner reflector cube, a reflective marker, a type of light source that contrasts with an environment in which the 105 virtual reality (VR) viewer operates, or some combination thereof . In embodiments where locators 120 are active (i.e., a light emitting diode 5 (LED) or other type of light emitting device), the locators

120 they can emit light in the visible band (from about 380 nanometers to 750 nanometers), in the infrared (IR) band (from about 750 nanometers to 1 millimeter), in the ultraviolet band (from 10 nanometers to 380 nanometers), in some other 10 part of the electromagnetic spectrum, or in some combination thereof.

In some embodiments, the locators are located below an external surface of the virtual reality (VR) viewer 105, which is transparent to the wavelengths of light 15 emitted or reflected by the locators 120, or is thin enough to not substantially attenuate the wavelengths of light emitted or reflected by locators 120. Additionally, in some embodiments, the external surface or other portions of the virtual reality (VR) viewer 105 are opaque in the visible band. Accordingly, locators 120 can emit light in the infrared (IR) band below an external surface that is transparent in the infrared (IR) band, but opaque in the visible band.

The inertial measurement unit (IMU) 130 is an electronic device that generates rapid calibration data based on the measurement signals received from one or more of the position sensors 125. A position sensor 125 generates one or more measurement signals in motion response of virtual reality (VR) viewer 105. Examples of position sensors 125 include: one or more accelerometers, one or more gyros, one or more magnetometers, or any other suitable sensor type, or some combination thereof. The position sensors 125 can be located external to the inertial measurement unit (IMU) 130, internal to the inertial measurement unit (IMU) 130, or some combination thereof.

Based on the one or more measurement signals from one or more position sensors 125, the inertial measurement unit (IMU) 130 generates quick calibration data indicating an estimated position of the virtual reality (VR) viewer 105 relative to a starting position of the virtual reality (VR) viewer 105. For example, position sensors 125 include multiple accelerometers to measure translational motion (forward / backward, up / down, left / right), and multiple gyroscopes to measure rotational motion (eg, pitch, yaw, wobble) . In some embodiments, the inertial measurement unit (IMU) 130 rapidly samples the measurement signals and calculates the estimated position of the virtual reality display (VR) 105 from the sampled data. For example, the inertial measurement unit (IMU) 130 integrates the measurement signals received from the accelerometers over time to estimate a velocity vector, and integrates the velocity vector over time to determine an estimated position of a point (for example, the estimated intermediate position) in the virtual reality (VR) viewer 105. Alternatively, the inertial measurement unit (IMU) 130 provides the sampled measurement signals to the virtual reality console (VR) 110, which determines the rapid calibration data. The reference point is a point that can be used to describe the position of the virtual reality (VR) viewer 105. Although the reference point can generally be defined as a point in space, however, in practice, the reference point is defined as a point within the virtual reality (VR) viewer 105 (for example, the center of the unit inertial measurement (IMU) 130).

The inertial measurement unit (IMU) 130 receives one or more calibration parameters from the virtual reality (VR) console 110. As discussed further below, the one or more calibration parameters are used to keep track of the display of the virtual reality (VR) 105. Based on a received calibration parameter (for example, the parameters of the inertial measurement unit (IMU)), the inertial measurement unit (IMU) 130 can adjust its operation (for example, the change of sampling frequency, etc. ). In some embodiments, as further described below, certain calibration parameters cause the inertial measurement unit (IMU) 130 to offset an estimated position of the virtual reality display (VR) 105 to correct for position errors that may occur when only Certain parts of the virtual reality (VR) viewer 105 are visible to the image processing device 135. In some embodiments, certain calibration parameters cause the inertial measurement unit (IMU) 130 to update a reference point start position such that it corresponds to the next reference point calibration position. Updating the reference point starting position as the next calibrated reference point position helps reduce the accumulated error associated with the determined estimated position. Accumulated error, also known as drift error, causes the estimated position of the reference point to drift away from the actual position of the reference point over time.

The image processing device 135 generates slow calibration data in accordance with the calibration parameters received from the virtual reality (VR) console 110. The slow calibration data includes one or more images showing the observed positions of the locators 120 that are detectable by the image processing device 135. Image processing device 135 can include one or more cameras, one or more video cameras, any other device capable of capturing images, including one or more of locators 120, or some combination thereof. Furthermore, image processing device 135 can include one or more filters (eg, used to increase the signal-to-noise ratio). Image processing device 135 is configured to detect light emitted or reflected from locators 120 in a field of view of image processing device 135. In embodiments where locators 120 include passive elements (eg, a retroreflector), image processing device 135 may include a light source that illuminates all or some of locators 120, which retro-reflect light toward the light source in image processing device 135. Slow calibration data is communicated from image processing device 135 to virtual reality (VR) console 110. Image processing device 135 receives one or more calibration parameters from virtual reality (VR) console 110, and can adjust one or more image processing parameters (eg, focal length, focus, frame rate, scale ISO sensitivity, sensor temperature, shutter speed, aperture, etc.) based on the calibration parameters.

Virtual reality (VR) input interface 140 is a device that allows a user to send action requests to virtual reality (VR) console 110. An action request is a request to perform a particular action. For example, a request for action can be to start or end an application or to perform a particular action within the application. Virtual reality (VR) input interface 140 may include one or more input devices. Examples of input devices include: a keyboard, mouse, game controller, or any other suitable device for receiving action requests and communicating action requests received to the virtual reality (VR) console. 110. An action request received by the virtual reality (VR) input interface 140 communicates with the virtual reality (VR) console 110, which performs an action corresponding to the action request. In some embodiments, virtual reality (VR) input interface 140 may provide haptic feedback to the user in accordance with instructions received from virtual reality (VR) console 110. For example, haptic feedback is provided when an action request is received, or the virtual reality (VR) console 110 communicates instructions to the virtual reality (VR) input interface 140 by making the virtual reality (VR) input interface ) 140 generate haptic feedback when the virtual reality console (VR) 110 performs an action.

Virtual reality (VR) console 110 provides means to virtual reality (VR) viewer 105 for presentation to the user in accordance with information received from one or more of: image processing device 135, virtual reality viewer (VR) 105, and the virtual reality input interface (VR) 140. In the example shown in Figure 1, the virtual reality (VR) console 110 includes a media store 145, a tracking module 150, and a virtual reality (VR) engine 155. Some modes of the console virtual reality (VR) 110 have modules different from those described in relation to figure 1. In a similar way, the functions described further below can be distributed among the components of the virtual reality (VR) console 110 in a different way from what is described here.

Application store 145 stores one or more applications to be executed by virtual reality console (VR) 110. An application is a group of instructions, which when executed by a processor, generates the means for presentation to the user. Media generated by an application may be displayed in response to input received from the user by movement of virtual reality (VR) viewer 105 or virtual reality (VR) interface device 140. Examples of applications include: gaming applications, conference applications, video playback application, or other suitable applications.

Tracking module 150 calibrates the environment of system 100 using one or more calibration parameters. As further described in relation to Figures 3 to 5, the tracking module 150 can adjust one or more calibration parameters to reduce the error in determining the position of the virtual reality (VR) viewer 105. For example, the tracking module 150 adjusts the focus of the image processing device 135 to obtain a more accurate position for the locators observed in the virtual reality (VR) viewer 105. Furthermore, the calibration performed by the tracking module 150 also counts for the information received from the inertial measurement unit (IMU) 130. Additionally, as discussed in greater detail below in relation to Figures 4 and 5, if that tracking of the virtual reality (VR) viewer 105 is lost (for example, the image processing device 135 loses the line of sight of at least a threshold number of locators 120), tracking module 140 recalibrates some or all of the environments in system 100. As used herein, loss of tracking can generally refer to a loss of calibration of the image processing device 135 or of the inertial measurement unit (IMU) 130, a loss of relative positions of one or more bodies stiffness in the virtual reality (VR) viewer 105, a loss of position of the virtual reality (VR) viewer 105 relative to the image processing device 135, or some combination thereof.

Re-calibration of the environment of system 100 is generally transparent to the user. In some embodiments, the tracking module 150 may request the user to move the virtual reality (VR) viewer 105 to an orientation where one or more sides of the virtual reality (VR) viewer 105 are visible to the processing device of images 135. For example, tracking module 150 prompts the user to look up, look down, look left, look right, or look in another specified direction such that one or more sides of the virtual reality (VR) viewer ) 105 are visible to the image processing device 135. Once a threshold number of locators 120 in virtual reality (VR) viewer 105 are projected by image processing device 135, tracking module 150 resets the calibration. In some embodiments, the tracking module 150 can continuously calibrate the environment of system 100 or it can calibrate the environment of system 100 at periodic intervals to maintain accurate tracking of the virtual reality (VR) viewer 105.

Tracking module 150 can calibrate a system 100 environment that includes a virtual reality (VR) viewer 105 comprising one or more rigid bodies (eg, see Figures 2A and 2B). Additionally, as further described below in conjunction with the figures. 3 and 5, calibration can count for a virtual reality (VR) viewer 105, including two rigid bodies that are coupled in a non-rigid manner (eg, coupled to each other by means of an elastic band). The two rigid bodies can be a front rigid body that includes the inertial measurement unit (IMU) 130 that is placed in front of the user's eyes, and a rear rigid body that is placed at the rear of the user's head. This configuration of the front rigid body and the rear rigid body allows a user to rotate 360 degrees relative to the image processing device 135. However, because the ratio of the front rigid body to the rear rigid body is not necessarily fixed, the environment of system 100 may lose the calibration of the position of the front rigid body relative to the rear rigid body. Furthermore, as discussed in detail below with respect to FIG. 6, in some embodiments, if tracking between multiple rigid bodies is lost in virtual reality (VR) viewer 105, tracking module 150 may shift the position of a rigid body until re-calibration can occur. In these cases, in some embodiments, the tracking module 150 can determine an offset value to the intermediate estimated position of the virtual reality display (VR) 105 and supply it to the inertial measurement unit (IMU) 130 as a parameter of calibration.

Alternatively, the tracking module 150 can adjust a position vector describing the relative position of the front rigid body to the rear rigid body by the offset value. In some embodiments, tracking module 150 determines when to recalibrate based on a measured difference between the movement indicated by locators 120 in the rear rigid body, and the movement predicted from the rapid calibration data received from the tracking unit. inertial measurement (IMU) 130. Tracking module 150 recalibrates using the slow calibration data that includes one or more images including the front rigid body locators 120 and the rear rigid body locators.

Furthermore, the tracking module 150 tracks the movements of the virtual reality (VR) viewer 105 using the slow calibration data from the image processing device 135. As further described below in conjunction with FIG. 3, the tracking module 150 determines the positions of a reference point of the virtual reality (VR) viewer 105 using the locators observed from the slow calibration data and a model of the virtual reality (VR) viewer 105. The tracking module 150 also determines the positions of a reference point of the virtual reality (VR) viewer 105 using the position information from the quick calibration data. Additionally, in some embodiments, the tracking module 150 may use portions of the fast calibration data, the slow calibration data, or some combination thereof, to predict a future location of the viewer 105. Tracking module 150 provides the estimated or predicted future position of virtual reality (VR) viewer 105 to virtual reality (VR) engine 155.

Virtual reality (VR) engine 155 runs applications within the system environment and receives position information, acceleration information, speed information, predicted future positions, or some combination thereof, from the virtual reality (VR) viewer 105 from tracking module 150. Based on the received information, the virtual reality (VR) engine 155 determines the means to be provided to the virtual reality (VR) viewer 105 for presentation to the user. For example, if the information received indicates that the user has looked left, the virtual reality (VR) engine 155 generates the means for the virtual reality (VR) viewer 105 that reflects the movement of the user in a virtual environment. Additionally, virtual reality (VR) engine 155 performs an action within an application running in virtual reality (VR) console 110 in response to an action request received from the virtual reality (VR) input interface. 140 and provides feedback to the user that the action was performed. The feedback provided can be a visual or audible feedback through the virtual reality (VR) viewer 105 or a haptic feedback through the virtual reality (VR) input interface 140.

Figure 2A is a linear diagram of one embodiment of a virtual reality viewer. The virtual reality (VR) viewer 200 is an embodiment of the virtual reality (VR) viewer 105 and includes a front rigid body 205 and a band 210. The front rigid body 205 includes the electronic display 115 (not shown), the unit measurement sensor (IMU) 130, the one or more position sensors 125, and locators 120. In the embodiment shown in Figure 2A, the position sensors 125 are located within the inertial measurement unit (IMU) 130, and neither the position sensors 125 nor the inertial measurement unit (IMU) 130 are visible to the user. .

Locators 120 are located at fixed positions on the front rigid body 205 relative to one another, and relative to a reference point 215. In the example of Figure 2A, reference point 215 is at the center of unit

measuring sensor (IMU) 130. Each of the locators 120 emits light that can be detected by the image processing device 135. The locators 120, or portions of the locators 120, are located on a front side 220A, on a upper side 220B, on a lower side 220C, on a right side 220D, and on a left side 220E of the rigid front body 205 in the example of figure 2A.

FIG. 2B is a linear diagram of one embodiment of a virtual reality (VR) viewer 225 including a front rigid body 205 and a rear rigid body 230. The virtual reality (VR) viewer 225 shown in FIG. 2B , is an embodiment of the virtual reality (VR) viewer 105 where the front rigid body 205 and the rear rigid body 230 are coupled to each other through the band 210. Band 210 is not rigid (eg, is elastic) such that front rigid body 205 does not rigidly engage rear rigid body 210. Accordingly, rear rigid body 230 can be moved relative to the rigid body 205 and specifically can be moved relative to reference point 215. As discussed further below in conjunction with Figures 3 and 5, the rear rigid body 230 allows the virtual reality (VR) console 110 to keep track of the virtual reality (VR) viewer 105, even if the front rigid body 205 is not visible to image processing device 135. Locators 120 on the rear rigid body 230 are in fixed positions relative to each other, and relative to reference point 215 on the front rigid body 205. In the example of Figure 2B, one or more locators 120, or portions of locators 120, on the rear rigid body 230 are located on a front side 235A, on an upper side 235B, on a lower side 235C, on one side right 235D, and left 235E of the rear rigid body 230.

FIG. 3 is a block diagram of one embodiment of the tracking module 150 included in the virtual reality (VR) console 110. Some embodiments of the tracking module 150 have modules different from those described herein. In a similar way, the functionality described in relation to Figure 3 can be distributed among the components in a different way from what is described herein. In the example of Figure 3, the tracking module 150 includes a tracking database of 310, an initialization module 320, an estimation module 330, a parameter setting module 340 and a control module 350.

Tracking database 310 stores the information used by tracking module 150 to track one or more virtual reality (VR) viewers 105. For example, tracking database 310 stores one or more viewer models, one or more calibration parameter values, or any other appropriate information to track a virtual reality (VR) viewer 105. For reference above with respect to FIG. 1, a viewfinder model describes the ideal positions of each of locators 120 with respect to each other and to reference point 215. Each locator 120 is associated with a corresponding model locator in the viewfinder model; accordingly, a model locator corresponding to a locator 120 describes an ideal position of locator 120 according to the viewfinder model. Additionally, the viewer model may include information that describes changes in the model positions of locators 120 or reference point 215 as a function of the different calibration parameters. In some embodiments, the viewfinder model can describe model positions of locators 120 on a rear rigid body 230 relative to one another, model positions of a rear reference point describing a position of the rear rigid body 230, the default positions of the rear reference point relative to a reference point 215 on the front rigid body 205, the default locations of the model locations of locators 120 on the rear rigid body 230 relative to reference point 215, or some combination thereof.

Calibration parameters are parameters that can be adjusted to affect calibration of the 105 display. Sample calibration parameters include image processing parameters, Inertial Measurement Unit (IMU) parameters, or some combination of the themselves. Image processing parameters and inertial measurement unit (IMU) parameters can be included in the calibration parameters. Examples of image processing parameters include: focal length, focus, frame rate, ISO sensitivity scale, shutter speed, aperture, camera orientation, source triggering (in modes where image processing device 135 uses a source to illuminate reflective locators 120), the offset of a projection sensor relative to the center of a lens of the image processing device 135, the lens distortion parameters, the temperature of the sensor, or any other parameter used by the image processing device 135 to produce the slow calibration data. The Inertial Measurement Unit (IMU) parameters are the parameters that control the collection of rapid calibration data. Examples of inertial measurement unit (IMU) parameters include: a sampling frequency of one or more of the measurement signals from the position sensors 125, an output rate of the rapid calibration data, other suitable parameters used by the inertial measurement unit (IMU) 130 to generate the measurement data quick calibration, commands to feed the power to the inertial measurement unit (IMU) 130 on or off, commands to update the starting position to the current reference point position, offset information (eg offset to position information), or any other appropriate information.

Initialization module 320 initializes the system environment

100 using the information from the trace database 310, such as the calibration parameters retrieved from the trace database 310. In modes where the environment of system 100 was not previously calibrated, the default calibration parameters are retrieved from trace database 310. If the system environment 100 was previously calibrated, the adjusted calibration parameters can be retrieved from trace database 310. The initialization module 320 provides the retrieved calibration parameters to the inertial measurement unit (IMU) 130 and / or to the image processing device 130.

The estimation module 330 receives the slow calibration data and / or the rapid calibration data from the virtual reality display (VR) 105 and / or from the inertial measurement unit (IMU) 130. The slow calibration data is received from the image processing device 135 at a slow data rate (for example, at 20 Hz). Conversely, fast calibration data is received from the inertial measurement unit (IMU) 130 at a data rate (eg 200 Hz or higher) that is significantly faster than the data rate at which it is receive the slow calibration data. Accordingly, the quick calibration data can be used to determine the position information of the virtual reality (VR) viewer 105 among the images of the virtual reality (VR) viewer 105 included in the slow calibration data.

By using a viewer model from trace database 310 and slow calibration data from image processing device 135, estimation module 330 identifies model locators corresponding to one or more locators in the viewer virtual reality (VR) devices 135 identified from the images captured by the image processing device 135. The estimation module 330 extracts the locator information from the images in the slow calibration data, the locator information that describes the positions of the locators 120 observed relative to one another in a given image. For a given image, the locator information describes the relative positions between the locators 120 observed in the image. For example, if an image shows the observed locators A, B, and C, the locator information includes data describing the relative distances between A and B, A and C, and B and C. As described above, the model Viewfinder includes one or more model positions for locators in the 105 Virtual Reality (VR) Viewer. The estimation model 330 compares the relative positions of the observed locators 120 with the relative positions of the model locators in order to determine the correspondences between the observed locators 120 in the virtual reality (VR) display 105 and the model locators from the viewfinder model. In modalities where calibration is occurring for a 225 virtual reality (VR) viewer, including multiple rigid bodies, the model locators corresponding to the locators observed in both the front rigid body 205 and the rear rigid body 230 are they identify from at least one of the images of the slow calibration data.

Furthermore, based on the viewfinder model and information describing the model locators and the observed locators 120, the estimation module 330 generates the estimated positions for the observed locators 120. The estimation module 330 determines a projection matrix based on the viewer model and the information describing the model locators and the observed locators 120. The projection matrix is a mathematical construction that translates the ideal positions of locators 120, described by the viewfinder model, to the positions in an image plane, described by the images of the observed locators 120, of the image processing device. images 135. Accordingly, the estimation module 330 estimates the positions of the locators 120 observed using the projection matrix and the positions of the model locators described in the viewfinder model. One or more calibration parameters can be applied to the projection matrix, such that adjustments to one or more of the calibration parameters modify the estimated positions of the observed locators 120.

Estimation module 330 also extracts intermediate position information, intermediate speed information, intermediate acceleration information, or some combination thereof, from the quick calibration data. As the fast calibration data is received more frequently than the slow calibration data, the information extracted from the fast calibration data allows the estimation module 330 to determine the position information, the speed information, or Acceleration information during the time periods between images from the slow calibration data. The information of an intermediate estimated position (for example, an intermediate estimated position) describes a position of the reference point 215 at a time associated with an image, or at a time between the times associated with an image and a subsequent image from slow calibration data. The intermediate speed information describes a speed vector associated with reference point 215 at a time between a time associated with an image and a time associated with a subs image. subsequent from the slow calibration data. The intermediate acceleration information describes an acceleration vector associated with reference point 215 at a time between a time associated with an image and a time associated with a subsequent image from the slow calibration data. In some embodiments, the estimation module 330 is configured to obtain the intermediate estimated position information using the intermediate acceleration information or from the intermediate speed information. Estimation module 330 provides the intermediate position to parameter setting module 340.

The parameter setting module 340 adjusts one or more calibration parameters to adjust the estimated positions until the relative distances between the adjusted estimated positions of the observed locators 120 and the positions of their corresponding model locators are less than a threshold value. If a relative distance between an estimated position of an observed locator 120 and a position of its corresponding model locator is equal to or greater than a threshold value (for example, 1 millimeter), the parameter setting module 340 adjusts one or more parameters calibration (for example, image processing parameters) until the relative distance is less than the threshold value. For example, parameter setting module 340 modifies a calibration parameter, while other calibration parameters are kept fixed in order to determine a value for the calibration parameter being modified, resulting in less than a threshold distance between the estimated position of an observed locator 120 and a position of its corresponding model locator. The parameter setting module 340 can then set the calibration parameter to the determined value and repeat the process of modifying the values for the individual calibration parameters while maintaining other calibration parameters at constant values until the relative distances between the adjusted estimated positions of at least a threshold number of observed locators 120 and the positions of their corresponding model locators are less than the threshold value. By using the adjusted estimated positions of the observed locators 120, the parameter adjustment module 340 generates the calibrated positions of the reference point 215 for one or more frames of the slow calibration data.

In embodiments where the virtual reality (VR) viewer 105 includes two rigid bodies (eg, the virtual reality (VR) viewer 225), the parameter setting module 340 determines a position of the rear rigid body 230 relative to with reference point 215 on the rigid front body 205. In some embodiments, the parameter setting module 340 identifies a rear reference point on the rear rigid body 230 using the locators 120 observed on the rear rigid body 230 and their corresponding pattern locators. The parameter setting module 340 then identifies a position of the rear reference point relative to the reference point 215 on the front rigid body 205. Alternatively, virtual reality (VR) console 110 identifies the position of each locator 120 observed on the rear rigid body 230 relative to reference point 215 on the front rigid body 205. In some embodiments, the parameter setting module 340 generates the calibrated reference point positions 215 in response to determining that a threshold number of locators have been projected (observed locators) on one or more sides of each rigid body 205, 230, or that a threshold number of locators have been projected (locators 5 observed) on all sides of each rigid body 205, 230. For example, the threshold number of locators projected on one side of a rigid body 205, 230 is greater than or equal to zero. If the locator threshold number is not projected, the parameter setting module 340 can ask the user through the virtual reality viewer 10 (VR) 105 or through another suitable component, to orient the virtual reality viewer (VR) 105 in a specific direction relative to image processing device 135, or continue to move virtual reality (VR) viewer 105 until the locator threshold number is projected.

The parameter setting module 340 also determines a prediction function that predicts the positions of reference point 215 and adjusts one or more calibration parameters until the intermediate estimated positions of reference point 215 from the quick calibration data are within a threshold value of the predicted positions of reference point 215. For example, the prediction function is generated by fitting a curve to the series of calibrated positions. The parameter setting module 340 then adjusts one or more calibration parameters until a distance between the estimated intermediate positions 25 of the reference point 215 and the predicted positions of the reference point 215 is less than a threshold value. For example, the parameter setting module 340 can increase the sampling frequency of the inertial measurement unit (IMU) 140 until the distance between the intermediate estimated positions of reference point 215 and the predicted positions of reference point 215 is 1 millimeter or less. In other embodiments, the parameter setting module 340 adjusts one or more calibration parameters, such that the distances between each intermediate estimated position and a calibrated position (eg, CPi) of the reference point 215 associated with the image, are less than a distance value between the calibrated position (for example, CPi) of the reference point 215 associated with the image and the calibrated position of the reference point 215 associated with the subsequent image (eg GP2).

In some embodiments, the parameter setting module 340 updates the initial position of the inertial measurement unit (IMU) 130 to be the next calibrated position of reference point 215. As discussed above in relation to FIG. 1 and later in connection with FIG. 6, the inertial measurement unit (IMU) 130 collects the rapid calibration data relative to the reference point positions 215 previously determined by the unit. inertial measurement (IMU) 130. Consequently, the drift error increases the longer the inertial measurement unit (IMU) 130 collects the data without updating the initial position to a calibrated position. The parameter setting module 340 compares the estimated intermediate positions with an update threshold value. If one or more of the intermediate estimated positions exceed the update threshold value, the parameter setting module 340 communicates an instruction to the inertial measurement unit (IMU) 130 to update the initial position as the position associated with the next position calibrated. Alternatively, after determining a calibrated position, the parameter setting module 340 instructs the inertial measurement unit (IMU) 130 to update the starting position to the determined calibrated position. The parameter tuning module 340 stores the adjusted calibration parameter values in the tracking database 310 and can also provide the adjusted calibration parameters to other components in the virtual reality (VR) console 110.

Supervision module 350 monitors the environment of system 100 to determine the loss of calibration. In various embodiments, the monitoring module 350 monitors the relative distances between the adjusted estimated positions of the observed locators 120 and the positions of their corresponding model locators. If a relative distance between an adjusted estimated position of an observed locator and a position of its corresponding model locator is less than a threshold value (for example, 1 millimeter), the monitoring module 350 provides the calibrated reference point position 215 determined from the positions of the locators 120 observed to the virtual reality (VR) engine 155. Conversely, if the relative distance between an observed locator and its corresponding model locator is greater than the threshold value (for example, 1 millimeter), the monitoring module 350 determines that the calibration was lost and asks the adjustment module for parameters 340 to recalibrate system environment 100.

The relative distances determined by parameter setting module 340 between the intermediate estimated positions and their corresponding predicted positions are monitored. If a distance between a predicted position and its corresponding intermediate estimated position is less than a threshold value (for example, 1 millimeter), the monitoring module 350 provides the intermediate estimated position to the virtual reality (VR) engine 155. In some embodiments, the monitoring module 350 may also provide the intermediate speed information or the intermediate acceleration information extracted from the rapid calibration data to the virtual reality (VR) engine 155. Conversely, if the distance between the predicted position and its corresponding intermediate estimated position is greater than the threshold value, the monitoring module 350 determines that the calibration was lost and causes the environment of system 100 to reset the calibration.

In some cases, locators 120 on the rear rigid body 230 are visible only to image processing device 135. When only locators 120 are visible on the rear rigid body 230 for image processing device 135, in some embodiments, if a difference between the estimated position of the rear rigid body 230 (for example, generated from the observed locators 120 on the rear rigid body 230) and a predicted position of the rear rigid body 230 (eg generated using the quick calibration data) is greater than a threshold value, the monitoring module 350 determines that the calibration has been lost and causes the environment of the system 100 to reset the calibration. Furthermore, if the difference between the estimated position of the rear rigid body 230 and the predicted position of the rear rigid body 230 is greater than the threshold value, the virtual reality (VR) console 110 adjusts the predicted position of the rear rigid body 230 by a time offset value, such that the difference between the estimated position of the rear rigid body 230 and the predicted position of the rear rigid body 230 is less than the threshold value. The monitoring module 350 can then use the time offset value (or subsequently generated time offset values) to more accurately predict the position of the rear rigid body 230 until re-calibration can occur between the front rigid body 205 and the rigid rear body 230. Alternatively, if a difference between the estimated positions of locators 120 on the rear rigid body 230 and the positions of their corresponding model locators, relative to reference point 215, is greater than a threshold value, the modulus Monitor 350 determines that the calibration has been lost and causes the environment of system 100 to reset the calibration. In some embodiments, when the slow calibration data includes an image that includes a threshold number of locators on the front rigid body 205 and a threshold number of locators on the rear rigid body 230, the tracking module 150 begins the re-calibration. Additionally, in some embodiments, once the tracking is lost, the monitoring module 350 automatically prompts the user to adjust the virtual reality (VR) viewer 105 so that the locators are visible on both the rigid front body 205 as in the rigid rear body 230.

Calibration of virtual reality systems

FIG. 4 is a flow chart of one embodiment of a process for calibration of a virtual reality (VR) system, such as the environment of system 100 described above in connection with FIG. 1. In other embodiments, the method includes steps different, additional, or in less number than those represented in figure 4. Additionally, in some modalities, the steps described in relation to figure 4 can be performed in different orders.

Virtual Reality (VR) Console 110 initializes 410 the system environment using one or more calibration parameters. For example, virtual reality (VR) console 110 retrieves one or more calibration parameters associated with virtual reality (VR) viewer 105 from trace database 310. In some embodiments, the virtual reality (VR) console 110 retrieves the adjusted calibration parameter values from the trace database 310 if the image processing device 135 or inertial measurement unit (IMU) 130 they were previously calibrated for a particular 105 virtual reality (VR) viewer. If the image processing device 135 or inertial measurement unit (IMU) 130 was not previously calibrated for the virtual reality (VR) viewer 105, the virtual reality (VR) console 110 retrieves the default calibration parameters at from the tracking database 310. The virtual reality (VR) console 110 provides the calibration parameters to the inertial measurement unit (IMU) 130 or image processing device 135.

The virtual reality (VR) console 110 receives (420) the slow calibration data from the image processing device 135, and the fast calibration data from the inertial measurement unit (IMU) 130. The slow calibration data includes a series of images including one or more of the locators 120 in the virtual reality (VR) viewer 105. A locator 120 included in an image from the slow calibration data is referred to herein as an observed locator. Rapid calibration data may include one or more intermediate estimated positions of reference point 215 (for example, an inertial measurement unit (IMU) center 130). In other embodiments, the quick calibration data includes the intermediate acceleration information and / or the intermediate speed information from which the virtual reality console (VR) 110 determines one or more intermediate estimated positions of the reference point 215.

Based, at least in part, on the slow calibration data and a viewer model, the virtual reality console (VR) 110 identifies (430) the model locators, which are the locators in the viewer model. The virtual reality (VR) console 110 extracts the locator information describing the positions of the observed locators 120 relative to one another in the form of the slow calibration data, and compares the locator information with a model viewer retrieved from trace database 310 to identify (430) the model locators that correspond to the observed locators. Model locators are components of the viewer model, such that the identification (430) of a model locator associated with an observed locator allows the virtual reality console (VR) 110 to subsequently compare a position of the observed locator with the ideal position, from the model locator viewer model associated with the observed locator.

Using the viewer model, the virtual reality (VR) console 110 generates (440) the estimated positions of one or more of the observed locators 120. The viewfinder model describes the ideal positioning between locators 120 and reference point 215. In various modalities, the virtual reality (VR) console 110 uses the viewer model and locator information to determine a projection matrix to translate the ideal positions in the viewer model to positions in a plane of the image from the image processing device 135. The virtual reality (VR) console 110 uses the projection matrix to estimate the positions of the observed locators. Accordingly, the estimated position of an observed locator 120 identifies an ideal position of the observed locator 120 in the image plane of the images from the slow calibration data.

Based, at least in part, on the relative distances between the estimated positions of one or more observed locators 120 and the positions of the model locators corresponding to the one or more observed locators 120, the virtual reality (VR) console 110 adjusts (450) one or more calibration parameters that adjust the estimated positions of the one or more locators 120, such that a relative distance between the estimated positions of the observed locators 120 and the positions of their corresponding model locators from the viewfinder model is less than a threshold value (eg, 1 millimeter). Adjusting the calibration parameters affects the projection matrix (eg, change in focal length, etc.), such that changing one or more calibration parameters can affect the estimated positions of the locators 120 observed. If the distances between the estimated positions of the observed locators 120 and the positions of their corresponding model locators are equal to or greater than the threshold value, in one mode, the virtual reality console (VR) 110 adjusts (450) a parameter of calibration, while the other calibration parameters are held fixed to determine a value for the calibration parameter being adjusted that results in a distance between the estimated position of an observed locator 120 and the position of its corresponding model locator being less than the threshold value. The calibration parameter can then be set to the determined value, while another calibration parameter is modified such that the distance between an estimated position of an additional locator 120 and an additional position of a model locator corresponding to the additional locator is less than the threshold value. Various calibration parameters (450) can be adjusted as described above, such that the relative distances between the adjusted estimated positions of at least a threshold number of observed locators 120 and the positions of their corresponding model locators are less than the threshold value. If the distances between the estimated positions of at least a threshold number of the observed locators 120 and the positions of their corresponding model locators are less than the threshold value, the calibration parameters are not adjusted (450).

Virtual reality (VR) console 110 determines (460) whether a threshold number of observed locators 120 are on each side of front rigid body 205 (i.e., front side 220A, top side 220B, bottom side 220C, the right side 220D, and the left side 220E). If the threshold number of observed locators 120 are associated with each side, the virtual reality (VR) console 110 generates (470) the calibrated reference point 215 positions for one or more frames of the slow calibration data using the estimated positions adjusted from observed locators. In modalities where the virtual reality (VR) viewer 105 includes multiple rigid bodies, the virtual reality (VR) console 110 generates the calibrated positions of the reference point 215 that respond to the determination that a threshold number of locators are projected (observed locators) on one or more sides of each rigid body 205, 230, or that they respond to the determination that a threshold number of locators are projected (observed locators) on all sides of each rigid body 205, 230. If the observed threshold number of locators 120 are not associated with each side, virtual reality (VR) console 110 may communicate a message to the user through virtual reality (VR) viewer 105 or another component, to reposition the virtual reality (VR) viewer 105 such that slow calibration data can be captured including locators on one or more sides of the virtual reality (VR) viewer 150.

The virtual reality (VR) console 110 adjusts (480) one or more calibration parameters until the intermediate estimated positions of the virtual reality (VR) viewer 105 received from the quick calibration data are within a threshold distance of predicted positions for virtual reality viewer (VR) 105 or reference point 215, wherein the predicted positions are determined from the calibrated positions of the reference point 215 associated with various images from the slow calibration data. In some embodiments, the virtual reality (VR) console 110 determines a predicted position of the reference point 215 by generating (for example, by means of a curve fit) a prediction function using the calibrated positions of the reference point reference 215 associated with different images from the slow calibration data. Virtual reality (VR) console 110 adjusts one or more of the calibration parameters until the distances between the intermediate estimated positions of reference point 215 and the predicted positions of reference point 215 are less than a threshold distance. For example, virtual reality console (VR) 110 can increase the sampling frequency of inertial measurement unit (IMU) 130 until the distances between the estimated intermediate positions of reference point 215 and the predicted positions of reference point 215 are all 1 millimeter or less, or until the distances between at least a threshold number of intermediate estimated positions of reference point 215 and the predicted positions of reference point 215 are less than 1 millimeter. In other embodiments, virtual reality (VR) console 110 determines a predicted reference point position 215 as a position between a calibrated position of reference point 215 associated with an image from the slow calibration data and a calibrated position of reference point 215 associated with a subsequent image from the slow calibration data. The virtual reality (VR) console 110 then adjusts (480) one or more calibration parameters such that the distances between each intermediate estimated position and a calibrated position (eg CPi) of the reference point 215 associated with the image , are less than a distance between the calibrated position (for example, CPi) of the reference point 215 associated with the image and the calibrated position of the reference point 215 associated with the subsequent image (for example, CP2). Additionally, the virtual reality console (VR) 110 can update the starting position of the inertial measurement unit (IMU) 130 to be the calibrated position of the reference point 215.

In some embodiments, the virtual reality (VR) console 110 stores the adjusted calibration parameter values in the tracking database 310 or provides the adjusted calibration parameter values to other components of the virtual reality console ( VR) 110. Adjusted calibration values can reduce calibration times for subsequent operations in the system 100 environment, improving the user experience.

Virtual reality (VR) console 110 monitors (490) the environment of system 100 to determine loss of calibration. For example, virtual reality (VR) console 110 monitors the relative distances between the adjusted estimated positions of the observed locators 120 and the positions of their corresponding model locators. If a relative distance between an adjusted estimated position of an observed locator 120 and a position of its corresponding model locator is less than a threshold value (for example, 1 millimeter), the virtual reality (VR) console 110 provides the calibrated position to the virtual reality (VR) engine 155. Conversely, if the relative distance between an estimated position of an observed locator and a position of its corresponding model locator is greater than (or equal to or exceeds) the threshold value (for example, 1 millimeter), the virtual reality (VR) 110 determines that the calibration was lost, receives (420) the slow calibration data and the fast calibration data, and performs the functions identified above to recalibrate the environment of system 110.

Additionally, virtual reality console (VR) 110 monitors (490) the relative distances between the estimated intermediate positions of reference point 215 and the predicted positions of reference point 215. For example, if a distance between a predicted position curve of reference point 215 and an intermediate estimated position of reference point 215 is less than a threshold distance (for example, 1 millimeter), the virtual reality console (VR) 110 provides the intermediate estimated position to the virtual reality (VR) engine 155. In some embodiments, the virtual reality (VR) console 110 may also provide the intermediate speed information or the intermediate acceleration information extracted from the rapid calibration data to the virtual reality (VR) engine 155. Conversely, if the distance between the predicted position of the reference point 215 and an intermediate estimated position of the reference point 215 is greater than, or equal to or exceeds, the threshold distance, the virtual reality console (VR) 110 determines that the calibration was lost, receives (420) the slow calibration data and the quick calibration data, and performs the functions identified above to recalibrate the environment of system 100.

In some embodiments, the inertial measurement unit (IMU) 130 and image processing device 135 can be calibrated simultaneously. To simultaneously calibrate the inertial measurement unit (IMU) 130 and image processing device 135, the virtual reality (VR) console 110 estimates the positions of reference point 215 for a series of images using the estimated positions of the locators observed.

Additionally, the virtual reality (VR) console 110 uses the quick calibration data, including the intermediate estimated positions of the reference point 215 in the particular time values that correspond to the images from the slow calibration data when calibrating the inertial measurement unit (IMU) 130 and the image processing device 5 135. When the calibration parameters are adjusted simultaneously with the inertial measurement unit (IMU) 130 and image processing device 135, the virtual reality console (VR) 110: (1) adjusts the estimated positions of the locators observed such that the relative distance 10 between the adjusted estimated positions of the observed locators and the positions of their corresponding model locators is less than a threshold value; and (2) adjusts the estimated positions for the reference point such that the relative distance between the estimated positions for the reference point at the particular time values corresponding to the images from the slow calibration data and the positions of a model reference point determined from the model locators, be less than the threshold value.

FIG. 5 is a flowchart illustrating one embodiment of a process for restoring calibration between two rigid bodies of a virtual reality viewer 225 included in the environment of system 100. In other embodiments, the procedure includes different, additional steps, or in smaller numbers than those represented in figure 5. Additionally, in some 25 modalities, the steps described in relation to figure 5 can be carried out in different orders.

Virtual reality (VR) console 110 receives (510) the slow calibration data, including images showing a front threshold number of locators 120 on a rigid front body 205 and a rear threshold number (for example, at least one ) of the locators 120 in a rigid rear body 230 of the virtual reality (VR) viewer 225. A locator 120 included in an image from the slow calibration data is referred to herein as an observed locator. As previously described in relation to Figures 2 to 4, the virtual reality (VR) console 110 receives (150) the slow calibration data from the image processing device 135, and the fast calibration data from the unit inertial measurement (IMU) 130. The quick calibration data may also include the intermediate acceleration information and / or the intermediate speed information from which the virtual reality console (VR) 110 determines one or more intermediate estimated positions of the reference point 215 of the virtual reality (VR) viewer 225.

Based at least in part on slow calibration data and a viewfinder model, the virtual reality console (VR) 110 identifies (520) the model locators, which are locators in the viewer model. The virtual reality (VR) console 110 extracts the locator information describing the positions of the observed locators 120 relative to one another from the slow calibration data, and compares the locator information with a model of viewer retrieved from trace database 310 to identify (520) the model locators corresponding to the observed locators 120. In at least one of the images, the model locators corresponding to the locators observed in both the front rigid body 205 and the rear rigid body 230 of the virtual reality (VR) viewer 225 are identified. Model locators are components of the viewer model, such that identification (520) of a model locator associated with an observed locator allows the virtual reality (VR) console 110 to subsequently compare a position of the observed locator with the ideal position, from the model locator viewer model associated with the observed locator.

Using the viewer model, the virtual reality (VR) console 110 generates (530) the estimated positions of one or more of the observed locators 120. The viewfinder model describes the ideal positioning between locators 120 and reference point 215. In various modalities, the virtual reality (VR) console 110 uses the viewfinder model and locator information to determine a projection matrix for translation of ideal positions in the viewfinder model to positions in an image plane from image processing device 135. Virtual reality (VR) console 110 uses the projection matrix to estimate the locator positions

120 observed. Accordingly, the estimated position of an observed locator 120 identifies an ideal position of the observed locator 120 in the image plane of the images from the slow calibration data.

Based at least in part, on the relative distances between the estimated positions of one or more observed locators 120 and the positions of the model locators corresponding to the one or more observed locators 120, the virtual reality console (VR) 110 adjusts ( 540) the relative distance 10 between the estimated positions of the locators observed on the front rigid body 205 and the positions of their corresponding model locators, in less than a threshold value (for example, 1 millimeter). Adjusting the calibration parameters affects the projection matrix (eg, change in focal length, etc.), such that changing one or more calibration parameters can affect the estimated locator positions 120 observed. If the distances between the estimated positions of the observed locators 120 and the positions of their corresponding model locators are equal to or greater than 20 above the threshold value, in one mode, the virtual reality console (VR) 110 adjusts (540) a parameter calibration, while other fixed calibration parameters are maintained to determine a value for the calibration parameter being adjusted, which results in a distance between the estimated position of an observed locator 120 and the position of its corresponding model locator being less than the threshold value. The adjustment (540) of the calibration parameters is further described above in relation to FIG. 4. IF the distances between the estimated positions of at least a threshold number of the observed locators 120 and the positions of their corresponding model locators are less than the threshold value, the calibration parameters are unadjusted (540).

After adjusting (540) the calibration parameters such that at least a threshold number of the relative distances between the estimated positions of the observed locators and the positions of their corresponding model locators, are less than the threshold value, virtual reality console (VR) 110 generates (550) the calibrated positions of the reference point 215 associated with one or more images from the slow calibration data using the adjusted estimated positions of the observed locators 120. In some embodiments, the virtual reality (VR) console 110 generates the calibrated positions of reference point 215 in response to determining that a threshold number of locators are projected (observed locators) on one or more sides of each rigid body 205 , 230, or the determination that a threshold number of locators are projected (observed locators) on all sides of each rigid body 205, 230. If the threshold number of locators is not projected (on one side of a rigid body 205, 230, or on all sides of each rigid body 205, 230), the virtual reality console (VR) 110 can ask the user, for the virtual reality (VR) viewer 105 or by means of another suitable component, which orients the virtual reality (VR) viewer 105 in a specific direction relative to the image processing device 135, or continue to move the virtual reality (VR) viewer 105 until the locator threshold number is projected.

Virtual reality (VR) console 110 also determines (560) a position of the rear rigid body 230 relative to reference point 215. In some embodiments, virtual reality (VR) console 110 identifies a rear reference point at the rear rigid body 230 using the observed locators 120 and their corresponding model locators. Virtual reality (VR) console 110 then identifies the position of the rear reference point relative to reference point 215 on the front rigid body 205 such that the rear reference point is positioned relative to the point of reference. reference 215 using a position vector. Alternatively, virtual reality (VR) console 110 identifies the position of each locator observed on the rigid rear body 230 relative to reference point 215, such that the positions of each locator observed on the rigid body rear 230 are positioned relative to reference point 215 by their own position vector.

The virtual reality console (VR) 110 adjusts (570) one or more calibration parameters, such that the intermediate estimated positions of reference point 215 are within a threshold distance from the predicted positions of reference point 215. Adjustment of the calibration parameters such that the intermediate estimated positions of the reference point 215 are within a threshold value of the predicted positions of the reference point, is further described above in connection with Figure 4. After adjusting (570) one or more calibration parameters, the virtual reality console (VR) 110 monitors (580) to determine the calibration loss of the system environment 100, as described above in connection with Figure 4.

When the loss of calibration is monitored (580), the virtual reality (VR) console 110 uses the images from the slow calibration data, which may include the observed positions of locators 120 on the front rigid body 205 , in the rigid rear body 230, or in some combination thereof. In some embodiments, the threshold value between a position of an observed locator 120 and a position of its corresponding model locator may vary depending on the rigid body in which the observed locator 120 is located. For example, the threshold value may be 1 millimeter for locators 120 observed on the front rigid body 205, and 2 millimeters for locators 120 observed on the rear rigid body 230.

Additionally, in some scenarios, the image processing device 135 is not capable of viewing the locators 120 on the front rigid body 205, but is capable of viewing the locators on the rear rigid body 230. In these scenarios, tracking is monitored using the process described below with respect to figure 6.

When the slow calibration data includes an image that includes a threshold number of locators on the front rigid body 205 and a threshold number of locators on the rear rigid body 230, the steps described above in relation to FIG. 5 are repeated to restore the system environment calibration 100. In some embodiments, when tracking is lost, virtual reality (VR) console 110 automatically prompts the user to adjust virtual reality (VR) viewer 105 such that locators on both the front rigid body 205 and rear rigid body 230, are visible to image processing device 135. The request presented to the user can provide the user with specific instructions for positioning the virtual reality (VR) viewer 105, so that the locators on both the front rigid body 205 and the rear rigid body 230 are visible to the user. image processing device 135.

FIG. 6 is a flow chart illustrating one embodiment of a process for maintaining a mating relationship between two rigid bodies of a virtual reality viewer 225 included in the environment of system 100. In other embodiments, the method includes steps different, additional, or in fewer numbers than those represented by figure 6. Additionally, in some embodiments, the steps described in relation to figure 6 can be performed in different orders.

The virtual reality (VR) console 110 receives (610) the slow calibration data from the image processing device 135. The slow calibration data includes a series of 10 images that includes an image associated with an image time value. , and having only the locators 120 observed in the rear rigid body 230 visible to the image processing device 135. An image time value is a time value when the image was captured by image processing device 15 135. Additionally, virtual reality console (VR) 110 receives (620), from the inertial measurement unit ( IMU) 130, the quick calibration data including the Intermediate estimated positions of a reference point 215 for a series of time values including the time value of the image.

Based on the slow calibration data, the virtual reality console (VR) 110 determines (630) an observed position of the rear rigid body 230 at the time value of the image. To determine (620) the observed position of the rear rigid body 230, 25 the virtual reality (VR) console 110 extracts the locator information describing the positions of the observed locators 120 on the rear rigid body 230 relative to the others, from the slow calibration data, and compares the locator information with a viewer model retrieved from the tracking database 310, to identify the model locators corresponding to the observed locators 120. After identifying the model locators, the virtual reality (VR) console 110 determines the observed locators 120 corresponding to each model locator, and determines a rear reference point for the rear rigid body 230 using the positions of the observed locators 120 . In some embodiments, the observed position of the rear rigid body 230 is the position of the rear reference point. In alternative embodiments, the observed position of the rear rigid body 230 may be the observed position of one or more of the observed locators.

Virtual reality console (VR) 110 determines (640) a predicted position of the rear rigid body 230 at the image time value using the quick calibration data and a position vector. The position vector describes a calibrated offset between the front rigid body 205 and the rear rigid body 230. For example, the position vector describes a calibrated offset between the reference point 215 associated with the front rigid body 205 and a rear reference point associated with the rear rigid body 230. Additionally, in some embodiments, the position vector may include one or more sub-vectors each describing the relative calibrated offset between reference point 215 and different locators in the rear rigid body 230.

From the quick calibration data, the virtual reality (VR) console 110 determines an intermediate estimated position of the reference point 215 in the front rigid body 205. In some embodiments, the virtual reality (VR) console 110 determines the predicted position of the rear rigid body 230 as a position relative to the position of the reference point 215 based on the position vector. For example, the position vector identifies a relative positioning of a rear reference point on the rear rigid body 230 to reference point 215. Alternatively, the position vector identifies the relative position of one or more locators 120 ( including the observed locators) on the rear rigid body 230 relative to reference point 215.

The virtual reality console (VR) 110 determines (650) if a difference between the observed position and the predicted position is greater than a threshold value (for example, 1 millimeter). If the difference is less than the threshold value, the virtual reality (VR) 225 trace is maintained, and the slow calibration data is received (610), and the process proceeds as described above. However, if the difference between the observed position of the rear rigid body 230 exceeds the threshold value, the virtual reality console (VR) 110 determines that the virtual reality viewer (VR) 105 tracking is lost, and adjusts (660) the position predicted by an offset value. The offset value is determined such that the difference between the observed position of the rear rigid body 230 and the predicted position of the rear rigid body 230 is less than the threshold value. For example, virtual reality (VR) console 110 uses the position vector modified by the offset value to more accurately determine the position of the rear rigid body 230 from the quick calibration data. Alternatively, the virtual reality console (VR) 110 communicates an instruction to the inertial measurement unit (IMU) 130 to compensate the estimated intermediate positions based on the offset value without modifying the position vector.

Based on the quick calibration data and the adjusted vector, the virtual reality console (VR) 110 determines (670) the subsequent predicted positions of the rear rigid body until re-calibration occurs (for example, as described above with respect to figure 5). In some embodiments, when tracking is lost, virtual reality (VR) console 110 prompts the user to adjust virtual reality (VR) viewer 105 so that locators on both the front rigid body 205 and the rear rigid body 230 are visible to the image processing device 135. The message presented to the user may provide the user with specific instructions for positioning the virtual reality (VR) viewer 105 such that the locators on both the front rigid body 205 and the rear rigid body 230 are visible to the image processing device. Images 135 to facilitate the recalibration described in detail above with reference to Figure 5.

Figure 7 illustrates an example graph 700 illustrating a series of calibrated positions of a virtual reality display 105. In Figure 7, the vertical axis represents position, and the horizontal axis represents time. Graph 700 includes a series of calibrated 710A-C positions from a 105 Virtual Reality (VR) viewer reference point, at times Ti, T2 and T3, respectively. Graph 700 also includes a series of intermediate estimated positions 715A-D and 720A-H from the reference point. The calibrated positions 710A-C are generated using the slow calibration data from an image processing device 135, and the intermediate estimated positions 715A-D and 720A-H are generated using the rapid calibration data from the inertial measurement unit ( IMU) 130 included in a virtual reality (VR) viewer 105. Note that the relative time scales of the 710A-C calibrated positions and the 715A-D intermediate estimated positions are different, and that the 715A-D and 720A-H intermediate estimated positions are determined more frequently than the 710A calibrated positions. -C.

Graph 700 shows a predicted position curve 725 described by a prediction function that describes the predicted position of the reference point. The prediction function is generated by fitting a curve to the calibrated positions 710A-C, and by determining a function that describes the fitted curve. Any suitable method can be used to determine the position function from the calibrated positions 710A-C.

In the example of Figure 7, the intermediate estimated positions 715A-D are the initial intermediate estimated positions determined using the quick calibration data before adjusting the calibration parameters. The intermediate estimated position 715A is relatively close to the predicted position curve in Figure 7, but as time progresses, the intermediate estimated positions move further away from the predicted position curve 725, the intermediate estimated position 715D being Figure 7 is the furthest from the predicted position curve 725. The difference between the predicted position curve and the intermediate estimated position can be attributed to a combination of actual user movements, drift error, as well as additional factors. As discussed above, because the inertial measurement unit (IMU) 130 determines an intermediate estimated position relative to a previously determined position, the error compounds result in a larger deviation between the predicted position curve 725 and the Estimated 715 intermediate positions over time. To account for drift error, virtual reality (VR) console 110 can update an inertial measurement unit (IMU) position 130 as the subsequent calibration position. The inertial measurement unit (IMU) 130 then generates a quick calibration with respect to the updated starting position and the estimated intermediate positions determined after the starting position. In this mode, the virtual reality console (VR) 110 updates the starting point as the calibrated position 710B.

Another way to reduce the error associated with estimated intermediate positions is by increasing the frequency at which intermediate estimated positions are determined. In the example of Figure 7, the virtual reality (VR) console 110 determines the intermediate estimated positions 720A-H at twice the frequency of the intermediate estimated positions 715A-D, resulting in a smaller difference between the intermediate estimated positions 720A-H and predicted position curve 725. Additional configuration information

A rigid body tracking system is presented for tracking the unpredictable movement of rigid bodies. This rigid body tracking system can be useful for virtual reality, augmented reality, games, education, training and therapy.

One embodiment of the rigid body tracking system is to track the movement of a head mount display attached to the user's head. In this case, the light emitting diodes (LEDs) are attached to the surface of the screen. An inertial measurement unit is mounted inside the screen and can include a sensor that measures angular velocity, such as one or more gyros. It may additionally include one or more accelerometers and one or more magnetometers. A separate camera is placed in a fixed location, in front of the user. In some embodiments, the rigid body tracking system is the virtual reality (VR) system 100.

In another embodiment, the rigid body being tracked can be held in the user's hand. For example, this could allow hand position and orientation to be maintained in a virtual reality experience. In another embodiment, the camera can track multiple rigid bodies, including more than one viewfinder and more than one object held by the additional hand.

The sensors that make up the inertial measurement unit provide sufficient measurements to estimate the orientation of a rigid body. These sensors include a gyroscope, an accelerometer, and possibly a magnetometer. If the rigid body is in view of a stationary camera and at an appropriate distance, then additional orientation information can be inferred. Additionally, the rigid body position is inferred. The tracking method combines measurements from all detection sources, including one or more cameras, in such a way that the position and orientation of the rigid body are accurately maintained. The placement, number, and modulation of light emitting diodes (LEDs) on the rigid body surface are crucial to the rigid body tracking system. These factors together lead to reliable, high-precision tracking of the rigid body.

Please note that the various features of the present invention can be practiced alone or in combination. These and other features of the present invention will be described in greater detail later in the detailed description of the invention and in conjunction with Figures 8 to 10.

The rigid body tracking system works by merging sensor data from two sources: an inertial measurement unit inside the rigid body and a stationary camera that faces the body. The most common modality is that the rigid body is a head-mounted screen that is attached to a human head. In this case, the position and orientation of the head should be tracked with low latency and jitter. The information provided by the sources is complementary in that the inertial measurement unit provides the most powerful limitations on head orientation, and the camera provides the most powerful limitations on head position 20. Together, they provide the inputs to the filters that estimate the position and orientation of the head.

The most important component of the inertial measurement input is a high-frequency three-axis gyroscope, which measures the angular velocity of the rigid body. After numerical integration of the sensor data that accumulates over time, the current orientation is estimated relative to the initial orientation. To account for dead reckoning errors that accumulate over time, an accelerometer or other sensor can be used to estimate the direction below to compensate for tilt errors. A magnetometer may possibly be included in the inertial measurement to compensate for errors in the estimated orientation with respect to rotations about the vertical axis (parallel to gravity). This is particularly useful when the rigid body is not in the camera's field of view; otherwise, the camera may alternatively provide this correction.

The camera captures frames at a standard rate of 60 Hz, although both higher and lower speeds can offer advantages in other modes. The resolution is 640 x 480 (standard VGA), and other resolutions can also be used. Images are sent over a serial communication (USB) link to the core processor. There is also a serial link between the camera and the virtual reality (VR) viewer. This is used to command the camera to open and close the shutter for short precise time intervals. A typical exposure time time is 0.1 milliseconds. The lens can be of any diopter, including narrow, wide-angle, or fish-eye vision. The camera may or may not have an infrared (IR) filter on its lens.

Imagine a user who is wearing a virtual reality (VR) viewer and is about to be magically transported from the real world to a virtual world. Ideally, the user's head movements in the real world should be perfectly reflected in the virtual world for the user's brain to be completely fooled. When images of the virtual world are presented to the user's eyes, they must correspond exactly to what the user expects to see based on the user's sense of immersion. A critical component to achieving this is head tracking, which involves collecting data from the sensors and processing them to determine how a user's head is moving.

You can think of a user's head and virtual reality (VR) viewer as a whole as a rigid body moving through 3D space. A rigid body has up to six degrees of freedom (DOF), which means that six independent parameters are needed to fully specify its position and orientation in space. Using these six parameters, we can calculate the coordinates of any point on the body. (It took a long time to find out in the history of physics, with most of the credit being given to Euler and Lagrange in the late 18th century.) Although there are many ways to choose these parameters, they are usually carried out in such a way that three correspond to the orientation (the way the body rotates) and three correspond to the position (where the body is placed). These can be referred to as the yaw, pitch, and wobble orientation parameters. In the above methods, only these three parameters were tracked, resulting in the 3 degrees of freedom (3-DOF) scan. Using these, a virtual reality (VR) system would know which direction your head is pointing, but unfortunately you have to guess where the user's eyes might be located.

Using all three parameters, a 3D rotation is applied to the base of the head model, and the eyes move to a reasonable location, but it is not necessarily correct. The first problem is measuring the head model: How big is a user's head? A bigger problem is that there is nothing to prevent a user from moving the base of the head. For example, when a user looks forward and moves from side to side, his head rotates very little, but the position changes a lot. Alternatively, if a user leans forward on his hip, while keeping his neck stiff, the base of his head will go a long way. Failure to take into account this type of movements of the base of the head causes a vestibular mismatch, which can contribute to simulator dizziness. The problem is that the sensors in a user's inner ear do not agree with what their eyes are seeing. The inner ear detects movement, but the eyes cannot see it.

The rigid body tracking system tracks all six parameters directly from the sensor data, rather than having to guess the three additional position parameters. This is 6-degree-of-freedom (6-DOF) tracking, which allows the rigid body tracking system to reliably estimate both the direction a user is in and the position of the eyes while moving their head around. This assumes that some fixed, person-specific amounts, such as interpupillary distance (IPD), have been measured. In addition to a reduction in vestibular mismatch, the level of immersion is incredible! A user can move her head back and forth to judge depth. While standing on a plank of a pirate, a user can bend over to look at the awful waters. The possibilities are endless!

VISION BASED POSITION TRACKING

Position tracking is not a new problem in virtual reality (VR). There have been many attempts using a variety of technologies: single or multiple cameras, structured light, time of flight, and magnetic sensors to name a few. Solutions vary based on cost, power consumption, accuracy, computation required, and environmental limitations. To this end, some modalities describe a position tracking solution that is based on a low-cost, low-resolution single camera.

Camera-based position tracking, also known as posture estimation, is a major problem for many robotics and computer vision applications. The relevant literature is huge. However, most of this literature has nothing to do with precision, computational resources, and latency. When navigating in a car in the desert, a position estimate that deviates by a few inches is pretty good, and there's no need to update the position very often. By contrast, in the context of virtual reality (VR), tracking must be very accurate, with low latency, and cannot take computing resources away from rendered content.

In some embodiments, the position tracking system is made up of three components: a set of markers built into the virtual reality (VR) viewer (for example, the virtual reality (VR) viewer 105), an inertial measurement unit (IMU) and an external camera. Figure 8 shows a high level flow diagram of a rigid body tracking system.

The rigid body tracking system acquires images from the virtual reality (VR) viewer. Through Image processing, the rigid body tracking system extracts the image position from the markers. Each marker is an infrared (IR) light emitting diode (LED). Based on the image positions of the markers, and on the known 3D model of the virtual reality (VR) viewer, the rigid body tracking system calculates a 6D posture. The algorithm takes advantage of the information available from the inertial measurement unit (IMU) (gyroscope, magnetometer and accelerometer). After the fusion of the vision data and the inertial measurement unit (MIU), we determine the optimal viewer posture. The result is provided to the application for rendering. We now return to a more detailed analysis of the individual components of the algorithm.

The rigid body tracking system identifies each of the visible markers (light emitting diodes (LEDs)). In each captured image, the rigid body tracking system detects various bright spots. Each point corresponds to a light emitting diode (LED). The rigid body tracking system identifies which of the light emitting diodes (LEDs) physically produces each point. Once this is done, the rigid body tracking system associates a known 3D model of the light emitting diodes (LEDs) in the virtual reality (VR) viewer with their observed image projections. With this association, and given a calibrated camera, rigid body tracking can estimate the posture that best explains the observed dot pattern.

As a result, the identification problem is difficult. In order to identify individual light emitting diodes (LEDs), the rigid body tracking system extracts some unique information about each. There are many possible ways to achieve just that. For example, the rigid body tracking system could use a variety of visible light emitting diodes (LEDs) and could distinguish them based on color. Another interesting approach is to incorporate geometric information in the form of lines or other shapes. Based on the way the points are organized, the rigid body tracking system can determine your individual identifications. And also the spatial arrangement, such as allowing only coplanar markers to simplify their identification.

Most of these ideas are computationally complex, are not strong on occlusion and noise, or impose strict requirements on the geometry and appearance of the virtual reality (VR) viewer. In some embodiments, infrared (IR) light emitting diodes (LEDs) are used in a general 3D configuration. Light Emitting Diodes (LEDs) are indistinguishable based on appearance. They all look like bright spots. Additionally, there is also no special geometric arrangement that can simplify your identification. Bright spots are also very abundant in images; Noise, other light sources, and our Light Emitting Diodes (LEDs) all look the same. And finally, because the light emitting diodes (LEDs) are scattered throughout the virtual reality (VR) viewer, only some of the light emitting diodes (LEDs) are visible in any given image, and some diodes Light emitters (LEDs) can be obstructed by the user's hands.

Light emitting diodes (LEDs) are integrated into the virtual reality (VR) viewer. In some embodiments, a solution is based on modulating the brightness of light emitting diodes (LEDs). Over time, each light emitting diode (LED) displays a unique light pattern. Because light emitting diodes (LEDs) operate in the near infrared (IR) spectrum, they are invisible to the human eye. However, in the infrared (IR) spectrum, light emitting diodes (LEDs) are very bright and have a wide illumination field (about 120 degrees). Camera 5 is designed to be sensitive to light in the near infrared (IR) frequency, and has a wide field of view lens. These properties (bright light emitting diodes (LEDs) and wide field of view) allow for a large tracking volume.

A vision algorithm detects individual light emitting diodes (LEDs) and tracks them across multiple frames. Next, analyze the displayed light pattern to determine the unique identification of each light-emitting diode (LED). Because the rigid body tracking system decodes each light-emitting diode (LED) individually, this approach is strong for noise and occlusion. However, it uses multiple frames before decoding is achieved. It also means that in any given frame there may be some new (not yet decoded) light emitting diodes (LEDs), and some decoded light emitting diodes (LEDs) may be out of sight. The reconstruction algorithm 20 has to be robust to false positives.

Note that the camera has a very fast shutter (less than a millisecond long) and the light emitting diodes (LEDs) are synchronized so that they only light up when the shutter is open. As a result, the Rigid Body Tracking System 25 can greatly reduce the amount of ambient light collected by the camera, save energy, minimize blur during fast head movement, and most importantly, maintain known synchronization between the camera and the virtual reality (VR) viewer. This can be important when the rigid body tracking system fuses the sensors between the Inertial Motion Unit (IMU) and vision measurements.

RECONSTRUCTION OF THE POSTURE

Once the identification of each light emitting diode (LED) is known, and given a known 3D model of light emitting diodes (LEDs), the rigid body tracking system is ready to solve a classic projective geometry problem - which is the 6D posture (orientation and 3D position) of the virtual reality (VR) viewer that best explains the observed projection of light-emitting diodes (LEDs).

The rigid body tracking system has two types of posture reconstruction modes: bootstrapping and incremental.

The first variant, bootstrapping, occurs when the rigid body tracking system first perceives images from light-emitting diodes (LEDs). The rigid body tracking system previously solved the identification problem, which means you may have an assumption about the identifications of some of the bright spots in the box. But the rigid body tracking system must be careful: some of the identifications may be wrong, some of the points may not have been identified yet, and some of the points may be due to other light sources in the environment. During bootstrapping, the rigid body tracking system has to resolve all these conflicts quickly and calculate the posture of the virtual reality viewer (VR).

Bootstrapping is a computationally very efficient method for generating posture hypotheses based on decoded light emitting diodes (LEDs). The rigid body tracking system sorts through these hypotheses using a Random Consensus Sampling Variant (RanSAC) to find the subset of Light Emitting Diodes (LEDs) that were correctly identified.

In order to understand what these posture hypotheses pose, an ABC triangle projected onto a flat triangle of the image is taken. A posture hypothesis works backward; It tells the rigid body tracking system what the position and orientation of the triangle is that explains the image. In the case of 3 points A, B and C, there cannot be more than one posture with the same projection. It is important to note that this ambiguity of posture disappears when the points are not coplanar, and decreases with more than 3 points.

The last part of bootstrapping is to refine the reconstructed posture by including as many Light Emitting Diodes (LEDs) as possible. The rigid body tracking system does this by projecting our 3D model onto the image using the calculated posture. Next, the rigid body tracking system matches the expected light emitting diode (LED) positions for observations. This allows the rigid body tracking system to dramatically increase the number of light emitting diodes (LEDs) identified and, as a result, refine our calculated posture. The coincidence of the expected light-emitting diode (LED) image positions with the observations turns out to be the key to solving the second version of the posture reconstruction problem: the incremental problem.

Figure 9 illustrates the observed bright spots (the largest dots) and the predicted projections (the smallest dots). Matching the pairs of predicted and observed light emitting diodes (LEDs) can be challenging in some cases.

It is worth noting that matching the expected and observed light emitting diode (LED) positions can be quite difficult, particularly with rapid head movements and because all light emitting diodes (LEDs) have the same appearance.

The second variant of the posture estimation problem is much easier. Fortunately, it's the one the rigid body tracking system has to solve most of the time! When the rigid body tracking system is in an incremental mode, the rigid body tracking system can use the posture of the virtual reality (VR) viewer from the table above as a starting point for calculating your posture in the current frame. How does it work? In a word: prediction. During incremental mode, the rigid body tracking system fuses information from vision and the inertial unit of measurement (IMU) to calculate where the virtual reality (VR) viewer is expected to be. The rigid body tracking system then matches the prediction with the observations and makes the necessary corrections.

Incremental estimation has been applied to many different problems. For example, many optimization algorithms use an iterative approach in which each step gets a little closer to the solution. Another example is the problem where measurements are provided in real time (online). A wide variety of filters have been proposed, among which the most popular are those of the Bayesian family of filters. This family includes the Kalman Filter, the Extended Kalman Filter, the Sequential Monte Cario filters, and others. Real-time posture estimation is also often addressed using a Bayesian filter.

The rigid body tracking system uses a custom filter that fuses information and can make predictions. The starting point is the position and orientation calculated in the table above. It's a good starting point, but you can certainly move around a bit between the camera frames (16.6 ms @ 60 Hz).

If the rigid body tracking system sees the posture on the last frames, some prediction can be made based on the speeds. Now we are a little closer. Fortunately, the rigid body tracking system also has an inertial measurement unit (IMU). This means that between the previous and current image, the rigid body tracking system actually has approximately 16 angular velocity and linear acceleration measurements. Our custom filter fuses all of this information, and provides history-based posture estimates, which are based on positions, speeds, and accelerations. The end result is pretty close to the actual stance in the current box.

The rigid body tracking system uses the predicted posture to project the Light Emitting Diodes (LEDs) onto the image plane. The rigid body tracking system then matches the predicted light emitting diode (LED) image coordinates with the observations. If the prediction is correct, the pairings are perfect. And finally, the rigid body tracking system calculates the necessary correction to the estimated posture, resulting in a perfect match.

Figure 10 illustrates the posture optimization step. A 2D horizontal plane is shown, where the optimal solution is the global minimum. The rigid body tracking system starts with a predicted posture based on history, speed, and acceleration. The solution is iteratively improved using gradient descent optimization, until the rigid body tracking system finds the optimal posture. Incremental posture estimation is computationally efficient, and is the way the rigid body tracking system expects to be most of the time. Once bootstrapping is successful, and as long as the virtual reality (VR) viewer is visible to the camera, the system can continually go through this process of: predicting posture - matching observations to refining posture.

The top three considerations of a rigid body tracking system are occlusion strength, efficient computing, and precision. The method is inherently robust to occlusion. During identification, the rigid body tracking system can recognize individual light emitting diodes (LEDs). One condition is seeing a few light-emitting diodes (LEDs) across multiple frames. Posture reconstruction is also robust to occlusion. The rigid body tracking system is capable of verifying a prediction and refining the posture estimate of the virtual reality (VR) viewer by viewing a few light-emitting diodes (LEDs) per frame.

The requirements for accuracy of head tracking in virtual reality (VR) are high. The rigid body tracking system provides the necessary stability (<0.1 millimeter and <0.1 degree) through careful design of the placement of light emitting diodes (LEDs) in the virtual reality (VR) viewer, and a Reliable synchronization between light emitting diodes (LEDs), inertial measurement unit (IMU) and camera. The merging of all the information together is what ultimately allows accurate prediction and uniform tracking.

An interesting property of position tracking is grounding. For example, the rigid body tracking system can now give meaning to looking ahead. It simply means looking at the camera. In contrast to gyroscopes, accelerometers, and magnetometers, vision does not drift. Consequently, with position tracking, the rigid body tracking system has a simple and reliable solution to drift correction.

Position tracking opens up some interesting questions. Scale is a good example. Would a user expect a 1 to 1 correlation between movement in the real world and in the virtual world? Probably yes, because any mismatch can alter the user's vestibular system. But then again, there are some use cases where speed control based on head displacement would be helpful - the farther the user is from the center, the faster the user moves. Another interesting question is how to motivate the user to stay in the field of view of the camera.

Another example is when a user leaves the field of view, the rigid body tracking system loses position tracking, but still has orientation tracking thanks to the inertial measurement unit (IMU). Every time the rigid body tracking system reacquires position tracking, it goes back from 3D to 6D. What should the rigid body tracking system do after re-entering 6D? Two possible solutions are snapping to the correct position as soon as vision is regained, and slowly interpolating to the correct position, but both solutions are troubling in different ways.

Areas of interest

A rigid body tracking system that achieves high precision and robustness to occlusion due to a high plurality of modulated light emitting diodes (LEDs) arranged on the body surface, where the additional components of the system are: at least one stationary digital camera, which is external to the body being tracked; an inertial measurement unit, which is rigidly attached to the body being tracked; and the digital hardware that receives and sends the information between the components of the inertial measurement unit, the camera, and the main CPU.

The rigid body tracking system, where inertial measurement contains a three-axis gyroscope to measure angular velocity.

The rigid body tracking system, where the inertial measurement contains a three-axis accelerometer.

The rigid body tracking system, where the inertial measurement can contain a three axis magnetometer.

The rigid body tracking system, where the surface light emitting diodes (LEDs) are arranged in a careful pattern that is not coplanar and provides sufficient separation between neighboring light emitting diodes (LEDs).

The rigid body tracking system, wherein the light emitting diodes (LEDs) are modulated in such a way that they can maintain one of two or more previously determined infrared brightness levels 10 for a desired time interval.

The rigid body tracking system, where the modulation of light emitting diodes (LEDs) is controlled by the digital hardware component.

The rigid body tracking system, where the modulation of light emitting diodes (LEDs) can be in amplitude, frequency, some combination, or by other means of signal encoding.

The rigid body tracking system, where light emitting diodes (LEDs) can be in the visible light spectrum or 20 in the infrared spectrum.

The rigid body tracking system, where the digital hardware component generates and records timestamps of less than one millisecond, for times when the camera shutter is open or the inertial measurement unit 25 provides a new measurement.

A method for rigid body tracking, useful in conjunction with a head-mounted display, comprising the head tracking method: updating body orientation estimates from high angular velocity measurements frequency (at least 1,000 Hz) obtained by a gyroscope in the inertial measurement unit; updating the estimated body position from images containing a uniquely identified subset of the light emitting diodes (LEDs); improve computational efficiency and estimation precision by tightly integrating measurements from both the inertial measurement unit and from one or more cameras; and provide estimates of body position and orientation, with the added ability of predicting future positions and orientations.

The method as mentioned above, where accelerometer measurements are used to compensate for navigation errors due to dead reckoning of the tilt orientation.

The method as mentioned above, where camera images, and possibly magnetometer measurements, are used to compensate for yaw orientation dead reckoning errors.

The method as mentioned above, where low-level image processing is performed to extract the locations of the centers of the light emitting diodes (LEDs) in the image with sub-pixel precision.

The method as mentioned above, where the modulation levels of the light emitting diodes (LEDs) provide a digitally encoded identifier on the consecutive images of the camera, thus solving the problem of identifying the light emitting diodes (LED).

The method as mentioned above, where the position and orientation estimates from the images are calculated incrementally from frame to frame.

The method as mentioned above, where gyroscope and accelerometer measurements are used during the time interval between consecutive shutter openings.

The method as mentioned above, where precise prediction estimates of position and orientation are made by combining the estimate in the table above with the cumulative measurements of the gyroscope and accelerometer.

The method as mentioned above, where a method iteratively disturbs the predictive estimation until the new position and orientation estimates optimize the error, which is the mismatch between the locations of the centers of the light-emitting diodes (LEDs). ) expected in the image and their locations measured in the image.

Compendium

The foregoing description of the modalities of description has been presented for the purpose of illustration; It is not intended to be exhaustive or to limit disclosure to the precise forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the foregoing description.

Parts of this description describe the modalities of disclosure in terms of algorithms and symbolic representations of operations on information. These descriptions and algorithmic representations are commonly used by experts in data processing techniques to effectively convey the substance of their work to other experts in this field. These operations, although described functionally, computationally or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Additionally, it has also proven convenient, at times, to refer to these operations configurations as modules, without losing the generality. The operations described and their associated modules can be incorporated into software, firmware, hardware, or any combination thereof.

Any of the steps, operations or processes described in this document can be carried out or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product that comprises a computer program code that contains a computer-readable medium, which can be executed by a computer processor to perform any or all of the steps , operations or processes described.

Disclosure modalities may also refer to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and / or may comprise a general purpose computing device selectively activated or reconfigured by a computer program stored in the computer. This computer program can be stored in a non-transient, tangible, computer-readable storage medium, or any type of means suitable for storing electronic instructions, which can be coupled to a busbar of a computer system. Additionally, any computing system referenced in the specification may include a single processor or may be architectures that employ multiple processor designs for increased computing power.

Disclosure modalities may also refer to a product that is produced through a computing process described in this document. This product may comprise information resulting from a computing process, where the information is stored on a non-transient, tangible, computer-readable storage medium, and may include any form of a computer program product or other combination of data. described in this document.

Finally, the language used in the specification has been selected primarily for ease of reading and for instructional purposes, and may not have been selected to delimit or circumscribe the subject matter of the invention. Accordingly, the scope of the disclosure is intended not to be limited by this detailed description, but rather by any claim arising out of a request based on this. Accordingly, the description of the modalities is intended to be illustrative, but not limiting, of the scope of the disclosure, which is stipulated in the following claims.

Contents6

33 members in 11 offices

This recordMember, by publication datePriority claim1 more office is in the list below

Priority claims11

Priority claims
Document	Office	Kind	Date
61923895	United States of America	–	2014-01-06
201461923895	United States of America	P	2014-01-06
62088085	United States of America	–	2014-12-05
62088088	United States of America	–	2014-12-05
201462088088	United States of America	P	2014-12-05
201462088085	United States of America	P	2014-12-05
14589755	United States of America	–	2015-01-05
14589774	United States of America	–	2015-01-05
201514589774	United States of America	A	2015-01-05
201514589755	United States of America	A	2015-01-05
2015010344	United States of America	W	2015-01-06

Members33

Family members, oldest first
Document	Office	Kind
CA2930773A1	Canada	A1	2015-07-09
US2015193949A1	United States of America	A1	2015-07-09
US2015193983A1	United States of America	A1	2015-07-09
WO2015103621A1	World Intellectual Property Organization (WIPO)	A1	2015-07-09
EP3017591A1	European Patent Office (EPO)	A1	2016-05-11
AU2015203932A1	Australia	A1	2016-06-09
CN105850113A	China	A	2016-08-10
KR20160105796A	Republic of Korea	A	2016-09-07
MX2016008908AThis record	Mexico	A	2016-09-16
US9524580B2	United States of America	B2	2016-12-20
JP2017503418A	Japan	A	2017-01-26
US2017053454A1	United States of America	A1	2017-02-23
US9600925B2	United States of America	B2	2017-03-21
JP6110573B2	Japan	B2	2017-04-05
AU2015203932B2	Australia	B2	2017-04-06
US2017147066A1	United States of America	A1	2017-05-25
MX348608B	Mexico	B	2017-06-21
CA2930773C	Canada	C	2017-07-11
KR20170086707A	Republic of Korea	A	2017-07-26
KR101762297B1	Republic of Korea	B1	2017-07-28
BR112016013563A2	Brazil	A2	2017-08-08
JP2017142813A	Japan	A	2017-08-17
IL245813A	Israel	A	2017-08-31
EP3017591A4	European Patent Office (EPO)	A4	2017-09-27
US9779540B2	United States of America	B2	2017-10-03
CN105850113B	China	B	2018-02-13
CN108107592A	China	A	2018-06-01
US10001834B2	United States of America	B2	2018-06-19
JP6381711B2	Japan	B2	2018-08-29
EP3017591B1	European Patent Office (EPO)	B1	2018-12-12
CN108107592B	China	B	2019-11-05
BR112016013563A8	Brazil	A8	2020-05-19
KR102121994B1	Republic of Korea	B1	2020-06-11