Gaze-based object placement within a virtual reality environment

Virtual reality computing devices such as head-mounted display (HMD) systems and handheld mobile devices (eg, smartphones, tablet computers, etc.) are in the field of view of the user and / or in the field of view of the device's camera. It can be configured to display the virtual reality environment to the user. Similarly, mobile devices may use the camera's viewfinder window to display such information.

A description of this background technique is provided to provide a brief background on the outline and detailed description of the invention that follows. This background is not intended to be used as an aid in determining the scope of the claims, and the claims may be any or all of the shortcomings or problems presented above. It is also not intended to be considered as limiting to the implementation that resolves.

An HMD device operating in a real-world physical environment can position the virtual object in the virtual reality environment with respect to the device user's projected gaze so that the virtual object can be placed in the environment with high accuracy. Consists of a sensor package that allows you to determine the intersection of. Data from the sensor package can be used to apply surface reproduction of the physical environment to determine the user's view position in the virtual world. The line of sight originating from the view position is projected outwards, and the cursor or similar indicator is placed on the HMD display at the closest intersection of the line of sight with the virtual world, such as a virtual object, floor / ground, etc. Rendered. In response to user input, such as gestures, voice dialogue, or manipulation of controls (eg, buttons or other user interface objects), the virtual object is placed at the intersection between the projected gaze and the virtual reality environment.

This gaze-based virtual object placement can be used in a variety of applications and usage scenarios, and by changing the view position and / or gaze, the user can easily place the virtual object at a desired position in the virtual environment. Allows you to. For example, a user of an HMD device can place a marker indicating a point of interest on the terrain of a virtual world in a survey or map application, or in a game scenario, a user can play a virtual object such as an avatar or a game object. It can be placed in a specific position in the virtual world as part of play.

A brief description of the present invention is provided to provide a simplified description of the selection of concepts further described below in the detailed description. The description of the present invention is not intended to identify the main or essential features of the claimed subject matter and is used as an aid in determining the scope of the claimed subject matter. Not even intended. Moreover, the subject matter of the claims is not limited to implementations that resolve any or all of the shortcomings presented anywhere in the disclosure. It may be recognized that the subject matter described above may be implemented as a computer controlled device, computer process, computing system, or as a product such as one or more computer readable storage media. These and various other features may become apparent by reading the detailed description below and looking at the relevant drawings.

<figref num="1">It is a figure which shows the example virtual reality environment, and a part of the virtual reality environment is rendered in the field of view of the user of an HMD device.</figref>

<figref num="2">It is a figure which shows the example real-world environment where the user of the HMD device is placed.</figref>

<figref num="3">It is a figure which shows the example gaze line projected from the view position of the HMD device which intersects a point in a virtual world.</figref>

<figref num="4">It is a figure which shows a mode that a virtual object is arranged at an intersection in response to a user input.</figref>

<figref num="5">It is a figure which shows how the cursor is rendered at the intersection of the projected gaze lines in the virtual reality environment in the field of view of the HMD device user.</figref>

<figref num="6">It is a figure which shows how the example virtual object is arranged at the intersection of the projected gaze lines in a virtual reality environment.</figref>

<figref num="7">It is a figure which shows how the cursor is rendered at the intersection of the projected gaze lines in the virtual reality environment in the field of view of the HMD device user.</figref>

<figref num="8">It is a figure which shows how the example virtual object is arranged at the intersection of the projected gaze lines in a virtual reality environment.</figref>

<figref num="9">It is a figure which shows the appearance that the projected gaze line of an example is rotated downward.</figref>

<figref num="10">It is a figure which shows the appearance that the projected gaze line of an example is rotated upward.</figref>

<figref num="11">It is a figure which shows the example large virtual object clipped by the user's field of view.</figref>

<figref num="12">FIG. 5 shows a large virtual object placed in a virtual world using a rotated gaze so that the object is not clipped to the top edge of the field of view.</figref>

<figref num="13">It is a figure which shows the example virtual object clipped by the user's field of view.</figref>

<figref num="14">FIG. 5 shows a virtual object placed in a virtual world using a rotated gaze so that the object is not clipped to the lower edge of the field of view and is placed more centrally in the field of view.</figref>

<figref num="15">It is a figure which shows how the surface reproduction data associated with a real world environment is captured by an HMD device.</figref>

<figref num="16">It is a figure which shows the exemplary user interface supported by the HMD device and the exemplary data provided by the HMD sensor package.</figref>

<figref num="17">It is a block diagram of an exemplary surface reproduction pipeline.</figref>

<figref num="18">FIG. 6 is a flow chart of an exemplary method that can be performed using an HMD device.</figref><figref num="19">FIG. 6 is a flow chart of an exemplary method that can be performed using an HMD device.</figref><figref num="20">FIG. 6 is a flow chart of an exemplary method that can be performed using an HMD device.</figref>

<figref num="21">It is a perspective view of the Example of a virtual reality HMD device.</figref>

<figref num="22">It is a functional block diagram of an embodiment of a virtual reality HMD device.</figref>

<figref num="23">FIG. 3 is a perspective front view of an exemplary enclosed visor that can be used as a component of a virtual reality HMD device.</figref><figref num="24">FIG. 3 is a perspective front view of an exemplary enclosed visor that can be used as a component of a virtual reality HMD device.</figref>

<figref num="25">It is a figure of the closed type visor when it is partially disassembled.</figref>

<figref num="26">It is a front view of the very thin line of a closed type visor.</figref>

<figref num="27">It is a perspective rear view of the closed type visor.</figref>

<figref num="28">It is a figure which shows the example computing system.</figref>

Similar reference numbers indicate similar elements in the drawings. Elements are not scaled and drawn unless otherwise specified.

Users typically explore, navigate, and move within a virtual reality environment rendered by an HMD device by moving within the corresponding real-world physical environment (eg, through some form of movement). Can be done. In the explanatory example as illustrated in FIG. 1, the user 102 can experience the virtual reality environment 100 with the HMD device 104. The virtual reality environment 100 is visually rendered in three dimensions (3D) and may include audio and / or touch / tactile sensations in some implementations. In this particular non-limiting example, an application running on the HMD device 104 supports a virtual reality environment 100, including streets, along with various buildings, stores, and the like. As the user repositions or orients his head within the physical real world environment 200 illustrated in FIG. 2 and / or moves, the user's view of the virtual reality environment 100 may change. The field of view (represented by the dashed area 100 in Figure 1) can be adjusted in size and shape, so that the device is visually immersed in the HMD device experience to provide the user with a strong presence in the virtual world. Other features can be controlled. Although virtual reality environments are illustrated and described herein, this gaze-based object placement can also be applied to mixed reality environments and scenarios.

During the course of a given user experience with the HMD device 104, various new virtual objects can be introduced into the virtual reality environment. Can be used to enhance or control the experience, such as objects that are essential to the experience, such as avatars, terrains, markers, flags, buildings, etc., or interactive elements, including menus, widgets, notifications, etc. Any of a given user interface supported on the HMD device, including objects, or any of a variety of objects that can be used to facilitate the user experience can be introduced.

As illustrated in FIG. 3, the user's view position and / or gaze direction of the HMD device can be used to place virtual objects within the virtual world rendered on the HMD display. The gaze line 302 is projected from the view position of the HMD device 104. The view position begins between the eyes of user 102 and the point of the forward line (ray) 302 as shown. The line is cast into the virtual reality environment 100, and the closest intersection 304 with the virtual reality environment 100 is determined. Cursor 305 or other suitable indicator is typically displayed at an intersection. As illustrated in FIG. 4, when the HMD device 104 receives the input 402 from the user, the device can place the virtual object 405 at the intersection. The user input 402 may vary from implementation to implementation and may include, for example, sensing gestures, voice commands or language inputs, physical or virtual control operations supported by the HMD device, and the like.

In some implementations, the gaze can be projected so that the cursor can be rendered at any point within the field of view 110. For example, in a typical scenario, the cursor can be placed in a fixed location in or near the center of the field of view. Since the field of view and view position are typically determined by tracking the position and orientation of the user's head (as described in more detail below), the user is required to adjust the field of view. You can place the cursor on the object of interest in the virtual world using only the movement of the part.

In an alternative implementation, the HMD device may be configured to project a gaze line to allow the cursor to be rendered at different positions within a given field of view, depending on the position of the user's eyes. .. Eye position can be detected using, for example, an inward facing sensor that can be incorporated into the HMD device 104. Such eye position detection is called gaze tracking and is described in more detail below. Thus, the user can place the cursor by using a combination of head and eye movements in these alternative implementations. For example, the user may use the movement of the head to place the virtual object of interest in the field of view, and then use the movement of the eyes to place the cursor over the object.

The HMD device is also configured to provide the user with the option to choose between fixed and variable cursor positions, for example by disabling or enabling eye tracking, respectively, in some cases. obtain. For example, a cursor fixed to the field of view may be suitable for some usage scenarios that provide an effective way to position the cursor solely by head movement. In other usage scenarios, the combination of head and eye movements may be more advantageous. In general, the use of either fixed or variable cursor placement can make it possible to place virtual objects in a virtual world with high accuracy.

FIG. 5 shows the cursor 305 rendered at the intersection of gazes projected into the virtual reality environment 100 within the field of view 110 of the user of the HMD device (user not shown for clarity of disclosure). Is illustrated. Typically, the cursor is dynamically rendered in 3D on the HMD display using a size proportional to the distance of the cursor from the user in the virtual world (ie, the closer it is, the larger it is, and the further it is, the smaller it is). Will be done. FIG. 6 illustrates an exemplary virtual object 605 placed at the intersection of a projected gaze and a virtual reality environment in response to user input such as gestures, voice commands or button presses. Virtual object 605 is intended to be exemplary and the rendered virtual object may differ in size, shape and position in the field of view from those shown.

FIG. 7 illustrates another concrete example of the cursor 305 rendered at the intersection between the projection line and the virtual reality environment 700, where the virtual reality environment 700 is undulating and wavy. Includes outdoor views with terrain with hills. FIG. 8 illustrates flag 805, which is an exemplary virtual object placed at an intersection.

In a typical implementation, the field of view 110 for the HMD device 104 may be relatively limited. Therefore, the virtual object introduced in the virtual reality environment 700 may not fit the user's field of view or may be clipped at the edge of the field of view. This can occur, for example, when the user interacts with a relatively large virtual object, which can be offensive to the user and / or reduce the quality of the user experience. In other situations, a given virtual object may extend from the intersection to cause clipping to the edge of the field of view, and / or the user experience is when the virtual object is placed more centered in the field of view. Can be more efficient.

Therefore, in some implementations, the HMD device 104 may be configured to rotate the line of sight from the view position from the original projection. Such rotations are generally performed in any direction (eg, left, right, up, down or a combination thereof) to fit a given use case, and the angle of rotation can be varied. As illustrated exemplary in FIG. 9, the rotation may be directed downwards (in FIG. 9, the original gaze is indicated by reference number 905 and the rotated gaze is indicated by reference number 910). ). As illustrated exemplary in FIG. 10, the rotation may be directed upwards (in FIG. 10, the original gaze is indicated by reference number 1005 and the rotated gaze is indicated by reference number 1010). ..

FIG. 11 shows an exemplary virtual object 1105. The illustrated virtual object 1105 is relatively large and is clipped relative to the top of the field of view 110 when placed at the intersection of the original gaze 905 (FIG. 5) and the virtual reality environment 700. For comparison, FIG. 12 illustrates a virtual object 1105 when placed at the intersection of a downwardly rotated gaze 910 and a virtual reality space 700. In a typical implementation, the projection line is rotated sufficiently downward so that it is cast through the lower part of the field of view 110. As shown, the virtual object 1105 is not clipped to the top edge of the field of view 110 because there is so much margin in the field of view.

FIG. 13 illustrates an exemplary virtual object 1305 arranged as a banner hanging from the intersection of the original gaze line 1005 (FIG. 10) and the clouds in the virtual reality environment 700. As shown, the object 1305 is clipped at the bottom edge of the field of view 1100, with some of the content on the banner (as represented by the geometry) outside the field of view. Also, the content may not be effectively presented when displayed in the lower part of the field of view, which in some cases may cause user discomfort. For comparison, FIG. 14 illustrates a virtual object 1305 located at the intersection of the upwardly rotated gaze line 1010 and the virtual reality environment 700. As shown, the virtual object is not clipped to the bottom edge of the field of view 110 and the banner content is centered more in the field of view, which enhances the effectiveness of the user experience. User comfort can be improved in some situations.

The HMD device 104 is configured to acquire surface reproduction data 1500 by sensing the user's position in the physical environment using the integrated sensor package 1505, as illustrated in FIG. As described in more detail below, the sensor package can include a depth sensor or depth sensing camera system. In an alternative implementation, depth data can be derived using appropriate stereoscopic image analysis techniques.

As illustrated in FIG. 16, the sensor package 1505 can support a variety of functions, including surface reproduction 1610. 3D of the user's head within the physical real-world environment 200, including head posture, so that surface reproduction can be used, for example, for head tracking and to determine the view position of the virtual world. The (three-dimensional) position and orientation 1615 can be determined. The sensor package can also support gaze tracking 1620 to identify the user's gaze direction 1625. The user's gaze direction 1625 can be used with head position and orientation data. The HMD device 104 can also be configured to expose a user interface (UI) 1630 that can not only display system messages, prompts, etc., but also present controls that can be manipulated by the user. The control may be virtual or physical in some cases. The UI1630 can also be configured to operate with gestures and voice sensed using, for example, voice commands or natural language.

FIG. 17 illustrates an exemplary surface reproduction data pipeline 1700 for acquiring surface reproduction data for the real world environment 200. It should be emphasized that the disclosed techniques are exemplary and that other techniques and methods may be utilized depending on the specific implementation requirements. Input the raw depth sensor data 1702 into the sensor's 3D pose estimate (block 1704). Sensor pose tracking can be achieved, for example, by using an ICP (iterative closest point) alignment between the predicted surface and the current sensor measurements. Each depth measurement of the sensor can be integrated into a volumetric representation, for example using a surface encoded as a signed distance field (SDF) (block 1706). Using a loop, the SDF is raycast to an estimated frame (block 1708) and the depth map is aligned with the dense surface. Prediction) is provided. Therefore, when the user 102 looks around the virtual world, the surface reproduction data associated with the real world environment 200 can be collected and analyzed to determine the position and orientation of the user's head.

18, 19, and 20 are flowcharts of the exemplary method. Unless otherwise stated, the methods or steps shown in the flowchart and described in the accompanying text are not restricted to a particular order or order. In addition, some of these methods or steps can occur at the same time or be performed at the same time, and not all methods or steps need to be performed, depending on the implementation requirements. The method or step may be optionally utilized.

The method 1800 illustrated in FIG. 18 can be performed by an HMD device that supports rendering in a virtual reality environment. In step 1805, sensor data is acquired that describes the physical environment in the real world adjacent to the user of the HMD device. The sensor data can include, for example, depth data using a depth sensor integrated into the HMD device, or can be obtained from an external sensor or source. Depth-to-stereo imaging analyzes may be used to create depth data. In step 1810, sensor data is used to reproduce the geometry of the physical environment, for example using surface reproduction.

In step 1815, the reproduced shape of the physical environment is used to perform tracking of the user's head to determine the current field of view and view position of the virtual reality environment. In step 1820, the gaze line is projected outward from the view position along the gaze direction of the user. Gaze detection may also be implemented using, for example, an inward sensor built into the sensor package 1505 (FIG. 15).

At step 1825, the intersection between the projected gaze and the virtual reality environment is identified. At step 1830, the HMD device exposes the user interface and receives user input. The user interface can be configured according to the needs of a given implementation and may include physical or virtual controls that can be manipulated by the user, such controllers in some cases voice and / or Can support gestures. In step 1835, for example, when a virtual object is too large to fit in the field of view, in some cases the projection line is rotated downwards to some virtual along the top edge of the field of view. Clipping of objects (eg large virtual objects) can be avoided. In some cases, the rotated projection line is cast through the lower part of the field of view. At step 1840, virtual objects can be placed at intersections in response to user input.

The method 1900 illustrated in FIG. 19 is a display for rendering a virtual reality environment using one or more processors and a variable field of view, a sensor package, and software code that can be used to implement the method. It can be performed by an HMD device that has one or more memory devices that store computer-readable instructions such as. In step 1905, surface reproduction data is generated using the sensor package built into the HMD device. The sensor package may include a depth sensor or camera system. In step 1910, surface reproduction data is used to dynamically track the user's view position in a virtual reality environment. A variety of suitable surface reproduction techniques are available, including those illustrated in the pipeline of FIG. 17, in which multiple overlapping surfaces are integrated.

In step 1915, the sensor package is used to dynamically track the user's gaze direction, for example using an inward sensor. In step 1920, locate the intersection between a ray projected from a view position along the user's gaze direction and a point in the virtual reality environment in the field of view. In step 1925, the HMD device is manipulated to render the cursor at the intersection found. Rendering is typically performed dynamically so that the cursor position in the virtual reality environment is updated as the user moves and the field of view and gaze direction change. In some cases, where the virtual object to be placed will be clipped in the field of view, in step 1930, when the object is placed at the intersection of the rotated line and the virtual reality environment, the object is in the current field of view. The projection line can be rotated downward so that it is not clipped by the top edge of. In step 1935, the HMD device is manipulated to place the virtual object at the intersection found. In a typical implementation, this placement is done in response to user input to the UI exposed by the HMD device.

Method 2000 of FIG. 20 can be performed by instructions stored on an HMD device that is operating in a real world environment and has a display that renders a virtual reality environment. In step 2005, a surface reproduction model of the real world environment is dynamically generated using data from the sensor package on the HMD device. As the user travels within the real-world environment, the model can be updated, for example, frame by frame or in other appropriate units. In step 2010, the data from the sensor package is used to generate a gaze line projected from the user's view position along the direction of the user's gaze.

In step 2015, the current surface reproduction model is used to determine the user's current field of view in the virtual reality environment. In step 2020, the cursor is rendered at the intersection between the projected gaze and the virtual reality environment. In step 2025, the user input is received, for example, in the UI exposed by the HMD device. In step 2030, if the virtual object to be placed extends out of the field of view, the projected gaze can be rotated downwards. In some implementations, the degree of rotation is sufficient to allow the virtual object to be placed so that the space above the object in the field of view is not obstructed. In step 2035, the virtual object is placed at the intersection in response to the received user input.

Next, moving on to the details of various exemplary implementations, virtual reality or mixed reality display devices with this configuration are near-, such as, but not limited to, HMD devices 104 and / or other portable / mobile devices. eye) It may take any suitable shape, including the device. FIG. 21 shows one particular embodiment of the see-through mixed reality display system 2100, and FIG. 22 shows a functional block diagram of the system 2100. However, while some implementations may use see-through displays, others have opaque (ie, non-see-through) displays that use, for example, camera-based pass-through or outward facing sensors. It should be emphasized that it may be used.

The display system 2100 uses a lens 2102 for images (eg, using projection onto the lens 2102, using one or more waveguide systems incorporated into the lens 2102, and / or any other. It comprises one or more lenses 2102 that form part of the see-through display subsystem 2104 so that it can be displayed (in the appropriate manner). The display system 2100 further comprises one or more outward image sensors 2106 configured to capture images of the background scene and / or physical environment being viewed by the user, such as voice commands from the user. It may include one or more microphones 2108 that are configured to detect a sound. The outward-facing image sensor 2106 may include one or more depth sensors and / or one or more two-dimensional image sensors. In an alternative configuration, instead of incorporating a see-through display subsystem, as described above, a virtual reality or mixed reality display system displays a mixed reality image through a viewfinder mode for an outward-facing image sensor. May be good.

The display system 2100 may further include a gaze detection subsystem 2110 configured to detect the gaze direction or focus direction or position of each eye of the user, as described above. The gaze detection subsystem 2110 may be configured to determine the gaze direction of each of the user's eyes in any suitable manner. For example, in the illustrated embodiment, the gaze detection subsystem 2110 is one or more glint sources configured to reflect the glint of light from each user's eye, such as an infrared light source. ) 2112 and one or more image sensors 2114 configured to capture an image of each user's eyeball, such as an inward facing sensor. Changes in brightness from the position of the user's eyeball and / or the user's pupil, as determined from the image data collected using the image sensor 2114, can be used to determine the gaze direction.

In addition, the position where the gaze projected from the user's eyes intersects the external display is used to determine the object the user is gaze at (eg, the displayed virtual object and / or the actual background object). Can be done. The gaze detection subsystem 2110 may have any suitable number and arrangement of light sources and image sensors. In some implementations, the gaze detection subsystem 2110 may be omitted.

The display system 2100 may include additional sensors. For example, the display system 2100 may include a Global Positioning System (GPS) subsystem 2166 to allow the display system 2100 to be positioned. This can help identify real-world objects such as buildings that can be placed within the user's adjacent physical environment.

The display system 2100 is one or more motion sensors 2118 (eg, inertia) that detect the movement and position / orientation / orientation of the user's head when the user is wearing the system as part of an augmented reality HMD device. Sensors, multi-axis gyroscope sensors or accelerometers) may be further included. Motion data is potentially eye-tracking glow data and outward-facing for gaze detection and for image stabilization that helps correct blurring in the image from the outward-facing image sensor 2106. May be used with image data. The use of motion data makes it possible to track changes in gaze position, even if the image data from the outward image sensor 2106 cannot be resolved.

In addition, motion sensors 2118, as well as microphones 2108 and gaze detection subsystem 2110, may also be used as user input devices, allowing the user to use through eye, neck and / or head gestures, and in part. In the case of, it is possible to interact with the display system 2100 via verbal commands. The sensors illustrated in FIGS. 21 and 22 and described in the accompanying text are included for illustrative purposes and are any other suitable sensors to meet the specific implementation needs of the augmented reality HMD device. It can be understood that the combination of and / or sensors can be used and is not intended to be limiting in any way. For example, in some implementations, a biosensor (eg, for detecting heart rate and respiratory rate, blood pressure, brain activity, body temperature, etc.) or a biosensor (eg, for detecting temperature, humidity, altitude, UV (ultraviolet) light level, etc.) ) An environment sensor may be used.

The display system 2100 can further include a controller 2120 with a logical subsystem 2122 and a data storage subsystem 2124, which through the communication subsystem 2126 are sensors, gaze detection subsystem 2110, display subsystem 2104 and /. Or communicate with other components. Communication subsystem 2126 can also facilitate the display system to operate in conjunction with remotely located resources such as processing, storage, power, data and services. That is, in some implementations, the HMD device can operate as part of a system that can distribute resources and capabilities among different components and subsystems.

The storage subsystem 2124, for example, receives and interprets input from sensors, identifies the user's position and movement, uses surface reproduction and other techniques to identify real objects, and among other tasks, objects. Instructions that can be executed by the logical subsystem 2122 may be included to dim / fade the display based on the distance to the object so that it can be seen by the user.

The display system 2100 is configured with one or more audio transducers 2128 (eg speakers, earphones, etc.) so that audio can be used as part of an augmented reality experience. The power management subsystem 2130 may also include one or more batteries 2132 and / or protection circuit modules (PCM), as well as associated charging interfaces 2134 and / remote power interfaces for powering components within the display system 2100. Good.

It will be appreciated that the displayed display devices 104 and 2100 shown are shown for purposes of illustration and are therefore not intended to be limiting. It is also understood that the display device may include additional and / or alternative sensors, cameras, microphones, input devices, output devices, etc. to those illustrated without departing from the scope of this configuration. Yeah. In addition, the physical configuration of the display device and its various sensors and subcomponents may take a variety of different forms without departing from the scope of this configuration.

23-27 show an exemplary alternative implementation of the virtual or mixed reality display system 2300 that can be used as a component of an HMD device. In this example, the system 2300 uses a see-through sealed visor 2302, which is configured to protect the internal optical assembly utilized in the see-through display subsystem. The visor 2302 typically has an HMD, such as a head-mounted / holding system and other subsystems, including sensors, power management, controllers, etc., as exemplified in connection with FIGS. 21 and 22. Interfaces other components of the device (not shown). Appropriate interface elements (not shown), including snaps, protrusions and other fasteners, can also be incorporated within the visor 2302.

Visors include see-through front shield 2304 and back shield 2306. These shields can be formed using transparent material to facilitate unobstructed vision of the optical display and the surrounding real-world environment. Treatments such as coloring, mirroring, antireflection, anti-fog and other coatings may be applied to the front and back shields, and various colors and finishes may be used, the front and back shields are shown in FIG. 24. Mounted on chassis 2405 as shown in the figure partially presented in. In FIG. 24, the shield cover 2410 is shown to be disassembled from the visor 2302.

The enclosed visor 2302 is delicate, including the optical display subassembly 2502 (shown in the exploded view of FIG. 25) when the HMD device is attached and used during operation and normal handling for cleaning. Internal components can be physically protected. The visor 2302 can also protect the optical display subassembly 2502 from damage that could cause environmental elements and HMD devices to be dropped, bumped, impacted, and the like. The optical display subassembly 2502 is mounted within the sealed visor so that the shield does not contact the subassembly when deflected by a drop or impact.

As illustrated in FIGS. 25 and 27, the back shield 2306 is ergonomically configured to interface with the user's nose and nose pad 2704 (FIG. 27) and includes other comfort features ( For example, it can be molded in and / or added as a separate component. The sealed visor 2302 can also incorporate some level of optical diopter curvature (ie, eye formulation) into the molded shield in some cases.

FIG. 28 schematically illustrates a non-limiting embodiment of a computing system 2800 that can be used when implementing one or more of the configurations, arrays, methods or processes described above. The HMD device 104 can be a non-limiting example of the computing system 2800. The computing system 2800 is illustrated in a simplified form. It can be understood that virtually any computer architecture may be used without departing from the scope of this configuration. In different embodiments, the computing system 2800 is a display device, a wearable computing device, a machine frame computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a home entertainment computer, a network computing device, a game device, a mobile compute. It may take the form of a computing device, a mobile communication device (for example, a smartphone), or the like.

The computing system 2800 includes a logical subsystem 2802 and a storage subsystem 2804. The computing system 2800 may optionally include display subsystem 2806, input subsystem 2808, communication subsystem 2810 and / or other components not shown in FIG. 28.

The logical subsystem 2802 contains one or more physical devices that are configured to execute instructions. For example, the logical subsystem 2802 may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures or other logical structures. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, or otherwise achieve the desired result.

The logical subsystem 2802 may include one or more processors configured to execute software instructions. Alternatively, the logical subsystem 2802 may include one or more hardware or firmware logical machines configured to execute hardware or firmware instructions. The processor of the logical subsystem 2802 may be single-core or multi-core, and the programs executed on the processor may be configured for sequential processing, parallel processing, or distributed processing. The logical subsystem 2802 may optionally include individual components distributed among two or more devices that can be remotely deployed and / or configured for collaborative processing. Aspects of the logical subsystem 2802 can be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

Storage subsystem 2804 includes one or more physical devices configured to hold data and / or instructions that can be executed by logical subsystem 2802 to perform the methods and processes described herein. .. Once such methods and processes are implemented, the state of storage subsystem 2804 can be transformed-eg to hold different data.

Storage subsystem 2804 may include removable media and / or internal devices. The storage subsystem 2804 is, among other things, optical memory devices (eg CD (compact disc), DVD (digital versatile disc), HD-DVD (high resolution DVD), Blu-ray® disc, etc.), Semiconductor memory devices (eg RAM (random access memory), ROM (read-only memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable ROM), etc.) and / or magnetic memory devices (eg hard disk drives, floppy disks) It can include drives, tape drives, MRAM (Magnetoresistive RAM), etc.). Storage subsystem 2804 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable and / or content addressable devices. ..

As can be recognized, the storage subsystem 2804 includes one or more physical devices, excluding the propagating signal itself. However, in some implementations, the aspects of the instructions described herein use a communication medium to create a pure signal (eg, an electromagnetic signal, an optical signal, etc.) as opposed to being stored on a storage device. ) May be propagated. In addition, data and / or other forms of information regarding this configuration may be propagated by pure signals.

In some embodiments, the aspects of the logical subsystem 2802 and the storage subsystem 2804 may be integrated together into one or more hardware logical components in which the functions described herein can be established. Such hardware logic components include, for example, field programmable gate arrays (FPGAs), application-specific integrated circuits (PASIC / ASIC), application-specific and application-specific standards (PSSP / ASSP), and system-on-chip (system-on-chip). It may include a SOC) system and a coupled programmable logic device (CPLD).

The display subsystem 2806 can be used to present a visual representation of the data held by the storage subsystem 2804. This visual representation may take the form of a graphical user interface (GUI). When the methods and processes described change the data held by the storage subsystem and therefore transform the state of the storage subsystem, the state of the display subsystem 2806 also sees the change in the underlying data. Can be transformed to represent. The display subsystem 2806 can include one or more display devices that use virtually any type of technology. Such display devices may in some cases be combined with the logical subsystem 2802 and / or storage subsystem 2804 in a shared enclosure, or in other cases such display devices may be peripherals. It may be a display device of.

The input subsystem 2808, when included, may include or interface with one or more user input devices such as a keyboard, mouse, touch screen or game controller. In some embodiments, the input subsystem may include or interface with selected Natural User Input (NUI) components. Such components may be integrated or peripherals, and the conversion and / or processing of input actions may be processed onboard or offboard. Illustrated NUI components are microphones for speech and / or speech recognition; infrared, color, stereoscopic and / or depth cameras for machine vision and / or gesture recognition; heads for motion detection and / or intent recognition. It may include trackers, eye trackers, accelerometers and / or gyroscopes; as well as electric field sensing components for assessing brain activity.

The communications subsystem 2810, when included, may be configured to communicatively combine the computing system 2800 with one or more other computing devices. The communication subsystem 2810 may include wired and / or wireless communication devices that are compatible with one or more different communication protocols. As a non-limiting example, a communication subsystem may be configured for communication over a wireless telephone network or a wired or wireless local or wide area network. In some embodiments, the communication subsystem allows the computing system 2800 to send and / or receive messages from other devices using a network such as the Internet. Can be.

Various exemplary embodiments of gaze-based object placement in a virtual reality environment according to the present invention are presented below as examples rather than a comprehensive list of all embodiments. One example includes a method performed by a head-mounted display (HMD) device that supports rendering of a virtual reality environment: the method: sensor data showing the physical environment of the real world adjacent to the user of the HMD device. With the steps to acquire; with the steps to reproduce the shape of the physical environment using sensor data; with the recreated shape to track the user's head and gaze within the physical environment, the field of view and view The step of determining the position; the step of projecting the gaze outward from the view position; the step of identifying the intersection between the projected gaze and the virtual reality environment; the current field of view in response to user input. Includes steps to place virtual objects at intersections within;

In another example, the sensor data includes depth data, further including the step of generating sensor data using a depth sensor and applying surface reproduction techniques to reproduce the shape of the physical environment. In another example, the above method is depth-from-stereo imaging. It further includes the step of generating depth data using analyses). In another example, the method further comprises identifying the intersection between the projected gaze and the virtual reality environment, which is closest to the HMD device. In another example, the method further comprises exposing a user interface (UI) for receiving user input, which provides user control or supports gesture recognition or speech recognition. In another example, the method further comprises rotating the projected gaze so that the placed virtual object is not clipped along the edge of the field of view. In another example, the method further comprises performing a rotation such that the line of sight is projected through the lower part of the field of view. In another example, the method further comprises performing a rotation when the object to be placed cannot be rendered to fit as a whole in the field of view. The virtual object placed in another example is an interactive element that supports the user interface or user experience, which is one of a menu, widget, or notification. In another example, the method uses one or more inward sensors located on the HMD device to determine the gaze direction and project the gaze line from the view position along the user's gaze direction. Is further included.

A further example includes a head-mounted display (HMD) device that can be operated by the user in a physical environment, the HMD device being with one or more sensors; for rendering a virtual reality environment to the user. A display, in which the field of view of the virtual reality environment to be rendered changes, at least in part, depending on the orientation of the user's head in the physical environment, with the display; with the sensor package; And when run by one or more processors, the step of using the sensor package to generate surface reproduction data for at least part of the physical environment and the user using the surface reproduction data for the virtual reality environment. The step of dynamically tracking the view position of the computer and the step of locating the line projected from the view position along the user's gaze direction and the point of the virtual reality environment in the current field of view. It comprises one or more memory devices that store computer-readable instructions that perform methods including manipulating the HMD device and rendering the cursor at the intersection.

In another example, a head-mounted display (HMD) device further comprises using a sensor package to dynamically track the gaze direction. In another example, the head-mounted display (HMD) device further includes a user interface (UI) that manipulates the HMD device to place virtual objects at intersections in response to user input to the UI. Including doing. In another example, a head-mounted display (HMD) device further comprises dynamically repositioning the cursor as the field of view or gaze direction changes. In another example, a head-mounted display (HMD) device also uses a surface reproduction data pipeline that implements a volumetric method to create multiple overlapping surfaces. Includes modeling the physical environment. In another example, a head-mounted display (HMD) device also has a projected line down so that when rendered on the display, the virtual object being placed is not clipped by the top edge of the current field of view. Further includes rotating.

A further example is one or more computers readable that store computer-executable instructions for rendering a virtual reality environment within the variable field of view of a head-mounted display (HMD) device located within the real-world environment. It includes memory and the method uses data from the sensor package embedded in the HMD device to dynamically generate a surface reproduction model of the real world environment and b) project it from the user's view position on the HMD device. The steps to generate the gaze to be done; the step to determine the field of view of the virtual reality environment using the model; the step to receive the input to the HMD device from the user; in response to the received user input, in the field of view In, a step of arranging a virtual object at an intersection between the line of sight and the virtual reality environment is provided.

In another example, one or more computer-readable memory has a step of tracking the user's gaze direction, a step of projecting a gaze from a view position along the gaze direction, and a gaze and virtual reality in the field of view. It further includes a step of rendering the cursor at the intersection with the environment. In another example, the one or more computer-readable memory further includes a step of determining when the virtual object being placed extends out of the field of view, and a step of rotating the gaze in response to that determination. In another example, one or more computer-readable memory further comprises rotating the line of sight at an angle sufficient to provide a visual field portion that is not blocked by the virtual object being placed.

Although the subject matter is described in terms specific to structural features and / or methodical behaviors, the subject matter defined in the appended claims may not necessarily be limited to the particular features or behaviors described above. Will be understood. Rather, the particular features and behaviors described above are disclosed as exemplary forms that implement the claims.