Virtual measurement tool for a wearable visualization device

[0001] At least one embodiment of the present invention relates to display-related technology, and more particularly to virtual measurement tools for wearable visualization devices such as augmented reality or virtual reality display devices.

[0002] For thousands of years, humans have invented and relied on various types of measurement tools to quantify and better understand their environment. For example, rulers have been relied on for centuries to measure relatively short spatial distances. The tape measure is a modern adaptation of the ruler, and more recently, laser rulers and other active measuring tools have been invented.

[0003] However, simple spatial measurement tools, such as conventional rulers, tape measures, and laser rulers, which are affordable to the average person, have certain drawbacks. For example, these tools lack the ability to make more complex measurements such as area and volume measurements. Also, in many situations, one may want to measure an object in one place and determine if the object fits in another. For example, a person may want to buy a piece of new furniture for his home. Typically, in that situation, the person measures the available space at home, then goes to a furniture store, measures some of the furniture of interest, and sees if those furniture fit in that space. (And vice versa). In that case, this person needs to either remember or write down the dimensions of this available space (or this furniture product), which is inconvenient.

<p num="0004">[0004] The technology presented here is virtual in wearable visualization devices such as headsets, glasses, or goggles equipped to provide users with augmented reality and / or virtual reality ("AR / VR") experiences. Includes techniques for generating and displaying measurement tools (also referred to simply as "tools" in the discussion below). In certain embodiments, the device is in a three-dimensional (3D) space in which the user is based, for example, by using gesture recognition, eye tracking, speech recognition, or any combination thereof. Generate tools by determining multiple points at different locations in an environment (eg, a room). The device presents the tool to the user so that it appears to the user in real time with a real-world view.</p>

<p num="0005">[0005] In various embodiments, the tool may be displayed to the user as a holographic ruler or similar measurement tool. The points used to define the tool can be pinned to different points in 3D space so that the tool is fixed to the user in space, even when the user moves in that 3D space. It seems to stay in the same place and orientation. At least one of the points can be fixed to a corresponding point on the physical object. For example, with gesture recognition, line-of-sight tracking, and / or speech recognition, the user can use tools in any of six degrees of freedom (eg, in translation along or around any of the three orthogonal axes). Can also be moved to specify or adjust the size, shape, units, and other characteristics of the tool.</p><p num="0006">[0006] In some cases, the tool can basically be displayed as just a line or a very thin rectangle between two user-specified points in space. However, in another example, the tool takes the form of a two-dimensional (2D) polygon with vertices at three or more user-specified points, or a 3D volume with vertices at four or more user-specified points. Can be done. In any of these embodiments, the tool can include a scale that includes values and units when displayed to the user. In addition, the device has a length value between any two of the determined points, an area value between any three or more of the determined points, or of the determined points. Volume values between any four or more of can be automatically calculated and displayed to the user. Also, in certain embodiments, the device allows the user to save the state of the tool in memory, including any corresponding measurements and settings, and reload / redisplay that state at different locations. Make it possible. The device can include a depth camera or other similar sensor for measuring the distance from the device to an object in the 3D space in which the user is (eg, a room). Based on the distance information, the device can generate a 3D mesh model of the face in its 3D space, and the 3D mesh model can be used to identify the spatial coordinates of a plurality of determined points. .. One or more of the determined points can be spatially associated with one or more of the objects in 3D space.</p><p num="0007">[0007] Other aspects of the technique will become apparent from the accompanying figures and detailed description.</p><p num="0008">[0008] An overview of the present invention is provided to introduce in a simplified form the selected concepts described below in the form for carrying out the invention. The outline of the present invention is not intended to identify the important or essential features of the claimed subject matter, but is intended to be used to limit the scope of the claimed subject matter. It's not a thing.</p><p num="0009">[0009] One or more embodiments of the present invention are shown by way of example and are not limited in the drawings of the accompanying drawings. In the drawings, similar reference symbols indicate similar elements.</p>

<figref num="1">[0010] It is a figure which shows an example of an AR / VR headset.</figref><figref num="2">[0011] A high-level block diagram of a particular component of an AR / VR headset.</figref><figref num="3A">[0012] It is a figure which shows the example of the user's field of view by the AR / VR headset.</figref><figref num="3B">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3C">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3D">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3E">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3F">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3G">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3H">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3I">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3J">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3K">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3L">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="3M">It is a figure which shows the example of the user's field of view by an AR / VR headset.</figref><figref num="4">[0013] It is a figure which shows an example of the process which can be performed by the headset coupled with the virtual measurement tool.</figref><figref num="5">[0014] It is a figure which shows an example of the process which provides a virtual measurement tool in more detail.</figref><figref num="6">[0015] It is a figure which shows the process of generating and displaying a virtual measurement tool in more detail according to a scenario example.</figref>

[0016] In this description, references to "one embodiment" or "one embodiment", etc., refer to at least one embodiment of the technique in which the particular feature, function, structure, or property described herein is introduced herein. It means that it is included in the form. When such language appears herein, it does not necessarily refer to the same embodiment. On the other hand, the embodiments referred to are also not necessarily mutually exclusive.

The techniques presented herein include wearable visualization devices that generate and display virtual (eg, holographic) measurement tools ("tools") such as holographic rulers. The visualization device can be, for example, a headset, glasses, or goggles equipped to provide the user with an AR / VR experience. This tool allows the user of the device (eg, the wearer) to easily measure the distance, area, and volume associated with an object or space in its vicinity. The device allows the user to easily use and operate the tool, for example, with gestures, gaze, or utterances, or any combination thereof. Users can customize the tool to any length, size, or shape they need. In addition, the state of the tool can be stored in memory and reloaded / redisplayed in different environments.

[0018] Figure 1 shows an example of an AR / VR headset that can provide a virtual measurement tool according to the techniques presented here. However, the techniques presented here can be implemented in essentially any type of visualization device that can superimpose (superimpose) machine-generated images on the real-world field of view of the user's environment. Please note that. The headset 1 shown includes a headband 2, which allows the headset 1 to be detachably worn on the user's head. The headset 1 can be easily held in place by the rigidity of the headband 2 and / or by a fastening mechanism not shown in FIG. The headhand 2 is fitted with one or more transparent or translucent lenses 3, which include one or more transparent or translucent AR / VR display devices 4, each of which contains one or more transparent or translucent AR / VR display devices 4. Images can be overlaid in the field of view of the user's environment for one or both eyes. The details of AR / VR display device 4 are not closely related to the technique introduced here, and this technology is a display device that can superimpose machine-generated images on the real-world field of view of the user's environment in real time. Any known or convenient mechanism known in the art and having such performance is available.

[0019] The headset 1 also includes a microphone 5 for inputting speech from the user (for example, for use when recognizing voice commands) and one or more audio speakers for outputting sound to the user. 6 and one or more eye tracking cameras 7 for use in tracking the position and orientation of the user's head in real-world space, and one or more eye tracking cameras 7 for use by the eye tracking camera 7. Illuminated light source 8 and one or more depth cameras 9 for use in detecting and measuring distances to nearby surfaces, and capturing standard footage of the user's environment and / or said. One or more outwardly directed visible spectrum cameras 10 for use in locating the user in the environment, and a circuit that controls at least some of the aforementioned elements and performs related data processing functions. Includes configuration 11 and. The circuit configuration 11 can include, for example, one or more processors and one or more memories. Note that in other embodiments, the aforementioned components can be positioned at different locations on the headset 1. In addition, some embodiments may include some of the components described above being optional and / or additional components not described above.

[0020] Figure 2 is a high-level block diagram of a particular component of the AR / VR headset 20 according to some embodiments of the technique presented here. The headset 20 and components in FIG. 2 may represent the headset 1 in FIG. In Figure 2, the functional components of the headset 20 are: processor 21, memory 22, transparent or translucent AR / VR display device 23, audio speaker 24, depth camera 25, eye tracking camera 26, It includes one or more instances of each of the microphone 27 and the communication device 28, all of which are coupled together (directly or indirectly) by the interconnect 29. The interconnect 29 may or may be one or more conductive traces, buses, point-to-point connections, controllers, adapters, wireless links, and / or other conventional connected devices and / or media. At least some of these can operate independently of each other.

[0021] The processor 21 controls the entire operation of the headset 20 individually and / or collectively to perform various data processing functions. In addition, the processor 21 can provide at least some of the computational and data processing capabilities for generating and displaying the virtual measurement tools described above. Each processor 21 may include, for example, one or more general purpose programmable microprocessors, digital signal processors (DSPs), mobile application processors, microcontrollers, application specific integrated circuits (ASICs), or programmable gate arrays (PGAs). Alternatively, it can be a combination of such devices, or can include them.

[0022] The data and instructions (codes) 30 that make up the processor 21 to perform the aspects of the technique presented here can be stored in one or more memories 22. Each memory 22 can be, or can include, one or more physical storage devices. The physical storage device may be random access memory (RAM), read-only memory (ROM) (which can be erasable and programmable), flash memory, a small hard disk drive, or other suitable type of storage device, or , May be in the form of a combination of such devices.

[0023] By one or more communication devices 28, the headset 20 can receive data and / or commands to a separate external processing system such as a personal computer or game console and send the data and / or commands. be able to. Each communication device 28 may include, for example, a universal serial bus (USB) adapter, a Wi-Fi® transceiver, a Bluetooth® or Bluetooth Low Energy (BLE) transceiver, an Ethernet® adapter, a cable modem, etc. It can be, or can include, a DSL modem, a cellular transceiver (eg, 3G, LTE / 4G or 5G), or a baseband processor, or a combination thereof.

[0024] Each depth camera 25 can apply, for example, a time-of-flight principle that measures the distance to a nearby object. The distance information acquired by the depth camera 25 is used (eg, by processor 21) to build a 3D mesh model of the surface in the user's environment. Each eye tracking camera 26 is from the pupil and / or corneal reflections of near-infrared light emitted by one or more near-infrared sources on the headset, such as the illumination light source 7 in FIG. It can be a near-infrared camera that detects the direction of the line of sight based on mirror reflection. To enable the detection of such reflections, the inner surface of the headset lens (eg, lens 3 in FIG. 1) may be coated with a substance that reflects IR light but transmits visible light. Such substances are known in the art. By this technique, the illumination from the IR light source can bounce off the inner surface of the lens with respect to the user's eye, and the illumination is reflected back (possibly through the inner surface of the lens again) to the eye tracking camera.

[0025] While any or all of the components described above can be fully integrated with respect to their functionality described above, in some embodiments one or more processors 21 are associated with other components. It should be noted that it provides at least some of the processing capabilities provided. For example, at least part of the data processing for depth detection associated with the depth camera 25 may be performed by the processor 21. Similarly, at least part of the data processing for gaze tracking associated with the gaze tracking camera 26 may be performed by the processor 21. Similarly, at least part of the image processing that supports the AR / VR display device 23 may be performed by the processor 21, and so on.

An example of how an AR / VR headset can provide a virtual measurement tool is described here with reference to FIGS. 3A-3H. 3A-3H show various examples of the user's field of view with an AR / VR headset (eg, with lens 3 and display device 4 in FIG. 1). In particular, Figure 3A shows the central part of the field of view that a user wearing a headset would have while standing in his room while wearing the headset (peripheral vision is limited in page size). Therefore, it is cut off in this figure). The user can see, for example, the sofa 31 and the chair 32 placed around the coffee table 32. The headset may display one or more holographic icons 34, or other user interface elements in the user's field of view, to allow the user to use the various features of the headset. For example, one of the user interface elements may be the icon 35 (or other equivalent element) for selecting / starting the action of the virtual measurement tool.

[0027] While the headset is being manipulated, it uses the headset's depth camera from the user on all surfaces in the vicinity of the user (eg, within a few meters), or at least all nearby surfaces within the user's field of view. Build a 3D mesh model that includes the distance (ie from the headset). Techniques for generating 3D mesh models of nearby surfaces by using depth detection (eg, time of flight) are known in the art and need not be described herein. Thus, the 3D mesh model in the example of FIG. 3A is not only the at least all visible surfaces of the sofa 31, chair 32, and coffee table 33, but also the walls, floors, ceilings, windows, and possibly walls of the room, etc. Even smaller features such as curtains and works of art (not shown) that are attached to the table will be modeled. The 3D mesh model can be stored in the memory on the headset. By using a 3D mesh model and image data from a visual tracking system (eg camera 10), the circuit configuration in the headset (eg processor 21) at any given time identifies the exact location of the user in the room. can do. The 3D mesh model can be automatically updated frequently, such as several times per second.

[0028] Now suppose the user wants to replace the coffee table 33 with a new one, but wants to replace it with a coffee table of similar size and keep it in the same place in the room. Therefore, the user can decide to use the tool to measure the dimensions of the coffee table 33. To do this, the user first enters a command to select or start a tool. Unless otherwise specified, this command, like all other user commands described in this description, is, for example, a hand gesture, a voice command, or an action based on the user's line of sight (eg, a displayed holo). It can be a paused action of the user's line of sight on a graphic icon), or a combination of these types of inputs.

[0029] In this example, after the user selects the tool, the user inputs to the headset to specify the two points 37, which in this example are the endpoints that the user of the virtual measurement tool first wanted. In other embodiments, the tool may be displayed first at a given default location and orientation of space for the user. In this example scenario, point 37 corresponds to a separate corner on the top surface of the coffee table 33. Appropriate commands such as, for example, by making a finger-directed "tap" gesture at each corner of the coffee table (from the user's point of view), or by pointing at each corner and "aligning the points". Each point 37 can be specified by speaking. By correlating the user's input with a 3D mesh model of the room already created, the processor in the headset can identify the most likely 3D spatial coordinates that the user intended to identify. However, it should be noted that point 37 in this context does not necessarily coincide with the corner of the physical object. For example, the user can specify the tool endpoint 37 as being near the user or even on any surface (recognized by the headset) floating in the air. As seen in this example, if the user's input appears to specify a point on a physical object, the processor will associate the point with the object and fix the point. This process of automatically positioning and fixing an endpoint to a point on a physical object is called a "snap." Snap features act like real-world magnetic forces, and the virtual ruler 38 is physical until the user explicitly indicates intent to pull the virtual ruler 38 apart by some input (eg, gaze, utterance, or gesture). It will appear to "stick" to the target object.

[0030] In this example, if the user specifies two points 37, the headset displays a holographic (virtual) line 38, or virtual ruler, that connects the two points 37. Thus, in this example, the line 38 extends along one of the longer edges of the top surface of the coffee table 33. Line 38 may be annotated with hash marks and / or numbers indicating units such as feet and inches and / or fractions thereof.

[0031] As seen in this example, when the virtual ruler 38 is anchored to an object, the default headset will allow the user to move around the room unless the user makes an input that modifies its functionality. , The user can adjust its display so that the virtual ruler 38 appears to remain fixed to the object in the same orientation. The user may choose to unfix the virtual ruler 38 from the object and move the ruler 38 around in space, as shown in FIGS. 3C and 3D. In FIG. 3C, for example, the user raises (translates) the virtual ruler 38 vertically from the coffee table 33. In Figure 3D, the user is rotating the virtual ruler 38 about a vertical axis. The user can move the virtual ruler 38 in translation along any of the three orthogonal axes (eg, x, y, and z) and center the ruler around any of the three orthogonal axes. Can be rotated. This can also be achieved by voice commands, gestures, or changes in the user's line of sight, or any suitable command such as a combination thereof.

[0032] Instead of fixing the virtual ruler 38 to an object from the beginning, the user instead instantiates the virtual ruler 38 so that it initially floats in space, and then (optionally). The ruler 38 can be "snapped" to a physical object. The virtual ruler 38 can snap to any edge or face represented in the 3D mesh of the local environment. The headset can infer the user's intention to snap based on any of a variety of inputs, such as voice commands, gestures, or pauses in the line of sight of the user's object. This judgment / reasoning may be based on how close the physical object is to the user and / or how central the object is in the user's field of view.

Virtual measurement tools, such as those described herein, take the form of (2D) polygons by allowing the user to specify three or more related points rather than just two endpoints. You can also have. In such an example, the headset can automatically calculate and display to the user the value of the area of the polygon, in addition to the length of each side of the polygon. For example, here referring to FIG. 3E, the user may want to know how much area the coffee table 33 occupies, and accordingly the user puts the tool on the top surface of the coffee table 33. It can be defined to have the form of the corresponding rectangle 40. Although not shown in FIG. 3E, the display of the polygonal embodiment of the tool can also include units and values as in the linear embodiment. The headset can also automatically calculate and display its area (for example, "8ft" in this example.²"). In some cases, the user may specify all three or more points first when defining the first endpoint as described above (Figure 3B), and alternatively, the user (Figure 3B) You can first define the tool as just a line between two points (as described above), and then add one or more additional points to extend the tool to a polygon or 3D volume. Headsets can use any of a variety of techniques to infer the user's intent in this regard. For example, if the user first specifies three or more points on a physical object that are relatively close together or all the same in time, it can be inferred that the user wants to define the tool as a polygon. it can. If the user first defines the tool as a line, the user may later add one or more points, for example by a command (eg, say "add points") to convert the line to a polygon. Yes, or the headset can infer the user's intention to add points based on the user's behavior. As seen in the example of a linear measurement tool (eg virtual ruler 38), the user can move the polygonal tool by translation and rotation.

[0034] Similarly, the tool can also have the form of a 3D object by allowing the user to specify four or more related points. In such an example, the headset can automatically calculate and display to the user not only the value of the volume of the tool, but also the value of any area and length of each side of the object. For example, referring here to FIG. 3F, the user can define the tool as a rectangular box 50 that represents the "cover" of the exterior space of the coffee table. Although not shown in FIG. 3E, the representation of the polygonal embodiment of the tool may include units and values as in the linear embodiment. The headset can also automatically calculate and display the volume of the tool (box 50), as shown (for example, "8ft" in this example.³"). Users can also move 3D tools by translation and rotation, as seen in the examples of linear and 2D virtual measurement tools.

[0035] In some cases, the headset allows the user to save the current state of the tool, including any corresponding measurements and settings, in memory and reload / redisplay it in a different location. For example, in this example, the user may want to save the tool in its current form and redisplay it elsewhere, such as in a furniture store. Therefore, as shown in Figure 3G, the user has the appropriate command (eg, by saying "save" or by making the appropriate hand gesture to select the corresponding icon 34 to be displayed). Can be entered. Later, when the user visits the furniture store, as shown in Figure 3H, the user can select the corresponding icon 34 to be displayed, either by saying "load" or by the appropriate command. You can have the headset load the tool from memory and redisplay the tool (with proper hand gestures). By adjusting the position and orientation of the tool, the user can adapt to a physical object in the store (eg, a new coffee table) and allow the user to measure that object.

[0036] Various other usage scenarios for virtual measurement tools are possible. For example, a headset allows the user to specify a series of three or more endpoints that automatically calculate and display the sum of the lengths of the segments defined by those three or more endpoints. be able to. An example of this usage scenario is shown in Figure 3I. Here, the virtual ruler 58 is made up of two connected linear segments 61 defined by three endpoints 63, and the length of each segment and the sum of the lengths of the two segments are shown. In addition, by using the face recognition feature of the headset, as shown in Figure 3J, the user can generate one or more endpoints over time (or based on a distance threshold). The face can be "wrapped" with a virtual ruler 59, and the headset can automatically calculate and display the length of each segment and the sum of the lengths of the segments.

[0037] Moreover, the virtual measurement tool does not have to be instantiated with a straight line. For example, as shown in Figure 3K, the user can define the virtual ruler 70 as a curved / irregular line (eg, by using hand gestures), and the headset is still (eg, 1). To calculate the total length of a virtual ruler (by dividing the virtual ruler into one or more radii centered on one or more corresponding center points and then calculating the length of each radius). Can be done. Regardless of whether the tool is in the form of linear or curved / irregular segments (or combinations thereof), the user "snaps" its endpoints together and encloses the shape 72 etc. in Figure 3L. 2D shape can be formed. In that case, the headset can automatically calculate and display the area enclosed by the newly defined shape. In addition, as shown in Figure 3M, the user can create a 3D shape (such as volume 74) from any 2D shape by entering the appropriate command. In this case, the headset can also automatically calculate and display the total volume enclosed by the 3D shape.

[0038] FIG. 4 shows an example of a process that can be performed by a headset (eg, by processor 21) to provide a virtual measurement tool, according to some embodiments. First, in step 401, the headset makes multiple points at different locations in the 3D space in which the user is, based on input from the user by using gesture recognition, eye tracking, and / or speech recognition, etc. Generate a virtual measurement tool by defining each. Then, in step 402, the headset displays the virtual measurement tool to the user so that the virtual measurement tool appears to the user in real time in the 3D space in which the user is located, overlapping the real-world field of view.

[0039] FIG. 5 shows in more detail an example of a process of providing a virtual measurement tool according to some embodiments. When the headset is first powered on and initialized, the headset in step 501 uses its depth sensor to measure the distance from the headset to a nearby surface in the user's environment. The headset then in step 502 generates a 3D mesh model of those faces based on the measured distances. Any known or convenient technique for generating 3D mesh model faces can be used at this step. After a while, the headset receives user input to select the virtual measurement tool in step 503, not necessarily as a result of step 502. The headset then performs user input (eg, one or more gestures, voice commands, and / or line-of-sight-based commands) to specify two or more points of space in the user's environment in step 504. Receive. In step 505, the headset determines user-specified points by identifying (at least in part) the most likely 3D coordinates of each user-specified point based on the 3D mesh model. In step 506, the headset displays the measurement tool to the user using the points determined as the tool's endpoints or vertices.

[0040] Figure 6 shows in more detail the process of generating and displaying tools according to an example scenario. In step 601, the headset receives user input (eg, one or more gestures, voice commands, and / or line-of-sight-based commands) that specify two or more points in space. In step 602, the headset identifies the most probable 3D coordinates of each point based on the 3D mesh model. In this example, this step further involves associating at least one of the points with a point on the object in the vicinity of the user, which can further include fixing the point of the object. As a result, as the user moves through this environment, the point (which defines the endpoint or vertex of the tool) remains fixed to the object from the user's point of view.

[0041] In the example scenario shown, if the user specifies only two points (step 603), the headset will specify the units and values as a line connecting those two points in step 606 (optionally). Define and display the measurement tool. The headset also calculates the length of the line and displays it to the user. The process then proceeds to step 604. In step 604, if the user specifies three or more points and gives instructions (either explicitly or implicitly) that they want to make a 2D measurement (eg, area), then step 608. The headset in defines and displays the measurement tool as a polygon connecting three or more points. The headset also calculates the area of the polygon in step 609, displays it, and then proceeds to step 604. In step 604, if the user specifies four or more points and gives instructions (either explicitly or implicitly) that he wants to make a 3D measurement (eg, volume), then step 610. The headset in defines and displays a measuring tool as a 3D volume that connects four or more points. The headset also calculates and displays the volume surrounded by the tool in step 611.

[0042] In a variant of the technique described above, the virtual measurement tool can be instantiated and / or used by multiple users working together in a shared AR environment. For example, two or more users, each using a visualization device as described above, can measure the shared physical space together, establishing points in the real world that contribute to the overall measurement of space and markup, respectively. can do. In such embodiments, the two or more visualization devices can communicate with each other either directly or by a separate processing device (eg, a computer), or the visualization device is a separate processing device that coordinates the measurement. Can communicate with and display all the features of the visualization device.

[0043] Based on this, a virtual (holographic) measurement tool for use in a wearable AR / VR display system is described.

[0044] The machine mounting operations described above can be implemented either completely by a programmable circuit configuration programmed / configured by software, by a dedicated circuit configuration, or by a combination of such forms. Such dedicated circuit configurations (if any) include, for example, one or more application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), system-on-chip systems (if any). It can be in the form of SOC) or the like.

The software for implementing the techniques presented herein may be stored on a machine-readable storage medium and may be run by one or more general purpose or dedicated programmable microprocessors. The term "machine readable medium" includes any mechanism that can store information in a form accessible by the machine, as used herein (machines are, for example, computers, network devices, mobile phones, mobile phones). Information terminals (PDAs), manufacturing tools, any device with one or more processors, etc.). For example, machine-accessible media include recordable / non-recordable media (eg, read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.) and the like. ..

[0046] Certain embodiments of the techniques presented herein are summarized in the following numbered examples.

<p num="0042">1. Based on at least one of recognizing at least one gesture of the user, tracking the user's line of sight, or recognizing the user's speech by the visualization device worn by the user. , Generate a virtual measurement tool by determining multiple points at different locations in the 3D space where the user is, and the virtual measurement tool tells the user the reality of the 3D space where the user is. A method that includes displaying a virtual measurement tool to the user by a visualization device so that it appears to overlap the field of view.</p><p num="0043">2. Generating a virtual measurement tool makes it appear to the user that the virtual measurement tool stays in a fixed location and orientation of space as the user moves through its three-dimensional space. The method described in Example 1, wherein a plurality of points are fixed to corresponding different points in a three-dimensional space.</p><p num="0044">3. Generating a virtual measurement tool involves spatially associating at least one of a plurality of points with a corresponding point on a physical object in the three-dimensional space in which the user is located. The method described in Example 1 or Example 2.</p><p num="0045">4. Generating a virtual measurement tool is described in any of Examples 1-3, comprising generating at least a portion of the virtual measurement tool as a line between two of a plurality of points. How to be done.</p><p num="0046">5. Generating a virtual measurement tool is described in any of Examples 1 to 4, including generating the virtual measurement tool as a polygon having vertices at three or more of a plurality of points. How to be done.</p><p num="0047">6. Generating a virtual measurement tool comprises generating a virtual measurement tool as a three-dimensional volume having vertices at four or more of a plurality of points, according to any one of Examples 1-5. The method described.</p><p num="0048">7. The method described in any of Examples 1-6, wherein displaying the virtual measurement tool comprises displaying a measurement scale on or in close proximity to the virtual measurement tool.</p><p num="0049">8. Performed further including calculating the length, area, or volume by the visualization device based on multiple points and outputting the length, area, or volume to the user by the visualization device. The method described in any of Examples 1-7.</p><p num="0050">[0055] 9. The 3D space in which the user is located is the 1st 3D space, and in response to the 1st user command, the virtual measurement tool is saved in memory and the visualization device displays the virtual measurement tool. In response to abandonment and a second user command after the user moves into the second 3D space, the virtual measurement tool is read from memory and virtual while the user is in the 2nd 3D space. Described in any of Examples 1-8, further comprising redisplaying the measurement tool to the user, and redisplaying comprising spatially associating the virtual measurement tool with an object in a second three-dimensional space. How to be done.</p><p num="0051">[0056] 10. Using a depth sensor to measure the distance from the visualization device to an object in the user's 3D space, and based on the measured distance, the surface in the user's 3D space. Identifying the spatial coordinates of multiple points, including generating a 3D mesh model of the 3D mesh model and using the 3D mesh model to identify the spatial coordinates of multiple points based on at least one user input. Using a 3D mesh model to do this involves determining the location of at least one of a plurality of points that is spatially associated with one of the above objects, of Examples 1-9. The method described in either.</p><p num="0052">11. Identifying adjustments to the location or orientation of the virtual measurement tool by at least one of recognizing the user's gestures, tracking the user's line of sight, or recognizing the user's utterances. The method according to any of Examples 1 to 10, further comprising adjusting the location or orientation of the virtual linear measurement tool displayed to the user based on the adjustment.</p><p num="0053">[0058] 12. Using the depth sensor on the head-mounted visualization device to measure the distance from the visualization device to an object in the first enclosed space where the user of the visualization device is, and the measured distance. To generate a 3D mesh model of a face in a first enclosed space, and to associate multiple points with one of the above objects, at least one of a plurality of points. To generate a virtual measurement tool with a visualization device, each of which is determined at a different location in the first enclosed space, according to at least one input from the user, including identifying one location. , The above at least one input comprises at least one of the user's gesture, the user's line of sight, or the user's speech, and the virtual measurement tool is first enclosed with the user. A method that includes displaying the virtual measurement tool to the user by a visualization device so that it appears to overlap the actual view of the space, and the above display is on or on the virtual measurement tool. Generating a virtual measurement tool, including displaying the measurement scale in close proximity, means that the virtual measurement tool is fixed to the user as the user moves through the first enclosed space. Recognize user gestures by fixing multiple points to corresponding different points in the first enclosed space and adjusting the virtual measurement tool to the location or orientation so that it appears to stay in place and orientation. Identifying by at least one of doing, tracking the user's line of sight, or recognizing the user's speech, and the location of the virtual linear measurement tool displayed to the user based on that adjustment. Or methods, including adjusting the orientation.</p><p num="0054">13. The method described in Example 12, wherein generating a virtual measurement tool comprises generating at least a portion of the virtual measurement tool as a line between two of a plurality of points.</p><p num="0055">[0060] 14. Generating a virtual measurement tool means generating at least a portion of the virtual measurement tool as a polygon with vertices at three or more of the points, or of multiple points. The method according to Example 12 or Example 13, comprising at least one of generating at least a portion of a virtual measurement tool as a three-dimensional volume having vertices in four or more.</p><p num="0056">15. An embodiment further comprising calculating a length, area, or volume by a visualization device based on a plurality of points and outputting the length, area, or volume to the user by the visualization device. The method described in any of 12-14.</p><p num="0057">16. A head mount for attaching the head-mounted visualization device to the user's head and at least partially for displaying the generated image combined with the head mount to the user. A transparent display surface, an input subsystem configured to receive input from the user for gesture recognition and gaze detection, a depth sensor to locate objects in the user's environment, and a display surface. , Input subsystem, and virtual measurement by determining multiple points at different locations in the user's environment according to at least one input from the user that is coupled to the input subsystem and received through the input subsystem. Generate a tool so that at least one location of multiple points is identified to be spatially associated with one of the objects in the user's environment, and on the display surface, distance, area, Or display the virtual measurement tool to the user with volume instructions so that the virtual measurement tool appears to the user to stay in a fixed location and orientation of space as the user moves through the environment. A head-mounted visualization device, including a processor configured in.</p><p num="0058">[0063] 17. The processor identifies an adjustment to the location or orientation of the virtual measurement tool based on at least one of the user's gestures or the user's line of sight, and based on that adjustment to the user. The head-worn visualization device according to Example 16, further configured to adjust the location or orientation of the virtual linear measurement tool displayed.</p><p num="0059">[0064] 18. The head-worn visualization device of Example 16, wherein the processor is configured to generate a virtual measurement tool as a polygon with vertices at three or more of a plurality of points.</p><p num="0060">19. The head described in any of Examples 16-18, wherein the processor is configured to generate a virtual measurement tool as a three-dimensional volume with vertices at four or more of a plurality of points. Part-mounted visualization device.</p><p num="0061">[0066] 20. Including more memory, the processor should stop displaying the virtual measurement tool on the display surface to store the virtual measurement tool in memory in response to the first user input, and the user. In response to a second user input after moving to the second environment, the user reads the virtual measurement tool from memory and spatially associates the virtual measurement tool with an object in the second environment. The head-worn visualization device according to any of Examples 16-19, further configured to redisplay the virtual measurement tool to the user on the display surface while in the second environment.</p><p num="0062">21. Different in the three-dimensional space in which the user is located, based on at least one of recognizing at least one gesture of the user, tracking the user's line of sight, or recognizing the user's utterance. A means of generating a virtual measurement tool by determining multiple points at a location, and a virtual measurement tool to the user so that it appears to overlap the actual view of the three-dimensional space in which the user is located. A device that includes a means of displaying virtual measurement tools.</p><p num="0063">22. The means by which the virtual measurement tool is generated is to make the virtual measurement tool appear to the user to stay in a fixed location and orientation of the space as the user moves through the three-dimensional space. 21. The apparatus of Example 21, comprising means for fixing a plurality of points to corresponding different points in a three-dimensional space.</p><p num="0064">[0069] 23. Means for generating a virtual measurement tool include means for spatially associating at least one of a plurality of points with a corresponding point on a physical object in the three-dimensional space in which the user is located. The device according to Example 21 or Example 22.</p><p num="0065">24. The means for generating a virtual measurement tool is described in any of Examples 21-23, including means for generating at least a part of the virtual measurement tool as a line between two of a plurality of points. Equipment to be done.</p><p num="0066">25. The means for generating a virtual measurement tool is described in any of Examples 21-24, comprising means for generating a virtual measurement tool as a polygon having vertices at three or more of a plurality of points. Equipment to be done.</p><p num="0067">26. Any of Examples 21-25, wherein the means for generating the virtual measurement tool includes means for generating the virtual measurement tool as a three-dimensional volume having vertices at four or more of a plurality of points. The device described.</p><p num="0068">27. The apparatus according to any of Examples 21-26, wherein the means for displaying the virtual measurement tool includes means for displaying the measurement scale on or in close proximity to the virtual measurement tool.</p><p num="0069">28. Any of Examples 21-27, further comprising a means of calculating a length, area, or volume based on a plurality of points and a means of outputting the length, area, or volume to the user. The device described in.</p><p num="0070">29. The three-dimensional space in which the user is located is the first three-dimensional space, and a means for saving the virtual measurement tool in the memory in response to the first user command and a means for stopping the display of the virtual measurement tool. And, in response to the second user command after the user moves to the second 3D space, while the user is in the 2nd 3D space, the virtual measurement tool is read from the memory and the virtual measurement tool is used. A device according to any of Examples 21-28, further comprising a means of redisplaying to the user, wherein redisplaying comprises spatially associating a virtual measurement tool with an object in a second three-dimensional space. ..</p><p num="0071">30. Means of using a depth sensor to measure the distance from the visualization device to an object in the user's 3D space, and the surface in the user's 3D space based on the measured distance. Further include means for generating a 3D mesh model of the 3D mesh model and a means for using the 3D mesh model to identify the spatial coordinates of multiple points based on at least one user input, and identify the spatial coordinates of multiple points. Means of using a 3D mesh model to do so include means of identifying at least one of a plurality of points that are spatially associated with one of the above objects, of Examples 21-29. The device described in either.</p><p num="0072">31. A means of identifying adjustments to the location or orientation of a virtual measurement tool by at least one of recognizing a user's gesture, tracking the user's line of sight, or recognizing a user's utterance. The apparatus according to any one of Examples 21 to 30, further comprising a means for adjusting the location or orientation of the virtual linear measurement tool displayed to the user based on the adjustment.</p><p num="0073">[0078] Any or all of the features and functions described above will be apparent to those skilled in the art, as otherwise described above, or any such embodiment is incompatible with its function or structure. Except in some cases, they can be combined with each other. Unless contrary to physical potential, (i) the methods / steps described herein may be performed in any order and / or in any combination, and (ii) the components of each embodiment may be arbitrary. It is assumed that they can be combined in a way.</p><p num="0074">[0079] Although the subject matter is described in a language specific to structural features and / or behaviors, it is understood that the subject matter defined in the appended claims is not necessarily limited to the particular features or behaviors described above. I want to be. More precisely, the particular features or behaviors mentioned above are disclosed as examples of implementing the claims, and other equivalent features and behaviors are intended to be within the claims.</p>