Head-mounted augmented reality video presentation device and virtual display object operation method

The present invention relates to a head-mounted augmented reality image presenting device, and more particularly to a head-mounted augmented reality image presenting device capable of easily operating a virtual display object displayed in front of the eyes and a method for operating the virtual display object. ..

Software that presents images of augmented reality has been developed and is being used. For example, application software for camera-equipped mobile phones acquires information (such as tourist information around the area) that matches the scene taken by the camera via a network, and captures a virtual display (icon, etc.) that indicates the existence of this information. And display it on the display screen of the mobile phone. Then, when the user selects the virtual display object on the display screen using a touch panel or the like, the virtual display object is enlarged to browse detailed information (history of tourist facilities, etc.) or copy the information to the local recording area. You can do it. It is also possible to generate a virtual display object to which the user himself / herself has given information.

Even if such an augmented reality image is displayed on a small display screen of a mobile phone, there is a problem that it is difficult to see. It also lacks a sense of presence. Therefore, by connecting a head-mounted display device (HMD) to the mobile phone and displaying the captured image and the virtual display object, the user can see a large and easy-to-see image with a sense of reality.

However, when the head-mounted display device is used, the user's eyes are blocked by the display screen that covers the front of the eyes, and there arises a problem that the virtual display object on the display screen cannot be operated by the mobile phone. ..

As a means for solving this problem, for example, as described in Patent Document 1 below, a gyro sensor is attached to the user's hand, the position of the hand is grasped by the head-mounted display device, and the position of the hand is displayed on the display screen. By displaying the image of the hand in the virtual space, it is conceivable to operate the virtual display object using the image of the hand. Alternatively, it is also possible to attach an infrared camera to the display screen side of the head-mounted display device, detect the line of sight of the user, and issue an operation instruction.

However, when a sensor such as a gyro sensor or a special device such as an infrared camera is provided, there is a problem that the cost of the head-mounted display device increases and the device configuration becomes large.

<p><patcit num="1"><text>Japanese Patent Application Laid-Open No. 2003-337962</text></patcit></p>

<p> An object of the present invention is a head-mounted augmented reality image presentation device and its virtual display object, which can easily operate a virtual display object displayed on the head-mounted display device without using a device such as an expensive sensor. The purpose is to provide an operation method.</p>

<p> The head-mounted extended reality image presentation device and its virtual display object operation method of the present invention detect an imaging means for photographing a subject image, a position detecting means for detecting the position of the imaging means, and an imaging direction of the imaging means. The direction detecting means to be used, the calculation means for calculating the shooting range of the subject image from the detection result of the position detecting means and the detection result of the direction detecting means, and the shooting range calculated by the calculation means are transmitted to an external database. Then, the virtual object data acquisition means for acquiring the virtual object data corresponding to the shooting range from the database and the imaging means are made to capture a pointer image as a part of the subject image to show the subject image and the virtual object data. An image synthesizing means for synthesizing a virtual display object to generate a composite image and a head-mounted display means for displaying the composite image are provided, and the mutual relationship between the pointer image in the composite image and the virtual display object. When the predetermined condition is satisfied, it is assumed that the required operation by the pointer image for the virtual display object is executed, and the required function corresponding to the required operation is expressed in the virtual object data corresponding to the virtual display object. It is a feature.</p>

<p> According to the present invention, it is possible to intuitively and easily operate a virtual display object displayed on a head-mounted display means without using a device such as an expensive sensor.</p>

<figref num="1">It is a functional block diagram of the head mount type augmented reality image presenting apparatus which concerns on one Embodiment of this invention.</figref><figref num="2">It is a figure which shows the image example displayed on the display screen shown in FIG.</figref><figref num="3">It is a figure which shows the capture example of the fingertip image which concerns on another embodiment.</figref><figref num="4">It is a figure which shows the operation procedure when a fingertip image is used as a pointer image.</figref><figref num="5">It is explanatory drawing of the operation example using a fingertip image.</figref><figref num="6">It is a figure explaining another operation example using a fingertip image.</figref><figref num="7">It is a figure explaining another operation example using the image of a hand.</figref><figref num="8">It is a figure which captures a marker image as a pointer image.</figref><figref num="9">It is a figure which shows the operation procedure when a marker image is used as a pointer image.</figref><figref num="10">It is explanatory drawing of the operation example using a marker image.</figref><figref num="11">It is explanatory drawing of another operation example using a marker image.</figref><figref num="12">It is a figure which captures a point light source image as a pointer image.</figref><figref num="13">It is a figure which shows the operation procedure when a point light source image is used as a pointer image.</figref><figref num="14">It is explanatory drawing of the operation example using a point light source image.</figref><figref num="15">It is a figure explaining another operation example using a point light source image.</figref>

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a functional block diagram of the head-mounted augmented reality image presentation device according to the first embodiment of the present invention. The head-mounted augmented reality image presentation device 10 of the present embodiment includes a central processing unit (CPU) 11 that controls the device 10 in an integrated manner, and a bus 12 connected to the CPU 11.

The bus 12 includes an imaging control unit 15 connected to the camera 14, a main memory 16, a signal processing unit 17 that digitally processes the image signal captured by the camera 14, a photographing lens 14a, and an imaging sensor 14b. With a measuring unit 18 that measures the position and orientation of the camera 14, a GPS (global positioning sensor) 26 that acquires position information, a geomagnetic sensor 27 that acquires direction information, a virtual display, etc., which will be described later, and an image sensor 14b. An image synthesizing unit 19 that synthesizes a captured and signal-processed image is connected.

Further, the bus 12 includes a communication control unit 21 having an antenna 20, a pointer detection / recognition unit 22 and a pointer monitoring unit 23, which will be described in detail later, and an image display control unit 25 to which a display screen unit (EVF) 24 is connected. Is connected.

The head-mounted augmented reality image presentation device 10 communicates with an external database through a communication control unit 21 to acquire virtual object data. The external database 30 in which a large amount of virtual object data (including various information such as text data and image data) is stored is provided with a virtual object data search unit 31, a communication control unit 32, and an antenna 33. There is.

The head-mounted augmented reality image presenting device 10 calculates the shooting range of the subject image captured by the image sensor 14b by the camera position and orientation measuring unit 18 based on the information acquired by the GPS 26 and the geomagnetic sensor 27. Then, the direction and position data corresponding to the shooting range of the subject image is transmitted to the communication control unit 32 of the database 30 through its own communication control unit 21 and the antenna 20.

The virtual object data search unit 31 searches the database 30 for virtual object data corresponding to the shooting range of the subject image based on the received data, and retrieves the searched virtual object data in the head-mounted augmented reality image presentation device 10. Return to.

The CPU 11 of the head-mounted augmented reality image presentation device 10 converts the received virtual object data into an icon and passes it to the image compositing unit 19, and the image compositing unit 19 transfers this icon to the image captured by the image sensor 14b as a virtual display object. And display it on the display screen 24.

The head-mounted augmented reality image presentation device 10 has a built-in communication control unit 21 and an antenna 20, but these two functions may be substituted by the functions of a mobile phone.

Since some mobile phones are equipped with a camera, it is possible to substitute the camera mounted on the mobile phone for the image pickup control unit 15 and the camera 14 of the head-mounted augmented reality image presentation device 10. Further, the GPS sensor 26 and the geomagnetic sensor 27 may be substituted with the GPS or the geomagnetic sensor mounted on the mobile phone. If there are overlapping functions (camera function and GPS function in the above example) among the functions of the mobile phone and the functions of the head-mounted augmented reality image presentation device 10, the functions of the head-mounted augmented reality image presentation device 10 are prioritized. It is preferable to use the function of the mobile phone when a special instruction is input from the user.

FIG. 2 is a diagram showing an example of a display screen of the head-mounted augmented reality image presentation device 10 shown in FIG. The scenery taken by the camera 14 is displayed on the screen 24. The CPU 11 executes the augmented reality (AR) application software, and displays the virtual object data icons acquired from the virtual object database 30 as described above with virtual display objects (also called air tags) 1a, 1b, ... indicate. Each air tag 1a, 1b, ... Is displayed for each location such as a building corresponding to each of the subject images.

For example, in order for the user to browse the information given to the air tag here, it is necessary to select a desired air tag from the display screen from a plurality of displayed air tags and start the application software for browsing. In order to express a specific function for the desired air tag in this way, the user needs to interact with the air tag.

However, when the user wears the head-mounted augmented reality image presentation device 10, the display device (display screen in FIG. 1) 24 covers the entire vicinity of the user's eyes, so that the user cannot see his / her own hands. Therefore, it is not possible to operate a device such as a touch panel or a hardware key.

Therefore, in the present embodiment, the image of the fingertip of the user is taken by the camera 14 or the camera of the mobile phone, and the image of the fingertip is displayed on the screen 24. When shooting a fingertip with the camera 14, the user raises his / her fingertip in front of the camera 14 (in front of the head-mounted augmented reality image presenting device 10) so that the fingertip image 2 is captured at the same time as the landscape. Then, as described next, the CPU 11 shifts to the user interface processing for AR by using the fingertip image 2.

FIG. 3 is a diagram showing a user capturing an image of his / her fingertip from a camera of a mobile phone. In FIGS. 1 and 2, the camera 14 captures the image of the fingertip of the landscape image, but in FIG. 3, the camera attached to the mobile phone 5 captures the image of the landscape and the image of the fingertip. Then, the CPU 11 of the head-mounted augmented reality image presentation device 10 connected to the mobile phone 5 by a cable takes in the image data transmitted from the mobile phone 5 and displays it on the display screen 24.

FIG. 4 is an explanatory diagram of a user interface processing procedure for operating the air tag using the user's fingertip image as a pointer. First, the captured image is captured (step S1), the hue is observed, and the flesh-colored region (hand, fingertip) is detected (step S2). Next, the arrangement coordinate data of the virtual display object (air tag) is acquired (step S3), and the linear distance between the contour of the skin color region and the air tag is calculated (step S4).

In parallel with this, a difference image is generated between the skin color detection result (step S2a) of the captured image of the previous frame and the skin color detection result (step S2) of the current frame (step S5), and the motion vector of the skin color region is calculated (step). S6).

After step S4 and step S6, the process proceeds to step S7 to recognize what kind of operation instruction the operation instruction is from the calculation results so far, and also recognize the air tag to be operated. The recognition method will be described later. In the next step S8, the function based on the above operation instruction is expressed for the air tag to be operated. That is, if the operation instruction is an erasure instruction, the air tag to be operated is erased, and the process returns to step S1.

If the air tag to be operated cannot be recognized in step S7, or if the operation instruction cannot be recognized, the process returns from step S7 to step S1.

By performing such user interface processing, it is possible to operate the air tag without using a special device such as a gyro sensor.

5 to 7 are explanatory views of the above-mentioned operation instructions, FIG. 5 (a) shows a grasping operation, and FIG. 5 (b) shows a releasing operation. The increase area and decrease area of the skin color area are calculated from the frame-to-frame difference of the skin color area 2, and the state where the area of the increase area is less than a certain value continues for a certain period of time, and as shown in the left figure of FIG. 5 (a). , If the air tag 1 touches the skin color area (fingertip image) 2 at two or more points, the air tag 1 is regarded as "pinched", and thereafter, as shown in the right figure of Fig. 5 (a), the air tag 1 is used as the fingertip. Move it by following the movement of image 2. Alternatively, when the skin color region 2 is closer to the air tag 1 than a certain level, it may be regarded as "pinched".

As shown in the right figure of Fig. 5 (b) from the "pinched" state of the air tag 1 (Fig. 5 (b) left figure), when the skin color area 2 is moved at a certain speed or more, the air tag 1 is "released". The air tag 1 is not made to follow the skin color region 2.

FIG. 6 is an explanatory diagram of an operation instruction for discarding (erasing) the air tag 1. The air tag 1 is discarded by superimposing the air tag 1 on the icon that expresses the function by following the movement of the fingertip image 2 with the air tag 1 pinched, and in this case, the trash can icon 4 to be discarded.

FIG. 7 is an explanatory diagram for performing an operation of manually dispelling the air tag 1. When the entire hand is photographed with a camera, the skin color area 2 moves at a certain speed or more, and the air tag 1 is present in the middle of the moved path, the air tag 1 moves the skin color area 2 at a speed, a speed according to the direction, and a direction. Make it blow away.

That is, the increase region and the decrease region of the skin color region are calculated from the difference between the captured image frames of the skin color region 2, and when the area of the increase region is equal to or larger than a certain value and the air tag is in the increase region, an action force is generated on the air tag. Let me. Alternatively, if there is an air tag on the extension line of the line segment connecting the center coordinates of the increasing region and the center coordinates of the decreasing region and in the direction from the decreasing region to the increasing region, an acting force is generated on the air tag.

The acting force depends on the following parameters, that is, the area of the increasing region (the larger the acting force, the greater the acting force), and the direction of the line segment connecting the center coordinates of the increasing region and the center coordinates of the decreasing region (force vector of the acting force). Let me. In such a process, the user's action of paying the air tag is recognized, and the air tag is given an action force.

In this way, the user can express various functions on the air tag by moving his / her hand or fingertip like a gesture.

In the above-described embodiment, the user operates the air tag using an image of his / her hand or fingertip as a pointer, but another object other than the fingertip or the like can be used as a pointer.

FIG. 8 shows an example in which a camera-taken image of a stick-shaped marker is used as a pointer. The marker 6 is photographed by the camera of the mobile phone 5, the photographed image is sent to the head-mounted augmented reality image presenting device 10, and this is used as a pointer. Marker 6 describes a two-dimensional geometric pattern. In the illustrated example, a pattern meaning "download", i.e. an arrow that captures something in the box, is described.

FIG. 9 is an explanatory diagram of a user interface processing procedure for operating the air tag using the marker image 6 shown in FIG. 8 as a pointer. First, the marker image 6 is captured from the camera (step S11). Then, in the next step S12, after binarizing the luminance component of the marker image 6, the marker image 6 is detected and recognized by template matching with the marker image registered in advance, and the recognized marker is detected. Calculate the relative position and orientation with the camera.

In the next step S13, the arrangement coordinate data of the air tag 1 is acquired, and in step S14, the distance between the center of the marker image 6 and the air tag 1 is calculated.

In parallel with this, in step S15, the position and attitude data of the marker of the captured image of the previous frame are retained, and in step S16, the position and attitude data of the marker held in step S15 and in the current frame of step S12. The motion vector of the marker image 6 is calculated from the difference between the marker position and the posture data.

Then, based on the motion vector of this marker image and the distance between the center of the marker image and the air tag obtained in step S14, the air tag to be operated by the user is recognized as which air tag, and the operation instruction is also recognized ( Step S17).

In the next step S18, the function is expressed for the air tag to be operated based on the recognition result in step S17, and the process returns to step S11. If any recognition becomes impossible in step S17, the process returns to step S11.

This makes it possible to operate the air tag intuitively without using a special device.

FIG. 10 is an explanatory diagram of an operation instruction using the marker image 6. If you want to download the air tag 1, bring the marker image 6 close to the air tag 1 and bring them close to each other for a certain period of time, or overlap them. As a result, the data corresponding to the air tag 1 is downloaded.

In the marker image, patterns are drawn on the front and back of the flat mount, and different functions are associated with each. A trash can pattern is drawn on the back of the marker image 6, and when the trash can pattern side is directed toward the camera side, the state as shown in FIG. 11 is obtained. Then, by bringing this trash can pattern closer to the corresponding air tag and bringing it closer for a certain period of time, this air tag can be discarded.

In the above example, the two patterns of markers have been described, but various functions can be realized by preparing patterns corresponding to various other functions. A plurality of markers may be arranged not only on a flat mount but also on a polyhedral mount. Further, a display device such as a liquid crystal display may be configured to switch and display patterns indicating various functions according to a user's instruction.

With such a configuration, it is possible to express various functions on the air tag.

FIG. 12 shows an example in which a light emitting body (LED) attached to the tip of a stick is used as a pointer, and the LED 7 is photographed by a camera of a mobile phone and sent to the head-mounted augmented reality image presenting device 10.

FIG. 13 is an explanatory diagram of a user interface processing procedure for operating the air tag using the LED image 7 shown in FIG. 12 as a pointer. First, the LED image 7 is captured from the camera (step S21). Then, in the next step S22, after binarizing the luminance component of the LED image 7, the LED point light source is detected. At this time, the detected color component or light emission pattern of the LED point light source may be recognized. After that, the arrangement coordinate data of the air tag 1 is acquired in step S23, and the distance between the LED point light source 7 and the air tag 1 is calculated in step S24.

In parallel with this, in step S25, the position of the LED point light source in the captured image of the previous frame is held, and in step S26, the motion vector of the LED point light source 7 is calculated from the difference from the captured image of the current frame. Based on this motion vector and the distance obtained in step S24, the air tag to be operated by the user is recognized as which air tag, and the operation instruction is also recognized (step S27).

In the next step S28, the function is expressed for the air tag to be operated based on the recognition result in step S27, and the process returns to step S21. If any recognition becomes impossible in step S27, the process returns to step S21.

As a result, the air tag can be operated intuitively.

14 and 15 are explanatory views of operation instructions using the LED point light source. The LED point light source used in the present embodiment can change the emission color to yellow or red according to the instruction of the user, and the expression function is changed for each color. The expression function may be changed by changing the light emission pattern (for example, blinking speed) instead of the color. The expression function may be changed by the combination of the color and the emission pattern.

The emission color of the LED point light source 7 is changed to yellow, and as shown in FIG. 14, the LED point light source is brought close to the air tag 1 and kept close for a certain period of time. As a result, the application software corresponding to this air tag is executed.

The emission color of the LED point light source 7 is changed to red, and as shown in FIG. 15, the LED point light source is brought close to the air tag 1 and kept close for a certain period of time. This replaces the air tag with another air tag.

According to this embodiment, various functions can be exhibited in the air tag only by moving the LED point light source.

As described above, the head-mounted extended reality image presentation device and the virtual display object operation method of the embodiment include an imaging means for capturing a subject image, a position detecting means for detecting the position of the imaging means, and the imaging. A direction detecting means for detecting the photographing direction of the means, a calculating means for calculating the photographing range of the subject image from the detection result of the position detecting means and the detection result of the direction detecting means, and the photographing range calculated by the calculating means. Is transmitted to an external database to acquire virtual object data corresponding to the shooting range from the database, and the imaging means is made to capture a pointer image as a part of the subject image to obtain the subject image. A head-mounted extended reality image presenting apparatus including an image synthesizing means for generating a composite image by synthesizing a virtual display object showing virtual object data and a head-mounted display means for displaying the composite image. When the mutual relationship between the pointer image and the virtual display object in the composite image satisfies a predetermined condition, it is assumed that the required operation by the pointer image for the virtual display object is executed, and the virtual object corresponding to the virtual display object is executed. It is characterized in that a required function corresponding to the required operation is expressed in the data.

Further, the predetermined conditions of the head-mounted augmented reality image presenting device of the embodiment and the method of operating the virtual display object thereof are determined by the movement of the pointer image.

Further, in the head-mounted augmented reality image presentation device and its virtual display object operation method of the embodiment, the pointer image is an image of a user's hand wearing the head-mounted display means, and the control means is the pointer image. The range is identified by skin color detection, and the movement of the pointer image is recognized from the difference between frames in the skin color region.

Further, in the head-mounted augmented reality image presenting device of the embodiment and the method of operating the virtual display object, the virtual display object is displayed by giving an acting force corresponding to the movement by the movement of the image of the hand to the virtual display object. It is characterized by controlling the position.

Further, the head-mounted augmented reality image presenting device of the embodiment and the method of operating the virtual display object are such that the virtual display is performed when the movement of the image of the hand is continuously in contact with the virtual display object at two or more points for a predetermined time. It is regarded as an operation of pinching an object, the display position of the virtual display object is moved by following the movement of the pointer image, and the pointer image is released when the pointer image moves from the virtual display object at a predetermined speed or higher. It is characterized in that only the pointer image is deemed to be moved.

Further, the predetermined conditions of the head-mounted augmented reality image presentation device of the embodiment and the method of operating the virtual display object thereof are determined by the two-dimensional geometric pattern of the pointer image.

Further, in the head-mounted augmented reality image presentation device of the embodiment and the method of operating the virtual display object, the pointer image is an image of a marker, and the marker identifies the content of the function with respect to the virtual object data. It is characterized in that a two-dimensional geometric pattern is displayed.

Further, in the head-mounted augmented reality image presentation device and the virtual display object operation method of the embodiment, the image of the marker is an image of a polyhedron, and different two-dimensional geometric patterns are displayed on each surface of the polyhedron. It is characterized by being.

Further, in the head-mounted augmented reality image presentation device and the virtual display object operation method of the embodiment, the marker is a display means, and a plurality of types of the two-dimensional geometric patterns to be displayed on the display means are prepared and switched and displayed. It is characterized by that.

Further, the predetermined conditions of the head-mounted augmented reality image presenting device of the embodiment and the method of operating the virtual display object thereof are determined by at least one of the display color, the display pattern, and the change of the display pattern of the pointer image. It is characterized by.

Further, the head-mounted augmented reality image presenting device of the embodiment and the method of operating the virtual display object thereof are characterized in that the pointer image is an image of a point light source.

Further, the head-mounted augmented reality image presenting device of the embodiment and the method of operating the virtual display object thereof are characterized in that different functions are identified by the emission color or emission pattern of the point light source.

According to each of the above-described embodiments, even if the user's eyes are covered with a head-mounted display screen and the user cannot see his / her own hand, the virtual display object in the composite image can be displayed without any special sensors or the like. It becomes possible to operate.

The head-mounted augmented reality image presenting device according to the present invention can easily operate a virtual display object in an image without using an expensive pointer device while displaying the augmented reality image on the head-mounted display device. This is useful for popularizing head-mounted augmented reality image presentation devices.

1,1a, 1b Virtual display (air tag) 2 User's fingertip image 5 Mobile phone 6 Marker image 7 LED point light source image 10 Head mound type augmented reality image presentation device 11 CPU 14 Built-in camera 19 Image synthesis section 21 Built-in communication control unit 22 Pointer detection recognition unit 23 Pointer monitoring unit 24 Display screen 26 GPS 27 Geomagnetic sensor 30 Virtual database 31 Virtual object data search unit