Composite reality image presentation device

【0001】

[Technical field to which the invention belongs]

The present invention relates to a mixed reality image presenting device.

【0002】

[Conventional technology]

In recent years, a so-called mixed reality image is provided to an observer by superimposing and synthesizing a virtual image generated by a computer on a real image directly observed by the observer or an image taken in the immediate vicinity of the observer. A technology called (MR: Mixed Reality) has been proposed and is being used for games and attractions. In addition, at construction sites such as airfields and factory construction, application to more realistic simulations and confirmation of the progress of the construction at the planned construction site is also being studied.

【0003】

An overview of mixed reality is given in VOL.52, No.3, pp.266-272 (1998) of the Journal of the Institute of Image Information and Television Engineers.

【0004】

Further, Japanese Patent Application Laid-Open No. 2002-163670 shows an example of the above-mentioned mixed reality image presenting device, and superimposes a virtual space image on a real space where changes in color tone and brightness are large to create a mixed reality image. A mixed reality presenting device to be presented and a control method thereof are disclosed.

【0005】

[Patent Document 1]

JP-A-2002-163670 [0006]

[Non-Patent Document 1]

Journal of the Institute of Image Information and Television Engineers, VOL.52, No.3, pp.266-272 (1998) [0007]

[Problems to be Solved by the Invention]

In the conventional composite reality image presenting device including the above-mentioned Japanese Patent Application Laid-Open No. 2002-163670, some processing is not performed on the real image taken by the CCD camera or the like as the real image capturing unit, and the reality in the captured state is not performed. The image and the CG image as a virtual image whose hue, brightness, etc. are adjusted are combined to generate a composite image, which is intended to give the observer a high sense of reality. It wasn't.

【0008】

The present invention has been made by paying attention to such a problem, and an object of the present invention is to provide a mixed reality image presenting device capable of giving an observer a high sense of reality.

【0009】

[Means for solving problems]

In order to achieve the above object, the composite reality image presenting device according to the first invention is a composite reality image presenting device that synthesizes a real image and a virtual image and presents the composite reality image to an observer. A real image capture unit for capturing a real image, a layer image generation unit that generates a layer image based on the real image captured by the real image capture unit, and a layer image generation unit that generates a layer image. It is provided with an image compositing unit that synthesizes the layer image and the virtual image to generate a composite image, and an image presenting unit that presents the composite image generated by the image compositing unit to the observer as a composite reality image. To do.

【0010】

Further, according to the second invention, in the mixed reality image presenting apparatus according to the first invention, the reality image capture unit has a plurality of units, and the difference is calculated based on the reality image captured by each reality image acquisition unit. It has a parallax detection unit for detection.

【0011】

Further, according to the third invention, in the composite reality image presenting apparatus according to the second invention, the layer image generation unit generates a layer image having a new parallax based on the parallax detected by the disparity detection unit. , The image synthesizing unit synthesizes the layer image and the virtual image.

【0012】

Further, the fourth invention is the composite reality image presenting apparatus according to the third invention, in which the image synthesizing unit uses a layer image having the new parallax only for a specific region of the real image, and the specific region is used. For the regions other than the above, the layer image having the parallax detected by the parallax detection unit is used for compositing with the virtual image.

【0013】

Further, the fifth invention further includes a parallax limiting means for limiting the amount of the parallax in generating the layer image having the new parallax in the mixed reality image presenting apparatus according to the third invention.

【0014】

Further, the sixth invention is the viewpoint position detection unit for detecting the viewpoint position of the observer and the viewpoint posture detection for detecting the viewpoint posture of the observer in the composite reality image presenting apparatus according to the first or third invention. The layer image generation unit further includes a unit, and the layer image generation unit is based on the viewpoint position detected by the viewpoint position detection unit, the viewpoint posture detected by the viewpoint posture detection unit, and an instruction from the outside. The conditions including the timing of generating the image, the range of the layer image, the arrangement, the contrast, and the color tone are determined.

【0015】

Further, the seventh invention is the composite reality image presenting apparatus according to the sixth invention, in the case of an emergency, or the viewpoint position detected by the viewpoint position detection unit and the viewpoint attitude detection unit. When the detected viewpoint orientation exceeds a predetermined control range, the image synthesizing unit stops the synthesizing process of the layer image and the real image, and sends only the real image to the image presenting unit.

【0016】

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram of a mixed reality image presentation device according to an embodiment of the present invention. In FIG. 1, the real image capture units (R) 11 and (L) 13 are parts that capture a real image by capturing a subject in the real space 10, and are captured by a CCD camera as an image sensor and the CCD camera. This includes a signal processing unit that performs predetermined signal processing on the captured image data obtained by the above process, and a memory that stores the captured image data that has undergone signal processing. The real image capture unit (R) 11 is a portion for acquiring captured image data to be presented to the right eye of the observer. The real image capture unit (L) 13 is a portion for acquiring captured image data to be presented to the left eye of the observer.

【0017】

The parallax detection unit 12 detects the parallax of the photographer's left and right eyes based on the captured image data from the real image capturing unit (R) 11 and the captured image data from the real image capturing unit (L) 13. It is the part to do. Further, the layer generation unit 14, which is a feature of the present invention, has a plurality of layer generation units 14 based on the captured image data from the real image capture unit (R) 11 and the captured image data from the real image capture unit (L) 13. Generate a layer image (image layer). A specific example of the layer image will be described later.

【0018】

The viewpoint position detection unit 18 is a portion that detects the viewpoint position of the observer. The viewpoint posture detection unit 19 is a part that detects the viewpoint attitude of the observer. The web terminal 20 is a part that acquires desired virtual image data from the Internet. The virtual image storage unit 21 is a portion that stores virtual image data, and is realized by, for example, a hard disk of a PC (personal computer). Specific examples of virtual images will be described later.

【0019】

The virtual image generation unit 17 is a virtual image stored in the virtual image storage unit 21, virtual image data from the web terminal 20, viewpoint position data from the viewpoint position detection unit 18, and viewpoint posture detection unit 19. This is a part that generates virtual image data in the virtual space based on the viewpoint attitude data. The image composition unit 15 generates a composite image based on the layer image from the layer-generation unit 14 and the virtual image data from the virtual image generation unit 17. The mixed reality image display unit (R) 22-1, (L) 22-2 as an image presentation unit presents the composite image from the image composition unit 15 as a mixed reality image to the left and right eyes of the observer 25. To do. The parallax limiter 16 as a parallax limiting means is a means for limiting the amount of parallax in generating a layer image having an arbitrary new parallax from a layer image of a real image. Controller 24, various fingers for its operation to each unit described above is a portion that gives the shows.

【0020】

The layer generation unit 14 can generate a layer image having a new parallax based on the parallax detected by the parallax detection unit 12. Then, the image synthesizing unit 15 uses the layer image having the new parallax only for the specific region of the real image, and uses the layer image having the parallax detected by the parallax detecting unit 12 for the region other than the specific region to create a virtual image. It is conceivable to synthesize with.

【0021】

Further, the layer generation unit 14 generates a layer image based on the viewpoint position detected by the viewpoint position detection unit 18, the viewpoint posture detected by the viewpoint posture detection unit 19, and an instruction from the external controller 24. It is conceivable to determine conditions including timing, layer image range, placement, contrast and color tone.

【0022】

FIG. 2 is a diagram showing an example in which the mixed reality image presentation device of the present embodiment is applied to a specific system configuration. Here, the real image capture units (R) 11 and (L) 13 in FIG. 1 are realized as CCD cameras (R) 111-1, (L) 111-2, respectively. The viewpoint position detection unit 18 is realized as a GPS (Global Positioning System) 118. The viewpoint attitude detection unit 19 is realized as an HMS (Head Motion Sensor) 119. The mixed reality image display units (R) 22-1, (L) 22-2 are realized as, for example, a video see-through type HMD (Head Mounted Display) 122. The controller 24 is realized by the controller 124, which is an operating means that can be operated by the observer.

【0023】

Further, the virtual image storage unit 21 of FIG. 1 is realized by the hard disk in the PC 150 connected to the controller 124, but can also be obtained from the Internet by the web terminal 20 via a wireless line or the like.

【0024】

The observer can operate the controller 124 to extract virtual image data from the web terminal 20 or the virtual image storage unit 21, or select a specific layer image from a plurality of layer images.

【0025】

It is also possible to realize the functions of the GPS 118 and the controller 124 described above by the PC 150.

【0026】

FIG. 3 is a flowchart for explaining the operation of the mixed reality image presenting device according to the present embodiment. First, it is determined based on the instruction from the controller 24 whether to extract the virtual image data from the virtual image storage unit 21 or the web terminal 20 (step S1). Next, the viewpoint position data of the observer detected by the viewpoint position detection unit 18 (GPS118) is transferred to the virtual image generation unit 17 (step S2). Next, the viewpoint attitude data of the observer detected by the viewpoint attitude detection unit 19 (HMS119) is transferred to the virtual image generation unit 17 (step S3).

【0027】

Next, the virtual image generation unit 17 generates a virtual image based on the viewpoint position, the posture position, and the instruction from the controller 24 (step S4). A specific example of the instruction from the controller 24 will be described later. Next, the real image data is transferred from the real image capturing units (R) 11 and (L) 13 to the parallax detecting unit 12 to detect the parallax (step S5). Next, the layer generation unit 14 generates a predetermined layer image based on the real image data sent from the real image acquisition units (R) 11 and (L) 13 and the parallax information from the parallax detection unit 12. (Step S6). Specifically, a plurality of types of layer images are generated by reading out the real image data stored in advance and performing known image processing. A specific example of the layer image will be described later.

【0028】

Next, the layer image selected according to the purpose from the plurality of layer images generated from the real image data and the virtual image data generated by the virtual image generation unit 17 are transferred to the image composition unit 15 and combined. The process related to superimposition is performed to generate a composite image (step S7).

【0029】

Next, the generated composite image is transferred to the mixed reality image display unit (R) 22-1, (L) 22-2 for display (step S8), and is presented to the left and right eyes of the observer 25. Next, it is determined whether to continue the screen display according to the instruction from the controller 124 (step S9), and if the screen display is to be continued, the process returns to step S1 and the above process is repeated, and the screen display is displayed in step S9. The process ends when it is determined to end.

【0030】

FIG. 4 is a flowchart for explaining other operations of the mixed reality image presenting apparatus according to the present embodiment. Here, it will be described assuming that some kind of emergency occurs while displaying an image.

【0031】

First, it is determined based on the instruction from the controller 24 whether to extract the virtual image data from the virtual image storage unit 21 or the web terminal 20 (step S20). Next, it is determined whether or not there is an emergency instruction from the controller 24 (step S21), and if it is an emergency instruction, only the actual image is displayed to make the observer recognize that it is an emergency. (Step S31) is performed, and a predetermined warning is superimposed and displayed (Step S32) to end the process.

【0032】

If it is determined in step S21 that the instruction is not an urgent instruction, then the observer's viewpoint position data detected by the viewpoint position detection unit 18 (GPS118) is transferred to the virtual image generation unit 17 ( Step S22). Next, the viewpoint attitude data of the observer detected by the viewpoint attitude detection unit 19 (HMS119) is transferred to the virtual image generation unit 17 (step S23).

【0033】

Next, based on the observer's viewpoint position data and viewpoint attitude data, it is determined whether or not the range that can be controlled by the controller 24 is exceeded (step S24), and if it is exceeded, the above steps S31 and S32 are executed. And end the process.

【0034】

If it is determined in step S24 that the controllable range is not exceeded, then the virtual image generation unit 17 generates a virtual image based on the viewpoint position, the posture position, and the instruction from the controller 24. (Step S25). Next, the real image data is transferred from the real image capturing units (R) 11 and (L) 13 to the parallax detecting unit 12 to detect the parallax (step S26). Next, the layer generation unit 14 generates a predetermined layer image based on the real image data sent from the real image acquisition units (R) 11 and (L) 13 and the parallax information from the parallax detection unit 12. (Step S27). Specifically, a plurality of types of layer images are generated by reading out the real image data stored in advance and performing known image processing. A specific example of the layer image will be described later.

【0035】

Next, the layer image selected according to the purpose from the plurality of layer images generated from the real image data and the virtual image data generated by the virtual image generation unit 17 are transferred to the image composition unit 15 and combined. The process related to superimposition is performed to generate a composite image (step S28). Next, the generated composite image is transferred to the mixed reality image display unit (R) 22-1, (L) 22-2 for display (step S29), and is presented to the left and right eyes of the observer 25. Next, it is determined whether to continue the screen display according to the instruction from the controller 124 (step S30), and if the screen display is to be continued, the process returns to step S20 and the above process is repeated, and the screen display is displayed in step S30. If it is determined to end, the process ends.

【0036】

FIG. 5 is a diagram for explaining a specific example of layer image generation in the layer generation unit 14, and shows a situation in which persons A and B are performing a show on the stage 300 and the audience 301 is watching this. ing. In FIG. 5, the image of the portion other than the portion indicated by C is defined as the real image 302 captured from the real image capturing units (R) 11 and (L) 13. Conventionally, the real image 302 is combined with the virtual image 303 representing the person C without any processing, but the layer generation unit 14 according to the present invention has detected the viewpoint position, the posture position, and the controller. Based on the instructions from 24, for example, the following layer image can be generated. As a specific example of the instruction from the controller 24, when the controller 24 detects a specific voice or detects a specific landmark (mark) or image inserted in the real image, an instruction to generate a layer image is issued. And so on. Of course, the generated layer image is not limited to the following.

【0037】

1. Layer image in which the part B is erased from the real image 302 shown in Fig. 5. 2. In the real image 302 shown in Fig. 5, the layer image showing the effect of shining a spotlight on the part B 3. The real image shown in Fig. 5. In 302, the layer image layer generation unit 14 that emphasizes the three-dimensional effect of the portion B generates an image obtained by synthesizing an arbitrary layer image selected from these layer images and the virtual image 303, and is in the eyes of the observer. By presenting it, the observer can feel a high sense of reality. In addition, by synthesizing layer images according to the situation with virtual images, it is possible to realize space saving, low cost, and shortening of production days for various attractions, stage lighting such as shows at stage 300, stage equipment, etc. It is possible to provide a mixed reality image with a larger amount of information.

【0038】

FIG. 6 shows a specific example in which the present invention is applied to attractions at theme parks. The observer 401 riding on the roller coaster 402 can enjoy various attractions by wearing the mixed reality image presenting device 400 according to the present invention.

【0039】

FIGS. 7 (A) and 7 (B) show specific examples in which the present invention is applied to a city planning simulation. Here, the image captured from the real image capturing units (R) 11 and (L) 13 is referred to as the real image 500. This reality image 500 shows the street of a certain city, but when it is planned to construct a building as shown in Fig. 7 (B) in the part indicated by A of such a reality image 500. Think. In this case, conventionally, the situation after construction was grasped by inserting the pre-generated virtual image 501 into the portion shown by A in FIG. 7 (A), but in the present invention, the real image 500 is synthesized as it is. Instead, the following layer image is generated based on the real image 500, combined with the virtual image 501, and presented to give the observer a high sense of reality.

【0040】

1. Only the part indicated by A of the real image 500 (the part where the virtual image 501 is inserted) is generated as a layer image with the color tone and contrast changed or a black and white layer image. By synthesizing and displaying this with the virtual image 501, which is a building scheduled to be constructed, and presenting it to the observer, the observer can confirm the difference between the current townscape and the townscape after construction. ..

【0041】

2. From the real image 500, generate a layer image of daytime brightness and night scene, combine this with the virtual image 501, display it, and present it to the observer. You can see how it differs from nighttime.

【0042】

[Effect of the invention]

According to the present invention, it is possible to provide a mixed reality image presenting device that can make an observer feel a high sense of reality.

[Simple explanation of drawings]

FIG. 1 is a functional block diagram of a mixed reality image presentation device according to an embodiment of the present invention.

FIG. 2 is a diagram showing an example in which the mixed reality image presentation device of the present embodiment is applied to a specific system configuration.

FIG. 3 is a flowchart for explaining the operation of the mixed reality image presenting device according to the present embodiment.

FIG. 4 is a flowchart for explaining other operations of the mixed reality image presenting apparatus according to the present embodiment.

FIG. 5 is a diagram for explaining a specific example of layer image generation in the layer generation unit 14.

FIG. 6 is a diagram showing a specific example of applying the present invention to attractions at theme parks.

FIG. 7 is a diagram showing a specific example in which the present invention is applied to a city planning simulation.

[Explanation of symbols]

10 ... Real space, 11 ... Real image capture section (R), 12 ... Parallax detection section, 13 ... Real image capture section (L), 14 ... Layer generation section, 15 ... image composition unit, 16 ... parallax limiter, 17 ... virtual image generation unit, 18 ... viewpoint position detection unit, 19 ... viewpoint orientation detection unit, 20 ... web terminal, 21. .. Virtual image storage unit, 22-1 ... Composite reality image display unit (R), 22-2 ... Composite reality image display unit (L), 24 ... Controller, 25 ... Observation Person.