Head-mounted display

The present invention relates to a head-mounted display that can be worn on the user's head to view an image.

Conventionally, a head-mounted display as shown in Patent Document 1 has been known. Such a head-mounted display is composed of a head-mounted portion that is attached to the user's head and an image generation portion that is attached to the front portion of the head-mounted portion and allows the user to visually recognize an image. It is expected that such a head-mounted display will be widely used in work sites such as production factories in the future. For example, if the image generation unit makes the operator visually recognize the instruction sheet and the design drawing, the worker can perform more reliable work by looking at the instruction sheet and the design drawing as needed. Since the worker does not have to pick up and see the instructions and blueprints on the paper medium, he / she can see the instructions and blueprints displayed on the image display while working without stopping the work. Not only can workability be improved, but work mistakes can be prevented.

Conventionally, in order to switch the image generated by the image generation unit, it was necessary to press a physical button, but if oil is attached to the hand, the button slips due to the oil, and the above There was a problem that it was difficult to operate the buttons. Further, when the hand is dirty with oil or the like, there is a problem that the button and its surroundings become dirty. Further, since the button is provided on the head-mounted display, the button cannot be seen directly while the head-mounted display is attached, and the button must be operated by fumbling, which causes a problem that workability deteriorates. It was.

<patcit num="1"><text>Japanese Unexamined Patent Publication No. 9-211374</text></patcit>

<p> The present invention has been made to solve the above problems and provide a head-mounted display capable of performing operations such as switching screens without touching physical buttons.</p>

<p> The invention according to claim 1 made to solve the above problems includes a head-mounted portion to be mounted on the user's head and a head-mounted portion. In a head-mounted display that is attached to the head-mounted portion and has an image generating portion that allows the user to visually recognize an image. A virtual panel generation means that generates a virtual panel image, which is an image in which virtual panels to which each command is assigned are visually recognized as if they are stacked in the depth direction, and outputs the virtual panel image to the image generation unit. A hand position motion detecting means for detecting the position and motion of the user's hand in the depth direction, The virtual panel to be selected by the user is detected from the position in the depth direction of the user's hand detected by the hand position motion detecting means, and the command assigned to the virtual panel is selected from the user's selection motion. It is characterized by having a command selection means.</p><p> The invention according to claim 2 is the invention according to claim 1. Hand position motion detection means, Multiple imaging devices attached to the front of the head-mounted portion at intervals, It is composed of a hand coordinate calculation means that calculates the imaging coordinates of the user's hand from the images captured by each imaging device and calculates the coordinates in the depth direction of the user's hand from each imaging coordinate. The command selection means calculates the position and operation of the user's hand in the depth direction based on the coordinates of the user's hand in the depth direction calculated by the hand coordinate calculation means, and calculates the position and operation of the user's hand in the depth direction. And, based on the operation, the command assigned to the virtual panel is selected.</p><p> The invention according to claim 3 is the invention according to claim 1. Hand position motion detection means, Multiple imaging devices attached to the front of the head-mounted portion at intervals, A hand coordinate calculation means that calculates the imaging coordinates of the user's hand from the images captured by each imaging device and calculates the coordinates in the depth direction of the user's hand from the imaging coordinates. It consists of a sensor attached to the user's hand that detects the movement of the user's hand and generates a hand movement signal. The command selection method is Based on the coordinates in the hand depth direction of the user calculated by the hand coordinate calculation means, the virtual panel that the user is trying to select is recognized, and the virtual panel is recognized. The user determines the selection operation of the virtual panel from the hand operation signal detected by the sensor, and selects a command assigned to the virtual panel.</p><p> The invention according to claim 4 is the invention according to claim 1 to 3. Virtual panel generation means Before generating the virtual panel image A marker image that is virtually displayed at a near position and a distant position is generated from the user and output to the image generation unit. From the position information in the depth direction of the user's hand detected by the hand position motion detecting means from the selection motion of each marker image of the user, the coordinates in the depth direction of the far position and the near position are calculated. It is characterized in that a plurality of virtual panels are stacked and arranged between the calculated near position and the far position to generate a virtual panel image.</p><p> The invention according to claim 5 is the invention according to claim 1 to 4. The virtual panel generation means is characterized in that the virtual panel at a position of the user's hand detected by the command selection means is highlighted.</p><p> The invention according to claim 6 is the invention according to claim 1 to 5. The command selection means is the user's hand detected by the hand position motion detection means.<u style="single">position</u>However, when it is determined that the hand position is within a predetermined range for a certain period of time or more, the hand position motion detecting means selects a virtual panel at a position of the user's hand detected by the hand position motion detecting means.</p><p> The invention according to claim 7 is the invention according to claim 1 to 5, wherein the command selection means is a selection operation in which the user's hand motion detected by the hand position motion detection means is preset. When it is determined that there is, the hand position motion detecting means selects a virtual panel at a certain position of the user's hand detected.</p>

<p> The invention according to claim 1 is to generate a virtual panel image which is an image in which virtual panels to which each command is assigned are visually recognized as if they are stacked in the depth direction, and output the virtual panel to the image generation unit. The virtual means that the user intends to select from the means, the hand position motion detecting means for detecting the depth direction position and motion of the user's hand, and the depth direction position of the user's hand detected by the hand position motion detecting means. It is characterized by having a command selection means for detecting a panel and selecting a command assigned to the virtual panel from the selection operation of the user. Therefore, the user can perform operations such as switching screens without touching physical buttons. If the user's head moves even slightly, the position of the virtual panel displayed by the head-mounted display and the detected hand will deviate significantly in the plane direction perpendicular to the user's line-of-sight direction, but with respect to the head-mounted display. The positional relationship of the user's hand in the depth direction is almost unchanged. Normally, the head of a user wearing a head-mounted display is constantly moving, albeit slightly, but even with such natural movement of the user's head, in the present invention, the depth direction of the user's hand with respect to the head-mounted display Since it was decided to detect the position of and select the command, the positional relationship between the virtual panel and the user's hand in the depth direction hardly changes, so there are extremely few false positives, the reliability is high, and the command is selected. It becomes possible. In addition, the virtual panel image can be displayed on the image display unit regardless of the direction in which the user's head is turned, and even in a situation where the user is forced to perform unnatural work, the user can display the virtual panel on the virtual panel. It is possible to select the assigned command, which is very useful.</p><p> According to the second aspect of the present invention, the user can use the hand position motion detecting means from a plurality of imaging devices attached to the front portion of the head mounting portion at intervals and the captured images captured by the respective imaging devices. It is composed of a hand coordinate calculation means that calculates the imaging coordinates of the hand and calculates the coordinates in the depth direction of the user's hand from each imaging coordinate, and the command selection means is the user calculated by the hand coordinate calculation means. Calculates the position and movement of the user's hand in the depth direction based on the depth direction coordinates of the user's hand, and selects the command assigned to the virtual panel based on the position and movement of the user's hand in the depth direction. It is characterized by doing. Therefore, the user does not need to wear the sensor on his / her hand to select a command, and he / she can freely use his / her hand. Further, even if the user's head shakes, the plurality of imaging devices also move as the user's head moves, and even if the position of the user's hand with respect to the photographing device changes, the difference in the user's hand imaged by the plurality of imaging devices. Does not change, so that the coordinates of the user's hand in the depth direction with respect to the head-mounted display can be stably detected.</p><p> According to the third aspect of the present invention, the user can use the hand position motion detecting means from a plurality of imaging devices attached to the front portion of the head mounting portion at intervals and the captured images captured by the respective imaging devices. A hand coordinate calculation means that calculates the imaging coordinates of the hand and calculates the coordinates in the depth direction of the user's hand from each imaging coordinate, and the user's hand that detects the motion of the user's hand and generates a hand motion signal. The command selection means recognizes the virtual panel that the user is trying to select based on the coordinates in the hand depth direction of the user calculated by the hand coordinate calculation means, and the sensor recognizes the virtual panel attached to the user. The user determines the selection operation of the virtual panel from the detected manual operation signal, and selects a command assigned to the virtual panel. Therefore, even if the user's head moves and the image is taken as if the hand was shaken by the image pickup device, the sensor attached to the user's hand detects the selection action of the virtual panel. The movement of the head can reliably prevent the virtual panel from being erroneously selected, and can reliably detect the virtual panel that the user wants to select.</p><p> According to the fourth aspect of the present invention, the virtual panel generating means generates a marker image that is virtually displayed at a near position and a distant position from the user's eyeball before generating the virtual panel image, and causes the image generation unit to generate the marker image. The coordinates in the depth direction of the distant position and the near position are calculated from the position information in the depth direction of the user's hand detected by the hand position motion detecting means from the output and the selection operation of each marker image of the user. It is characterized in that a plurality of virtual panels are stacked and arranged between a near position and a distant position to generate a virtual panel image. In this way, if the user can determine the correspondence between the position that the user wants to recognize as the position in the depth direction of the marker image and the actual position in the depth direction of the hand, the positions are stacked in the depth direction. Even if the virtual panel is displayed on a flat surface so that the user feels it, it is possible to absorb individual differences for each user and generate the virtual panel in the operating range of the hand that is easy for the user to operate.</p><p> The invention according to claim 5 is characterized in that the virtual panel generation means highlights a virtual panel at a position of a user's hand detected by the command selection means. Therefore, the user can check the virtual panel corresponding to the command to be selected before selecting the command, and it is possible to avoid an erroneous operation. The virtual panels are displayed so that the user feels that they are stacked in the depth direction, but since it is a flat image, it is difficult to correspond the positions of the virtual panels that appear to be stacked and the positions in the depth direction of the hands. However, in the invention according to claim 5, since the virtual panel at the position where the hand is located is highlighted, the user can adjust the depth position of the hand while looking at the highlighted virtual panel and desire without error. It is possible to select the virtual panel of.</p><p> According to the sixth aspect of the present invention, when the command selection means determines that the user's hand movement detected by the hand position movement detecting means is within a predetermined range for a certain period of time or longer, the hand position movement detecting means. It is characterized by selecting a virtual panel at a position of the user's hand detected by. Therefore, the user can select a command by simply stopping the hand for a certain period of time or more, which is user-friendly.</p><p> According to the invention of claim 7, when the command selection means determines that the user's hand motion detected by the hand position motion detection means is a preset selection motion, the hand position motion detection means. It is characterized by selecting a virtual panel at a position of the user's hand detected by. Therefore, since the command is selected only by the preset selection operation, it is possible to avoid accidentally selecting the command. If the hand movement that does not occur even if the hand is stopped and the head is moved while the imaging device is imaging the hand is set in advance as the selection movement, erroneous detection is performed even if the head moves. It is possible to select a virtual panel at a position where the user has a hand without having to do so.</p>

(Overview of Head Mounted Display of the Present Invention) Hereinafter, preferred embodiments of the present invention will be shown with reference to the drawings. FIG. 1 is an overall view of the head-mounted display 100 of the first embodiment. The head-mounted display 100 of the first embodiment includes a head-mounted unit 10, an image generation unit 20, image pickup devices 31, 32, and a control unit 40.

In the embodiment shown in the figure, the head-mounted portion 10 has a frame shape of eyeglasses, but may have a helmet shape or the like, and includes all those having a structure worn on the user's head.

The image generation unit 20 is attached to the front side of the head mounting unit 10. The image generation unit 20 generates an image and makes the user visually recognize the image. In the present embodiment, the user is made to visually recognize the image by scanning the laser beam directly into the user's eyeball. The image includes a still image file, a text file, and a moving image file. The user can visually recognize the outside world at the same time as visually recognizing the image generated by the image generation unit 20. In the embodiment shown in FIG. 1, the image generation unit 20 is attached to one side front portion of the head mounting portion 10, but the image generation unit 20 is attached to both side front portions of the head mounting portion 10. The user may be configured to visually recognize the image generated by the image generation unit 20 with both eyes.

The image pickup devices 31 and 32 are attached to the front portion of the head mounting portion 10 at regular intervals. The image pickup devices 31 and 32 are composed of an image sensor, an imaging optical system, and the like.

The control unit 40 controls the head-mounted display 100 of the present invention, and is connected to the image generation unit 20, the image pickup devices 31, and 32.

The outline of the present invention will be described with reference to FIGS. 2 and 3, and the outline of the present invention will be described below with reference to explanatory views. FIG. 2 is a diagram showing the user's field of view. Reference numeral 500 is a virtual panel image generated by the image generation unit 20. The virtual panel image 500 is an image in which a plurality of virtual panels 501 appear to be stacked and arranged in the depth direction (direction far from the front of the user's line of sight). In the embodiment shown in the figure, the virtual panel 501 is displayed so as to look down slightly from above when viewed from the user. That is, the front virtual panel 501 displays the lower end of the virtual panel 501 on the lower side, and the back virtual panel 501 displays the upper end of the virtual panel 501 on the upper side. Each command is assigned to each virtual panel 501.

When the user's hand (finger) 601 is in a virtual panel 501 for a certain period of time or longer, the command corresponding to the virtual panel 501 is executed. In the first embodiment, as shown in FIG. 3A, the left and right imaging devices 31 and 32 detect the coordinates (z) in the depth direction of the user's hand (finger) 601 and the control unit 40 detects the coordinates (z) in the depth direction. It is determined whether or not the user's hand (finger) 601 is on the virtual panel 501 for a certain period of time or longer. Note that the user's hand (finger) 601 is in a certain virtual panel 501. For example, when selecting a virtual panel 501 stacked in the back, the virtual panel 501 in the foreground is sequentially penetrated. It means that the user's hand (finger) 601 is on the virtual panel 501 at the front. In the implementation siblings shown in FIG. 2 (B), the user's hand (finger) 601 penetrates the first virtual panel 501a and the second virtual panel 501b, and the third virtual panel 501c does not. From the front to the back, the third virtual panel 501c, where the user's hand (finger) 601 penetrates the virtual panels 501a and b, is selected.

If the user's head moves even slightly, the positions of the virtual panel 501 displayed by the image generator 20 and the detected user's hand (finger) 601 are large in the plane direction perpendicular to the user's line-of-sight direction. Although it shifts, the positional relationship of the user's hand in the depth direction with respect to the head-mounted display 100 is almost unchanged. Normally, the head of the user wearing the head-mounted display 100 is constantly moving, albeit slightly, but even with such natural movement of the user's head, in the present invention, the user's hand with respect to the head-mounted display 100 is used. Since it was decided to detect the position in the depth direction and select the command, the positional relationship between the virtual panel and the user's hand in the depth direction hardly changes, so there are extremely few false positives, the reliability is high, and the command can be executed. It becomes possible to select.

As shown in Fig. 2, when the virtual panel 501 that gives a sense of depth as if it is stacked concentrically around the user at predetermined intervals from the front to the back is displayed, the user's head Since the positional relationship between the virtual panel 501 and the depth direction does not change even if there is movement, it is possible to select a command with less false detection.

The commands assigned to the virtual panel 501 include commands for manipulating the images generated by the image generation unit 20. For example, a command for advancing an image, a command for moving an image back, a command for enlarging / reducing an image, a command for displaying a list of images, and the like are included. The virtual panel image 500 may have a tree structure as shown in FIG. 12B. For example, in the example shown in (B) of FIG. 12, the user selects the "select image" command in the tree hierarchy, selects the "display list of folders" command in the second hierarchy, and selects the "display folder list" command in the third hierarchy. If you select the "Select Folder a" command, select the "Select Folder A" command in the 4th layer, and select the "Show File 1" command in the bottom layer, File 1 will be the image generator. Displayed at 20. Alternatively, when the user selects the "Proceed to next file" command in the lowest layer while the file is displayed in the image generation unit 20, the next file in the same folder is displayed in the image generation unit 20. Will be done.

(Block diagram of the head-mounted display of the first embodiment) The control unit 40 includes CPU 41, RAM 42, ROM 43, non-volatile memory 44, right image pickup device controller 45, right image pickup device VRAM 46, left image pickup device controller 47, left image pickup device VRAM 48, image generator controller 49, and image generator. It consists of VRAM50 and interface 51, which are connected to each other by bus 55.

The CPU (abbreviation of Central Processing Unit) 41 cooperates with RAM (abbreviation of Random Access Memory) 42 and ROM (abbreviation of Read Only Memory) 43 to perform various operations and processes.

The RAM 42 temporarily stores the program processed by the CPU 41 and the data processed by the CPU 41 in the address space.

Various programs and parameters that control the head-mounted display 100 are stored in the ROM 43. The various programs are processed by the CPU 41 to realize various functions. The ROM 43 stores programs and data such as a virtual panel generation program 43a, a hand position detection program 43b, a hand coordinate calculation means 43c, a command selection means 43d, and a hand matching data 43e, which will be described later. It is also possible to store these programs and data in the non-volatile memory 44.

Virtual panel generating program 43a is to the image generator controller 49, life generates a virtual panel image 500 outputs the instruction. The "virtual panel generation means" for generating the virtual panel image 500 is composed of a virtual panel generation program 43a, a CPU 41, a RAM 42, an image generation unit controller 49, and a VRAM 50 for an image generation unit.

The hand position detection program 43b collates the "images" captured by the right image pickup device 31 and the left image pickup device 32 with the hand matching data 43e described later, detects the user's hand, and detects the position of the user's hand. Then, the "imaging coordinates", which are the coordinate positions of the user's hand up and down (y) and left and right (x) in the "image captured", are calculated.

The hand coordinate calculation means 43c calculates the "hand coordinates" of the user's hand from each "imaging coordinate" (two-dimensional coordinates) of the user's hand calculated by the hand position detection program 43b. The "hand coordinates" are three-dimensional coordinates of the position of the user's hand with reference to the head-mounted display 100.

The command selection means 43d calculates the position and motion of the user's hand in the depth direction based on the "hand coordinates" of the position of the user's hand calculated by the hand coordinate calculation means 43c, and calculates the position and movement of the user's hand in the depth direction. Select the command assigned to the virtual panel 501 based on its position and behavior.

The hand matching data 43e is a set of data representing the contours of a large number of hands, fingers, and the like.

The virtual panel generation program 43a, the hand position detection program 43b, the hand coordinate calculation means 43c, and the command selection means 43d may be configured as an ASIC (Application Specific Integrated Circuit).

The non-volatile memory 44 stores various settings and image files of the head-mounted display 100.

The right image pickup device controller 45 is connected to the bus 55, the right image pickup device 31, and the right image pickup device VRAM 46. The right image pickup device controller 45 controls the right image pickup device 31 and generates an "image capture image" by gradation-expressing the output of the image sensor of the right image pickup device 31 described later for each color of RGB, and is used for the right image pickup device. Save to VRAM46.

The VRAM 46 for the right image pickup device is connected to the right image pickup device controller 45 and the bus 55. The VRAM 46 for the right image pickup device delivers the stored "captured image" to the bus 55.

The left image pickup device controller 47 is connected to the left image pickup device 32. The left image pickup device controller 47 has the same structure as the right image pickup device controller 45. The left image pickup device VRAM 48 is connected to the left image pickup device controller 47 and the bus 55. The VRAM 48 for the left image pickup device has the same structure as the VRAM 46 for the right image pickup device.

The image generation unit controller 49 is connected to the image generation unit 20, the image generation unit VRAM 50, and the bus 55. The image generation unit controller 49 has a so-called GPU (Graphics Processing Unit), receives a command from the virtual panel generation program 43a, generates a virtual panel image 500 which is pixel data, and stores it in the VRAM 50 for the image generation unit. , The virtual panel image 500 stored in the VRAM 50 for the image generation unit is output to the image generation unit 20 as an image signal. Further, the image generation unit 20 generates pixel data "image data" from the "image signal" and "image file" input to the interface 51, saves the image data in the image generation unit VRAM50, and also stores the image data in the image generation unit VRAM50. The "image data" saved in the VRAM 50 for use is output to the image generation unit 20 as an "image signal". Further, the image generation unit controller 49 outputs a control signal to the image generation unit 20. The control signal includes control signals such as turning on / off the power of the image generation unit 20, adjusting the image position, and adjusting the focus.

The image generator VRAM 50 is connected to the image generator controller 49 and the bus 55. The VRAM 50 for the image generation unit stores the virtual panel image 500 generated by the image generation unit controller 49, and outputs the virtual panel image 500 to the image generation unit controller 49.

Interface 51 transforms the physical and logical form of the signal. The interface 51 is connected to the operation switch 60, the power lamp 61, and the bus 55. The operation switch 60 is a switch for performing operations such as turning the power of the head-mounted display 100 into an ON / OFF state. The power lamp 61 is lit when the head-mounted display 100 is in the ON state. Further, an "image signal" and an "image file" are input to the interface 51. The "image signal" input to the interface 51 is input to the VRAM 50 for the image generation unit via the bus 55. Further, the "image file" input to the interface 51 is stored in at least one of the RAM 42 and the non-volatile memory 44 via the bus 55.

The right image pickup device 31 and the left image pickup device 32 include an image sensor, an imaging optical system, and a focus device. The image sensor detects the light on the sensor surface and outputs a two-dimensional image signal. Image sensors include CCD (Charge Coupled Device Image SenSor) and CMOS (Complementary Mwtal Oxide Semiconductor). The imaging optical system is composed of a single lens or a plurality of lenses, and forms an incoming image on an image sensor. The focus device is a device that focuses an image that has entered the imaging optical system so that it is imaged on an image sensor. The image of the image to be imaged formed by the imaging optical system is converted into an image signal by the image sensor, and the image signal is output to the right image pickup apparatus controller 45. When the imaging optical system realizes pan focus, the focus device is unnecessary.

The "hand position motion detecting means" for detecting the position or motion of the user's hand includes the hand position detection program 43b, CPU41, RAM42, right image pickup device 31, right image pickup device controller 45, right image pickup device VRAM46, and left image pickup device 32. It is composed of a left image pickup device controller 47 and a left image pickup device VRAM 48.

The image generation unit 20 is a device that generates an image and makes the user visually recognize the image, and has a configuration as shown in FIG. 5, for example. The image generation unit 20 includes an image light generation unit 21, a collimating optical system 22, a horizontal scanning unit 23, a vertical scanning unit 24, a relay optical system 25, and a relay optical system 26.

The image light generation unit 21 is a device that reads out an image signal output by the image generation unit controller 49 for each dot clock, and intensity-modulates the image light according to the read image signal to generate image light. The image generation unit 21 includes a signal processing circuit 211, a light source unit 212, and a photosynthesis unit 213.

The signal processing circuit 211 is connected to the image generator controller 49. The signal processing circuit 211 has B (blue), G (green), and R (red) image signals 214a, which are elements for generating image light, based on the image signal input from the image generator controller 49. ~ 214c is generated and output to the light source unit 212. Further, the signal processing circuit 211 is connected to the horizontal scanning control circuit 23b of the horizontal scanning unit 23, which will be described later, and is horizontally driven to the horizontal scanning control circuit 23b based on the image signal input from the image generator controller 49. Output signal 215. Further, the signal processing circuit 211 is connected to the vertical scanning control circuit 24b described later, and outputs the vertical drive signal 216 to the vertical scanning control circuit 24b based on the image signal input from the image generator controller 49.

The light source unit 212 is composed of a B laser driver 212a, a G laser driver 212b, an R laser driver 212c, a B laser 212d, a G laser 212e, and an R laser 212f. The B laser driver 212a drives the B laser 212d based on the B (blue) image signal 214a output from the signal processing circuit 211 for each dot clock. The B laser 212d emits intensity-modulated blue laser light based on the B (blue) image signal 214a. Similarly, the G laser 212e and the R laser 212f also emit intensity-modulated green laser light and red laser light, respectively.

Each laser 212d to 212f includes a semiconductor laser and a fixed laser with a harmonic generation function. When a semiconductor laser is used, the drive current is directly modulated to modulate the intensity of the laser beam. When a fixed laser with a harmonic generation function is used, each laser 212d to 212f is provided with an external modulator to modulate the intensity of the laser beam.

The photosynthetic unit 213 is composed of collimating optical systems 213a to 213c, dichroic mirrors 213d to 213f, and coupled optical systems 213g. The collimating optical systems 213a to 213c are respectively arranged in front of the lasers 212d to 212f, and the laser light emitted by each of the lasers 212d to 212f is made into parallel light. The dichroic mirrors 213d to 213f are respectively arranged in front of the collimating optical systems 213a to 213c, and each of the laser beams parallelized by the collimating optical systems 213a to 213c can only emit laser light having a wavelength in a predetermined range. Optionally reflect or transmit.

The coupled optical system 213g is arranged in front of the dichroic mirrors 213d to 213f. The blue laser beam transmitted through the dichroic mirror 213d and the green laser beam and the red laser beam reflected by the dichroic mirrors 213e and 213f are incident on the coupled optical system 213g. The coupled optical system 213g collects the laser beams of each of the three primary colors and causes them to enter the optical fiber 27.

In order to irradiate the laser beam incident on the optical fiber 27 as an image, the horizontal scanning unit 23 and the vertical scanning unit 24 scan the laser beam in the horizontal and vertical directions to generate a scanned image light.

The horizontal scanning unit 23 includes a resonance type deflection element 23a, a horizontal scanning control circuit 23b, and a horizontal scanning angle detection circuit 23c. The laser beam incident on the optical fiber 27 is collimated by the collimated optical system 22 and incident on the resonance type deflection element 23a. The resonance type deflection element 23a has a reflecting surface 23d that is oscillated by the horizontal scanning control circuit 23b, and the incident laser light is reflected by the oscillating reflecting surface 23d and scanned in the horizontal direction. The horizontal scanning control circuit 23b generates a drive signal that swings the reflecting surface 23d of the resonance type deflection element 23a based on the horizontal drive signal 215 output from the signal processing circuit 211. The horizontal scanning angle detection circuit 23c detects the swing state such as the swing range and swing frequency of the reflection surface 23d of the resonance type deflection element 23a based on the displacement signal output from the resonance type deflection element 23a. A signal indicating the rocking state is output to the image generator controller 49.

The vertical scanning unit 24 includes a deflection element 24a, a vertical scanning control circuit 24b, and a vertical scanning angle detection circuit 24c. The deflection element 24a has a reflecting surface 24d that is oscillated by the vertical scanning control circuit 24b, and the incident laser light is reflected by the oscillating reflecting surface 24d and scanned in the vertical direction to scan two-dimensionally. The resulting image light is emitted to the relay optical system 26. The vertical scanning control circuit 24b generates a drive signal that swings the reflection surface 24d of the deflection element 24a based on the vertical drive signal 216 output from the signal processing circuit 211. The vertical scanning angle detection circuit 24c detects the swing state such as the swing range and the swing frequency of the reflecting surface 24d of the deflection element 24a based on the displacement signal output from the deflection element 24a, and determines the swing state. The indicated signal is output to the image generator controller 49.

The relay optical system 25 is arranged between the resonance type deflection element 23a and the deflection element 24a. The relay optical system 25 converges the laser beam scanned in the horizontal direction on the reflection surface 23d of the resonance type deflection element 23a and causes the laser beam to be incident on the reflection surface 24d of the deflection element 24a.

The signal processing circuit 211 outputs the horizontal drive signal 215 and the vertical drive signal 216 to the horizontal scan control circuit 23b and the vertical scan control circuit 24b, respectively, based on the image signal input from the image generator controller 49, and outputs the reflective surface. The brightness of the generated image light is adjusted by changing the scanning angles of 23d and 24d.

The scanning angles of the reflecting surfaces 23d and 24d changed in this way are detected as detection signals by the horizontal scanning angle detection circuit 23c and the vertical scanning angle detection circuit 24c, and the detection signal is input to the image generator controller 49 to be a horizontal drive signal. It is fed back to the 215 and the vertical drive signal 216.

The relay optical system 26 has lens systems 26a and 26b having a positive refractive power. The image light emitted from the deflection element 24a is converted into convergent image light by the lens system 26a so that the center lines of the image light are substantially parallel to each other and the center lines of the image light are substantially parallel to each other. The convergent image lights are scanned image lights that are substantially parallel to each other by the lens system 26b, and the center lines of these scanned image lights are focused so as to converge on the user's pupil Ea.

In the present embodiment, the laser beam incident from the optical fiber 27 is scanned horizontally by the horizontal scanning unit 23 and then vertically scanned by the vertical scanning unit 24. The arrangement of the vertical scanning unit 24 may be changed so that the vertical scanning unit 24 scans in the vertical direction and then the horizontal scanning unit 23 scans in the horizontal direction.

Such a method of scanning a laser beam on the retina to display an image has a deep depth of focus, and even if the user moves his / her hand in the depth direction and the focal position of the user's eye moves in the depth direction. The virtual panels can be clearly seen, and a sense of depth can be felt in the display of the stacked virtual panels.

The embodiment in which the image generation unit 20 directly irradiates the user's eyeball E with an image light obtained by scanning a laser beam whose intensity is modulated according to the image in two dimensions to allow the user to visually recognize the image has been described. The image generation unit 20 is not limited to this embodiment, and may be a device such as an LCD (Liquid Crystal Display) or an organic EL arranged on the front surface of the user's eyeball, which allows the user to visually recognize an image.

(Main process of the first embodiment) FIG. 6 shows a flow chart of the main processing of the first embodiment, and the flow will be described below. When the main flow process of the first embodiment starts, the process proceeds to the process of S11 calibration. In the process of S11, the virtual panel generation program 43a executes the "calibration" process. Details will be described later with reference to FIGS. 7 and 8, but in this process, the virtual panel generation program 43a determines the coordinates (z coordinates) in the depth direction in which the virtual panels 501 are stacked and arranged. When the processing of S11 is completed, the process proceeds to the processing of S12 "Virtual panel display".

In the process of S12 "virtual panel display", the virtual panel generation program 43a issues an instruction to generate the virtual panel image 500 based on the coordinates (z coordinates) in the depth direction of each virtual panel 501 determined in the process of S11. Output to the image generator controller 49. Based on the above command, the controller 49 for the image generation unit stacks and arranges the virtual panels 501 to generate the virtual panel image 500 which is bitmap data, stores it in the VRAM 50 for the image generation unit, and also stores the virtual panel image 20 in the image generation unit 20. The image generation unit 20 generates and displays the virtual panel image 500. When the processing of S12 is completed, the process proceeds to the processing of S13 "hand position detection".

In the process of S13 hand position detection, which will be described in detail later with reference to FIGS. 9, 10 and 11, the hand coordinate calculation means 43c detects the user's hand (finger) 601 and the user's hand. (Finger) Calculate the "hand coordinates" of 601. When the processing of S13 is completed, the process proceeds to S14 "Recognize the selected virtual panel".

In the process of S14 "Recognize the selected virtual panel", the command selection means 43d is set to the depth coordinate (z coordinate) of the "hand coordinate" of the user's hand (finger) 601 calculated by the hand coordinate calculation means 43c. Based on this, the virtual panel 501 selected by the user is recognized. As shown in (B) of FIG. 3, in the command selection means 43d, the coordinates in the depth direction (z direction) of the "hand coordinates" of the user's hand (finger) 601 are defined in each virtual panel 501. Coordinates (z1, z2, z3 ... ) Is determined, and among the virtual panels 501 that have penetrated, the closest virtual panel 501 is recognized as the virtual panel 501 selected by the user. In FIG. 3B, z1', z2', and z3'represent the range determined to be the closest to the coordinates (z1, z2, z3) defined in the virtual panel 501, respectively. If the coordinates (z coordinates) in the depth direction of the "hand coordinates" of the user's hand (finger) 601 are within the range of z2', it is determined that the second virtual panel 501b is selected. Alternatively, if the depth direction coordinates (z coordinates) of the "hand coordinates" of the user's hand (finger) 601 are within the range of z3', it is determined that the third virtual panel 501c is selected. As for the virtual panel 501a in the front row, there is no virtual panel 501 that penetrates in front of it, so the user's hand (finger) is in front of the virtual panel 501a in the front row so as not to penetrate the virtual panel 501a in the front row. If there is 601, the command selection means 43d recognizes the front row virtual panel 501a as the user-selected virtual panel 501. Alternatively, as shown in FIG. 3 (C), the command selection means 43d sets the virtual panel 501 closest to the depth (z direction) coordinates of the "hand coordinates" of the user's hand (finger) 601. It may be recognized as the virtual panel 501 selected by the user. In FIG. 3C, z1'', z2'', and z3'' represent the ranges judged to be closest to the coordinates (z1, z2, z3) defined in the virtual panel 501, respectively. ing. When the processing of S14 is completed, the process proceeds to S15 "Highlight selected touch panel".

In the process of S15 "Highlight selected touch panel", the virtual panel generation program 43a blinks the number of the virtual panel 501 recognized by the command selection means 43b in the process of S14, for example, as shown in FIG. The highlighting command is output to the image generator controller 49. It is also possible to highlight the virtual panel 501 by performing image processing such as blinking the virtual panel 501 itself, changing the color of the virtual panel 501, and displaying the virtual panel 501 in a large size. By highlighting the virtual panel 501 in this way, the user can recognize the virtual panel 501 to be selected. The virtual panel 501 is displayed so that the user feels that it is stacked in the depth direction, but because of the flat image, the position of the virtual panel 501 that appears to be stacked and the user's hand (finger) 601 with respect to the head-mounted display 100 It is difficult to correspond to the position in the depth direction of. However, in the present embodiment, the virtual panel 501 at the position where the hand (finger) 601 is located is highlighted. Therefore, while looking at the highlighted virtual panel 501, the hand (finger) 601 with respect to the head-mounted display 100 is highlighted. The depth position can be adjusted by the user to select the desired virtual panel 501 without error. When the processing of S15 is completed, the process proceeds to the judgment processing of S16 "Pressing the virtual panel?".

In the determination process of S16 "virtual panel pressed?", The command selection means 43d determines whether or not the virtual panel 501 selected in the process of S14 and highlighted in the process of S15 is pressed. Specifically, the command selection means 43d determines whether or not the user's hand has stopped within the set range for a certain period of time or longer. The setting range is a range in which the virtual panel 501 is displayed in the left-right / up-down direction in the left-right / up-down direction (x, y-direction), and is a range in which the virtual panel 501 is displayed in the left-right / up-down direction. The range (z1', z2', z3' ...) specified in each virtual panel 501 shown in B). If the command selection means 43d determines that the user's hand has stopped within the set range for a certain period of time or longer, the process proceeds to S17 "Virtual panel press display". On the other hand, when the command selection means 43d determines that the user's hand has not stopped within the set range for a certain period of time or more and is out of the set range, the process returns to the process of S13.

In the process of S17 "virtual panel press display", the virtual panel generation program 43a outputs a command to display that the virtual panel 501 selected in the process of S14 is pressed to the image generation unit controller 49. The display indicating that the virtual panel 501 has been pressed includes displaying the characters "pressed", changing the color of the selected virtual panel 501, and the like. When the processing of S17 is completed, the process proceeds to the processing of S18 "command selection".

In the process of S18 "command selection", select the command corresponding to the virtual panel 501 highlighted in the process of S17. When the processing of S18 is completed, the process proceeds to the judgment processing of S19 "final selection?".

In the judgment process of S19 "final selection?", The command selection means 43d has a virtual panel 501 corresponding to the command selected in the process of S18 in the virtual panel image 500 (shown in FIG. 12) at the lowest level. Determine if it is 501. When the command selection means 43d determines that the virtual panel 501 corresponding to the command selected in the process of S18 is the virtual panel 501 in the virtual panel image 500 of the lowest layer, the process of S20 "command execution" is performed. Proceed to. On the other hand, if the command selection means 43d determines that the virtual panel 501 corresponding to the command selected in the process of S18 is not the virtual panel 501 in the virtual panel image 500 in the lowest layer, S25 "next layer" Proceed to the process of "Display virtual panel image".

In the process of S25 "displaying the virtual panel image of the next layer", the virtual panel generation program 43a outputs an instruction to generate the virtual panel image 500 of the next layer (lower layer) to the image generation unit controller 49. The controller 49 for the image generation unit generates the virtual panel image 500 and outputs the virtual panel image 500 to the image generation unit 20 according to the instruction of the virtual panel generation program 43a. When the processing of S25 is completed, the process returns to the processing of S13. If commands can be selected in such a hierarchical structure, complicated operations can be performed only by detecting the depth direction of the hand.

In the process of S20 "command execution", the command selection means 43d executes the command selected in the process of S18. When the processing of S20 is completed, the process proceeds to the processing of S21 "image generation".

In the processing of S21 "image generation", the image generation unit controller 49 generates an image based on the instruction executed by the CPU 41 in the processing of S20, outputs the image to the VRAM50 for the image generation unit, and outputs the image to the image generation unit 20. To do. The commands executed in the processing of S20 include enlargement / reduction / movement of the image generated by the image generation unit 20, processing for advancing the next image, processing for returning to the previous image, and the like. When the processing of S21 is completed, the flow of the main processing is completed.

If the command executed in the process of S20 does not change the image generated by the image generation unit 20 (for example, saving the currently displayed image), the process of S21 is not executed.

In the present embodiment, when the user stops his / her hand in the setting range of the virtual panel 501 for a certain period of time or longer in the process of S16, the command selection means 43d determines that the virtual panel 501 has been pressed. However, the processing of S16 is not limited to this, and when the user performs a predetermined operation, for example, when the user performs an operation such as folding the hand (finger) 601 or waving the hand (finger) 601. It may be determined that the command selection means 43d has pressed the virtual panel 501 based on the "hand coordinates" of the user's hand (finger) 601 calculated by the calculation means 43c. If the movement of the user's hand (finger) 601 is set as a selection movement that does not occur with the movement of the head, erroneous detection will not occur with the movement of the head, and a highly reliable pressing judgment can be made. It can be carried out. For example, the selection operation includes an operation of folding the hand (finger) 601 or shaking the hand (finger) 601 faster than the speed of shaking the head.

(Calibration process) FIG. 7 shows a flow diagram of the calibration process, FIG. 8 shows an explanatory diagram of the calibration process, and the calibration process will be described below. Note that FIG. 8 is a diagram showing the user's field of view. In the calibration process, the image generator 20 displays the marker image 505 at a position where the user feels close (near position) ((A) in FIG. 8), and the image is moved to a position where the user feels far (far position). The generation unit 20 displays the marker image 506 ((B) in FIG. 8), causes the user to select the marker images 505 and 506, and arranges the virtual panel 501 between the marker images 505 and 506. This is the process of calculating the coordinates. In this way, by performing the calibration process, the individual difference in the perspective of the image generated by the image generation unit 20 for each user is absorbed, and the user does not feel a sense of discomfort.

When the calibration process starts, in the process of S31 "Marker image display for near position adjustment", as shown in (A) of FIG. 8, the virtual panel generation program 43a displays the marker image 505 that the user can see in the near position. The generated command is output to the image generator controller 49. As shown in (A) of FIG. 8, the image generation unit 20 generates and displays a large marker image 505 on the lower side. When the processing of S31 is completed, the process proceeds to the processing of S32 "hand position detection".

In the process of S32 hand position detection, the hand position detection program 43b executes the hand position detection process. Details will be described later with reference to FIG. 9, but in this process, the hand position detection program 43b detects the positions of the user and the hand (finger) 601 and the hand coordinate calculation means 43c is based on the head mount display 100. Calculate the "hand coordinates", which are the three-dimensional coordinates of the position of the user's hand (finger) 601. When the processing of S32 is completed, the process proceeds to the judgment processing of S33 "Is there a hand in the set position?".

In the determination process of S33 "Is there a hand at the set position?", The virtual panel generation program 43a determines whether or not there is a user's hand at the set position. The set position is a position within the range of the x-coordinate and the y-coordinate where the marker image 505 is displayed. When the virtual panel generation program 43a determines that the user's hand (finger) 601 is within the range in which the marker image 505 is displayed, the determination process of S34 "is there a set time hand at the set position?" Proceed to. On the other hand, when the virtual panel generation program 43a determines that the user's hand (finger) 601 is not within the range in which the marker image 505 is displayed, the process returns to the process of S32.

In the determination process of S34 "Is there a set time hand at the set position?", The virtual panel generation program 43a determines whether or not the set time user's hand (finger) 601 is present at the set position. When the virtual panel generation program 43a determines that the user's hand (finger) 601 for the set time is at the set position, the process proceeds to S35 Near position setting (Z1). On the other hand, when the virtual panel generation program 43a determines that the user's hand (finger) 601 does not exist at the set position for the set time, the process returns to S32.

In the process of S35 "Near position setting (Z1)", the virtual panel generation program 43a is the coordinate (Z coordinate) in the depth direction of the "hand coordinates" of the user's hand (finger) 601 detected in the process of S32. Is set as the near position (Z1). When the processing of S35 is completed, the process proceeds to the processing of S36 "marker image display for distance position adjustment".

In the process of S36 "marker image display for distant position adjustment", as shown in (B) of FIG. 8, the virtual panel generation program 43a issues an image generation command to generate a marker image 506 that the user can see at a distant position. Output to the unit controller 49. As shown in (B) of FIG. 8, the image generation unit 20 generates a small marker image 506 on the upper side. When the processing of S36 is completed, the process proceeds to the processing of S37 Hand position detection.

In the process of S37 hand position detection, the hand position detection program 43b executes the hand position detection process. The processing of S37 is the same as the processing of S32. When the processing of S37 is completed, the process proceeds to the processing of S38 "Is there a hand at the set position?".

In the determination process of S38 "Is there a hand at the set position?", The virtual panel generation program 43a determines whether or not there is a user's hand at the set position. When the virtual panel generation program 43a determines that the user's hand (finger) 601 is within the range in which the marker image 506 is displayed, the determination process of S39 "is there a set time hand at the set position?" Proceed to. On the other hand, when the virtual panel generation program 43a determines that the user's hand (finger) 601 is not within the range in which the marker image 506 is displayed, the process returns to the process of S37.

In the determination process of S39 "Is there a set time hand at the set position?", The virtual panel generation program 43a determines whether or not the set time user's hand (finger) 601 is present at the set position. When the virtual panel generation program 43a determines that the user's hand (finger) 601 for the set time is at the set position, the process proceeds to S40 "far position setting (Zn)". On the other hand, when the virtual panel generation program 43a determines that the user's hand (finger) 601 does not exist at the set position for the set time, the process returns to the process of S37.

In the process of S40 "far position setting (Zn)", the virtual panel generation program 43a is the coordinate (Z coordinate) in the depth direction of the "hand coordinates" of the user's hand (finger) 601 detected in the process of S39. Is set as the far position (Zn). When the processing of S40 is completed, the process proceeds to the processing of S41 "determining the coordinates of n virtual panels".

In the process of S41 "determining the coordinates of n virtual panels", the virtual panel generation program 43a uses the coordinates (Z1) in the depth direction set in the process of S35 as the coordinates in the depth direction of the virtual panel 501 displayed in the foreground. Determined as (Z1), the coordinates in the depth direction (Zn) set in the processing of S40 are determined as the coordinates in the depth direction (Zn) of the virtual panel 501 to be displayed farthest, and are displayed the most troublesome and farthest. Calculated by dividing the difference between Zn and Z1 by (n-1) and adding the value to Z1 for the coordinates in each depth direction of the virtual panel 501 in which n-2 sheets are stacked and arranged between the virtual panels 501. The coordinates (Z coordinates: Z1, Z2 ... Zn) in the depth direction of each virtual panel 501 are determined. When the processing of S41 is completed, the calibration processing is completed, and the process proceeds to the processing of S12 in the main flow of FIG.

By such "calibration processing", the correspondence between the position that the user wants to recognize as the position in the depth direction of the marker images 505 and 506 and the actual position in the depth direction of the user's hand (finger) 601 with respect to the head-mounted display 100. Can be decided by the user himself. For this reason, even if the virtual panel 501 is displayed on a flat surface so that the user feels that it is stacked in the depth direction, it absorbs individual differences for each user and is virtual in the operating range of the hand that is easy for the user to operate. It is possible to generate a panel 501.

(Hand position detection process) FIG. 9 shows a flow diagram of the hand position detection process, and the flow will be described below. When the hand position detection process starts, the process proceeds to S51 Right image pickup device imaging. In the processing of S51, the right image pickup device 31 performs the image pickup process, and sequentially outputs the output signal of the image sensor of the right image pickup device 31 to the right image pickup device controller 45. The right image pickup device controller 45 is based on the signals of the R (red) G (green) B (blue) primary colors of each pixel of the image sensor of the right image pickup device 31, and each primary color (R, G,) for each pixel. B) is expressed in gradation, an "image" is generated, and stored in VRAM46 for the right image pickup device. When the processing of S51 is completed, the process proceeds to the processing of S52 "color analysis".

In the process of S52 "color analysis", the hand position detection program 43b executes the hand detection process. The details will be described later with reference to FIG. 10, but in this process, the hand position detection program 43b performs a color analysis process of the captured image captured by the right image pickup device 31, and the color components of the hand (finger). Extract the pixels. When the processing of S52 is completed, the process proceeds to the judgment processing of S53 "Detect hand?".

In the judgment process of S53 "Hand detected?", The hand position detection program 43b collates the set of pixels extracted by the process of S52 (S62) with the hand matching data 43e stored in ROM43, and S52 (S62). ) Is determined whether or not the set of pixels extracted by the process is a hand (finger). In the processing of S53, the hand position detection program 43b uses a predetermined threshold value to determine whether or not the set of pixels extracted in the processing of S52 (S62) is a hand (finger). When the hand position detection program 43b determines that the set of pixels extracted by the process of S52 (S62) is the hand (finger), the process proceeds to the process of S54 "calculation of imaging coordinates of the hand". On the other hand, if the hand position detection program 43b does not determine that the set of pixels extracted by the process of S52 (S62) is the hand (finger), the process returns to the process of S52.

In the processing of S54 "calculation of imaging coordinates of the hand", the coordinates in the vertical (y) and horizontal (x) directions (hereinafter referred to as imaging coordinates) on the captured image of the hand (finger) detected in the judgment processing of S53 are used. calculate. In this embodiment, the imaging coordinates of the fingertip are calculated. That is, the shooting coordinates at which the coordinates in the vertical (y) direction on the captured image of the detected hand (finger) 601 are maximized are the coordinates of the fingertip. When the processing of S54 is completed, the process proceeds to the processing of S55 Left imaging device imaging.

In the processing of S55 "left imaging device imaging", the left imaging device 32 performs imaging processing in the same manner as the processing of S51, and the "imaging image" captured by the left imaging device 32 is transferred to the VRAM 48 for the left imaging device. It will be saved. When the processing of S55 is completed, the process proceeds to the processing of S56 "color analysis".

In the process of S56 "color analysis", the hand position detection program 43b executes the hand detection process. The processing of S56 is the same as the processing of S52. When the processing of S56 is completed, the process proceeds to the judgment processing of S57 "Detect hand?".

In the judgment process of S57 "Hand detected?", The hand position detection program 43b collates the set of pixels extracted by the process of S56 (S62) with the hand matching data 43e stored in ROM43, and S56 (S62). ) Is determined whether or not the set of pixels extracted by the process is a hand (finger). When the hand position detection program 43b determines that the set of pixels extracted by the process of S56 (S62) is the hand (finger), the process proceeds to the process of S58 "calculation of imaging coordinates of the hand". On the other hand, if the hand position detection program 43b does not determine that the set of pixels extracted in the process of S56 (S62) is the hand (finger), the process returns to the process of S52.

In the process of S58 "calculation of imaging coordinates of the hand", the imaging coordinates of the hand (finger) detected in the judgment process of S57 are calculated. When the processing of S58 is completed, the process proceeds to the processing of S59 "calculation of hand coordinates".

In the process of S59 "hand coordinate calculation", the hand coordinate calculation means 43c is a three-dimensional coordinate with the head-mounted display 100 as the reference origin based on each "imaging coordinate" calculated in the process of S54 and the process of S58. Calculate "hand coordinates". As shown in FIG. 11, for example, when the user's hand (finger) 601 is located at the center of the left and right image pickup devices 31 and 32, the user's hand (finger) 601 imaged by the right image pickup device 31 The imaging coordinates are shifted to the left in the lateral direction (x direction) from the center of the imaging range of the right imaging device 31. Further, the imaging coordinates of the user's hand (finger) 601 imaged by the left imaging device 32 shift to the right side in the lateral direction (x direction) from the center of the imaging range of the left imaging device 31. From the amount of deviation (parallax) of the imaging coordinates, the coordinates (z coordinates) in the depth direction of the user's hand (finger) 601 based on the head-mounted display 100 are calculated. Even if the user's hand (finger) 601 is not located at the center of the left and right imaging devices 31 and 32, the "imaging coordinates" of the user's hand (finger) 601 imaged by the left and right imaging devices 31 and 32. Is different in the lateral direction (x direction), so the coordinates in the depth direction (z direction) are calculated from the "imaging coordinates" of the user's hand (finger) 601 imaged by the left and right imaging devices 31 and 32. In this way, the hand coordinate calculation means 43c is the deviation amount of the "imaging coordinates" of the user's hand (finger) 601 at the same position, which is imaged by the left and right imaging devices 31 and 32 arranged at regular intervals. Based on the (parallax), the coordinates (z coordinates) in the depth direction of the user's finger 601 with respect to the head-mounted display 100 are calculated.

In this way, by imaging the user's finger 601 with the image pickup devices 31 and 32 attached to the head-mounted portion 10 at intervals, the coordinates in the depth direction of the user's finger 601 with respect to the head-mounted display 100 ( Since we decided to calculate the z-coordinate), even if the user's head shakes, the imaging devices 31 and 32 also move as the user's head moves, and the position of the user's finger 601 changes with respect to the captured images 31 and 32. Even if the imaging devices 31 and 32 capture images, the difference between the user's finger 601 and the user's finger 601 does not change, so that the depth coordinate (z coordinate) of the user's finger 601 with respect to the head-mounted display 100 can be stably detected. ..

Further, the hand coordinate calculation means 43c averages the lateral coordinates (x coordinates) of the "imaging coordinates" of the user's hand (finger) 601 imaged by the right imaging device 31 and the left imaging device 32, and the user's hand. (Finger) Calculate the horizontal coordinates (x coordinates) of 601. Further, the hand coordinate calculation means 43c averages or either of the vertical coordinates (y coordinates) of the "imaging coordinates" of the user's hand (finger) 601 imaged by the right imaging device 31 and the left imaging device 32. Select one and calculate the vertical coordinates (y coordinates) of the user's hand (finger) 601. In this way, the hand coordinate calculation means 43c calculates "hand coordinates" which are three-dimensional coordinates of the position of the user's hand (finger) 601 with respect to the head-mounted display 100. When the processing of S59 is completed, the "hand position detection processing" is completed, and the process returns to the "calibration processing" shown in FIG. At this time, the user's head is constantly moving, and the imaging devices 31 and 32 are also moved accordingly. Normally, there are many movements in which the head rotates around the neck, so the x-coordinate and y-coordinate change according to the movements of the head and the imaging devices 31 and 32 due to the movements of the imaging devices 31 and 32, but in the depth direction. The z coordinate, which is the coordinate of, hardly changes.

(Color analysis processing) A flow chart of the color analysis process is shown in FIG. 10, and the process will be described below. When the color analysis process starts, in the process of S61 "RGB decomposition", the hand position detection program 43b determines the strength of the gradation of each primary color of RGB for each pixel of the "image captured image" stored in the VRAM46 for the right image pickup device. Performs digitization processing. When the processing of S61 is completed, the process proceeds to the processing of S63 "setting color area extraction".

In the processing of S63 "set color area extraction", the hand position detection program 43b is the range of the hand setting color (skin color) for each pixel in which the intensity of the gradation of each primary color of RGB is quantified in the processing of S61. It is determined whether or not it is inside, and the pixels of the set color of the hand are extracted. When the processing of S63 is completed, the color analysis processing is completed.

(Head-mounted display of the second embodiment) FIG. 13 shows a block diagram of the head-mounted display 110 of the second embodiment, and FIG. 15 shows an overall view of the head-mounted display 110 of the second embodiment. The head-mounted display 110 of the second embodiment will be described. The second embodiment is an embodiment in which the pressing of the virtual panel 501 is determined by the movement of the user's hand detected by the sensor 70 attached to the user's hand.

The head-mounted display 110 of the second embodiment, in addition to the head-mounted display 100 of the first embodiment, further includes a sensor 70 attached to the user's hand and a "hand operation signal" detected by the sensor 70. It has a sensor interface 52 that performs physical and logical conversion and delivers it to the bus 55. The "manual operation signal" delivered to the bus 55 is stored in the RAM 42.

The sensor 70 includes, for example, an accelerometer that detects the acceleration of the movement of the user's hand (finger) 601 and a potentiometer or strain gauge that detects the movement of the user's hand (for example, bending the finger 601). It is a sensor that detects the movement of the hand and generates a "hand movement signal". The sensor 70 of the embodiment shown in FIG. 15B is a sensor that detects the bending and stretching of the user's finger 601.

FIG. 14 shows a flow chart of the main processing of the head-mounted display 110 of the second embodiment, and the flow will be described below. When the main flow process of the second embodiment starts, the process proceeds to the process of S111 calibration. In the process of S111, the virtual panel generation program 43a executes the "calibration" process. In this process, the virtual panel generation program 43a determines the coordinates (z coordinates) in the depth direction of each virtual panel 501. The process of S111 is the same as the calibration process (shown in FIG. 7) of the first embodiment described above. When the processing of S111 is completed, the process proceeds to the processing of S112 "Virtual panel display".

In the process of S112 "virtual panel display", the virtual panel generation program 43a issues an instruction to generate the virtual panel image 500 based on the coordinates (z coordinates) in the depth direction of each virtual panel 501 determined in the process of S111. Output to the image generator controller 49. Based on the above command, the image generation unit controller 49 stacks and arranges virtual panels 501 to generate a virtual panel image 500, stores the virtual panel image 500 in the image generation unit VRAM 50, and outputs the image to the image generation unit 20. The generation unit 20 generates and displays the virtual panel image 500. When the processing of S112 is completed, the process proceeds to the processing of S113 Hand position detection.

In the process of S113 "hand position detection", the hand coordinate calculation means 43c detects the user's hand (finger) 601 and calculates the "hand coordinate" of the user's hand (finger) 601. The process of S113 is the same process as the hand position detection process (shown in FIG. 9) of the first embodiment described above. When the processing of S113 is completed, the process proceeds to S114 "Recognize selected virtual panel".

In the process of S114 "Recognize the selected virtual panel", the command selection means 43d is set to the depth coordinate (z coordinate) of the "hand coordinate" of the user's hand (finger) 601 calculated by the hand coordinate calculation means 43c. Based on this, the virtual panel 501 selected by the user is recognized. As shown in (B) of FIG. 3, in the command selection means 43d, the coordinates in the depth direction (z direction) of the "hand coordinates" of the user's hand (finger) 601 are defined in each virtual panel 501. It determines whether or not it is within the range (z1, z2, z3 ...) and recognizes the virtual panel 501 selected by the user. When the processing of S114 is completed, the process proceeds to S115 "Highlight selected touch panel".

In the process of S115 "highlight selected touch panel", the virtual panel generation program 43a blinks the number of the virtual panel 501 recognized by the command selection means 43d in the process of S114, for example, as shown in FIG. Outputs the highlighting command to the image generator controller 49. It is also possible to highlight the virtual panel 501 by performing image processing such as blinking the virtual panel 501 itself, changing the color of the virtual panel 501, and displaying the virtual panel 501 in a large size. By highlighting the virtual panel 501 in this way, the user can recognize the virtual panel 501 to be selected. The virtual panel 501 is displayed so that the user feels that it is stacked in the depth direction, but because of the flat image, the position of the virtual panel 501 that appears to be stacked and the user's hand (finger) 601 with respect to the head-mounted display 100. It is difficult to correspond to the position in the depth direction of. However, in the present embodiment, since the virtual panel 501 at the position where the hand (finger) 601 is located is highlighted, the hand (finger) 601 with respect to the head-mounted display 100 is highlighted while looking at the highlighted virtual panel 501. The depth position can be adjusted by the user to select the desired virtual panel 501 without error. When the processing of S115 is completed, the process proceeds to the judgment processing of S116 "Manual operation information input?".

In the determination process of S116 manual operation information input?, The command selection means 43d determines whether or not the hand operation signal has been input to the sensor interface 52. When the command selection means 43d determines that the "hand motion signal" has been input to the sensor interface 52, the process proceeds to the determination process of S118 "is the hand motion signal the pressing level?". On the other hand, when the command selection means 43d determines that the "hand operation signal" has not been input to the sensor interface 52, the process returns to the process of S113.

In the determination process of S118 "Is the hand motion signal the pressing level?", The command selection means 43d determines the strength of the "hand motion signal" input in the determination process of S116, and the "hand motion signal" is the pressing level. Judge whether or not. When the command selection means 43d determines that the "manual operation signal" input in the determination process of S116 is the pressing level, the process proceeds to the process of S119 "virtual panel pressing display". On the other hand, when the command selection means 43d determines that the "hand operation signal" input in the determination process of S116 is not the pressing level, the process returns to the process of S113. Even if the user's head moves and the image pickup devices 31 and 32 capture the image as if the user's hand (finger) 601 was shaken, in the present embodiment, the user's hand (finger) 601 is attached. Since it is decided that the sensor 70 detects the selection operation of the virtual panel 501, it is surely prevented that the virtual panel 501 is erroneously selected due to the movement of the user's head, and the virtual panel 501 desired to be selected by the user is surely prevented. Can be reliably detected.

In the process of S119 "virtual panel press display", the virtual panel generation program 43a has the coordinates in the depth direction (z direction) of the "hand coordinates" of the user's hand (finger) 601 calculated in the process of S117, respectively. Judges whether or not it is within the range (z1', z2', z3' ...) specified in the virtual panel 501 of the user, recognizes the virtual panel 501 selected by the user, and presses the virtual panel 501. Outputs a command to display the fact to the image generator controller 49. The display indicating that the virtual panel 501 has been pressed includes displaying the characters "pressed", changing the color of the selected virtual panel 501, and the like. When the processing of S119 is completed, the process proceeds to the processing of S120 "command selection".

In the S120 "command selection" process, select the command corresponding to the virtual panel 501 highlighted in the S119 process. When the processing of S120 is completed, the process proceeds to the judgment processing of S121 "final selection?".

In the judgment process of S121 "final selection?", In the command selection means 43d, the virtual panel 501 corresponding to the command selected in the process of S120 is located in the lowermost virtual panel image 500 (shown in FIG. 12). Judge whether or not. When the command selection means 43d determines that the virtual panel 501 corresponding to the command selected in the processing of S120 is the virtual panel 501 in the virtual panel image 500 at the bottom layer, the processing of S122 "command execution" is performed. move on. On the other hand, if the command selection means 43d determines that the virtual panel 501 corresponding to the command selected in the processing of S120 is not the virtual panel 501 in the virtual panel image 500 of the lowest layer, S125 "virtual of the next layer". Proceed to the process of "Display panel image".

In the process of S125 displaying the virtual panel image of the next layer, the virtual panel generation program 43a outputs an instruction to generate the virtual panel image 500 of the next layer (lower layer) to the image generation unit controller 49. The controller 49 for the image generation unit generates the virtual panel image 500 and outputs the virtual panel image 500 to the image generation unit 20 according to the instruction of the virtual panel generation program 43a. When the processing of S125 is completed, the process returns to the processing of S113.

In the process of S122 "command execution", the command selection means 43d executes the command selected in the process of S120. When the processing of S122 is completed, the process proceeds to the processing of S123 "image generation".

In the processing of S123 "image generation", the image generation unit controller 49 generates an image based on the instruction executed by the CPU 41 in the processing of S122, outputs the image to the VRAM 50 for the image generation unit, and outputs the image to the image generation unit 20. To do. The command executed in the processing of S122 includes enlargement / reduction / movement of the image generated by the image generation unit 20, processing for advancing to the next image, processing for returning to the previous image, and the like. When the processing of S123 is completed, the main processing flow ends.

If the command executed in the process of S122 does not change the image generated by the image generation unit 20 (for example, saving the currently displayed image), the process of S123 is not executed.

In addition, in this specification, "hand" is a substitute for a hand, for example, a writing instrument such as a pencil or a pen held in the hand, a glove worn on the hand, a marker or a light spot provided on a remote controller or the like. It is assumed that the ones that can be operated are also included. Further, in the embodiment described above, the two imaging devices 31 and 32 are attached to the front portion of the head mounting portion 10 at predetermined intervals, but three or more imaging devices are attached to the head mounting portion 10. It may be attached to the front portion at a predetermined interval, and the position and movement of the user's hand (finger) 601 in the depth direction may be detected by the three or more imaging devices.

As described above, the present invention has been described in relation to the most practical and preferable embodiments at present, but the present invention is not limited to the embodiments disclosed in the present specification. It is possible to make appropriate changes within the scope of the claims and within the scope not contrary to the gist or idea of the invention that can be read from the entire specification, and the head-mounted display with such changes must also be understood as being included in the technical scope. Must be.

<figref num="1">It is an overall view of the head-mounted display of this invention. (First Embodiment)</figref><figref num="2">It is the figure which showed the user's vision.</figref><figref num="3">It is a virtual panel selection explanatory diagram.</figref><figref num="4">It is a block diagram of this invention. (First Embodiment)</figref><figref num="5">It is a block diagram of an image generation part.</figref><figref num="6">It is a flow chart of the main process of this invention. (First Embodiment)</figref><figref num="7">It is a flow chart of the calibration process. (First Embodiment)</figref><figref num="8">It is explanatory drawing of the calibration process.</figref><figref num="9">It is a flow chart of a hand position detection process. (First Embodiment)</figref><figref num="10">It is a flow chart of a color analysis process. (First Embodiment)</figref><figref num="11">It is explanatory drawing which calculates the hand coordinate from the imaging coordinate.</figref><figref num="12">It is explanatory drawing of the tree structure of a virtual panel.</figref><figref num="13">It is a block diagram of the head-mounted display of the second embodiment.</figref><figref num="14">It is a flow chart of the main process of 2nd Embodiment.</figref><figref num="15">It is an overall view of the head-mounted display of the second embodiment.</figref>

Code description

10 Head mounting part 20 Image generator 21 Image light generator 22 Collimated optics 23 Horizontal scanning unit 23a Resonant deflection element 23b Horizontal scanning control circuit 23c Horizontal scanning angle detection circuit 23d Reflective surface 24 Vertical scanning section 24a deflection element 24b vertical scan control circuit 24c vertical scan detection circuit 24d reflective surface 25 Relay optics 26 Relay optics 26a Lens system 26b lens system 27 optical fiber 31 Right imager 32 Left imager 40 Control unit 41 CPU 42 RAM 43 ROM 43a Virtual panel generator 43b Hand position detection program 43c Hand coordinate calculation means 43d Command selection method 43e Hand matching data 44 Non-volatile memory 45 Right imager controller 46 VRAM for right imager 47 Left imager controller 48 VRAM for left imager 49 Image generator controller 50 VRAM for image generator 51 interface 52 Sensor interface 55 bus 60 Operation switch 61 Power lamp 70 sensor 100 head-mounted display (first embodiment) 110 Head-mounted display (second embodiment) 211 Signal processing circuit 212 Light source 212a B laser driver 212b G laser driver 212c R laser driver 212d B laser 212e G laser 212f R laser 213 Photosynthesis 213a ~ 213c Collimated optics 213d ~ 213f Dynamic mirror 213g Coupled optics 214a B (blue) image signal 214b G (green) image signal 214c R (red) image signal 215 Horizontal drive signal 216 Vertical drive signal 500 virtual panel image 501 virtual panel 505 Marker image (near position) 506 Marker image (far position) 601 User's hand (finger) E eyeball Ea pupil