Three-dimensional image pickup system

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Industrial application field]

The present invention relates to a three-dimensional image capturing apparatus that captures a plurality of parallax images from different directions of an object for three-dimensional image reproduction / recording.

【0002】

[Conventional technology]

Conventional 3D image capturing devices include (1) a lenticula method in which images for the left and right eyes are combined and displayed on the display and a stereoscopic image is reproduced through a lenticular lens, and (2) for the left and right eyes on the display. Polarized glasses method in which images are displayed alternately or simultaneously and the observer wears polarized glasses to observe a stereoscopic image, (3) Shutter glasses method, (4) A lens and a small display are placed in front of the observer. A 3D image that captures multiple disparity images in different directions using multiple video cameras that support a head-mounted display (HMD) 3D display that displays images for the left eye and right eye, respectively. There is an imaging device.

【0003】

Next, the operation of the conventional three-dimensional image capturing apparatus will be described with reference to FIG.

【0004】

A conventional three-dimensional image capturing device is usually configured by combining a plurality of video cameras. Figure 8 shows an example using two cameras. The video camera 102 includes an image sensor 100 and a lens 101. The video camera 105 includes an image sensor 103 and a lens 104. The plane including the optical axes 106 and 107 of the lenses 101 and 104 is taken as the xz plane, and the origin is taken as the center of the lens 101.

【0005】

The center of the lens 104 is on the x-axis, and its coordinates are (x, z) = (a, 0).

【0006】

Normally, the optical axes 106 and 107 of each video camera are set to face slightly inward. The intersection P at that time is set at an appropriate distance in the Z-axis direction. Also, this intersection is often fixed. In Fig. 8, the intersection P is assumed to be (x, z) = (a / 2, z).₀). At this time, the focus of the video cameras 102 and 105 is adjusted to the point P.

【0007】

Now, disk-shaped object objects A, B, and C are placed at different distances in the Z-axis direction. Temporarily, the (x, z) coordinates of the centers of the target objects A, B, and C are set to (a / 2, z).₀), (A / 2, z₁), (A / 2, z₂), Passing through the midpoint of the center of each of the lenses 101 and 104, and lining up on a straight line parallel to the z-axis (z).₁> z₂)。

【0008】

FIG. 9A shows an image captured by the image sensor 100. FIG. 9B shows an image captured by the image sensor 103. The centers O and O'of the captured images in FIGS. 9 (a) and 9 (b) correspond to the imaging points at the point P in FIG. There is no binocular parallax in the captured image of the target object A captured by the video cameras 102 and 105. That is, no parallax occurs between the eyes of an object that includes the intersection P of the optical axes of the video cameras 102 and 105 and is in a plane perpendicular to the plane including the two optical axes. Objects located in front of and behind this plane are imaged with binocular parallax by video cameras 102 and 105. The farther away from this plane, the greater the amount of binocular parallax.

【0009】

In Figure 9, | z₀-z₁| <| Z₀-z₂Depending on the condition of |, the parallax amount (the amount of deviation from the center O or O'of the image pickup surface) of the captured image corresponding to the target objects B and C is | b | <| c | and | b'| <| c'. | Also, the signs of b and b'or c and c'are reversed.

【0010】

Next, an example of the conventional display will be described by a direct-view lenticular method.

【0011】

Images captured by two video cameras (Figs. 9 (a) and 9 (b)) are chopped into thin strips and woven alternately to create an image as shown in Fig. 10. At this time, the centers O and O'of each captured image are overlapped at substantially the same point to be created. Also, the horizontal resolution of each captured image is halved.

【0012】

This image is displayed on a display such as a liquid crystal panel. A lenticular lens is placed on the front of this display. One cylindrical lens that makes up a wrench character lens corresponds to a set of two strips made from two images.

【0013】

In the 3D display with the above settings, the space including the point P and based on the relative position is reproduced with reference to the plane perpendicular to the xz plane. The plane including the point P and perpendicular to the xz plane is reproduced on the display plane. That is, the target object A is on the display surface from the display surface (z).₀-z₁The target object B is in front of the distance () from the same plane (z).₀-z₂The target object C is reproduced before the distance of).

【0014】

The display on which the image of FIG. 10 is displayed is placed in a certain space, and the observer looks at the display at a certain distance. The distance between the video camera and the optical axis of the camera does not always match the distance between the display and the observer.

【0015】

[Problems to be Solved by the Invention]

However, the conventional three-dimensional image capturing apparatus described above cannot capture a three-dimensional image as the photographer intends. Further, the conventional three-dimensional image capturing apparatus can capture only a specific limited space set in advance. Furthermore, even if the setting conditions of the 3D image capturing device are changed so that the degree of freedom is slightly increased, an accurate 3D image cannot be reproduced unless the information is given to the display side and the displayed image is controlled. ..

【0016】

The optical axes of a plurality of video cameras constituting a conventional three-dimensional image capturing device are set and fixed so as to intersect at a certain intersection in advance. The space captured by the conventional 3D image capturing device is limited to the space centered on this intersection. That is, unless the optical axis of the video camera is changed, the space that can be photographed is limited. The space can be moved by rotating the entire conventional three-dimensional image capturing apparatus in the horizontal direction, but it cannot be easily moved in the depth direction.

【0017】

Further, in the conventional stereoscopic display device, since the image frames taken by the conventional three-dimensional image capturing device are superimposed and displayed, the plane including the intersection of the optical axes and perpendicular to each optical axis is displayed on the stereoscopic display. It is reproduced on the surface, but the distance at which the observer sees the display does not change. Therefore, if the internal convergence angle of the optical axis of the video camera can be changed and the intersection of the optical axes is changed, the reproduced stereoscopic image appears to move back and forth. That is, it is very uncomfortable to feel that the target object approaches or moves away just by moving the viewpoint (intersection of the optical axis) of the camera.

【0018】

Conventional 3D image capturing devices include a plurality of video cameras, but it is not easy to adjust the focus, convergence angle, zoom-up / zoom-back of each camera.

【0019】

The conventional 3D image capturing device is equipped with one or two viewfinders to check the captured image. In the case of one unit, check with a two-dimensional image instead of a three-dimensional image. You can see what is reflected, but you cannot recognize what kind of stereoscopic image is being captured. In the case of two units, a stereoscopic image can be seen, but in that case, it is a stereoscopic image of the head mounted display system. The appearance of the reproduced stereoscopic image differs depending on the reproduction method of the stereoscopic image. If it is an imaging device for lenticular display, confirmation with a finder is not optimal. In addition, the eyepiece must be in contact with the viewfinder, and there is a limit to the posture of holding the 3D image capturing system device.

【0020】

The two-dimensional image capturing device can handle the image with a considerable degree of freedom, and can capture an image almost as intended by the photographer. Compared to that, the degree of freedom in handling the conventional 3D image capturing device is extremely poor, and it is far from the intended shooting. Also, the operability is very poor.

【0021】

An object of the present invention is that, in view of the above-mentioned problems in the conventional three-dimensional imaging apparatus, three-dimensional images can be captured while being excellent in operability, reproducing accurate depth information, and constantly confirming a reproduced stereoscopic image. The purpose is to provide an imaging system.

【0022】

[Means for solving problems]

An object of the present invention is from a photographing means that outputs an image signal corresponding to a photographed subject, a variable means that is connected to the photographing means and changes the convergence angle of the photographing means, and a photographing means that is connected to the photographing means. A processing means that converts the output image signal by a specific method and moves the frame of the image signal in a predetermined direction according to the convergence angle of the photographing means, and an image that is connected to the processing means and processed by the processing means. It is achieved by a three-dimensional imaging system including a display means for reproducing a stereoscopic image based on a signal and a control means connected to a processing means for controlling the amount of movement of a frame.

【0023】

Further, an object of the present invention is a photographing means that outputs an image signal corresponding to a photographed subject, a variable means that is connected to the photographing means and changes the convergence angle of the photographing means, and a photographing means that is connected to the photographing means. It is connected to a display means for reproducing a stereoscopic image based on an image signal output from the means, a viewpoint detection means for the photographer to detect the position of the photographer's gazing point on the display means, and a viewpoint detection means. Focus adjustment means that detects the focus state on the display means based on the detected position and outputs the state signal, and focus adjustment that is connected to the focus detection means and adjusts the focus state of the photographing means based on the state signal. It is also achieved by a three-dimensional imaging system including means and a convergence angle control means connected to the viewpoint detecting means and adjusting the convergence angle of the photographing means based on the detected position.

【0024】

Further, an object of the present invention is a photographing means that outputs an image signal corresponding to a photographed subject, a display means that reproduces a three-dimensional image based on the image signal output from the photographing means, a display means, and a photographer. Congestion that is connected to the photographer position detecting means that detects the distance and outputs the distance signal, and adjusts and changes the convergence angle of the photographing means based on the distance signal that is connected to the photographer position detecting means and is output from the photographer position detecting means. The angle adjusting variable means, the interval adjusting variable means connected to the photographer position detecting means, and the interval adjusting variable means for adjusting and changing the interval of the photographing means based on the distance signal output from the photographer position detecting means, and the photographer position detecting means. It is also achieved by a three-dimensional image capturing system provided with a magnification factor adjusting means that is connected and adjusts the image magnifying power of the photographing means based on a distance signal output from the photographer position detecting means.

【0025】

[Action]

In the three-dimensional image capturing system of the present invention, the photographing means outputs an image signal corresponding to the photographed subject, the variable means is connected to the photographing means to change the convergence angle of the photographing means, and the processing means becomes the photographing means. The image signal that is connected and output from the photographing means is converted by a specific method, the frame of the image signal is moved in a predetermined direction according to the convergence angle of the photographing means, and the control means is connected to the processing means. A display means that reproduces a stereoscopic image based on an image signal processed by the processing means, and a display means that is connected to the processing means and controls the amount of movement of the frame.

【0026】

Further, in the three-dimensional image capturing system of the present invention, the photographing means outputs an image signal corresponding to the photographed subject, the variable means is connected to the photographing means to change the convergence angle of the photographing means, and the display means photographs. The stereoscopic image is reproduced based on the image signal connected to the means and output from the photographing means, the photographer detects the position of the photographer's gazing point on the display means in the viewpoint detecting means, and the focus detecting means is the viewpoint. The focusing state on the display means based on the detected position is detected and the state signal is output, and the focus adjusting means is connected to the focus detecting means and the photographing means is based on the state signal. The focus state of the image is adjusted, and the convergence angle control means is connected to the viewpoint detecting means, and the convergence angle of the photographing means based on the detected position is adjusted.

【0027】

Further, in the three-dimensional image capturing system of the present invention, the photographing means outputs an image signal corresponding to the photographed subject, and the display means reproduces a stereoscopic image based on the image signal output from the photographing means, and the photographer's position. The detecting means detects the distance between the display means and the photographer and outputs a distance signal, and the convergence angle adjusting variable means is connected to the photographer position detecting means and is based on the distance signal output from the photographer position detecting means. The convergence angle of the photographing means is adjusted and variable, and the interval adjusting variable means is connected to the photographer position detecting means, and the interval of the photographing means is adjusted and changed based on the distance signal output from the photographer position detecting means, and the magnification factor is increased. The adjusting means is connected to the photographer position detecting means and adjusts the image magnification of the photographing means based on the distance signal output from the photographer position detecting means.

【0028】

[Example]

Hereinafter, examples of the three-dimensional imaging system of the present invention will be described in detail with reference to the drawings.

【0029】

FIG. 1 is a block diagram showing a configuration of an embodiment of the three-dimensional imaging system of the present invention.

【0030】

The three-dimensional imaging system of FIG. 1 includes video cameras 1 and 2, which are imaging means for capturing a difference image, a stereoscopic display 6 which is a display means for displaying a stereoscopic image captured by a video camera 1 and a video camera 2, and imaging. The video camera 1, via the turntables 18, 19, which are variable means for controlling the direction of the optical axis of the lenses of the line-of-sight detection camera 7, the video cameras 1 and 2, for detecting the line of sight of a person. camera moving base 8 for fixing the 2, camera 1, the rotary base 19, the camera moving base 8, the image signal processing unit 21 is a processing unit to be described later, Four debris detection unit 22, controls the video tape recorder 26 The controller 20, which is a control means, the image signal processing unit 21, which converts the image signal captured by the cameras 1 and 2 into a signal suitable for the display of the stereoscopic display 6, the image signal from the cameras 1 and 2, and the viewpoint described later. Focus detection unit 22, cameras 1, 2 that input the position information of the viewpoint obtained by the detection unit 24, detect the focusing state of the image signal near the target point corresponding to the position of the viewpoint, and send the information to the controller 20. By inputting the image signal and the position information of the viewpoint obtained by the viewpoint detection unit 24 described later, the target object on the image signal corresponding to the position of the viewpoint is grasped and the region on the captured image is detected. The target point detection unit 23 that sends the position information to the controller 20, the viewpoint detection unit 24 that determines the photographer's viewpoint based on the image signal obtained from the line-of-sight detection camera 7, and sends the information to the controller 20. The photographer's position detection unit 25, camera 1, which extracts the outline of the photographer's face from the image captured by the camera 7 and determines the distance between the photographer and the line-of-sight detection camera 7 and sends the information to the controller 20. A video tape recorder that records the image signal obtained in step 2 and the signal of the convergence angle, reproduces the stereoscopic image of the stereoscopic display 6, sends the reproduced signal to the image signal processing unit 21, and sends the information of the convergence angle to the controller 20. It has 26.

【0031】

The video cameras 1 and 2 are imaged on the lenses 14 and 15, the image sensors 16 and 17 for photoelectric conversion of the optical image, the focusing controllers 10 and 11 for controlling the focusing of the lenses 14 and 15, and the image sensor, respectively. Built-in zoom controllers 12 and 13 that control the size of the optical image.

【0032】

The video cameras 1 and 2 are fixed to the camera moving table 8 via the rotating tables 18 and 19 that control the direction of the optical axis of the lens, and are configured so that the distance between the video camera 1 and the video camera 2 can be changed. ing.

【0033】

FIG. 2 is a schematic explanatory view showing the appearance of the three-dimensional imaging system of FIG.

【0034】

Next, the operation of each of the above components will be described with reference to FIGS. 1 and 2.

【0035】

The 3D image capturing system shown in FIG. 1 is equipped with a plurality of video cameras 1 and 2 (two in this embodiment) for capturing parallax images, and is configured to capture parallax images in different directions. There is.

【0036】

The parallax image captured by the video cameras 1 and 2 is displayed in real time on the stereoscopic display 6 built into the 3D image capturing system shown in FIG. In addition, the image signal and the signal of the convergence angle of the video cameras 1 and 2 are output to the outside.

【0037】

The stereoscopic display 6 that displays the stereoscopic image captured by the video cameras 1 and 2 is configured by combining the liquid crystal panel 5 and the lenticular lens 4. The photographer looks at the stereoscopic display 6 and controls the video cameras 1 and 2 so as to obtain a desired captured image while checking the currently captured image.

【0038】

The stereoscopic display 6 is not limited to the lenticular type, and may be a polarized glasses type or a shutter glasses type. The liquid crystal panel 5 can also be replaced with another flat panel display, electroluminescence (EL), plasma display, or light emitting diode array.

【0039】

The line-of-sight detection camera 7 for detecting the line-of-sight of the photographer detects where the photographer is looking at the stereoscopic image reproduced on the stereoscopic display 6, and determines the distance between the photographer and the stereoscopic display 6. It is also used for general and detection.

【0040】

Hereinafter, the components of FIG. 1 will be described in detail.

【0041】

The video camera 1 changes the focusing of the lens 14 based on the lens 14 that forms an optical image on the image sensor 16, the image sensor 16 that photoelectrically converts the formed optical image, and the signal sent from the controller 20. It is composed of a focusing controller 10 for making the lens and a zoom controller 12 for changing the size of an optical image imaged on an image sensor based on a signal sent from the controller 20.

【0042】

The video camera 2 changes the focusing of the lens 15 based on the signal sent from the lens 15 that forms an optical image on the image sensor 17, the image sensor 17 that photoelectrically converts the formed optical image, and the controller 20. It is composed of a focusing controller 11 and a zoom controller 13 that change the size of an optical image imaged on an image sensor based on a signal sent from the controller 20.

【0043】

The image signals captured by the image pickup elements 16 and 17 are sent to the image signal processing unit 21 and converted into signals suitable for display on the lenticular stereo display 6. The stereoscopic display 6 displays a stereoscopic image based on the parallax image captured by the video cameras 1 and 2.

【0044】

The line-of-sight detection camera 7 captures the photographer's face in the stereoscopic observation area of the stereoscopic display 6 and sends the obtained image signal to the viewpoint detection unit 24.

【0045】

The viewpoint detection unit 24 obtains the direction of the face and the direction of the line of sight from the features of the photographer's face, determines where the photographer is looking at the stereoscopic display 6 (viewpoint), and sends the information to the controller 20.

【0046】

The focus detection unit 22 inputs the image signals output from the image sensors 16 and 17, and also inputs the position information of the viewpoint obtained by the viewpoint detection unit 24 from the controller 20. The focus detection unit 22 detects the focusing state of the image signal near the target point corresponding to the position of the viewpoint, and sends the information to the controller 20. The controller 20 sends a focusing control signal to the focusing controllers 10 and 11 based on the information, and focuses the focusing on the target object corresponding to the viewpoint on the stereoscopic display 6. The target point detection unit 23 inputs an image signal from the image sensors 16 and 17, and the position information of the viewpoint obtained by the viewpoint detection unit 24 from the controller 20, and inputs the target object on the image signal corresponding to the position of the viewpoint. It grasps, detects which area on the captured image it is located, and sends the position information to the controller 20. The controller 20 sends a signal to the turntables 18 and 19 so that the target object is at the center position on the captured image.

【0047】

The photographer position detection unit 25 stores in advance the size of the photographer's face viewed at a certain reference distance, sequentially compares the size of the face captured by the line-of-sight detection camera 7, and compares the size of the face captured by the line-of-sight detection camera 7 with the photographer. The distance between the detection cameras 7 is roughly detected. Thereby, it is detected whether the photographer is closer or farther than the reference distance. The controller 20 recognizes this operation and sends a signal to the zoom controllers 12 and 13 and also sends a signal to the camera moving table 8.

【0048】

The zoom controllers 12 and 13 change the size of the formed optical image according to the signal sent from the control 20.

【0049】

The camera moving table 8 controls the distance between the video cameras 1 and 2 (the principal points of the lenses 14 and 15) according to the signal sent from the controller 20.

【0050】

The video tape recorder 26 records the image signals taken by the image sensors 16 and 17 and the signals of the convergence angles of the video cameras 1 and 2. The videotape recorder 26 may be replaced with a writable optical disc recorder. It is also possible to reproduce the stereoscopic image of the stereoscopic display 6 by reproducing the tape on which the stereoscopic image is recorded on the video tape recorder 26. At that time, the video tape recorder 26 sends the reproduced signal to the image signal processing unit 21, and the convergence angle information to the controller 20. The controller 20 calculates the amount of horizontal movement of the image frame and sends the signal to the image signal processing unit 21, and the image signal processing unit 21 processes the horizontal movement of the image frame to produce an image of the stereoscopic display 6. Send a signal to reproduce a stereoscopic image.

【0051】

Next, the operation of the three-dimensional imaging system of FIGS. 1 and 2 will be described with reference to the flowchart of FIG.

【0052】

The subject is photographed with the lenses 14 and 15 and the image sensors 16 and 17 included in the video cameras 1 and 2 (step S1). The image signal processing unit 21 converts the image signals obtained by the image sensors 16 and 17 for stereoscopic display and displays the stereoscopic image on the stereoscopic display 6 (step S2).

【0053】

The photographer looks at the reproduced stereoscopic image, gazes at the reproduced image of the subject to be photographed, captures the state with the line-of-sight detection camera 7, and uses the image processing of the viewpoint detection unit 24 to gaze at the viewpoint on the stereoscopic display 6. Find out (step S3). The controller 20 knows the position information of the gazing point.

【0054】

The focus detection unit 22 obtains the position information of the gazing point from the controller 20, detects the focusing state in the vicinity of the frame of the image signal obtained from the image sensors 16 and 17 (step S4), and the information is obtained. To controller 20 (step S5).

【0055】

The controller 20 sends a signal to the focusing controllers 10 and 11 according to the information to change the focusing state (step S6). A feedback loop is formed between the focus detection unit 22, the controller 20, and the focusing controllers 10 and 11, and each operation is repeatedly performed until the image near the gazing point is in focus (step S7).

【0056】

Next, the operation of aligning the optical axes of the video cameras 1 and 2 with the subject to be photographed is shown.

【0057】

The target point detection unit 23 obtains the position information of the gazing point from the controller 20 and detects in the horizontal direction the position of the subject to be photographed in each image frame photographed by the image sensors 16 and 17. Then (step S8), the difference between the position and the midpoint in the horizontal direction is taken and the value is sent to the controller 20. The controller 20 sends a signal to move the turntables 18 and 19 in the direction of decreasing the value (step S9), sends a signal to the turntables 18,19, and the turntables 18 and 19 send the signal. Rotate based on (step S10). In this way, the subject to be photographed is always centered in the horizontal direction in the frame captured by the video cameras 1 and 2, and the optical axis of each video camera 1 and 2 is aligned with the subject (step S11). The controller 20 grasps the information of the internal convergence angle of each of the video cameras 1 and 2 at that time.

【0058】

From the internal convergence angle of each video camera 1 and 2 and the distance between each video camera 1 and 2, the distance from the lenses 14 and 15 of each video camera 1 and 2 to the intersection of the optical axes is calculated (step S12), and the video camera In order to accurately display the depth information captured by 1 and 2 on the stereoscopic display 6, the image frames taken by the respective image pickup elements 16 and 17 are shifted in the horizontal direction (step S13) and displayed on the stereoscopic display 6. (Step S14). The amount of deviation of the image frame can be calculated geometrically. The controller 20 calculates the amount of deviation, sends it to the image signal processing unit 21, and displays the stereoscopic image with the adjusted convergence angle on the stereoscopic display 6.

【0059】

The photographer position detection unit 25 extracts the outline of the photographer's face from the image captured by the line-of-sight detection camera 7, and in advance, the photographer's face when the line-of-sight detection camera 7 and the photographer are at a reference distance. The size of the face is memorized and the sizes of the faces that are input sequentially are compared to determine whether the photographer is closer or farther than the reference distance to the line-of-sight detection camera 7. Then, the information is sent to the controller 20 (step S15).

【0060】

The controller 20 sends a zoom-up (telephoto) signal to the zoom controllers 12 and 13 when the photographer position detection unit 25 receives the information "close", and the zoom controller 12 and 13 receive the information "far". A zoom-down (wide-angle) signal is sent to 13, and in parallel, the controller 20 calculates an appropriate interval between the video cameras 1 and 2 and sends a signal to the camera moving table 8 to adjust the interval (step S16). , S17, S18). The image signals taken by the image sensors 16 and 17 and the information on the convergence angles of the video cameras 1 and 2 at that time are output to the outside (step S19). Alternatively, the image signals captured by the image sensors 16 and 17 and the information on the convergence angles of the video cameras 1 and 2 at that time are recorded on the videotape recorder 26 (step S20).

【0061】

When the series of processes is completed, the process returns to the start and the series of processes is repeated again.

【0062】

The flow shown in FIG. 3 is an example, and the order of processing may be changed or labor may be saved.

【0063】

Next, each of the above processes will be described in detail.

【0064】

The line-of-sight detection camera 7 is fixed with a position and an optical system that captures the entire head of the photographer (observer of the stereoscopic display 6) within the area where the stereoscopic image reproduced by the stereoscopic display 6 can be observed. .. In order to know the relative position of the line-of-sight detection camera 7 and the photographer's head, the line-of-sight detection camera 7 must be fixed.

【0065】

The viewpoint detection unit 24 first extracts the contour of the face from the image signal obtained from the viewpoint detection unit camera 7. Then, the orientation of the face with respect to the stereoscopic display surface is obtained. Next, both eyes are extracted and the line-of-sight direction is obtained from the position of the pupil. Further, the approximate distance between the line-of-sight detection camera 7 and the photographer's head is obtained from the size of the photographer reflected in the line-of-sight detection camera 7. The position of the viewpoint on the stereoscopic display 6 that the photographer is gazing at can be determined from the direction of the face, the direction of the line of sight, and the distance to the photographer's head. Strictly speaking, since the stereoscopic display 6 displays an image including binocular parallax, the viewpoints should be different for the right eye and the left eye. However, in general, the subject is not seen in front of the eyes, and the subject also has a certain size, so that the viewpoints of the right eye and the left eye can be regarded as the same.

【0066】

The focus detection unit 22 extracts a certain region centered on a point corresponding to the position of the viewpoint for each image frame taken by the image sensors 16 and 17, and the high-frequency component of the image signal included in the region. Measure the size. The magnitude of the high frequency component of the image signal reflects the focusing state. As a result, the focusing state is detected. This process is performed for each image frame taken by the image sensors 16 and 17.

【0067】

Next, the relationship between the convergence angle of the camera and the captured image will be described with reference to FIGS. 4a, 4b, and 4c.

【0068】

FIG. 4a shows the positional relationship between the camera and the subject.

【0069】

The x-axis is the straight line passing through the principal point of the lens 14 of the video camera 1 and the principal point of the lens 15 of the video camera 2, and the principal point of the lens 14 is the origin. The z-axis orthogonal to the x-axis is set in the plane including the optical axes 30 and 31 of the video cameras 1 and 2.

【0070】

It is assumed that there are small subjects C (), A (), and B () in the xz plane. Let the viewing angle of the lens 14 be φ1 and the viewing angle of the lens 15 be φ2. The overlapping space of the conical spaces cut out at the respective viewing angles φ1 and φ2 is the space that can be reproduced stereoscopically. In the xz plane, it is represented by the shaded area (actually, it extends more in the z-axis direction) in Fig. 4a.

【0071】

It is assumed that the optical axis 30 of the lens 14 and the optical axis 31 of the lens 15 intersect at the position where the subject A is placed (at a distance Z1). Let θ1 and θ2 be the angles (convergence angles) formed by the optical axis 30 and the optical axis 31 with the z-axis, respectively. The convergence angles θ1 and θ2 are not always the same, but if they are significantly different, the distance from each lens to the subject will be different, and the sizes of the captured images will not match, making stereoscopic viewing difficult. In addition, the viewing angles φ1 and φ2 must always be set to the same angle (zoom magnification) in order to match the sizes of the captured images.

【0072】

The captured images captured by the image sensor 16 and the image sensor 17 are shown in FIGS. 4b and 4c. The object on the optical axis 30 is projected at the center point 34 of the image frame, and similarly, the object on the optical axis 31 is projected at the center point 35 of the image frame.

【0073】

In FIGS. 4b and 4c, the subject A is reflected at the center points 34 and 35. The positions where subjects B and C are reflected in each image frame are different in the horizontal direction. This is binocular parallax. Subject A has no binocular parallax. That is, binocular parallax is not attached to an object existing in a plane including the intersection of the optical axes and perpendicular to the xz plane (a plane including two optical axes).

【0074】

The convergence angles of the optical axes 30 and 31 are accurately measured by a rotary encoder built in the turntables 18 and 19. The distance between the principal points of the lens 14 and the lens 15 is set to about 65 mm, which matches the average distance of the human eye.

【0075】

By simply bringing the subject you want to shoot to the center of each image frame, you can align each optical axis with the subject you want to shoot. However, the vertical position in the image frame depends on the tilt of each optical axis, and since the image pickup device is held and supported by the hand, the accuracy is not so high. Therefore, the subject to be photographed may be aligned with the horizontal center lines 32 and 33 of each image frame. The position of the subject to be photographed in each image frame is detected by the target point detection unit 23.

【0076】

Next, the stereoscopic image reproduction will be described with reference to FIG. Similar to FIGS. 4a, 4b, and 4c, the left eye and the right eye are placed at the positions of the lenses 14 and 15, and the x-axis and the z-axis are taken. It is assumed that the stereo display 6 is located on a plane perpendicular to the xz plane at a distance Zd.

【0077】

If the stereoscopic display 6 is of the lenticular type, each captured image is cut into elongated strips for stereoscopic display, and the images are woven one by one to create an image. If you look at it through a lenticular, you can see a stereoscopic image separated into two images.

【0078】

When the image frames shown in FIGS. 4b and 4c are combined by aligning the center points 34 and 35 and displayed at the center of the stereoscopic display 6, the stereoscopic image is reproduced with reference to the surface of the stereoscopic display 6. That is, the object existing at the distance Z1 (the intersection of the optical axis 30 and the optical axis 31) is reproduced at the distance Zd. The distance Z1 can take various values, but the distance Zd is almost unchanged. Depth information changes each time the position of the intersection of the optical axis 30 and the optical axis 31 is changed.

【0079】

Therefore, the congestion parallax due to the change in the convergence angle is provided and corrected. Instead of superimposing the centers 34 and 35 of the image frame on the center D of the stereoscopic display 6, the display is shifted. The distance Zd between the eye and the stereoscopic display 6 is fixed. Let point D be the center of the stereoscopic display 6. Let θd be the angle between the line of sight and the z-axis when both eyes see point D. The convergence angle parallax d1 of the center 34 of the image frame and the center D of the stereoscopic display 6 is d1 = Zd (tanθ1-tanθd) ......... (1) Given in, the amount of convergence parallax d2 at the center 34 of the image frame and the center D of the stereoscopic display 6 is d2 = Zd {tan (-θ2) -tan (-θd)} ......... (2) Given in.

【0080】

In this way, even if the convergence angle of the optical axis of the camera changes, the sense of depth captured by the camera at the time of shooting is accurately reproduced.

【0081】

When playing back on an external stereoscopic display, information on the image signal taken by each camera and the convergence angle of each camera is required. The distance Zd and the angle θd are values determined by the positional relationship between the stereoscopic display 6 and the observer.

【0082】

Next, the processing at the time of zooming will be described. With zooming, you can easily obtain an enlarged view (telephoto) and a reduced view (wide angle) of the subject without moving the position of the camera. Figure 6 shows the relationship between the subject and the lens in zooming with a monocular.

【0083】

A zoom lens is usually composed of a plurality of lenses, but is omitted in FIG. The image sensor 41 is at the imaging position of the zoom lens 40.

【0084】

Let a be the size of the field of view (subject) with the zoom lens 40 in the state of a standard lens, the viewing angle of φ1, and the distance Z away. A standard lens is a lens that can take an image that looks the same size as the real thing when displayed on a display of a standard size that is far from the standard distance. Suppose you zoom in and the field of view becomes b. At that time, the viewing angle is reduced to φ2. Since the size of the image sensor 41 is invariant, the image is a / b times larger.

【0085】

From another point of view, there is a point where the field of view becomes the size of b when approaching the subject on the optical axis 42 while keeping the viewing angle at φ1. The distance between the subject and the lens at that time is Z'. An image taken by zooming in at a distance Z is close to a distance Z'and is almost the same as an image taken with a standard lens. The distance Z'is calculated by the following equation (3).

【0086】

Z'= (b / a) Z ......... (3) Figure 7 shows the case of two zoom lenses. Zoom lenses 50 and 51 are installed at intervals of c. When the zoom lenses 50 and 51 are standard lenses, the distance c is usually adjusted to the average distance of the human eye, about 65 mm. Image sensors 52 and 53 are placed at the respective imaging positions.

【0087】

Let a be the size of the field of view when desired with a standard lens at a distance Z. Suppose that each lens zooms in and the field of view becomes b. As described in FIG. 6, this is equivalent to an image taken by the lens approaching a distance Z'. However, the distance between the lenses (the distance between the optical axis 54 and the optical axis 55) is reduced to d.

【0088】

The distance between the lenses is related to the relative size of the observer and the subject when the stereoscopic image is reproduced. The smaller the distance between the lenses, the more the observer feels like a dwarf. In order to reproduce the stereoscopic effect that can be seen when a person of normal size approaches by zooming in, the sense of the lens must be expanded.

【0089】

The distance between the optical axis 56 and the optical axis 57 may be c (about 65 mm) at a position Z'away from the subject. The distance between the lenses (distance between the optical axis 56 and the optical axis 57) e when the distance Z at that time is separated is expressed by Eq. (4).

【0090】

e = (Z / Z ) c ......... (4) Equation (4) is from Equation (3) e = (a / b) c ............ (5) Will be.

【0091】

That is, in order to prevent the relationship between the relative size of the observer and the subject from being broken when the magnification of the image is changed by using a zoom lens in a 3D image capturing system, the magnification of the image is adjusted. , You have to change the distance between the cameras.

【0092】

Each image signal captured by the image sensors 16 and 17 and the signal of the convergence angle of the video cameras 1 and 2 are output to the outside. It is also possible to incorporate a video tape recorder into the three-dimensional image capturing system to record each image signal and each convergence angle signal. By doing so, it is possible to comprehensively capture, display, and record a three-dimensional image with only one device.

【0093】

[Effect of the invention]

The three-dimensional image capturing system of the present invention is connected to a photographing means that outputs an image signal corresponding to a photographed subject, a variable means that is connected to the photographing means and changes the convergence angle of the photographing means, and a photographing means. A processing means that converts the image signal output from the cage photographing means by a specific method and moves a frame of the image signal in a predetermined direction according to the convergence angle of the photographing means, and a processing means that is connected to the processing means and is connected to the processing means. Since it is provided with a display means for reproducing a stereoscopic image based on the processed image signal and a control means connected to the processing means for controlling the amount of movement of the frame, the stereoscopic image is determined according to the convergence angle of the photographing means. By providing the convergence parallax in the image composition at the time of reproduction, it is possible to display the stereoscopic image that always reproduces an accurate sense of depth on the display means even if the convergence angle of the photographing means is changed. As a result, it is possible to take a three-dimensional image while constantly checking the reproduced stereoscopic image by reproducing accurate depth information with excellent operability.

【0094】

Further, the three-dimensional image capturing system of the present invention is connected to a photographing means that outputs an image signal corresponding to the photographed subject, a variable means that is connected to the photographing means and changes the convergence angle of the photographing means, and a photographing means. It is connected to a display means for reproducing a stereoscopic image based on an image signal output from the photographing means, a viewpoint detecting means for the photographer to detect the position of the photographer's gazing point on the display means, and a viewpoint detecting means. The focus detection means that detects the focus state on the display means based on the detected position and outputs the state signal, and the focus state of the photographing means that is connected to the focus detection means and is based on the state signal. Since it is provided with the focus adjusting means for adjusting and the convergence angle controlling means for adjusting the convergence angle of the photographing means based on the detected position connected to the viewpoint detecting means, the convergence angle of the photographing means can be freely changed. By making it possible, it is possible to widen the shooting area and always clearly capture the subject to be shot. In addition, it is possible to shoot while checking the stereoscopic image being shot by the display means. Then, the photographer can recognize the subject to be photographed and set the convergence angle and the focus state of the photographing means by gazing at the reproduced image on the display means. As a result, the operability of the tertiary device can be remarkably improved.

【0095】

Further, the three-dimensional image capturing system of the present invention includes a photographing means that outputs an image signal corresponding to the photographed subject, a display means that reproduces a stereoscopic image based on the image signal output from the photographing means, and a display means. The convergence angle of the photographing means is determined based on the photographer position detecting means that detects the distance to the photographer and outputs a distance signal and the distance signal that is connected to the photographer position detecting means and is output from the photographer position detecting means. An adjustable convergence angle adjusting variable means, an interval adjusting variable means connected to the photographer position detecting means, and an interval adjusting variable means for adjusting and changing the interval of the photographing means based on a distance signal output from the photographer position detecting means, and a photographer. Since it is connected to the position detecting means and has a magnifying power adjusting means for adjusting the image magnifying power of the photographing means based on the distance signal output from the photographer's position detecting means, the distance between the photographing means and the photographer can be increased. By roughly detecting, controlling zoom up / down, and adjusting the distance between the shooting means according to the zoom magnification, the relative size between the subject and the photographer can be maintained, and the distance between the shooting means can be maintained. Images that move closer to or further away from the subject while maintaining their positions can be taken by simply changing the magnification of the shooting means. As a result, the shooting conditions are relaxed and the number of images that can be shot increases.

[Simple explanation of drawings]

[Figure 1]

It is a block diagram which shows the structure of one Example of the 3D image taking system of this invention.

[Figure 2]

It is an external perspective view which shows the outline of the appearance of the 3D image taking system of FIG.

[Fig. 3a]

This is the first half of the flowchart for explaining a series of processes of the three-dimensional image capturing system of FIG.

[Fig. 3b]

This is the latter half of the flowchart for explaining a series of processes of the three-dimensional image capturing system of FIG.

[Fig. 4]

It is explanatory drawing of the relationship between the convergence angle of a camera and the captured image in the 3D image taking system of FIG.

[Fig. 5]

It is explanatory drawing of the relationship between the display image and 3D reproduction in the 3D image taking system of FIG.

[Fig. 6]

It is explanatory drawing of the relationship between the monocular zoom camera and the shooting distance in the 3D image shooting system of FIG.

[Fig. 7]

It is explanatory drawing of the relationship between the twin-lens zoom camera and the lens spacing in the 3D image taking system of FIG.

[Fig. 8]

It is explanatory drawing of the operation of the conventional 3D image taking apparatus.

[Fig. 9]

It is explanatory drawing of the image taken by the 3D image taking apparatus shown in FIG.

[Fig. 10]

It is explanatory drawing of the conventional display image.

[Fig. 11]

It is explanatory drawing of the conventional 3D reproduction.

[Explanation of symbols]

1,2 camcorder 4 Lenticular lens 5 LCD panel 6 stereoscopic display 7 Eye-gaze detection camera 8 Camera moving table 10,11 Focusing controller 12,13 Zoom controller 14,15 lens 16,17 Image sensor 18,19 turntable 20 controller 21 Image signal processing unit 22 Focus detector 23 Target point detector 24 viewpoint detector 25 Photographer position detector 26 Videotape recorder