3-dimensional shape detection device, 3-dimensional shape detection method, 3-dimensional shape detection program

The present invention relates to a three-dimensional shape detection device, a three-dimensional shape detection method, and a three-dimensional shape detection program that can detect the boundary of pattern light at high speed with subpixel accuracy without increasing the number of imaged patterns of pattern light.

Conventionally, one slit light is sequentially projected on a target object, an image is input by an imaging means for each direction φ of each projected light, and the direction θ in which the target object is viewed from the imaging means is determined from the trajectory of the slit light in the image. There is known a three-dimensional shape detecting device that detects a three-dimensional shape of a target object by using a slit light projection method that detects the position of the target object.

However, in this slit light projection method, the resolution is determined by the number of slits, and one video frame is required to obtain one slit image. Therefore, in order to obtain a large number of slit images. Then, there was a problem that it took a long time.

Therefore, in order to solve this problem, the so-called spatial coding method is proposed in Non-Patent Document 1 below. In the space code method, n vertical striped pattern lights are projected onto the target object to create a space of 2.ⁿDivide into thin fan-shaped areas. An n-bit binary code (spatial code) can be assigned to each area, and in n times of pattern light projection, 2ⁿSince an image equivalent to the slit image of a book can be obtained, it is possible to perform high-speed measurement as compared with the slit light projection method.

However, the division of the pattern light is finite and the boundary of the pattern light detected due to the relationship with the resolution of the image sensor contains an error, and as a result, the three-dimensional shape of the target object is detected with high accuracy. There was a problem that it could not be done.

Therefore, in order to solve such a problem, Non-Patent Document 2 describes two types of pattern light: a positive (positive image) pattern light and a negative (negative image) pattern light in which the positive pattern light and the light / darkness are inverted. Is projected onto the target object, the boundary of the pattern light is inserted from the brightness distribution of the captured image, and the boundary of the pattern light is detected with subpixel accuracy, thereby reducing the error included in the boundary coordinates of the pattern light. However, a technique for detecting the three-dimensional shape of a target object with high accuracy is disclosed.<nplcit num="1"><text>Kosuke Sato, 1 other person, "Distance image input by spatial coding", IEICE Transactions, 85 / 3Vol.J 68-D No3 p369 ~ 375</text></nplcit><nplcit num="2"><text>Kosuke Sato, 1 other person, "3D image measurement", Shokodo Co., Ltd., p109 ~ 117</text></nplcit>

<p> However, when the method described in Non-Patent Document 2 described above is used to detect the boundary of the pattern light with subpixel accuracy, the positive / negative pattern light projection image 2 * n and the non-projection image 1 It is necessary to take an image of the image, and there is a problem that the number of times the pattern light is projected and the number of images of the pattern light are increased, and it takes a long time to measure.</p><p> Further, in the above-mentioned spatial code method, in order to reduce the error in coding, the projected pattern light is not brightened with a pure binary code but is called a gray code (alternative binary code). It was necessary to project a special pattern of light.</p><p> The present invention has been made to solve the above-mentioned problems, and is capable of detecting the boundary of the pattern light at high speed with subpixel accuracy without increasing the number of imaged patterns of the pattern light. It is an object of the present invention to provide a device, a three-dimensional shape detection method, and a three-dimensional shape detection program.</p>

<p> In order to achieve this object, the three-dimensional shape detecting device according to claim 1 is a projection means for projecting each of a plurality of types of patterned lights in which light and dark are alternately arranged on a subject in chronological order, and the projection means thereof. An imaging means that captures a subject in a state in which each pattern light is projected from the ground, and a luminance image generating means that generates a plurality of luminance images obtained by calculating the luminance of each pixel from each captured image captured by the imaging means. A code image generation means that generates a code image in which a predetermined code is assigned to each pixel based on the result of performing threshold processing with a predetermined threshold on a plurality of luminance images generated by the luminance image generation means. It has a code of interest at a three-dimensional shape calculating means that calculates a three-dimensional shape of a subject using a code image generated by the code image generating means and a detection position in a direction that intersects the pattern light in the code image. Brightness and darkness at positions corresponding to the first pixel detecting means that detects the first pixel that is adjacent to the pixel and has a code different from the code of interest, and the first pixel that is detected by the first pixel detecting means. A luminance image extracting means for extracting a luminance image having the boundary of the above from the plurality of luminance images, a pixel area specifying means for specifying a pixel area composed of pixels in a predetermined area adjacent to the first pixel, and a pixel area thereof. An approximate expression calculation means for calculating an approximate expression representing a change in brightness in the extracted luminance image in a pixel region specified by the specific means, and a position having a predetermined luminance-related threshold in the approximate expression calculated by the approximate expression calculation means. Is provided, and the boundary coordinate detecting means for detecting the boundary coordinates of the code of interest based on the calculation result is provided, and the three-dimensional shape calculating means is detected by the boundary coordinate detecting means using the code image. The three-dimensional shape of the subject is calculated based on the boundary coordinates.</p><p> According to the three-dimensional shape detection device according to claim 1, the first pixel, which is adjacent to the code of interest and has a code different from the code of interest, is the first pixel from the detection position in the direction orthogonal to the pattern in the code image. It is detected in pixel units by the image detecting means. When the first pixel is detected, a luminance image having a boundary between light and dark at a position corresponding to the first pixel is extracted from the plurality of luminance images by the luminance image extraction means. Then, in the pixel area specified by the pixel area specifying means, an approximate expression representing a change in brightness in the extracted luminance image is calculated by the approximate expression calculation means. By calculating the position having a predetermined threshold value for brightness in this approximate expression by the boundary coordinate detecting means, the boundary position with subpixel accuracy at the detection position can be calculated. That is, the boundary coordinates consisting of the coordinates of the detection position and the coordinates of the position having a predetermined threshold value regarding the brightness in the approximate expression are calculated, and the three-dimensional shape of the subject is calculated by the three-dimensional shape calculation means using the boundary coordinates. Will be done.</p><p> The three-dimensional shape detecting device according to claim 2 has a resolution in which the boundary coordinates of the code of interest detected by the boundary coordinate detecting means in the three-dimensional shape detecting device according to claim 1 are higher than the resolution of the imaging means. It is calculated by.</p><p> The three-dimensional shape detecting device according to claim 3 is the three-dimensional shape detecting device according to claim 1 or 2, wherein the pixel area specifying means is a pixel array including the first pixel with reference to the code image. Separately, the pixel region is specified from at least one or more pixel rows continuously arranged from the pixel row including the first pixel, and the approximation formula calculation means calculates the approximation formula for each pixel row of the pixel region. The calculation is performed, and the boundary coordinate detecting means calculates a position having a predetermined threshold value for brightness for each of the approximate expressions, and detects the boundary coordinates of the code of interest based on the calculation result.</p><p> The three-dimensional shape detecting device according to claim 4 is the three-dimensional shape detecting device according to claim 1 or 2, wherein the luminance image extracting means extracts at least one extracted luminance image and specifies the pixel area. The means specifies the pixel region with reference to the code image for each extracted luminance image, and the approximation formula calculating means calculates the approximation formula for each pixel string in the pixel region detected from each extracted luminance image. The calculation is performed, and the boundary coordinate detecting means calculates a position having a predetermined threshold value regarding the brightness for each of the approximate expressions, and detects the boundary coordinates of the code of interest based on the calculation result.</p><p> The three-dimensional shape detecting device according to claim 5 is the three-dimensional shape detecting device according to claim 1 or 2, wherein the luminance image extracting means extracts at least one extracted luminance image and specifies the pixel area. The means means that, for each extracted luminance image, the code image is referred to, and the pixel sequence including the first pixel is separated from the pixel sequence including the first pixel, and at least one row or more of the pixel rows consecutively arranged from the pixel sequence including the first pixel is described. The pixel region is specified, the approximation formula calculation means calculates the approximation formula for each pixel sequence in the pixel region for each extracted luminance image, and the boundary coordinate detecting means approximates each extracted luminance image. A position having a predetermined threshold value for brightness is calculated for each equation, and the boundary coordinates of the code of interest are detected based on the calculation result.</p><p> The three-dimensional shape detecting device according to claim 6 is the three-dimensional shape detecting device according to any one of claims 3 to 5, wherein the boundary coordinate detecting means is a weighted average value of positions calculated for each of the approximate expressions. Alternatively, the median value is calculated, and the boundary coordinates of the code of interest are detected based on the calculation result.</p><p> The three-dimensional shape detecting device according to claim 7 is the three-dimensional shape detecting device according to claim 3, wherein the boundary coordinate detecting means calculates an approximate expression of a position calculated for each of the approximate expressions, and calculates the approximate expression. Based on the result, the boundary coordinates of the code of interest are detected.</p><p> The three-dimensional shape detecting device according to claim 8 is the three-dimensional shape detecting device according to claim 5, wherein the boundary coordinate detecting means obtains an approximate expression of a position calculated for each of the approximate expressions in units of extracted brightness images. Calculate, calculate the boundary coordinates of the focus code based on the calculation result, calculate the weighted average or median for the boundary coordinates of the focus code calculated for each extracted brightness image unit, and based on the result. The boundary coordinates of the code of interest are detected.</p><p> The three-dimensional shape detection method according to claim 9 has a projection step of projecting each of a plurality of types of pattern lights in which light and dark are alternately arranged on a subject in time series, and each pattern light is projected from the projection step. Generated by an imaging step of capturing a subject in a state of being in a state, a luminance image generation step of generating a plurality of luminance images obtained by calculating the brightness of each pixel from each captured image captured by the luminance image, and a luminance image generation step. A code image generation step of generating a code image in which a predetermined code is assigned to each pixel based on the result of performing threshold processing with a predetermined threshold for a plurality of luminance images to be performed, and a code image generation step of the code image generation step. After the three-dimensional shape calculation step of calculating the three-dimensional shape of the subject using the code image and the code image generation step, the code of interest is set at the detection position in the direction intersecting the pattern light in the code image. Bright and dark at the position corresponding to the first pixel detected in the first pixel detection step, which is adjacent to the pixel and has a code different from the code of interest. A luminance image extraction step of extracting a luminance image having a boundary with and from the plurality of luminance images, a pixel region specifying step of specifying a pixel region composed of pixels of a predetermined region adjacent to the first pixel, and a pixel region specifying step thereof. It has a predetermined luminance-related threshold in the approximation formula calculation step of calculating the approximation formula representing the change in brightness in the extracted luminance image in the pixel region specified in the region identification step and the approximation formula calculated by the approximation formula calculation step. A boundary coordinate detection step of calculating a position and detecting the boundary coordinates of the code of interest based on the calculation result is provided, and the three-dimensional shape calculation step is detected by the boundary coordinate detection step using the code image. The three-dimensional shape of the subject is calculated based on the boundary coordinates.</p><p> According to the three-dimensional shape detection method according to claim 9, the first pixel adjacent to the code of interest and having a code different from the code of interest is the first pixel from the detection position in the direction orthogonal to the pattern in the code image. It is detected in pixel units by the image detection step. When the first pixel is detected, a luminance image having a boundary between light and dark at a position corresponding to the first pixel is extracted from the plurality of luminance images by the luminance image extraction step. Then, in the pixel area specified in the pixel area specifying step, an approximate expression representing a change in brightness in the extracted luminance image is calculated by the approximate expression calculation step. By calculating the position having a predetermined threshold value for brightness in this approximate expression by the boundary coordinate detection step, the boundary position with subpixel accuracy at the detection position can be calculated. That is, the boundary coordinates consisting of the coordinates of the detection position and the coordinates of the position having a predetermined threshold value for brightness in the approximate expression are calculated, and the three-dimensional shape of the subject is calculated by the three-dimensional shape calculation step using these boundary coordinates. Will be done.</p><p> The three-dimensional shape detection program according to claim 10 has a projection step of projecting each of a plurality of types of pattern lights in which light and dark are alternately arranged on a subject in time series, and each pattern light is projected from the projection step. A luminance image generation step that generates a plurality of luminance images in which the brightness of each pixel is calculated from each luminance image captured by the imaging step, and a luminance image generation step that captures the subject in the state of being in the state, and the luminance image generation step. A code image generation step for generating a code image to which a predetermined code is assigned to each pixel based on the result of performing threshold processing with a predetermined threshold for a plurality of luminance images to be performed, and a code image generation step for generating the code image. At the three-dimensional shape calculation step of calculating the three-dimensional shape of the subject using the code image and the detection position in the direction intersecting the pattern light in the code image, the pixel having the attention code is adjacent to the pixel and the attention thereof. A luminance image having a boundary between light and dark at a position corresponding to the first pixel detection step for detecting the first pixel having a code different from the code and the first pixel detected in the first pixel detection step. A luminance image extraction step for extracting from the plurality of luminance images, a pixel region specifying step for specifying a pixel region composed of pixels in a predetermined region adjacent to the first pixel, and pixels specified in the pixel region specifying step. In the region, the approximation formula calculation step for calculating the approximation formula representing the change in brightness in the extracted luminance image and the position having a predetermined luminance-related threshold in the approximation formula calculated by the approximation formula calculation step are calculated, and the calculation result thereof. The three-dimensional shape calculation step includes a boundary coordinate detection step for detecting the boundary coordinates of the code of interest based on the above code image, and the subject is based on the boundary coordinates detected in the boundary coordinate detection step using the code image. Calculate the three-dimensional shape of.</p><p> According to the three-dimensional shape detection program according to claim 10, the first pixel adjacent to the code of interest and having a code different from the code of interest is the first pixel from the detection position in the direction orthogonal to the pattern in the code image. It is detected on a pixel-by-pixel basis by the image detection step. When the first pixel is detected, a luminance image having a boundary between light and dark at a position corresponding to the first pixel is extracted from the plurality of luminance images by the luminance image extraction step. Then, in the pixel region specified in the pixel region specifying step, an approximate expression representing a change in brightness in the extracted luminance image is calculated by the approximate expression calculation step. By calculating the position having a predetermined threshold value for brightness in this approximate expression by the boundary coordinate detection step, the boundary position with subpixel accuracy at the detection position can be calculated. That is, the boundary coordinates consisting of the coordinates of the detection position and the coordinates of the position having a predetermined threshold value for brightness in the approximate expression are calculated, and the three-dimensional shape of the subject is calculated by the three-dimensional shape calculation step using these boundary coordinates. Will be done.</p>

<p> According to the three-dimensional shape detection device according to claim 1, the boundary coordinates of the pattern light at a predetermined detection position are calculated by an approximate expression representing a change in brightness in a predetermined pixel region including the boundary coordinates. And it is detected as a position having a predetermined threshold value regarding the brightness in the approximate expression, so that the boundary between light and dark at the detection position can be calculated with higher accuracy than the boundary between light and dark in the code image. .. Therefore, if the three-dimensional shape is calculated using the result, there is an effect that the three-dimensional shape of the subject can be calculated with high accuracy.</p><p> According to the three-dimensional shape detecting device according to claim 2, in addition to the effect of the three-dimensional shape detecting device according to claim 1, the boundary coordinates of the code of interest detected by the boundary coordinate detecting means or the boundary coordinate detecting step are Since it is calculated with a resolution (sub-pixel accuracy) higher than the resolution of the imaging means or the imaging process, there is an effect that the three-dimensional shape of the subject can be calculated with high accuracy.</p><p> According to the three-dimensional shape detection device according to claim 3, in addition to the effect of the three-dimensional shape detection device according to claim 1 or 2, the boundary coordinates are calculated based on at least two or more columns of pixels. Since the position having a predetermined threshold value is calculated for each of at least two or more approximate expressions, the accuracy of detecting the boundary coordinates can be improved as compared with the case of obtaining the boundary coordinates using one approximate expression. Therefore, there is an effect that the three-dimensional shape of the subject can be calculated with higher accuracy.</p><p> According to the 3D shape detection device according to claim 4, in addition to the effect of the 3D shape detection device according to claim 1 or 2, the boundary coordinates are at least 2 or more based on the extracted brightness image of at least 2 or more. Since the position having a predetermined threshold value is calculated for each of the approximate expressions of, the accuracy of detecting the boundary coordinates can be improved as compared with the case of obtaining the boundary coordinates using one approximate expression. Therefore, there is an effect that the three-dimensional shape of the subject can be calculated with higher accuracy.</p><p> According to the three-dimensional shape detection device according to claim 5, in addition to the effect of the three-dimensional shape detection device according to claim 1 or 2, the boundary coordinates are at least 2 for each of at least two or more extracted brightness images. Since the position having a predetermined threshold value is calculated for each of at least 4 or more approximation formulas calculated based on the pixel rows of columns or more, the boundary coordinates are detected more than when the boundary coordinates are obtained using one approximation formula. It is possible to improve the accuracy of the operation. Therefore, there is an effect that the three-dimensional shape of the subject can be calculated with higher accuracy.</p><p> According to the three-dimensional shape detecting apparatus according to claim 6, in addition to the effects of the three-dimensional shape detecting apparatus according to claims 3 to 5, the boundary coordinate detecting means or the boundary coordinate detecting step is calculated for each approximate expression. Since the weighted average value or median value of the position is calculated and the boundary coordinates of the code of interest are detected based on the calculation result, the burden on the calculation device can be reduced by adding a simple calculation, and the processing speed is increased. It has the effect of being able to.</p><p> According to the three-dimensional shape detecting device according to claim 7, in addition to the effect of the three-dimensional shape detecting device according to claim 3, the boundary coordinate detecting means or the boundary coordinate detecting step is a position calculated for each approximate expression. Since the approximate expression of is calculated and the boundary coordinates of the code of interest are detected based on the calculation result, it is possible to calculate the three-dimensional shape with higher accuracy than calculating the weighted average value or the median value. effective.</p><p> According to the three-dimensional shape detecting device according to claim 8, in addition to the effect of the three-dimensional shape detecting device according to claim 5, the boundary coordinate detecting means is a position calculated for each approximate expression in units of extracted brightness images. Calculate the approximate expression of, calculate the boundary coordinates of the focus code based on the calculation result, calculate the weighted average or median for the boundary coordinates of the focus code calculated for each extracted brightness image, and based on the result. Since the boundary coordinates of the code of interest are detected, the burden on the arithmetic unit can be reduced by simply adding calculations, and there is an effect that the processing speed can be increased.</p><p> According to the three-dimensional shape detection method according to claim 9, the boundary coordinates of the pattern light at a predetermined detection position are calculated by an approximate expression representing a change in brightness in a predetermined pixel region including the boundary coordinates. And it is detected as a position having a predetermined threshold value regarding the brightness in the approximate expression, so that the boundary between light and dark at the detection position can be calculated with higher accuracy than the boundary between light and dark in the code image. .. Therefore, if the three-dimensional shape is calculated using the result, there is an effect that the three-dimensional shape of the subject can be calculated with high accuracy.</p><p> According to the three-dimensional shape detection program according to claim 10, the boundary coordinates of the pattern light at a predetermined detection position are calculated by an approximate expression representing a change in brightness in a predetermined pixel region including the boundary coordinates. And it is detected as a position having a predetermined threshold value regarding the brightness in the approximate expression, so that the boundary between light and dark at the detection position can be calculated with higher accuracy than the boundary between light and dark in the code image. .. Therefore, if the three-dimensional shape is calculated using the result, there is an effect that the three-dimensional shape of the subject can be calculated with high accuracy.</p>

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1 is an external perspective view of the image input / output device 1. The projection device and the three-dimensional shape detection device of the present invention are devices included in the image input / output device 1.

The image input / output device 1 has a digital camera mode that functions as a digital camera, a webcam mode that functions as a webcam, a stereoscopic image mode for detecting a three-dimensional shape and acquiring a stereoscopic image, and a flat surface of a curved document or the like. It is a device provided with various modes such as a flattened image mode for acquiring a digital image.

In FIG. 1, in order to detect the three-dimensional shape of the document P as a subject, particularly in the stereoscopic image mode and the flattened image mode, a striped pattern light formed by alternately arranging light and dark from an image projection unit 13 described later is emitted. The state of projection is illustrated.

The image input / output device 1 is connected to an image pickup head 2 formed in a substantially box shape, a pipe-shaped arm member 3 having one end connected to the image pickup head 2, and the other end of the arm member 3, and is viewed in a plan view. It has a base 4 formed in a substantially L shape.

The image pickup head 2 is a case in which an image projection unit 13 and an image image pickup unit 14, which will be described later, are included therein. In front of the image pickup head 2, a tubular lens barrel 5 is arranged in the center thereof, a finder 6 is arranged diagonally above the lens barrel 5, and a flash 7 is arranged on the opposite side of the finder 6. Further, a part of the lens of the imaging optical system 21 which is a part of the image capturing unit 14 described later is exposed on the outer surface between the finder 6 and the flash 7, and the image of the subject is input from this exposed portion. To.

The lens barrel 5 is a cover that protrudes from the front of the image pickup head 2 and contains a projection optical system 20 that is a part of the image projection unit 13 inside. The lens barrel 5 holds the projection optical system 20 so that the entire projection optical system 20 can be moved for focus adjustment and is prevented from being damaged. Further, from the end surface of the lens barrel 5, a part of the lens of the projection optical system 20 which is a part of the image projection unit 13 is exposed to the outer surface, and the image signal light is projected from this exposed part toward the projection surface. To.

The finder 6 is composed of an optical lens arranged from the back surface to the front surface of the image pickup head 2. When the user looks into the image from the back of the image pickup apparatus 1, a range that substantially matches the range in which the image pickup optical system 21 forms an image on the CCD 22 can be seen.

The flash 7 is, for example, a light source for supplementing a required amount of light in a digital camera mode, and is composed of a discharge tube filled with xenon gas. Therefore, it can be used repeatedly by discharging from a capacitor (not shown) built in the image pickup head 2.

Further, on the upper surface of the image pickup head 2, a release button 8 is arranged on the front side, a mode changeover switch 9 is arranged behind the release button 8, and a monitor LCD 10 is arranged on the opposite side of the mode changeover switch 9.

The release button 8 is composed of a two-stage push button type switch that can be set to two types of states, a "half-pressed state" and a "full-pressed state". The state of the release button 8 is managed by the processor 15, which will be described later, and the well-known autofocus (AF) and autoexposure (AF) functions are activated in the "half-pressed state", and the focus, aperture, and shutter speed are adjusted. Imaging is performed in the "fully pressed state".

The mode changeover switch 9 is a switch that can be set to various modes such as a digital camera mode, a webcam mode, a stereoscopic image mode, a flattened image mode, and an off mode. The state of the mode changeover switch 9 is managed by the processor 15, and the processing of each mode is executed when the state of the mode changeover switch 9 is detected by the processor 15.

The monitor LCD 10 is composed of a liquid crystal display, and receives an image signal from the processor 15 to display an image to the user. For example, the monitor LCD 10 displays an captured image in the digital camera mode or the webcam mode, a three-dimensional shape detection result image in the stereoscopic image mode, a flattened image in the flattened image mode, and the like.

Further, on the side surface of the image pickup head 2, an antenna 11 as an RF (radio frequency) interface and a connecting member 12 for connecting the image pickup head 2 and the arm member 3 are arranged above the antenna 11.

The antenna 11 transmits the captured image data acquired in the digital camera mode, the stereoscopic image data acquired in the stereoscopic image mode, and the like to the external interface by wireless communication via the RF driver 24 described later.

The connecting member 12 is formed in a ring shape having a female screw formed on the inner peripheral surface, and is rotatably fixed to the side surface of the image pickup head 2. Further, a male screw is formed on one end side of the arm member 3. By fitting the female screw and the male screw, the image pickup head 2 and the arm member 3 can be detachably connected to each other, and the image pickup head 2 can be fixed at an arbitrary angle. Therefore, the image pickup head 2 can be removed and used as a normal digital camera (digital camera).

The arm member 3 is for holding the image pickup head 2 in a changeable position at a predetermined image pickup position, and is composed of a bellows-shaped pipe that can be bent into an arbitrary shape. Therefore, the image pickup head 2 can be directed to an arbitrary position.

The base 4 is mounted on a mounting table such as a desk and supports the image pickup head 2 and the arm member 3. Since it is formed in a substantially L-shape in a plan view, it can stably support the image pickup head 2 and the like. In addition, since the base 4 and the arm member 3 are detachably connected to each other, it is convenient to carry and can be stored in a small space.

FIG. 2 is a diagram schematically showing the internal configuration of the image pickup head 2. The image projection unit 13, the image imaging unit 14, and the processor 15 are mainly built in the image pickup head 2.

The image projection unit 13 is a unit for projecting an arbitrary projected image on a projection surface, and includes a substrate 16 and a plurality of LEDs 17 (hereinafter collectively referred to as LED array 17A) along the projection direction. It includes a light source lens 18, a projection LCD 19, and a projection optical system 20. The image projection unit 13 will be described in detail with reference to FIG.

The image capturing unit 14 is a unit for capturing a document P as a subject, and includes an imaging optical system 21 and a CCD 22 along the light input direction.

The imaging optical system 21 is composed of a plurality of lenses, has a well-known autofocus function, and automatically adjusts the focal length and the aperture to form an image of external light on the CCD 22.

The CCD 22 is configured by arranging photoelectric conversion elements such as CCD (Charge Coupled Device) elements in a matrix, and signals according to the color and intensity of the light of the image formed on the surface via the imaging optical system 21. Is generated, converted to digital data, and output to the processor 15.

Processor 15 includes flash 7, release button 8, mode selector switch 9, monitor LCD 10 via monitor LCD driver 23, antenna 11 via RF driver 24, battery 26 via power interface 25, external memory 27, cache. Each of the LED array 17A via the memory 28, the light source driver 29, the projection LCD 19 via the projection LCD driver 30, and the CCD 22 via the CCD interface 31 is electrically connected and managed by the processor 15.

The external memory 27 is a detachable flash ROM, and stores captured images and three-dimensional information captured in the digital camera mode, the webcam mode, and the stereoscopic image mode. Specifically, an SD card, a compact flash (registered trademark) card, or the like can be used.

The cache memory 28 is a high-speed storage device. For example, the captured image captured in the digital camera mode is transferred to the cache memory 28 at high speed, image processing is performed by the processor 15, and then the image is stored in the external memory 27. Specifically, SDRAM, DDRRAM and the like can be used.

The power supply interface 25 is composed of the battery 26, the light source driver 29 is composed of the LED array 17A, the projection LCD driver 30 is composed of the projection LED 19, and the CCD interface 31 is composed of various ICs (Integrated Circuits) that control the CCD 22. ing.

FIG. 3A is an enlarged view of the image projection unit 13, FIG. 3B is a plan view of the light source lens 18, and FIG. 3C is a diagram showing the arrangement relationship between the projection LCD 19 and the CCD 22. As described above, the image projection unit 13 includes a substrate 16, an LED array 17A, a light source lens 18, a projection LCD 19, and a projection optical system 20 along the projection direction.

The board 16 is for mounting the LED array 17A and for electrical wiring with the LED array 17A. Specifically, an aluminum substrate coated with an insulating resin and then patterned by electroless plating, or a single-layer or multi-layered substrate having a color epoxy substrate as a core can be used.

The LED array 17A is a light source that emits radial light toward the projected LCD 19, and a plurality of LEDs 17 (light emitting diodes) are arranged in a staggered pattern on the substrate 16 and adhered via a silver paste. In addition, it is electrically connected via a bonding wire.

By using a plurality of LEDs 17 as a light source in this way, the efficiency of converting electricity into light (electric light conversion efficiency) is improved as compared with the case of using an incandescent lamp, a halogen lamp, etc. as a light source, and at the same time, infrared rays and infrared rays are used. The generation of ultraviolet rays can be suppressed. Therefore, it can be driven with low power consumption, and can save power and extend the service life. In addition, the temperature rise of the device can be reduced.

As described above, since the LED 17 generates extremely low heat rays as compared with the halogen lamp and the like, a resin lens can be adopted as the light source lens 18 and the projection optical system 20 described later. Therefore, each lens 18 and 20 can be constructed at low cost and light weight as compared with the case of adopting a glass lens.

Further, each LED 17 constituting the LED array 17A emits the same emission color, and is configured to emit an amber color using four elements of Al, In, Ga, and P as a material. Therefore, it is not necessary to consider the correction of chromatic aberration that occurs when a plurality of emission colors are emitted, and it is not necessary to adopt an achromatic lens as the projection optical system 20 in order to correct the chromatic aberration. The degree of design freedom can be improved.

In addition, by adopting an amber LED made of a four-element material, which has a high electro-light conversion rate of about 80 lumen / W compared to other emission colors, it is possible to further achieve high brightness, power saving, and long life. The effect of arranging each LED 17 in a staggered pattern will be described with reference to FIG.

Specifically, the LED array 17A consists of 59 LEDs 17, each LED 17 is driven by 50mW (20mA, 2.5V), and in the end, all 59 LEDs 17 are driven by approximately 3W of power consumption. In addition, the brightness as the luminous flux value when the light emitted from each LED 17 passes through the light source lens 18 and the projection LCD 19 and is emitted from the projection optical system 20 is about 25 ANSI lumens even in the case of full irradiation. Is set to.

By adopting this brightness, for example, in the stereoscopic image mode, when detecting the three-dimensional shape of a subject such as the face of a person or an animal, the person or the animal does not give glare to the person or the animal, and the person or the animal looks at the eyes. It is possible to detect a three-dimensional shape that is not crushed.

The light source lens 18 is a lens that collects light radiated from the LED array 17A, and the material thereof is an optical resin typified by acrylic.

Specifically, the light source lens 18 includes a convex lens portion 18a that is projected toward the projection LED 19 side at a position facing each LED 17 of the LED array 17A, and a base portion 18b that supports the lens portion 18a. , The epoxy encapsulant 18c for the purpose of sealing the LED 17 that is filled in the opening containing the LED array 17A and adhering the substrate 16 and the light source lens 18 in the internal space of the base portion 18b, and the base portion 18b. It is provided with a positioning pin 18d that is projected from the side of the substrate 16 and connects the light source lens 18 and the substrate 16.

The light source lens 18 is fixed on the substrate 16 by inserting the positioning pin 18d into the elongated hole 16 formed in the substrate 16 while including the LED array 17A inside the aperture.

Therefore, the light source lens 18 can be arranged in a small space. In addition to the function of mounting the LED array 17A on the board 16, the function of supporting the light source lens 18 is also provided, so that no separate component for supporting the light source lens 18 is required and the number of components is reduced. be able to.

Further, each lens unit 18a is arranged at a position facing each LED 17 of the LED array 17A in a one-to-one relationship.

Therefore, the radial light emitted from each LED 17 is efficiently collected by each lens unit 18 facing each LED 17, and is irradiated to the projected LED 19 as highly directional synchrotron radiation as shown in the figure. The reason why the directivity is improved in this way is that the in-plane transmittance unevenness can be suppressed by injecting light substantially perpendicular to the projection LCD 19. At the same time, the projection optical system 20 has telecentric characteristics and its incident NA is about 0.1, so that only light within the vertical ± 5 ° is regulated so that it can pass through the internal diaphragm. Therefore, it is important to improve the image quality by aligning the emission angles of the light from the LED 17 vertically and putting most of the luminous flux within ± 5 °.

The projection LCD 19 is a spatial modulation element that spatially modulates the light that has passed through the light source lens 18 and is focused, and outputs the image signal light toward the projection optical system 20, and specifically, the vertical and horizontal directions. It is composed of plate-shaped liquid crystal displays with different ratios.

Further, as shown in (C), each pixel constituting the projected LCD 19 has one pixel array arranged in a straight line along the longitudinal direction of the liquid crystal display, and the one pixel array is a liquid crystal display. Other pixel rows that are displaced by a predetermined interval in the longitudinal direction of the above are alternately arranged in parallel.

In (C), it is assumed that the front surface of the image pickup head 2 is directed to the front side of the paper surface, light is emitted from the back side of the paper surface toward the projection LCD 19, and the subject image is formed on the CCD 22 from the labor side of the paper surface.

By arranging the pixels constituting the projection LCD 19 in a staggered manner in the longitudinal direction in this way, the light spatially modulated by the projection LCD 19 is halved in the direction orthogonal to the longitudinal direction (short direction). Can be controlled with. Therefore, the projection pattern can be controlled at a fine pitch, the resolution can be increased, and the three-dimensional shape can be detected with high accuracy.

In particular, in the stereoscopic image mode and the flattened image mode described later, when projecting a striped pattern light in which light and dark are alternately arranged toward the subject in order to detect the three-dimensional shape of the subject, the stripe direction is determined. By matching the projection LCD 19 in the lateral direction, the boundary between light and dark can be controlled at 1/2 pitch, so that the three-dimensional shape can be detected with high accuracy as well.

Further, inside the image pickup head 2, the projection LCD 19 and the CCD 22 are arranged in the relationship shown in (C). Specifically, since the wide surface of the projection LCD 19 and the wide surface of the CCD 22 are arranged so as to face substantially the same direction, when the image projected on the projection surface from the projection LCD 19 is imaged on the CCD 22, The projected image can be imaged as it is without bending the projected image with a half mirror or the like.

Further, the CCD 22 is arranged on the longitudinal direction side (the direction side in which the pixel array extends) of the projection LCD 19. Therefore, especially in the stereoscopic image mode and the flattened image mode, when detecting the three-dimensional shape of the subject using the principle of triangulation, the inclination formed by the CCD22 and the subject should be controlled at 1/2 pitch. Therefore, it is possible to detect a three-dimensional shape with high accuracy as well.

The projection optical system 20 is a plurality of lenses that project the image signal light that has passed through the projection LED 19 toward the projection surface, and is composed of a telecentric lens made of a combination of glass and resin. Telecentric means that the main ray passing through the projection optical system 20 is parallel to the optical axis in the space on the incident side, and the position of the exit pupil is infinite. By making it telecentric in this way, as described above, only the light passing through the projection LCD 19 at a vertical ± 5 ° can be projected, so that the image quality can be improved.

FIG. 4 is a diagram for explaining the arrangement of the LED array 17A. (a) is a diagram showing the illuminance distribution of the light passing through the light source lens 18, (b) is a plan view showing the arrangement state of the LED array 17A, and (c) shows the combined illuminance distribution on the projection LCD 19 surface. It is a figure.

As shown in (a), the light that has passed through the light source lens 18 reaches the surface of the projected LCD 19 as light having a half-width spread half-width θ (= approximately 5 °) and (a) an illuminance distribution as shown on the left side. Designed to do.

Further, as shown in (b), a plurality of LEDs 17 are arranged in a staggered pattern on 16 on the substrate. Specifically, an LED row in which a plurality of LEDs 17 are arranged in series at a d pitch is arranged in parallel at a pitch of 3 / 2d, and the LED rows are moved 1 / 2d in the same direction every other row. It is arranged so that it will be in a state.

In other words, the distance between LED17 of 1 and LCD17 around LED17 of 1 is set to be d (triangular grid arrangement).

The length of d is determined to be less than or equal to the full width at half maximum (FWHM) of the illuminance distribution formed in the projection LCD 19 by the light emitted from one of the LEDs 17.

Therefore, the combined illuminance distribution of the light passing through the light source lens 18 and reaching the surface of the projected LCD 19 becomes a substantially linear shape including a small ripple as shown in (c), and the light is substantially uniformly applied to the surface of the projected LCD 19. Can be irradiated. Therefore, the uneven illuminance in the projection LCD 19 can be suppressed, and as a result, a high-quality image can be projected.

FIG. 5 is an electrical block diagram of the image input / output device 1. The description of the above-described configuration will be omitted. Processor 15 includes a CPU 35, a ROM 36, and a RAM 37.

The CPU35 uses the RAM37 to detect the pressing operation of the release button 8, imports the image data from the CCD22, transfers and stores the image data, and the mode changeover switch 9 according to the processing by the program stored in the ROM36. Performs various processes such as detection of the state of.

The ROM 36 includes a camera control program 36a, a pattern light photographing program 36b, a brightness image generation program 36c, a code image generation program 36d, a code boundary extraction program 36e, a lens aberration correction program 36f, and a triangulation calculation program 36g. , The manuscript orientation calculation program 36h, and the plane conversion program 36i are stored.

The camera control program 36a is a program related to the control of the entire image pickup apparatus 1 including the main processing shown in FIG.

The pattern light photographing program 36b is a program that captures a state in which the pattern light is projected onto the subject and a state in which the pattern light is not projected in order to detect the three-dimensional shape of the document P.

The luminance image generation program 36c takes the difference between the pattern light image obtained by capturing the pattern light projected state by the pattern light photographing program 36b and the pattern light non-image image obtained by capturing the pattern light not projected state, and projects the difference. It is a program that generates a luminance image of the pattern light.

In addition, a plurality of types of pattern light are projected in time series and imaged for each pattern light, and the difference between each of the captured images with pattern light and the image without pattern light is taken, and a plurality of types of pattern light are obtained. A brightness image is generated.

The code image generation program 36d is a program that superimposes a plurality of luminance images generated by the luminance image generation program 36c and generates a code image to which a predetermined code is assigned to each pixel.

The code boundary extraction program 36e uses the code image generated by the code image generation program 36d and the luminance image generated by the luminance image generation program 36c to obtain the boundary coordinates of the code with subpixel accuracy. It is a program.

The lens aberration correction program 36f is a program that corrects the aberration of the imaging optical system 20 with respect to the boundary coordinates of the code obtained by the code boundary extraction program 36e with subpixel accuracy.

The triangulation calculation program 36g is a program that calculates the three-dimensional coordinates of the real space related to the boundary coordinates from the boundary coordinates of the code whose aberration is corrected by the lens aberration correction program 36f.

The manuscript orientation calculation program 36h is a program that estimates and obtains the three-dimensional shape of the manuscript P from the three-dimensional coordinates calculated by the triangulation calculation program 36g.

The plane conversion program 36i is a program that generates a flattened image as if it was captured from the front of the manuscript P, based on the three-dimensional shape of the manuscript P calculated by the manuscript posture calculation program 36h.

The RAM 37 includes an image storage unit 37a with pattern light, an image storage unit 37b without pattern light, a brightness image storage unit 37c, a code image storage unit 37d, a code boundary coordinate storage unit 37e, and an ID storage unit 37f. Aberration correction coordinate storage unit 37g, 3D coordinate storage unit 37h, document orientation calculation result storage unit 37i, plane conversion result storage unit 37j, projected image storage unit 37k, and working area 37l are allocated as storage areas. ing.

The pattern light image storage unit 37a stores a pattern light image in which the pattern light is projected onto the document P by the pattern light photographing program 36b. The pattern lightless image storage unit 37b stores a pattern lightless image obtained by capturing a state in which the pattern light is not projected on the document P by the pattern light photographing program 36b.

The luminance image storage unit 37c stores the luminance image generated by the luminance image generation program 36c. The code image storage unit 37d stores the code image generated by the code image generation program 36d. The code boundary coordinate storage unit 37e stores the boundary coordinates of each code obtained by the subpixel accuracy extracted by the code boundary extraction program 36e. The ID storage unit 37f stores an ID or the like assigned to a luminance image having a change in brightness at a pixel position having a boundary. The aberration correction coordinate storage unit 37g stores the boundary coordinates of the code whose aberration has been corrected by the lens aberration correction program 36f. The three-dimensional shape coordinate storage unit 37h stores the three-dimensional coordinates in the real space calculated by the triangulation calculation program 36g.

The document posture calculation result storage unit 37i stores parameters related to the three-dimensional shape of the document P calculated by the document posture calculation program 36h. The plane conversion result storage unit 37j stores the plane conversion result generated by the plane conversion program 36i. The projected image storage unit 37k stores the image information projected from the image projection unit 13. The working area 37l stores data that is temporarily used for operations on the CPU 15.

FIG. 6 is a flowchart of the main process. Details of each of the digital camera processing (S605), webcam processing (S607), stereoscopic image processing (S607), and flattening image processing (S611) in this main processing will be described later.

In the main process, first, when the power is turned on (S601), the processor 15 and other interfaces are initialized (S602).

Then, a key scan for determining the state of the mode changeover switch 9 is performed (S603), it is determined whether or not the mode changeover switch 9 is set to the digital camera mode (S604), and if it is the digital camera mode (S604: Yes), Move to digital camera processing, which will be described later (S605).

On the other hand, if it is not the digital camera mode (S604: No), it is determined whether the mode changeover switch 9 is set to the webcam mode (S606), and if it is the webcam mode (S606: Yes), the process shifts to the webcam processing described later. (S607).

On the other hand, if it is not webcam mode (S605: No), it is determined whether the mode selector switch 9 is set to stereoscopic image mode (S608), and if it is stereoscopic image mode (S608: Yes), stereoscopic image processing described later. (S609).

On the other hand, if it is not the stereoscopic image mode (S608: No), it is determined whether the mode changeover switch 9 is set to the flattened image mode (S610), and if it is the flattened image mode (S610: Yes), it will be described later. Shift to flattened image processing (S611).

On the other hand, if it is not in the flattened image mode (S610: No), it is determined whether or not the mode selector switch 9 is in the off mode (S612), and if it is not in the off mode (S612: No), the processing from S603 is repeated. If it is in the off mode (S612: Yes), the process is terminated.

FIG. 7 is a flowchart of digital camera processing (S605 in FIG. 6). The digital camera process is a process of acquiring an image captured by the image capturing unit 14.

In this process, first, a high resolution setting signal is transmitted to CCD22 (S701). This makes it possible to provide the user with a high-quality captured image.

Next, the finder image (the image in the range visible through the finder 6) is displayed on the monitor LCD 10 (S702). Therefore, the user can confirm the captured image (imaging range) by the image displayed on the monitor LCD 10 before the actual imaging without looking into the finder 6.

Next, the release button 8 is scanned (S703a) to determine whether the release button 8 has been pressed halfway (S703b). If pressed halfway (S703b: Yes), it activates the autofocus (AF) and autoexposure (AE) functions and adjusts focus, aperture, and shutter speed (S703c). If it is not pressed halfway (S703b: No), the process from S703a is repeated.

Then, the release button 8 is scanned again (S703d) to determine whether the release button 8 has been fully pressed (S703e). If it is fully pressed (S703e: Yes), it is determined whether it is in flash mode (S704).

As a result, in flash mode (S704: Yes), flash 7 is projected (S705), shooting (S706), and not in flash mode (S704: No), without flash 7 being projected. Take a picture (S706). In the judgment of S703e, if it is not fully pressed (S703e: No), the process from S703a is repeated.

Next, the captured image captured is transferred from the CCD 22 to the cache memory 28 (S707), and the captured image stored in the cache memory 28 is displayed on the monitor LCD 10 (S708). In this way, by transferring the captured image to the cache memory 28, the captured image can be displayed on the monitor LCD 10 at a higher speed than when the captured image is transferred to the main memory. Then, the captured image is stored in the external memory 27 (S709).

Finally, it is determined whether or not there is a change in the mode changeover switch 9 (S710), if there is no change (S710: Yes), the processing from S702 is repeated, and if there is a change (S710: No), the relevant End the process.

FIG. 8 is a flowchart of webcam processing (S607 in FIG. 6). The webcam process is a process of transmitting an captured image (including a still image and a moving image) captured by the image capturing unit 14 to an external network. In this embodiment, it is assumed that a moving image is transmitted to an external network as an captured image.

In this process, the low resolution setting signal is sent to CCD22 (S801), the well-known autofocus and autoexposure functions are activated, the focus, aperture, and shutter speed are adjusted (S802), and then shooting is started. (S803).

Then, the captured image is displayed on the monitor LCD10 (S804), the finder image is stored in the projection image storage unit 37k (S805), the projection process described later is performed (S806), and the image is stored in the projection image storage unit 37k. The image is projected on the projection surface.

In addition, the captured image is transferred from the CCD 22 to the cache memory 28 (S807), and the captured image transferred to the cache memory 28 is transmitted to the external network via the RF interface (S808).

Finally, it is determined whether or not the mode changeover switch 9 has changed (S809), and if there is no change (S809: Yes), the processing from S802 is repeated, and if there is a change (S809: No), The process is terminated.

FIG. 9 is a flowchart of the projection process (S806 in FIG. 8). This process is a process of projecting an image stored in the shadow image storage unit 37k from the projection image projection unit 13 onto the projection surface. In this process, first, it is confirmed whether or not the image is stored in the projected image storage unit 37k (S901). If it is stored (S901: Yes), the image stored in the projection image storage unit 37k is transferred to the projection LCD driver 30 (S902), and the image signal corresponding to the image is projected from the projection LCD driver 30 on the projection LCD19. And display the image on the projection LCD (S903).

Next, the light source driver 29 is driven (S904), the LED array 17A is turned on by the electric signal from the light source driver 29 (S905), and the process is terminated.

In this way, when the LED array 17A is turned on, the light emitted from the LED array 17A reaches the projection LCD 19 via the light source lens 18, and the projection LCD 19 is spatially modulated according to the image signal transmitted from the projection LCD driver 30. It is applied and output as an image signal light. Then, the image signal light output from the projected LCD 19 is projected as a projected image on the projection surface via the projection optical system 20.

FIG. 10 is a flowchart of stereoscopic image processing (S609 in FIG. 6). The stereoscopic image processing is a process of detecting a three-dimensional shape of a subject and acquiring, displaying, and projecting a three-dimensional shape detection result image as the stereoscopic image.

In this process, first, a high resolution setting signal is transmitted to CCD22 (S1001), and a finder image is displayed on monitor LCD10 (S1002).

Next, the release button 8 is scanned (S1003a) to determine whether the release button 8 has been pressed halfway (S1003b). If pressed halfway (S1003b: Yes), it activates the autofocus (AF) and autoexposure (AE) functions and adjusts focus, aperture, and shutter speed (S1003c). If it is not pressed halfway (S1003b: No), the process from S1003a is repeated.

Then, the release button 8 is scanned again (S1003d) to determine whether the release button 8 is fully pressed (S1003e). If it is fully pressed (S1003e: Yes), it is determined whether it is in flash mode (S1003f).

As a result, if it is in flash mode (S1003f: Yes), flash 7 is projected (S1003g), it is shot (S1003h), and if it is not in flash mode (S1003f: No), flash 7 is not projected. Take a picture (S1003h). In the judgment of S1003e, if it is not fully pressed (S1003e: No), the process from S1003a is repeated.

Next, the three-dimensional shape detection process described later is performed to detect the three-dimensional shape of the subject (S1006).

Next, the 3D shape detection result in the 3D shape detection process (S1006) is stored in the external memory 27 (S1007), and the 3D shape detection result is displayed on the monitor LCD 10 (S1008). The three-dimensional shape detection result is displayed as an aggregate of three-dimensional coordinates (XYZ) in the real space of each measurement vertex.

Next, the 3D shape detection result image as a stereoscopic image (3D CG image) in which the measurement vertices as the 3D shape detection result are connected by polygons and the surface is displayed is stored in the projection image storage unit 37k (S1009). , Perform the same projection processing as the projection processing of S806 in FIG. 8 (S1010). In this case, the coordinates on the projected LCD19 with respect to the obtained three-dimensional coordinates can be obtained by using the inverse function of the equation for converting the coordinates on the projected LCD19 described in FIG. 18 into the three-dimensional space coordinates. , 3D shape result coordinates can be projected on the projection plane.

Then, it is determined whether or not there is a change in the mode changeover switch 9 (S1011), and if there is no change (S1011: Yes), the process from S702 is repeated, and if there is a change (S1011: No), the process is performed. finish.

FIG. 11 (a) is a diagram for explaining the principle of the spatial coding method used for detecting a three-dimensional shape in the above-mentioned three-dimensional shape detection process (S1006 in FIG. 10), and FIG. 11 (b) is a diagram for explaining the principle of the spatial coding method. It is a figure which shows the pattern light different from (a). Either of these (a) or (b) may be used for the pattern light, and further, a gray level code which is a multi-gradation code may be used.

For details on this spatial coding method, see Kosuke Sato and one other person, "Distance image input by spatial coding", IEICE Journal, 85/3 Vol.J 68-D No3 p369 ~ 375. It is disclosed.

The spatial code method is a type of method for detecting the three-dimensional shape of a subject based on triangulation between the projected light and the observed image. As shown in (a), the distance between the projected light source L and the observer O is It is characterized in that it is installed separated by D and the space is divided into elongated fan-shaped areas and coded.

When the three mask patterns A, B, and C in the figure are projected in order from the MSB, each fan-shaped area is coded as bright "1" and dark "0" by the mask. For example, the region including the point P is coded as 001 (A = 0, B = 0, C = 1) because the masks A and B are not exposed to light and the mask C is bright.

A code corresponding to the direction φ is assigned to each fan-shaped region, and each can be regarded as one slit ray. Therefore, the scene is photographed with a camera as an observation device for each mask, and the light and dark patterns are binarized to form each bit plane of the memory.

The position (address) in the horizontal direction of the obtained multi-bit plane image corresponds to the observation direction θ, and the contents of the memory at this address give the projected light code, that is, φ. The coordinates of the point of interest are determined from these θ and φ.

In addition, as the mask pattern used in this method, the case where a pure binary code such as mask patterns A, B, and C is used is shown in (a), but when the mask position shift occurs, the boundary of the region is shown. There is a risk of large errors.

For example, point Q in (a) indicates the boundary between region 3 (011) and region 4 (100), but if mask A 1 shifts, the code in region 7 (111) may occur. In other words, large errors can occur where the Hamming distance is 2 or more between adjacent regions.

Therefore, as the mask pattern used in this method, as shown in (b), by using a code in which the Hamming distance is always 1 between adjacent regions, it is possible to avoid the coding error as described above. It is said that.

FIG. 12A is a flowchart of the three-dimensional shape detection process (S1006 in FIG. 10). In this process, first, an imaging process is performed (S1210). In this imaging process, a striped pattern light (see FIG. 1) formed by alternately arranging light and dark is emitted from the image projection unit 13 using a plurality of pure binary code mask patterns shown in FIG. 11 (a). This is a process of sequentially projecting onto a subject and acquiring an image with pattern light that captures a state in which each pattern light is projected and an image without pattern light that captures a state in which pattern light is not projected.

When the imaging process is completed (S1210), the three-dimensional measurement process is performed (S1220). The three-dimensional measurement process is a process of actually measuring the three-dimensional shape of the subject by using the image with the pattern light and the image without the pattern light acquired by the imaging process. When the three-dimensional measurement process is completed (S1220) in this way, the process is completed.

FIG. 12B is a flowchart of the imaging process (S1210 in FIG. 12A). This process is executed based on the pattern light photographing program 36a. First, the image capturing unit 14 captures the subject without projecting the pattern light from the image projection unit 13, thereby acquiring a pattern light-free image (S1211). ). The acquired pattern lightless image is stored in the pattern lightless image storage unit 37b.

Next, the counter i is initialized (S1212), and it is determined whether or not the value of the counter i is the maximum value imax (S1213). The maximum value imax is determined by the number of mask patterns used. For example, when using 8 types of mask patterns, the maximum imax (= 8) is obtained.

Then, as a result of the judgment, if the value of the counter i is smaller than the maximum value imax (S1213: Yes), the mask pattern of the i number among the mask patterns to be used is displayed on the projection LCD19, and the mask pattern of the i number is displayed. The pattern light of No. i projected by is projected onto the projection surface (S1214), and the state in which the pattern light is projected is photographed by the image capturing unit 14 (S1215).

In this way, an image with pattern light that captures the state in which the pattern light of No. i is projected on the subject is acquired. The acquired image with pattern light is stored in the image storage unit 37a with pattern light.

When the shooting is finished, the projection of the pattern light of No. i is finished (S1216), "1" is added to the counter i to project the next pattern light (S1217), and the processing from S1213 is repeated.

Then, when it is determined that the value of the counter i is larger than the maximum value imax (S1213: No), the process is terminated. That is, in this imaging process, one image without pattern light and an image with maximum value imax of pattern light are acquired.

FIG. 12 (c) is a flowchart of the three-dimensional measurement process (S1220 in FIG. 12 (a)). This process is executed based on the luminance image generation program 36c, and first, a luminance image is generated (S1221). Here, the luminance is a Y value in the YCbCr space, and is a value calculated from Y = 0.2989, R + 0.5866, G + 0.1145, and B from the RGB values of each pixel. By obtaining the Y value for each pixel, a luminance image for each pattern with and without light is generated. The generated luminance image is stored in the luminance image storage unit 37c. In addition, a number corresponding to the pattern light number is assigned to each luminance image.

Next, the code image generation program 36d generates a coded image for each pixel by combining the generated luminance images by using the spatial code method described above (S1222).

This code image can be generated by comparing each pixel of the luminance image related to the image with patterned light stored in the luminance image storage unit 37c with a preset luminance threshold value and combining the results. The generated code image is stored in the code image storage unit 37d.

Next, the code boundary extraction program 36e performs a code boundary coordinate detection process (S1223) described later, and detects the boundary coordinates of the code assigned to each pixel with subpixel accuracy.

Next, the lens aberration correction program 36f performs lens aberration correction processing (S1224). By this processing, it is possible to correct the error of the code boundary coordinates detected by S1223, which includes an error due to the influence of distortion of the imaging optical system 21 and the like.

Next, the triangulation calculation program 36g is used to perform real-space conversion processing based on the triangulation principle (S1225). The code boundary coordinates in the CCD space after the aberration correction is performed by this processing are converted into the three-dimensional coordinates in the real space, and the three-dimensional coordinates as the three-dimensional shape detection result are obtained.

FIG. 13 is a diagram for explaining the outline of the code boundary coordinate detection process (S1223 in FIG. 12). In the upper figure, the boundary between light and dark of the actual pattern light in the CCD space is indicated by the boundary line K, the pattern light is coded by the above-mentioned spatial code method, and the boundary between the code 1 and the other code is shown by the thick line in the figure. It is a figure shown by.

That is, since the coding in the above-mentioned spatial coding method is performed for each pixel, an error in subpixel accuracy occurs between the actual pattern light boundary line K and the coded boundary line (thick line in the figure). Therefore, this code boundary coordinate detection process aims to detect the boundary coordinates of the code with subpixel accuracy.

In this process, first, at a certain detection position (hereinafter referred to as "curCCDX"), the first pixel G that changes from a certain code of interest (hereinafter referred to as "curCode") to another code is detected (first pixel detection step). ..

For example, in curCCDX, when each pixel is detected in order from the top, the pixels have curCode up to the boundary (thick line), but the curCode changes in the pixel next to the boundary, that is, the first pixel G. , This is detected as the first pixel G.

Next, at the pixel position of the first pixel G, all the luminance images having a change in brightness are extracted from the luminance images stored in the luminance image storage unit 37c in S1221 of FIG. 12 (luminance image extraction step). ).

Next, move the detection position to the left side of "2" to specify the pixel area to be used for approximation, and at the position of the detection position curCCDX-2, refer to the code image and start from the code of interest (curCode). Search for a pixel (boundary pixel (pixel H at the detection position of curCCDX-2)) that changes to the code of, and a predetermined range centered on that pixel (in the case of this embodiment, -3 pixels and +2 pixels in the Y-axis direction). (Range of) specifies the pixel range (part of the pixel area specifying means).

Next, within the predetermined range, as shown in the graph on the lower left side of the figure, an approximate expression (indicated by the solid line in the figure) relating to the pixel position and the brightness in the Y direction is obtained, and the approximate expression is used. Find the Y coordinate Y1 at the intersection with the luminance threshold bTh (part of the boundary coordinate detection process).

The brightness threshold value bTh may be calculated from a predetermined range (for example, half of the average brightness of each pixel), or may be a fixed value given in advance. As a result, the boundary between light and dark can be detected with subpixel accuracy.

Next, the detection position is moved from curCCDX-2 to the right side by "1", and the same processing as described above is performed in curCCDX-1 to obtain the representative value in curCCDX-1 (a part of the boundary coordinate detection process).

In this way, in the pixel area (see the downward-sloping shaded area in the figure) composed of a predetermined range in the Y-axis direction centered on the boundary pixel and a range from curCCDX-2 to curCCDX + 2 in the X-axis direction. Obtain a representative value at each detection position.

The processing so far is performed on all the luminance images having pixels that change from curCode to other codes, and the weighted average value of the representative value for each luminance image is finally adopted as the boundary coordinates in curCode (boundary coordinate detection step). Part of).

As a result, the boundary coordinates of the code can be detected with high accuracy and subpixel accuracy, and by using these boundary coordinates to perform the real space conversion process (S1225 in FIG. 12) based on the above-mentioned triangulation principle, the high The three-dimensional shape of the subject can be detected with high accuracy.

Further, since the boundary coordinates can be detected with subpixel accuracy by using the approximation formula calculated based on the luminance image in this way, the number of images to be imaged is not increased as in the conventional case, and it is pure binary. It may be a pattern light shaded with a code, and it is not necessary to use a gray code which is a special pattern light.

In this embodiment, the range of "-3" to "+2" in the Y-axis direction and the range of curCCDX-2 to curCCDX + 2 as the detection position in the X-axis direction at each detection position centering on the boundary pixel. The region composed of and has been described as a pixel region for obtaining an approximation, but the range of the Y-axis and the X-axis of this pixel region is not limited to these. For example, only a predetermined range in the Y-axis direction centered on the boundary pixel at the detection position of curCCDX may be set as the pixel area.

FIG. 14 is a flowchart of the code boundary coordinate detection process (S1223 in FIG. 12). This process is executed based on the code boundary extraction program 36e. First, each element of the code boundary coordinate sequence in the CCD space is initialized (S1401), and curCCDX is set as the start coordinate (S1402).

Next, it is determined whether curCCDX is below the end coordinates (S1403), and if it is below the end coordinates (S1403: Yes), curCode is set to "0" (S1404). That is, curCode is initially set to the minimum value.

Next, determine if the curCode is less than the maximum code (S1405). If the curCode is smaller than the maximum code (S1405: Yes), the code image is referenced in curCCDX to look for the curCode pixel (S1406) and determine if the curCode pixel exists (S1407).

As a result, if a pixel with curCode exists (S1407: Yes), in curCCDX, a pixel with a code larger than that curCode is searched for by referring to the code image (S1408), and a pixel with a curCode larger than that curCode is found. Determine if it exists (S1409).

As a result, if a pixel with a code larger than curCode exists (S1409: Yes), the boundary described later is obtained with subpixel accuracy (S1410). Then, in order to obtain the boundary coordinates for the next curCode, "1" is added to the curCode (S1411), and the process from S1405 is repeated.

That is, since the boundary exists at the pixel position of the pixel having the curCode or the pixel position of the pixel having the Code larger than the curCode, in this embodiment, the boundary is tentatively set to the pixel position of the pixel having the curCode larger than the curCode. The process proceeds on the assumption that it is in.

Also, if the curCode does not exist (S1407: No), or if there are no pixels with a code larger than the curCode (S1409: No), the curCode should be "" to find the boundary coordinates for the next curCode. 1 is added (S1411), and the process from S1405 is repeated.

In this way, for curCode from 0 to maximum code, the processing from S1405 to S1411 is repeated, and when curCode becomes larger than the maximum code (S1405: No), "dCCDX" is added to curCCDX to change the detection position (S1412). ), At the new detection position, the process from S1403 is repeated in the same manner as described above.

Then, the curCCDX is changed, and when the curCCDX finally becomes larger than the end coordinates (S1403), that is, when the detection from the start coordinate to the end coordinate is completed, the process is terminated.

FIG. 15 is a flowchart of a process (S1410 in FIG. 14) for obtaining the code boundary coordinates with subpixel accuracy.

In this process, first, among the luminance images stored in the luminance image storage unit 37c in S1221 of FIG. 12, the brightness changes at the pixel positions of the pixels having a code larger than the curCode detected in S1409 of FIG. Extract the entire luminance image with (S1501).

Then, the mask pattern number of the extracted luminance image is stored in the array PatID [], and the number of images of the extracted luminance image is stored in noPatID (S1502). The arrays PatID [] and noPatID are stored in the ID storage unit 37f.

Next, the counter i is initialized (S1503), and it is determined whether or not the value of the counter i is smaller than noPatID (S1504). As a result, if it is determined to be small (S1504: Yes), the CCDY value of the boundary is obtained for the luminance image having the mask pattern number of PatID [i] corresponding to the counter i, and the value is stored in fCCDY [i]. (S1505).

When the processing of S1505 is completed, "1" is added to the counter i (S1506), and the processing from S1504 is repeated. Then, when it is determined in S1504 that the value of the counter i is larger than noPatID (S1504: No), that is, when the processing of S1505 is completed for all the luminance images extracted by S1501, the fCCDY obtained by the processing of S1505 is completed. Calculate the weighted average value of [i] and use the result as the boundary value (S1507).

Instead of the weighted average value, the median value of fCCDY [i] obtained by the processing of S1505 can be calculated and the result can be used as the boundary value, or the boundary value can be calculated by statistical calculation.

That is, the boundary coordinates are represented by the coordinates of curCCDX and the weighted average value obtained by S1507, and the boundary coordinates are stored in the code boundary coordinate storage unit 37e to end the process.

FIG. 16 is a flowchart of a process (S1505 of FIG. 15) for obtaining the CCDY value of the boundary for the luminance image having the mask pattern number of PatID [i].

In this process, first, among "curCCDX-dx" and "0", the process represented by "ccdx = MAX (curCCDX-dx, 0)" that sets the larger value as ccdx is performed, and the counter j is set. Initialize (S1601).

Specifically, "0" in S1601 means the minimum value of the CCDX value. For example, the curCCDX value as the detection position is now "1" and the preset dx value is "2". If so, "curCCDX-dx" becomes "-1", which is smaller than the minimum value of "0", so the subsequent processing at "-1" is the processing to set "ccdx = 0". I do.

That is, for a position smaller than the minimum value of the CCDX value, a process of excluding the subsequent processes is performed.

The value of this "dx" can be set to an appropriate integer including "0" in advance. In the example described with reference to FIG. 13, this "dx" is set to "2", and FIG. 13 According to the example of, this ccdx will be set to "curCCDX-2".

Next, it is determined whether or not ccdx &lt;= MIN (curCCDX + dx, ccdW-1) (S1602). In other words, "MIN (curCCDX + dx, ccdW-1)" on the left side is smaller than "curCCDX + dx" and "ccdW-1", which is the maximum CCDX value "ccdW" minus "1". Since it means that it is a value, compare the magnitude of that value with the "ccdx" value.

That is, for a position larger than the maximum value of the CCDX value, a process of excluding the subsequent processes is performed.

Then, as a result of the judgment, if ccdx is smaller than MIN (curCCDX + dx, ccdW-1) (S1602: Yes), the existence of the boundary is referred to by referring to the code image and the luminance image to which PatID [i] is assigned. Find the eCCDY value of the pixel position of the pixel to be used (S1603).

For example, assuming that the detection position is curCCDX-1 shown in FIG. 13, pixel I is detected as a pixel candidate having a boundary, and the eCCDY value is obtained at the position of pixel I.

Next, from the luminance image having the mask pattern number of PatID [i], the luminance in the ccdy direction is in the range of MAX (eCCDY-dy, 0) &lt;= ccdy &lt;= MIN (eCCDY + dy-1, ccdH-1). Find the approximate polynomial Bt = fb (ccdy) for (S1604).

Next, the ccdy value at which the approximate polynomial Bt and the luminance threshold bTh intersect is obtained, and the value is stored in efCCDY [j] (S1605). With these S1604 and S1605, it is possible to detect the boundary coordinates with subpixel accuracy.

Next, "1" is added to each of ccdx and counter j (S1605), and the processing from S1602 is repeated. That is, the boundary of subpixel accuracy is detected at each detection position within a predetermined range on the left and right of the curCCDX.

Then, in S1602, when it is judged that "ccdx" is larger than "MIN (curCCDX + dx, ccdW-1)" (S1602: No), efCCDY [j calculated in the range from curCCDX-dx to curCCDX + dx ], The approximate polynomial of ccdy = fy (ccdx) is obtained (S1606). Since each value detected in S1605 by this process is used, the detection accuracy of the boundary coordinates can be improved as compared with the case where the boundary coordinates are detected at one detection position.

The intersection of the approximate polynomial obtained in this way and curCCDX is set as the CCDY value of the boundary for the luminance image having the mask pattern number of PatID [i] (S1607), and the processing is terminated. As shown in the flowchart of FIG. 15, the processing up to this point is executed on one sheet of all the extracted brightness images, the weighted average value is calculated for the obtained boundary coordinates, and the result is final. Since the boundary coordinates are used (S1507), the detection accuracy of the boundary coordinates can be further improved.

FIG. 17 is a diagram for explaining a lens aberration correction process (S1224 in FIG. 12). In the lens aberration correction processing, as shown in FIG. 17A, the incident light flux is deviated from the position to be imaged by the ideal lens due to the aberration of the imaging optical system 21, and the imaged pixels are subjected to the lens aberration correction processing. This is a process of correcting the position to the position where the image should be originally formed.

As shown in FIG. 17B, for example, this aberration correction is data obtained by calculating the aberration of the optical system in the imaging range of the imaging optical system 21 with the half-angle of view hfa, which is the angle of the incident light, as a parameter. Correct based on.

This aberration correction processing is executed based on the lens aberration correction program 36f, is performed on the code boundary coordinates stored in the code boundary coordinate storage unit 37e, and the data obtained by the aberration correction processing is stored in the aberration correction coordinate storage unit 37g. It is stored.

Specifically, the following camera calibration (approximate formula) (1) to (3) is used to convert arbitrary point coordinates (ccdx, ccdy) in the real image to coordinates (ccdcx, ccdcy) in the ideal camera image. To correct.

In this embodiment, the aberration amount dist (%) is described as dist = f (hfa) using the half angle of view hfa (deg). Further, the focal length of the imaging optical system 21 is focal length (mm), the ccd pixel length is pixellength (mm), and the center coordinates of the lens in the CCD 22 are (Centx, Centy).

(1) ccdcx = (ccdx-Centx) / (1 + dist / 100) + Centx (2) ccdcy = (ccdy-Centy) / (1 + dist / 100) + Centy (3) hfa = arctan [(((((() ccdx-Centx)² + (Ccdy-Centy)²)^0.5) × pixellength / focallength] Fig. 18 is a diagram for explaining the method of calculating the 3D coordinates in the 3D space from the coordinates in the CCD space in the real space conversion process based on the triangulation principle (S1225 in Fig. 12). is there.

In the real space conversion process based on the triangulation principle, the triangulation calculation program 36g calculates the three-dimensional coordinates in the three-dimensional space for the aberration-corrected code boundary coordinates stored in the aberration-corrected coordinate storage unit 37g. The three-dimensional coordinates calculated in this way are stored in the three-dimensional coordinate storage unit 37h.

In this embodiment, as the coordinate system of the image input / output device 1 for the laterally curved document P to be imaged, the optical axis direction of the image pickup optical system 21 is the Z axis, and the image pickup optical system 21 is along the Z axis. The origin is at a point VPZ away from the principal point position, the X-axis is in the horizontal direction and the Y-axis is in the vertical direction with respect to the image input / output device 1.

Further, the projection angle θp from the image projection unit 13 to the three-dimensional space (X, Y, Z), the distance between the optical axis of the imaging lens optical system 20 and the optical axis of the image projection unit 13 are D, and the imaging optical system 21. Let the field of view in the Y direction be from Yftop to Yfbottom, the field of view in the X direction be from Xfstart to Xfend, the length (height) of CCD22 in the Y-axis direction be Hc, and the length (width) in the X-axis direction be Wc. The projection angle θp is given based on the code assigned to each pixel.

In this case, the three-dimensional spatial position (X, Y, Z) corresponding to the arbitrary coordinates (ccdx, ccdy) of CCD22 intersects the point on the image plane of CCD22, the projection point of the pattern light, and the XY plane. It can be obtained by solving five equations for the triangle formed by points. (1) Y =-(tanθp) Z + PPZ + tanθp-D + cmp (Xtarget) (2) Y =-(Ytarget / VPZ) Z + Ytarget (3) X =-(Xtarget / VP) Z + Xtarget ( 4) Ytarget = Yftop-(ccdcy / Hc) × (Yftop-Yfbottom) (5) Xtarget = Xfstart + (ccdcx / Wc) × (Xfend-Xfstart) The cmp (Xtarget) in (1) is the imaging optical system 20. It is a function that corrects the deviation between the image projection unit 13 and the image projection unit 13, and can be regarded as cmp (Xtarget) = 0 in an ideal case where there is no deviation.

On the other hand, as described above, the relationship between the arbitrary coordinates (lcdcx, lcdcy) on the projection LCD 19 included in the image projection unit 13 and the three-dimensional coordinates (X, Y, Z) in the three-dimensional space is as follows (X, Y, Z). It can be expressed by the equations 1) to (4).

In this embodiment, the principal point position (0,0, PPZ) of the image projection unit 13, the Y-direction field of view of the image projection unit 13 from Ypftop to Ypfbottom, the X-direction field of view from Xpfstart to Xpfend, and Y of the projection LED19. Let the length (height) in the axial direction be Hp and the length (width) in the X-axis direction be Wp. (1) Y =-(Yptarget / PPZ) Z + Yptarget (2) X =-(Xptarget / PPZ) Z + Xptarget (3) Yptarget = Ypftop-(lcdcy / Hp) × (Xpftop-Xpfbottom) (4) Xptarget = Xpfstart + (lcdcx / Wp) × (Xpfend-Xpfstart) By using this relational expression, by giving the three-dimensional spatial coordinates (X, Y, Z) to the above equations (1) to (4), the LCD Spatial coordinates (lcdcx, lcdcy) can be calculated. Therefore, for example, it is possible to calculate an LCD element pattern for projecting an arbitrary shape and characters in a three-dimensional space.

FIG. 19 is a flowchart of flattened image processing (S611 in FIG. 6). The flattened image processing is, for example, a case where a curved document P as shown in FIG. 1 is imaged or a case where a rectangular document is imaged from an oblique direction (the captured image becomes trapezoidal). Is also a process of acquiring and displaying a flattened image in which the original is not curved or is flattened as if it was captured from a vertical direction.

In this process, first, a high resolution setting signal is transmitted to CCD22 (S1901), and a finder image is displayed on monitor LCD10 (S1902).

Next, the release button 8 is scanned (S1903a) to determine whether the release button 8 has been pressed halfway (S1903b). If pressed halfway (S1903b: Yes), it activates the autofocus (AF) and autoexposure (AE) functions and adjusts focus, aperture, and shutter speed (S1903c). If it is not pressed halfway (S1903b: No), the process from S1903a is repeated.

Then, the release button 8 is scanned again (S1903d) to determine whether the release button 8 has been fully pressed (S1903e). If it is fully pressed (S1903e: Yes), it is determined whether it is in flash mode (S1903f).

As a result, if it is in flash mode (S1903f: Yes), flash 7 is projected (S1903g), it is shot (S1903h), and if it is not in flash mode (S1903f: No), flash 7 is not projected. Take a picture (S1903h). In the judgment of S1903e, if it is not fully pressed (S1903e: No), the process from S1903a is repeated.

Next, the three-dimensional shape detection process, which is the same process as the three-dimensional shape detection process (S1006 in FIG. 10) described above, is performed to detect the three-dimensional shape of the subject (S1906).

Next, based on the three-dimensional shape detection result obtained by the three-dimensional shape detection process (S1906), the document posture calculation process for calculating the posture of the document P is performed (S1907). By this process, the position L, the angle θ, and the curvature φ (x) of the document P with respect to the image input device 1 are calculated as the posture parameters of the document P.

Next, based on the calculation result, a plane conversion process described later is performed (S1908), and even if the document P is curved, a flattened image flattened in a non-curved state is generated.

Next, the flattened image obtained by the plane change processing (S1908) is stored in the external memory 27 (S1909), and the flattened image is displayed on the monitor LCD 10 (S1910).

Then, it is determined whether or not there is a change in the mode changeover switch 9 (S1911), and as a result, if there is no change (S1911: Yes), the processing from S702 is repeated again, and if there is a change (S1911: No). ), End the process.

FIG. 20 is a diagram for explaining a document posture calculation process (S1907 in FIG. 19). As a precondition for a manuscript such as a book, it is assumed that the curvature of the manuscript P is uniform in the y direction. In this manuscript orientation calculation process, first, as shown in FIG. 20 (a), the points arranged in two columns at the three-dimensional space position are approximated by the regression curve from the coordinate data related to the code boundary stored in the three-dimensional coordinate storage unit 37h. Find the two curves.

For example, it can be obtained from the upper and lower quarters of the position information (the boundary between the code 63 and the code 64 and the boundary between the code 191 and the code 192) of the upper and lower quarters of the projected pattern light.

Assuming a straight line connecting the points where the positions of the two curves in the X-axis direction are "0", the point where this straight line intersects the Z-axis, that is, the point where the optical axis intersects the document P, is the point of the document P3. The dimensional space position (0,0, L) is defined, and the angle formed by this straight line with the XY plane is defined as the inclination θ around the X axis of the original P.

Next, as shown in FIG. 20 (b), the original P is rotationally transformed in the opposite direction by the inclination θ around the X axis obtained earlier, that is, the original P is parallel to the XY plane. Is assumed.

Then, as shown in FIG. 20 (c), the displacement in the Z-axis direction of the cross section of the document P in the XZ plane can be represented by the curvature φ (X) as a function of X. In this way, the position L, the angle θ, and the curvature φ (x) of the document P are calculated as the document posture parameters, and the process is completed.

FIG. 21 is a flowchart of the plane conversion process (S1908 in FIG. 19). In this process, first, the processing area of the process is allocated to the working area 37l of the RAM 37, and the variable of the counter b used for the process is set to the initial value (b = 0) (S2101).

Next, 4 of the pattern lightless image stored in the pattern lightless image storage unit 37b based on the position L of the document P, the inclination θ, and the curvature φ (x) based on the calculation result of the document orientation calculation program 36h. It is obtained by the inverse conversion of the curvature (a process equivalent to the "curvature process" described later) that moves each corner point by -L in the Z direction, rotates by -θ in the X-axis direction, and further makes it φ (x). A rectangular area formed by points (that is, a rectangular area in which the surface on which the characters of the original P are written is an image observed from a substantially orthogonal direction) is set, and the pixels included in this rectangular area are set. Find the number a (S2102).

Next, the coordinates on the pattern lightless image corresponding to each pixel constituting the set rectangular region are obtained, and the pixel information of each pixel of the flattened image is set from the pixel information around the coordinates.

That is, first, it is determined whether or not the counter b has reached the number of pixels a (S2103). If the counter b does not reach the number of pixels a (S2103: No), a curvature calculation process is performed to rotate and move one pixel constituting the rectangular region by curvature φ (x) around the Y axis (S2104). Tilt around the X-axis and move it by θ rotation (S2105), and shift it by the distance L in the Z-axis direction (S2106).

Next, the coordinates (ccdcx, ccdcy) on the CCD image captured by the ideal camera are obtained (S2107) from the obtained three-dimensional space position by the inverse function of the previous triangulation survey, and the imaging optical system used is used. According to the aberration characteristics of 20, the coordinates (ccdx, ccdy) on the CCD image taken by the actual camera are obtained by the inverse function of the previous camera calibration (S2108), and the pixels of the pattern lightless image corresponding to this position. The state of is obtained and stored in the working area 37l of RAM37 (S2109).

Then, "1" is added to the counter b in order to execute the above-mentioned processes S2103 to S2109 for the next pixel (S2110).

In this way, when the processing from S2104 to S2110 is repeated until the counter b reaches the number of pixels a (S2103: Yes), in S2101, the processing area allocated to the working area 37l to execute the processing is released (S2111). ), End the process.

FIG. 22 (a) is a diagram for explaining the outline of the bending process (S2104 in FIG. 21), and FIG. 22 (b) shows the manuscript P flattened by the plane conversion process (S1908 in FIG. 19). There is. The details of this bending process are disclosed in detail in the journal of the Institute of Electronics, Information and Communication Engineers DII Vol.J86-D2 No.3 p409 "Curved document photography with an eye scanner".

Curved Z = φ (x) is a method of least squares of a three-dimensional shape composed of the obtained code boundary coordinate sequence (real space) and a cross-sectional shape cut in a plane parallel to the XZ plane at an arbitrary Y value. It is expressed by an expression approximated by a polynomial.

When flattening a curved curved surface, as shown in (a), the flattened points corresponding to the points on Z = φ (x) are from Z = φ (0) to Z = φ (x). Will be associated by the length of the curve.

By the plane conversion process including such a bending process, for example, even when the document P in a curved state as shown in FIG. 1 is imaged, as shown in FIG. 22 (b), a flattened plane image is obtained. And the accuracy of OCR processing can be improved by using the flattened image in this way, so that the characters, figures, etc. described in the manuscript can be clearly recognized from the image. ..

FIG. 23 is a diagram for explaining the light source lens 60 of the second embodiment regarding the light source lens 18 of the first embodiment described above, and FIG. 23 is a side view showing the light source lens 60 of the second embodiment. , (B) are plan views showing the light source lens 60 of the second embodiment. The same members as described above are designated by the same reference numerals, and the description thereof will be omitted.

The light source lens 18 in the first embodiment is configured by arranging the lens portions 18a integrally on the base 18b from the convex aspherical shape corresponding to each LED 17, whereas in the second embodiment. The light source lens 50 is formed by separately forming a cannonball-shaped resin lens containing each of the LEDs 17.

In this way, by configuring the light source lens 50 including each LED 17 separately, the position of each LED 17 and the corresponding optical lens 50 is determined on a one-to-one basis, so that the relative positions are determined. The accuracy can be improved, and there is an effect that the light emission directions are aligned.

On the other hand, when the lens arrays are aligned on the substrate 16 together, the light emission directions are different due to the positioning error when each LED 17 is die-bonded and the difference in the linear expansion coefficient between the lens array and the substrate. There is a risk of becoming.

Therefore, the surface of the projection LCD 19 is irradiated with light in which the incident direction of the light from the LED 17 is aligned perpendicular to the surface of the projection LCD 19, and the light can pass uniformly through the diaphragm of the projection optical system 20. Illumination unevenness can be suppressed, and as a result, a high-quality image can be projected. The LED 17 included in the light source lens 50 is mounted on the substrate 16 via an electrode 51 composed of a lead and a reflector.

Further, on the outer peripheral surface of the light source lens 50 of the group 1 in the second embodiment, a frame-shaped elastic fixing member 52 that bundles the light source lenses 50 and regulates them in a predetermined direction is arranged. The fixing member 52 is made of a resin material such as rubber or plastic.

Since the light source lens 50 of the second embodiment is configured separately for each LED 17, the angle of the optical axis formed by the convex tip of each light source lens 50 is correctly aligned and faces the projection LCD 19. It is difficult to install it.

Therefore, the fixing member 52 surrounds a group of light source lenses 50, and the outer peripheral surfaces of the light source lenses 50 are brought into contact with each other so that the optical axes of the light source lenses 50 face the projected LCD 19 at the correct angle. By restricting the position of, it is possible to irradiate light from each light source lens 50 toward the projection LCD 19 substantially vertically. Therefore, the light aligned vertically to the surface of the projection LCD 19 is irradiated so that the light can pass through the diaphragm of the projection lens uniformly, so that the uneven illuminance of the projected image can be suppressed. Therefore, it is possible to project a higher quality image.

The fixing member 52 may have a rigidity predetermined to a predetermined size in advance, and is made of a material having an elastic force, and the position of each light source lens 50 is set to a predetermined position by the elastic force. It may be regulated to.

FIG. 24 is a diagram for explaining a second embodiment regarding the fixing member 52 that restricts the light source lens 50 to a predetermined position described with reference to FIG. 23, and FIG. 24A is a perspective view showing a state in which the light source lens 50 is fixed. And (b) is a partial cross-sectional view. The same members as described above are designated by the same reference numerals, and the description thereof will be omitted.

The fixing member 60 of the second embodiment is formed in a plate shape in which a through hole 60a having a conical cross-sectional view having a cross section along the outer peripheral surface of each light source lens 50 is bored. Each light source lens 50 is inserted into and fixed to each through hole 60a.

Further, an elastic urging plate 61 is interposed between the fixing member 60 and the substrate 16, and further, an electrode 51 is provided between the urging plate 61 and the lower surface of each light source lens 50. An elastic annular O-ring 62 is arranged so as to surround the ring.

The LED 17 included in the light source lens 50 is mounted on the substrate 16 via an electrode 51 penetrating a through hole formed in the urging plate 61 and the substrate 16.

According to the fixing member 60 described above, each light source lens 50 is fixed by penetrating through each through hole 60a having a cross section along the outer peripheral surface of the light source lens, so that it is more reliable than the fixing member 50 described above. The optical axis of the light source lens 50 can be fixed so as to face the projection LCD 19 at the correct angle.

Further, at the time of assembly, the LED 17 can be urged and fixed at the correct position by the urging force of the O-ring 62.

In addition, the impact force that may occur when the device 1 is transported can be absorbed by the elastic force of the O-ring 62, and the position of the light source lens 50 shifts due to the impact of the impact, resulting in the light source lens. It is possible to prevent the inconvenience of not being able to irradiate light vertically from 50 toward LCD 19.

In the above embodiment, the processing of S1211, S1215 of FIG. 12B corresponds to the imaging means according to claim 1, the imaging step according to claim 9, and the imaging step according to claim 10. The process of S1221 in FIG. 12C corresponds to the luminance image generation means according to claim 1, the luminance image generation step according to claim 9, and the luminance image generation step according to claim 10. The code image generation means according to claim 1, the code image generation step according to claim 9, and the code image generation step according to claim 10 correspond to the process of S1222 in FIG. 12 (c). The process of S1006 in FIG. 10 corresponds to the three-dimensional shape calculation means according to claim 1, the three-dimensional shape calculation step according to claim 9, and the three-dimensional shape calculation step according to claim 10.

The process of S1408 in FIG. 14 corresponds to the first pixel detection means according to claim 1, the first pixel detection step according to claim 9, and the first pixel detection step according to claim 10. The process of S1501 in FIG. 15 corresponds to the luminance image extraction means according to claim 1, the luminance image extraction step according to claim 9, and the luminance image extraction step according to claim 10. The pixel area specifying means according to claim 1, the pixel area specifying step according to claim 9, and the pixel area specifying step according to claim 10 correspond to some processes of S1603 and S1604 of FIG. The approximate expression calculation means according to claim 1, the approximate expression calculation step according to claim 9, and the approximate expression calculation step according to claim 10 correspond to a part of S1604 in FIG. The processing of S1505 and S1507 of FIG. 15 corresponds to the boundary coordinate calculation means according to claim 1, the boundary coordinate calculation step according to claim 9, and the boundary coordinate calculation step according to claim 10.

Although the present invention has been described above based on the above examples, the present invention is not limited to the above examples, and it is easily inferred that various improvements and modifications can be made without departing from the gist of the present invention. It can be done.

For example, in the above embodiment, the process of acquiring and displaying a flattened image as a flattened image mode has been described, but a well-known OCR function is installed and the flattened flat image is read by this OCR function. It may be configured as follows. In such a case, the text written on the document can be read with higher accuracy than when the document in a curved state is read by the OCR function.

Further, in S1501 of FIG. 15 in the above embodiment, a case where all the luminance images having a change in brightness are extracted and a provisional CCDY value is obtained for all of them has been described. It does not have to be, and if it is one or more, it is not limited to that number. Boundary coordinates can be obtained at high speed by reducing the number of sheets to be extracted.

Further, in S1507 of FIG. 15 in the above embodiment, fCCDY [i] is weighted averaged, and in S1607 of FIG. 16, each value is averaged using efCCDY [j] as an approximate polynomial. The averaging method is not limited to these, for example, a method of taking a simple average value of each value, a method of adopting the median value of each value, a method of calculating an approximate expression of each value, and an approximate expression thereof. A method of using the detection position in the above as boundary coordinates, a method of obtaining by statistical calculation, or the like may be used.

Further, for example, in the three-dimensional shape detection process in the flattened image mode in the above embodiment, in order to detect the three-dimensional shape of the original P, a striped pattern light formed by alternately arranging a plurality of types of light and dark is projected. Although the case has been described, the light for detecting the three-dimensional shape is not limited to the pattern light.

For example, as shown in FIG. 25, when the three-dimensional shape of a curved original is easily detected, two strip-shaped slit lights 70 and 71 may be projected from the image projection unit 13. In this case, the three-dimensional shape can be detected at high speed from only two captured images as compared with the case of projecting eight pattern lights.

<figref num="1">It is an external perspective view of an image input / output device.</figref><figref num="2">It is a figure which shows the internal structure of an image pickup head.</figref><figref num="3">(a) is an enlarged view of the image projection unit, (b) is a plan view of the light source lens, and (c) is a front view of the projection LCD 19.</figref><figref num="4">It is a figure for demonstrating the arrangement of the LED array.</figref><figref num="5">It is an electric block diagram of an image input / output device.</figref><figref num="6">It is a flowchart of a main process.</figref><figref num="7">It is a flowchart of digital camera processing.</figref><figref num="8">It is a flowchart of webcam processing.</figref><figref num="9">It is a flowchart of a projection process.</figref><figref num="10">It is a flowchart of stereoscopic image processing.</figref><figref num="11">(a) is a diagram for explaining the principle of the spatial code method, and (b) is a diagram showing a mask pattern (Gray code) different from (a).</figref><figref num="12">(a) is a flowchart of the three-dimensional shape detection process. (b) is a flowchart of the imaging process. (c) is a flowchart of the three-dimensional measurement process.</figref><figref num="13">It is a figure for demonstrating the outline of the code boundary coordinate detection process.</figref><figref num="14">It is a flowchart of code boundary coordinate detection processing.</figref><figref num="15">It is a flowchart of the process which obtains a code boundary coordinate with a subpixel accuracy.</figref><figref num="16">It is a flowchart of the process of obtaining the CCDY value of a boundary for a luminance image having a mask pattern number of PatID [i].</figref><figref num="17">It is a figure for demonstrating the lens aberration correction processing.</figref><figref num="18">It is a figure for demonstrating the method of calculating 3D coordinates in 3D space from coordinates in CCD space.</figref><figref num="19">It is a flowchart of flattened image processing.</figref><figref num="20">It is a figure for demonstrating the document posture calculation process.</figref><figref num="21">It is a flowchart of a plane conversion process.</figref><figref num="22">(a) is a diagram for explaining the outline of the curvature calculation process, and (b) is a diagram showing a planarized image flattened by the planar transformation process.</figref><figref num="23">(a) is a side view showing the light source lens 60 of the second embodiment, and (b) is a plan view showing the light source lens 60 of the second embodiment.</figref><figref num="24">(a) is a perspective view showing a state in which the light source lens 50 is fixed, and (b) is a partial cross-sectional view thereof.</figref><figref num="25">It is a figure which shows another example as a pattern light to be projected on a subject.</figref>

Code description

1 Image input / output device (including 3D shape detection device) 13 Image projection unit (projection means) 14 Image imaging unit (part of imaging means) 18,50 Light source lens (part of projection means) 19 Projection LCD (projection) Part of means) 20 Projection optical system (projection means) 21 Imaging optical system (part of imaging means) 22 CCD (part of imaging means)