Method and imaging apparatus for generating a dynamic range expanded video signal

This invention relates to an imaging apparatus whose dynamic range is expanded, a video camera including the same, and a method of generating an expanded dynamic range video signal.

An imaging apparatus is known for generating an expanded dynamic range video signal by combining video signals generated at substantially the same time with different exposure intervals. Such an imaging apparatus is disclosed in Japanese Patent Application Provisional Publication No. JP-A-07131718A. A video signal processing circuit is known, which has a generating circuit for Kantenkontrastierungs- or Edge enhancement signals that generates an edge enhancement signal from a video signal, and a gamma correction circuit for compensating a gamma of the video signal, wherein the edge enhancement signal is not subjected to the gamma correction and added to the gamma corrected video signal. Such a video signal processing circuit is disclosed in Japanese Patent Application Provisional Publication No. JP-A-63-209373.

EP-0 595 299A discloses an imaging apparatus and method wherein a short exposure signal and a long exposure signal are combined to produce a wide dynamic range signal.

The object of the present invention is to provide a better expanded dynamic range imaging device, a better video camera incorporating the same, and a better method of generating an expanded dynamic range video signal.

According to the present invention there is provided an imaging apparatus comprising:

Imaging means including a drive means for receiving an optical image and producing a first video signal having a first exposure interval and a second video signal having a second exposure interval substantially at the same time, the second exposure interval being shorter than the first exposure interval, the first one and second video signal a first or have second effective luminance detection area which are different but whose combination is a continuous luminance detection area,

Synchronizing means for synchronizing the first video signal with the second video signal all corresponding frames of the first and second video signals;

Mixing control signal generating means for generating a mixture control signal indicating a mixing ratio of the first and second video signals according to the first and second video signals; and

Video signal generating means for generating a composite video signal from the first and second video signals from the synchronizing means in accordance with the mixing control signal and levels the first and second video signals to have an expanded dynamic range such that the first effective luminance detection area is associated with the second effective luminance detection area, characterized by:

Edge enhancement signal generating means for generating an edge enhancement signal from the combined video signal;

Edge enhancement amount control means for controlling an amount of the edge enhancement signal in accordance with the mixture control signal; and

Adding means for adding the edge enhancement signal from the edge enhancement amount control means and the combined video signal and outputting the edge enhancement video signal.

The imaging device may further comprise a coring amount control signal generating circuit for generating a coring amount control signal according to the mixture control signal, and a coring circuit for performing a coring operation on the edge enhancement signal according to the coring amount control signal.

According to the present invention, there is also provided a method of generating a combined video signal from an optical image, comprising the steps of:

receiving the optical image and generating a first video signal having a first exposure interval and a second video signal having a second exposure interval at substantially the same time, wherein the second exposure interval is shorter than the first exposure interval, the first and second video signals are first and second effective, respectively Have luminance detection range which are different but whose combination is a continuous luminance detection range;

the first video signal is synchronized with the second video signal every corresponding frames of the first and second video signals;

a mixture control signal indicative of a mixing ratio of the first and second video signals is generated in accordance with the synchronized first and second video signals; and

from the synchronized first and second video signals according to the mixture control signal and levels of the first and second video signals, a combined video signal is generated to have an expanded detection area such that the first effective luminance detection area is connected to the second effective luminance detection area; characterized by the further steps of:

an edge enhancement signal is generated from the combined video signal;

in accordance with the mixture control signal, an amount of the edge enhancement signal is controlled; and

the amount-controlled edge enhancement signal and the combined video signal are added together and an edge-enhanced video signal is output.

The method of the invention may further comprise the steps of: generating a coring amount control signal in accordance with the mixture control signal; and a coring operation is performed on the edge enhancement signal in accordance with the coring amount control signal.

The features of the present invention will become more readily apparent from the following detailed description of exemplary embodiments and the accompanying drawings, in which:

Fig. 1 is a block diagram of an imaging apparatus of a first example;

Fig. 2 is a block diagram of the first example showing the structure of the synchronizing circuit shown in Fig. 1;

Figs. 3A to 3E are timing charts of the first example for illustrating the synchronizing operation;

Fig. 4 is a timing chart of the first example showing the exposure interval identification signal shown in Fig. 1;

Fig. 5 is a timing chart of the first example showing the gain control signal shown in Fig. 1;

Figs. 6A to 6C are graphic drawings of the first example showing the operations for setting the gain and combining;

Figs. 7A to 7C are graphic drawings of the first example showing another example of the operations for setting the gain and combining;

Fig. 8 is a block diagram of an imaging device of a first embodiment of the invention;

Figs. 9A and 9B are graphs used in explaining the operation of the first embodiment;

Fig. 10 is a block diagram of an imaging apparatus of a second embodiment of the invention;

Figs. 11A to 11D are graphs used in explaining the operation of the second embodiment;

Fig. 12 is a block diagram of an imaging apparatus of a third embodiment of the invention;

Fig. 13 is a block diagram of an imaging apparatus of a fourth embodiment of the invention;

Fig. 14 is a block diagram of an imaging apparatus of a second example;

Fig. 15 is a block diagram of an imaging apparatus of a third example;

Figs. 16 and 17 are graphs illustrating a dynamic range compression operation of the third example;

Fig. 18 is a block diagram of a nonlinear processing circuit of the third example; and

Figs. 19A to 19C are graphic drawings of the third example illustrating the non-linear processing.

The same or corresponding elements or parts are designated by the same reference numerals in all drawings.

(FIRST EXAMPLE)

Fig. 1 is a block diagram of an imaging apparatus described as a first example. An imaging means 1010 receives an optical image through a lens unit 10 thereon and alternately generates a long exposure video signal having a first exposure interval and an exposed video signal having a short interval with a second exposure interval substantially at the same time (slightly different timings, ie consecutive two frames) under the control of a drive unit 1020. The second exposure interval is shorter than the first exposure interval. The long exposure video signal and the short exposure video signal have first and second effective detection areas 11 and 12, respectively. A preprocessing circuit 1030 performs pre-processing on the long exposure video signal and short exposure video signal. The preprocessing circuit 1030 includes a CDS circuit (not shown) for suppressing noise components in the long exposure analog video signal and short exposure analog video signal from the correlation double sampling means 1010, an automatic gain control amplifier (not shown) for amplifying the long video signal Exposure and the short exposure video signal from the CDS circuit, wherein the gain is automatically controlled, a clamp circuit for clamping the output of the automatic gain control amplifier to feed it into the following a / d converter 1040. The A / D converter 1040 performs a / d conversion on the long exposure video signal and the short exposure video signal into a long exposure digital video signal and a short exposure digital video signal. The output 1041 of the a / d converter 1040 is provided to a synchronizing circuit 1050.

The synchronizing circuit 1050 synchronizes the digital long exposure digital video signal with the short exposure digital video signal and simultaneously outputs the digital long exposure digital video signal and the short exposure digital video signal with the set minute time difference set in parallel.

An exposure ratio detection circuit 1140 responsive to the drive circuit 1020 detects an exposure ratio between the first and second exposure intervals, and outputs a gain control signal 1141. A gain adjusting circuit 1150 adjusts a difference between gains of the first and second video signals from the synchronizing circuit 1050 according to the exposure ratio in the gain control signal 1141 from the exposure ratio detecting circuit 1140, ie, a gain of the short exposure video signal from the synchronizing circuit 1050 is set.

A video signal combining circuit 1080 includes a mixture control signal generating circuit 1083 for generating a mixture control signal indicating a mixing ratio k between the long exposure video signal and the short exposure video signals, and combining the long exposure signal from the synchronizing circuit 1050 with the short exposure video signal from gain adjusting circuit 1150 according to the gain setting circuit 1150 according to FIG. 7B, and levels of the long exposure video signal and the short exposure video signal having the gain set to have an expanded detection area, so that the first effective detection area 11 is connected to the second effective detection area 12. The combined video signal 1080 shows linearity because the gain of the short exposure video signal 1070 is set.

The combined video signal 1081 is supplied to a gamma adjustment circuit 1090 and an edge contrast enhancement signal generation circuit 1100. The gamma adjustment circuit 1090 adjusts the gamma of the combined video signal 1081. The edge enhancement signal generation circuit 1100 generates an edge enhancement signal from the composite video signal 1081 and supplies the edge enhancement signal to a coring circuit 1110. The coring circuit 1110 removes noise components whose levels are less than a predetermined level and provides the edge enhancement signal to a multiplexer 1120. The multiplexer 1120 multiplies the edge enhancement signal by an edge enhancement control signal and provides the edge enhancement signal to an adder 1130. The adder 1130 adds the edge enhancement signal from the multiplexer 1120 to the gamma adjusted video signal 1091 to generate an output video signal 1131. The lens unit 10 is further provided for the imaging device to provide a video camera.

Fig. 2 is a block diagram of the first example showing the structure of the synchronizing circuit shown in Fig. 1; The synchronizing circuit 1050 includes a memory for storing the output of the a / d converter 1041, a selector 10513 for outputting either the output 1041 of the a / d converter or the output 10512 of the memory 10511 according to a signal 1021 for identifying exposure intervals of the Drive circuit 1020 to selectively output the video signal 1060 with long exposure, and a selector 10514 for outputting either the output 1041 of the a / d converter or the output 10512 of the memory 10511 according to the exposure interval identification signal 1021 from the drive circuit 1020 to selectively output the short exposure video signal 1060.

FIGS. 3A to 3E are timing charts for illustrating the synchronizing operation by the synchronizing circuit 1050.

The imaging means 1010 alternately outputs the long exposure video signal and the short exposure video signal as shown in FIG. 3A as the output 1041 of the a / d converter 1040. The memory 10511 outputs the output 1041 of the a / d converter 1040 with a delay of one frame. Therefore, a frame of the short exposure video signal on the line 10515 is synchronized with the corresponding frame of the long exposure video signal from the memory 10511. For the next frame interval, the long exposure video signal on line 10515 is synchronized with the corresponding frame of the short exposure video signal from memory 10511. This operation is repeated as shown in Figs. 3A and 3B. The exposure interval identification signal 1021 changes its output level between "2" and "64" every frame (field) as shown in Fig. 3C. The selector 10513 performs the switching operation to switch, as shown in FIG. 3E, only the continuous exposure video signal 1060 is shown continuously. The selector 10514 performs the switching operation to continuously output only the short exposure video signal 1070 as shown in FIG. 3D. A frame of the short exposure video signal from the selector 10514 is synchronized with the corresponding frame of the long exposure video signal 1060 from the selector 10513 as shown in Figs. 3D and 3E.

Fig. 4 is a timing chart showing the exposure interval identification signal 1021. Figs. The drive circuit 1020 generates the exposure interval identification signal 1021 which alternately shows a high level value "64" for the long exposure interval and a low level value "2" for the short exposure interval, as shown in FIG.

FIG. 5 is a timing chart showing the gain control signal 1141. The exposure ratio judging circuit 1140 sets the exposure ratio "32" from the high level value "64" for the long exposure interval (frame) and the low level value "2" for the short exposure interval (frame) as shown in FIG.

Figs. 6A to 6C are graphic drawings showing the operations for setting the gain and combining.

As shown in Fig. 6A, a level of the long exposure video signal 1060 increases with the amount of received light up to a saturation level (SAT) at a saturation amount. After the saturation amount (larger), the level of the long exposure video signal 1060 is constant. The long exposure video signal saturates with a relatively small amount of light because the exposure interval is relatively long. At the black level range, on the other hand, the noise level is relatively low. Therefore, the long exposure video signal has the first effective detection area 11.

In the short exposure video signal 1070 as shown in Fig. 6B, a level of the short exposure video signal 1070 increases with the amount of received light to the saturation level at a low γ. The short exposure video signal saturates with a relatively high amount of light because the exposure interval is relatively short. On the other hand, in the black level range, the noise level is relatively high. Therefore, the short exposure video signal has the second effective detection area 12.

As shown in FIG. 6C, the long exposure video signal 1060 is combined with the short exposure video signal 1070 to provide an expanded detection area 13.

Figs. 7A to 7C are graphic drawings showing another example of the operations for setting the gain and combining.

7B shows the mixing control signal 1082. The mixing control signal 1082 represents the mixing ratio k = 0 at the long exposure region 15, the mixing ratio k = 1 at the short exposure region 17, and the mixing ratio at the mixing region 16 increases from k = 0 to k = 1 proportional to.

The long-exposure video signal is modified by a ratio (1-k) at the mixing area 16, and the short-exposure video signal is modified at the mixing area 16 by a mixing ratio k, as shown in Fig. 7A. The combining circuit 1080 combines the short exposure video signal from the gain adjustment circuit 1150 with the long exposure video signal 1060 by adding the modified long exposure video signal to the short exposure video signal as shown in FIG. 7A and 7C to provide the combined video signal with an expanded detection area 13 such that the first effective detection area 11 is connected to the second effective detection area 12.

In this example device, the gain of the short exposure video signal is adjusted by the gain adjustment circuit 1150. However, it is also possible to adjust the gain of the long exposure video signal to match its gamma to that of the short exposure video signal.

(FIRST EMBODIMENT)

Fig. 8 is a block diagram of an imaging apparatus of a first embodiment of the present invention.

The imaging device of the first embodiment has substantially the same structure as that of the first example. The difference is that the gain adjustment circuit 1150 and the exposure ratio detection circuit 1140 are omitted, and further, a flank gain amount control signal generating circuit 1160 and a multiplier 1170 are provided.

Figs. 9A and 9B are graphic drawings of the first embodiment. Fig. 9A shows the mixing control signal which is also shown in Fig. 7A. 9B shows a control signal 1161 for the edge enhancement amount.

The edge enhancement amount control circuit 1160 generates the edge enhancement amount control signal 1161 in accordance with the mixture control signal 1082 as shown in Figs. 9A and 9B.

The multiplier 1170 controls the amount of the edge enhancement signal 1101 according to the edge enhancement amount control signal 1161, and supplies the edge enhancement signal subjected to the edge enhancement amount control to the coring processing circuit 1110. The edge enhancement amount control signal 1161 in the long exposure region 15 indicates a coefficient "1" for the multiplier 1170 and, for the short exposure region 17, for example, a coefficient "2". Therefore, the edge enhancement signal is controlled by the multiplier 1170 to have a larger edge enhancement signal at the short exposure region 17.

In this embodiment, the gain adjustment circuit 1150 and the exposure ratio detection circuit 1140 are omitted. However, it is also possible that these circuits are further provided for the imaging device of the second embodiment similarly to the first example.

SECOND EMBODIMENT

Fig. 10 is a block diagram of an imaging apparatus of a second embodiment.

The imaging device of the second embodiment has substantially the same structure as that of the first embodiment. The difference is that further a coring amount control circuit 1180 and a multiplier 1190 are provided and a coring amount can be controlled, ie a coring circuit 1112 is provided.

Figs. 11A to 11D are graphic drawings of the second embodiment, in which Fig. 11A shows the mixing control signal 1082, which is also shown in Fig. 7A.

The coring amount control circuit 1180 generates the coring amount control signal 1181 according to the mixing control signal 1082.

The multiplier 1190 multiplies the coring amount control signal 1181 by a coefficient, and supplies and a final coring amount control signal to the coring circuit 1112. Therefore, the coring circuit 1112 controls noise components in the edge enhancement signal:

Fig. 11C shows the case that the coring amount is "1", and Fig. 11D shows the case that the coring amount is "1.5". The noise components in the edge-contrast enhanced video signal at the short exposure area 17, which is conspicuous in the reproduced image, are then suppressed, so that the noise in the output video signal is improved.

In this embodiment, the gain adjustment circuit 1150 and the exposure ratio detection circuit 1140 are omitted. However, it is also possible that these circuits similar to the first example are further provided for the imaging device of the third embodiment.

(THIRD EMBODIMENT)

Fig. 12 is a block diagram of an imaging apparatus of a third embodiment.

The imaging device of the third embodiment has substantially the same structure as that of the first embodiment. The difference is that the exposure ratio detection circuit 1140 and a multiplier 1200 are further provided.

The multiplier 1200 controls the edge enhancement amount control signal 1161 in accordance with the gain control signal 1141 indicating the exposure ratio between the long exposure interval and the short exposure interval. The total amount of the edge enhancement signal is controlled, ie weighted, by the multiplier 1170 according to the edge enhancement amount control signal derived from the composite signal 1082 and the exposure ratio derived amplification control signal 1141. The edges at the short exposure area, which is rather flat because of the short exposure, can then be further improved.

In this embodiment, the gain adjustment circuit 1150 is omitted. However, it is also possible that, similarly to the first example, the gain adjusting circuit 1150 is provided for the imaging device of the third embodiment. In addition, the edge enhancement amount control circuit 1160 as the modification of this embodiment can be omitted.

FOURTH EMBODIMENT

Fig. 13 is a block diagram of an imaging apparatus of a fourth embodiment.

The imaging device of the fourth embodiment has substantially the same structure as that of the third embodiment. The difference is that there is further provided a multiplier 1210 and the coring amount control circuit 1180 and the multiplier 1190 used in the second embodiment.

The coring amount control circuit 1180 generates the coring amount control signal 1181 according to the mixture control signal 1082 as mentioned in the second embodiment. The multiplier 1210 controls the correction amount control signal 1181 according to the gain control signal 1141 indicating the exposure ratio between the long exposure interval and the short exposure interval. The second correction amount control signal 1211 is controlled, ie, weighted, by the multiplier 1210 in accordance with the correction amount control signal 1181 derived from the mixture control signal 1082 and the exposure ratio derived gain control signal 1141. Noise components in the edge enhancement signal from the multiplier 1170 are canceled by the coring circuit 1112.

The flanks at the short exposure area, which is rather flat because of the short exposure, are further improved, and the noise component coring at the short exposure area, which is rather flat because of the short exposure, may continue by providing the flank gain control circuit 1160 be improved. In addition, the noise component in the edge-contrasted or Edge amplified video signal at the area 17 with short exposure, which is obvious in the reproduced image, suppressed, so that the noise in the output video signal is improved.

In this embodiment, the gain adjustment circuit 1150 is omitted. However, it is also possible that, similarly to the second embodiment, the gain adjusting circuit 1150 is provided in the imaging device of the fourth embodiment. In addition, as a modification in this embodiment, it is also possible that either the edge enhancement amount control circuit 1160 and the multiplier 1170 and 1200 or the coring amount control circuit 1180 and the multipliers 1190 and 1210 are omitted.

(SECOND EXAMPLE)

Fig. 14 is a block diagram of an imaging apparatus of a second example.

The imaging device of the second example has substantially the same structure as that of the first example. The difference is that there is further provided a prism unit 1000 for separating an incident image into color images, ie a red image, a green image and a blue image, and the imaging means 1010 provide the red image, the green image, and the like. receive a blue image and the processing circuits each including the preprocessing circuit 1030, the A / D converter 1040, the synchronizing circuit 1050, the gain adjusting circuit 1110, the combining circuit 1080 for the red, green and blue images, respectively, and a camera processing circuit 1090 is provided for processing the respective color video signal. For color separation, 1000 units of dichroic mirrors may be used instead of the prism unit.

The imaging means 1010 of each color receives an optical image from the prism 1000 and generates at substantially the same time a first video signal having a first exposure interval and a second video signal having a second exposure interval, the second exposure interval being shorter than the first exposure interval and the first and second Video signal have a first and second effective detection area 11 and 12 respectively. The sync circuit 1050 for each color synchronizes the first video signal with the second video signal. The exposure interval ratio detection circuit 1100 responsive to the drive circuit 1020 detects an exposure ratio between the first and second exposure intervals. The gain adjustment circuit 1110 for each color adjusts the gain of the second video signal from the synchronizing circuit 1050 according to the exposure ratio. The mixture control signal generating circuit 1083 for each color generates a mixture control signal indicating a mixing ratio of the first and second video signals. The combining circuit 1080 for each color generates a combined video signal from the first and second video signals according to the mixing control signal and levels of the first and second video signals to have an expanded detection area such that the first effective detection area is connected to the second effective detection area in the first embodiment has been described.

The camera processing circuit 1090 processes the red, green and blue color combined video signals 1081 to produce a red output signal, a green output signal and a blue output signal.

The first to fourth embodiments are usable for the imaging device of the second example.

(THIRD EXAMPLE)

Fig. 15 is a block diagram of an imaging apparatus of a third example.

The imaging device of the third example has substantially the same structure as that of the second example. The difference is that there is further provided a maximum detection circuit 1130, a non-linear processing circuit 1120, and a display dynamic range setting circuit 1140.

The display dynamic range setting circuit 1140 generates or receives and supplies data 1141 of a display dynamic range (DISPMAX). The maximum detection circuit 1130 detects the maximum level among the third red, third green and third blue video signals for one frame period. The non-linear processing circuit 1120 responsive to the display dynamic range (DISPMAX) data 1141 generates and outputs red, green and blue display signals having non-linear characteristics, so that the maximum level becomes equal to or less than the data of the display dynamic range if the detected maximum level is greater than the data dynamic range data, and outputs the combined red, green, and blue video signals as they are, if the detected maximum level is greater than the data of the display dynamic range.

If the dynamic range of a display device receiving the output red, blue and green signals from this imaging device is less than the dynamic range of the red, blue and green signals being output, it is necessary to compress the dynamic range of the red, blue and green signals that are output ,

The non-linear processing circuit 1120 compresses the dynamic range of the combined video signal to a display video signal according to the detected maximum high light level among the third red, third green and third blue video signals for a frame period, saturation data and data (DISPMAX) 1141 for setting a display dynamic range by an internally dividing operation.

Figs. 16 and 17 are graphic drawings of the third example illustrating the operation of compressing the dynamic range.

It is assumed that the display dynamic range (DISPMAX) 1141 is 20 and the maximum value (RGBMAX) is 25. The characteristic (in terms of gradations characteristic) of the combined video signals 1081 is bent above the saturation level (SAT). In fact, the level of the compressed-range video signal is provided by internally dividing the level of the combined video signal by using the maximum (RGBMAX), the saturation level (SAT), and the data (DISPMAX) 1041 to set the display dynamic range.

The maximum detection circuit 1130 detects a maximum level among the combined video signals 1081 of red, green and blue for one frame, and supplies the maximum level (RGBMAX) 1131 to the non-linear processing circuit 1120. The non-linear processing circuit 1120 generates the compressed-area video data according to the detected maximum value, the saturation data, and the display area (DISPMAX) 1141.

As shown in Fig. 16, a level A of the combined video signal is shifted to A ", and a level S is shifted to S" (= RGBMAX = DISPMAX) to prevent saturation. However, if the level A is smaller than the saturation level (SAT), the non-linear processing is not performed, ie A = A ".

More specifically, if the level A is greater than the saturation level (SAT), assuming that n is a control coefficient to control the nonlinear characteristic curve, the level A "after nonlinear processing is given as follows:

If RGBMAX - DISPMAX ≥ 0 and A ≥ SAT,

αA = (A - SAT) / (RGBMAX - SAT)

A "= A - (αA) n · (RGBMAX - DISPMAX)

If RGBMAX - DISPMAX ≥ 0 and A <SAT,

αA = 0

A "= 0

If RGBMAX - DISPMAX <0,

αA = 0

A "= A.

Fig. 17 shows the curves for providing the non-linear characteristic when the control coefficient n is changed from one to three.

Fig. 18 is a block diagram of the non-linear processing circuit 1120 of the third example.

The nonlinear processing circuit 1120 includes a judging circuit 20, subtractors 21 to 24, multipliers 25 and 26, a divider 27, switches 28 and 29, and generates in response to the combined video signal 1081, the maximum value 1131, the saturation data 1082, and a display Maximum value 1141 is a nonlinear video signal 1121. The control coefficient n for the non-linear characteristic is determined by the number of multipliers 25 and connections around the multipliers 25.

The display dynamic range (DISPMAX) 1141 may be generated by a ROM or RAM that stores the display dynamic range (DISPMAX) 1141, set manually, or sent from the display device to be connected. In addition, the saturation value SAT can be reduced by an operator as desired.

Figs. 19A to 19C are graphic drawings of the third example illustrating the non-linear processing.

It is assumed that the display dynamic range (DISPMAX) 1141 = 20, the maximum level among the red, green and blue video signals (RGBMAX) = 25 and the saturation level (SAT) of the long exposure video signal = 3 and that the red , green and blue video signals each show level R: G: B = 20: 5: 25. The level of the red video signal is shifted from 20 to 17 by the non-linear processing, the level of the green video signal remains about 5 because the original level "5" is near the cut-off point "3", which is the saturation level, and the level of the blue one Video signal is shifted from 25 to 20 by non-linear processing. That is, the output levels of the non-linear circuits 1020 are then R: G: B = 17: 5: 20, so that saturation in the display device to be connected to this imaging device prevents and the ratio of levels of the red, green and blue video signals, ie, a hue , is essentially maintained.

The first to fourth embodiments are usable for the imaging device of the third example.