Image shake detecting device

This invention relates to a device for detecting image shake which is particularly suitable for an automatic image stabilizing device of a video camera or the like and also to a device for controlling the image stabilizing device.

In optical image pickup devices of various types including video cameras and the like. Not only does image blur prevent easy image recognition, it also degrades recognition accuracy, whether used in the industrial or consumer markets. For example, in the case of the video camera, the camera is often operated while the operator is moving or while the camera is on a moving vehicle. In such a case, it is inevitable to get a shake-type image scan according to the photographing circumstances or the subject to be photographed.

To solve this problem, a device for detecting image shake according to various methods has been proposed. According to one of these methods, the movement of the camera is physically detected by means of an acceleration sensor (an angular velocity sensor), and an optical system is corrected for the movement according to the direction and the amount of movement. In another conceivable method, the parallel motion amount of the entire picture is detected by a video signal and expressed in a motion vector. Then the optical system is compensated based on the vector.

According to the method of using the acceleration sensor, the size of the device becomes bulky and therefore requires much space and weight. Incidentally, this leads to a complex structure arrangement. Consequently, this method is hardly suitable for a home video camera, such as a built-in camera VCR, which needs to be compact in size and weight.

With regard to the method of calculating and generating the motion vector of the image plane from the video signal, some camera movement deliberately caused by the user is mistaken for image shake. This device would also respond to a movement of the object, which is in fact no image blur. This process is therefore fraught with serious problems.

The above-mentioned known image shake detection apparatuses include, for example, an image stabilization camera disclosed in Japanese Laid-Open Patent Application JP-A-61248681. The camera of this kind is designed as follows: an optical image is converted into an electric signal by means of an image conversion system consisting of a lens system and a photoelectric conversion element. A television picture signal is obtained by signal processing operation performed in a predetermined manner with respect to the electrical signal by a signal processing circuit. The thus obtained image signal is sent both to a device for detecting image shake and to a monitor. From the image signal, a correlation is detected between two image planes obtained in a given time interval to find out the magnitude and direction of any image jitter. Then, a drive control circuit and a motor for controlling and moving the objective system are operated to set this on the basis of the result of the detection of image shake. The camera is thus set up to allow a stable picture, even if it shakes.

However, the image flicker detection device thus conceived is unable to detect a difference between the movement of an object which occurs only in a part of the image plane and a jitter of the entire image plane. To solve this problem, the image jitter detection sensitivity of the device must be designed to be variable for different areas of the image plane.

In connection with this problem, there has been proposed a device for detecting image blur which has been proposed in an article entitled "About an Image Plane Shake Compensating Device" which is described in "The Technical Report" of the Televison Society, vol. 11, no , 3, pages 43 to 48, PPOE, 87-12 (May 1987). In this device, the entire image plane is divided into 140 area blocks. A jitter detecting switch is arbitrarily turned on or off for each of these areas, and the image jitter is detected only from the area for which the jitter detecting switch is turned on in accordance with a dot pitch matching method.

In realizing this means for detecting image shake, however, an image as a reference must be temporarily stored in an image memory whose changed density value is kept intact. To cope with this requirement, the device requires the use of an analog-to-digital converter (hereinafter referred to as A / D converter) and a memory of relatively large memory space. In addition to this deficiency, the device is designed to have one image plane superimposed on another and offset from another by a certain amount of the vector to find a vector that gives the greatest degree of match. Consequently, the operation of the device involves a large amount of calculation. The device thus requires large scale circuitry and requires a long computation time.

Incidentally, it is extremely difficult to incorporate a jitter detecting device in a compact video camera capable of real-time processing and which has to be housed in a compact circuit because the aforementioned device is known in the art In the art, the application of extensive circuitry including the A / D converter, frame buffer, arithmetic circuitry, etc. and a long processing time is needed.

A control system which is capable of instantaneously effecting the image jitter compensation by driving a photo-taking optical system based on the information on a detected jitter by the aforementioned image jitter detection means is further constructed as follows, and various methods for correcting image shift are known , One of these various methods which has been recently developed is equipped with optical compensating members such as a lens whose optical axis is changeable by a motor and a variable vertex angle prism. With the image obtained by an optical system including these optical compensators, the degree of the deflection and the size of the image to be compensated are determined on the basis of the image offset from the image. Then, the so-called feedback control is performed to deflect the image by driving the aforesaid optical compensators according to the information of the result of the determination.

In the device of this type, a variable vertex prism is used which is designed with a variable vertex angle as the optical image steering means. A motor is used to drive the prism. The degree of deflection of the image is determined in the following manner: the images of at least two television camera image planes having a time difference with each other are compared with each other. The deflection of the variable vertex angle prism is then controlled in such a way that there is a reduction in the offset that occurs between the compared images, so that the images are always achieved in a consistent state. When the object to be photographed is traced or when the object is moving, the variable vertex angle prism is driven so as to reduce the degree of positional change of the image of the object on the image plane. If the image fluctuates by panning the camera or for some reason, the image is stabilized by driving the variable vertex angle prism in the same manner as in the case of tracking the object to reduce the degree of positional change of the image of the object. The control is thus actually carried out in the same way.

For the television camera, an interlaced scanning method or the like is generally used. used. If the television camera uses the NTSC system, an image plane is generated and transmitted in each period of 1/60 sec (one field period). When the feedback control method is used to detect the state of the image produced by such a television camera, the date of the feedback system consists of samples scattered at intervals of at least 1/60 sec. Incidentally, the feedback action delays by one period of the time required for the detection operation plus a period required for the image plane transmission (about 1/60 sec). The control output of the device is therefore unstable with any rapid movement of the object. The appeal of the facility is poor. When the motor used to drive the optical compensator has a good starting characteristic, a feedback coefficient exceeding unity tends to oscillate. Another disadvantage of the prior art device is a poor frequency response. These problems stem from the fact that the control is performed using the control algorithm of a continuous time system while the sampling by detecting means has a dispersive time delay.

Furthermore, there has been proposed an apparatus having means for detecting image shake for objects appearing in an image plane and arranged for tracking a moving object and continuously executing the control for accurate automatic focusing and accurate automatic exposure control. However, as already mentioned, it is difficult to exactly make a difference between the movement of only one object in a part of the image plane and the movement of the entire image planes due to the movement of the camera. According to the aforementioned method disclosed in Japanese Laid-Open Patent Application JP-A-61248681, the degree of accuracy will be lower when there is no illumination difference because the method uses a difference in luminance between the background and the object to be photographed. According to the technique disclosed in the aforementioned Television Society Technical Report, it is difficult to pinpoint the areas in which a plurality of objects are in motion. In other words, all the prior art methods nact have both advantages and benefits. Consequently, with the prior art image-shake detection apparatus which is in a compact video camera, one is unable to adequately cope with each image condition.

With regard to the device for detecting image blur of this type, further examples are given in U.S. Pat. 5.- Patent US-A-4788596. This patent application discloses an image stabilizing camera designed to compensate the optical axis of the objective by detecting image blurring of the edge component of the image of an object to be photographed; and an arrangement for recognizing an image by a histogram representing the distribution of the feature of the image.

Furthermore, document US-A-4 410 914 discloses a television picture stabilization system which detects and measures a discontinuity of the television picture. Camera movement references in the form of a successor window for reference selection of the vertical and horizontal stabilization are provided during two successive partial images of the camera image, each of which consists of a horizontal or vertical edge of an object. The movement of the selected camera movement references is continuously measured from field to field. The horizontal position of the vertical tracking window along the vertical reference edge and the vertical position of the horizontal tracking window along the vertical reference edge, each responsive to side-by-side movement to be measured and measured up and down motion, are stored separately in memory means. Based on a detected motion, vertical and horizontal corrections of the television picture are made to eliminate the detected discontinuity, and the respective position of the vertical and horizontal follow-up windows is automatically changed. The starting position of the horizontal and vertical tracking windows is selected by an operator by positioning joystick-controlled cursors over the desired reference edges.

This invention is intended to solve the aforementioned problems of the prior art.

Accordingly, it is an object of the present invention to provide a jitter detection apparatus which is designed, on the one hand, to measure any deviations occurring at a given feature point within each image plane during the process of scanning image planes, and which are temporarily different from each other, and, on the other hand, to produce the magnitude of the dithering of an image on the basis of the value so obtained.

It is a further object of the invention to provide an image-shake detection apparatus and a compensation device which has the detection sensitivity in a variable manner for the magnitude of the jitter, and which is capable of shaking the entire image plane from one locally to distinguish occurring tremors on the image plane exactly, so that the device always compensates for any image blur unerringly.

It is still another object of the invention indicate a device for detecting image blur, which allows processing almost in real time, if an amount of image jitter can be calculated briefly during a vertical blanking period, and it also allows the size, to reduce weight and costs because a simple circuit arrangement of these in principle the need to use an A / D converter, the storage and the computing means bypasses on a large scale.

It is an object of the invention to provide a control device for an optical image sensing system which is very stable and exhibits good response.

Another object of the invention is to provide a control unit for an optical image sensing system, that is able to perform an image stabilization and an object style action, to effectively compensate for any image offsets, the AU, resulting in a movement of the object to be photographed, and which is advantageously applicable not only to an optical photo-taking system, but also for image processing systems of various types considering the following: in terms of a sample period for scattered information and the time delay of the information detected, that exists, if the controlled system the Bildabtastsystem a television camera or the like. , wherein the amount of control operation for a feedback control system is calculated and provided to the controlled system. This stabilizes the control output as far as possible and improves the response and the frequency response. With the controlled system stabilized in this manner, the adjustment range of the feedback coefficients for the system can be extended.

It is an object of the invention to provide a compact control device which requires neither special sensors nor special optical parts.

According to the present invention, this object is achieved by a device for detecting image shakes, which detects an image shake of an image scanning plane on the basis of an image signal output from an image scanning means, with: reading means for converting the image signal into a digital signal and for reading out a distribution of a level of the digital signal, whereby a predetermined feature information of the image is generated from the image scanning plane; Detecting means for detecting the image movement by calculating the difference of the generation time of the predetermined feature information among a plurality of sub-images; and computing means for calculating a magnitude and a direction of the jitter of the image based on an output of the detecting means; which is characterized in that the determining means calculates the difference by detecting the difference of a position of the predetermined feature information among the plurality of fields based on clock signals; and sensitivity control means are provided for varying a sensitivity of the detection means; wherein the sensitivity control means is designed to control a sensitivity of the detection means by changing the resolving power of the arithmetic operation of the detection means and wherein the read-out means reads out the predetermined feature information from a signal pattern, which is generated by converting a luminance component of the image signal coming from the image scanning means into a digital value.

Further objects and features of the invention will become apparent from the following description of embodiments thereof in conjunction with the accompanying drawings. In the drawing mean:

FIG. 1 Fig. 10 is a block diagram showing an image shake detection apparatus according to a first explanatory example, shown as an application to a camera-mounted type VTR. FIG. 2 Fig. 10 is a block diagram showing the internal structure of the image shake detection circuit of the first illustrative example. FIG. 3 Fig. 10 is a block diagram showing an edge detection circuit included in the image shake detection circuit. Figs. 4 (a) and 4 (b) illustrate a jitter direction area provided on an image scanning plane. FIG. 5 is a flowchart showing the algorithm of the processes of the image jitter detection and compensation.

Fig. 6 is a flow chart showing, as a second explanatory example, the algorithm of image squint detecting and compensating operations.

Fig. 7 is a flowchart showing, as a third explanatory example, the algorithm of image detection and compensation operations. Fig. 8 illustrates a method of detecting a jitter that has occurred in an image.

FIG. 9 Fig. 10 is a block diagram showing an image shake detection device provided as a first embodiment of the invention. The illustrated device is applied to a camera of image stabilization type. Figs. 10 (a) to 10 (c) and 11 (a) to 11 (c) are timing charts showing the arrangement and operation of a gate pulse generating circuit included in the image jitter detection circuit of the invention. Figs. 12 (a) to 12 (f) show a timing chart of a method of detecting image shake in the vertical direction. Figs. 13 (a) and 13 (b) show an arrangement for detecting a region in which movement occurs in an image scanning plane and the control over the jitter detection sensitivity.

FIG. 14 Fig. 10 is a block diagram showing a controller of an image pickup optical system included according to the invention in a second embodiment thereof. FIG. 15 FIG. 3 is a block diagram illustrating a significant portion of a map shown in FIG. 14 shown second embodiment. FIG. 16 Fig. 10 is a graph showing the following property of the same controller. FIG. 17 Fig. 4 is a graph showing the result of a simulation of the drag action of the control device of the present invention obtained with the device herein applied to a feedback control system.

Fig. 18 is a block diagram showing a controller provided according to this invention as a third embodiment thereof. Fig. 19 is a block diagram showing the control means provided in a fourth embodiment of the invention shown as an application in a mounting robot.

Fig. 20 is a block diagram showing the arrangement of a video camera to which the invention is applied as a fifth embodiment thereof. Figs. 21 (a) to 21 (d) and Figs. 22 (a) to 22 (c) illustrate procedures to be taken by the fifth embodiment for range determination.

Fig. 23 is a block diagram showing an automatic focusing device provided according to the invention as a sixth embodiment thereof.

Fig. 24 is an illustration of an operation for determining a distance measuring range according to an optical process. Fig. 25 is an illustration of the characteristic of the optical process. Fig. 26 is a diagram of an operation decided to decide a restart by detecting a movement of an object in the distance measuring area.

Hereinafter, means for detecting image shake embodying the present invention will be described with reference to the accompanying drawings.

Fig. 1 shows a video camera to which the image shake detection means according to a first explanatory example is applied. The first explanatory example is characterized by the following: a camera pan is never misinterpreted as a movement of an object to be photographed. A movement of the camera is never misunderstood as a movement of the object. The device thus effectively allows a camera tremble to compensate.

In the case of Fig. 1 For example, the device applied to a built-in camera-type video recorder is shown whose arrangement is shown in a schematic block diagram. In Fig. 1 For example, a variable vertex angle prism 1 is formed by filling the space between two parallel flat glass plates with a liquid and sealing, and the space is filled with something like a rubber bellows. The vertex angle of the variable vertex angle prism 1 is thus set to be variable for the compensation of image jitter. The dithering motion of an optical image can be compensated in any direction, whereby the optical axis of the prism 1 can be tilted up, down, left or right by applying a load to a predetermined peripheral portion of the variable apex prism 1 by means of Prism drive part 8. A taking lens 11 forms an optical pickup system (or image pickup system) in conjunction with the variable apex prism 1. An image sensor 2 composed of a photoelectric conversion element such as a CCD or the like is provided for converting an image of the object on the photoelectric conversion planes, from which an electric signal corresponding to the image of the object is generated. A camera signal processing circuit 3 is composed of an amplifier circuit, a matrix circuit, a gamma correction circuit, an encoder, etc., and is configured to convert a signal received from the image sensor 2 into a luminance signal and a color carrier signal. A VTR signal processing circuit 4 is provided for frequency modulation of the luminance signal, conversion to a lower frequency band of the color carrier signal, and then output of these processed signals in a multiplexed state. A magnetic head 5 is ready to receive a video signal on a magnetic tape 6 which is output from the VTR signal processing circuit 4. A clock pulse generating circuit 7 is provided for generating a clock pulse for driving the image sensor 2 and also for supplying an image jitter detecting circuit 9 having various timing signals. The image jitter detecting circuit 9 is provided for detecting the image jitter and supplying a jitter compensation signal to the prism driving part 8 which will be described later. A keypad 10 is provided with various operation switches including a switch for selecting a jitter compensation mode which will be described later.

The switches provided in the operating part 10 also contain a switch SW1, used to select ON or OFF of a picture jitter detection and compensation operation, and a switch SW2, which is provided for selecting one of the following operating modes: an operating mode 1, in which a picture jitter detection area A1 is selected; a mode 2, in which a detection range A2 is selected; a mode 3, in which the detection range A1 or A2 is set automatically; an operating mode 4, in which the detection range A1 and A2 is set automatically, and the ON and OFF setting of the image jitter detection action is automatically set.

The image shake detection means according to this illustrated example works to detect image shake suitably for the state of an image of an object according to a principle described below. 1;

As shown in Figs. 4 (a) and 4 (b), the image shake detection means according to the illustrated example is capable of detecting an image shift occurring either in the detection area A1 which is in a detection area A or the detection area A2 is located, which frames the image A.

FIG. Fig. 4 (a) shows the case where an image is picked up by an object which is in size in the detection area A1 in the central part of the image scanning plane 100 with the background fixed. In this case, the image offset, ie a picture jitter detected by a video signal which is the image of the fixed background in the peripheral detection area A2, neglecting the object allowed to move freely and all around the central part of the image scanning plane 100. *** "

In the case of Fig. 4 (b), the camera follows an object so as to always keep it in the central part of the image scanning plane 100 while the article is moving at a relatively high speed. In this case, the moving object is fixed almost in the middle detection area A1 relative to the image scanning plane 100. In the meantime, the background of the subject will become fluid. Consequently, the jitter of the image is detected by a video signal according to the image generated in the middle detection area A1. However, if the image of the entire image sensing plane 100 is flowing, the image jitter detection action is not performed.

The operating part 10 allows the operator to set the camera in a desired mode by manually operating the mode selecting switches SW1 and SW2. However, it is also possible to automatically execute the operation mode setting operation, as will be described later.

FIG. 2 shows an internal structure of the image jitter detecting circuit 9. In Fig. 2 An A / D converting circuit 91 for converting the luminance signal into a digital signal coming from the camera signal processing circuit 3 is arranged. In this case, the A / D conversion need not be performed with a high degree of accuracy to reproduce the luminance signal without deterioration. It can be carried out almost with a degree of patterning, as far as it allows the determination of the image movement. In other words, the A / D conversion can be performed at a relatively low sampling frequency with a relatively low quantization number or bit number. An edge detecting circuit 92i5t for detecting an edge portion of the image displayed on the image scanning plane 100 is provided. The edge detection circuit 92 is composed, for example, of a delay circuit 92a having a delay time corresponding to a plurality of samples generated by the A / D conversion circuit 91, and a subtraction circuit 92b as shown in FIG. 3 is shown. A delayed luminance signal, generated by delaying an input luminance signal, is subtracted from the input luminance signal which has been converted to a digital signal. This arrangement allows the edge detecting circuit 92 to adequately recognize any edge portion of the image of the object that suddenly changes. This edge detection circuit 92 is provided for automatically selecting either the detection area A1 or the detection area A2, as will be described later. The method used for the edge detection circuit 92 may be replaced by any other method as long as a high-frequency component or the degree of sharpness of the image can be expressed by this other method. A gate 93 operates under the control of the control signal coming from the control microcomputer 97. The gate circuit 93 is thus designed to selectively pass either a video signal belonging to the center detection section A1 of the image sensing plane 100 or a video signal belonging to the peripheral detection section A2 of the image sensing plane 100. An integrating circuit 94 is designed to integrate the video (light) signal that has passed through the gate 93 for one field period. A memory 95 is designed to be able to store a sub-period portion of the output signal of the A / D converter circuit 91. A memory scan judgment 96 is designed to generate a read / write clock pulse and an address for writing or reading from the memory 95 under the control of the control microcomputer 97.

The image jitter detecting circuit 9 in Fig. 8, which uses the video signal, performs a jitter detection operation in the manner described below.

In Fig. 8th Reference 20 denotes an image scanning plane. Reference numerals 21 and 22 denote points of objects which are generated at certain times. A reference symbol H denotes a histogram obtained in the horizontal direction from the level of the A / D converted luminance signal which is generated at the same time. A symbol V means a histogram obtained in the vertical direction from the level of the A / D converted luminance signal which is also generated at the same time. Further, reference numerals 21 'and 22' denote the locations of the objects 21 and 22 which are generated after a predetermined period of time has elapsed. The reference symbols H 'and V' represent histograms respectively obtained on the basis of the luminance signal level distribution, then in the horizontal and vertical directions. The positions of the center points SH, SV, SH 'and SV' in the horizontal and vertical directions are respectively obtained from the histograms at different times. Then, motion vectors {GH} and {GV} are calculated based on the shift of the center of gravity that takes place as the passage of time progresses. Then a vector {G} is calculated as follows: vector {G} = {GH} + {GV}. This composite vector {G} represents the motion of the entire image plane, showing the direction and amount of jitter that has occurred.

The operation of the image-shake detection device is as follows. An incident light representing an image to be photographed is brought to the image plane of the image sensor 2 via the variable apex prism 1. The image is photoelectrically converted and delivered as a video signal. The video signal thus output from the image sensor 2 is processed by the camera signal processing circuit 3 and by the VTR signal processing circuit 4. The processed signal is supplied to the magnetic head 5 to be picked up on the magnetic tape 6. Further, the luminance signal output from the camera signal processing circuit 3 is supplied to the image jitter detecting circuit 9. In the image jitter detecting circuit 9 which is in the image jitter detecting circuit 9 in the FIG. 2 is constructed as shown, the video signal is sampled in a predetermined cycle by means of the A / D converter circuit 91 to be converted into a digital signal. The digital signal thus obtained is supplied to the edge detecting circuit 92 to provide an edge portion of the image, ie, a portion of sudden change detected by the edge detecting circuit 92. In the meantime, the digital signal is also supplied to the memory 95 to obtain a partial image of the image data stored therein.

An edge signal output from the detection circuit 92 is supplied to the gate 93. Then, the control microcomputer 97 extracts only a portion of the edge signal representing either the direction of the detection range Al or the detection range A2, which varies according to the state of the one shown in FIG. 4 (a) or 4 (b). The thus extracted signal is integrated for the duration of one field. The integrated signal is then provided to the control microcomputer 97. Meanwhile, the data obtained by A / D conversion of an image plane portion of the luminance signal is stored in the memory 95. Then, the thus stored information is supplied to the control microcomputer 97.

The control microcomputer 97 selects the detection area A1 or the detection area A2 by controlling the gate 93. The integrated edge values ​​of the detection ranges are compared with each other to determine which one of them has a larger integrated value. The luminance signal datum corresponding to the thus-detected area stored in the memory 95 is then read out from the memory 95 to obtain a histogram of the horizontal and vertical directions in the foregoing FIG. 8th to execute described manner. The center of gravity of the image generated at the image scanning level is calculated and extracted from the histograms. The center of gravity thus determined is then compared with another center of gravity obtained from the image on the image scanning plane after a predetermined period of time has elapsed. Thereby, by computation, a motion vector {G} indicating the magnitude and direction of an image offset, that is, an image shake which is obtained by calculation during the course of time, is obtained. Then, the date of this vector {G} is supplied to the prism driving part 8. According to this date, the prism driving part 8 changes the vertex angle of the variable vertex angle prism 1 in such a manner that the amount and the direction of the image offset are compensated. Such compensation can be effected for image shake or skew.

FIG. 5 FIG. 12 shows an algorithm to be executed when the operator operates the operation part 10 of the image shake detection device to manually select the shake detection area A1 or A2 (selection between modes 1 and 2) while monitoring the image scanning plane. As mentioned above, the control panel 10 includes, among other things, the switch SWI provided for initiating an automatic jitter detection and compensation operation at the beginning or to come to an end; and the mode selection switch SW2 which enables the manual selection of either the detection range A1 or the detection range A2.

In Fig. 5 First, the switch SW1 is checked in step S1 if it has been turned on. If not, the processing proceeds to step S2. In step 82, the image jitter detection circuit 9 is turned off, and then the processing returns to step S1. If the switch SW1 is found turned on in step S1, the processing proceeds to step S3. Step S3: The switch SW2 checks whether the central detection area A1 or the peripheral detection area A2 of the image scanning plane 100 is selected. If the peripheral detection area A2 has been found to be selected, the processing comes to step S4. Step 84: The control microcomputer 97 reads out from the memory 95 the luminance signal data of the detection area A2 shown in FIG. 4 (a) to prepare the horizontal and vertical histograms, as shown in FIG. 8th shown, and calculates the priorities SH2 and SV2. Thereafter, the processing proceeds to step 85. Step S5: The centers of gravity SH2 and SV2 are compared by calculation with emphases "SH2 and SV2" generated a predetermined period before to find out any difference. A vector representing any change in center of gravity is analyzed. The processing then goes to step 86. Step 86: Image jitter compensation data obtained by analyzing the vector is supplied to the prism driving section 8. Step S7: The variable vertex angle prism 1 is driven to change the optical axis thereof in the direction of offset of the image shift by controlling the vertical angle of the variable vertex angle prism 1 according to the supplied data. These steps are suitable for the case where an image is captured by resizing a main object within a central portion of the image sensing plane with background objects. An image swing can be detected with respect to the background, and the variable apex prism 1 can compensate for this image jitter.

After completion of the compensation, the processing proceeds to step 88. Step S8: The previously stored

Center of gravity values ​​"SH2 and SV2" are then replaced by the horizontal and vertical center of gravity values ​​SH2 and SV2 to store the values ​​SH2 and SV2 until a next determination. The processing then returns to step S1.

When the center detection area A1 of the image scanning plane is judged to be selected at step 83, indicating that the object is moving at a relatively high speed, as shown in FIG. 4 (b), and that the camera follows the object by holding the image approximately in the middle detection section A1, the processing goes to step 89 to detect image jitter with respect to the object. Step 89: The control microcomputer 97 reads out from the memory 95 the luminance signal data for the detection area A2 shown in FIG. 4 (b) is shown. Histograms for the horizontal and vertical directions are shown in FIG. 8th prepared way. The focal points SH1 and SV1 of the image are generated from this histogram. Step S10: As with steps 85 to S7, the center of gravity values ​​SH1 and SV1 are then compared with previous center of gravity values ​​"SH1 and SV1" generated, for example, for an immediately preceding field. Step S: The information thus obtained regarding the size and direction of the image jitter is supplied to the prism driving part 8. Step S12: The optical axis of the variable apex angle prism 1 is adjusted to compensate for image jitter. Thereafter, the processing goes to step S13. Step 813: The center-of-gravity values ​​SH1 and SV1 of the current image plane are stored instead of the previous center-of-gravity values ​​"SH1 and SV1", and then the processing returns to step S1.

With these repeated steps, the image jitter compensation can be accomplished with respect to the detection range manually selected by the operator.

While the jitter detection area A1 or A2 has been manually selected by the user in the case of the above-described operation, the detection area can be automatically selected. The explanatory example automatically selects, in accordance with the state of the subject to be photographed, the detection range for detecting the amount of image jitter with the edge detection circuit 92, the gate 93, and the integrator 94 which takes place in the image scanning plane.

In Fig. 6, which is a flow chart, the aforementioned explanatory example operates as follows. In order to automatically set the shake detection area A1 or A2 of the image scanning plane, the switches SW1 and SW2 of the operating section 10 are operated to select the mode 3 in which the jitter detection and the detection range can be performed automatically.

In step S21 of FIG. 6, in which the control microcomputer is provided in the image jitter detecting circuit 9, controls and causes the gate 93 to pass only the signal of the detection area A2 which is outside the detection area A of the image scanning plane 100. As a result, the edge information of the detection area A2 is supplied from the edge detection circuit 92 to the integrating circuit 94. Then, an edge integration value generated by the integration action of the integrating circuit 94 is supplied to the control microcomputer 97 and stored as a data F2. The operation then proceeds to step S22. Step 22: The gate 93 is controlled and caused to pass signals of the detection area A1 which are within the image-pick-up frame A. Then, an edge integration value of the detection area A1 is also supplied from the integrating circuit 94 to the microcomputer 97 as the data F1. Step S23: The edge integration values ​​of the detection areas A1 and A2, which are inside and outside the lock-up image A, are compared with each other by a data calculation operation F2 "- F1" to find out whether the result of the calculation is not less than zero. In other words, a check is made to find out which of the two areas has a larger part of the image that is efficiently adapted for image jitter detection. If the detection ranges A1 and A2 differ in size, the edge integration values ​​thereof can not be compared on a similar basis. Consequently, in such a case, a normalization or weighting process is suitable for carrying out the different size between the two detection areas.

If the result of the comparison between the data F1 and F2 is judged to be F2-F1 ≥ 0 in step S23, it indicates that the edge integration value of the detection range A2 is either greater than or equal to the detection period A1; the processing then proceeds to step S24. In step S24, the detection range A2 is selected. Then, image data corresponding to the detection area A2 and stored in the memory 95 is taken out and supplied to the control microcomputer 97. Thereafter, steps S24 to S28 are executed in the same manner as in the case of steps S24 to S28 of the flowchart of FIG. 5, to obtain the center of gravity values ​​SH2 and SV2 of the image coming from the detection area A2; to compare the degree of jitter by comparing the values ​​SH2 and AV2 with previously stored values ​​SH2 "and SV2"; to generate a motion vector and to provide the information of the amount and the direction of the jitter to the prism driving part 8. The prism driving part 8 then changes the vertex angle of the variable vertex angle prism 1 for compensation and adjustment of the optical axis thereof in the direction of compensation of the image in which the jitter has occurred. Upon completion of the variable peak angle prism 1 compensation, the centroid sH2 ​​and SV2 are again stored to be ready for the next detection operation.

If the result of the calculation on the data F1 and F2 shows that F2-F1 <0 in step S23, indicating that the edge integration value of the detection area A2 is smaller than that of the detection area A1, the flow comes to step S29. In step S29, the center detection area A1 is selected. Image data corresponding to the detection area A1 and stored in the memory 95 is used by the control microcomputer 97. Then, the steps S29 to S33 are repeated in the same manner as in the steps S9 to S13 of the flowchart of FIG. 5 executed. In other words, the amount of image jitter is detected with respect to the center detection area A1. The variable vertex angle prism 1 is driven to compensate for its optical axis for image shake. The illustrative example is thus designed to compare the edge integration values ​​of the detection areas A1 and A2 with each other to automatically select one of those areas which requires a larger amount of edge components in determining the amount of image coverage and is capable of limiting the scope of the Image shake under a condition that ensures a high detection accuracy.

In the case of Fig. 4 (a), a main object to be photographed moves in the middle part of the image scanning plane against a stationary background. In such case, the high frequency component and the edge component of the picture signal are mostly extracted from the background. The background in this case gives a larger edge integration value. FIG. Fig. 4 (b) shows another case where the object is held against a moving background in the center detection area. In this case, the edge integration value of the center detection area becomes larger. The illustrative example always selects the detection range having a larger edge integration value than the other, so that image shake can be accurately detected and compensated for with a high degree of reliability.

FIG. 7 shows a flowchart of a third illustrative example. The explanatory example is shown in an operating state 4. The operation state 4 is designed so that, in addition to the automatic adjustment processing for setting an image jitter detection region, the illustrative example automatically makes a decision as to whether the image jitter detection action is to be performed. In this case, the switch SW2 is operated to select the automatic setting mode of the detection ranges A1 and A2 and to automatically start the image jitter detection process.

In step 841 of FIG. 7 For example, the control microcomputer 97 accommodated in the image jitter detection circuit 9 will first control to cause the gate 93 to provide a gate for the detection area A2. Then, the edge integration value of the detection range A2 is taken in as the data F2. Thereafter, the processing proceeds to a step S42. Step S42: The edge integration value data F2 is checked if it is larger than a predetermined level value THI. If so, the flow of operation comes to steps S43 to S47 to perform an image jitter detection operation for the detection area A2; and the variable vertex angle prism 1 is driven so as to compensate for its optical axis for image shake. The steps S43 to S47 are carried out in exactly the same manner as the steps S24 to S28, and hence a description will be omitted in the following description.

If it is found in step S42 that the edge integration value F2 is not greater than the level value THI, the flow comes to step S48. Step S48: The control microcomputer 97 controls and causes the gate circuit 93 to provide a gate for the detection area A1. Then, the edge integration value of the detection area A1 is taken in as the date F1. The processing then proceeds to step S49. Step S49: The edge integration value Fl is compared with a predetermined level value TH2. If it is found that the value F1 is greater than the value TH2, the processing executes the steps S51 to S55. In steps S51 to S55, the image jitter detection operation with respect to the detection area A1 is performed. The variable vertex angle prism 1 is driven to compensate for its optical axis from image jitter. These steps S51 to S55 are performed in exactly the same way as the steps S29 to S33 of the flowchart of FIG. 6 Therefore, details will be omitted from the following description.

If, in step S49, the result of the check is that F2 ≦ TH2, the processing proceeds to step S50 to return to step S41 without performing image jitter detection and compensation action.

In other words, the image jitter detection action is performed with respect to the detection area A2 when the edge integration value obtained from the detection area A2 is sufficient for the image jitter detection; or with respect to the center detection area A1, when the information regarding the image obtained in the detection area A2 is judged to be insufficient. Further, if the image jitter detection is not possible or reliable, since the edge integration values ​​obtained for both the detection area A1 and A2 are insufficient, the explanatory example can be controlled and made to perform no image jitter detection action. For example, in the case of a pivoting movement of the camera or the like. For example, the image of the overall image scanning plane is fluent or moving, including both the background and the object image in the center part of the image scanning plane. Then, both the high frequency component and the edge component become lower, and the edge integration value also drops. In this case, the image jitter detection and compensation actions are disabled.

Further, the jitter detection and compensation action is also not performed when jitter compensation is not required, when there is almost no image on the image scanning plane, or when there is a featureless, flat image because the edge integration value also decreases in such a case.

The illustrative example is capable of reliably performing the image jitter detection and compensation actions in all situations. Incidentally, the mode of operation of the illustrative example is selectable either manually or automatically according to the circumstances of the subject.

As described above, the image shake detection means for detecting image shake and compensation for shaking is designed according to the circumstances of the object with respect to either the image of the subject or that of the background, which is more suitable for jitter detection. Consequently, the apparatus is capable of always making the most appropriate compensation for jitter in all cases, even in the case where an image of a moving object is picked up at an imaging position which is fixed, and otherwise when the camera is moved to follow a moving object.

Further, since the described operation is carried out by using the video signal, the explanatory example may be arranged in common and accompanied by other functions, including automatic focusing, automatic exposure control, and automatic dragging function. The control and selection of the detection range can be handled with the video signal without having to resort to any additional sensor. This is a great advantage of the illustrative example.

Figs. 9 to 13 (b) relate to an image shake detecting device which is arranged according to the invention as a first embodiment thereof. The first embodiment is designed to be able to perform the image jitter detection and compensation operations in real time without misunderstanding image shake of a moving image that occurs only in a part of the image scanning plane, and vice versa. For this purpose, the jitter of the image is performed on the basis of the deviation of the generation timing of a feature point appearing on the image plane; and the detection sensitivity of the image flicker detection device is designed to be variable and adjustable to enable the device to cope with all types of motion that occur in the image scanning plane. The details of the first embodiment will be described below with reference to the previously cited drawing.

FIG. 9 Fig. 10 is a block diagram showing an apparatus for detecting image shake applied in a video camera. The representation contains an object OB; a prism 101 having a variable vertex angle; a lens 102, an image sensor 103, ie an image pickup device comprising a two-dimensional CCD or the like and is designed to convert the conversion into an electrical signal from an object image imaged on its image scanning plane by the objective 102; a signal processing circuit 104 for executing a gamma correction operation, a blanking operation, a sync signal adding operation and the like. with respect to the image signal output from the image sensor 103 for the purpose of converting the signal, for example. into a standard television signal of the NTSC system; a video signal output terminal OUT; a luminance signal Y; a horizontal synchronizing signal H.SYNC and a vertical synchronizing signal V.SYNC. A feature extraction circuit 105 is provided, that is, a read-out means for detecting a characteristic feature of an image of the object from the luminance signal Y output from the signal processing circuit 104. For example, the feature extraction circuit 105 is a binary-coding circuit arranged to generate, for example, a binary signal representing the edge portion of the luminance signal Y.

A delay circuit 106 is provided for delaying the binary-coded edge signal by a predetermined period of time output from the feature readout circuit 105. Area designation circuits 107 and 108 are provided for designating a dither scan area of ​​the image scan plane in accordance with commands 81 issued from a control microcomputer 128 which will be described later. A gate pulse generating circuit 109 is prepared to generate a gate pulse signal for a period of time during which the output timing of a signal deviates according to the range set by the area designation circuits 107 and 108, so that a shift of the mark point is indicated. The illustration also includes a quartz oscillator 110; Frequency dividers 111 and 112 serving to frequency-divide the output of the quartz oscillator 110; a frequency target ratio setting circuit 113 adapted to change the frequency dividing ratio of the frequency divider 112 from a ratio over another one between at least two different frequency dividing ratios in accordance with one

Control signal S2 is set from the control microcomputer 128; AND circuits 114 and 115; and pulse counters 116, 117 and 118. A sorter 119 is provided for generating a difference between the outputs of the pulse counters 116 and 117. A divider 120 is provided for dividing the output signal from the subtracter 119 by the output signal of the pulse counter 118. The output of the divider 120 is generated as a component representing the magnitude of the jitter in one direction. A prism driving mechanism 121 includes an actuator such as a piezoelectric element or the like which is provided to change the vertex angle of the variable vertex angle prism 101 by driving the prism 101 having a variable vertex angle. A drive circuit 122 is provided for driving the prism drive mechanism 121 according to information on the amount of jitter that the divider 120 outputs in the direction of mutual use of the jitter circumference.

Blocks 124 through 127 represent circuit elements provided for detecting the position of any part of the image scanning plane that has local image motion on the image plane. Any local image movement, which takes place at the image scanning level, can be prevented from to be held for a dither motion of the entire image scanning plane thanks to these circuit elements, which are composed of: an area determining circuit 124, which is set up to generate a gate signal, to divide the image scanning plane into an m × n number of blocks, as shown in Figs. 13 (a) and 13 (b); a binary coding circuit 125, which is designed to convert the luminance signal of the video signal according to each of the mxn blocks into a binary value using a predetermined manipulated variable level; a delay circuit 126, which is set up to delay a partial image of an image signal by one period in accordance with the one image plane, which is binarized by the binary coding circuit 125 for each of the m × n blocks or areas; and a comparison circuit 127, which serves to compare each area of ​​the image plane of the current field, which is binarized by the binary coding circuit 125 with that of the image plane, that precedes a drawing, and for generating a movement range information P, which indicates the location on the image plane of any area, in which a change of the binary information has been found. This information P is supplied to the control microcomputer 128.

The control microcomputer 128 performs overall control of all the circuit elements, including the area designation circuits 107 and 108, the frequency division ratio setting circuit 113, and so forth. The control microcomputer 128 generates control signals S1, S2 and S3 for controlling the generation of gate signals to be generated in the area designation circuits 107, 108 and 104, respectively. 124th The control signal S2 is generated based on the movement range information P (P1 and P2) supplied from the comparison circuit 127. The frequency dividing ratio of the frequency dividing ratio setting circuit 113 is controlled by the control signal S2.

In Figs. 10 (a) to 10 (c), reference portions A to K respectively show signals obtained on the signal lines A to K shown in various parts of Fig. 9. The image shake detection apparatus described in the first embodiment This invention operates in the following manner.

An incident optical image passing through the variable apex prism 101 and the objective 102 is converted by the image sensor 103 into an image signal. The image signal to be used in the image shake detection means is supplied to the label readout circuit 105. The flag readout circuit 105 converts the edge information of this signal into a binary signal. The binary signal is delayed by the delay circuit 106 for a predetermined period of time before being supplied to the area designation circuit 107 and simultaneously supplied to the area designation circuit 108 without delay. The delay time of the delay circuit 106 is set to a value which is a multiple of one field period. Thus, the delay circuit 106 indicates edge information, that is, an identifying point of an image plane which is the same as the one currently under the scanning, but which is obtained by a predetermined period before the current image plane. With the feature point detection signals being binary signals supplied to the area designation circuits 107 and 108, the area designation circuits 107 and 108 then read out only the signal parts belonging to a set of detection areas on the image scanning plane. The signal parts thus read out are supplied to the gate pulse generating circuit 109, respectively.

The jitter detection range is set for the following reason. For example, when the jitter detection range is close to a peripheral portion of the image sensing plane, the characteristic point tends to be jittered outside the image sensing level. Under these circumstances, it often becomes impossible to correlate two fields. Consequently, the image jitter detection area is preferably not set in a given peripheral area of ​​the image sensing plane (several percent, for example).

In view of this fact, the jitter detection range of this embodiment is set in the manner shown in Figs. 13 (a) and 13 (b). In this drawing, a peripheral area of ​​the image scanning planes, measured in two blocks width, is set to a non-detection area by setting a jitter detection clock signal to "zero". The arrangement of Figs. 13 (a) and 13 (b) will be described later in detail.

The gate pulse generating circuit 109 receives the output signals A and B of the area determining circuits 107 and 107, respectively. 108th The gate pulse generating circuit 109 checks a correlation between these signals A and B and generates a gate pulse signal of a pulse width corresponding to an offset time difference between pulses representing one and the same image. This gate pulse generating circuit 109 is designed to generate a pulse signal H which is generated when the picture of the current field is shifted from the picture of a reference picture in the scanning direction; a pulse signal 1 generated when the current picture is shifted in the direction opposite to the sampling means; and a pulse signal E generated every time an identifying point of the image disappears during a scanning operation. The longer these gate pulse signals H and 1 are, the greater the amount of image jitter. The gate pulse generating circuit 109 generates the logical product (AND) of the gate pulses H and 1 and a clock pulse C, which is generated by frequency division of the oscillator frequency quartz oscillator 110; passes the clock pulses to the pulse counters 116 and 117, while the gate pulses H and I are generated; measures the lengths of the gate pulses by counting the number of clock pulses; and generates the amount of image jitter with respect to an identification point of the image, which is represented by the pulse signal E. When a plurality of feature points are on the entire image plane, the amount of image flare of the entire image plane can be obtained by dividing the total number of feature points by the total number of pulse counts according to the amount of image flicker represented by the gate pulse signals H and 1. This arithmetic operation is carried out during a vertical blanking period.

The movement of the pictures is generally complex. All feature points do not move in the same direction, depending on the circumstances of the object and its background. In view of this, therefore, this embodiment is designed so that a gate pulse length of the signals of two channels A and B are generated for different fields and a difference between them is found out by the subtracter 119. The subtraction operation of the subtractor 119 and the sub-operation of the divider 120 are performed only once per field and can be performed in integers. The length of time required to compute the operations is short, and the arithmetic circuits can be simple and compact.

The information regarding the image squitter amount output from the divider 120 by the above-mentioned arrangement and the operations is supplied to the drive circuit 122. Upon receipt of the information, the drive circuit 122 actuates the actuator of the prism drive mechanism 121 to thereby drive the prism 101 having a variable vertex angle in the direction of the staggered image cir- cular periphery. This causes a compensation of the image jitter.

A practical gate pulse generation algorithm required by the aforementioned operation of the gate pulse generating circuit 109 will be described below with reference to the timing chart of Figs. 10 (a) to 10 (c).

Fig. 10 (a) shows a subject pattern (picture) OB in a moving state to the right (as viewed in the drawing) during an interval period between two temporally different fields 1 and 2. Fig. 10 (b) shows it moving Left. Fig. 10 (c) shows this in a state of motionlessness but with a binarized image pattern approximated by a change in lighting conditions or the like.

In FIGS. 10 (a) to 10 (c), parts A to G respectively denote the waveform of the signal shown on the associated signal line in FIG. 9 occurs. Part A shows the binary signal of the reference sub-picture. This binary signal A represents an image obtained for a previous field preceding the current field by a predetermined number of fields (one field period before the case of this embodiment). Part B shows the binary signal of the image in the current field. Part C shows a clock signal output from the crystal oscillator 110 via the frequency divider 112.

Part D shows the logical sum (OR) of signals A and B. Part E shows the logical product (AND) of signals A and B. The part F shows a pulse signal which rises at the leading edge part of the signal A and drops at the rear part of the signal B. The part G shows a pulse signal which rises at the leading edge of the signal B and which drops at the trailing edge of the signal A. The part H shows a pulse signal representing a logical difference obtained from the difference between the signals F and G. The length of this pulse signal H, ie its pulse width, indicates the amount of movement of the object OB to the right. Part 1 shows a pulse signal representing a logical difference obtained from the difference between the signals G and F. The length of the signal I, ie its width, indicates the extent of movement of the object OB to the left.

The total sum of the pulses of the pulse signal E obtained per scan line represents the number of flag points obtained in one scan line. Thus, in this embodiment, the signals H, I and E are designed to be output from the gate pulse generating circuit 109. This arrangement can be changed to output the signals F and G instead of the signals H and I. However, the arrangement with respect to the pulse counters 116 and 117 is preferable because it allows small maximum counts. Consequently, this is advantageous in terms of the scope of the circuit arrangement.

With the gate pulses generated in this manner, logical products are obtained from the clock pulse signal C and the signals H and I by means of the AND circuits 114 and 115 for generating the signals J and K. Then, the amount of movement of the object, that is, the image squitter circumference can be measured by counting the number of pulses.

In the case of Fig. 10 (c), the object does not move much, and the pulse width of the binary signal becomes narrower with time. The length of the periods of the times t5 and t6 represent information of the degree of movement. This information is also sufficiently readable, since the subtractor 119 is designed to produce a difference between the pulse counts.

The image movement taking place at the image scanning level can be detected finer and more accurately by changing the frequency dividing ratio of the frequency divider 112 to increase the clock signal frequency. In other words, the detection sensitivity of the device can be increased by adjusting the frequency dividing ratio of the frequency divider 112. Furthermore, the image jitter detection sensitivity can be lowered by decreasing the clock signal frequency. If only part of the image is moving, erroneous action can be avoided at best from the erroneous use of the local motion to jitter the entire image plane by lowering the detection sensitivity for the moving part as described above.

The frequency dividing ratio setting circuit 113 is thus arranged so that the frequency dividing ratio of the frequency divider 112 is adjusted and capable of changing the change of the clock signal frequency during the sampling operation. As a result, the detection sensitivity of the device can be increased or decreased for different parts of the image scanning plane.

When taking an image containing, for example, a local motion, such as a dog wagging, the image flicker detection device of the present invention applied to a video camera equipped with an image stabilizer is preferably designed so that she does not notice the wiggling tail of the dog, because it is not a shivering picture.

In this regard, the embodiment is designed to accommodate the above need in the following way: the image position of the subject matter, or the dog, is determined by the same method. The dither detection sensitivity is increased by raising the clock signal frequency for the areas of the image sensing plane that differ from the area where the dog's image is reduced by lowering the clock signal frequency near the dog's image. Further, in view of the degree to which the detection sensitivity is changed, the sensitivity is preferably changed gradually by a plurality of steps with respect to a possible error in automatically detecting the image position of the tail of the dog, and that the shading of the tail in FIG near the image of the dog exists. This allows the camera to work stably.

The above-described arrangement for changing the detection sensitivity, either continuously or stepwise by changing the clock signal frequency, is a feature of the detection method according to the present invention.

The position of the area with a local and partial image movement within an image of an object on the image scanning plane, such as the image of the dog, is designed to be detectable with each other by means of the area-leveling circuit 124, the binary-coding circuit 125, the delay circuit 126 and the comparison circuit 127 ,

Fig. 13 (a) shows an image movement area on the image scanning plane divided into m × n blocks. The comparison circuit 127 is arranged to provide information on a horizontal movement range including, for example, data P1 and P2 for left and right positions in the horizontal direction of the movement range M.

Fig. 13 (b) shows a jitter detection sensitivity distribution for the movement range M, a range around the movement range M, and an area farther from the movement range M. The axis of abscissa shows the position coordinate for each block according to the horizontal direction of the image scanning plane. The axis of the ordinate shows the clock signal frequency for jitter detection.

In Fig. 13 (a), for each of the mxn blocks on the image sample, the luminance level of the image signal output from the signal processing circuit 104 is binarized. The image scanning plane is compared in the comparison circuit 127 with a previously obtained image plane and has been delayed by one field period by the delay circuit 126. The position on the image plane of any block showing the same change in information detected by this comparison. Then, the position of any partial movement occurring locally on the image plane is thereby detected and generated as a movement area information P.

It is assumed that the mxn blocks are denoted by ai, j, and that the total number of 16 areas ai, j (i = 0 to 3; j = 0 to 3) is detected on the image plane as the movement area M with some movement occurred Then, the portion of the image plane having the movement range M is set to go to zero by causing the frequency of the clock pulse signal to detect the amount of jitter to be a non-detection range. Since other areas near the movement area M tend to be touched by the movement, the lock sensitivity is not immediately raised for nearby areas. In the case of this embodiment, by increasing the clock signal frequency, the jitter detection sensitivity is gradually increased for other areas in accordance with the fact that they take place further away from the movement area M. These other areas include near areas M1 having a width corresponding to two blocks, areas M2 lying on the outside of the nearby areas M1 and having a width corresponding to two blocks, and area M 3 lying on the outside of the areas M2. The clock signal frequency is set to be raised in accordance with the distance from other areas of the movement area M. It is assumed that the clock signal frequency for the area M3 for which the embodiment is arranged has a maximum jitter detection sensitivity with fc, the clock signal frequency being set to fc / 2 (1/2 sensitivity) for the area M2, fc / 4 (FIG. 1/4 sensitivity) for the near range M1 and zero for the range of motion M.

This concept allows the device to detect the position of any local motion that occurs in the image scan plane, determine the image jitter from an area with no local image motion, and compensate for the image jitter thus detected.

Further, there are areas M4 each having a width corresponding to two blocks, which are largely outside the image scanning plane, and which are arranged as non-detection areas for the purpose of avoiding a detection error, adverse effect, and shading of the image of the object, etc., as already mentioned.

While the embodiment in the foregoing is designed to detect a range of movement in the horizontal direction, a range of movement in the vertical direction can also be detected in exactly the same way.

As described above, the lock sensitivity is set to be changed every other block that is away from the movement range. However, of course, the sensitivity change number of blocks is not limited to two blocks, but may be changed by any number considering the suitability according to the total number of blocks provided on the image scanning plane.

The stepwise adjustment of the detection sensitivity described above can also be set up for a continuous adjustment concept.

To ensure that the image-shake detection device is able to cope with any of the recordable types of changes that occur in the image patterns, the device must be able to operate without error, even under such circumstances as described in US Pat Figures 11 (a) to 11 (c) are shown.

FIG. Fig. 11 (a) shows the case that an object pattern existed in a preceding field 1 over a predetermined period before it disappeared and no longer exists in the current field 2. FIG. 11 (b) shows the further case that two object patterns existing in the preceding field 1 overlap each other to appear as a pattern in the current field 2. FIG. 11 (c) shows the further case that strong jitter brings one object pattern closer to another object pattern. In order to ensure accurate operation even under these conditions, the device must be equipped with some image pattern discriminating means, in addition to the circuitry previously described.

In view of this problem, errors provided in the discriminating means capable of discriminating the aforementioned circumstances can be prevented so as to prevent the device from obtaining any data in the aforementioned circumstances.

For executing this discrimination method, the device is designed, for example, as follows: under circumstances shown in Figs. 11 (a) and 11 (b), the number of pulses of the pulse signal E existing in the pulse signal D corresponding to those shown in Figs. 9 and 10 (a) to 10 (c) is the logical sum of the signals A and B which is counted. In cases where object patterns do not differ in size and are therefore considered to be fully correlated, as shown in Figs. 10 (a) to 10 (C), the number of signal pulses E (the logical product of signals A and B) ) which are in each signal pulse D (the logical sum of the signals A and B), always equal to 1. In the case of FIGS. 11 (a) and 11 (b), on the other hand, the aforementioned number of pulses 0 and / or 2 or greater than 2. Consequently, the aforementioned discriminating means counting the number of signals E is designed to determine the validity of the data only when the counted number is equal to 1, because in this case the two fields are correlatable with respect to the feature point. In all other cases, the date is set to invalid. Under the condition of FIG. 11 (c), the discriminating means further checks the signal A or the signal B for its size of the image pattern and does not decide to deal with any image pattern smaller than the maximum measured value of the image shake. This concept prevents the erroneous operation of the device.

The values ​​of the constants to be used that are currently compensating for the image blur are the following. If the image jitter results from the jitter of the camera, the upper limit of the image jitter to be compensated is set to about ± 25% of the image plane with the degree of compensation, about 1/5 to 16 of the currently detected jitter amount.

Further, the gate pulse generating circuit 109 may be composed of a logic circuit having TTL or the like. Thus, the gate pulse generating circuit 109 may be easily designed and integrated in a compact form. The logic of the gate pulse generating circuit 109 of the invention is not limited to the logic used in this embodiment.

In the case of the described embodiment, the means for detecting image jitter movement in the horizontal direction of the image plane is established during the process of field scanning. The number of data samples is increased for higher accuracy according to the number of subject patterns (or feature points) in the increasing image plane. The image jitter detection is theoretically possible only with a horizontal scanning line. Consequently, a one-directional dithering operation can be performed by using a one-dimensional CCD (a line sensor) instead of the two-dimensional CCD.

Next, a method for detecting image shake in the vertical direction will be described below. There is no vertical scan line for a video signal. Consequently, the above description with reference to Figs. 10 (a) to 10 (c) can not be applied thereto. Thus, the embodiment is provided with a device for forming a luminance signal in the vertical direction in a manner described below.

Figures 12 (a) to 12 (f) show by way of example a way in which this method can be carried out.

The determination is conceived in the vertical direction of the image plane as indicated by the vertical line in FIG. 12 (f) indicated to be performed. A point at which a vertical image shake from a in Fig. 12 (a) is detected on the detection line within the image plane. A time t10 required to reach this point is obtained from the horizontal synchronizing pulse. Then, a pulse train based on this value is generated after the lapse of time tlo from the horizontal synchronizing pulse in each horizontal scanning operation, as shown in FIG. 12 (b). A luminance signal is separated from a video signal obtained between the pulses. Any change in the level of the level of the pulse train, as shown in FIG. 12 (c) represents a change in luminance that has taken place in the vertical direction. The height is sampled and held to obtain a signal waveform as shown in FIG. 12 (d) is shown. Then, an arbitrary conversion process or the like. applied to this signal waveform to thereby read out a feature point in the same manner as in the case of the horizontal detection action. This concept allows the use of the detection circuit described above. The detection sensitivity may be increased in terms of the method of reading the feature point by obtaining the zero crossing point of a secondary differential signal instead of using the binary edge.

The device embodying the inventive method with the concept described above can be made of ordinary components such as TTL, C-MOS, etc. be formed. The number of gates required is relatively small, so that integration can be done without difficulty. Consequently, then can be built at a low cost. The device used in a compact video camera makes it possible to reduce the size and cost compared with the conventional stabilization type camera.

As mentioned above, the image shake detection means according to the present invention is designed to be able to measure the deviation from feature points occurring in each image plane during a scanning operation on the image planes obtained at different timings; the detection of the magnitude of the image jitter from the measured value is thus achieved, and the detection sensitivity is variable. Consequently, this device is able to exactly make a distinction between a jitter of the entire image plane and a partial image movement occurring locally within the image plane, so that the compensation of image jitter can always be carried out extremely accurately.

The arithmetic operation for finding the amount of image jitter can be performed briefly during the vertical blanking period, so that the processes of the embodiment can be performed almost in real time.

Since the operation principle of the device does not require an A / D converter, a large memory, and no large computing area, the embodiment enables the simplification of the structure and the reduction in size, weight, and cost.

In the described embodiment, the invention is applied to a camera of image stabilization type, but the invention is not limited to the camera of this type. The invention allows a wide range of applications including, for example, an image recognition device for industrial robots and an image sensing system for industrial image measuring instruments.

Figures 14 to 19 show a second embodiment of the invention. The second embodiment relates to the improvement of the control portion of an optical pickup system to be controlled by the image shake detection apparatus of the invention. The embodiment improves the stability and responsiveness of the control part in the control of the image pickup system on the basis of the result of the image jitter detection.

The details of the second embodiment are described below with reference to FIGS. 14 to 19.

FIG. 14 Fig. 16 is a block diagram showing the second embodiment in a state where it is applied to the image stabilization and object tracking device of a television camera. The representation contains an object OB; a prism 202 having a variable vertex angle; a taking lens 203; an image sensor 204 including a CCD or the like. is and which serves to generate an electrical signal by photoelectric conversion of the image information, generated by the taking lens system 203 on the image scanning plane; a video signal processing circuit 205, intended to carry out a signal processing action on the image signal, which is output from the image sensor 204 including a gamma correction, a blanking process, the addition of sync signals, etc., and for generating a video signal 205a, which agrees with the NTSC system; a monitor 206, for example, an electronic viewfinder or the like; an image recorder 207, which is for example a video recorder; an image offset detection circuit 208, which is ready to detect a jitter or movement condition of an image, which is generated at the image scanning level of the image sensor 204; and an image offset information signal 208a, which is output from the offset detection circuit 208.

For example, the image skew detecting circuit 208 always compares the edge portion of an image or the location of the center of gravity with the resp. and that of the immediately preceding image plane, and calculates and finds out a change in this point in which an image plane (a sub-image) is transmitted. Since the process is carried out by means of electronic circuits, the result of the calculation is carried out almost simultaneously with the end of the transmission. FIG. 15 Fig. 14 shows, by way of example, the internal structure of the image offset detecting circuit 208 which is provided for detecting the edge distribution state of an image of an object occurring on a plurality of (2) image planes obtained at different times and detecting any image movement through one Change of the found difference between the two image planes.

In the image offset detection circuit 208 in FIG. 15 the following applies: the edge-distribution detecting circuit 281 is provided for detecting the edge-distribution state of an image of an object existing on the image-scanning plane by the image signal 205a generated by the video-signal processing circuit 205. A feature of the subject image is determined, for example, from the number of edges distributed in a predetermined direction, which is the vertical or horizontal direction. A delay circuit 282 is provided for storing an amount of the detection information corresponding to an image plane (or a field), and is generated by the edge distribution detecting circuit 281; and for generating the information by delaying by one period of one field. A comparison circuit 283 compares the edge distribution information, which is output and received directly from the edge distribution detection circuit 281, with the previous edge distribution information received by the delay circuit 282. The comparison circuit 283 is thus designed to generate an image offset information signal 208a indicating the size and direction of the image offset.

The characteristic of the image plane is thus determined from the distribution on the image plane. The detected characteristic of the image plane of each current field is compared with that of the immediately preceding field. Any positional change, ie, trembling of the image or movement of the object occurring during the period of one field is thus detected. The jitter detection information corresponding to the change in the position of the image can thus be generated in one cycle of one field.

In Fig. 14 Reference numeral 209 denotes a control circuit. The control circuit 209 is designed to generate a control signal 209a for controlling the variable apex angle prism 202 based on the image offset information signal 208a. FIG. 14 further shows an adder-subtracting circuit 291 provided for receiving the image offset information signal 208a output from the image offset detecting circuit 208 and adding it to (or subtracting from) the feedback output signal; which comes from a multiplier circuit 294. A further multiplier circuit 292 is provided, which is designed to multiply the output signal of the adder-subtractor 291 by a predetermined coefficient and to generate and supply a control signal to a motor 210, which serves to drive the prism 202 having a variable vertex angle. A memory 293 is provided for storing the control information supplied from the multiplying circuit 292 to the motor 210. Further, the memory 293 is provided for generating the input control information after lapse of one period of two fields in the order of the input. In other words, the memory 292 operates on the basis of a FIFO (input sequence processing memory). The multiplier circuit 294 is arranged to multiply the information stored in the memory 293 by a predetermined coefficient and feed the result of the multiplication back to the adder-subtracting circuit 291st

The variable vertex angle prism driving motor 210 is configured to change the vertex angle of the variable vertex angle prism 202 through the variable vertex angle prism 202 in accordance with the control signal 209a coming from the control circuit 209. The motor 210 is controlled to cause the variable vertex angle prism 202 to track the image offset or dither from the previous image plane and to cancel the image offset.

Between the video signal processing circuit 205 and the image offset detection circuit 208, there is a switch SW for selecting between execution and non-execution of the following action. When the switch SW is open, the control system that controls the variable apex angle prism stops operating and causes a mode of not tracking the image jitter or offset, and does not compensate the image shift.

The optical image pickup control system provided by the present invention operates in the manner described below.

When an object to be photographed relative to the television camera due to a panning or the like. A change of the subject image on the image scanning plane of the image sensor 204 is made to pass through the prism 202 having a variable vertex angle and through the taking lens 203. With the image converted into an electric signal, the change occurs in the image output signal 205a of the video signal processing circuit 205. The output image signal 205a of the video signal processing circuit 205 is supplied to the monitor 206 having an electronic viewfinder or the like. and the image recorder, ie, the VCR 207. The image can thus be captured while being viewed through the viewfinder.

The image signal 205a output from the video signal processing circuit 205 is also supplied to the image offset detection circuit 208. In the image shift detecting circuit 208 constructed as shown in Fig. 15, the degree of movement of the object OB is determined by calculation. As a result, an image skew detection signal 208a is supplied to the control circuit 209.

The control circuit 209 is supplied with information regarding the position of the subject image obtained at least in the vertical or horizontal direction of the image scanning plane. According to this information, the control circuit 209 causes the motor 210 to rotate in the direction of reducing the change resulting from the positional change of the subject image occurring in the image signal 205a output from the video signal processing circuit 205. The angle of the prism 202 having a variable vertex angle is thus changed by the rotation of the motor 210. In other words, the optical axis of the variable apex prism 202 is set in the direction in which the image moves in such a manner as to follow (tracking) the object. This feedback control is repeatedly performed in synchronism with a cycle set to a value of an integer multiple of the sampling period of the image sensor 204 which is 1/60 sec (one field period). However, in the case of high-speed image offset following property, the integer is preferably set to 1.

Next, the control sequence according to the invention will be described in comparison with the conventional control system. With respect to the arrangement shown in Fig. 14, the conventional device is designed to supply the image offset information signal 208a of the image offset detection circuit 208 directly to the multiplier circuit 292. The conventional device is not provided with the multiplier circuit 294 and the memory 293.

On the other hand, the device of this invention is equipped with the adder-subtractor 291, the multiplier 294 and the memory 293. This difference from the conventional device allows the device of the invention to execute a control algorithm as described below.

FIG. 16 Fig. 12 is a graphic representation of time series of the operation of tracking the positional change of the subject image. In Fig. 16 The axis of abscissa indicates the period of time from the beginning of object tracking, that is, the start of the image jitter compensation action. The time is graduated in units of partial images. The axis of the ordinate shows the position of the object OB as being smaller than the image signal 205a. Each of the reference symbols t1, t2, ... means a time indicating a partial image.

It is now assumed that the object movement causes the image on the image scanning plane to move in a direction at a constant speed.

Referring again to Fig. 16, a reference symbol A indicates the location of object movement occurring at the image scanning level, with no subsequent action being taken for the movement. The place shows that the place of the

Object OB changes gradually and deviates on the image plane according to the passage of time.

A symbol B denotes a subject post-action performed by the image jitter and object tracking device of the conventional television camera. A conventional device can be regarded as being designed to supply the output of the image offset detection circuit 208 of FIG. 14 directly to the multiplier circuit 292 without the adder-subtractor 291, the memory 293 and the multiplier circuit 294 ,

Another symbol C indicates the location of the object movement which is carried out with an object following action by the circuit of the embodiment shown in FIG.

First, the object follow-up action of the conventional device will be described. The coefficient to be used by the multiplier circuit 292 is assumed to be "1". It is assumed that the motor 210 rotates at a speed proportional to the input voltage and that it operates without delay. It is also assumed that the object follow-up action starts at time t1, and that images generated by the image sensor 204 at times t1 and t2 require one field period for transmission. Information regarding any object image offset output from the image offset detecting circuit 208 as a result of comparing these images is obtained at time t3. The motor 210 starts driving according to the value of this information. The motor 210 then operates to drive the variable apex angle prism 202 in such a way as to compensate for the attitude change of the object. During a period between times t3 and t4, the subject image appears to rest on the image plane. The result of the image comparison made at times t2 and t3 is obtained at time t4. As a result, the subject image becomes quiet and does not change even during the period between times t4 and t5. In other words, with the motor 210 moved on the basis of the degree of movement of the subject image detected between the times t1 and t3, the position of the object OB at the image scanning level during the time period between the times t3 and t5 remains unchanged. When the images taken in the image sensor 204 are compared with each other at the times t3 and t4, the result of the comparison is zero. With the obtained value at time t5, the image is then considered to have set in a position, and the rotation of the motor 210 is stopped at this time t5, despite the continued current movement of the object B, such as through the Place A shown. As a result, the position of the subject image on the image scanning plane again undergoes a change at time t5. This offset is detected between times t5 and t6 and between times t6 and t7. Then, the motor 210 is driven again at time t7 to leave the image position of the object OB unchanged on the image scanning plane. Thus, according to the conventional control method, the above-described phenomenon is repeated every two fields. The object position on the image scanning plane varies stepwise with time and never assumes a fixed value.

Successor accuracy or image stabilization efficiency can be expressed as follows:

Image Stabilization Efficiency = Object position change on the image plane caused by

Subsequent action / object position change is obtained on the image plane without subsequent action.

Accordingly, the image stabilization efficiency according to the conventional control method is only about 1/2.

Further, an attempt to increase the image-holding efficiency by increasing the multiplier of the multiplying circuit to a value greater than "1" tends to cause oscillations. When the output signals 210a and 209a of the motor 210 and the multiplier circuit 292 are designed to be delaying, the responsiveness of the control system is degraded to avoid these oscillations.

According to the arrangement according to the invention, which in Fig. 14 is shown, the object follow-up operation is performed with the switch SW closed in the following manner: the memory 293 is designed to always store the control information 209a output from the multiplier circuit 292; and for obtaining information on the value of a timing at which an image is outputted from the image offset detection circuit 208 according to the value of the image offset information signal 208a, is picked up by the image sensor 204.

In other words, for the image offset detection information 208a obtained at time t5, for example, the memory 293 is caused to generate an output control information 209a of the control circuit 209 sampled at the time t3 as the image on which the detection information is based. At object succession start time t1, all data is initialized to become zero.

The coefficient of the multiplier circuit 292 is also set to "1" in the same manner as described above. If the object following action allows to start at time t1, the output of memory 293 remains at zero from time t1 to time t3. Consequently, the follow-up characteristic of the device according to the invention coincides with the conventional device during a period between the times t1 and t5.

In other words, the memory 293 has no input signal and remains in its initialized state at the times t1 and t2. At time t3, the image offset information signal 208a obtained between times t1 and t2 is supplied to the control circuit 209. This image offset information signal 208a is received by the adder-subtractor 291. The output control information 209a of the control circuit 209, which has been multiply multiplied by an integer multiple from the multiplier circuit 292, is supplied to the motor 210, and also supplied and stored to and from the memory 293 at the same time. At the subsequent time t4, the image offset information signal 208a obtained between the times t2 and t3 is supplied to the control circuit 209 to be supplied to the motor 210 and the memory through the adder-subtracter circuit 291 and the multiplier circuit 292. In the meantime, the control information 209a obtained at the time t3 is already stored in the memory 293 and kept in the memory 293 at the time t4. Thus, at time t4, the output of memory 293 will be zero as before. Until the next times t5, the object tracking characteristic of the device according to the invention coincides with the object tracking characteristic B of the conventional control device.

However, when the image offset information signal 208a picked up between the images in the image sensor 204 at times t4 and t5 is output from the image offset detection circuit 208 at time tS, the control information 209a is sent to the motor 210 via the adder-subtractors - Circuit 291 and the multiplier circuit 292 included in the control circuit 209 are supplied. This control information signal 209a then controls and causes the motor 210 to rotate in the direction of slowing down the image offset. At the same time, the control information 209a is also supplied from the memory 2g3 and stored. Meanwhile, the previous control information 209a stored at time t3 preceding by two field periods is output from the memory 293 and multiplied by an integer multiple (× 1) from the multiplier circuit 294 and supplied to the adder-subtraction circuit 291. The control information 209a, which is supplied at time t5, is obtained by adding the multiplied value with the image offset information signal 208a. This allows the motor 210 to operate continuously between the times tS and t6 based on the control information 209a obtained at the time t3 even if the value of the image offset information signal 208a becomes zero, because a change of the image information at the times t3 and t4 as a result of the control operation with respect to the prism 202 having a variable vertex angle was performed by the drive motor 210 at time t3. Unlike the follow-up characteristic of the conventional control device which allows a positional deviation from the object OB to increase again, the device according to the invention continuously executes the following action without such an increase in the positional deviation even after the time t5 as indicated by the characteristic curve C in FIG , 16 as long as the control is in the working range of the prism 202 having a variable vertex angle. In other words, the device according to the invention enables the television camera to perfectly follow an object moving with respect to the camera.

The embodiment increases the following efficiency for the following reason. In terms of the fact in that the value of the image offset information signal 208a of the image offset detection circuit 208, which is obtained temporarily distributed by the operation described above, contains a time delay, is the control circuit 209 designed to determine the amount of image compensation at the time in which the value of the image offset information signal 208a is sampled; and to prepare a next control instruction by setting an excess or deficiency in the amount of compensation.

When the moving speed of an object changes while its image is changed either by the movement of the camera due to a camera panning or the like or by the movement of an object, the amount of displacement of the motor 210 is caused by inertia or viscosity, the optical works Image scanning device according to the invention as follows.

Fig. 17 shows the result of the simulation performed by a computer with respect to the movement of the object OB, which is assumed to have a sinusoid with a frequency of 1 Hz and an amplitude corresponding to 60 pixels, and the displacement amount 210a of the motor 210 is a transfer function of the secondary delay system with respect to a transition characteristic. In order to obtain a response step C (t), the following formula is used for the simulation:

where μ is an attenuation coefficient and ωn is a natural angular frequency. In the case of> 1, the step response C (t) is expressed as follows:

In this case, = 1.5 and ωn = 2 are substituted as values ​​for a current control system, and one cycle (one second) of the curve is shown in the drawing.

In Fig. 17 the axis of the abscissa shows the time and is divided into units of a partial image. The axis of the ordinate shows the change position of the object and is divided into units of two picture elements. The origin represents a tax start time. Curves indicate object tracking states, including characteristic curves A ', B' and C ', which respectively correspond to curves A, B, and C of FIG. 16 correspond. Comparing the curves B 'and C', one of them is closer to the axis of abscissa and closer to the straight line representing a degree of loss of image shake, less deviation from the object and a higher degree of stability and thus shows a better one Follow-up and image stabilization effect as the other.

The curve B 'was obtained by setting the multiplier of the multiplier circuit 292 to "1.5" to decrease the amplitude. Nevertheless, in the case of the curve B ', the amplitude covered over ± 30 picture elements and the stabilizing effect was around 0.5, showing the unstable state of the picture.

If the multiplier described above is set above 1.5, oscillations would occur. If set to a value less than 1.5, the stabilizing effect decreases. In this case of the curve C ', the multipliers of the multipliers 292 and 294 have been set to "3". However, no vibrations occurred and the amplitude covered only ± 10 pixels, thus documenting the stable state of the image. The arrangement according to the invention exhibits the above-described conspicuous effect even when using the same frequency and the same transmission coefficient of the motor as in the conventional control system which does not have the memory 293 and the multiplier circuit 294.

The control system according to the invention can be constructed either of a digital circuit or an analog circuit. In the case of the analog circuit, the use of the memory 293 may be replaced by a sample and hold circuit or by a delay element. In this case, the control circuit 209 may be designed to have a drive clock signal. Then, the efficient control can be accomplished by synchronizing the timing of the drive clock signal with that of the CCD or the video signal.

Further, regarding the image compensation method, the use of the variable apex angle prism 202 can be replaced by an optical compensation method by which the position of the taking lens 203 and that of the image sensor 204 are shifted parallel and perpendicular to the optical axis. The algorithm according to the invention is also applicable to this method.

When an element such as a piezoelectric element or a plunger which causes an output offset in proportion to the input voltage is used instead of the motor 210, an integrator must be connected between the control circuit 209 and the motor 210. Furthermore, the coefficients to be used for the multipliers 292 and 294 need not have fixed values. One coefficient may be changed to another according to the operating conditions. It is also possible to design these circuits either to increase the input power to the second power or to compute a square root. Furthermore, the coefficient of the multiplier circuit 292 and that of the multiplier circuit 294 need not coincide with each other. The multiplier circuit 294 may be designed as an observer using a modulated function on the secondary delay of the motor 210. If the jitter circumference of the motor 210 can be detected directly, a modification as shown in FIG. 18 shown used.

In Fig. 18 are to the arrangement of FIG. 14 a sensor 211 has been added to detect the magnitude of the jitter and the output of the motor 210; a driver circuit 212 which serves to control the motor 210 in accordance with a signal coming from the sensor 211; and a differential circuit 213. Motor 210, sensor 211 and driver circuit 212 form a closed loop system. This control system is for executing a feedback control in accordance with the control signal 209a to make the jitter amount 210a of the motor 210 an opposite value. The sensor 211 directly detects the control operation amount 210a. The value thus determined is stored in the memory 293. The computation control algorithm based on this invention may then be executed using the stored value, as previously described. The output of the sensor 211 is position information such as length or angle. The differential circuit 213 is provided for the purpose of changing this position information into a position change amount of an image obtained at a time preceded by two fields from the current image.

FIG. 19 shows another application example of the optical image sensing device according to the invention. In this case, the control device according to the invention is applied to an automatic assembly robot. This control system is shown in a block diagram. The illustration includes a television camera 220, an image offset detection circuit 212 for detecting an image offset by an image processing operation; a computer 222 for issuing commands; a control circuit 223 which embodies a control algorithm according to the invention; a robot hand 224; and a part 225 subjected to the assembling process. The control system works as follows. When the computer 222 gives a command for detecting the position of the part 225, the television camera 220 takes this information of the position of the part 225 and that of the robot hand 224 in the form of an image. The image offset detection circuit 221 calculates a difference between the command given by the computer 222 and the image obtained by the television camera 220. The control circuit 223 gives a drive command to the robot hand 224 based on the result of the calculation. In this case, a driving action is instructed to work in the direction of reducing the difference between the image of the television camera 220 and the command issued from the computer 222.

Further, in each of the embodiments of Figs. 14 and 18, the image offset detection information output 208a of the image offset detection circuit 208 represents a difference between the positions of the object obtained at the time interval corresponding to one field. In the case of the in Fig. 19 Otherwise, the output signal of the image offset detection circuit 221 is a position information. Thus, using the circuitry of FIG. 14 or 18 for the in Fig. 19 Embodiment required to calculate a change amount per sampling period by providing a differential circuit before the control circuit 223.

The use of the control algorithm of the invention for this robot allows accurate positioning, accurate speed operation and feedback coefficient adjustment within a wider range for ease of adjustment.

According to this invention, as previously described, the optical image sensing system controller is designed to calculate the amount of control operation for the feedback control system in consideration of the sampling period; which is required to scan such scattered detection information, like the information, which is to be handled in the control of the image sensing system of the television camera; and for supplying the controlled system with the information regarding the calculated value. As a result, the control operation becomes extremely stable. Both the response and the frequency response are improved. Furthermore, the system stabilization allows to provide a wider feedback coefficient adjustment range. Thus, with the invention applied to the optical image scanning system and other image processing devices of other types, image stabilization and object tracking functions can be advantageously performed to compensate for such image displacement resulting from movement of the object.

Furthermore, the device according to the invention can be accommodated in a compact size and can only be designed with electrical circuits without recourse to any additional sensors or additional optical parts.

While this invention is applied to a feedback control system for an optical image sensing system in the case of the above-described embodiment, the invention is not limited thereto but applicable to any other system having a feedback system for controlling a controlled system based on temporarily distributed detection.

Figures 20 to 22 (c) show a fifth embodiment of the invention. In this case, the image shake detection means according to the invention is applied to a tracking type automatic focusing device, an automatic tracking type exposure control device, etc. for the purpose of setting a tracking range for tracking a movement of an object occurring in the image plane.

The fifth embodiment is designed to obtain motion vectors from a plurality of areas set in the image plane. An optical history is statistically processed to determine an object tracking area and an image jitter detection area.

The details of the embodiment are described below with reference to the drawing.

FIG. 20 Fig. 12 is a block diagram showing an arrangement of a video camera provided with the image jitter compensation device arranged according to the present invention. The figure shows an object OB; a prism 312 having a variable vertex angle, which has a variable optical axis, a taking lens 314, an image sensor 316, for example, composed of a two-dimensional CCD; a signal processing circuit 318, which performs signal processing on an image signal, that comes from the image sensor 316, including the processing of gamma correction, the blanking and the addition of sync signals, such that a television signal, for example of the NTSC standard, is output from the output terminal 320; a luminance signal Y, a horizontal synchronizing signal H. SVNC; a vertical sync signal V.SYNC, a delay circuit 322, which is for delaying the luminance signal Y by a predetermined period of time and is composed of, for example, a field memory of the FIFO type (input sequence processing memory); a block unit pulse generating circuit 324, which serves to generate a gate pulse signal, to control the video signal under one scan in such a way that it is divided into a predetermined number of blocks, which are set at the image level; and scheduling circuits 326 and 328, which divide the luminance signal Y by logically driving it according to the pulses, which come from the block unit pulse generating circuit 324. The one-shot circuits 326 and 328 are designed to output the input luminance signal in units of blocks set on the image plane. More specifically, each of them consists of a gate circuit arranged to be opened and closed by pulses from the block unit pulse generating circuit 324 and a memory adapted to store signal parts permitted to pass through the gate circuit.

In Fig. 20 a motion vector detection circuit 330 is provided for comparing the signal of the currently obtained on the image plane with the signal output from the delay circuit 322, whereby a previous image plane preceding the current image by a predetermined period of time; and for generating a motion vector by detecting any motion encountered in each of the scheduled blocks. A memory 332 is arranged to store the motion vector information for each part of the image plane. A statistical processing circuit 334 is for preparing a histogram indicating the magnitude and frequency of each motion vector. A threshold determination circuit 336 is arranged to recognize the shape of the histogram and to determine a threshold to be described later. A range determining circuit 338 is arranged to look up and read out the blocks from the histogram which are within the threshold determined by the threshold determining circuit 336.

An image shake detecting circuit 340 is arranged to detect the amount of image shake from the luminance signal. For example, the image shake detection circuit 340 is composed of a correlation calculation circuit arranged to perform a matching action of a representing point. An actuator 342 is for changing the apex angle of the variable apex angle prism 312. A drive circuit 344 serves to drive the actuator 342 according to the output of the image shake detection circuit 340. The deflection angle of the optical output axis with respect to the input optical axis of the variable apex angle prism 312 is controllable by adjusting the apex angle of the variable apex angle prism 312.

The video camera of FIG. 20 works as follows. An object image passing through the variable apex angle prism 312 passes through the taking lens 314 onto the image sensor 316. The image sensor 316 then generates an image signal. The signal processing circuit 318 performs the above-described processing operation on the output signal of the image sensor 316. A luminance signal Y output from the signal processing circuit 318 is supplied directly to the subcircuit 328 and also indirectly to the subcircuit 326 through the delay circuit 322. The delay circuit 322 delays the luminance signal by the period of one field (about 16.7 msec) before being supplied to the subcircuit 326. The subcircuits 326 and 328 are arranged to divide an entire image plane mxn blocks in accordance with the number of pulses coming from the block division pulse generating circuit 324. In this case, it is assumed that m = 20 and that n = 14 so as to have a total of 280 blocks.

The motion vector detection circuit 330 generates the motion vectors for each block by a method called time-space gradient method. This procedure is from B. K. P. Horn, et al. discussed in "Artificial Intelligence", 17, pp. 185-203, 1981. It allows real-time processing by special hardware. This motion vector for the whole image plane obtained by this method shows a movement taking place at each block called "optical history". The actions of the circuit elements 330 to 338 will be described with reference to Figs. 21 (a) to 21 (d). FIG. Fig. 21 (a) shows by way of example an image plane of a currently photographed partial image. FIG. Fig. 21 (b) shows an optical history obtained by accumulating a difference over a predetermined period of time occurring between the current picture and an immediately preceding field. FIG. Fig. 21 (c) shows histograms representing the optical path in size, taken in the direction X and Y. FIG. Fig. 21 (d) shows the portions of the areas recognized in this embodiment.

At this point, a moving picture of a moving object is recorded. As a result of the photographer's intention to avoid the movement of the camera, the movement of the background is less than that of the object. the motion vectors detected by the motion vector detection circuit 330 are accumulated by the memory 332 for a predetermined duration (one second, for example). Thereafter, the accumulated vectors are supplied to the statistical processing circuit 334. The statistical processing circuit 334 prepares the histograms shown in FIG. 21 (c), by ranking these vectors according to the sizes of the X and Y components of each vector. In Fig. 21 (c), the upper half of the drawing shows the vector histogram for the direction X and the lower half the vector histogram for the other direction Y. In these histograms, each of the axes in the X and Y directions show positive and negative quantities separated by an origin null set, for example at the center of the image plane. The frequency values ​​of the vectors are displayed in the vertical direction of the drawing. Threshold determination circuit 336 determines thresholds from the shape of these two histograms. In either of the two directions X and Y, one finds a very small value near the distribution part having a peak very close to zero in each direction X and Y, and the position of this value is determined to be the threshold. Threshold values ​​are thus obtained on both the positive and the negative sides. In Fig. 21 (c) are reference symbols Thx1, Thx2, Thy1 and Thy2 these thresholds.

The threshold values ​​thus determined are supplied to the area determining circuit 338. The area determining circuit 338 detects the blocks which are lower in the range of threshold values ​​among the motion vectors stored in the memory 332. For example, it is assumed that a block located in the i-th position in the direction of m in the jth position toward n is expressed as a block Bij, and that in the block bij the amount of movement in the direction x is expressed as uij and the amount of movement in the direction y with Vij, then the circuit 338 gives the following condition:

Thx1 <Uij (Thx2 and

Thy1 <Vij (Thy2.

Blocks satisfying these conditions are assumed to be "on" while the remainder are "off". Then, a relationship between "on" and "off" blocks becomes as in Fig. 21 (d). In Fig. 21 (d), a hatched part indicates the area of ​​"on" blocks which approximately coincides with the background area shown in Fig. 21 (a).

The amount of image shake detecting circuit 340 calculates and obtains an amount of movement from the "on" area. The detection method to be used by the image flicker detection circuit 340 is not limited to the illustrated dot fitting method. The movement can be detected at high speed by image processing. The method may be replaced by any other method including control over the execution ("on") and non-execution ("off") of the calculation of the correlation for each of the blocks. The amount of jitter obtained by the extent of image jitter detection circuit 340 is sent to the drive control circuit 344. The actuator 342 then drives the variable apex angle prism 312 in such a manner that the output (amount of jitter) of the image shake detecting circuit 340 is reduced, that is, the movement in the image plane due to the jitter of the camera is reduced.

The motion vector detection circuit 330 and the extent of image jitter detection circuit 340 are preferably arranged to perform arithmetic operation for each field. However, the processing operation of the circuit elements 332 to 338 may be carried out once in several 100 msec to several sec because: the camera crosstalk frequency is within a range of 1 to 3 Hz, and the frequency of mechanical vibration of vehicles or the like. are higher than these. Thus, any non-indicative object tracking action can be avoided by setting the cycle of statistical computation of these circuit elements to a value greater than these periods.

In the above description, the histograms have been described as being prepared in the X and Y directions. However, this can be changed to prepare a histogram for a two-dimensional space XY. When the optical history obtained by use of images is obtained for a predetermined period of time from the motion vector detecting circuit 330, the computing cycle of the switching elements 332 to 338 must be changed accordingly.

When the photographer next wishes to perform a panning on following an object, the embodiment operates as follows. It is assumed that a person B, who is located in the central part of the image plane, as shown in FIG. 21 (a), to be followed. In this case, the optical progression of the in Fig. 22 (a). Thus, the photographer tries to keep the person B always in a predetermined position in the image plane. As a result, the motion vectors in the image of the person B become small. In the meantime, the motion vectors in the rest of the image plane including the background become large. FIG. FIG. 22 (b) shows histograms corresponding to those of FIG. 21 (c) are the same. FIG. FIG. 22 (c) shows an "on" area (broken-line portion) set by the area determining circuit 338. The thresholds Thx1, Thx2, Thy1 and Thy2 and the ranges are determined by the same procedure described above.

In the above-described embodiment, when the movement or deformation of the object is periodically repeated, and when the statistical histogram processing periods of the circuit elements 32 to 38 coincide, a repeated motion dynamic range can not be discriminated from a static range. At this point, the change size (distribution) of the vector is obtained for each individual block. Any area that does not have the change amount larger than a predetermined value is set by a successor area. It is assumed, for example, that average vector values ​​are obtained in a processing period T of the statistical processing for the vectors Uijt and Vijt of a block Bij at the time t, and are, and the degrees of distribution of the vectors Sxij and Syij are, where the vector distribution can be expressed as :

The values ​​Sxij and Syij of each block are compared with each other. It is also possible to perform this comparison in the square root of the values ​​Sxij and Syij (standard deviation).

As described above, the subject tracking area of ​​this invention is determined by statistical processing of the optical history. Consequently, this embodiment is capable of automatically discriminating an object stably positioned on the image plane with respect to other objects, even if these objects are hardly distinguishable from each other by their luminance, or even if there are a plurality of moving objects on the image plane gives. The area setting method of the present invention is not limited to a television camera. There is a wide range of applications including industrial television camera and surveillance camera. The embodiment of the invention has notable advantages for practical applications.

Figures 23 to 26 show a sixth embodiment of the invention. In this case, an image movement detecting means which utilizes the optical path of the motion vectors explained above is applied to an automatic focusing device. First of all, general information about the sixth embodiment will be set forth below.

The video camera has recently come into use with an automatic focusing device which uses the video signal of the camera and is capable of detecting a focus state using the information of the image existing in an image plane regardless of the distance to the object to be photographed. Thus, instead of the focus detecting method and performing the focusing detection by projection of infrared rays or ultrasonic waves and detection of the reflected waves, the automatic focusing device has come to this place.

The focus detecting method of this kind using the video signal can be roughly classified into two methods.

In one of the two methods, the video signal is modulated to be forcibly brought into a blurring state by slightly vibrating either a component of the objective body of the taking optical system or an optical axis image sensor with an actuator such as a bimorph element, a plunger coil, or The like (hereinafter referred to as the modulation method).

In the other method, the taking-out optical system is driven in such a manner as to cause, for example, a high-frequency component of the video signal to be set to a maximum value (hereinafter referred to as an empirical method).

In the modulation method, a focusing degree varies a little, and the modulated signal almost stays at zero after a focus state is reached even when the taking optical system vibrates. In case of disarming a modulated signal is generated. Then, the phase of the modulated signal is inverted with respect to the modulation signal, depending on whether it is in a near-focus state or a far-focus state. To this end, the focus adjustment member (which is a DC motor in general) is restarted according to the presence or absence of this modulation signal. The direction of the restart is determined according to the phase of the modulated signal. This allows the actuator to begin a proper restart. However, the modulation method requires the modulation actuator, which requires a high degree of accuracy and must be capable of performing a high-precision action. As a result, the structure of the camera becomes complex and large. As a result, not only is it necessary to perform a complex adjustment, but it is also unfavorable in terms of electric power consumption.

In the case of the empirical method, the lens or the image sensor is moved in such a manner that a signal component corresponding to the degree of focusing is brought to a maximum value. After reaching a focus state, a recovered focus signal is stored. Thereafter, a current focus signal is compared with the stored signal. If the result of the comparison indicates the occurrence of any change, the focus actuator is restarted. This method does not require any complex focusing control drive systems to achieve simplification and size reduction of the camera. A drawback of the empirical method, however, is that when the actuator is restarted after it has been brought to a halt, when the focus condition has been reached, the information regarding the direction in which the lens or the image sensor is to be driven is often not reached , This requires empirical shifting of the lens in one direction or the other direction. When the focus signal increases in the direction thus indicated that the lens position reaches a focus point, it is possible for the actuator to further drive the lens. However, when the focus signal decreases, indicating that the lens is moving in the wrong direction, the drive direction of the actuator is reversed. Since this method theoretically requires some kind of empirical action, it does not represent a stable automatic focusing operation.

In the sixth embodiment, the inventive method of jitter detection using the optical path of the motion vectors is applied to a simple arrangement of the above-mentioned empirical method in such a manner that the information on the lens drive direction becomes available. The automatic focusing device, which is arranged according to the sixth embodiment, consists of: a focus detecting means, set up to perform the focus adjustment by detecting a degree of focus with respect to an object, who appears in the picture plane; a motion vector detecting means for generating a motion vector of an image in each of the plurality of divided areas of the image plane; and control means, which serve to calculate the degree of movement of the image on the basis of the motion vector information, which are output by the motion vector detecting means, and for updating the focus detecting means, when the degree of movement runs out of a predetermined range.

In the direction in which the object and the camera move relative to each other, thus detected by the information relating to the image motion vector, the automatic focusing device can be restarted according to this information. Unlike the modulation method which requires a complex and high-precision mechanism including the modulation actuator, etc., the embodiment is capable of stably, quickly and with high accuracy, performing an automatic focusing action with a simple structure fundamentally based on the empirical method.

In Figs. 23 to 26, the components which are either the same or which are provided to perform the same operations as corresponding parts of Figs. 20 to 22 (c) are given the same reference numerals and symbols, and the details of them already have been omitted in the description below.

FIG. 23 is a block diagram, which differs from the arrangement of FIG. 20 in the following points: the variable apex angle prism 312, the prism drive actuator 342 and the drive control circuit 344 are included. Instead of these, there are arranged a focus adjusting member 362 for focusing the taking lens 314 and a drive control circuit 360 which controls the actuator 362; There are also provided auto focus detecting blocks 348 to 358 for detecting movement of the object based on the motion vector information coming from the motion vector detecting circuit 330. A restarting detection circuit 346 is provided for restarting the automatic focusing device.

In Fig. 23, the restarting detection circuit 346 for receiving the output from the motion vector detecting circuit 330, the outputs of a distance measuring range setting circuit 348 and the output of a comparison circuit 358, and determining whether to restart the automatic focusing means are determined according to these output signals is.

Next, an automatic focusing adjustment system is constructed as follows. The range measuring range setting circuit 348 is composed of a gate circuit serving to cause only a luminance signal included in a luminance signal Y coming from the signal processing circuit 318 and corresponding to a focus detecting area (distance measuring area) set to be on the image scanning plane , The distance measuring range setting circuit 348 detects the distance measuring range on the image scanning level based on the output of the range determining circuit 338. A high-pass filter 350 is for reading out a high-frequency component from the luminance signal corresponding to the distance measuring range given by the distance measuring range setting circuit 348. A detection circuit 352 is for conversion into a DC signal whose high-frequency component is filtered out by the high-pass filter 350. An integrating circuit 354 is for integration for a predetermined duration of the DC signal output from the detection circuit 352. Further, the integrating circuit 354 is designed to be controlled in its integration sensitivity by a control signal 348a coming from the distance measuring range setting circuit 348 in accordance with the distance as the distance measuring range changes, so that range correction is performed. A delay circuit 356 is for delaying the output signal (hereinafter referred to as a focusing signal) of the integrating circuit 354 for a predetermined period of time corresponding to, for example, one field. The comparison circuit 358 is for comparing a currently obtained focusing signal with a focusing signal obtained one field period previously and delayed by the delay circuit 356. A driving circuit 360 is for driving and controlling an actuator (a DC servomotor, for example) which serves to control the focusing state of the taking lens 314.

The automatic focusing device, which is arranged according to the present invention, operates as follows.

As previously mentioned, the area determining circuit 338 is arranged to generate information regarding the patterns of the object parts A, B, and C and the pattern of the background area D, as shown in FIG. 21 (d). The distance measuring range setting circuit 348 performs a gating operation on the basis of this information in such a manner that only the luminance signal of a main subject portion is transmitted, and supplies it to a high-pass filter 358. In other words, an area where the main object is located is thus used as a distance measuring area. In high-pass filter 358, the high-frequency component is read from the supplied luminance signal. The read-out high-frequency component is converted into a DC signal from the detection circuit 352, and is integrated by the integrating circuit 354 for a predetermined period of time. The integrating circuit then generates a focusing signal. The focus signal is supplied to the comparison circuit 358 together with a focus signal relating to a previous image plane which was previously acquired one frame and delayed by one field period from the delay circuit 356. These focus signals of a current image plane and the previous image plane are compared with each other in the comparison circuit 358. The result of the comparison is supplied to the drive circuit 360 through the restart control determination circuit 346. The drive circuit 360 then controls the focus adjustment member 362 and causes the taking lens 314 to control in the direction of increasing the level of the focus signal. When the focus signal level reaches a maximum value and then drops again, the assumed position of the lens at the maximum level of the focus signal has reached the focus. The taking lens 314 is returned to this position and the automatic focusing action is completed.

When the integration outputs of the integrating circuit 354 are compared by the comparing circuit 358 during the automatic focusing operation, when the size of the distance measuring area previously found one field period ago differs from that of the distance measuring area of ​​the current field, the integration signals differ in level, though the focus states are unchanged. In such a case, the difference of the level could result in an erroneous determination, and the level comparison then does not correspond to the sharpness. To solve this problem, a control signal 348a, which represents the size of the distance measuring range, is supplied to the integrating circuit 354. Any area difference between the distance measuring ranges of two sub-picture planes is compensated thanks to this control signal. In other words, the integration signal output level of the integrating circuit 354 is normalized by a range value of the distance measuring range. In this embodiment, the above-described compensation is accomplished by changing the integration sensitivity of the integrating circuit 354 from one value to another.

The practical arrangement of the distance measuring range setting means is as described above. In the statistical processing performed subsequent to the memory 332, the statistical processing circuit 334, the threshold determination circuit 336 as described above, the histograms for both directions X and Y of the image scanning plane are prepared. However, this can be changed to prepare a histogram for a two-dimensional XY space. Furthermore, in the case where the optical history is obtained by the motion vector detecting circuit 330 using an image accumulated over a predetermined period of time, the cycle of operation of the group of circuit elements from the memory 332 to the area determining circuit 338 must be changed in accordance with the image accumulating time.

As described above, the distance measuring range is set according to the output of the range determining circuit 338. However, in the usual photographing, a main object is located in the central part of the image plane. In addition, it is desirable to avoid a distance measurement confusion between the main object and the background. Consequently, an area in and around the central part of the image plane can be selected alone as a distance measuring area. In this case, areas A and C can be completely disregarded. It is also possible to use a window function in such a way that the middle part of the image plane is weighted with 100% and four diagonal corners get 0% of the weighting of the area between these parts, which vary continuously.

The restart of the automatic focusing device on the basis of the motion vector is currently set in the following manner. Owing to the simplification of the illustration, the distance measuring range is assumed to be in the middle part of the image plane and around it, for example as viewed through the region B in FIG. 21 (d). FIG. 24 shows an optical history, which is obtained in the area B under these circumstances. In the case of this optical path, the object corresponding to the area B moves a little forward to the upper right side of the image plane as the camera approaches. The motion vectors are divided into blocks, as a whole, to the right with the directions and sizes that diverge to the peripheral areas. This indicates that the object is approaching the camera and at the same time moving to the right side.

Fig. 25 shows the characteristic of an optical path of this kind. In Fig. 25, a taking lens 314 is identical to the taking lens 314 of Fig. 23. Reference numeral 364 denotes the object corresponding to the region B of Fig. 21 (d).

Numeral 368 is an image that is generated on the image scanning plane and represents the object. Reference numeral 366 denotes a state of the object obtained when it moves upward, as shown in FIG. 24 while reaching the taking lens 314. Reference numeral 370 denotes the image of the moving object. A vectorial magnitude of the optical path, which represents the upward movement, as shown in the drawing, does not change independently of their positions on the image plane. The beam path resulting from the object approaching the lens is caused by the extent of the subject image and its vector size, as its position in the image plane changes accordingly. The size of the vector is zero on the optical axis of the taking lens 314, ie, in the center of the image plane, and increases in accordance with the distance from the central part of the image plane. The optical progression of FIG. 24 is a composite optical path consisting of the vectors resulting from these different factors. Further, while the images 368 and 370 on the image scan plane are shown at different positions on the optical axis, they are actually generated on one and the same planes.

Next, an arithmetic operation executed by the restarting determination circuit 364 will be described. FIG. 26 is a graphical representation of the area B of FIG. 21 (d). Coordinates x and y are obtained in a center of the image plane set as an origin (0, 0). The coordinates of blocks in the distance measuring range B are assumed to be (i, j). The vector of the optical path in each block is assumed to be A (i, j). Furthermore, the unit vector in the x direction is assumed to be "i" and the unit vector in the y direction to be "y" ei. The optical profile can be rewritten as A (i, j) = iAx + jAy.

The offset of the optical gradient vector due to the expansion or contraction of the image means divergence of vectors of the image plane. The divergence "divA" of vector A can be expressed as follows:

The divergence represents the change in the distance between the object and the camera.

The restart-start determining circuit 346 generates the distance-measuring-range control information from the distance measuring range setting circuit 348, and also generates information regarding the motion of the vectors of each block of the image scanning plane from the motion vector detecting circuit 330. The restarting determining circuit 346 then calculates the sum from "divA" as follows:

d = ΣB divA

The sum of the divergence "divA" in the region B changes in accordance with the expansion or contraction of the subject image on the image scanning plane.

In Fig. 24, the sum "d" becomes larger than zero because the image has expanded. As the object moves away from the taking lens 314, the sum becomes smaller than zero.

Consequently, a state of "d = 0" continues after reaching a focus state, the distance to the object remains unchanged, and the lens remains at rest. If the sum "d" is not equal to zero after taking the in-focus state, the distance between the camera and the subject is considered changed, and the autofocusing device is restarted. In this case, the focus adjustment direction may be set by a positive or negative value of the sum "d". More specifically, the object can be regarded as having come closer to the taking lens 314 when the sum "d" is greater than zero. At this point, the focus is on the back of the object and a so-called wide-focus state is achieved. If the sum "d" is less than zero, the object has moved farther away from the lens. At this point, a Naheinstellzustand the lens is achieved.

With the divergence "divA" of the vector of the optical path in the distance measuring range thus examined, any occurring change of the object distance and the direction of the change can be accurately detected. This allows the automatic focusing device to be restarted if necessary after reaching the in-focus state. Even in the case where the empirical method is used for the automatic focusing device, the information about close-up or remote adjustment can be accurately obtained without performing a trial action. The arrangement of this embodiment thus enables the automatic focusing device to be able to be restarted quickly and accurately.

According to the invention, a change of the object distance can be reliably detected by performing the statistical process on the motion vectors for calculating the divergence of the vectors of the optical history, even if the event of a plurality of objects in the image scanning plane, a complex background or a hardly readable image feature a small luminance difference, etc. are given. The sensitivity and accuracy of the embodiment are thus significantly higher than those of conventional devices.

As already mentioned, according to an arrangement of the embodiment of the invention, the optical plane of the image plane is generated by detecting the motion vectors for each of the plurality of blocks on the image scanning plane; the occurrence or non-occurrence of any movement of the object is detected from the offset of the optical path; A distinction is made exactly between an approaching movement and a partial movement; and the need to restart for an automatic focus adjustment action is determined on the basis of the thus obtained information. Thus, unlike the conventional modulation method, the embodiment avoids the necessity of resorting to high-precision and complex mechanisms for positively modulating the focusing signal by generating a focus state or a defocus state by obtaining information from a near-focus or focus-focusing state. The automatic focusing device according to this invention is thus designed to be able to accurately and precisely determine the necessity for restarting the focusing adjustment action from the image signal by an efficient and stable operation.