Autofocus image acquisition system

Background

0001The invention pertains to biometrics and particularly to acquisition of biometric images.

0002PRIOR ART image acquisition systems include: <patcit id="pcit0001" dnum="US20050264678A"><text>US 2005/0264678</text></patcit> which discloses a camera with an autofocus function for capturing a series of images when a lens is in different positions and analysing the images to identify a best-focused image; <patcit id="pcit0002" dnum="WO2005093510A"><text>W02005/093510</text></patcit> discloses a digital camera comprising an image sensor for capturing an image, focusing is achieved by a series of images having differing focus provided by the lens arrangement being captured by an image sensor and stored in memory for later analysis to derive an in-focus image: <patcit id="pcit0003" dnum="US20070286590A"><text>US 2007/0286590</text></patcit> discloses an imaging apparatus comprising a lens operating range computing part configured to estimate a focus position based on information about subject distance. <patcit id="pcit0004" dnum="US2007064145A"><text>US 2007/064145</text></patcit> discloses an autofocus control apparatus and method of controlling that apparatus.

Summary

0003The present invention in its various aspects is as set out in the appended claims. The invention is an image standoff acquisition system for capturing images of an eye or eyes of a non-cooperating subject. The invention may overcome a need for exact focusing by capturing a rapid sequence of frames while sweeping through the focus range of optics of an acquisition system or camera. The focus range may be effected with moving the lens, the subject, the camera, the image sensor in the camera, or a combination of two or more items. After the sequence is captured, then a frame of the sequence having an iris most in focus may be selected.

Brief Description of the Drawing

0004<ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">Figure 1</figref> is a block diagram of an image acquisition system;</li><li><figref idref="f0002">Figure 2</figref> is a more detailed diagram of the image acquisition system;</li><li><figref idref="f0003">Figure 3</figref> is a diagram of an example image on a camera sensor;</li><li><figref idref="f0004">Figure 4</figref> is a diagram of a window on an eye of an image of a subject;</li><li><figref idref="f0005">Figure 5</figref> is a diagram showing focus distances between an acquisition camera and a subject;</li><li><figref idref="f0006">Figure 5a</figref> is a graph showing the nominal focus distance versus time;</li><li><figref idref="f0007">Figure 6</figref> is a graph showing focusing distance versus time, and trigger signal voltage for image acquisition;</li><li><figref idref="f0008">Figure 7</figref> is a diagram illustrating lateral and radial movement of a subject relative to an acquisition camera;</li><li><figref idref="f0009">Figure 8</figref> is a graph of capture/shutter trigger signal versus time for exposures and frames of images;</li><li><figref idref="f0010">Figure 9</figref> is a graph of intensity variance versus focus distance for captured images;</li><li><figref idref="f0011">Figure 10</figref> is a diagram simulating some of the captured images of an eye of a subject at various focus distances shown in the graph of <figref idref="f0010">Figure 9</figref>; and</li><li><figref idref="f0012">Figure 11</figref> is a diagram of an iris image cropped from a focused image in <figref idref="f0011">Figure 10</figref>.</li></ul>

Description

0005Iris patterns may contain many very small details that need to be correctly recorded for analysis and identification. Capturing those details may require an optical system with a very large spatial frequency bandwidth. Such system may have very sensitive focusing in that its depth of field can be as low as a fraction of a millimeter. Given the depth profile of a human face, a high quality system cannot necessarily be focused on it in its entirety. The system may need to find within a frame the eye to be imaged and then focus on it. The focusing should be very precise. It is this requirement that makes many current systems so user unfriendly, because in order to lower their engineering complexity and thus cost, the systems shift the burden onto the subject in hope that the subject's cooperation will eventually result in a well focused iris image. To obtain a well focused iris image, the subject may be commanded to move back and forth, left to right, and so on, until the subject eventually positions its eye into the system's sweet spot. Experience shows that it may require much patience and willingness to cooperate by the subject. Alternatively, handheld devices like those used in the military need to be moved by the user to get the eye into a crosshair and achieve focus. If the user is not well trained or works under stress, capturing a good image may again become a time consuming challenge. Since the amount of iris details needed to be captured may depend on an intended security level, in that the higher the security level, the more precise solution is required to capture adequate images.

0006In an iris image acquisition system, the optics alone is not necessarily the costliest part. Cost may be primarily and progressively driven by the degree of subject's lack of cooperation and the user's lack of skill which the system can tolerate and still work reliably. The present system may address these issues of cooperation and skill in a way that requires neither optical autofocusing nor precise range finding.

0007Until recently, cameras offered not really very large image sizes and had low frame rates. The present system may build upon recent advances such as large image sizes and high frame rates. Other technologies may be incorporated.

0008The system may have a camera that takes a fast sequence of frames while the optical focus lens position is incremented or varied so that each frame is taken with a somewhat different focus adjustment. The focus lens position may execute a full sweep of its adjustability range, very much like if one turns the focus ring on a classical camera lens objective from end to end, i.e., from a focus at infinity to a focus at the nearest working distance of the objective, while shooting pictures in a rapid succession during the turn of the focus ring.

0009There may be the stop-and-go approach, when in each iteration, the system first resets the lens' focus and then takes a shot. There may be the continuous approach, when for instance, four or so shots are taken while the focus lens is moving, without stopping during the image acquisition. For this "continuous focus lens sweep" to work well, the image exposure time (T_E) should be shorter than the time (T_F) it takes the lens to get out of its depth of field.

0010Once the sequence has been captured, each frame may eventually be checked to note if the subject of the camera is at least approximately in focus by measuring its contrast or, in a more detailed way, spatial frequency content in a number of small patches selected in the frame post-processing. Patch selection may follow a predefined pattern or it may be random. A first check of the frames may result in discarding most of the images in the sequence for being very much out of focus. The selection may be done after the sequence has been taken but it could be done in real time though the latter processing or autofocusing could be more costly.

0011A significant aspect is that the frames which survive a first pass will be analyzed further to locate one or two eyes of a person in them, and then the spatial frequency content will be measured over the eyes only. Eye finding may be generated by an algorithm. The eye in the image or images may be localized to and followed by a window. Such window may be described herein relative to <figref idref="f0003">Figures 3</figref>, <figref idref="f0004">4</figref> and <figref idref="f0011">10</figref>. The frames having the highest spatial frequency content over the eyes in them may then be the best eye and/or iris images in the sequence, and be either in or nearly in focus.

0012The rate at which the frames are to be taken, may define a minimum degree of cooperation required from the subject. If the subject is not moving, holding its head more or less steady, and is reasonably close to the camera, then frame rates on the order of hundreds of frames per second may suffice, without a need for very high illumination iradiance levels which may be harmful to the subject's eyes. Near IR flash may be used which may be increased with intensity at farther distances or shorter exposure times.

0013The frame sequence may be processed either offline or in real time, depending on the investment one is willing to make in the necessary computational hardware for the present system. Another approach may include culling out the unlikely candidate frames in real time, storing only those with promise, and analyzing them off-line after the entire focus sweep sequence is completed. Off-line processing of the frames may be done within the system in a second or so.

0014The system may have preset focusing prior to image capturing, whether it be either manual focusing done through manipulating the mutual position between the subject and the device, or focusing based on a distance or focus mechanism and then setting the focus back 100mm, for instance, or so behind the subject or a focus sweet spot for image acquisition, such as an iris being coincident with the camera optics object plane.

0015The system may have a focus lens suitably instrumented with a motor drive. During frame capture, the focus lens position may sweep an entire focus range in either continuous motion or discrete steps. In the latter case, frame grabbing may be synchronized so that a frame is taken when the lens stops at each consecutive step of a series of steps. A camera may have a fixed focus and the camera moves instead or the subject is asked to step forward or backward, or the sensor in the camera may be moved.

0016Measuring a quality of focus may rely on measuring image intensity variance over a patch, or rely on approaches based on spectral analysis. The measuring may be done off-line. For example, the variance value changing from one image to the next image may indicate whether the images examined are moving toward or away from the focus. The system is not limited to a particular approach of focus quality measurements or certain algorithms for face and eye finding.

0017<figref idref="f0001">Figure 1</figref> is a diagram of, for example, an iris image standoff acquisition system 10. The system may include an optics assembly 11 having an adjustable focus for capturing eye 28 details of a subject 20. Alternatively, an adjustable focus may be achieved by moving the camera sensor to and from the subject, moving the camera to and from the subject, or having the subject move to and from the camera. The assembly 11 may be optically coupled to a camera 16. There may be an illuminator 48 in case of a need for added light or a flash for obtaining a fast image take of the subject. The focus of assembly 11 may be adjusted with a signal from a computer / processor 30 (referred to herein as a computer). Shutter control, and imagery capture and receipt may be managed by the computer 30. Computer 30 may be a PC or other equivalent device. It may contain software appropriate for effecting standoff image acquisition of an iris. Computer 30 may be connected to a database 32, a network, enforcement agency data center, an immigration processing center, and/or so on, the latter items of which are shown as "other" 38 in <figref idref="f0001">Figure 1</figref>.

0018<figref idref="f0002">Figure 2</figref> is a diagram of an illustrative example of the iris image standoff acquisition system 10. Focus of the subject may be achieved in several ways. One way is to have the subject 20 move across the focusing distance 34 of the camera 16 as indicated by motion arrow 73. Another way is to move the camera 16 across the focusing distance 34 as indicated by motion arrow 71. Another way is to move the sensor 27 across the focusing distance 34 as indicated by motion arrow 72. Still another way is to move the lens 12 across the focusing distance 34 as indicated by motion arrow 13. In each of the indicated ways of moving across the focusing distance 34, the other ways are held still in that just one item is moved. Moving across the focusing distance 34 indicates that the movement begins at one side of the distance 34 and ends up on the other side of the focusing distance 34. During movement across the focusing distance for each of these ways, a sequence of images may be captured at a fast rate. Each of the movements may be "continuous" or "stop and go".

0019For illustrative purposes, the way of moving the lens 12 across the focusing distance 34 may be example for description herein. The optics assembly 11 of <figref idref="f0002">Figure 2</figref> may have a lens 12 which may be moved back and forth in direction 13 with a housing 14 for holding the lens which may be used for focusing an image of subject 20 or a portion of it on a sensor array 27. The lens 12 or sensor array 27 may be moved with a motor, a drive or mover mechanism 18 which may be connected to housing 14. The lens housing 14 may be moved in a continuous mode resulting in a "continuous focus sweep" approach, or in a step mode resulting in a "stop-and-go" approach having focusing increments.

0020In the "stop-and-go approach, at each stop, the lens focus maybe set and then a picture is taken and detected at sensor array 27 in camera 16. The lens 12 may be set again for another focus at the next step and then a picture is taken and so on. The optics assembly 11 may contain one or more lenses.

0021In the continuous approach, a sequence of images or pictures may be taken while the lens housing 14 is moving. The focusing of lens 12 does not necessarily stop during image acquisition or picture taking. Relative to the "continuous focus lens sweep" approach, image or picture exposure time should be shorter than the time it takes the lens to get out of its depth of field. Light 23 from subject 20 may be conveyed through lens 12 of lens housing 14, and into camera 16 onto array 27.

0022A mechanical or electronic shutter may be controlled with a signal from a shutter control 25 of a subsystem 26 which may be a part of the computer 30. The shutter may be electronically controlled, or may in effect be a picking off or an electronically receiving an image from sensor array 27 for a specified duration as desired. The exposure time of the image sensor for an image may be less than 100 milliseconds. In some instances, it may be less than 10 milliseconds or even less than 2 milliseconds, depending on the design of the system. The shutter may be in effect an illuminator 48 or other non-mechanical type of device. Alternatively, the shutter could be a mechanical mechanism.

0023There might be no explicit shutter as such in system 10. The image capture or acquisition may occur during a time of a flash or a constant supplemental illumination, such as LED sources, to assist in image exposure or capture. During no-flash time, an image sensor may be set, configured or designed not to detect any light. There may be a threshold which a light intensity, whether IR or visible, has to reach before the camera sensor 27 will sense and capture an image projected to it. The duration of the illumination or flash, particularly relative to sensor 27, may be equivalent to the speed of a shutter opening. For instance, system 10 may have an IR illuminator or flash 48 which may provide a basis for a short duration exposure of an image on the sensor 27. The IR illuminator 48 may be of a wavelength which is not readily visible to but may have some effect on the subject 20 such as a person. There may also be a visible light illuminator in place of having an intensity which may be inconspicuous to the person targeted by system 10.

0024IR flash or illuminator 48 may be electronically controlled by a shutter signal from shutter control 25. Alternatively, sensor 27 may capture or acquire an image by being electronically controlled in terms of the amount of time the sensor is allowed to be sensitive to light from the subject. Such sensor control may emulate a shutter effect. Camera 16 may monitor subject 20 and its eye or eyes 28 for purposes of aiming, focusing and capturing an image of the subject. The focusing change of lens 12 may be provided by a sweep signal from a module 33 to the drive or mover mechanism 18. An input to module 33 may be a "preset done" signal from a preset module 36 which occurs when a preset signal from a module 37 indicating an object plane or focusing distance (d_F) 34 for an initial time (t₁ of a start of a sequence of images to be captured of subject 20. The preset signal (d_F(t₁)) of module 37 may be based on an estimate of the distance 34 between the subject 20 and camera 16. A point of distance 34 measurement from camera 16 may be lens 12 or some other item of the camera 16 arrangement. A module 39 may provide a signal of the distance 34 estimate <i>d̃_s</i> to preset module 37. The signal from module 39 may also go to a decision item represented by a symbol 36 which asks the question whether the preset has been done. This signal indicates that the preset has been done and a "yes" signal may go to the sweep module 33 and the shutter control module 25 for a go-ahead of the sweep and shutter control to begin. If the signal from module 39 has not been received by preset module 37 and the decision item at symbol 36, then a "no" signal may go to a "wait" module 47 which means that modules 25 and 33 should wait until such signal has been sent before starting.

0025<figref idref="f0003">Figure 3</figref> shows an image 29 of subject 20 on sensor 27. Image 29 may be forwarded on to computer 30. Sensor 27 may be, for example, a 2560 by 1920 pixel array providing about 4.9 mega-pixels of imagery of the subject. A window or portion 31 of a target or an area of interest may be extracted from image 29. It may be a VGA format with a pixel size image of 640 by 480 pixels, or be some other size such as 480 by 290 pixels as used in the present example. An eye 28 of the subject or person may be the target to be extracted with window 31. Eye finding software may be used here in conjunction with the window. The window covers a portion of the camera's field of view which should include the target, e.g., eye or iris, in full but need not be much larger than required for better overall accurate focusing on the target (i.e., eye 28). Having the larger image 29 of subject 20, the pointing or aiming of the camera 16 is not necessarily so critical. Further, there may a significant aiming or pointing error tolerance of system 10. A diagram of window 31 is shown in <figref idref="f0004">Figure 4</figref>. About 150 pixels may be allowed for a diameter of an iris 46 of eye 28. The iris may be somewhat centered in window 31 as shown by the pixel dimensions. However, it need not necessarily be centered. Also, window 31 may have other pixel dimensions. The dimensions may be adjustable.

0026<figref idref="f0005">Figure 5</figref> is a diagram of an estimate <i>d̃_s</i> 49 of actual distance 34 between subject 20 and camera 16, e.g., lens 16. The estimate 49 of the distance, <i>d_s</i> 34, may be more or less than actual distance 34. For an illustrative example, the estimate 49 may be less than distance 34. When an estimate is received by the preset module 37 and forwarded on to the sweep module 33, the focusing distance of lens 12 for camera 16 may be set at a distance behind the estimated distance 49 for the initial start of focusing and taking images of the subject 20. The initial focusing distance may be about 100 mm (or other distance) behind the estimated distance 49, and be designated as d_F(t₁) in <figref idref="f0005">Figure 5</figref>. The distance at the other end of the focusing range may be about 100 mm or so ahead of the estimated distance 49, and designated as d_F(t_N), where N may indicate the number of frame periods, or frame shots taken t_F milliseconds apart, or it may be the last unit of N units of time for the focus sweep, whether continuous or discrete, of the subject by lens 12 of optics assembly 11. Starting at time t₁, the frames are taken periodically at times t₁, t₂, t₃, ... through t_N. Somewhere in the course of the focus sweep between t₁ and t_N, say at t_L, the focusing distance d_F(t_L) equals the actual distance 34. <figref idref="f0006">Figure 5a</figref> shows a graph of the nominal focus distance "d_F(t)" with the plus/minus delta distance of focus "+/-½Δ_DOF" versus time "t". In the Figure, t_L is intentionally chosen so that d_F(t_L) happens to fall within the depth of the field +/- Δ_DOF of both the (n-1)th and nth frames. In more typical cases, however, the iris will be found in focus in just one frame.

0027<figref idref="f0007">Figure 6</figref> illustrates graphically focusing distance d_F between the camera 16 or its lens 12 and subject 20, particularly an eye or eyes 28, and more particularly an iris or irises 46 of the eyes, over a number of time increments t, as shown by line 51. In the graphs of <figref idref="f0007">Figure 6</figref>, there are about ten instances 52 of time through the range of focusing at which an image of subject 20 may be captured. There could be more or less than ten instances. At each instance of time 52, a trigger signal voltage 53 may go from shutter control module 25 to camera 16 for the shutter or other mechanism to initiate a capture or acquisition of an image of subject 20. At one of these instances of time 52, an image of an iris 46 of at least one eye 28 of subject 20 may be captured on array 27 at an in-focus distance relative to the subject 20 and camera 16.

0028Movement of subject 20, as illustrated in <figref idref="f0008">Figure 7</figref>, is not an intended aspect of the present system 10. Even though subject 20 may have some inadvertent movement, it typically is not critical for obtaining a good focus of the subject. Lateral movement 54 of subject 20 does not necessarily affect the focus distance of the subject from the camera 16, unless it is particularly large. This movement may be arrested by choosing short exposure time. The present system may address radial or forward / backward movement 55 through appropriately choosing the frame rate. As long as individual frames overlap, the present system is immune to radial movement. However, since subject 20 may be directed to take a certain position as a steady subject, the radial movement 55 would not necessarily be a factor relative to attaining an in-focus image, particularly since the exposure time of camera 16 would be about one to two milliseconds. If there is to be a concern, the exposure time T_E may be shortened to avoid effects of movement. The lateral movement 54 or radial movement 55 would not necessarily exceed the depth of field of the optics assembly 11 for a given focus.

0029<figref idref="f0009">Figure 8</figref> is a diagram showing a relationship of exposure time T_E 56 and a frame period T_F 57. A shutter, exposure, capture or trigger signal may be indicated by a pulse 58 where there is an exposure time 56 of the subject 20 on image array 27 of camera 16. Time t₁ may begin when the exposure time T_E starts at the rising edge or beginning of pulse 58. The exposure time T_E ends at the falling edge or end of the pulse 58. The next exposure time T_E may begin at the end of t₁ and the start of pulse 58 at t₂. The same may occur for each time t₃, t₄, t₅, and so on, until pulse 58 at t_N, where N is the total number of pulses.

0030The depth of field of the optics 11 may, for one example, be about 10 mm. That the subject 20 moves forward, for instance, or that a focus that moves forward at a velocity V_F, is a factor to consider. Δd may be regarded as a depth of field. The formula T_F ≤ Δd/V_F applies. If T_F is much shorter than Δd/V_F, then there may be a waste of resources. If T_F is longer than Δd/V_F, then the system may be unworkable because of gaps in focus coverage. Thus, in the present illustrative example, for still subjects, T_F may be a period of up to 10 ms where the velocity V_F approaches 1 meter/sec. The exposure time may be relatively much shorter such as about one to two milliseconds. Focusing distance would sweep 0.2 m during which the actual subject distance d_s is within the depth of field of at least one frame. A sequence of images may be taken and processing relating to them may generally be done later or could be done in real-time. The processing may take only several seconds; however, this time is large relative to 10 milliseconds multiplied by the number of exposures. For 20 images, the time would be 200 milliseconds for the total image acquisition. A goal is to have at least one frame in the depth of field of the subject. This approach may permit a high frame rate (e.g., 100 frames per second) of image acquisition.

0031The above numbers may assume that the relative velocity |v_f-v_s| = 1 m/s. If this is the case, the system may be at its maximum speed and the 20 images will provide no difference overlap. For smaller relative velocities, there may be an overlap. The smaller the relative velocity, the larger may be the overlap.

0032<figref idref="f0010">Figure 9</figref> is a graph of intensity variance as a function of distance difference. There may be an actual in-focus point at distance 34 and an estimate in-focus point at distance 49. Distance 62 may represent an increment of the difference depth of focus at distance 34 which, as an illustrative example, may be about 25 mm. Other depths of focus 63, at and relative to distance 49, may be chosen to be about 25 millimeters and the timing of image capture can be such that the images may have about a 5 millimeter or so depth of focus overlap with adjacent depths of focus 63. That means a depth of focus 63 may extend about 12.5 mm on each side of its center. The depths of focus 63 may overlap other depths of focus 63 relative to 20 mm increments of distance difference throughout about a 200 mm portion shown in graph 35. The focus change may sweep forward (left to right) or backwards (right to left) on the graph. The example noted herein may be a sweep forward version. Thus, the focusing distance adjustment beginning at t₁ may be designated as d_F(t₁) = <i>d̃_s</i>-100 mm at line 59 and ending at t_N may be designated as d_F(t_N) - <i>d̃_s</i> + 100 mm at line 67, as noted relative to <figref idref="f0005">Figure 5</figref>. T_F may equal 20 ms. So, one may have an image acquired a -100 mm, -80 mm, -60 mm, -40 mm, -20 mm, 0 mm, +20 mm, +40 mm, +60 mm, +80 mm, +100 mm, 11 images in total, at lines 59, 60, 41, 42, 43, 49, 44, 45, 65, 66 and 67, respectively, and so on, relative to the estimated focus distance <i>d̃_s</i> 49. This setup may be performed under an assumption of a focus velocity being about one meter per second. As long as the focus distance estimate is within 100 mm of the depth of field 62 of the actual focus distance, then there should be at least one image taken within the depth of field of focus of the subject. Also, as long as T_F is less than or equal to the depth of focus Δd divided by V_F, then an image of the subject within the depth of field 62 of focus should be acquirable. For example, it may be that the estimated distance of focus is at line 49, which can turn out to be, for instance, about 15 mm closer to lens 12 of the camera than the actual distance 34 of focus from subject 20, which may be from contrast analysis processing of the images for selecting the image which is in the best focus. The distances as represented in <figref idref="f0010">Figure 9</figref> are for illustrative purposes and not necessarily drawn to scale.

0033The curves of graph 35 are not necessarily smooth, due to sensor noise and window location uncertainty image discretization, as indicated by an example magnification 68. The lens 12 focus may be set, for instance, at infinity to start and to its closest focus to end, and then be changed through its focus range as a sequence of images of the subject 20 is captured during the change of focus. For illustrative purposes, five images at focus distances 41, 42, 43, 49 and 44 of the right eye may be captured. Incidentally, just three images might be sufficient. Images at the distances 41, 42, 43, 49 and 44 as designated by the lines may be regarded as images 41, 42, 43, 49 and 44, respectively. These images may be a cropped image 31 of image 29 from <figref idref="f0003">Figure 3</figref> taken at various focus distances. Images 41, 42, 49 and 44, as shown in <figref idref="f0011">Figure 10</figref>, may be regarded as out of focus as indicated by blurriness, poor clarity, low contrast, a lack of information discernable in iris 46, and so forth in view of image 43, to an observer. However, mechanisms are available to indicate quality of focus. From left to right of the images, as the focus changes as the focus distance is changed, iris 46 details or information appears to be most unambiguous and discernable, and thus best focused in image 43. However, the best focus of the image 43 may be machine evaluated within system 10 via an aspect of computer 30 in view of the intensity variance / contrast and to be detected before a human eye sees it.

0034Specifically, graph 35 shows the intensity variance increasing as the focus is improving for the sequence of lines 41 through 43. At lines 49 and 44 the focus appears to degrade as the intensity variance decreases. It may be noted that the best focus may be at line 43 which appears within the depth of field 62 at the peak of the intensity variance of graph 35, which coincides with the capturing of image 43 within the depth of field 62.

0035<figref idref="f0012">Figure 11</figref> is a diagram of an iris 46 image 69 cropped from the focused captured image 43 in <figref idref="f0011">Figure 10</figref>. This iris image 69 may be available for analysis, storage, matching, and so forth.

0036In the present specification, some of the matter may be of a hypothetical or prophetic nature although stated in another manner or tense.

0037The invention has been described with respect to at least one illustrative example.