Adapted vocabularies for matching image signatures with fisher vectors

BACKGROUND

0001The exemplary embodiment relates to image processing and finds particular application in connection with object detection in images.

0002There are many cases where it is desirable to match objects in images acquired by different cameras in different locations. For example, still cameras or video cameras may be positioned to acquire images for use in automated or semi-automated toll assessment for toll roads and bridges, automated monitoring of a parking facility, camera based enforcement of speed limits or other traffic regulations, monitoring of carpool lanes, roadway usage studies, and the like. Depending upon the application, the vehicle images that are acquired may be an image of the entire vehicle, or an image of a portion of the vehicle, such as the rear license plate.

0003One problem with matching an object in different images (referred to as re-identification) is that the imaging conditions may be different. The difference in imaging conditions may be due to various reasons, such as cameras placed at different angles, differences in backgrounds, lighting conditions, due for example, to the time of the day or different weather conditions, camera settings, camera resolution or other camera characteristics, amount of motion blur, and post-processing. In general, if the difference in imaging conditions is significant, then it may impact computer vision tasks, such as object recognition or image matching. One reason is that even when the same features are extracted in both instances, the imaging conditions can strongly affect the feature distribution. This means that the assumptions of the classifier trained for one set of conditions do not always hold for the other.

0004For image matching, a feature-based representation of a captured image is often generated. For example, one method of representing an image or a part of an image is with a Fisher Vector (FV). In this method, it is assumed that a generative model exists (such as a Gaussian Mixture Model (GMM)) from which descriptors of image patches are emitted, and the Fisher Vector components are the gradient of the log-likelihood of the descriptor with respect to one or more parameters of the model. Each patch used for training can thus be characterized by a vector of weights, one (or more) weight(s) for each of a set of Gaussian functions forming the mixture model. Given a new image, a representation can be generated (often called an image signature) based on the characterization of its patches with respect to the trained GMM.

0005In a typical transportation application, cameras are placed at various strategic locations: for example, at various toll booths, and each camera is independently trained and thereafter used to generate representations of vehicles at (or passing through) the location. If two representations match, it can be assumed that the vehicles are the same. However, even small variations between the images captured with different cameras can impact performance significantly.

0006Domain adaption techniques have been developed for adapting data from one domain to use in another. <nplcit id="ncit0001" npl-type="s"><text>Jiang, J., "A literature survey on domain adaptation of statistical classifiers," Technical report pp. 1 - 12 (2008</text></nplcit>), and <nplcit id="ncit0002" npl-type="s"><text>Beijbom, O. "Domain adaptations for computer vision applications," Technical report, arXiv:1211.4860v1 [cs.CV] 20 pp. 1 - 9 (Nov. 2012</text></nplcit>) provide surveys focusing on learning theory and natural language processing applications and computer vision applications. Some approaches focus on transforming the feature space in order to bring the domains closer. In some cases, an unsupervised transformation, generally based on PCA projections, is used. See, <nplcit id="ncit0003" npl-type="s"><text>Gopalan, R., et al., "Domain adaptation for object recognition: An unsupervised approach," ICCV, pp. 999 - 1006 (2011</text></nplcit>); <nplcit id="ncit0004" npl-type="s"><text>Gong, B., et al., "Geodesic flow kernel for unsupervised domain adaptation," CVPR, pp. 2066 - 2073 (2012</text></nplcit>); and <nplcit id="ncit0005" npl-type="s"><text>Fernando, B., et al., "Unsupervised visual domain adaptation using subspace alignment," ICCV, pp. 2960 - 2967 (2013</text></nplcit>). In others, metric learning that exploits class labels (in general both in the source and in the target domain) is used to learn a transformation of the feature space such that in this new space the instances of the same class become closer to each other than to instances from other classes, independently of the domain to which they belong. See,<nplcit id="ncit0006" npl-type="s"><text> Zha, Z.-J., et al., "Robust distance metric learning with auxiliary knowledge," IJCAI, pp 1327 - 1332 (2009</text></nplcit>); <nplcit id="ncit0007" npl-type="s"><text>Saenko, K., et al., "Adapting visual category models to new domains," ECCV, Vol. 6314 of Lecture Notes in Computer Science, pp. 213 - 226 (2010</text></nplcit>); <nplcit id="ncit0008" npl-type="s"><text>Kulis, B., et al., "What you saw is not what you get: Domain adaptation using asymmetric kernel transforms," CVPR, pp. 1785 - 1792 (2011</text></nplcit>); and <nplcit id="ncit0009" npl-type="s"><text>Hoffman, J., et al., "Discovering latent domains for multisource domain adaptation," ECCV, Vol. Part II, pp. 702 - 715 (2012</text></nplcit>).

0007Many of these techniques are geared toward classification problems and would therefore be difficult to apply to a matching problem, such as re-identification, where there is no notion of class. Others require significant amounts of training data, which is not practical for many applications.

0008The exemplary embodiment provides a system and method for generating image representations, such as Fisher Vectors, which reduces the effect of difference in imaging conditions on image matching.

INCORPORATION BY REFERENCE

0009The following references, the disclosures of which are incorporated herein by reference in their entireties, are mentioned:

0010Methods for computing Fisher Vectors are described, for example, in <patcit id="pcit0001" dnum="US20120076401A"><text>U.S. Pub. No. 20120076401, published March 29, 2012, entitled IMAGE CLASSIFICATION EMPLOYING IMAGE VECTORS COMPRESSED USING VECTOR QUANTIZATION, by Jorge Sánchez, et al.</text></patcit>, <patcit id="pcit0002" dnum="US20120045134A"><text>U.S. Pub. No. 20120045134, published February 23, 2012, entitled LARGE SCALE IMAGE CLASSIFICATION, by Florent Perronnin, et al.</text></patcit>, <nplcit id="ncit0010" npl-type="s"><text>Jorge Sánchez, et al., "High-dimensional signature compression for large-scale image classification," in CVPR 2011</text></nplcit>, <nplcit id="ncit0011" npl-type="s"><text>Jorge Sánchez and Thomas Mensink, "Improving the fisher kernel for large-scale image classification," Proc. 11th European Conference on Computer Vision (ECCV): Part IV, pp. 143-156 (2010</text></nplcit>), <nplcit id="ncit0012" npl-type="s"><text>Jorge Sánchez, et al., "Image Classification with the Fisher Vector: Theory and Practice," International Journal of Computer Vision (IJCV) 105(3): 222 - 245 (2013</text></nplcit>), hereinafter, Sánchez 2013.

0011<patcit id="pcit0003" dnum="US47721514A"><text>U.S. Application Ser. No. 14/477,215, filed September 4, 2014, entitled DOMAIN ADAPTATION FOR IMAGE CLASSIFICATION WITH CLASS PRIORS, by Boris Chidlovskii and Gabriela Csurka</text></patcit>, discloses a labeling system with a boosting classifier trained to classify an image belonging to a target domain and represented by a feature vector. Labeled feature vectors representing training images for both the target domain and a set of source domains are provided for training. Training involves generating base classifiers and base classifier weights of the boost classifier in an iterative process. At one of the iterations, a set of sub-iterations is performed, in which a candidate base classifier is trained on a training set combining the target domain training set and the source domain training set and the candidate base classifier with lowest error for the target domain training set is selected. Given a feature vector representing the image to be labeled, a label is generated for the image using the learned weights and selected candidate base classifiers.

0012<patcit id="pcit0004" dnum="US20090144033A"><text>U.S. Pub. No. 20090144033, published June 4, 2009, entitled OBJECT COMPARISON, RETRIEVAL, AND CATEGORIZATION METHODS AND APPARATUSES, by Yan Liu, et al</text></patcit>., discloses adapting N universal mixture model components to a first object to generate N corresponding first object mixture model components, where N is an integer greater than or equal to two; and generating a similarity measure based on component-by-component comparison of the N first object mixture model components with corresponding N second object mixture model components obtained by adaptation of the N universal mixture model components to a second object.

0013<patcit id="pcit0005" dnum="US50483714A"><text>U.S. Application Ser. No. 14/504,837, filed October 2, 2014, entitled SYSTEM FOR DOMAIN ADAPTATION WITH A DOMAIN-SPECIFIC CLASS MEANS CLASSIFIER, by Gabriela Csurka, et al.</text></patcit>, and <patcit id="pcit0006" dnum="US20140029839A"><text>U.S. Pub. No. 20140029839, published January 30, 2014, entitled METRIC LEARNING FOR NEAREST CLASS MEAN CLASSIFIERS, by Thomas Mensink, et al.</text></patcit> disclose methods for learning classifiers using out-of-domain labeled data.

0014The following relate generally to methods for vehicle and license plate recognition: <patcit id="pcit0007" dnum="US05499813A"><text>U.S. Patent Application No. 14/054,998, filed October 16, 2013, entitled "DELAYED VEHICLE IDENTIFICATION FOR PRIVACY ENFORCEMENT," by José Antonio Rodríguez Serrano, et al.</text></patcit>; <patcit id="pcit0008" dnum="US49741714A"><text>U.S. Application Ser. No. 14/497417, filed 26 September, 2014, entitled MULTI-QUERY PRIVACY-PRESERVING PARKING MANAGEMENT SYSTEM AND METHOD, by José Antonio Rodriguez-Serrano</text></patcit>; <patcit id="pcit0009" dnum="US20130129151A"><text>U.S. Pub. No. 20130129151, published May 23, 2013, entitled METHODS AND SYSTEMS FOR IMPROVED LICENSE PLATE SIGNATURE MATCHING BY SIMILARITY LEARNING ON SYNTHETIC IMAGES, by José Antonio Rodríguez Serrano, et al.</text></patcit>; <patcit id="pcit0010" dnum="US20130129152A"><text>U.S. Pub. No. 20130129152, published May 23, 2013, entitled METHODS AND SYSTEMS FOR IMPROVING YIELD IN WANTED VEHICLE SEARCHES, by José Antonio Rodríguez Serrano, et al.</text></patcit>; <patcit id="pcit0011" dnum="US20130182909A"><text>U.S. Pub. No. 20130182909, published July 18, 2013, entitled IMAGE SEGMENTATION BASED ON APPROXIMATION OF SEGMENTATION SIMILARITY, by José Antonio Rodríguez Serrano</text></patcit>; <patcit id="pcit0012" dnum="US20130259314A"><text>U.S. Pub. No. 20130259314, published October 3, 2013, entitled METHODS AND SYSTEMS FOR ENHANCING THE PERFORMANCE OF AUTOMATED LICENSE PLATE RECOGNITION APPLICATIONS UTILIZING MULTIPLE RESULTS, by Vladimir Kozitsky, et al.</text></patcit>; <patcit id="pcit0013" dnum="US20130336538A"><text>U.S. Pub. No. 20130336538, published December 19, 2013, entitled OCCUPANCY DETECTION FOR MANAGED LANE ENFORCEMENT BASED ON LOCALIZATION AND CLASSIFICATION OF WINDSHIELD IMAGES, by Sandra Skaff, et al.</text></patcit>; <patcit id="pcit0014" dnum="US20140056520A"><text>U.S. Pub. No. 20140056520, published February 27, 2014, entitled REGION REFOCUSING FOR DATA-DRIVEN OBJECT LOCALIZATION, by José Antonio Rodríguez Serrano</text></patcit>; <patcit id="pcit0015" dnum="US20140063263A"><text>U.S. Pub. No. 20140063263, published March 6, 2014, entitled SYSTEM AND METHOD FOR OBJECT TRACKING AND TIMING ACROSS MULTIPLE CAMERA VIEWS, by Edgar A. Bernal, et al.</text></patcit>; and <patcit id="pcit0016" dnum="US20140270381A"><text>U.S. Pub. No. 20140270381, published September 18, 2014, entitled METHODS AND SYSTEM FOR AUTOMATED IN-FIELD HIERARCHICAL TRAINING OF A VEHICLE DETECTION SYSTEM, by Wencheng Wu, et al.</text></patcit>

BRIEF DESCRIPTION

0015In accordance with one aspect of the exemplary embodiment, a method includes providing a universal generative model of local descriptors, adapting the universal generative model to a first camera to obtain a first camera-dependent generative model. The universal generative model is also adapted to a second camera to obtain a second camera-dependent generative model (or the universal generative model can be used as the second camera-dependent generative model). From a first image captured by the first camera, a first image-level descriptor is extracted, using the first camera-dependent generative model. From a second image captured by the second camera, a second image-level descriptor is extracted using the second camera-dependent generative model. A similarity is computed between the first image-level descriptor and the second image-level descriptor. Information is output, based on the computed similarity.

0016At least one of the adapting the universal generative model to the first and second cameras, extracting the first and second image-level descriptors and the computing of the similarity may be performed with a computer processor.

0017In accordance with another aspect of the exemplary embodiment, a system includes memory which stores a universal generative model of local descriptors. An adaptation component adapts the universal generative model to a first camera to obtain a first camera-dependent generative model and adapts the universal generative model to a second camera to obtain a second camera-dependent generative model. A processor implements the adaptation component.

0018In accordance with another aspect of the exemplary embodiment, a method for generating a system for object reidentification includes providing a universal generative model generated using local descriptors extracted from images in a training set. The universal generative model is adapted to a first camera to obtain a first camera-dependent generative model using local descriptors extracted from images captured by the first camera. The universal generative model is adapted to a second camera to obtain a second camera-dependent generative model using local descriptors extracted from images captured by the first camera. A component is provided for computing at least one of an image-level representation of a first image using the first camera-dependent generative model, and an image-level representation of a first image using the first camera-dependent generative model. A component is provided for computing a similarity between the first image-level descriptor and the second image-level descriptor.

0019The adaptation may be performed with a computer processor.

BRIEF DESCRIPTION OF THE DRAWINGS

0020<ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">FIGURE 1</figref> is functional block diagram of a system for object re-identification in accordance with one aspect of the exemplary embodiment;</li><li><figref idref="f0002 f0003">FIGURE 2</figref>, which for ease of illustration has been split into <figref idref="f0002">FIGURE 2A</figref> and <figref idref="f0003">FIGURE 2B</figref>, is flow chart illustrating a method for object re-identification in accordance with another aspect of the exemplary embodiment;</li><li><figref idref="f0004">FIGURE 3</figref> graphically illustrates the generation of adapted GMMs;</li><li><figref idref="f0005">FIGURE 4</figref> is a plot of <i>p</i>(<i>k</i>|<i>x_i</i>), averaged across the features <i>x_i</i> from each lane before adaptation; and</li><li><figref idref="f0006">FIGURE 5</figref> is a plot of <i>p</i>(<i>k</i>|<i>x_i</i>), averaged across the features <i>x_i</i> from each lane after adaptation.</li></ul>

DETAILED DESCRIPTION

0021The exemplary embodiment relates to object matching and will be described with particular reference to object re-identification. In aspects of the exemplary embodiment, a system and method for object reidentification are described. The system and method can be used to determine whether two images contain the same object instance, for example, whether two images are of the same car (e.g., whether the same vehicle is being observed at the entry and exit of a facility), the same bicycle, the same airplane, or the like. To match two images, a representation is extracted based on local descriptors that are representative of the pixels of the image. As an example, each representation may include one or more Fisher Vectors that are representative of the whole image or representative of a region of interest (ROI) in the image predicted to include the object (e.g., a license plate). A similarity metric between the two Fisher Vectors is then computed. The Fisher Vectors encode deviations with respect to a generative model. A threshold on the similarity metric may then be used to decide whether the two images are of the same object instance.

0022In other aspects of the exemplary embodiment, a system and method for generating the object reidentification system are described.

0023As noted above, where the imaging conditions are different for capture of two images of an object, then the extracted Fisher Vectors may not be similar, despite containing the same object instance. In the exemplary embodiment, a universal generative model is learned <i>a priori.</i> A shift in the imaging conditions is reflected in the form of a corresponding shift in the model. To counter this shift, the parameters of the generative model are adapted in an unsupervised manner using images captured under each of the two imaging conditions (e.g., from two cameras). The result is two camera-specific generative models, each with a respective "adapted visual vocabulary". The deviations from these models encoded in the form of Fisher Vectors, is then shifted towards deviations in image content, rather than the imaging conditions.

0024With reference to <figref idref="f0001">FIGURE 1</figref>, a system <b>10</b> for object re-identification is shown. The system <b>10</b> receives test images <b>12, 14</b> from cameras <b>16, 18</b> and processes the images <b>12, 14</b> to determine whether they include the same object.

0025As used herein, the term "object" refers to an inanimate object (or group of inanimate objects), such as a vehicle, license plate, other article of manufacture, building, or the like, or to an animate object (or group of inanimate objects), such as a person or group of people, or an animal or group of animals. In particular, the term "object" refers to a physical object that can be imaged by cameras <b>16, 18</b> used by the system.

0026In the exemplary embodiment, the object is a vehicle such as a motorized vehicle, or its license plate which includes an identifier for the vehicle, including the license number of the vehicle. However, it is to be appreciated that other moving objects and their identifiers are also contemplated, such as people and their respective fingerprints and/or eye scans, shipping parcels and their tracking or address information, and the like. Vehicle identification may be desired at multiple locations and/or at different times, for example at the entry and exit of a car park or at different points on a road. For the sake of simplification, two locations X and Y which are spaced from each other by a distance Z are considered, although it is to be appreciated that X and Y may be at or close to the same location.

0027In one example embodiment, the aim is to find correspondences between the vehicle identities at X and Y, and when a correspondence between two identities has been established, to determine whether a certain condition is met that depends on information captured at X and Y. If the condition is met, a specified action is triggered, such as license plate recognition.

0028As used herein, the term "camera" refers to an imaging device that is capable of acquiring an image of an object of interest. By way of some nonlimiting examples, the camera may be: a conventional camera including photographic film or a digital imaging array (e.g., CCD array optionally with color filters) that is capable of acquiring color or monochrome images similar to those that would be observed by human vision; or an x-ray scanner including an x-ray sensitive imaging array operating in conjunction with an x-ray source to generate an x-ray projection image; or a computed tomography scanner; or an infrared camera of the type used to acquire night or low-light images; or a document scanner operating to generate document page images; or the like. It is contemplated for the camera to be specialized for the specific object labeling task, for example including wavelength selective filters so as to be sensitive to wavelength(s) emitted by a fluorescent dye that is expected to be present on or in the object. In general the camera may be a still camera (that is, acquires a single static image) or a video camera (that is, acquires a time sequence of images commonly called "frames" - here the image pre-processing may entail selecting a frame of a video burst that optimally captures the object of interest). In some types of cameras, such as a computed tomography scanner, data acquired from the object by the camera may undergo image reconstruction processing in order to generate the image.

0029The system <b>10</b> illustrated in <figref idref="f0001">FIGURE 1</figref> includes memory <b>20</b> which stores instructions <b>22</b> for performing the method described with reference to <figref idref="f0002 f0003">FIGURE 2</figref> and a processor device <b>24</b> in communication with the memory for executing the instructions. One or more network interfaces <b>26, 28</b> are provided for receiving the images <b>12, 14</b> (or features extracted therefrom) from the cameras <b>16, 18</b> and outputting information <b>30</b> based thereon. Hardware components <b>20, 24, 26, 28</b> of the system communicate via a data/control bus <b>32.</b> A set <b>34</b> of general training images, may be stored in a database <b>38,</b> e.g., in system memory <b>20</b> or in associated memory communicatively connected with the system. The training images <b>34</b> are not specific to either one of the cameras <b>16, 18,</b> but may be representative of the type of image to be captured by the cameras.

0030Briefly, the instructions <b>22</b> include a feature extraction component <b>40,</b> a training component <b>42,</b> an adaptation component <b>44,</b> a signature generation component <b>46,</b> a matching component <b>48,</b> optionally, a process implementation component <b>50,</b> and an information output component <b>52.</b>

0031The feature extraction component <b>40</b> extracts local features and generates local descriptors <b>54, 56</b> from images, such as images <b>12, 14</b> and training images <b>34,</b> etc. As will be appreciated, the training images <b>34</b> and images <b>12, 14</b> acquired by the cameras may be cropped or otherwise preprocessed prior to extraction of the local descriptors.

0032The training component <b>42</b> trains a universal generative model <b>60</b> of local descriptors using the local descriptors generated from the set <b>34</b> of general training images as training data. The training images <b>34</b> may have been acquired from a set of cameras which, in one embodiment, may include the cameras <b>16, 18.</b> The training images <b>34</b> may thus be similar, in visual content, to the images acquired by cameras <b>16, 18.</b> The generative model <b>60</b> is referred to as a Universal Background Model (UBM). The UBM <b>60</b> may be a Gaussian mixture model (GMM) which includes parameters for a set of Gaussian functions.

0033The adaptation component <b>44</b> adapts the generative model <b>60</b> to each camera <b>16, 18</b> to obtain a respective camera-specific model <b>62, 64,</b> which in the exemplary embodiment are GMMs. To adapt the UBM <b>60,</b> local descriptors extracted from a respective set of unlabeled training samples <b>66, 68</b> captured by each of the cameras <b>16, 18</b> (or specific to the capture locations) are used to adapt the generative model <b>60</b> to suit the different imaging conditions at the respective locations. As will be appreciated, while two cameras <b>16, 18</b> are illustrated, any number of cameras may be provided with a respective camera-dependent model <b>62, 64.</b>

0034Given the local descriptors <b>54</b> extracted from a first test image <b>12</b> and the generative model <b>62</b> of the first camera <b>16,</b> the signature generation component <b>46,</b> extracts a first image-level descriptor (image signature) <b>70,</b> denoted SI_x,. Given the local descriptors <b>56</b> extracted from a second test image <b>14</b> and the generative model <b>64</b> of the second camera <b>18,</b> the signature generation component <b>46</b> (which could be a separate signature generation component) extracts a second image-level descriptor <b>72,</b> denoted SI_Y. The image-level descriptors <b>70, 72</b> may each be a soft bag-of-visual words or a Fisher Vector.

0035The matching component <b>48</b> computes a matching between the image signatures <b>70, 72,</b> computed from the different images, by computing a similarity, e.g., a cosine distance between SI_x and SI_Y. If the similarity at least meets a threshold, the two images (or the objects captured within them), may be inferred to be a match, i.e., the same object.

0036The process implementation component <b>50</b> may implement a process based on the output of the matching component. For example, in the case of images of vehicles, if the vehicles are a match, then license plate information may be extracted from one or both of the images. This may be conditional on a further condition being met, such as a computed speed of the vehicle exceeding a threshold, or the like. This may be computed based on information such as timestamps M_X, M_Y<b>74</b>, <b>76,</b> associated with the capture of the respective images <b>12, 14.</b>

0037The information output component <b>52</b> outputs information <b>30,</b> e.g., to an output device <b>80,</b> such as a remote computer, printer, display device (e.g., computer screen), or remote memory storage device, or combination thereof, based on the inference made by the matching component or other information, based on the information computed by the process implementation component <b>50.</b>

0038The system <b>10</b> may communicate, via input/output interface <b>28,</b> with one or more of a display device, such as an LCD screen or computer monitor, for displaying information to users, and a user input device, such as a keyboard or touch or writable screen, and/or a cursor control device, such as mouse, trackball, or the like, for inputting text and for communicating user input information and command selections to the processor <b>24.</b> The display device and user input device are illustrated as being part of a client computing device <b>80,</b> although in other embodiments, they may be directly linked to the computer <b>82</b> hosting the system.

0039The system <b>10</b> may be resident on one or more computing devices <b>82,</b> such as a PC, such as a desktop, a laptop, palmtop computer, portable digital assistant (PDA), server computer, cellular telephone, tablet computer, pager, combination thereof, or other computing device capable of executing instructions for performing the exemplary method. As will be appreciated parts of the system <b>10</b> may be distributed over two or more computing devices. For example, local descriptors for images <b>12, 14</b> may be computed by processing components <b>84, 86</b> of the respective cameras <b>16, 18,</b> and may be stored in database <b>38</b> accessible to the computing device <b>82.</b>

0040The memory <b>20</b> may represent any type of non-transitory computer readable medium such as random access memory (RAM), read only memory (ROM), magnetic disk or tape, optical disk, flash memory, or holographic memory. In one embodiment, the memory <b>20</b> comprises a combination of random access memory and read only memory. In some embodiments, the processor <b>24</b> and memory <b>20</b> may be combined in a single chip.

0041The network interface <b>26, 28</b> allows the computer <b>82</b> to communicate with other devices via a wired or wireless link, such as a computer network, such as a local area network (LAN) or wide area network (WAN), or the Internet, and may comprise a modulator/demodulator (MODEM) a router, a cable, and and/or Ethernet port.

0042The digital processor <b>24</b> can be variously embodied, such as by a single-core processor, a dual-core processor (or more generally by a multiple-core processor), a digital processor and cooperating math coprocessor, a digital controller, or the like. The digital processor <b>24,</b> in addition to controlling the operation of the computer <b>82,</b> executes instructions stored in memory <b>20</b> for performing the method outlined in <figref idref="f0002 f0003">FIGURE 2</figref>.

0043In some embodiments, the system <b>10</b> can be used simply as a system for generating parts of the reidentification system, e.g., containing the adaptation component <b>44,</b> while other components may be omitted. In other embodiments, the system <b>10</b> can be used for employing the already adapted camera-dependent generative models <b>62, 64,</b> while other components, such as the training and adaptation components <b>42, 44</b> may be omitted.

0044The term "software," as used herein, is intended to encompass any collection or set of instructions executable by a computer or other digital system so as to configure the computer or other digital system to perform the task that is the intent of the software. The term "software" as used herein is intended to encompass such instructions stored in storage medium such as RAM, a hard disk, optical disk, or so forth, and is also intended to encompass so-called "firmware" that is software stored on a ROM or so forth. Such software may be organized in various ways, and may include software components organized as libraries, Internet-based programs stored on a remote server or so forth, source code, interpretive code, object code, directly executable code, and so forth. It is contemplated that the software may invoke system-level code or calls to other software residing on a server or other location to perform certain functions.

0045<figref idref="f0002 f0003">FIGURE 2</figref> illustrates a method for object re-identification, which may be performed with the system of <figref idref="f0001">FIGURE 1</figref>. Starting with <figref idref="f0002">FIGURE 2A</figref>, the method begins at S100.

0046At S102, a universal generative model UBM <b>60</b> of local descriptors is provided. This may include training the UBM <b>60</b> with the training component <b>42.</b> The UBM <b>60</b> includes parameters for a set of <i>K</i> Gaussian functions (also referred to as Gaussians or mixtures). In some embodiments, the universal generative model UBM <b>60</b> may have been previously trained and may simply be stored in memory <b>20</b> accessible to the system <b>10.</b>

0047At S104, the universal generative model <b>60</b> is adapted to a first camera <b>16</b> to obtain a first camera-dependent generative model <b>62.</b> This is performed using descriptors extracted from a first set of training images <b>66</b> captured by the first camera <b>16</b> (i.e., using descriptors only from images captured by the first camera and not by the second camera). The first camera-dependent generative model <b>62</b> includes parameters for a set of <i>K</i> Gaussians generated by adapting the parameters of the UBM <b>60</b> using the descriptors extracted from the first set of training images <b>66.</b>

0048At S106, the universal generative model <b>60</b> is adapted to a second camera <b>18</b> to obtain a second camera-dependent generative model <b>64.</b> This is performed using descriptors extracted from a second set of training images <b>68</b> captured by the second camera <b>18</b> (i.e., using descriptors only from images captured by the second camera and not by the first camera). The second camera-dependent generative model <b>64</b> includes parameters for a set of <i>K</i> Gaussians generated by adapting the parameters of the UBM <b>60</b> using the descriptors extracted from the second set of training images <b>68.</b> GMMs <b>62</b> and <b>64</b> thus include parameters for the same number of Gaussian functions as the UBM <b>60.</b> In another embodiment, the universal generative model is used as the second camera-dependent generative model (and may thus be generated with descriptors acquired solely or primarily from images captured by that camera).

0049At S108, local descriptors <b>54</b> are extracted from a first test image <b>12</b> captured by the first camera <b>16.</b>

0050At S110, local descriptors <b>56</b> are extracted from a second test image <b>14</b> captured by the second camera <b>18.</b>

0051At S112, a first image-level descriptor <b>70</b> is generated based on the first image local descriptors <b>54</b> and the adapted generative modelu <maths id="math0001"><math display="inline"><mrow><msubsup><mi>u</mi><mi mathvariant="normal">Θ</mi><mi>c</mi></msubsup><mi mathvariant="normal">x</mi></mrow></math><img file="EP3035239A1_D0001.tif" /></maths><b>62</b> of the first camera <b>16.</b>

0052At S114, a second image-level descriptor <b>72</b> is generated based on the second image local descriptors <b>56</b> and the adapted generative model <maths id="math0002"><math display="inline"><mrow><msubsup><mi>u</mi><mi mathvariant="normal">Θ</mi><mi>c</mi></msubsup><mi mathvariant="normal">y</mi></mrow></math><img file="EP3035239A1_D0002.tif" /></maths> 64 of the second camera <b>18.</b>

0053At S116, the first and second image-level descriptors <b>70, 72</b> are compared. Continuing on to <figref idref="f0003">FIGURE 2B</figref>, if at S118, the similarity between the signatures meets a threshold (and/or is the highest similarity of a set of images being compared), then at S120, a match is confirmed.

0054At S122, a process may be implemented based on the determination of a match.

0055At S124, information <b>30</b> is output, such as whether or not there is a match or other information computed when a match is found (or in another embodiment, information may be computed when no match is found).

0056The method ends at S126 or may return to S108 and or S110 for processing a new test image or images captured by one of the cameras.

0057As will be appreciated, once the adaptation steps have been performed, the method can begin at S108. Alternatively, the method can proceed from S106 to the provision of components for performing subsequent steps of the method.

0058The method is illustrated graphically in <figref idref="f0004">FIGURE 3</figref>. In particular, it can be seen that each camera-dependent mixture model is formed by shifting (by mean adaptation) and/or scaling (by variance adaptation) at least some of the original Gaussians (shown by the ovals) of the UBM slightly to reflect the camera-specific descriptors (shown by the stars). The adapted Gaussians of the camera-dependent mixture models still represent the same general visual classes as in the UBM (as illustrated by their overlap with the original Gaussians) and thus are comparable in the image-level representations.

0059The method illustrated in <figref idref="f0002 f0003">FIGURES 2</figref> and <figref idref="f0004">3</figref> may be implemented in a computer program product that may be executed on a computer. The computer program product may comprise a non-transitory computer-readable recording medium on which a control program is recorded (stored), such as a disk, hard drive, or the like. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, or any other magnetic storage medium, CD-ROM, DVD, or any other optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, or other memory chip or cartridge, or any other non-transitory medium from which a computer can read and use. The computer program product may be integral with the computer <b>82,</b> (for example, an internal hard drive of RAM), or may be separate (for example, an external hard drive operatively connected with the computer <b>82</b>), or may be separate and accessed via a digital data network such as a local area network (LAN) or the Internet (for example, as a redundant array of inexpensive of independent disks (RAID) or other network server storage that is indirectly accessed by the computer <b>82,</b> via a digital network).

0060Alternatively, the method may be implemented in transitory media, such as a transmittable carrier wave in which the control program is embodied as a data signal using transmission media, such as acoustic or light waves, such as those generated during radio wave and infrared data communications, and the like.

0061The exemplary method may be implemented on one or more general purpose computers, special purpose computer(s), a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, a hardwired electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PLD, PLA, FPGA, Graphical card CPU (GPU), or PAL, or the like. In general, any device, capable of implementing a finite state machine that is in turn capable of implementing the flowchart shown in <figref idref="f0002 f0003">FIGURE 2</figref>, can be used to implement the method for object reidentification. As will be appreciated, while the steps of the method may all be computer implemented, in some embodiments one or more of the steps may be at least partially performed manually.

0062Further details of the system and method will now be described.

Image capture

0063Captured images <b>12, 14</b> (I_X and I_Y) and training images <b>34, 66, 68</b> may be received by the system <b>10</b> in any convenient file format, such as JPEG, GIF, JBIG, BMP, TIFF, or the like or other common file format used for images and which may optionally be converted to another suitable format prior to processing. The input images may be stored in data memory during processing. The images may be individual images, such as photographs, or images extracted from sequences of images, such as video images. In general, each input digital image includes image data for an array of pixels forming the image. The image data may include colorant values, such as grayscale values, for each of a set of color separations, such as <i>L*a*b*</i> or RGB, or be expressed in another other color space in which different colors can be represented. In general, "grayscale" refers to the optical density value of any single color channel, however expressed (<i>L*a*b*</i>, RGB, YCbCr, etc.). The method is suitable for black and white (monochrome) images and multicolor images. The word "color" is used to refer to any aspect of color which may be specified, including, but not limited to, absolute color values, such as hue, chroma, and lightness, and relative color values, such as differences in hue, chroma, and lightness. In some embodiments, color can refer to a non-visible region of the electromagnetic spectrum, such as the Near Infrared (NIR) region, which is from about 800 nm to 2500 nm.

0064The image capture at a given location X, Y may be triggered in any suitable manner. In one embodiment, a loop sensor may be locally positioned, e.g., in the ground, which detects the presence of a vehicle and triggers a shot (and possibly a flash). In one embodiment, the camera <b>16, 18</b> shoots a video comprising a sequence of images and a motion detection algorithm or an object detection algorithm (or the combination of both) is employed which detects the presence of a vehicle in the image sequence and selects one image from the sequence accordingly. Vehicle detection techniques are well known and disclosed for example, in <patcit id="pcit0017" dnum="US4433325A"><text>U.S. Patent Nos. 4,433,325</text></patcit>, <patcit id="pcit0018" dnum="US5083200A"><text>5,083,200</text></patcit>, <patcit id="pcit0019" dnum="US5592567A"><text>5,592,567</text></patcit>, <patcit id="pcit0020" dnum="US5809161A"><text>5,809,161</text></patcit>, <patcit id="pcit0021" dnum="US5995900A"><text>5,995,900</text></patcit>, <patcit id="pcit0022" dnum="US6996255B"><text>6,996,255</text></patcit>, and <patcit id="pcit0023" dnum="US20140270381A"><text>U.S. Pub. Nos. 20140270381</text></patcit>, and <patcit id="pcit0024" dnum="US20140063263A"><text>20140063263</text></patcit>, the disclosures of all of which are incorporated herein in their entireties by reference. The image can be captured in full color, monochrome, NIR (near infrared), or a combination thereof.

0065In some embodiments, the same image capture device could be used for capturing both images I_X and I_Y, such as in a car park, for example, by rotating/moving the camera.

0066The associated metadata <b>74, 76</b> may include sufficient information to enable a determination to be made as to whether a predetermined condition has been met. One or more different types of information may be acquired, such as one or more of time stamp, current speed of the vehicle, GPS location, payment information, weather information, and the like.

Extracting local descriptors (S108, S110)

0067Local descriptors are extracted from training and test images <b>34, 66, 68, 12, 14</b> in the same manner. For example, a set of patches of the optionally cropped image are extracted, e.g., densely, on a grid at one or at multiple scales. The patches can be obtained by image segmentation, by applying specific interest point detectors, by considering a regular grid, or simply by the random sampling of image patches. In the exemplary embodiment, the patches are extracted on a regular grid, optionally at multiple scales, over the entire cropped image, or at least a part or a majority of the image. For example, at least 10 or at least 20 or at least 50 patches are extracted from each cropped image. Each patch may comprise at least 40 or at least 100 pixels, and up to 1,000,000 pixels or more.

0068For each patch, low-level features are extracted, such as shape, color and/or gradient (SIFT) features (see, <nplcit id="ncit0013" npl-type="s"><text>D. Lowe, "Distinctive image features from scale-invariant keypoints," IJCV, 2004</text></nplcit>). A patch descriptor, such as a vector or histogram, which is a representation of the extracted low level features for a patch, is generated. Based on the descriptors of all the patches, on overall image signature of the image is generated. In particular, statistics are computed on these patch descriptors and then the statistics are aggregated.

0069As an example, the local descriptors include gradient features, such as SIFT descriptors, one per patch. In one illustrative example employing SIFT features, the local descriptors are extracted from 32×32 pixel patches on regular grids (every 16 pixels) at five scales. The dimensionality of these descriptors can be reduced from 128 to 32 dimensions, for example, through Principal Component Analysis (PCA). Other suitable local descriptors which can be extracted include simple 96-dimensional color features in which a patch is subdivided into 4×4 sub-regions and in each sub-region the mean and standard deviation are computed for the three channels (R, G and B). These are merely illustrative examples, and additional and/or other features can be used.

Extracting image signatures (S112, S114)

0070Each image-level descriptor or "image signature" <b>70, 72,</b> etc., is a fixed-length vectorial representation of the (optionally cropped) image in a <i>D-</i>dimensional space. In one embodiment, the image-level descriptor is based on the Fisher Vector (FV). See, for example, <nplcit id="ncit0014" npl-type="s"><text>Perronnin and Dance, "Fisher kernels on visual vocabularies for image categorization," CVPR, 2007</text></nplcit>; <nplcit id="ncit0015" npl-type="s"><text>Perronnin, et al., "Improving the Fisher kernel for large-scale image classification", ECCV, 143-156 (2010</text></nplcit>); <nplcit id="ncit0016" npl-type="s"><text>Sánchez, et al., "High-dimensional signature compression for large-scale image classification," in CVPR 2011</text></nplcit>; <patcit id="pcit0025" dnum="US20120076401A"><text>U.S. Pub. No. 20120076401, published March 29 2012, entitled IMAGE CLASSIFICATION EMPLOYING IMAGE VECTORS COMPRESSED USING VECTOR QUANTIZATION, by Jorge Sánchez, et al.</text></patcit>; and <patcit id="pcit0026" dnum="US20120045134A"><text>U.S. Pub. No. 20120045134, published February 23, 2012, entitled LARGE SCALE IMAGE CLASSIFICATION, by Florent Perronnin, et al.</text></patcit> the disclosures of which are incorporated herein by reference in their entireties. Fisher vectors have demonstrated good results in a variety of applications including image classification, image retrieval, object detection, and person re-identification. Hence, any improvement that can be brought on top of Fisher Vectors will have a significant impact on many applications. In the following, Fisher Vector extraction is described, followed by a description of an adaptation technique.

0071Fisher Vectors show robustness in the range of photometric and geometric variability found in license plate processing. Briefly, Fisher vectors work by aggregating the local patch descriptors into a fixed-length representation. As discussed above, SIFT and/or other local descriptors are extracted from patches extracted at multiple scales on a regular grid, and their dimensionality is optionally reduced using principal component analysis (PCA). Given a camera-specific Gaussian mixture model (GMM) estimated as described below, from which it is assumed all the local descriptors are emitted, the set of patches can be characterized by a vector of weights, one weight per for each of a set of (e.g., at least 5 or 10) Gaussian functions forming the mixture model, per parameter.

0072In the present case, suppose <i>X</i> = {<i>x</i>₁, <i>x</i>₂,...,<i>x_N</i>} is a set of <i>N</i> multidimensional feature vectors (local descriptors) extracted from image patches. Let <i>u</i>_Θ be the probability density function of the generative model producing such feature vectors, where 0 represents the parameters of the probability density function. Then the Fisher Vector is given by: <maths id="math0003" num="(1)"><math display="block"><mrow><msubsup><mi mathvariant="script">G</mi><mi mathvariant="normal">Θ</mi><mi>X</mi></msubsup><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><mrow><mstyle displaystyle="false"><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><msub><mi>L</mi><mi mathvariant="normal">Θ</mi></msub><mrow><msub><mrow><mo>∇</mo></mrow><mi mathvariant="normal">Θ</mi></msub></mrow><mi>log</mi><msub><mi>u</mi><mi mathvariant="normal">Θ</mi></msub><mfenced><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow></mrow></math><img file="EP3035239A1_D0003.tif" /></maths> where <i>L</i>_Θ is the square-root of the inverse of the Fisher Information Matrix of <i>u</i>_Θ. <maths id="math0004"><math display="inline"><mrow><mrow><mstyle displaystyle="false"><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>log</mi><msub><mi>u</mi><mi mathvariant="normal">Θ</mi></msub><mfenced><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow></mrow></math><img file="EP3035239A1_D0004.tif" /></maths> is the log-likelihood of the data samples (patch descriptors) <i>x_i</i> with respect to <i>u</i>_Θ. Thus, computing its gradient (∇_Θ) gives a measure of how the parameters of the generative model <i>u</i>_Θ should be modified to better fit the data samples in <i>X</i>. See, Sánchez 2013. The gradient can be computed with respect to one or more of the parameters of the GMM, such as the mean, variance, and/or weight.

The Universal Generative Model (S102)

0073In the exemplary method, the universal generative model <i>u</i>_Θ) <b>60</b> is a <i>K-</i>component Gaussian Mixture Model (GMM) with parameters 0 = {<i>π_k</i>,<i>µ_k</i>,Σ<i>_k,k =</i> 1,...,<i>K</i>}, where <i>π_k</i>, <i>µ_k</i> and Σ<i>_k</i> are respectively the weight, mean vector, and covariance matrix of the <i>k</i>-th Gaussian. For convenience, let: <maths id="math0005" num="(2)"><math display="block"><mrow><msub><mrow><mo>∀</mo></mrow><mi>k</mi></msub><mo>:</mo><msub><mi>π</mi><mi>k</mi></msub><mo>≥</mo><mn>0</mn><mo>,</mo><mspace width="1em" /><mi>and</mi><mspace width="1em" /><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover></mrow></mstyle><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow><mo>=</mo><mn>1</mn></mrow></math><img file="EP3035239A1_D0005.tif" /></maths>

0074Thus, for any feature vector <i>x_i</i>: <maths id="math0006" num="(3)"><math display="block"><mrow><msub><mi>u</mi><mi mathvariant="normal">Θ</mi></msub><mfenced><msub><mi>x</mi><mi>i</mi></msub></mfenced><mo>=</mo><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover></mrow></mstyle><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow><msub><mi>u</mi><mi>k</mi></msub><mfenced><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></math><img file="EP3035239A1_D0006.tif" /></maths> where, <maths id="math0007" num="(4)"><math display="block"><mrow><msub><mi>u</mi><mi>k</mi></msub><mfenced><msub><mi>x</mi><mi>i</mi></msub></mfenced><mo>=</mo><mfrac><mn>1</mn><mrow><msup><mfenced><mn>2</mn><mi>π</mi></mfenced><mrow><mfrac><mi>p</mi><mn>2</mn></mfrac></mrow></msup><msup><mfenced open="|" close="|"><mi mathvariant="normal">Σ</mi></mfenced><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac></mrow></msup></mrow></mfrac><mi>exp</mi><mfenced open="{" close="}"><mo>-</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><mfenced><msub><mi>x</mi><mi>i</mi></msub><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub></mfenced><msup><mi>ʹΣ</mi><mrow><mo>-</mo><mn>1</mn></mrow></msup><mfenced><msub><mi>x</mi><mi>i</mi></msub><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub></mfenced></mfenced></mrow></math><img file="EP3035239A1_D0007.tif" /></maths>

0075<i>p</i> is the dimensionality of the feature vectors, <i>x_i</i>. The GMM <b>60</b> is learned <i>apriori</i> using local descriptors extracted from training images <b>34</b> and may be referred to as a Universal Background Model (UBM (following the convention from speech processing, see, for example, <nplcit id="ncit0017" npl-type="s"><text>Douglas A. Reynolds, et al., "Speaker verification using adapted Gaussian mixture models," Digital Signal Processing, 10(13):19 - 41 (2000</text></nplcit>)).

0076After computing <i>L</i>_Θ under some assumptions as in Sánchez 2013, and considering diagonal covariance matrices Σ<i>_k</i>, the equations (derived from Equation (1)) for each of the different parameters of the GMM can be deduced. However, it has been found that deriving the Fisher Vectors using only the means gives good performance in matching applications. Hence, for the gradient with respect to the means ∑<i>_k</i>: <maths id="math0008" num="(5)"><math display="block"><mrow><msubsup><mi mathvariant="script">G</mi><mrow><msub><mi>μ</mi><mi>k</mi></msub></mrow><mi>X</mi></msubsup><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><mfrac><mn>1</mn><mrow><msqrt><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow></msqrt></mrow></mfrac><mrow><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><mfenced><mfrac><mrow><msub><mi>x</mi><mi>i</mi></msub><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub></mrow><mrow><msub><mi>σ</mi><mi>k</mi></msub></mrow></mfrac></mfenced></mrow></mrow></mrow></math><img file="EP3035239A1_D0008.tif" /></maths> where: <maths id="math0009" num="(6)"><math display="block"><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><mo>=</mo><mfrac><mrow><msub><mi>π</mi><mi>k</mi></msub><msub><mi>u</mi><mrow><msub><mi>θ</mi><mi>k</mi></msub></mrow></msub><mfenced><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow><mrow><mrow><mstyle displaystyle="false"><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover></mrow></mstyle><mrow><msub><mi>π</mi><mi>j</mi></msub><msub><mi>u</mi><mrow><msub><mi>θ</mi><mi>j</mi></msub></mrow></msub><mfenced><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow></mrow></mfrac></mrow></math><img file="EP3035239A1_D0009.tif" /></maths>

0077Here, <i>σ_k</i> are the diagonal entries of the diagonal covariance matrix for the <i>k</i>-th Gaussian and division implies an <i>element-wise</i> operation. Also, <i>u_{θ<sub2>k</sub2>}</i>(<i>x_i</i>) is the value of the pdf for <i>k</i>-th Gaussian at <i>x_i.</i> Note that <maths id="math0010"><math display="inline"><mrow><msubsup><mi mathvariant="script">G</mi><mrow><msub><mi>μ</mi><mi>k</mi></msub></mrow><mi>X</mi></msubsup></mrow></math><img file="EP3035239A1_D0010.tif" /></maths> is a <i>p</i> × 1 vector. The final Fisher Vector is the concatenation of the gradients <maths id="math0011"><math display="inline"><mrow><msubsup><mi mathvariant="script">G</mi><mrow><msub><mi>μ</mi><mi>k</mi></msub></mrow><mi>X</mi></msubsup></mrow></math><img file="EP3035239A1_D0011.tif" /></maths> for all <i>K</i> Gaussian functions in the mixture model. Hence, it is a <i>Kp</i>-dim vector. This may then be sign-square-rooted and ℓ₂-normalized. See, Sánchez 2013.

0078Suppose that cameras <b>16, 18</b> have differing imaging conditions. One approach to counter balance the imaging conditions would be to learn a GMM for each camera independently from scratch. However, this does not provide correspondence among different Gaussians from different GMMs. Thus, comparing Fisher Vectors across cameras would be difficult, since Fisher Vectors are constructed as a concatenation of gradient statistics for all the Gaussian s. In the present method the model parameters of the UBM <b>60</b> are adapted for the images from different cameras, to yield camera-specific GMMs <b>62, 64.</b> Afterwards, when these camera-specific GMMs <b>62, 64</b> are used to compute Fisher Vectors, they primarily encode the deviations which are specific to a particular image and not the differences in imaging conditions.

0079As will be appreciated, bag-of-word image-level descriptors are also generated by a Gaussian Mixture Model and thus the method is also applicable to these image-level descriptors. See, e.g., <nplcit id="ncit0018" npl-type="s"><text>Csurka, et al., "Visual Categorization with Bags of Key-points," in ECCV Workshop on Statistical Learning for Computer Vision (2004</text></nplcit>), <patcit id="pcit0027" dnum="US20080069456A"><text>U.S. Pub. No. 20080069456, published March 20, 2008, entitled BAGS OF VISUAL CONTEXT-DEPENDENT WORDS FOR GENERIC VISUAL CATEGORIZATION, by Florent Perronnin</text></patcit>, and <patcit id="pcit0028" dnum="US20120045134A"><text>U.S. Pub. No. 20120045134, published February 23, 2012, entitled LARGE SCALE IMAGE CLASSIFICATION, by Florent Perronnin, et al.</text></patcit>, the disclosures of which are incorporated by reference, for a fuller description of bag-of-word image-level descriptors.

Adapting the generative model (S104, S106)

0080The exemplary image-level descriptors <b>70, 72</b> are Fisher vectors, which are extracted from the test images <b>12, 14</b> in a similar manner. However, the generative models <b>62</b> and <b>64</b> used to generate them are different.

0081To explain the need for adaptation further, consider Equation (5), the gradient of the Fisher vector with respect to the mean vector: <maths id="math0012" num="(7)"><math display="block"><mrow><mtable columnalign="left"><mtr><mtd><mspace width="1em" /></mtd><mtd><msubsup><mi mathvariant="script">G</mi><mrow><msub><mi>μ</mi><mi>k</mi></msub></mrow><mi>X</mi></msubsup><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><mfrac><mn>1</mn><mrow><msqrt><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow></msqrt></mrow></mfrac><mrow><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><mfenced><mfrac><mrow><msub><mi>x</mi><mi>i</mi></msub><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub></mrow><mrow><msub><mi>σ</mi><mi>k</mi></msub></mrow></mfrac></mfenced></mrow></mrow></mtd></mtr><mtr><mtd><mspace width="1em" /></mtd><mtd><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><mfrac><mn>1</mn><mrow><msub><mi>σ</mi><mi>k</mi></msub><msqrt><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow></msqrt></mrow></mfrac><mfenced><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><msub><mi>x</mi><mi>i</mi></msub></mrow><mo>-</mo><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><msub><mi>μ</mi><mi>k</mi></msub></mrow></mfenced></mtd></mtr><mtr><mtd><mspace width="1em" /></mtd><mtd><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><mfrac><mn>1</mn><mrow><msub><mi>σ</mi><mi>k</mi></msub><msqrt><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow></msqrt></mrow></mfrac><mfenced><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><msub><mi>x</mi><mi>i</mi></msub></mrow><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mfenced></mtd></mtr><mtr><mtd><mspace width="1em" /></mtd><mtd><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><mfrac><mn>1</mn><mrow><msub><mi>σ</mi><mi>k</mi></msub><msqrt><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow></msqrt></mrow></mfrac><mfenced><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><msub><mi>x</mi><mi>i</mi></msub></mrow><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub><msub><mrow><mover><mi>n</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub></mfenced></mtd></mtr><mtr><mtd><mspace width="1em" /></mtd><mtd><mo>=</mo><mfrac><mrow><msub><mrow><mover><mi>n</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub></mrow><mi>N</mi></mfrac><mfrac><mn>1</mn><mrow><msub><mi>σ</mi><mi>k</mi></msub><msqrt><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow></msqrt></mrow></mfrac><mfenced><mfrac><mn>1</mn><mrow><msub><mrow><mover><mi>n</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub></mrow></mfrac><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced><msub><mi>x</mi><mi>i</mi></msub></mrow><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub></mfenced></mtd></mtr><mtr><mtd><mo>⇒</mo><msubsup><mi mathvariant="script">G</mi><mrow><msub><mi>μ</mi><mi>k</mi></msub></mrow><mi>X</mi></msubsup></mtd><mtd><mo>=</mo><mfrac><mrow><msub><mrow><mover><mi>π</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub></mrow><mrow><msub><mi>σ</mi><mi>k</mi></msub><msqrt><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow></msqrt></mrow></mfrac><mfenced><msub><mrow><mover><mi>m</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub><mo>-</mo><msub><mi>μ</mi><mi>k</mi></msub></mfenced></mtd></mtr></mtable></mrow></math><img file="EP3035239A1_D0012.tif" /></maths> where: <maths id="math0013"><math display="block"><mrow><msub><mrow><mover><mi>n</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub><mo>=</mo><mrow><mstyle displaystyle="false"><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow></mrow></math><img file="EP3035239A1_D0013.tif" /></maths><maths id="math0014"><math display="block"><mrow><msub><mrow><mover><mi>π</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub><mo>=</mo><mfrac><mrow><msub><mrow><mover><mi>n</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub></mrow><mi>N</mi></mfrac></mrow></math><img file="EP3035239A1_D0014.tif" /></maths><maths id="math0015"><math display="block"><mrow><msub><mrow><mover><mi>m</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mrow><mover><mi>n</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi></msub></mrow></mfrac><mrow><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow><msub><mi>x</mi><mi>i</mi></msub></mrow></math><img file="EP3035239A1_D0015.tif" /></maths>

0082Note that <i>m̂_k</i> and <i>π̂_k</i> are respectively the estimates of the mean vector and mixing weight of <i>k</i>-th Gaussian component given the data samples from <i>X.</i> Thus Equation (7) defines the Fisher Vector as a function of the difference between the estimated mean vector (<i>m̂_k</i>) and the respective UBM Gaussian mean vector (<i>µ_k</i>).

0083The support Ω<i>_k</i> of a Gaussian <i>k</i> is defined as: Ω<i>_k</i> = {<i>x</i>: <i>k</i> = argmax<i>_ip</i>(<i>i</i>|<i>x</i>)}. Hence, if the change in imaging conditions can be modeled by a piecewise-constant shift <i>δ_k</i> over the support of each Gaussian <i>k,</i> then the corrupted UBM mean <i>µ_k</i> and image-specific mean <i>m̂_k</i> become <i>µ_k</i> + <i>δ_k</i> and <i>m̂_k</i> + <i>δ_k</i> (making a hard assignment assumption, meaning that <i>p</i>(<i>i</i>|<i>x</i>) is assumed to be binary, which is reasonable for high-dimensional input vectors <i>x</i>). However, the influence of <i>δ_k</i> cancels-out in Equation (7) thus justifying the UBM adaptation strategy.

0084Various adaptation methods are contemplated. These include the maximum a posteriori (MAP) adaptation, the adaptation methods from the linear transformation family, e.g., Maximum Likelihood Linear Regression (MLLR), techniques from the speaker clustering family, e.g., Eigenvoices, e.g., adapted from speech recognition methods (see,<nplcit id="ncit0019" npl-type="s"><text> Douglas A. Reynolds, et al., "Speaker verification using adapted Gaussian mixture models," Digital Signal Processing, 10(13):19 - 41 (2000</text></nplcit>), hereinafter, Reynolds, et al.). The choice may depend on the amount of adaptation data available and the intended application. As an example MAP adaptation may be used.

MAP Adaptation

0085The exemplary adaptation process is fully unsupervised: given a set of images associated with a camera <b>16</b> or <b>18,</b> information on the content of these images (such as labels) is not needed to adapt the UBM <b>60</b> to generate the camera-specific models <b>62, 64.</b>

0086In MAP adaptation, a camera specific model <maths id="math0016"><math display="inline"><mrow><msubsup><mi>u</mi><mi mathvariant="normal">Θ</mi><mi>c</mi></msubsup></mrow></math><img file="EP3035239A1_D0016.tif" /></maths> is derived by updating the well-trained parameters in the UBM <i>u</i>_Θ, using the local descriptors extracted from the images <b>66</b> or <b>68</b> taken with the respective camera c. The adaptation can be performed in a two-step Expectation-Maximization (EM) iterative process (see, e.g., <nplcit id="ncit0020" npl-type="s"><text>Jeff A. Bilmes, et al. "A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and Hidden Markov models," Intern'I Computer Science Institute, 4(510), p. 126 (1998</text></nplcit>), for a discussion of EM). In the first step the estimates of the sufficient statistics of the features from the camera c are computed (probabilistic count, first and second moments) for each Gaussian in the UBM. These are then combined with the old sufficient statistics from the UBM Gaussians. This combination is performed through a data-dependent mixing coefficient. This coefficient puts more weight on the new sufficient statistics for the Gaussians that have a higher probabilistic count from the adaptation data.

0087In the following, the terms "optimization," "minimization," and similar phraseology are to be broadly construed as one of ordinary skill in the art would understand these terms. For example, these terms are not to be construed as being limited to the absolute global optimum value, absolute global minimum, and so forth. For example, minimization of a function may employ an iterative minimization algorithm that terminates at a stopping criterion before an absolute minimum is reached. It is also contemplated for the optimum or minimum value to be a local optimum or local minimum value.

0088Suppose <i>X =</i> {<i>x</i>₁,<i>x</i>₂,...,<i>x_{N<sub2>c</sub2>}</i>} is a set of <i>N_c</i> feature vectors (local descriptors) extracted from the overlapping or non-overlapping patches from the training images <b>66</b> or <b>68</b> from a camera c. For each Gaussian <i>k</i> and feature vector <i>x_i</i>, the probability <i>p</i>(<i>k</i>|<i>x_i</i>) is computed in the first, Expectation step, from Equation (6). Then, <i>p</i>(<i>k</i>|<i>x_i</i>) and <i>x_i</i> are used to compute the probabilistic count of each Gaussian <i>k</i> and first and second moments (see, Reynolds, et al.) according to Equations 8-10: <maths id="math0017" num="(8)"><math display="block"><mrow><msub><mi>n</mi><mi>k</mi></msub><mo>=</mo><mrow><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mrow><msub><mi>N</mi><mi>c</mi></msub></mrow></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow></mrow></math><img file="EP3035239A1_D0017.tif" /></maths><maths id="math0018" num="(9)"><math display="block"><mrow><msub><mi>m</mi><mi>k</mi></msub><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mi>n</mi><mi>k</mi></msub></mrow></mfrac><mrow><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mrow><msub><mi>N</mi><mi>c</mi></msub></mrow></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow><msub><mi>x</mi><mi>i</mi></msub></mrow></math><img file="EP3035239A1_D0018.tif" /></maths><maths id="math0019" num="(10)"><math display="block"><mrow><msub><mi>s</mi><mi>k</mi></msub><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mi>n</mi><mi>k</mi></msub></mrow></mfrac><mrow><mstyle displaystyle="false"><mrow><munderover><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mrow><msub><mi>N</mi><mi>c</mi></msub></mrow></munderover></mrow></mstyle><mrow><mi>p</mi><mfenced><mi>k</mi><mrow><mo>|</mo></mrow><msub><mi>x</mi><mi>i</mi></msub></mfenced></mrow></mrow><mi>diag</mi><mfenced><msub><mi>x</mi><mi>i</mi></msub><msub><mi>x</mi><mi>i</mi></msub><mi>ʹ</mi></mfenced></mrow></math><img file="EP3035239A1_D0019.tif" /></maths>

0089<i>n_k</i> is the responsibility of a Gaussian <i>k</i>- an estimation of how many points <i>x_i</i> are assigned to that Gaussian. <i>m_k</i> is the first order statistic- an estimate of the mean of all points assigned to Gaussian <i>k. s_k</i> is the second order statistic-an estimate of the statistical variance of all points in Gaussian <i>k</i>. In the second, Maximization step, these statistics are then used to update the UBM parameters for each Gaussian <i>k</i> using the following equations: <maths id="math0020" num="(11)"><math display="block"><mrow><mi>For the mixture weight</mi><mo>:</mo><mspace width="1em" /><msubsup><mrow><mover><mi>π</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi><mi>c</mi></msubsup><mo>=</mo><mfenced open="[" close="]"><msubsup><mi>τ</mi><mi>k</mi><mi>π</mi></msubsup><mfrac><mrow><msub><mi>n</mi><mi>k</mi></msub></mrow><mrow><msub><mi>N</mi><mi>c</mi></msub></mrow></mfrac><mo>+</mo><mfenced><mn>1</mn><mo>-</mo><msubsup><mi>τ</mi><mi>k</mi><mi>π</mi></msubsup></mfenced><msub><mi>π</mi><mi>k</mi></msub></mfenced><mi>α</mi></mrow></math><img file="EP3035239A1_D0020.tif" /></maths><maths id="math0021" num="(12)"><math display="block"><mrow><mi>For the mean vector</mi><mo>:</mo><mspace width="1em" /><msubsup><mrow><mover><mi>μ</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi><mi>c</mi></msubsup><mo>=</mo><msubsup><mi>τ</mi><mi>k</mi><mi>π</mi></msubsup><msub><mi>m</mi><mi>k</mi></msub><mo>+</mo><mfenced><mn>1</mn><mo>-</mo><msubsup><mi>τ</mi><mi>k</mi><mi>π</mi></msubsup></mfenced><msub><mi>μ</mi><mi>k</mi></msub></mrow></math><img file="EP3035239A1_D0021.tif" /></maths><maths id="math0022" num="(13)"><math display="block"><mrow><mi>For the covariance matrix</mi><mo>:</mo><mspace width="1em" /><msubsup><mrow><mover><mi>σ</mi><mrow><mo>^</mo></mrow></mover></mrow><mi>k</mi><mi>c</mi></msubsup><mo>=</mo><msubsup><mi>τ</mi><mi>k</mi><mi>σ</mi></msubsup><msub><mi>s</mi><mi>k</mi></msub><mo>+</mo><mfenced><mn>1</mn><mo>-</mo><msubsup><mi>τ</mi><mi>k</mi><mi>σ</mi></msubsup></mfenced><mfenced><msubsup><mi>σ</mi><mi>k</mi><mn>2</mn></msubsup><mo>+</mo><msubsup><mi>μ</mi><mi>k</mi><mn>2</mn></msubsup></mfenced><mo>-</mo><msup><mrow><mfenced><msubsup><mi>μ</mi><mi>k</mi><mi>c</mi></msubsup></mfenced></mrow><mn>2</mn></msup></mrow></math><img file="EP3035239A1_D0022.tif" /></maths>

0090The above two-step process is then repeated, several times with the updated UBM parameters, e.g., for <i>I</i> iterations or until convergence, or some other stopping criterion is met. <i>α</i> in Equation 11 is recomputed over all Gaussians to make sure that <maths id="math0023"><math display="inline"><mrow><mstyle displaystyle="false"><mrow><munder><mo>∑</mo><mi>k</mi></munder><mrow><msubsup><mi>π</mi><mi>k</mi><mi>c</mi></msubsup></mrow></mrow></mstyle><mo>=</mo><mn>1</mn></mrow></math><img file="EP3035239A1_D0023.tif" /></maths> holds true. The adaptation parameters <maths id="math0024"><math display="inline"><mrow><msubsup><mi>τ</mi><mi>k</mi><mi>ρ</mi></msubsup><mo>,</mo></mrow></math><img file="EP3035239A1_D0024.tif" /></maths> for each parameter <i>ρ</i> E {<i>π,µ,σ</i>}, are given by, <maths id="math0025" num="(14)"><math display="block"><mrow><msubsup><mi>τ</mi><mi>k</mi><mi>ρ</mi></msubsup><mo>=</mo><mfrac><mrow><msub><mi>n</mi><mi>k</mi></msub></mrow><mrow><msub><mi>n</mi><mi>k</mi></msub><mo>+</mo><msup><mi>r</mi><mi>ρ</mi></msup></mrow></mfrac></mrow></math><img file="EP3035239A1_D0025.tif" /></maths>

0091Here <i>r</i> is a design parameter which controls the level of adaptation between new and old statistics and can be estabished by cross-validation. Note that once <i>r</i> is fixed, the new statistics are emphasized more in those Gaussians which have a higher probabilistic count <i>n_k</i> for the adaptation data. In practice the same adaptation parameter can be used for all parameters of all Gaussians, i.e., <maths id="math0026"><math display="inline"><mrow><msubsup><mi>τ</mi><mi>k</mi><mi>ρ</mi></msubsup><mo>=</mo><mi>τ</mi><mn>.</mn></mrow></math><img file="EP3035239A1_D0026.tif" /></maths> As will be appreciated, fewer than three parameters may be considered, such as one or two of the exemplary parameters <i>π,µ,σ</i> are updated and/or used to generate the weights (dimension values) for each Fisher vector.

0092To include spatial information about the cropped image, the image can be partitioned into regions, such as at least three regions, the per-patch statistics aggregated at a region level, and then the region-level representations concatenated to form the image representation. See, for example, <nplcit id="ncit0021" npl-type="s"><text>S. Lazebnik, et al., "Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories," CVPR '06 Proc. 2006 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition - Volume 2, Pages 2169 - 2178</text></nplcit>.

0093The exemplary image signatures are of a fixed dimensionality <i>D</i>, i.e., each image representation has the same number of elements which is a function of the number of Gaussians and the number of parameters considered. For example, if a visual vocabulary of 32 Gaussians is used in the GMMs <b>60, 62, 64</b> and two of the parameters are considered. This results in a 32x2= 64-dimensional FV representation.

0094Using the FV representation has other advantages in license plate matching as it obscures the license plate number. It does not allow recognition of the license plate number from it since the original image cannot be reconstructed from it, and therefore it can be seen as a hash key of the original image as there is no feasible way to recover the original image (or the license plate number) from the image signature. This is because the identity of the individual patches is lost in the aggregation process. Consequently, this retains the privacy of the license plate number (or other private information in the image) until the need for use of that information has been established (for example, by establishing that the speed of the vehicle captured in the images has exceeded a threshold speed).

Comparing image signatures (S116, S118)

0095In some embodiments, the image signatures of at location X are stored in the database <b>38</b> and then a search for a closest match among these stored signatures is performed in an exhaustive manner for each image signature subsequently acquired at location Y. When the database <b>38</b> contains a larger number of images, an exhaustive search may be time consuming. In this case, an approximate search technique may be employed. Techniques for the approximate search of high-dimensional vectors are disclosed, for example, in <nplcit id="ncit0022" npl-type="s"><text>Jégou, et al., "Aggregating local image descriptors into compact codes," IEEE TPAMI, 34(9)1704-1716 (2012</text></nplcit>). The identity of the closest match is assigned, provided the similarity at least meets a predetermined threshold.

0096Since the Fisher vector is an explicit embedding of the Fisher kernel, the corresponding cosine similarity measure between two such image descriptors SI_X and SI_Y is the dot product SI_X^TSI_Y.

0097In one embodiment, the image signature (e.g., the FV) can be made more discriminative by applying a projection which is obtained, for instance, by learning a metric for embedding the image signature into a vectorial space where similarity is a good indicator for actual similarity (in terms of the character sequences forming the identifiers). As an example, a low-rank Mahalanobis metric can be employed, see for example, <nplcit id="ncit0023" npl-type="s"><text>Rodríguez-Serrano, et al., "Data-Driven Vehicle Identification by Image Matching". 12th European Conf. on Computer Vision (ECCV) Workshops, October 7-13, 2012, Lecture Notes in Computer Science, vol. 7584, pp. 536-545</text></nplcit>. See also <patcit id="pcit0029" dnum="US20140056520A"><text>U.S. Pub. Nos. 20140056520</text></patcit> and <patcit id="pcit0030" dnum="US20140219563A"><text>20140219563</text></patcit>, and <patcit id="pcit0031" dnum="US903218A"><text>U.S. Application Ser. No. 13/903,218</text></patcit>, the disclosures of which are incorporated by reference in their entireties, for descriptions of exemplary embedding techniques.

Process Implementation (S122)

0098License plate recognition methods which may be used at S122 are disclosed, for example, in <patcit id="pcit0032" dnum="US20130129151A"><text>U.S. Pub. Nos. 20130129151</text></patcit>, <patcit id="pcit0033" dnum="US20130129152A"><text>20130129152</text></patcit>, <patcit id="pcit0034" dnum="US20130182909A"><text>20130182909</text></patcit>, <patcit id="pcit0035" dnum="US20130259314A"><text>20130259314</text></patcit>, <patcit id="pcit0036" dnum="US20140056520A"><text>20140056520</text></patcit>, <patcit id="pcit0037" dnum="US20140270381A"><text>20140270381</text></patcit>, and <patcit id="pcit0038" dnum="US20140219563A"><text>20140219563</text></patcit> and <patcit id="pcit0039" dnum="US903218A"><text>U.S. Application Ser. No. 13/903,218</text></patcit>, and in <nplcit id="ncit0024" npl-type="s"><text>J-A. Rodríguez-Serrano, et al., "Data-Driven Vehicle Identification by Image Matching". 12th European Conf. on Computer Vision (ECCV) Workshops, October 7-13, 2012, Lecture Notes in Computer Science, vol. 7584, pp. 536-545</text></nplcit>. In one embodiment, Xerox License Plate Recognition (XLPR) software is employed. As will be appreciated, the license plate number and image of a license plate are exemplary only and are used to illustrate the exemplary embodiment. In other embodiments, a larger set of ASCII, UNICODE, and/or UTF-8 characters may be used as the alphabet.

0099The action initiated by the processing component <b>50</b> may depend on the type of application. In the case of license plates, the action triggered may be associated to the respective license plate number identified (e.g., prosecute the violator, alert parking operator, send parking fine, etc.). In the case of a car park, if the first and second images match (and optionally if a paid-for duration between the capture of the two images has not been exceeded), a barrier to the car park may automatically open.

0100While the illustrative embodiments disclosed herein are directed to camera-based vehicle labeling tasks, it will be appreciated that similar difficulties arise in other camera-based object labeling tasks in which multiple still or video cameras are used to acquire images of objects to be labeled (or in which the same camera is used to acquire the images over different time intervals and/or at different locations). For example, in a retail or advertising setting it may be useful to employ camera-based customer labeling as to gender, age, or so forth in order to provide targeted advertising. In this case the objects to be labeled are human beings (or parts, such as faces). In an event attendance monitoring system images may be labeled as to the number of people shown in the image. Objects may also be animals, or inanimate objects such as the illustrative vehicles. As further examples of camera-based labeling of inanimate objects of interest, in a retail assembly line articles of manufacture may be labeled as to the presence or absence of a certain defect based on an imaging technology that is capable of observing the defect. In the case of a security scanner, the camera may be an x-ray imager or other specialized imaging device, and the object labeling seeks to identify inanimate objects of concern such as firearms, knives, fluids, and so forth. These are merely illustrative examples. The adapted mixture models can also find use in classification (images from different cameras can be classified with the same classifier, which may have been trained on image-level descriptors extracted from training images <b>34</b> and respective labels).

0101Without intending to limit the scope of the exemplary embodiment, the following examples illustrate the applicability of the method to license plate matching.

Examples

0102The method was applied to the problem of vehicle license plate matching/re-identification. Cameras are positioned on various entry-exit lanes of a parking lot. The aim is to match the license plate of a vehicle exiting the parking lot to its license plate image which was captured when this vehicle entered the lot. However, the imaging conditions in both cases may be very different. The different imaging conditions may include different geometry, different camera quality, different lighting conditions etc. This scenario is a good candidate for adaptation of the UBM to learn camera-specific (lane-specific) GMMs before Fisher Vector computation, to aid in matching.

Datasets

0103Two in-house datasets coming from two real parking facilities in different cities are denoted A and B. Both datasets are pre-processed by extracting the license plate regions and normalizing the height to 50 pixels. Dataset A has 13,006 images (6503 entry-exit pairs) from 11 lanes/cameras. Five out of 11 are entry lanes while the rest are the exit lanes with varying distribution of entry-exit statistics. For dataset B, 9,629 images from two lanes were used.

Experimental Setup

0104Features and local descriptors were extracted from overlapping patches from the images. Each dataset is divided into three parts. The first part is used to learn the PCA model for dimensionality reduction of the local descriptors and UBM estimation, the second is used for UBM adaptation and the third is used for testing. During testing, the exiting license plates are matched with the entering ones and the results are reported as a percentage of correctly matched pairs.

0105In one test, after learning the UBM on the first part of Dataset A with images from all the lanes/cameras, the UBM is adapted to each of the different lanes/cameras of Dataset A to generate camera-specific GMMs, using images from the respective lanes in the second part. Then the images in the third part and the adapted GMMs are used to compute Fisher Vectors which are then used for matching.

0106In another test, the first part of the images from Dataset B is used to learn the UBM. This UBM is then adapted using the second part of Dataset A which is followed by testing on the third part of Dataset A. This mimics a scenario when the UBM is learned on images which are not from the same dataset/parking lot.

0107The results of image matching with adapted GMMs are compared to two baselines: <ol id="ol0001" compact="compact"><li>1. The system without adaptation ('No adapt.').</li><li>2. The system after mean and standard deviation adjustment ('mean-std adj.'): For each camera, the mean and standard deviation of its training samples (second part) is computed and test vectors standardized accordingly. The mean adjustment is a domain adaptation approach which was proposed for the VLAD image descriptor. See <nplcit id="ncit0025" npl-type="s"><text>Relja Arandjelovic, et al., "All about VLAD," Proc. 2013 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR '13), pp. 1578 - 1585 (2013</text></nplcit>) for a description of VLAD image descriptors. The VLAD image descriptor is closely related to the Fisher Vector, but does not have the probabilistic interpretation of the Fisher Vector. Mean and standard deviation adjustment is a natural extension of the mean adjustment approach. Note that this can be understood as a diagonal approximation of the subspace alignment approach of <nplcit id="ncit0026" npl-type="s"><text>Basura Fernando, et al., "Unsupervised visual domain adaptation using subspace alignment, ICCV, pp. 2960-2967 (2013</text></nplcit>), which is more practical when dealing with high-dimensional vectors.</li></ol>

0108There are various options for UBM adaptation. For example, all the UBM parameters are adapted or parameters or to be adapted are selected. In these examples, both means and variances are adapted. Weight adaptation was found to have little additional impact.

Results

0109Tables 1 - 4 show the results using different numbers of Gaussians in the GMMs (Nmix), with MAP adaptation (MAP Adapt.), without MAP adaptation (No Adapt.), and with mean and standard deviation adjustment (Mean-Std. Adj.). Tables 1 and 2 give the results when only the mean vectors are adapted, while Tables 3 and 4 give the results for the case when both mean vectors and covariance matrices are adapted. 'Inc.' indicates the performance improvement with MAP adaptation over no adaptation. <tables id="tabl0001" num="0001"><table frame="all"><title>Table 1: <u>Results for UBM learnt on Dataset A, followed by mean vector adaptation and testing on Dataset A</u></title><tgroup cols="5"><colspec colnum="1" colname="col1" colwidth="28mm" /><colspec colnum="2" colname="col2" colwidth="31mm" /><colspec colnum="3" colname="col3" colwidth="31mm" /><colspec colnum="4" colname="col4" colwidth="31mm" /><colspec colnum="5" colname="col5" colwidth="30mm" /><thead><row><entry align="center" valign="top">Nmix</entry><entry align="center" valign="top">No Adapt.</entry><entry align="center" valign="top">Mean-Std. Adj.</entry><entry align="center" valign="top">MAP Adapt.</entry><entry align="center" valign="top">Inc.</entry></row></thead><tbody><row><entry align="center">32</entry><entry align="center">81.89%</entry><entry align="center">81.41%</entry><entry align="center"><b>82.37%</b></entry><entry align="center">0.48%</entry></row><row><entry align="center">64</entry><entry align="center">82.77%</entry><entry align="center">82.17%</entry><entry align="center"><b>83.53%</b></entry><entry align="center">0.76%</entry></row><row><entry align="center">128</entry><entry align="center">83.45%</entry><entry align="center">83.01%</entry><entry align="center"><b>84.49%</b></entry><entry align="center">1.04%</entry></row></tbody></tgroup></table></tables><tables id="tabl0002" num="0002"><table frame="all"><title>Table 2: <u>Results for UBM learnt on Dataset B, followed by mean vector adaptation and testing on Dataset A</u></title><tgroup cols="5"><colspec colnum="1" colname="col1" colwidth="28mm" /><colspec colnum="2" colname="col2" colwidth="31mm" /><colspec colnum="3" colname="col3" colwidth="31mm" /><colspec colnum="4" colname="col4" colwidth="31mm" /><colspec colnum="5" colname="col5" colwidth="30mm" /><thead><row><entry align="center" valign="top">Nmix</entry><entry align="center" valign="top">No Adapt.</entry><entry align="center" valign="top">Mean-Std. Adj.</entry><entry align="center" valign="top">MAP Adapt.</entry><entry align="center" valign="top">Inc.</entry></row></thead><tbody><row><entry align="center">32</entry><entry align="center">81.05%</entry><entry align="center">78.81%</entry><entry align="center"><b>83.21%</b></entry><entry align="center">2.16%</entry></row><row><entry align="center">64</entry><entry align="center">84.13%</entry><entry align="center">83.01%</entry><entry align="center"><b>85.81%</b></entry><entry align="center">1.68%</entry></row><row><entry align="center">128</entry><entry align="center">84.89%</entry><entry align="center">84.17%</entry><entry align="center"><b>85.85%</b></entry><entry align="center">0.96%</entry></row></tbody></tgroup></table></tables><tables id="tabl0003" num="0003"><table frame="all"><title>Table 3: <u>Results for UBM learnt on Dataset A, followed by mean vector and covariance matrix adaptation and testing on Dataset A</u></title><tgroup cols="5"><colspec colnum="1" colname="col1" colwidth="31mm" /><colspec colnum="2" colname="col2" colwidth="35mm" /><colspec colnum="3" colname="col3" colwidth="35mm" /><colspec colnum="4" colname="col4" colwidth="35mm" /><colspec colnum="5" colname="col5" colwidth="33mm" /><thead><row><entry align="center" valign="top">Nmix</entry><entry align="center" valign="top">No Adapt.</entry><entry align="center" valign="top">Mean-Std. Adj.</entry><entry align="center" valign="top">MAP Adapt.</entry><entry align="center" valign="top">Inc.</entry></row></thead><tbody><row><entry align="center">32</entry><entry align="center">81.89%</entry><entry align="center">81.41%</entry><entry align="center"><b>82.01%</b></entry><entry align="center">0.12%</entry></row><row><entry align="center">64</entry><entry align="center">82.77%</entry><entry align="center">82.17%</entry><entry align="center"><b>83.13%</b></entry><entry align="center">0.36%</entry></row><row><entry align="center">128</entry><entry align="center">83.45%</entry><entry align="center">83.01%</entry><entry align="center"><b>84.73%</b></entry><entry align="center">1.28%</entry></row></tbody></tgroup></table></tables><tables id="tabl0004" num="0004"><table frame="all"><title>Table 4: <u>Results for UBM learnt on Dataset B, followed by mean vector and covariance matrix adaptation and testing on Dataset A</u></title><tgroup cols="5"><colspec colnum="1" colname="col1" colwidth="31mm" /><colspec colnum="2" colname="col2" colwidth="35mm" /><colspec colnum="3" colname="col3" colwidth="35mm" /><colspec colnum="4" colname="col4" colwidth="35mm" /><colspec colnum="5" colname="col5" colwidth="33mm" /><thead><row><entry align="center" valign="top">Nmix</entry><entry align="center" valign="top">No Adapt.</entry><entry align="center" valign="top">Mean-Std. Adj.</entry><entry align="center" valign="top">MAP Adapt.</entry><entry align="center" valign="top">Inc.</entry></row></thead><tbody><row><entry align="center">32</entry><entry align="center">81.05%</entry><entry align="center">78.81%</entry><entry align="center"><b>82.81%</b></entry><entry align="center">1.76%</entry></row><row><entry align="center">64</entry><entry align="center">84.13%</entry><entry align="center">83.01%</entry><entry align="center"><b>85.29%</b></entry><entry align="center">1.16%</entry></row><row><entry align="center">128</entry><entry align="center">84.89%</entry><entry align="center">84.17%</entry><entry align="center"><b>86.41%</b></entry><entry align="center">1.52%</entry></row></tbody></tgroup></table></tables>

0110It can be seen from Tables 1 - 4 that MAP adaptation gives performance improvement for all cases, while mean and standard deviation adjustment shows a decrease in performance over the No Adaptation baseline. The performance increase with MAP adaptation, over No Adaptation can be as high as 2.16% in these experiments.

Evaluation of Adaptation

0111To evaluate adaptation, the internal mechanics of assignment of features to different Gaussians in a UBM/GMM are evaluated. As an example, a UBM learned on Dataset B with 32 Gaussians was used. Then adaptation data from Dataset A was taken. The data from Dataset A contains images from 11 lanes. The images from each lane are then used to learn a lane/camera-dependent GMM. Then <i>p</i>(<i>k</i>|<i>x_i</i>) is computed from features <i>x_i</i> for each Gaussian <i>k</i> of the UBM and of each lane/camera-dependent GMM through Equation (6).

0112A plot of <i>p</i>(<i>k</i>|<i>x_i</i>) vs. <i>k</i>, averaged across features <i>x_i</i> from a particular lane in the adaptation dataset was obtained for the UBM (<figref idref="f0005">FIG. 4</figref>) and for lane/camera-dependent GMMs (<figref idref="f0006">FIG. 5</figref>). From <figref idref="f0005">FIGURE 4</figref>, it can be seen that when the same UBM is used, there is considerable spread across images from different lanes. For some lanes, some Gaussians have a higher <i>p</i>(<i>k</i>|<i>x_i</i>) while for some other lanes it may be considerably low. This shows that, although the UBM can represent the features, the representation may be different for different lanes/cameras. To counter this shift, the adaptation and then lane/camera specific-GMMs used in <figref idref="f0006">FIGURE 5</figref> result in the spread being considerably reduced. This shows that when these lane/camera specific-GMMs are used for Fisher Vector computation, the Fisher Vectors can encode the deviations which are due to a particular object in question and not due to the imaging conditions.

0113It will be appreciated that variants of the above-disclosed and other features and functions, or alternatives thereof, may be combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.