Method and apparatus for detecting suspicious activity using video analysis

CONTINUATION DATA

00011. This application is a continuation of U.S. Ser. No. 11/393,370 entitled “METHOD AND APPARATUS FOR DETECTING SUSPICIOUS ACTIVITY USING VIDEO ANALYSIS” Filed Mar. 29, 2007 (Attorney Docket Number STL06-01) and allowed Aug. 6, 2009, which is a CONTINUATION-IN-PART APPLICATION of earlier filed co-pending entitled “METHOD AND APPARATUS FOR AUDITING TRANSACTION ACTIVITY IN RETAIL AND OTHER ENVIRONMENTS USING VISUAL RECOGNITION”, having U.S. Ser. No. 11/157,127, Filed Jun. 20, 2005 (Attorney Docket Number STL05-02), which in turn claims priority to “METHOD AND APPARATUS FOR AUDITING TRANSACTION ACTIVITY IN RETAIL AND OTHER ENVIRONMENTS USING VISUAL RECOGNITION”, Filed Jun. 21, 2004, having U.S. Ser. No. 60/581,559 and “METHOD AND APPARATUS FOR DIGITAL ARCHIVING OF VIDEO”, Filed Nov. 15, 2004, U.S. Ser. No. 60/627,862.

00022: This application further claims the benefit of the filing date of Provisional patent application entitled “METHOD AND APPARATUS FOR DETECTING ITEM PASS THROUGH & MISIDENTIFICATION”, having U.S. Ser. No. 60/666,081, Filed Mar. 29, 2005 (Attorney Docket STL05-01p). The entire disclosure, teachings and contents of this provisional application are hereby incorporated by reference herein.

BACKGROUND

0003Retail establishments commonly utilize point of sale or other transaction terminals, often referred to as cash registers, to allow customers of those establishments to purchase items. As an example, in a conventional department store, supermarket or other retail establishment, a customer collects items for purchase throughout the store and places them in a shopping cart, basket, or simply carries them to a point of sale terminal to purchase those items in a transaction. The point of sale terminal may be staffed with an operator such as a cashier who is a person employed by the store to assist the customer in completing the transaction. In some cases, retail establishments have implemented self-checkout point of sale terminals in which the customer is the operator. In either case, the operator typically places items for purchase on a counter, conveyor belt or other item input area. The point of sale terminals include a scanning device such as a laser or optical scanner device that operates to identify a Uniform Product Code (UPC) label or bar code affixed to each item that the customer desires to purchase. The laser scanner is usually a peripheral device coupled to a computer that is part of the POS terminal. To scan an item, the operator picks up each item, one by one, from the item input area and passes that item over a scanning area such as glass window built into the counter or checkout area to allow the laser scanner to detect the UPC code. Once the point of sale computer identifies the UPC code on an item, the computer can perform a lookup in a database to determine the price and identity of the scanned item. Alternatively, in every case where the operator can scan the item, the operator may likewise enter the UPC or product identification code into the terminal manually or through an automatic product identification device such as an RFID reader. The term “scan” is defined generally to include all means of entering transaction items into a transaction terminal. Likewise, the term “scanner” is defined generally as any transaction terminal, automated and/or manual, for recording transaction information.

0004As the operator scans or enters each item for purchase, one by one, the point of sale terminal maintains an accumulated total purchase price for all of the items in the transaction. For each item that an operator successfully scans or enters, the point of sale terminal typically makes a beeping noise or tone to indicate to the operator that the item has been scanned by the point of sale terminal and in response, the operator places the item into an item output area such as a downstream conveyor belt or other area for retrieval of the items by the customer or for bagging of the items into a shopping bag. Once all items in the transaction are scanned in this manner, the operator indicates to the point of sale terminal that the scanning process is complete and the point of sale terminal displays a total purchase price to the customer who then pays the store for the items purchased in that transaction.

SUMMARY

0005Conventional systems that provide for purchase of items using a scanner or other automated identification of items via code suffer from a variety of deficiencies. In particular, operation of such systems can be compromised either knowingly or unknowingly by an operator in a manner that allows a customer to receive possession of one or more items without paying for them. In particular, such systems are susceptible to “pass-throughs”, also know as “sweethearting” in which an operator purposefully or accidentally fails to scan an item as that item moves through the transaction area. In such cases, the POS system never detects the un-scanned item and the item is thus never totaled into the purchase price. In such cases, the customer effectively receives the item for free. Retail chains lose millions of dollars a year to operator error or fraudulent activity of this nature.

0006In a non-fraudulent example, an operator may unknowingly pass an item through the scanning area during a transaction and place the item into the item output area such as a downstream conveyor belt, but no scan of the item took place. Perhaps the operator was not paying attention and did not notice (or did not care) that the scanner failed to beep during scanning of an item.

0007As an example of sweethearting that represents fraud on the retail establishment, an operator who may be assisting a customer who is personally known to the operator (e.g. a friend or conspirator of the operator) intentionally causes the POS system to either not scan the item as the operator moves the item through the transaction area, such as by covering the UPC label with their hand or moving the UPC code out of range of the scanner. In such situations, the item is included with other items that may or may not have also been scanned, and the customer or operator continues along as if nothing wrong happened. After the operator processes all items in the transaction, either via actual scanning which is often done for the less expensive items to make the transaction look legitimate to anyone who may be watching, or via sweethearting in which case the item not scanned is grouped with items that appear to have be scanned, the customer pays the operator a purchase price reflecting only the sum total of all scanned and entered items. After paying, the customer removes all items, scanned/entered and un-scanned, from the store, having only paid for those items that were scanned or entered.

0008In another fraudulent example know as label switching, the operator causes the POS system to scan an item that is different that the item being passed through the scanning area during the transaction. In such cases, a customer or operator may replace a UPC label of an original and often expensive item with a UPC label for another less expensive item. In such cases, a scan takes place but the wrong item is identified by the POS system. In this manner, the system will scan the item for a price that is substantially lower that the value of the item received by the customer.

0009Security system designers have attempted to develop conventional techniques for detecting fraudulent or accidental operator error in use of POS terminals. The conventional systems in the field of detection of pass-through and sweethearting provide for the detection of abnormally long “scan-gaps”. A “scan-gap” is the amount of time between consecutive scans at the point of sale terminal. When an item is passed through without scanning, the scan-gap increases until the next scan. By comparing the scan-gaps between scanned items of a transaction versus the average scan-gap for a given employee operator, the conventional scan-gap method seeks to identify incidents when an item has bypassed the scanner without being scanned.

0010The conventional scan-gap detection method is widely regarded to be impractical, as scan-gaps have been found to be a “noisy” measure at best. This is due to the fact that perfectly legitimate scan-gaps may vary widely due to delays such as those caused by weighing of produce, manual entry of unlabeled or un-scannable goods, and rescanning of items that did not get scanned on the first pass. As a result, scan-gaps are not a dependable metric and therefore conventional systems that attempt to use scan gaps as a method for detecting fraudulent activity are prone to problems.

0011In contrast, the system disclosed herein uses video data analysis techniques as will be explained to detect activity such as sweethearting or pass-throughs. In particular, the system disclosed herein detects incidents of theft or loss of inventory at the cash register, POS or other transaction terminal when an operator such as a customer or store employee passes one or more items around the scanner (or RFID reader) without being scanned, or when the operator scans or manually enters an incorrect code into the transaction terminal for an item. The system disclosed herein can also detect items which may be mislabeled with an incorrect bar code to be misread by the scanner or entered as the wrong item by the operator. Some embodiments utilize video analysis in conjunction with transaction scan data concerning items that were actually scanned by the POS terminal.

0012It is to be understood that point-of-sale terminals or cash registers that utilize scanning are only examples of transaction terminals and the system is not limited to detecting fraud in only retail environments. Additionally, scanning is not limited to laser scanning with a fixed scanner device, but can include handheld scanners, or Radio Frequency Identification (RFID) readers. The system is even applicable in situations where an operator manually enters a code or other item identification via a keyboard into the transaction terminal. The system disclosed herein is generally applicable to any environment where transaction data is available for comparison with video data associated with that transaction.

0013As an example, a system that utilizes RFID tags to identify items can benefit from the system disclosed herein. As an example, toll booth collection systems provide video data of vehicles traveling through the toll booths and provide for operators such as people, or automated scanners such as RFID vehicle transceiver reading systems, to collect toll fees from vehicles traveling on a highway. Fraud may occur is such systems, for example, if a vehicle is equipped with an improper transceiver (e.g. a truck is equipped with a car transceiver). Also, the terminal operator may refer to either store employee or customer, as in situations such as self-checkout transaction terminals.

0014More specifically, the system disclosed herein includes methods and apparatus for detecting a transaction outcome such as suspicious activity related to a transaction (e.g., purchase, refund, void, etc.) of items by a customer at a transaction terminal. The system obtains video data associated with a transaction area. The video data may be obtained, for example, from an elevated camera focused on a cash register check out or other transaction area in a supermarket or other retail establishment. The system applies an automated machine video analysis algorithm that is disclosed as part of the system to analyze at least a portion of the video data to obtain at least one video parameter concerning at least a portion of a transaction associated with the transaction area. As an example, the system can analyze the video data to track (e.g. identify the presence of) items involved in the transaction in the transaction area. This process can automatically identify the presence of an item involved in the transaction from the video data analysis. This can be done, for example, by automatically detecting item activity in the transaction area and/or detecting operator activity in the transaction area. Detection of item presence can include detecting removal of an item from a region of interest in the transaction area and/or detecting introduction of an item into a region of interest in the transaction area.

0015In one configuration, the video transaction parameter is reduced to a video count of how many items the video analysis algorithm identified as having been processed by the operator in the transaction. As an example, an item is processed when an operator moves the item through the transaction area, whether or not the item was scanned or entered. Thus the video count can detect and count items that are both processed and scanned/entered and items that are processed, but not scanned/entered.

0016In another configuration, the video transaction parameter is a sequence of detection events produced from one or more detectors performing analysis of all or part of the video data. A detector is generally an automated image processing algorithm applied to a region of interest of the video data. For example, the video data may cover a large portion of the transaction area that includes the operator (e.g. store employee and/or customer), an item input region, a scan region, and an item output region. A detector can analyze all or a portion of this area, such as just the input conveyor belt region of the video data of a point of sale terminal. An image isolation and comparison process can be applied to frames of the video data for one or more regions of interest to detect the presence of an item being introduced or removed from this region of interest. When an item is detected in this manner, a detection event is produced indicating the presence of an item, time of detection, and other characteristics such as the size of the item. There may be more than one region of interest and the sequence of events may include many events from different detectors, or there may be a single detector producing a single set of events for just one region of interest. By analyzing the video data in this manner, the system is able to detect, from video analysis, the presence of individual items in the transaction. In some configurations, the system can determine how many items were visually processed in the entire transaction.

0017The system obtains at least one transaction parameter originated from the transaction terminal associated with the transaction area. The expected transaction parameter in one configuration is a transaction count or other item presence indication obtained from the transaction data produced by the transaction terminal (e.g. point of sale terminal). As each item is scanned or otherwise identified, data is sent from the scanner device to a processor in the transaction terminal. The system disclosed herein accesses this data (this can be done in various ways, as will be explained) either on a scan-by-scan basis, or as a collective set of data from a database, to determine the presence (and in some cases the identity) of a number of items processed by the transaction.

0018Using the combination of video analysis in comparison with transaction data, the system can determine if the presence of the item identified in the analysis of the video data has a corresponding presence in the transaction data, and if not, identifies the suspicious activity. As an example, for each detector, the system can compare the set of detection events for that detector to at least a portion of the transaction data to identify at least one apparent discrepancy in a number of items detected by that detector from a number of items indicated in the portion of transaction data. Transaction data such as transaction count (e.g. scan count) or transaction item identity thus represents the presence of an item or a number of items scanned (for an entire transaction), while the detection event data or video count from the video analysis represents the presence (or number) of items that the operator causes to move through the transaction area. Many other embodiments are disclosed herein, details of which are provided in the detailed description section.

0019Other embodiments of the invention include any type of computerized device, workstation, handheld or laptop computer, POS or transaction terminal, or the like configured with software and/or circuitry (e.g., a processor) to process any or all of the method operations disclosed herein. The system may include the video camera(s) for obtaining the video, or the system may be a standalone computer that receives as input video data and scan data collected from one or more POS terminals in one or more retail locations. In other words, a computerized device or a processor that is programmed or configured to operate in any manner as explained herein is considered an embodiment of the invention. The system need not include the video camera and POS terminal, but instead may be an offsite computer system operated by a security service provider that receives the video and transaction data.

0020The processing may be done in real-time as the video and transaction data are collected to identify fraudulent or suspicious activity that may have just occurred (and may include notifying a security officer who can approach the operator and inspect the transaction items and receipt to determine if fraud has occurred), or alternatively, the processing may be done at some point after video and transaction data are collected for one or more transactions (i.e. may be post-processed). If post-processed, the identity of the operator can be maintained and tracked and a history of suspicious activities associated with that operator can be accumulated. During subsequent analysis of transactions for that operator (operators log into a POS terminal with a unique identity, and thus all transactions that the operator performs can be associated with that operator), the system disclosed herein can take into account the history for that operator to adjust a suspicion level assigned to the transaction outcome. Thus, an initial suspicion outcome may not be flagged as fraudulent, but if a second, third, fourth transaction outcome in a certain amount of time (e.g. over several hours, days, etc.) is detected for that same operator, the video associated with those transactions can be automatically identified and forwarded for review by another means, such as human review to confirm fraudulent activity is present.

0021In one configuration, transaction data is processed for many transactions from one or more POS terminals for one or more retails locations, and those transactions that are processed as explained herein that result in an indication of fraudulent or suspicious activity are flagged and the video data for only those transactions can then be further reviewed using other techniques, such as human review, to confirm the fraudulent or suspicious activity as initially identified by the automated (i.e. non-human or machine-based) processing explained herein. Further variations and alternative embodiments will be explained more fully in the detailed description section below.

0022Other embodiments of the invention that are disclosed herein include software programs to perform the steps and operations summarized above and disclosed in detail below. One such embodiment comprises a computer program product that has a computer-readable medium including computer program logic encoded thereon that, when performed in a computerized device having a coupling of a memory and a processor, programs the processor to perform the operations disclosed herein. Such arrangements are typically provided as software, code and/or other data (e.g., data structures) arranged or encoded on a computer readable medium such as an optical medium (e.g., CD-ROM), floppy or hard disk or other a medium such as firmware or microcode in one or more ROM or RAM or PROM chips or as an Application Specific Integrated Circuit (ASIC). The software or firmware or other such configurations can be installed onto a computerized device to cause the computerized device to perform the techniques explained herein as embodiments of the invention.

0023It is to be understood that the system of the invention can be embodied strictly as a software program, as software and hardware, or as hardware alone such as within a processor, or within an operating system. Example embodiments of the invention may be implemented within computer systems, processors, and computer program products and/or software applications manufactured by Stoplift, Inc. of Burlington, Mass., USA.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of embodiments of the invention, as illustrated in the accompanying drawings and figures in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, with emphasis instead being placed upon illustrating the embodiments, principles and concepts of the invention.

<figref idref="DRAWINGS">FIG. 1</figref> illustrates an example configuration of a network environment that includes a video surveillance system and computer system configured with a transaction monitor configured as disclosed herein.

<figref idref="DRAWINGS">FIG. 2</figref> is a flow chart of operation of one configuration of processing of the transaction monitor to detect pass-through activity within the transaction area.

<figref idref="DRAWINGS">FIG. 3A</figref> is a flow chart of processing that the transaction monitor performs in an example configuration in which misidentified items are detected.

<figref idref="DRAWINGS">FIG. 3B</figref> shows one method of detecting activity in a region of interest to indicate presence of an item.

<figref idref="DRAWINGS">FIG. 4</figref> shows processing that the transaction monitor performs in one configuration count items involved in a transaction using analysis of video data.

<figref idref="DRAWINGS">FIG. 5</figref> illustrates one method of processing that the transaction monitor can perform to provide object removal and introduction event detection.

<figref idref="DRAWINGS">FIG. 6</figref> is a flow chart showing processing steps that the transaction monitor employs in one configuration to provide customer and/or employee presence detection &amp; tracking.

<figref idref="DRAWINGS">FIG. 7</figref> is a flow chart of processing steps that show processing that the transaction monitor <b>32</b> performs to provide employee/customer object segmentation.

<figref idref="DRAWINGS">FIG. 8</figref> is a flow chart of processing operations that the transaction monitor can perform to provide an area comparison method used by detectors when performing video analysis.

<figref idref="DRAWINGS">FIG. 9</figref> is a flow chart of processing steps that the transaction monitor performs to provide key frame area comparison.

<figref idref="DRAWINGS">FIG. 10</figref> is a flow chart of processing steps that describe transaction monitor processing that provides a passthrough item detection method.

<figref idref="DRAWINGS">FIG. 11</figref> illustrates example of frames of video data and that show the appearance of the transaction area before and after detection of an event.

<figref idref="DRAWINGS">FIG. 12</figref> is a timeline of events that demonstrates how the transaction manager can determine if there is more than one visual item detection present within the same inter-transaction-item interval.

DETAILED DESCRIPTION

0038The system disclosed herein generally performs video counting or identification of items involved with transactions, as captured in video data, and compares this item identification information with transaction data obtained from a transaction terminal such as a point-of-sale register to identify situations that are suspicious and may indicate fraudulent activity or operator error. By obtaining video data originating from at least one video camera that monitors a transaction area, the system can automatically (i.e. no human involvement needed) analyze the video data to track items involved in a transaction in the transaction area. Using this information, the system can compare the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity.

0039<figref idref="DRAWINGS">FIG. 1</figref> is an illustration of an example environment <b>300</b> suitable for use in explaining example embodiment disclosed herein. Example environment <b>300</b> depicts a retail establishment in which customers <b>305</b> can purchase items <b>307</b>. A transaction terminal <b>34</b> such as a point-of-sale terminal or cash register is under control of an operator <b>308</b> such as a store employee to allow the customer <b>305</b> to purchase the items <b>307</b>. The transaction terminal <b>34</b> includes a scanning device <b>36</b> that is able to detect and scan or otherwise read item identities <b>310</b>, such as UPC barcode symbols or RFID tags affixed to each item <b>307</b> when those items <b>307</b> are brought within a predetermined proximity of the scanner device in the <b>36</b>.

0040In a typical operation of the environment <b>300</b>, the customer <b>305</b> approaches the transaction area <b>301</b> with a set of items <b>307</b> to be purchased. The items <b>307</b> may be contained, for example, with a shopping cart <b>311</b> or other item carrier transported by the customer <b>305</b> to the transaction area <b>301</b>. Alternatively, the customer <b>305</b> may carry the individual items <b>307</b> to the transaction area <b>301</b>. To make the purchase of the items <b>307</b>, the customer <b>305</b> removes the items <b>307</b> from shopping cart <b>311</b> (or from their hands if carrying the items) and places the items into an item input region generally designated as region <b>302</b>-<b>1</b> within the transaction area <b>301</b>. The item input region <b>302</b>-<b>1</b> may be a conveyor belt, countertop or other surface area onto which items to be purchased are placed prior to being detected and read by the scanner device <b>36</b> of the transaction terminal <b>34</b>.

0041The operator <b>308</b> such as a store employee interacts with the transaction terminal <b>34</b> by logging in or otherwise activating the transaction terminal <b>34</b>. This process may involve the operator <b>308</b> providing a unique operator identity to the transaction terminal <b>34</b>. During operation of the transaction terminal <b>34</b> by the operator <b>308</b>, the body of the operator <b>308</b> generally remains within an operator region <b>302</b>-<b>4</b> of the transaction area <b>301</b>. Once logged in, the operator <b>308</b> can begin selecting items for purchase <b>307</b> within the item input region <b>302</b>-<b>1</b>, such as by picking up the individual items <b>307</b> by hand. The operator <b>308</b> passes each item <b>307</b> from the item input region <b>302</b>-<b>1</b> over the scanner device <b>36</b> generally located within an item read region <b>302</b>-<b>2</b>. Assuming proper (i.e., non-fraudulent and non-error) operator operation, the operator <b>308</b> positions the item <b>307</b> such that the item identities <b>310</b> affixed to the item can be detected and scan or read by the scanner device <b>36</b>. In response to the scanner device <b>36</b> detecting an item identity <b>310</b> of the item <b>307</b> just scanned, the transaction terminal <b>34</b> register has the item <b>307</b> as an item to be purchased and usually produces a notification to the operator <b>308</b> such as a beeping noise or tone to indicate that the item <b>307</b> has been successfully identified. In response to the notification, the operator <b>308</b> moves the item <b>307</b> into the item output region <b>302</b>-<b>3</b> which may be another countertop, downstream conveyor belt or the like that holds items <b>307</b> to have been successfully scanned or read by or entered into the transaction terminal <b>34</b>.

0042The operator <b>308</b> repeats this process for each individual item <b>307</b> such that all items <b>307</b> to be purchased are moved from the item input region <b>302</b>-<b>1</b>, over or through the item read region <b>302</b>-<b>2</b> (during which scanning of the item takes place) and into the item output region <b>302</b>-<b>3</b>. In some cases, items <b>307</b> may not contain an affixed item identity <b>310</b> such as fruit, vegetables or the like. In such cases, after removal of the item <b>307</b> from the item input region <b>302</b>-<b>1</b>, the operator <b>308</b> manually enters the item identity into the transaction terminal <b>304</b> a keyboard or other manual input device to allow the transaction terminal <b>34</b> to register the item <b>307</b>. In this manner, after all items <b>307</b> have been identified to the transaction terminal <b>34</b>, the operator <b>308</b> can indicate to the transaction terminal <b>34</b> that the transaction is complete and the transaction terminal <b>34</b> calculates the total price of the items <b>307</b> to be purchased. The customer <b>305</b> then provides payment in that amount to the operator <b>308</b> and proceeds to remove the items <b>307</b> from the item output region <b>302</b>-<b>3</b> for transport out of the retail establishment.

0043As illustrated in <figref idref="DRAWINGS">FIG. 1</figref>, the environment <b>300</b> further includes a transaction monitor <b>32</b> configured in accordance with embodiments of the invention to detect suspicious activity related to a transaction. The environment <b>300</b> also includes a video source <b>30</b> such as one or more overhead video cameras that capture video of the transaction area <b>301</b>. Typically, the video source and will <b>30</b> is mounted in an elevated position sufficiently above the transaction area <b>301</b> to cover and capture video from the various regions <b>302</b>. The transaction monitor <b>32</b> in this example receives, as input, video data <b>320</b> from the video source <b>30</b> as well as transaction data <b>34</b> from the transaction terminal <b>34</b>. While the example environment <b>300</b> illustrates the transaction monitor <b>32</b> as receiving a transaction data <b>330</b> and video data <b>320</b> directly from the video source <b>30</b> and the transaction terminal <b>34</b>, is to be understood that the transaction monitor <b>32</b> may receive these inputs and either real-time or any later time after processing of items or entire transactions by operator <b>308</b> is complete. Additionally, it is not required that the transaction monitor <b>32</b> receive the transaction data <b>330</b> and video data <b>320</b> directly from the video source <b>30</b> and transaction terminal <b>34</b>. In an alternative configuration, these inputs can be received from a videotape machine (or from digital recorded media) or from the transaction database maintained by another computer system besides the transaction terminal <b>34</b>. The video source <b>30</b> may thus be a real-time source such as a camera, or a delayed source such as a recording device such as a VCR or DVR. The transaction terminal <b>34</b> likewise may provide real-time transaction data directly from a POS (e.g., cashier terminal or scanner) or the transaction data may be delayed data from a transaction log database in which POS data is stored.

0044In operation of the system disclosed herein, the transaction monitor <b>32</b> operates to identify suspicious activity associated with the transaction area <b>301</b> such as sweethearting or pass-through activities, by comparing the video data <b>320</b> and corresponding transaction data <b>330</b> in order to identify and report suspicious activity. Generally, this entails the transaction monitor <b>32</b> collecting video data <b>320</b> from the transaction area <b>301</b> including the transaction terminal <b>34</b> in which customers <b>305</b> purchase items <b>307</b> during a transaction. The video source <b>30</b> such as a camera is preferably mounted in an elevated position above the transaction area <b>301</b> to allow video capture of regions <b>302</b> from above, though the system is not limited as such.

0045Generally, the transaction monitor <b>32</b> applies automated (i.e. non-human) video analysis to at least a portion or segment of the overhead video data <b>320</b> to detect the presence of at least one item <b>307</b> associated with the transaction. The transaction monitor <b>32</b> compares the presence of the item associated with the transaction from the video data (automatically detected by image processing techniques as explained herein) to transaction data <b>330</b> indicating items actually purchased by the customer at the transaction terminal (i.e. items <b>307</b> read or scanned by the terminal <b>34</b>) to identify items in possession of the customer <b>305</b> that were not purchased at the transaction terminal (i.e. that were passed through the transaction area <b>301</b> without being scanned or entered into or read by the transaction terminal <b>34</b>). The discrepancy between the presence of one or more items identified via automated processing of the video data <b>320</b> in comparison to items identified within the transaction data <b>330</b> indicates suspicious activity that the system disclosed herein can detect. The suspicious activity may be the result of operator error on behalf of the operator <b>308</b>, or actual fraudulent activity that may include sweethearting or pass-throughs.

0046Depending upon the configuration, the transaction monitor <b>32</b> can analyze all or a portion of the video data captured from the transaction area to automatically detect items based, for example, on activity of objects that pass through the transaction area, activity of objects within a specific region of interest within the transaction area, activity of objects within a plurality of specific regions of interest within the transaction area, activity of objects entering into specific regions of interest within the transaction area and/or activity of objects exiting the specific regions of interest within transaction area.

0047Analysis of all or a portion of the video data produces, in one configuration, a set of detection events indicating detection of one or more items by at least one detector within at least one region of interest <b>302</b> of at least one portion of the video data. Depending upon the embodiment, the transaction monitor <b>32</b> can detect item detection from video analysis in only one region of interest <b>302</b>, or in many regions <b>302</b>. Notice in <figref idref="DRAWINGS">FIG. 1</figref> that the transaction area <b>301</b> is divided or enumerated into several regions <b>302</b>-<b>1</b> through <b>302</b>-N. Each of these areas or regions can be considered a region of interest <b>302</b> and the video data <b>320</b> can capture activity in some, all or only one of these areas that may be indicative of an item involved in the transaction.

0048The transaction monitor <b>32</b> applies a detector to perform image processing in a region of interest. The detector is generally an image processing algorithm that can detect the presence of an item in that region. Item presence can be detected, for example, by applying a detector processing to the input item region <b>302</b>. For each detector (one or more), the transaction monitor <b>32</b> compares the set of detection events for that detector to at least a portion of transaction data (i.e. the portion that contains transaction information that coincides with the video data) to identify at least one apparent discrepancy in a number of items detected by that detector from a number of items indicated in the portion of the transaction data. The transaction monitor <b>32</b> can identify an overall suspicion level for the transaction based on apparent discrepancies identified by the detectors(s).

0049In some configurations, video processing or analysis includes dividing the transaction area <b>301</b> into a plurality of regions (e.g. <b>302</b>-<b>1</b> and <b>302</b>-<b>3</b>) through which objects move in sequence during at least a portion of a transaction. The transaction monitor <b>32</b> can perform automated video detection of an item as the items move through the plurality of regions in a sequence to obtain a pattern represented by one or more video parameters. The pattern thus represents video events of items that moved through the regions during all or part of a transaction. The transaction monitor <b>32</b> can obtain transaction data <b>330</b> identifying items detected by the transaction terminal <b>34</b> during the portion of the transaction that corresponds to the video data that is analyzed, and can automatically comparing the video parameter to the transaction parameter by determining if the pattern representing video events of items from all or part of the transaction indicates a discrepancy from the transaction data identifying items detected by the transaction terminal during all or the same part of the transaction. If a discrepancy exists, the transaction monitor <b>32</b> identifies the transaction outcome to be a suspicious transaction.

0050By having different regions, sequences of different detection events can be used to identify existence of an item in the video data. In such cases, the transaction monitor <b>32</b> can concurrently compare sets of detection events from detectors with the transaction data to identify a discrepancy in a number of items processed in the transaction area. For example, performing automated video detection of the items can include identifying a removal event in an item input region <b>302</b>-<b>1</b> that indicates the operator <b>308</b> has removed an item <b>307</b> from the item input region <b>302</b>-<b>1</b>, and can also include identifying an introduction event in an item output area that indicates an operator has placed an item into the item output area. Thus a sequence of events such as removal, introduction, removal introduction, and so forth can be produced from the video analysis if multiple regions of video data are monitored. This sequence can be time synchronized with transaction data indicating, for example, scans of items, so that a legitimate pattern appears as removal, scan, introduction, removal, scan, introduction and so forth, whereas a suspicious pattern might appear as removal, scan, introduction, removal, introduction. Notice the second scan event is not present, indicating a potential fraudulent or otherwise suspicious activity. By applying an automated machine-based video analysis technique to detect a video event of an item associated with the transaction area, and by comparing this with transaction data indicating transaction events of items associated with the transaction area, the transaction monitor <b>32</b> can determining if the video event is not identified as a transaction event in the transaction data, and in response, can identify a specific segment <b>328</b> of the video data that indicates where the video event not identified as a transaction event exists.

0051As shown in <figref idref="DRAWINGS">FIG. 1</figref>, once suspicious activity is identified, the transaction monitor <b>32</b> can identify and transmit the specific segment of video data that indicates where the video event that is not identified as a transaction event exists (i.e. in the video clip <b>328</b>) to a reviewer to review the segment of video data <b>328</b> to review the suspicious activity of an operator with respect to purchase of items during the transaction.

0052In one configuration, the system disclosed herein provides an approach of actual counting of items or item detection events and is a more robust and accurate method of identifying incidents of items being passed through the transaction without being scanned. The system is unaffected by scan delays and since the system can determine that more items were part of a transaction than were scanned, the system can serve as a very clear indicator that theft or error on behalf of the operator has occurred.

0053Another kind of error, independent of item detection comparisons, is the misidentification of items. This may be due to fraud such as “ticket or label switching” where the barcode code or other item identity <b>310</b> may be overlaid with a bar code from lesser-priced item or “discounting” where the operator <b>308</b> intentionally manually enters a code or item identity into the transaction terminal <b>34</b> for a lesser priced item. In one configuration, the system disclosed herein provides an approach of isolating the item images for comparison directly from video of typically performed transactions. In such configurations, the transaction monitor <b>32</b> can perform image comparison to determine if an image of an item associated a detection event (e.g. an event indicating presence of an item in video data) substantially matches a previously stored image of an item. If the images do not substantially match, the transaction monitor <b>32</b> can identify the transaction as potentially including a label-switching event indicating potential suspicious activity. Because this approach allows the operator <b>308</b> to handle the item <b>307</b> normally, it does not require alteration of the manner in which transactions are typically performed. The system also does not impact or slow down the operator's performance of the transaction. This is of particular importance to professional operators as they are assessed on their speed of performance. Because the operator is not asked to alter his or her behavior, the system can be put in place to detect dishonest employees without their knowledge that anything new has been put in place.

0054Discussed below are a series of flow charts of various embodiments disclosed herein. It is to be understood that these are example embodiments provided to illustrate certain operations of certain configurations, and are not intended to cover all variations of the system disclosed herein.

0055<figref idref="DRAWINGS">FIG. 2</figref> is a flow chart of operation of one configuration of processing of the transaction monitor <b>32</b> to detect pass-through activity within the transaction area. In this example configuration, the transaction monitor <b>32</b> obtains video data originating from at least one video camera that monitors a transaction area <b>301</b>. The video clip <b>2</b> from video data <b>320</b> for at least a portion of one transaction and the corresponding transaction data <b>8</b> (from transaction data <b>330</b> in <figref idref="DRAWINGS">FIG. 1</figref>) for that transaction are analyzed to track items <b>307</b> involved in the transaction in the transaction area <b>301</b>. Any time span of video data <b>320</b> and corresponding transaction data <b>330</b> may be handled (i.e., a portion of or more than one transaction), but for the sake of clarity and simplicity, this example will discuss one transaction being handled at a time.

0056In one configuration, identifying the presence of an item can be done using an area differential technique. In this technique, the transaction monitor <b>32</b> defines a region of interest within the transaction area, such as the input item region <b>302</b>-<b>1</b>. For video of this region, the transaction monitor <b>32</b> operates a detector to automatically identify a first frame of video data (i.e. taken at a first time) that indicates a first set of items in the region of interest <b>302</b>-<b>1</b>. This may be, for example, the initial set of items <b>307</b> placed in that area <b>302</b>-<b>1</b> by the customer <b>305</b> for purchase. Thereafter, the transaction monitor <b>32</b> automatically identifies a second frame of video data (i.e. captured as a second or later time) that indicates a second set of items in the region of interest <b>302</b>, but for which the first set of items is visually distinct from the second set of items. This may be due to the removal of an item from the region <b>302</b>-<b>1</b> by the operator <b>308</b>. The transaction monitor <b>32</b> can automatically indicate the visual distinctness of the first set of items from the second set of items as an event indicating an item existed within the region of interest <b>302</b>-<b>1</b> of the video data.

0057In step <b>10</b>, the transaction monitor <b>32</b> obtains transaction data associated with the transaction terminal <b>34</b> associated with the transaction area <b>301</b>. The transaction data indicates if the item <b>307</b> was registered as a purchase item with the transaction terminal <b>34</b>.

0058In step <b>14</b>, the transaction monitor <b>32</b> compares the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity. In one configuration, the transaction monitor <b>32</b> identifies suspicious activity when the transaction data is missing transaction data for an item for which an event indicates the item existed within the region of interest <b>302</b>. This is an event-based comparison to associate each video detection event with a transaction event from the transaction data.

0059In another configuration of <figref idref="DRAWINGS">FIG. 2</figref>, the video clip <b>2</b> is analyzed in step <b>4</b> to visually detect the presence of items <b>307</b> actually involved in the transaction (event detection can be used). The transaction monitor <b>32</b> analyzes at least a portion of the video data to obtain at least one video parameter concerning at least a portion of a transaction associated with the transaction area <b>301</b>. The video parameter may thus be a video count of items whose presence was detected. The transaction monitor <b>32</b> also obtains at least one transaction parameter originated from a transaction terminal <b>34</b> associated with the transaction area. The transaction monitor <b>32</b> in step <b>10</b> analyzes the transaction data <b>330</b> to obtain records of items <b>307</b> involved in the transaction. From these records can be determined an expected count of items reflected in the transaction data. In step <b>14</b>, the transaction monitor <b>32</b> compares the actual or video count <b>6</b> against the expected, scan or transaction count <b>12</b>. If the counts do match, then the transaction monitor <b>32</b> can flag the transaction as non-suspicious as in step (<b>16</b>). If the counts do not match, then the transaction monitor <b>32</b> flags the transaction as suspicious (e.g. potentially fraudulent) in step <b>18</b>.

0060As explained above, the transaction monitor <b>32</b> identifies a video count of items detected within the transaction area using video analysis and identifies a transaction count of items within the transaction area by analyzing transaction data associated with at least a portion of the transaction. By comparing the video count to the transaction count, if the video count is different from the transaction count, the transaction monitor <b>32</b> can indicate a transaction outcome that represents suspicious activity, such as a fraudulent transaction or operator error. Depending upon the configuration, the transaction monitor <b>32</b> may provide additional information such as a suspicion level based on a metric such as the difference between the actual count and expected count.

0061In certain configurations, other information can be used to rank a suspicion level as being low, or high, or in a range. As an example, in a retail environment, each cashier operating a transaction terminal or cashier station typically logs into the system with a unique identification via his or her register (e.g. via a keyboard) to being processing transactions for customers. Once logged in, the system disclosed herein can performed as explained herein, and if a suspicious transaction is detected (e.g. a transaction count does not match a video count), the system can look at a past history of this particular cashier (based on their unique identity) that can be stored in a database that indicates how frequently this cashier performs transactions that are labeled as being suspicious. In this manner, if it is a rather infrequent occurrence, it may just be a mistake by the cashier (i.e. that person may have accidentally not scanned an item), whereas if the suspicious transaction history of that cashier indicates more frequent occurrences of suspicious transactions, this can be indicated in the suspicion level and can be used for subsequent processing (e.g. flagging the situation to a human operator for further review of the video).

0062In step <b>20</b> of <figref idref="DRAWINGS">FIG. 2</figref>, the transaction monitor <b>32</b> can assign a suspicion level to the segment of video <b>328</b>. The suspicion level indicates a level of suspicion produced from automated video analysis of video data in comparison to transaction data. The transaction monitor <b>32</b> can adjust a suspicion level associated with the transaction outcome based on many factors. Examples include: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0063">An amount of difference between the transaction count and the video count. For example, out of 20 detected items, if 10 are not transacted by the operator, then the suspicion level is much higher than if only 1 of 20 items was not transacted.</li><li id="ul0002-0002" num="0064">A history of an operator processing the transaction indicates that at least one former transaction has been identified for that operator that is indicated in the history as being suspicious. This example was discussed above.</li><li id="ul0002-0003" num="0065">A number of regions of interest in which an item were detected within the video data. For example, if a non-transacted item is visually detected by every detector for every region, then the suspicion/confidence level is higher than if the non-transacted item is detected by only one detector for one region.</li><li id="ul0002-0004" num="0066">A sequence of detection of an item in the video data within different regions of interest within the transaction area (e.g. item removal event followed by introduction event).</li><li id="ul0002-0005" num="0067">A discrepancy between a video identification of an item within the video data in comparison to a transaction identification of the item obtained from the transaction data. The transaction identification may be, for example, the price of the item and/or identity of the item.</li><li id="ul0002-0006" num="0068">A history of a register performing the transaction. This may indicate a faulty register, or this may indicate a register which is more prone to suspicious activity because of location or other factors.</li><li id="ul0002-0007" num="0069">A history of other items identified in the transaction that may be indicative of an item in the transaction for which transaction data may not be required. As an example, if a salad is identified as an item, followed by a fork that is not scanned (and thus initially identified as suspicious), the suspicion level may be adjusted down to reflect the fork is free and does not need to be scanned.</li></ul></li></ul>

0070<figref idref="DRAWINGS">FIG. 3A</figref> is a flow chart of processing that the transaction monitor <b>32</b> performs in an example configuration in which misidentified items are detected. In this configuration, the video data <b>40</b> for one transaction and the corresponding Transaction Data <b>46</b> for that transaction are analyzed. Any time span of video and corresponding transaction data may be handled (i.e., at least a portion of or more than one transaction), but for the sake of clarity and simplicity, this example will discuss one transaction being handled at a time.

0071In step <b>42</b>, the actual item images <b>44</b> are isolated from the video data <b>40</b>. One method of image isolation is described as part of the removal/introduction detection method (discussed below). In step <b>48</b>, the expected item images <b>52</b> corresponding to the items in the transaction data are extracted from a database of item images <b>50</b>. The database of item images may be organized in any fashion, but one way that is convenient in a retail environment would be by SKU number.

0072In terms of the item image database, while the system disclosed herein can rely on a pre-populated database of images, the transaction monitor <b>32</b> can also populate the database with images as the transaction monitor <b>32</b> proceeds to isolate more images from each consecutive transaction. In this manner, the retailer is not required go through the time and expense to provide a database ahead of time. Instead, the database can be essentially learned by capturing and storing video of enough transactions with the same items.

0073In step <b>54</b>, the actual item image is compared against an expected item image. It should be noted that the actual item images can be isolated and compared one at a time or as a group against their corresponding expected item images. In the case where the transaction data contains only a full list of items but no data with respect to the sequence or times at which they were scanned, there is no basis by which apriori correspondence between individual images can be established. Therefore the entire set of actual item images would need to be compared against the entire set of expected item images. If, however, sequence or timing transaction data is available that allows a synchronization process to associate scanned item data with video data (e.g. the timestamp of the scan is substantially synchronized with a timestamp of the video), then the correspondence between images can be established, e.g., the first actual item image is compared with the first expected item image, and so forth. In this case, the option exists to compare each individual actual item image against its one corresponding expected item image.

0074In step <b>56</b>, if the images are found not to match, then transaction is flagged as suspicious in step <b>58</b>. If, as mentioned above, the actual item images are being compared one-by-one against the expected item images, then the option exists to flag a specific item as suspicious rather than the entire transaction.

0075In step <b>60</b>, if the images are indeed found to match, then the transaction is considered non-suspicious. The actual item image or images <b>44</b> may be incorporated into the database of item images (<b>50</b>) if desired. For example, it may not be desirable to introduce new images into a professionally pre-populated database, whereas it is necessary for a “learning” database. Regardless, in the next step <b>62</b>, the transaction (or individual item as described above) would then be flagged as non-suspicious.

0076In this manner, the transaction monitor <b>32</b> performs image recognition of an item detected within the transaction area using automated video analysis to produce a video identity of an item. In doing so, the transaction monitor <b>32</b> obtains at least one transaction parameter by identifying an expected item identity of an item detected within the transaction area. The transaction monitor <b>32</b> then automatically compares the video parameter to the transaction parameter by comparing the video identity of the item to the expected identity of the item. If the video identity is different from the expected identity, the transaction monitor <b>32</b> indicates a transaction outcome that represents suspicious activity.

0077As discussed above, the items involved in a transaction can be counted within a particular region of interest or a combination of multiple regions of interest <b>302</b>. Regions of interest may include any area where items of the transaction may be. In a typical supermarket shopping scenario, for example, the regions of interest may include the shopping cart <b>302</b>-N, customer region <b>302</b>-<b>5</b>, an incoming conveyor belt region <b>302</b>-<b>1</b> (i.e., object input region or area), scanning region <b>302</b>-<b>2</b>, and an outgoing conveyor belt <b>302</b>-<b>3</b> or bagging area (i.e., object output area), and operator region <b>302</b>-<b>4</b>.

0078The operator may vary with the region of interest. For example, in the supermarket scenario, if items are being counted in the transfer from the cart <b>311</b> to the incoming belt region <b>302</b>-<b>1</b>, then the customer <b>305</b> doing the transfer may be considered the operator. Similarly, if the cashier operator <b>308</b> is scanning the items at the transaction terminal <b>34</b>, then the cashier <b>308</b> is considered the operator.

0079As discussed briefly above, in some configurations the counts across these regions of interest <b>302</b> are considered in combination to provide a more robust method of counting. For example, the number of detections across the incoming item input region <b>302</b>-<b>1</b>, scanner or item read region <b>302</b>-<b>2</b>, and bagging or other item output region <b>302</b>-<b>3</b> area in one configuration are compared to see if they coincide. If they do not, the average number of detections may be used. Likewise, the sequence of counts or detection events can also be taken into consideration. For example, each accurately counted item would be counted first at the incoming area as an item removal event, then at the scanner as a scan event, and then again at the bagging area as an item introduction event (each event or count being detected and produced by a detector analyzing that region of interest in the video data <b>320</b>).

0080In another configuration, the count of the item is seen as a sequence or pattern of a certain type of event at a certain area, such as an object removal event when the operator removes an object from the belt or object input region <b>302</b>-<b>1</b>, followed by another type of event such as an introduction event when the operator places the object into the object output region <b>302</b>-<b>3</b> or downstream conveyor belt. Thus a removal event (an operator picking up an object for scanning, and removing that object from the object input area) followed by an introduction event (the operator placing the item down on the output belt or object output area) is seen or interpreted by the system to be a video count.

0081If ever a count is registered only in one stage or region <b>302</b> (e.g. <b>302</b>-<b>1</b> item event detection) of the sequence but not the other one or two regions (no detection at regions <b>302</b>-<b>3</b> and/or <b>302</b>-<b>2</b>), then that one count or event may be considered an error and/or may be labeled as suspicious. For example, if the video data is analyzed and the scan or transaction data is analyzed, the system can determine which scan detection events (i.e. transaction events) or scan counts match up with which video counts (e.g. a video count being, for example, an item removal event followed by an item introduction event if analyzing two regions, or if analyzing only a single region such as the item input region <b>302</b>-<b>1</b>, then a video count or event can be a single removal event of an item <b>307</b> from that region <b>302</b>-<b>1</b> as detected by the video analysis). In such cases, if the scan counts or events match one to one with the video counts of events, there is no apparent fraudulent activity and the transaction is not flagged or labeled to further review. However, if, after time synchronizing the video data with the scan data (such time synchronization may be inherent in the data collection process when the video data is collected concurrently with the transaction or scan data, or may be done in post processing via algorithmic comparison of timestamps of scans with video frames or patterns of detection events), a pattern such as a removal event, followed by an introduction event (for a first object, with no scan or transaction event identifying the presence of the item in the transaction data), followed by another removal event (for a second object) is detected, then the transaction can be labeled as potentiality fraudulent or suspicious. In one configuration, the transaction monitor <b>32</b> correlates video timestamps of events from the analysis of the video data to transaction timestamps of items reflected as having been transacted in the transaction data to identify events indicating an item in the video data that does not have a corresponding record in the transaction data, thus indicating suspicious activity.

0082There are a number of methods which can be used individually or in combination with each other by which video analysis can be performed to identify an item count or detection event within one or more regions of interest <b>302</b>.

0083One of such processing techniques disclosed herein is to count items currently in a region of interest. One way to count the items currently in the region of interest is to count the items visible in a static image of the region of interest. This assumes that the individual items can be segmented and disambiguated well, and this approach may therefore be particularly challenging in the case of connected or overlapped items. In a supermarket or other retail scenario, for example, a small number of items slowly placed on a moving conveyor belt one by one may indeed be spread out on the belt such that they do not touch or overlap with each other. In such a case, counting the items against the background of the belt from a static image of the belt yields the accurate number of items in the transaction. If, however, there is a large number of items on the belt, then the items may begin to pile up against each other at the end of the belt. As they pile up, segmentation of specific items via video analysis may become very difficult and analysis of a static image may yield poor results concerning a count of items placed on the belt.

0084In another configuration, the transaction monitor <b>32</b> can utilize counting of periods of activity. Periods of activity within the region of interest may indicate the introduction, removal, or passing through of an item. Activity measures include simple detection and measurement of motion in the region. A “tripwire” (i.e., looking for motion along an edge of the region of interest) can take into account the direction of entry into the region. In a supermarket scenario, for example, a tripwire along the customer facing end of the incoming belt area may be used to count each time a customer reaches in to place a new item on the belt in the item input region <b>302</b>-<b>1</b>.

0085Two or more tripwires may be established and used to determine that motion of an item travels from one side of the region of interest across to another side of the region of interest. Around the scanner region <b>302</b>-<b>2</b> of a supermarket checkout, for example, two tripwires on either side of the scanner area can detect motion in a particular direction depending on the order in which they are triggered.

0086Another use of tripwires is to cover an entire large region of interest with a series of tripwires (perpendicular to the direction of motion of interest) to detect progression of motion from one end to the other. In a supermarket scenario, such a series of tripwires can be used to detect the forward progression of objects from the incoming area to the bagging area even as the objects are exchanged from one hand to another over the scanner area.

0087If an operator object (i.e. the portion of video data <b>320</b> containing the operator <b>308</b>) can be identified from the image, then the count can increment only upon the operator object itself (e.g. his or her arm or hand) entering and exiting the region. By use of skin detection and/or by identifying the extremities of the operator object <b>308</b>, the detector can be more accurate to trigger an item detection or count only if the operator's hand enters the region of interest (e.g. <b>302</b>-<b>1</b>). If the end of the operator object (i.e., his or her hand) has as part of its color histogram colors other than its own, then it can be considered more likely to be a hand entering or exiting the region of interest with an item.

0088<figref idref="DRAWINGS">FIG. 3B</figref> shows one method of detecting activity in a region of interest to indicate presence of an item. From an image from video source <b>430</b>, step <b>432</b> identifies objects and creates an object map. Step <b>434</b> applies relevance filters (e.g., skin detection, skin plus object color histogram, etc.) to make sure that only objects or activity of interest are being considered. For instance, in a video image of a retail store environment, an operator may reach across the scanner region in order to interact with the touch screen of the transaction terminal. Such activity in the region of interest is not indicative of item presence, and can therefore be ignored by filtering out cases where the operator object connects with the graphical region of the touch screen in the video image. Step <b>436</b> incorporates the current object map into a motion map of the motion over time in the image. Step <b>442</b> then analyzes the motion map to identify motion in the direction of motion <b>438</b> with the region of interest <b>440</b>. If step <b>444</b> determines that the motion has completed all the way across the region of interest, then it is recorded as a video detection event in step <b>446</b>. If step <b>448</b> determines that more video remains, then step <b>450</b> will advance to the next frame of video and continue from step <b>432</b>. If not, then the records of video detection events is returned in step <b>452</b>. As an example, with a direction of interest being from the item input region toward the item output region across the region of interest as the scanner area, the transaction monitor <b>32</b> could observe the scanner area for activity indicative of items present in the transaction.

0089In still other configurations, the transaction monitor <b>32</b> is able to count the introduction or removal of objects that may be items, or an operator or customer object. Objects being newly introduced or removed from one or more regions of interest <b>302</b> in one configuration is an indicator of item detection or count change. For example, in a supermarket scenario, if an item is removed from the incoming conveyor belt, or if it is introduced into the bagging area, (or both cases if considered in combination) then that indicates an additional item involved in the transaction.

0090One way to detect introduction or removal is to detect color histogram change in the region of interest. Another way to detect introduction or removal of an additional object (e.g. an item or an operator object) is to detect object appearance or disappearance. Such image detection algorithms as explained herein constantly incorporate static objects into the background image used for object segmentation. When a new object is added to the image, it will appear as the only one object in the foreground. Similarly, when an object is removed, it leaves behind a “ghost” (i.e., an alteration in the place of the image it once occupied) that will likewise appear as an object in the foreground. In either case, the object is subsequently counted and thereafter incorporated into the background to prepare for counting the next item.

0091One benefit of this method is that it readily facilitates isolation of an item image. When an object such as an item is introduced, that items image may be cut out as the isolated item image. When an item is removed (i.e., a ghost object appears), that item's image may be cut out from the frame that preceded removal. One challenge in the above kinds of methods is that the operator's arm itself will appear as an object in the region of interest. Two example configurations disclosed herein handle this challenge by either disregarding the operator's arm, or use only images without the arm in the region.

0092In order to disregard the arm object, it is first identified. This can be accomplished by checking all the objects in the video data to see which one extends (e.g. using edge detection) from the larger operator object <b>308</b> in the operator region <b>302</b>-<b>4</b> outside (or into) the region of interest <b>302</b>-<b>1</b>. This object can then be assumed to be the arm of the operator <b>308</b>. Skin detection may also be performed to further ensure that the object is indeed an arm and/or hand. Then the arm object may be removed from the region of interest's object map leaving only items of merchandise <b>307</b> that may have been introduced or removed. In order to use only images without the arm in the picture, a tripwire may be used along the edge of the region of interest closest to the operator <b>308</b> to see if any object is crossing it. If no object is crossing it in a particular frame of video, this indicates that the arm is not in the region of interest <b>302</b>-<b>1</b> in that frame. Using this logic, the frames before and after the entrance and exit of the arm can be isolated for determination of item presence. These frames can then be compared to identify item introduction or removal.

0093From the foregoing, the transaction monitor <b>32</b> is able to identify motion of an operator <b>308</b> within a region of interest (e.g. <b>302</b>-<b>1</b>) in the transaction area that indicates the presence of an item <b>307</b> for transacting (e.g., for purchase) within the region of interest. The transaction monitor <b>32</b> can indicate if a record of the item occurs within transaction data corresponding to identifying the motion of the operator can automatically identify a situation when motion of the operator within the region of interest <b>302</b>-<b>1</b> in the transaction area indicates the presence of the item <b>307</b> for transacting, but the record of the item does not occur in the transaction data. In response, the transaction monitor <b>32</b> can indicate suspicious activity.

0094As noted above, modeling the operator such as the cashier <b>308</b> (or a customer <b>305</b> if the environment <b>300</b> in <figref idref="DRAWINGS">FIG. 1</figref> is a self checkout terminal <b>34</b>) can be used to identify the motion associated with handling each item <b>307</b> in the transaction. In a supermarket scenario with a top down view, for example, the operator may be modeled as a torso with two arms extending out from it. The torso of the operator can be identified by its location at the transaction terminal <b>34</b>. The arms can then be identified as two faster moving extremities that extend from the torso. The handling of an item <b>307</b> may then be modeled as the extension of one hand to the incoming belt area or item input region <b>302</b>-<b>1</b>, followed by the bringing together of the two hands as the item is passed from one hand to the other around the scanner area or region <b>302</b>-<b>2</b>, then followed by the extension of the second hand toward the bagging area or item output region <b>302</b>-<b>3</b>.

0095As mentioned above, either a single region of interest may be monitored with one item counting technique, or multiple counting methods may used in a region of interest <b>302</b> to achieve a count for items in that region <b>302</b>. Similarly, the counts and sequence of counts from multiple regions of interest may be used in combination to achieve a more accurate count for the entire transaction.

0096<figref idref="DRAWINGS">FIG. 4</figref> shows processing that the transaction monitor <b>32</b> performs in one configuration count items involved in a transaction using analysis of video data <b>320</b>. One way to count the presence of items is to count the operator's removal of items or introduction of items from or to a region of interest. For example, whenever an operator lets go of an item within a region of interest such as the bagging area, that item can be safely assumed to be an item of merchandise being introduced into the bagging area. Conversely, one way to count removal of items is to count the removal of the items from the item input region <b>302</b>-<b>1</b>.

0097In <figref idref="DRAWINGS">FIG. 4</figref>, when the video source <b>70</b> outputs a frame of video, the first step <b>72</b> is to identify the objects within that image. One way this is typically done is by comparing the current frame against a model of the background so as to extract the foreground components, or the items, in the image. In the next step <b>74</b>, the operator object (i.e. a rendition of the operator <b>308</b> in the video frame) is isolated. One way to do this is to use apriori information of the likely location of the operator <b>308</b>. For example, if it is known that the operator <b>308</b> will stand in a particular confined location such as operator region <b>302</b>-<b>4</b>, then it may be assumed that the largest moving object in that region is the operator <b>308</b>. Another way to identify and isolate the operator object in video data is to identify the operator object reaching across the edge of the boundary of the region of interest <b>302</b>-<b>3</b> as the operator, for example, reaches into the bagging area to place another item <b>307</b>.

0098The next step <b>76</b> checks for the graphical introduction or removal event, such as when the operator object and an item (or the object's ghost as described below) within the region of interest separate from one another graphically. A removal event indicates the operator has picked up an item in the item input region <b>302</b>-<b>1</b>. In step <b>78</b>, if the removal has occurred, then the count is incremented in step <b>80</b>. In step <b>82</b>, if there is more video remaining, then processing advances to the next frame of video in step <b>84</b>, and the loop continues again with step <b>72</b>. When video for the transaction is finished, the count is returned in step <b>86</b>. In the case of item-by-item processing, each removal of an object from the item input area <b>302</b>-<b>1</b> (or introduction of an object into the item output area <b>302</b>-<b>3</b>) creates a corresponding removal (or introduction) event that can then be correlated with the transaction data (or the pair can be correlated, if using multiple regions of interest).

0099<figref idref="DRAWINGS">FIG. 5</figref> illustrates one method of processing that the transaction monitor <b>32</b> can perform to provide object removal and introduction event detection. In <figref idref="DRAWINGS">FIG. 5</figref>, the current image <b>90</b> and an updated background image <b>92</b> is taken as input by step <b>94</b> in which they are compared (by subtraction and thresholding) to create a binary object map <b>96</b>. This object map contains any new objects (i.e., items) in the image that are not part of the updated background, as well as the operator object which is also not part of the updated background. In the next step <b>98</b>, the current operator object <b>100</b> is isolated from the object map. In a supermarket scenario where the cashier operator <b>308</b> stands in a defined space (e.g. region <b>302</b>-<b>4</b>) before the register, this is done in one configuration by finding the largest moving object standing within the that space. (Alternatively, this can also be done by finding the object overlapping a smaller “tag” region which is likely to be touched by the operator.) This object is considered the operator object.

0100In step <b>104</b>, the current operator object <b>100</b> and the previous operator object <b>102</b> are used to define an immediate region of interest <b>106</b>. The non-overlapping region of the previous operator object makes up the immediate region of interest for the following reason: If an item were held by the operator <b>308</b> in a previous frame of video data, it would have been part of the previous operator object. Therefore, if that item were released in the current frame, it would have been left somewhere in the area of the previous operator object. And because it was released, it would not be part of the current operator object. Therefore, if an item were released since the previous frame, its object would appear in the region of the previous operator object that does not overlap with the current operator object. Likewise, if an item were picked up in the previous frame, its ghost (i.e., its absence where it was removed from the background layer) in this frame would appear in the same immediate region of interest described.

0101In the next step <b>110</b>, the object map <b>96</b> is checked to see if there is a new object (i.e., item) or object (i.e., item) ghost that has appeared in the immediate region of interest <b>106</b>. If not, then in the next step <b>120</b>, all regions outside the operator object are incorporated into updating of the background to be used for the next frame. Lastly, the occurrence of no removal/introduction is returned.

0102In step <b>110</b>, if there is indeed a new object in the Immediate Region of Interest, then the next step <b>112</b> is to isolate that object (i.e., item) image, i.e., copy it from the current frame. If an object removal (disconnection from ghost) is being detected, then the image is taken from a frame before the previous frame when the item was picked up.

0103In the next step <b>114</b>, the isolated image of the actual item will be compared with the database of expected objects (e.g., a database of items along with their photographs). Note, this is part of the item misidentification detection method described above. Then the object image is incorporated into the updating of the background to be used for the next frame. Again, in the next step <b>116</b>, all regions outside the operator object, including the removed or introduced object are incorporated into updating of the background to be used for the next frame. Lastly the occurrence of a removal (or introduction) is returned.

0104<figref idref="DRAWINGS">FIG. 11</figref> illustrates example of frames of video data <b>501</b> and <b>502</b> that show the appearance of the transaction area <b>301</b> before (video data frame <b>501</b>) and after (frames <b>502</b>) detection of an event. The detected event in this example happens to be an introduction event in which the operator <b>308</b> places an item <b>307</b> (a milk jug in this example). Each frame <b>501</b> and <b>502</b> is divided into four quadrants, an upper right and left, and a lower right a left. The upper left quadrant of each frame shows the original frame of video data, wherein the lower left quadrant shows the updated background image, produced as explained via the above processing. The upper right quadrant of each frame <b>501</b> and <b>502</b> shows the operator object <b>308</b>, and the lower right quadrant shows the difference binary image or map.

0105In the upper left quadrant of frame <b>501</b>, it can be seen that the operator <b>308</b> has placed an item into the item output region <b>302</b>-<b>3</b> but has not yet released their hand from the item <b>307</b>. In the corresponding quadrant in frame <b>502</b>, the operator <b>308</b> has at that point released the item <b>307</b> as an introduction into the item output region <b>302</b>-<b>3</b>. Notice the difference between the operator object <b>308</b> in the upper right quadrant of frame <b>501</b>, in which the operator object is considered to include the item in the hand of the operator, as opposed to the upper right quadrant of frame <b>502</b> that contains the operator object <b>308</b>, as well as the now separate item object <b>307</b>. The video analysis as previously discussed can detect this difference in images and can indicate this activity as an item detection event, in this case an item introduction event, accounting for the existence of that item in the transaction. In this manner, the video analysis is able to detect items involved in the transaction, and as explained above, the transaction monitor <b>32</b> can use this information in comparison with transaction data from the transaction terminal <b>34</b> to ensure that each item detected in the video data has a corresponding entry (e.g. scan, read, item identify or price) in the transaction data. If this corresponding transaction data does not exist for this object, the transaction monitor <b>32</b> can indicate suspicious activity.

0106Note that for regions of interest with moving backgrounds, such as conveyor belts in supermarkets, a method for adapting the background image is detailed below in the discussion adaptation for moving backgrounds. The system can also incorporate bottom-of-basket detection or in-basket detection to identify from an elevated view items existing underneath or within a possibly moving shopping cart, for example.

0107As exampled above, the transaction monitor <b>32</b> in one configuration can analyze at least a portion of the video data by defining at least one region of interest within the video data and calculating the object map that identifies a change between a current image of the region(s) of interest and an updated background image of the region(s) of interest. Then, by isolating at least one operator object within the region(s) of interest, the transaction monitor <b>32</b> can detect if an analysis of the object map and the operator object identifies either the removal or introduction of an object to the region of interest <b>302</b>. In this manner detection events can be maintained that indicate a count of presence of object in the video data. This can be done for a single region or for many regions <b>302</b>, and can be done on an item by item basis or for larger portions of a transaction or for the entire transaction (i.e. all items).

0108In the process of analyzing at least a portion of video data, the system needs to compensate for autogain functionality found in some cameras or other video sources. Autogain seeks to balance the brightness in a camera's three color channels over the entire image. This could lead to unfortunate shifts in color if a large noisance object (i.e. an object appearing in the scene but not part of the transaction or transaction area) appears within the field of view of the camera. The system needs to essentially undo that shift in color. The challenge is that the system cannot compute any statistics over the entire image, as the system will arrive at the same conclusion that the camera's autogain came to. Instead, the system needs to restrict its focus to the transaction area, more importantly, to the objects in the transaction area that that remained stationary but exhibit a color or brightness shift. One method of accomplishing this the following: The current image is compared (e.g., via subtraction and thresholding) from the current updated background image to get a noisy binary map of foreground/background pixels. The background pixels only are then used to compute the color statistics for the current image and the background image. These background pixels are used because they correspond to the same, static objects in the world, and they exhibit the color shift effect that needs to be correct for. Once the statistics the pixels labeled as background are computed, the change is applied to the current image to bring its colors into correspondence with the colors of the background image and therefore in correspondence with the rest of the images being processed by the system. Once this preprocessing step is done, the usual video processing steps proceed. In this way, the transaction monitor <b>32</b> can compensate for autogain being applied to the video source prior to video analysis.

0109For a configuration in which there are regions of interest being monitored for both removal and introduction of items, the transaction monitor <b>32</b> can define an object input region <b>302</b>-<b>1</b> as a first region of interest and can define an object output area <b>302</b>-<b>3</b> as a second region of interest. In this configuration, detecting an object removal event and/or an object introduction event can include detecting an object removal event when an operator removes an object from the object input area, and/or detecting an object introduction event when an operator places an object in the object output area. In response to detecting a removal event followed by an introduction event, the transaction monitor <b>32</b> can increment a video count as the at least one video transaction parameter in one configuration, or alternatively can generate respective events for the removal and introduction that are then processed during comparison to transaction data to identify items not in the transaction data but that appear in the video data.

0110In one configuration, at least one of the regions of interest includes a conveyor belt that translates a position of objects on the conveyor belt over time as captured within current and previous frames of video data. As an example, the item input area where removal detection events are generated as operator remove items <b>307</b> may be a conveyor belt that feeds the items <b>307</b> to the operator <b>308</b>. In such cases, the transaction monitor <b>32</b> can compensate for the movement or translation of the items as they move from position to position in successive video frames. In particular, the transaction monitor <b>32</b> can analyze at least a portion of the video data and can include correlating previous and current frames of video data within the region of interest to determine an amount of translation of the position of objects on the conveyor belt. The transaction monitor <b>32</b> can account for the amount of translation of the position of objects on the conveyor belt when detecting object removal events and/or object introduction events.

0111<figref idref="DRAWINGS">FIG. 8</figref> is a flow chart of processing operations that the transaction monitor <b>32</b> can perform to provide an area comparison method used by detectors when performing video analysis.

0112When the video source <b>200</b> provides a frame of video, the Video Source (<b>200</b>) outputs a frame of video. The first step <b>202</b> is to identify the objects within that image of that frame. One way this is done is by comparing the current frame against a model of the background so as to extract the foreground components, or the objects, in the image.

0113In the next step <b>204</b>, the operator object is isolated. One way to do this is to use apriori information of the likely location of the operator. For example, if it is known that the operator will stand in a particular confined location, then it may be assumed that the largest moving object in that location is the operator. Another way to do it would be to identify the object reaching across the edge of the boundary of the region of interest as the operator reaching into the bagging area to place another item.

0114In the next step <b>206</b>, the system checks if the operator object extends into the region of interest. In the next step <b>208</b>, if the operator object is in the region of interest, then the system checks, in step <b>210</b>, if the operator was in the region of interest in the previous frame as well. If not, then the previous frame is saved in step <b>212</b> as the key frame <b>1</b>. Key frame <b>1</b>, therefore, is the image of the region of interest before the operator object entered the region of interest. When compared against the image of the region of interest after the operator object leaves the region of interest, the system will be able to determine whether a new object was introduced or removed from the region of interest. Regardless of the decision in step <b>210</b>, the system will then proceed onto checking for more video in step <b>224</b>.

0115If step <b>208</b> decides that the operator object is not in the region of interest, then step <b>214</b> checks if key frame <b>1</b> is already saved. If no, then that indicates that the operator object had not yet entered the region of interest as of the previous frame. And since the Operator Object is still not in the region of interest in the current frame either, it is not worth examining the region of interest for any new object introduction/removal by the Operator. The system then proceeds onto checking for more video in step <b>224</b>.

0116If the decision in step <b>214</b> is yes, that indicates that operator object had previously already entered the region of interest and had not yet exited as of the last frame. Therefore, now that the operator object is not in the region of interest of any longer, the operator object must have just exited. Therefore, the current frame is saved in step <b>216</b> as key frame <b>2</b>.

0117In step <b>218</b>, the object areas for key frame <b>1</b> and key frame <b>2</b> are compared. This process is described in greater detail below.

0118If step <b>220</b> decides that there was indeed a substantive change (e.g., above a threshold amount) in the object area between key frames <b>1</b> &amp; <b>2</b>, then step <b>222</b> increments the item count or produces a detection event.

0119Regardless of the decision in step <b>220</b>, the system proceeds to step <b>223</b> where it resets (or deletes) key frames <b>1</b> &amp; <b>2</b> to signify that the operator object is no longer considered to be within the ROI.

0120The system then proceeds to step <b>224</b> to check if there is more video remaining. If no, then the item count is returned. If yes, step <b>228</b> advances the video the next frame and the system restarts the loop at step <b>202</b>.

0121<figref idref="DRAWINGS">FIG. 9</figref> is a flow chart of processing steps that the transaction monitor <b>32</b> performs to provide key frame area comparison. The diagram describes one example method by which two key frames can be compared to find change in the object area within the region of interest indicating object introduction or removal.

0122In this example, key frame <b>1</b> (<b>250</b>) is the frame of video before the Operator Object entered the region of interest. Key frame <b>2</b> (<b>254</b>) is the frame of video after the Operator Object has exited the region of interest. The Empty Base Image (<b>252</b>) is the image of the scene where the region of interest is without objects. The region of interest <b>259</b> is a binary map highlighting which region of the image is of interest.

0123Steps <b>256</b>, <b>258</b>, <b>260</b>, and <b>262</b> describe processing related to key frame <b>1</b>.

0124In step <b>256</b>, key frame <b>1</b> and the empty base image are compared (by subtraction and thresholding) to create a binary object map <b>258</b>. This object map contains any new objects in the image that are not part of the empty base image, including the operator object and other objects within the region of interest.

0125In step <b>260</b>, the objects within the region of interest are isolated by taking the object map <b>258</b> and masking it with the region of interest <b>259</b>, such that only the objects within the region of interest are left. One way to perform the masking is by performing an “AND” operation between the binary object map and the binary region of interest map. The new resulting object map is the key frame <b>1</b> object map <b>262</b>.

0126Steps <b>264</b>, <b>266</b>, <b>268</b>, and <b>270</b> perform the analogous creation of a key frame <b>2</b> object map <b>270</b> from key frame <b>2</b><b>254</b>.

0127In step <b>272</b>, the amount of translation within the region of interest between key frame <b>1</b> and key frame <b>2</b> is determined. This can be done so by using standard techniques like correlation, sum of squared differences, flow analysis etc. to register the region of interest and find the amount of translation that occurred between key frames <b>1</b> and <b>2</b>. Step <b>272</b> produces a translation amount <b>274</b>.

0128The translation amount <b>274</b> is applied by step <b>276</b> to register the key frame <b>2</b> object map <b>270</b> with the key frame <b>1</b> object map. The registered key frame <b>2</b> object map <b>278</b> is produced.

0129Step <b>280</b> compares the area of the key frame <b>1</b> object map <b>262</b> with the area of registered key frame <b>2</b> object map <b>278</b>. The area of each of the object maps can be computed as the sum of the binary object map.

0130In step <b>282</b>, if there was a significant enough change in the area (i.e., above a threshold), then return Yes that the area changed <b>284</b>. Otherwise, return No, that the area did not change <b>286</b>.

0131<figref idref="DRAWINGS">FIG. 10</figref> is a flow chart of processing steps that describe transaction monitor <b>32</b> processing that provides a passthrough item detection method. Just as passthroughs can be detected by comparing the expected and actual item counts over the time window duration of an entire transaction, so too is it the case that the time window can be of shorter be duration allowing the comparison to be done essentially on an item-by-item basis.

0132During the time window around each individual transaction item entry (i.e., scans, keyed-in entries, RFID reads, etc.), only one item is expected to be processed through the transaction. If the visual count is greater than one, then that indicates that suspicious activity has occurred. Similarly, during the time window around each individual actual visual item detection, there should be a corresponding POS item entry. If there is no POS entry, then it implies that the item actually processed was not expected according to the data, therefore signifying suspicious activity (e.g., a potential passthrough). This scenario is described in more detail below:

0133The item detection process <b>402</b> takes and advances video from the video source <b>400</b> until detecting the next item involved in the transaction. This can be done by any of variety of methods including but not limited to introduction removal detection, area difference comparison, scanner motion detection, etc. This produces an item detection time <b>404</b>.

0134Next, in step <b>410</b>, the system takes transaction item time data <b>408</b> from the transaction data source (<b>406</b>) and compares it with the item detection time <b>410</b>. The comparison indicates whether or not there is a transaction item time that corresponds to the item detection time.

0135Step <b>412</b> checks the result of the comparison. If there is a transaction time corresponding to the item detection time, then the expected and actual activity match, and the current visually detected item is flagged as non-suspicious <b>414</b>. If there is no transaction time corresponding to the item detection time, then the expected and actual activity have a discrepancy, and the current visually detected item is flagged as suspicious and entered into the suspicious item log <b>416</b>.

0136One method by which correspondence can be determined is by whether an as yet uncorresponded transaction item time is present within a sufficiently close time (e.g., less than some static or dynamic threshold) of the item detection. If so, then the detection can be matched to a transaction time; if not, then the item detection is considered uncorresponded. Another method involves keeping a running tally of the total number of items detected as well as the total number of transaction items. When the detected items outnumber the transacted items by more than some threshold (e.g., more than one item), then the last detected item is considered uncorresponded, and the tallies are reset until the next uncorresponded item. Another method is to see if there is more than one visual item detection present within the same inter-transaction-item interval as in the example in <figref idref="DRAWINGS">FIG. 12</figref>. If so, then at least one of the item detections must not correspond to a transaction item time. In such a case, the inter-transaction-item interval itself may be considered suspicious.

0137In Step <b>418</b>, if there is more video remaining, then system proceeds to step <b>402</b> to restart the loop. Otherwise, the system proceeds to step <b>420</b> to return a log of the suspicious items.

0138In this manner, the system provides a passthrough item detection method that can identify passthroughs as suspicious activity.

0139Other configurations include the ability to employ customer and/or employee presence and tracking. Such configurations can use an “over-the-shoulder” perspective camera shot in addition to the simpler top-down camera shot. In cases where the camera is behind where the cashier stands at the cash register looking over his or her shoulder at the counter, register and customer, the method can identify the employee as the object on the near side of the counter by identifying the object occluding the counter. Conversely, the customer object is identified as the object seen over the counter but cut off by the upper edge of the counter. In one configuration, once a customer or employee is identified, his or her location is tracked and labeled by the system.

0140<figref idref="DRAWINGS">FIG. 6</figref> is a flow chart showing processing steps that the transaction monitor <b>32</b> employs in one configuration to provide customer and/or employee presence detection &amp; tracking. In step <b>254</b>, the input image <b>250</b> and the background base image <b>252</b> are compared to produce an all objects binary map showing all non-background objects in the image. One typical method of such comparison is subtraction, followed by thresholding, followed by morphological operations to “clean” the binary map of noise. Next, in the next step <b>256</b>, employee/customer object segmentation is performed on the all objects binary map as described in the following section and in <figref idref="DRAWINGS">FIG. 7</figref> that described employee/customer object segmentation.

0141In step <b>268</b>, if an employee object exists, then the employee object is labeled in step <b>270</b>. In step <b>258</b>, if a customer object does not exist, then the process will advance to the next frame of video in step <b>284</b>. Otherwise, in step <b>260</b>, if the same customer object is present in the previous frame, then step <b>262</b> will determine if the customer object is stationary over time. If so, then the customer object is actually probably an inanimate object such as shopping cart. In that case, the process resets the customer flag in step <b>280</b> and then incorporates the stationary object into the background base image in step <b>282</b>. If the customer object is not stationary over time, then it is assumed to indeed be a human and the object is labeled as a customer in step <b>266</b>. Stationary or not, the process then advances to the next video frame in step <b>284</b>.

0142In step <b>260</b>, if the same customer object was not present in the previous frame, then the customer object is tracked in step <b>272</b>. The tracking is done by tracking shifts in the centroids of the objects of interest. If the customer object is near the counter region (e.g. <b>302</b>-<b>1</b>, <b>302</b>-<b>2</b>, <b>302</b>-<b>3</b>), i.e., as if standing at the counter, then it is determined whether or not is a stationary object like a cart. If so, then the process shifts to step <b>282</b> as described previously. If not, then the customer present flag in step <b>278</b> and the customer is labeled in step <b>266</b>.

0143<figref idref="DRAWINGS">FIG. 7</figref> is a flow chart of processing steps that show processing that the transaction monitor <b>32</b> performs to provide employee/customer object segmentation.

0144In step (<b>204</b>), the input image <b>200</b> and the background base image <b>202</b> are compared to produce the all objects binary map showing all non-background objects in the image. One typical method of such comparison is subtraction, followed by thresholding, followed by morphological operations to “clean” the binary map of noise.

0145In step <b>206</b>, the binary object map is checked for having any objects present. If not, then step <b>208</b> advances video to the next frame and processing continues again from the beginning of the loop. In step <b>206</b>, if objects are present in the binary object map, then they are undergo a labeling operation in step <b>210</b>. Next, in step <b>212</b>, the first labeled object is selected. In step <b>214</b>, if the object is found to only fall in the employee region of interest <b>302</b>-<b>4</b>, then the object is added to the employee only binary map in step <b>218</b>. The employee region of interest <b>302</b>-<b>4</b> is typically the near side of the counter where only an employee or other operator (versus Customer or Item on the Counter) would appear.

0146In step <b>216</b>, if the object is found to fall only in the customer region of interest <b>302</b>-<b>5</b>, then the object is added to the customer only binary map in step <b>228</b>. The customer region of interest <b>302</b>-<b>5</b> is typically just above the top edge of the counter top in the image where a customer would appear if standing at the counter.

0147Next, in step <b>220</b>, if the object is found to fall on in the counter region of interest (e.g. <b>302</b>-<b>1</b>, <b>302</b>-<b>2</b>, or <b>302</b>-<b>3</b>) (the counter top), then the object is likely to be an item on the counter and is added to the counter only binary map in step <b>222</b>. In step <b>224</b>, if the object appears in both the employee and counter regions of interest <b>302</b>-<b>4</b>, and one or more of <b>302</b>-<b>1</b>, <b>302</b>-<b>2</b> and <b>302</b>-<b>3</b>, but not the customer region of interest <b>302</b>-<b>5</b>, then the object is determined to be an employee or operator and is added to the employee only binary map in step <b>218</b>.

0148In step <b>226</b>, if the object is in both customer and counter region of interest <b>304</b>-<b>5</b> and one or more of <b>304</b>-<b>1</b>, <b>302</b>-<b>2</b> or <b>302</b>-<b>3</b>, but not the employee region of interest <b>302</b>-<b>4</b>, then the object is considered to be a customer and is added to the customer only binary map in step <b>228</b>.

0149In step <b>230</b>, if the object is in all three regions (Employee <b>302</b>-<b>4</b>, Customer <b>302</b>-<b>5</b>, and Counter—one or more of <b>302</b>-<b>1</b>, <b>302</b>-<b>2</b> and <b>302</b>-<b>3</b>) then it may be a merged object of the employee image partially overlapping with the image with the customer. Therefore the object is added to the employee and customer binary map in step <b>232</b>.

0150After the object has been processed, in step <b>234</b> it is removed from the all objects binary map so that the next labeled object can be processed.

0151In step <b>206</b>, if there are still objects remaining in the all objects binary map, then processing continues on the next object in step <b>210</b>. Otherwise, if all objects in the image have already been processed, the processing proceeds to the next frame of video in step <b>208</b>.

0152In this manner, the system can identify an operator or customer object so as to be able to distinguish this object from item objects in the video analysis.

0153While the system and method have been particularly shown and described with references to configurations thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims. Accordingly, the present invention is not intended to be limited by the example configurations provided above.