Electronic vehicle identification

TECHNICAL FIELD

This disclosure concerns electronic vehicle identification.

BACKGROUND

Communication facilities such as roads, bridges and tunnels generate road tolls, often the main source of income for many states and cities. A large number of cars, trucks and buses stopping at the payment booths per day to pay the toll can cause significant problems. For example, these objects can restrict the flow of traffic causing traffic jams and lane changes, often increasing the likelihood of accidents and more often bottlenecks. In addition, many people may have delays in reaching their destination, and goods may be delayed in accessing the market, and millions of gallons of fuel may be wasted as vehicles are idling. The environment may experience increased pollution as idling and slow moving vehicles emit pollutants (especially carbon dioxide and carbon monoxide) that can pose a significant threat to the health of drivers as well as payment booth operators.

Some payment booth systems may have a program that requires the driver to rent and then attach a radio transponder to the windshield of the vehicle connecting via radio frequency to the receiving units in the payment booths. However, such programs require drivers to search for the program and register it. These programs may require drivers to make a deposit with a credit card and execute automatic debit account instructions, which can effectively eliminate drivers with credit problems. These programs can also charge the user based on the minimum number of trips, regardless of the actual number of trips. For this reason, many drivers who travel infrequently on a toll road may have little benefit from investing time and money to participate in the program.

Payment booth systems usually include a lane transaction system that records the transaction of each vehicle on a road toll facility and an image system that takes photos of each vehicle that passes the road toll facility. If the lane transaction system detects a violation, the lane system usually sends a "violation" signal to the imaging system. The imaging system can respond to a "violation" signal by sending a photo related to the violation transaction to the internal system for vehicle identification and processing. If the "violation" signal is not received by the imaging system from the lane transaction system after taking a picture of the vehicle, the imaging system usually skips the images. Thus, the internal system only receives pictures of vehicles that have been infringed. After identifying the offending vehicle, the internal system sends the vehicle to justice and / or attempts to pay bills or other tolls.

One problem with existing technology used in road toll systems is that the integration of the imaging system with the lane transaction system can pose a risk to the lane system due to the increased demand for system sources (especially real-time or near real-time messaging to the imaging system). For this reason, it may be undesirable or impractical to integrate the imaging system directly into the lane system. System modifications can reduce the reliability of a proven system. The cost of integration into the legal system can be high.

The publication of the US Patent Application No. US 2004/0167861 discloses an electronic road toll management system including taking a picture of a vehicle triggered by a transaction event constituting an interaction between the vehicle and the object, determining the vehicle identifier based on the picture taken, checking whether the vehicle identifier matches the vehicle identifier provided by the website, and notifying the page about a match.

SUMMARY

General aspects are detailed in the accompanying independent claims.

In one embodiment, the method and / or device embodying the invention is at least part of a road toll system that allows electronic management of the payment of road tolls by vehicles passing a toll system without requiring the road toll system to communicate directly with the toll system image system road (namely, the lane transaction system is independent of the imaging system and does not need to send any signals, including "violation" signals to the imaging system.) Therefore, the toll system is configured to disconnect the imaging system from the lane transaction system, and thus, to minimize or eliminate the need to modify the lane transaction system when installing a new one image system.

In one general aspect, vehicle identification in the toll system includes accessing a set of toll transaction transactions. Each entry in the set indicates the toll transaction between the vehicle and the toll system and includes a transaction descriptor and transaction timestamp. You can access a series of photos of toll transactions. The series includes many photos, each of which is associated with a photo time stamp. The toll transaction entry is identified from the set as a violation transaction entry based on the transaction descriptor. A photo of the toll transaction is selected from the series. The transaction time stamp for the infringement transaction is compared, using a processing device, with the time stamp of the photo of the selected toll transaction transaction. The selected toll transaction photo is identified as a violation photo corresponding to the violation transaction entry based on the comparison result.

Embodiments may include one or more of the following features. For example, transaction time stamps included in the set of toll transaction entries and photo time stamps associated with multiple photos can be based on independent clocks.

Access to the set of toll transaction entries may include obtaining a set of toll transaction entries from the lane transaction system. Access to a series of photos of road toll transactions may include obtaining a series of photos of road toll transactions from an image system that is independent of the road lane transaction system. An imaging system independent of the lane transaction system may include an imaging system not receiving signals from the lane transaction system. An imaging system independent of the lane transaction system may include an imaging system having an internal clock independent of the internal lane of the lane transaction system. Transaction timestamps included in the set of toll transaction entries can be generated based on the internal lane transaction system clock, and photo time stamps associated with multiple photos can be generated based on the imaging's internal clock.

Receiving a set of toll transaction entries from a lane transaction system may include receiving a set of toll transaction entries in an email.

In another general aspect, vehicle identification in a road toll system includes accessing a set of road toll transaction entries. Each entry in the set indicates the toll transaction between the vehicle and the toll system and includes a transaction descriptor and transaction timestamp. Access to a series of photos of toll transactions is obtained. The series includes many photos, each of which is associated with a photo time stamp. The toll transaction entry from the set is identified as a violation transaction entry based on the transaction descriptor. The group of toll transaction entries is selected from a set of toll transaction entries based on the time stamp of the violation transaction entry. The toll transaction photo group is selected from the toll photo transaction series based on the selected toll transaction entry group. The toll transaction photo is identified from the toll transaction photo group as a violation photo corresponding to the violation transaction entry by connecting the toll transaction entry group to the toll transaction photo group.

Embodiments may include one or more of the following features. For example, selecting a group of toll transaction entries from a set of toll transaction entries may include identifying the first time gap having a specific length of time between the transaction timestamps chronologically sequential toll transaction transaction entries sets of toll transaction transactions, chronologically sequential toll transaction entries appear before the entry identified infringement transaction. Selecting a group of toll transaction entries from a set of toll transaction entries may additionally include adding a toll transaction entry to the toll transaction entry group if the toll transaction entry includes a transaction time stamp entering the time window starting at the time corresponding to the time stamp of the following transaction entry after the identified first time interval and ending at the time corresponding to transaction timestamp for the entry of an identified infringement transaction. The specific length of the first time interval may include a length of time between six and ten seconds.

Selecting a group of toll transaction entries from a set of toll transaction transactions may further include adding a toll transaction entry to the toll transaction entry group if the toll transaction entry includes a transaction time stamp entering the time window starting at the time corresponding to the transaction time stamp of an identified transaction entry violations and ending in a time corresponding to a certain amount of time after the time stamp transaction entry of the identified infringement transaction. The specified amount of time after the transaction timestamp of an identified violation transaction can be between thirty seconds and one minute.

Selecting a group of toll transaction photos may include selecting a toll transaction photo from the series of toll transaction photos corresponding to the transaction entry immediately following the identified first time interval. A toll transaction photo may be added to the toll transaction photo group if the toll transaction photo is associated with a photo time stamp entering the time window starting at the time corresponding to the photo time stamp associated with the selected photo of the toll transaction and ending at a specific time after Transaction timestamp of an identified violation transaction entry.

Selecting a group of toll transaction entries from a set of toll transaction entries may further include identifying a second time gap having a specified length of time between the time stamps of the chronologically sequential transaction toll transaction entries of the set of toll transaction entries, chronologically sequential toll transaction entries appear after the identified entry infringement transactions. The toll transaction entry may be added to the toll transaction entry group if the toll transaction entry includes a transaction time stamp entering the time window starting at the time corresponding to the transaction time stamp of the identified infringement transaction entry and ending at the time corresponding to the time stamp of the immediately preceding transaction entry identified second time interval.

Selecting a group of photos of road toll transactions may include selecting from the series of photos of road toll transactions the first photo of the road transaction corresponding to the transaction entry immediately following the identified first time interval. The second photo of the toll transaction corresponding to the transaction entry immediately before the identified second time interval is selected for a series of photos of the toll transaction. A toll transaction photo may be added to the toll transaction photo group if the toll transaction photo is related to a photo time stamp entering the time window starting at the time corresponding to the photo time stamp associated with the selected first photo of the toll transaction and ending at the time corresponding to photo timestamp associated with the selected second photo of the toll transaction.

Selecting a group of toll transaction entries from the set of toll transaction entries may include selecting a toll transaction entry from the set of toll transaction entries indicating a toll transaction between a toll system and a vehicle that has been positively identified, the selected toll transaction entry includes a transaction time stamp . that is earlier than the transaction timestamp contained in the entry of the identified infringement transaction. A toll transaction entry may be added to the toll transaction entry group if the toll transaction entry includes a transaction time stamp entering the time window starting at the time corresponding to the time stamp of the selected toll transaction and ending at the time corresponding to the transaction time stamp of the identified infringement transaction entry .

Selecting a group of toll transaction entries from a set of toll transaction transactions may further include adding a toll transaction entry to the toll transaction entry group if the toll transaction entry includes a transaction time stamp entering the time window starting at the time corresponding to the transaction time stamp of an identified transaction entry infringements and ending at a specified time after the entry transaction timestamp identified infringement transaction.

Identifying the toll transaction photo from the toll transaction photo group as a violation photo may include matching one-on-one toll transaction photo in the toll transaction photo group with each toll transaction entry in the toll transaction entry group. One-on-one matching of each toll transaction photo with each toll transaction entry may include arranging, in sequential chronological order, the toll transaction entries in the group of toll transaction entries based on the time stamps of the road toll transactions and arranging sequential chronological order, photos of road toll transactions in a group of photos of road toll transactions based on photo time stamps. Each toll transaction entry can be matched to the place in the toll transaction queue entry, and each toll transaction photo can be matched to the place in the toll transaction photo queue. The toll transaction entry can be selected, and the selected toll transaction entry can be matched to the toll transaction photo provided that the toll transaction entry is matched to the place in the toll transaction entry queue that corresponds to the place in the toll transaction photo queue matched to picture the toll transaction.

Additional toll transaction entries can be added to the toll transaction entry group if the number of toll transaction entries in the toll transaction entry group is less than the number of toll transactions photos in the toll transaction photo group. Additional toll transaction photos can be added to the toll transaction photo group if the number of toll transactions photos in the toll transaction photo group is less than the number of toll transactions entries in the toll transaction entries group.

The selected toll transaction entry and the matched photo of the toll transaction may be indicated as incorrectly matched, provided that the difference between the transaction time stamp of the selected toll transaction entry and the time stamp of the photo of the matched toll transaction photo is greater than the specified value. The value specified can be one second.

The time interval between two transactions can be calculated based on the time stamps of the road toll transactions of the chronologically sequential entries of the road toll transactions. The corresponding time interval between two transactions based on time stamps photos of two chronologically sequential photos of tolls can also be calculated, two chronologically sequential photos of tolls are matched to two chronologically sequential entries of toll transactions. Two chronologically sequential photos of tolls and two chronologically sequential entries of toll transactions can be indicated as incorrectly matched, provided that the difference between the time period and the corresponding time period is greater than the specified value. The specified value can be four seconds.

Identifying the toll transaction photo from the toll transaction photo group as a photo of the violation may include indicating as a photo of the toll transaction photo matching the place in the toll transaction photo queue that corresponds to the place in the toll transaction entry queue matching the violation transaction entry.

Details of one or more embodiments are detailed in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an embodiment of the electronic toll management system.

FIG. 2 is a flowchart of an example embodiment of an electronic road toll management system related to the management of tagged vehicle identifiers.

FIG. 3 is a flowchart of an embodiment of the electronic road toll management system related to toll management.

FIG. 4 is a flowchart of an embodiment of the electronic road toll management system related to toll management.

FIG. 5 is a flowchart of an embodiment of the electronic road toll management system associated with postal address verification.

FIG. 6 is a block diagram of an embodiment of the electronic toll management system.

FIG. 7 is a flowchart of an embodiment of the electronic road toll management system related to vehicle identification.

FIG. 8 is a flowchart of an embodiment of the electronic road toll management system associated with vehicle identification.

FIG. 9A-9C is a flowchart of an embodiment of the electronic road toll management system related to vehicle identification.

FIG. 10 is a block diagram of an embodiment of the electronic toll management system.

FIG. 11 is a group of transaction entries generated by the lane transaction system.

FIG. 12 is an illustration of a group of photo / sensor files.

FIG. 13 is a flowchart of an embodiment of the electronic toll management system associated with the selection of transaction entry groups and corresponding groups of photo / sensor files.

FIG. 14 is a flowchart of an embodiment of the electronic toll management system associated with photo / sensor file identification for each violation transaction entry.

FIG. 15 is an illustration of the group of transaction entries of FIG. 11 matched one-on-one with the group of photo / sensor files of FIG. 12.

FIG. 16 is a flowchart of an embodiment of the electronic road toll management system.

FIG. 17 is an example user interface.

FIG. 18 is a bar graph showing the time interval between the time stamp of entering a lane transaction and the corresponding time stamp of a transaction photo.

FIG. 19 is a bar graph showing the time interval between current and previous lane transactions by photo timestamps and transaction entry timestamps.

Similar reference symbols in different drawings indicate similar elements. DETAILED DESCRIPTION

In one embodiment, the toll system allows electronic management of the payment of tolls by vehicles passing through the toll object without requiring the toll road system of the toll system to directly communicate with the toll system image system. For this reason, the toll system is configured to disconnect the imaging system from the lane transaction system, and therefore, to minimize or eliminate the need to modify the lane transaction system when installing a new imaging system.

The above road toll system includes a computer road toll management system having a module for obtaining photos and transaction data of the road lane (ILDM). ILDM includes a lane transaction system, photo capture module, and video server.

The lane transaction system is configured to capture transaction data for each vehicle that passes or otherwise makes transactions with the toll object. Transaction related data may include, for example, transaction type, transaction time (e.g. transaction timestamp), vehicle classification data (e.g. number of vehicle axles), transponder information about the vehicle, if applicable, and accrued toll. The lane transaction system may be an existing or conventional lane transaction system. Therefore, when the lane transaction system may have the ability to send "violation" signals to the imaging system, this ability need not be used. The image system or image acquisition module can work independently of the lane transaction system and, therefore, does not need to receive any signals from the lane transaction system or directly from the lane system.

The lane transaction system is configured to periodically generate and send lane activity reports to the video server. The lane activity report includes data on lane transactions for vehicles that have passed through the toll object during a given time window (e.g. day). The lane activity report usually includes a chronological list of lane transaction entries, each of which corresponds to a single vehicle transaction with the toll object. Alternatively, the lane transaction system may allow access to a database with transaction data or copies of such data.

The image acquisition module uses sensors, such as, for example, laser sensors, to detect passing vehicles, usually when they enter or otherwise start passing through a toll object. Laser sensors trigger cameras and optionally other sensors that are configured to capture image data / sensors for each passing vehicle detected by the laser sensors. In particular, unlike conventional road toll systems, the image acquisition module does not have to receive "violation" signals directly from the road lane transaction system and does not have to discard photos in response to no reception of such signals.

The image acquisition module can upload to the video server a photo / sensor file for each vehicle that passes or makes a transaction at a road toll facility.

Each photo / sensor file may include data corresponding to at least one photo or image of the transacting vehicle (e.g., rear vehicle image), may optionally include sensor data, and may also include a time stamp indicating when the photo and optional sensor data have been taken.

The video server can receive the lane activity report from the lane transaction system and can receive photo / sensor files from the photo acquisition module. The video server synchronizes or adjusts each lane transaction entry in the lane activity report with a single image / sensor file received from the image acquisition module. For this reason, the video server determines one-on-one correspondence between lane transaction entries in the lane activity report and photo / sensor files.

The video server typically determines one-on-one correspondence between lane transaction entries and photo / sensor files by first syntactic analysis of the lane activity report into groups of chronologically sequential transaction entries separated by, or closed, transaction entries corresponding to "border transactions". A border transaction is a transaction that has a transaction entry that can usually be easily referred to an easily identifiable image or picture taken. For example, a borderline transaction may be a transaction involving a multi-axis vehicle (namely, a vehicle with three or more axles). If the lane transaction entry indicates that the vehicle making the transaction has three or more axles, the appropriate vehicle photo can be easily selected from the captured photos because usually the majority of the captured photos are photos of vehicles with only two axles.

Due to the above, lane transaction data can be parsed into groups of chronologically sequential transaction entries closed by borderline transaction entries, and photo / sensor files can be parsed into corresponding groups of chronologically sequential photo / sensor files closed by photo / sensor files having photos of borderline transactions (namely easily identified photos, which correspond to border transactions).

Once the transaction entry groups and the corresponding image / sensor file groups have been identified, the video server can link each transaction entry of the given transaction entry group to the image / sensor file from the corresponding image / sensor file group. The transaction entry for matching the photo / sensor file may use borderline transactions as a reference point and may match, in turn, each transaction entry after the borderline transaction entry with each image / sensor file following the image / sensor file having the corresponding borderline transaction photo. Due to the lack of synchronization between the lane transaction system and the image acquisition module, and the imperfect process of capturing transaction-related data and photos, the matching process usually involves adding completed transaction entries and / or photo / sensor files to ensure that the number of transaction entries in the group is same as the number of photo / sensor files in the corresponding group.

The video server can be configured to confirm whether the matching process was successful by checking if the differences between the transaction entry time stamps and the matching time / image file sensors are within the specified tolerance level. The video server can also be configured to check if the differences between the time intervals between transactions as determined from the transaction entry time stamps and the corresponding time intervals as specified from the matching time stamps of photo / sensor files are also within the specified tolerance level.

The video server can send customized photo / sensor files and transaction entries to the photo processing module of the computer toll management system. The photo processing module processes photo / sensor files for extracting vehicle identification data. The toll management computer uses vehicle identification data to identify vehicles. Once the vehicles have been identified, the toll management computer gains access to matched transaction data entries to identify the vehicles and fees, or otherwise allows payment to be received from the individual or company associated with the identified vehicle.

FIG. 1 is a block diagram of an embodiment of the electronic toll management system 10. The system 10 is configured to capture the vehicle identifier 31 interacting with the object 28 and to notify external systems 34 of such interaction. For example, system 10 may allow the toll authority to capture the vehicle identifier 31, such as license plate data, from a vehicle 30 traveling by road with tolls, and then to notify law enforcement authorities whether the captured vehicle identifier matches the registration plate previously marked by the services law enforcement.

The toll management system 10 may also manage the toll on the person associated with the vehicle 32 based on the interaction between the vehicle 30 and the object 28. For example, the system 10 may capture information from the license plate of the vehicle 30 and identify the registered owner of the vehicle. Then, the system will provide the owner, via a communication channel such as the Internet, with an account to pay the fee or dispute regarding the fee. The toll management system 10 may send a bill inviting page 32 to pay using a postal address that has been verified at one or more postal address sources. The system 10 may automatically capture a photo of the vehicle 30 upon activation by the interaction of the vehicle with the object. This photo capture can be done using photo processing technology without the need for a radio transponder (e.g. RFID device) in the vehicle.

The electronic toll management system 10 includes a toll management computer 12 which can be configured in a decentralized or centralized manner. Although one computer 12 is shown, one or more computers may be configured to implement the disclosed techniques. Computer 12 is connected to object 28, which may charge a fee for interacting with the object. Examples of facility 28 include a toll facility (managed by a toll authority) such as a toll road, bridge with tolls, tunnel, parking facility, or other facility. The fee can be based on the interaction between vehicle 30 and object 28. Examples of interactions that may involve a fee include the distance traveled by the vehicle through the object, the period of time the vehicle is present on the object, the type of vehicle interacting with the object, the speed at which the vehicle moves through the object, and the type of interaction between vehicle and object.

Object 28 can process vehicles including trucks, buses, or other vehicles. For easy explanation, the system 10 represents a single object 28 in interaction with one vehicle 30 and a person 32 associated with that vehicle. However, in other embodiments, the disclosed techniques may be configured to operate with one or more vehicles interacting with one or more objects extending at different geographical locations.

The toll management computer 12 includes a photo acquisition module 24 configured to detect the presence of a vehicle, acquire one or more vehicle photos, and send a photo (s) to the photo processing module 25 for further processing. Module 24 may include equipment that acquires photos based on the physical environment in which it is used. For example, for open road applications, photo acquisition equipment can be mounted above the road, on existing structures, or on purposely built gates. Some open road applications may also use equipment mounted in or next to the road. Lane based applications (or payment booths) may use equipment mounted on a physical structure next to each road lane instead of or in addition to equipment mounted on the top or on the road.

The image acquisition module 24 may include image components such as vehicle sensors, cameras, digitizing systems, or other components. Vehicle sensors can detect the presence of the vehicle and provide a signal that activates the camera to capture one or more images of the vehicle. Vehicle sensors may include one or more of the following:

(1) Laser, sound, microwave devices - these devices commonly used in Intelligent Transport Systems (ITS) applications can recognize the presence of a vehicle and provide information on vehicle size, classification, and / or speed. These sensors can be configured to provide additional vehicle information that can be used to identify the vehicle and its use in a road toll facility, including travel time and compliance with traffic regulations.

(2) Loops - these sensors can detect the presence and type of vehicle by recognizing the presence of metal masses using a wire loop embedded in the road. Loops can be used to support more sophisticated sensors. Loops can also be used as the primary data source for vehicle detection, classification of vehicle launch cameras, and delivery of vehicle signature data (e.g., based on the use of a loop system with an intelligent loop control program such as Diamond Consulting's IDRIS ® system from Buckinghamshire, United Kingdom ).

(3) Trans-beam sensors - these sensors can emit a continuous beam across the road, and detect the presence of the vehicle based on beam breaks. This type of sensor can be used in installations where traffic is channeled into the toll booth lanes.

(4) Optical sensors - the vehicle can be recognized using cameras for continuous monitoring of road images for changes indicating the presence of the vehicle. These cameras can also be used to record photos for vehicle identification.

Cameras can be used to capture photos of vehicles and their identification features. For example, they can be used to generate a vehicle identifier such as a vehicle registration number based on a photo of the license plate. Cameras can be analog or digital, and can take one or more photos of each vehicle.

Digitizing systems transform photos into digital form. If analog cameras are used, the cameras can be connected to separate computerized digital equipment. This computer equipment may include a specialized processing device for analog-to-digital conversion or may be based on an output device installed on a general purpose computer that may perform additional functions such as image processing. Lighting can be used to provide adequate and consistent conditions for obtaining a photo. Lighting may include gating lights or continuous lighting, and may emit visible and infrared light. When gating lights are used, they can be turned on by input from the vehicle sensor (s). Other sensors such as light sensors may be required to control the image acquisition module 24 and provide consistent results.

Once the photo acquisition module 24 has captured the vehicle photos, the photos can be sent to the photo processing module 25. The photo processing module 25 can be located in the same place as the photo acquisition module 24 and the photo computer 12, in a remote location, or in a combination of these places. Module 25 can process a single photo for each vehicle or multiple photos for each vehicle, depending on the functionality of the photo acquisition module 24 and / or business requirements (e.g., accuracy, jurisdictional requirements). If multiple photos are used, each photo can be processed and the results can be compared or combined to improve process accuracy. For example, more than one photo of a rear registration plate, or photos of both number plates, front and rear, can be processed and the results compared to determine the most likely registration number and / or confidence level. Photo processing may include identifying the vehicle's distinctive features (e.g., license plates) in the photo, and analyzing these features. The analysis may include optical character recognition (OCR), template matching, or other analytical techniques.

The toll management system 10 may include other systems capable of processing essentially in real time located at the location where the images are obtained to reduce data communication requirements. In an embodiment of local photo processing, the results can be compared to a list of authorized vehicles. If the vehicle is recognized as authorized, photos and / or data may be rejected rather than sent for further processing.

Photographs and data may be sent to a central processing facility such as an image database 14 working in conjunction with the billing device 22. This process may involve a computer network, but may also contain physical media from another computer located at the location of the acquisition (namely, the object 28). In general, information may be temporarily stored on a computer at the place where the images are obtained in case the network is unavailable.

Pictures received at a central location may not be processed. All raw photos can be taken care of as described above. The data resulting from the processing of the image (remote or central) can be divided into two categories. Data that meets the criteria applicable to the application or applicable to the jurisdiction of trust can be sent directly to the billing device 22. On the other hand, data results that do not meet the required confidence levels can be marked for additional processing. Additional processing may include, for example, determining whether multiple vehicle images are available, and independent image processing and comparison of results. This may include feature-by-feature comparisons of optical character recognition (OCR) results on a license plate photo. In another example, the photo (s) may be processed by one or more specialized algorithms for recognizing license plates of certain types or styles (such as tables from a specific jurisdiction). These algorithms can take into account the importance of the characters in each position on the license plate, the expected effect of certain pattern features (such as background images), or other style specific criteria. The processed image may be sent based on the results of the pre-processing, or may include processing by all available algorithms to determine the highest level of trust.

Preliminary data can be compared to other available data to increase the level of trust. Such techniques include:

(1) Comparison of OCR processed license plate data with lists of valid number plates within the billing system or vehicle registration authority of the relevant jurisdiction.

(2) Comparison of other data obtained from sensors at the shooting location (such as the size of the vehicle) to the known characteristics of the vehicle registered under the registration number recognized by the system, in the recognized jurisdiction or in many jurisdictions.

(3) comparing the registration of other data to records from other places (e.g. records of the same or similar vehicle using other objects on the same day, or using the same object at different times).

(4) Comparison of vehicle fingerprint data with stored vehicle fingerprint data lists. The use of vehicle fingerprint data to identify a vehicle is described in detail below.

(5) Manual review of photos or data to confirm or delete the results of automatic processing.

If additional processing provides a result with a particular level of trust, the resulting data can then be sent to the billing device 22. If the required level of trust cannot be achieved, the data can be stored for future reference or deleted.

The billing device 22 processes the information captured during the interaction between the vehicle and the toll object, including the vehicle identifier as defined by the photo processing module 25 to create a transaction event corresponding to the interaction between the vehicle and the object. The device 22 may store a transaction event in an accounting database 16 for subsequent payment processing. For example, the invoicing device 22, alone or in conjunction with customer management module 26 (described below), creates payment requests based on transaction events. Transaction event data may include individual fees based on the presence of the vehicle at specific points or facilities, or travel fees based on the origin of the vehicle and the destination associated with the facility. These transaction events can be compiled and invoiced, for example, by one or more of the following methods:

(1) Deduction of payment from an account established by the vehicle owner or user. For example, the accounting database 20 can be used to store the invoice record of each vehicle owner. In turn, each account entry may include a reference to more transaction events. A paper or electronic statement of payment can be issued and sent to the registered owner of the vehicle.

(2) Generating a paper bill and sending it to the vehicle owner using the postal address from the vehicle's registration record.

(3) Presentation of an electronic invoice for a specific invoice for the vehicle owner, kept either by computer 12 or a third party.

(4) Submission of the invoice to the competent vehicle registration authority or tax authorities, allowing the collection of a fee during the renewal process of the vehicle or during the tax collection process.

Billing can occur at regular intervals, or when transactions meet certain thresholds, such as the maximum time range or the maximum dollar amount of road tolls and other charges due. Owners can combine billing for many vehicles by setting up an account on the computer 12.

The client management module 26 may allow the user to interact with the toll management computer 12 through a communication channel such as a computer network (e.g., Internet, wired, wireless, etc.), telephone connection, or other channel. The user may include a person associated with the vehicle 22 (e.g., vehicle owner), a public or private body responsible for managing the facility 28, or another user. The client management module 26 includes a combination of a computer hardware module and software configured to interact with the client such as the account management module 26a, the dispute management module 26b and the fee processing module 26c. Module 26 has secure access techniques such as encryption, firewalls, passwords, or other techniques.

The account management module 26a allows users such as the vehicle driver to create an account in the system 10, link multiple vehicles to this account, review transactions for this account, review photos related to these transactions, and make payments to the account. In one embodiment, the user responsible for the facility may have access to accounting and aggregate information connected to the vehicle drivers who have used the facility.

The 26b dispute management module can allow clients to challenge specific transactions on their accounts and resolve disputes using a computer 12 or third parties. Disputes may arise during the billing situation. Module 26b can help you resolve such disputes in an automated manner. Module 26b can provide the customer with access to the "eSolutions" section of the website of the controlling / billing authority. Customers can submit a dispute and download a photo of their transaction, which is the subject of the dispute. If there is no match (namely, the customer's vehicle is not a vehicle in the photo frame), the invoice may be sent for third-party assessment, such as arbitration. In the most likely case, the photo will show that it was correctly billed for the customer's vehicle. Dispute management can use encrypted security, in which all text and photos are sent via a computer network (e.g., the Internet) using very strong encryption. Photographs of proof of presence may be embedded in the dispute resolution message as an electronic watermark.

The 26c fee processing module provides functionality for manual or electronic payment processing, depending on the transfer received. For example, if the transfer of the fee is in the form of a paper check, then scanning devices may be used to convert the paper information into an electronic format for further processing. On the other hand, if electronic payment has taken place, then standard electronic payment techniques may be used. The 26c fee processing module can support billing methods such as traditional mailing, electronic payment (e.g. using a credit card, debit card, smart card, or Automated Clearing House transactions), periodic billing (e.g. sending invoices monthly, quarterly, after reaching the threshold, or otherwise). The 26c fee processing module can support discounts and additional fees based on the frequency of use, method of payment, or time of use of the facility. The 26c toll processing module can also support toll collection methods such as traditional check processing, toll processing during vehicle registration renewal (with increasing interest), electronic payment, direct bank debit, credit cards, prepayment, customer initiated payments (as often as the customer wants), or provide discounts for various purposes.

The toll management computer 12 communicates with external systems 34 using one or more communication techniques compatible with the communication interfaces of the systems. For example, communication interfaces may include computer networks such as the Internet, Electronic Data Interchange (EDI), batch file transfers, message systems, or other interfaces. In one embodiment, external systems 34 include law enforcement services 36, postal authorities 38, vehicle registration authorities 40, insurance companies 42, service providers 44, financial systems 46 and internal security agencies 48. External systems 34 may include private or public organizations, which includes one or more geographical locations such as states, regions, countries, or other geographical locations.

The toll management computer 12 can connect and exchange information with law enforcement services 36. For example, when vehicles are identified, the computer can deliver transactions in essentially real time to law enforcement systems, in formats specified by law enforcement services. Transactions may also be submitted for vehicles carrying hazardous materials or in violation of traffic rules (e.g. speeding, weight violations, missing number plates). If suitable sensors are in place (e.g. laser / sound / microwave sensors as described above, weight sensors, radiation sensors). Alternatively, vehicle entries can be compiled and forwarded in series, based on lists provided by law enforcement services.

A database of tagged vehicle identifiers 20 can be used to store lists provided by law enforcement services. The term "marked" refers to the concept that law enforcement services provided a list of vehicle identifiers that these services indicated (indicated) that they wanted the toll object to be monitored. For example, when a vehicle has been stolen and reported to the police, the police may send a list of tagged vehicle identifiers to database 20. When a police-marked vehicle travels by the object, the photo processing module 24 determines the vehicle identifier associated with the vehicle and determines through certain interfaces that this particular vehicle is being searched by the law enforcement authority. Law enforcement agencies may want to be immediately informed of the location of the vehicle (and driver), the time it was detected in that place and the direction it was heading. Computer 12 can notify the law enforcement mobile units in substantially real time. In addition, law enforcement authorities can automatically mark vehicles based on the expiration of registration, the occurrence of the date of the traffic court, or other incident. This, in turn, can keep illegal drivers off the road and increase your income.

The toll management computer 12 can connect and exchange information with postal authorities 38. Since the disclosed techniques require road toll authorities to transform the collection of drivers' fees while traveling into collecting arrears, it is important that bills are sent to the correct driver / owner of the vehicle . To minimize the possibility of sending the invoice to the wrong person, computer 12 supports address compliance. For example, before sending the invoice, the computer 12 verifies that the address provided by the vehicle department matches the address provided by the postal authority. The vehicle database can then be updated with the most accurate address information associated with the vehicle owner. Because this occurs before sending the invoice, errors in the invoices can be reduced.

The toll management computer 12 can connect and exchange information with vehicle registration authorities 40. Registration authorities 40 provide an interface for the exchange of information related to vehicle owners, owner addresses, vehicle characteristics, or other information. Alternatively, this information may be accessed by third party data providers rather than through an interface to public vehicle registers. The accuracy of registers in various databases used by computer 12, including vehicle ownership and owner addresses, can be periodically verified against third party databases or government registers, including vehicle registers and address registers. This can help ensure the quality of property and address registers, and reduce errors in invoices and returned correspondence.

The toll management computer 12 can connect and exchange information with insurance companies 42. Insurance companies can mark vehicle identifiers in a similar way to law enforcement authorities 36. For example, the database of marked vehicle identifiers 20 may include vehicle license plate numbers with expired insurance indicating, that such a driver may drive illegally. The computer can notify law enforcement authorities, as well as insurance companies, whether a marked vehicle has been detected using a specific facility.

The toll management computer 12 can connect and exchange information with service providers 44. For example, the computer 12 may submit a set or interfaces in real time for delegating billing and charging functions to accounting service providers or collection agencies.

The toll computer 12 can connect and exchange information with financial systems 46. For example, to manage payment and collection, the computer 12 can connect to credit card processors, banks, electronic third party billing systems. Computer 12 can also exchange information with accounting systems.

The toll management computer 12 can connect and exchange information with the internal security agency 48. The internal security office can automatically provide a list of people for use in the database of marked vehicle identifiers 20. For example, registered drivers who are on a visa in this country can be automatically marked when the visa expires. The computer 12 will then notify the internal security office 48 that the marked vehicle identifier associated with the person was detected while driving in the country, including information about the time and place of the vehicle.

As described above, data captured from the toll site flows to the image database, and is obtained from the image database by the billing device. In another embodiment, the toll computer detects, for each vehicle, the interaction between the vehicle and the toll object, takes pictures and generates a data record. The data log may include the date, time and location of the transaction, reference to the photo file, and any other data available from the sensors on the object (e.g. speed, size). The photo can be forwarded to the photo processing module 25, which can generate a vehicle identifier, condition and confidence indicator for each vehicle.

This information can be added to the data register. (This process may appear after transmission to a central facility). The data log and photo file can be sent to a central facility. The image may be stored in an image database, and referenced if (a) additional processing is required to identify the vehicle, or (b) someone wants to verify the transaction. If the level of trust is sufficient, the data register can be submitted to the invoicing device, which can link it to the account and store it in the accounting database for subsequent invoicing. If no account exists, the vehicle identifier is sent to the appropriate state registration authority or third party service provider to determine the owner and establish the account. This process may be delayed until a sufficient number of transactions are collected for the vehicle to justify the invoice. If the level of trust is not sufficient, additional processing may be performed as described elsewhere.

The techniques described above describe the data flow based on a single end-to-end transaction, then looping to the beginning. In another embodiment, some of the functions described may be event-driven or scheduled, and may operate independently of each other. For example, there may not be a flow of control of the final processes to take a picture of the vehicle. The process of taking a picture of a vehicle can be initiated by an event, including the presence of the vehicle at the toll site.

In another embodiment, the system may be used for traffic monitoring and accident management. For example, if a decrease in average vehicle speed is detected, the computer may send a message to the highway control facility alerting controllers about the possibility of an accident. Authorized inspectors may connect to the on-site toll equipment to review camera images and determine if a response is required.

The operation of the toll management system 10 is explained with reference to FIG. 2-5.

FIG. 2 is a flowchart of an embodiment of an electronic road toll management system, especially process 100 for managing tagged vehicle identifiers 20 provided by external systems 34. To illustrate, in one example, law enforcement services 36 generate a list of tagged vehicle identifiers (e.g. license plate numbers) of drivers sought by the services and that the services 36 want to be notified when such vehicles have been identified using the tolls 28.

Computer 12 obtains (block 102) marked vehicle identifiers from a side such as law enforcement services 36. In one embodiment, these vehicle identifiers can be stored in the vehicle identifier database 20 for further processing.

Database 20 can be updated by new services as well as additional real-time information and / or batches. Law enforcement services accessed by a computer operate in many jurisdictions such as cities, cities, states, regions, countries, or other geographical purposes. As a result, computer 12 can process vehicle information across many jurisdictions and nationally.

Computer 12 (block 104) takes a picture of the vehicle triggered by a transaction event based on the interaction between the vehicle 30 and the object 28. For example, the photo acquisition module 24 can be used to obtain one or more pictures of the vehicle when traveling by an object such as a road toll. These photos can be stored in the image database 14 for further processing by the image processing module 25. Compression techniques can be applied to photos taken to help reduce the size of the database 14.

Computer 12 determines (block 106) the vehicle identifier based on the photo taken. For example, as discussed previously, the photo processing module 25 may apply photo analysis techniques to raw photos in an image database 14. These analytical techniques may extract a registration number from one or more vehicle license plate images. The extracted vehicle identifiers can be stored in the vehicle identifier database 18 for further processing.

Computer 12 compares (block 108) the captured vehicle identifier with the designated vehicle identifier. For example, computer 12 may compare the captured license plate number from the vehicle identifier database 18 with the registration number from the database of vehicle identifiers 20. As discussed above, automatic as well as manual techniques may be used to check the fit.

If the computer 12 detects a match (block 110) between registration numbers, then it checks (block 112) how the page associated with the tagged vehicle identifiers wants to be notified. This information may be stored in a vehicle identifier database 20 or other storage mechanism. On the other hand, if there is no match, the computer 12 resumes performing process 100 starting at block 102.

If the page indicates that it wants to be notified immediately (block 114), then the computer notifies (block 118) the person when a match occurs. In this example, a computer can notify law enforcement about a match in essentially real-time using wireless communication techniques or via a computer network.

On the other hand, if a person does not want to be notified immediately (block 114), then the computer 12 stores (block 116) a match for later notification after meeting certain criteria. In one embodiment, specific criteria may include gathering a specified number of matches and then sending messages to match the law enforcement authorities.

When the page has been notified (block 116) about the match or the match has been stored for later notification (block 116), the computer 12 resumes the process 100 starting from block 102.

FIG. 3 is a flowchart of an embodiment of the electronic road toll management system 10, especially process 200 for managing payment from a person associated with a vehicle that has interacted with an object. To illustrate, in one example, it was assumed that the toll authority decides to adopt disclosed techniques for managing payment processing, including billing and the collection of road tolls on cars using its toll road

Computer 12 takes (block 202) a photo of the vehicle triggered by a transaction event based on the interaction between the vehicle and the object. This function is similar to the process discussed above with respect to block 104 of FIG. 2. For example, the image acquisition module 24 may be used to obtain one or more images of a vehicle 30 when it crosses a road with tolls 28. These photos can be stored in the image database 14 for further processing by the image processing module 25.

Computer 12 determines (block 204) the vehicle identifier based on the photo taken. This function is also similar to the process discussed above with respect to block 106 of FIG. 2. For example, the photo processing module 25 may be used to extract a registration number from one or more vehicle license plate photos. These vehicle identifiers can be stored in the vehicle identifier database 18 for further processing.

Computer 12 determines (block 206) the person associated with the vehicle identifier by searching the registers' databases. For example, computer 12 may use the vehicle identifier from the vehicle identifier database 18 to search the vehicle registration authority database 40 to determine the registered vehicle owner associated with the vehicle identifier. Computer 12 can access vehicle information from one or more vehicle registration databases across many jurisdictions such as cities, cities, states, regions, countries or other geographical locations. In one embodiment, the computer 12 may keep a copy of the registration information from multiple registration authorities for further processing. Alternatively, the computer 12 may have access to multiple registration authorities and obtain registration information based on the request. In any case, these techniques allow the computer 12 to process vehicle information across multiple jurisdictions, and thus to process vehicles nationally.

Computer 12 checks (block 208) whether to demand payment from a person associated with the vehicle identifier. The payment request may depend on the payment processing information associated with the registered owner. For example, bills can be sent to the registered owner periodically (e.g. monthly) when a certain amount is reached, or as otherwise agreed.

If the computer 12 determines that payment is required (block 210), then it requests (block 214) payment from a person associated with the vehicle identifier based on the transaction event. As discussed above, a payment request can be generated using traditional postal service techniques or electronic techniques such as electronic payment. The bill amount may depend on information from a transaction event such as the nature of the interaction between the vehicle and the object. For example, a transaction event may indicate that the vehicle has arrived at a specific distance defined as the distance between the start and end point on the toll road. Accordingly, the invoice amount requested from the registered owner can be based on the distance traveled.

On the other hand, if the computer 12 determines that payment is not required (block 210), then it forwards (block 212) the transaction event to another party to manage the payment request. For example, the toll authority may decide that the computer 12 may perform image processing functions, and the billing and toll collection should be performed by a third party such as external systems 34. In one embodiment, the computer 12 may connect to service providers 44 and financial systems 48 to perform all or part of the billing and payment management functions. When the transaction event has been forwarded to a third party, the computer 12 resumes the functions of the process 200 starting in block 202.

If the computer processes the charges, the computer 12 processes (block 216) the payment response from the person associated with the vehicle identifier. In one embodiment, the accounting database 16, in combination with the billing device 22 and the customer management module 26, can be used to perform the billing and collection functions. As discussed above, the payment processing module 26c may assist electronic or manual payment processing depending on the received transfer. For example, computer 12 may provide an account for performing electronic payment processing over a computer network such as the Internet. The computer can also receive traditional payment such as a check.

When the payment has been processed (block 216), the computer 12 resumes execution of the process 200 starting in block 202.

FIG. 4 is a flowchart of an embodiment of the electronic road toll management system 10, especially the process 300 for managing payment through a communication channel from a person associated with a vehicle that has interacted with an object. To illustrate, it should be assumed that the toll authority responsible for the toll road uses the disclosed techniques and that the registered owner wants to make payment for the use of the toll road in an efficient and automatic way.

Computer 12 provides (block 302) an account for a person associated with the vehicle identifier. In one embodiment, the computer 12 in connection with the account management module 26a can provide customers with a website for opening an account for making electronic payment via a computer network such as the Internet. The website may also allow the customer to access and update account information such as payment history, amount due, preferred payment method, or other information.

Computer 12 receives (block 304) a request via a communication channel from a person for viewing the transaction event. For example, account 26a's fee module can make this request by extracting event information about a customer's account transaction from an accounting database 16. The information extracted may include image data of a specific transaction regarding the customer's vehicle and toll booths.

Computer 12 sends (block 306) a transaction event to person 32 via a communication channel. Transaction event information may include vehicle photos and a vehicle identifier (namely a license plate). Such data can be encrypted to ensure secure transmission over the Internet. Standard communication protocols such as hypertext information marking language (HTML) can be used to transmit information over the Internet.

Computer 12 determines (block 308) whether the person agrees to make the payment. For example, when a customer receives information regarding a transaction event, the customer can view the information to determine whether to make a payment based on whether the vehicle shown in the pictures is the customer's vehicle.

If computer 12 determines (block 310) that a person agrees to pay, then it processes (block 314) payment from the person by deducting the amount from the account based on the transaction event. For example, if the photographic information indicates that the transaction event data is accurate, then the customer can authorize payment as by making an electronic payment transaction.

On the other hand, if computer 12 determines (block 310) that a person does not agree to pay, then computer 12 processes (block 312) the contentious claim regarding payment from the person. In one embodiment, the dispute management module 26b can conduct a contentious claim submitted by the client using on-line techniques. Module 26b may conduct specific transactions regarding the customer's account, including the involvement of a third party to resolve the dispute.

When the payment has been processed (block 314) or the dispute has been resolved (block 312), the computer 12 resumes the execution of the process 300 starting in block 304.

F! G. 5 is a flowchart of an embodiment of an electronic road toll management system, especially process 400 for reconciling postal addresses from various sources. To illustrate, it is assumed that the toll authority used the disclosed techniques for processing the toll related to the use of the toll object. Because the disclosed techniques involve processing the toll after some time since the vehicle traveled through the toll authority, these techniques help ensure that payment is sent to the correct address of the registered vehicle owner.

Computer 12 determines (block 402) that the request for payment has been sent to the person associated with the vehicle identifier. As explained above, for example, payment requests may be generated periodically or based on a threshold amount.

Computer 12 gains access (block 404) to the vehicle registration authority for the postal address of the person associated with the vehicle identifier. For example, computer 12 may access one or more databases related to vehicle registration authorities 40 to obtain information such as the postal address of the registered owner of the vehicle.

Computer 12 gains access (block 406) to the postal authority for the postal address of the person associated with the vehicle identifier. For example, computer 12 may access one or more databases associated with postal authorities 38 to obtain information such as the postal address of the registered vehicle owner.

Computer 12 compares (block 408) the postal address from the vehicle registration authority to the postal address from the postal authority. For example, a computer compares email addresses from two authorities to determine if there is a discrepancy between information from databases.

If the computer 12 determines (block 410) that the addresses match, then it requests (block 414) payment from a person associated with the vehicle identifier using the postal address obtained from the postal authority. For example, the computer 12 may use the techniques described above to carry out payment processing, including billing and charging a fee from a registered owner.

On the other hand, if the computer 12 determines (block 410) that the addresses do not match, then it updates (block 412) the vehicle registration authority with a postal address from the postal authority.

For example, computer 12 may update the databases related to vehicle registration authorities 40 with the correct postal address obtained from postal authorities 38. Such techniques may help reduce the likelihood of sending the invoice to the wrong postal address resulting in a shortened time of payment transfer.

When the vehicle registration authority has been upgraded (block 412) or payment has been requested (block 414), the computer 12 resumes the process 400 starting in block 402 as explained above.

FIG. 6 is a block diagram of an embodiment of the electronic toll management system 600, which provides vehicle identification by extracting multiple vehicle identifiers for each vehicle that interacts with the toll object. The toll management system 600 includes a toll management computer 612. The toll management computer includes the image database 614, the accounting database 616, the vehicle identification database 618, the database of tagged vehicle identifiers 620, the billing device 622, the photo capture module 624, the photo processing module 625, and the customer management module 626. The toll management computer 612 connects or is integrated with the toll object 628, which interacts with the vehicle 630 and the person associated with the vehicle 632. The toll management computer 612 also connects to external systems 634.

Examples of each element included in the toll management system 600 of FIG. 6 are widely described with reference to FIG. 1. In particular, the toll management computer 612, image database 614, accounting database 616, vehicle identification database 618, database of designated vehicle identifiers 620, bill issuing device 622, photo acquisition module 624, photo processing module 625, management module customer 626, and toll object 628 usually have features comparable to and illustrate one possible embodiment, respectively, toll management computer 12 image database 14, accounting database 16, vehicle identification database 18, tagged vehicle identifier database 20, billing device 22, photo acquisition module 24, photo processing module 25, customer management module 26, and facility toll 28 from FIG. 1. Similarly, vehicle 630, person associated with vehicle 632, and external systems 634 typically have features comparable to vehicle 30, person connected to vehicle 32, and external systems 34 of FIG. 1.

The 618 vehicle identification database includes the extracted 6181 identifier database, the 6182 vehicle register database, and the 6183 read error database. The functions of the 6181-6183 database are described in more detail below.

System 600 is similar to system 10 and is configured to provide, for example, smaller amounts of vehicle identification errors by identifying each vehicle by using multiple vehicle identifiers. Two such identifiers are indicated as 631A and 631B. The vehicle identifier is preferably an identifier that uniquely or substantially uniquely identifies the vehicle, but may be an identifier that assists in the identification process by distinguishing the vehicle from other vehicles without necessarily having a unique identification of the vehicle. The identifiers 631A and 631B may be parts of the vehicle 630, as suggested by FIG. 6, but not necessarily. For example, identifiers 631A and / or 631B may be generated by the image processing module 625 based on the features of the vehicle 630.

As previously described, one example of a vehicle identifier is information from the vehicle's license plate, such as the license plate number and status. The photo processing module 625 can determine vehicle license plate information from the license plate photo using OCR, template matching, and other analytical techniques. The number plate can contain any character, but is usually limited to alphanumeric characters. The information on the license plate can usually be used to uniquely identify the vehicle.

Another example of a vehicle identifier is a vehicle detection tag as described in US Patent No. 6,747,687. The vehicle detection tag, referred to here as the vehicle's fingerprint, is a distilled set of data artifacts that represent the visual signature of the vehicle. The photo processing module 625 can generate vehicle fingerprints by processing the photo of the vehicle. However, to save processing time and storage needs, the generated fingerprint of the vehicle usually does not include normal "photographic" information that a person could recognize. Therefore, it is not usually possible to process the fingerprint of the vehicle to obtain the original photo of the vehicle. However, the fingerprints of some vehicles may include normal photographic information. A vehicle's fingerprint can usually be used to uniquely identify a vehicle.

In one embodiment, the camera in the photo acquisition module 624 takes a "still" photo of the back of each vehicle that passes through the toll object 628. For each vehicle, the photo processing module 625 recognizes visual paths that are unique to the vehicle and reduces them into an imprint vehicle finger. Because the license plate is a very unique feature, the 625 photo processing module usually maximizes the use of the license plate in creating the fingerprint of a vehicle. Notably, the vehicle's fingerprint also includes other vehicle parts in addition to the license plate and, therefore, vehicle identification by matching vehicle fingerprints is generally considered more accurate than identifying a vehicle by matching registration plate information. The vehicle fingerprint may include, for example, vehicle parts around the license plate and / or bumper parts and wheelbase.

Another example of a vehicle identifier is a vehicle signature generated using a laser scan (hereinafter referred to as a laser signature). The information from the laser signature that can be captured using a layered scan can include one or more electronic vehicle profiles from above, including the length, width, and height of the vehicle, the number of vehicle axles, and the 3D image of the vehicle. In one embodiment, the photo acquisition module 624 has two lasers per route lane, one is mounted above the road lane, and the other is attached along the road lane. A laser attached over a road lane typically scans the vehicle to capture the vehicle profile from above, and a laser attached along or over the road lane typically scans the vehicle to capture the number of vehicle axles. Together, two lasers can also generate a 3D image of the vehicle. The laser signature can be used to uniquely identify certain vehicles. For example, vehicles that have been modified to have a distinctive shape can be uniquely identified by a laser signature.

Another example of a vehicle identifier is a vehicle signature generated using a magnetic scan (hereinafter referred to as induction signature). The vehicle induction signature is a parameter that reflects the distribution of the metal in the vehicle and, therefore, can be used to classify the vehicle and, under certain circumstances, to uniquely identify the vehicle (e.g. if the metal distribution in a particular vehicle is unique to that vehicle due to unique modifications to that vehicle). The induction signature may include information that can be used to determine one or more number of axles (and probably the number of tires) of the vehicle, type of motor used in the vehicle, type or class of vehicle. In one embodiment, the photo acquisition module 624 includes a pair of vehicle detection loops, an axle detection loop, and a camera trigger loop in each lane of road.

When one or more vehicle identifiers are extracted by the image processing module 625, the image processing module 625 stores the extracted vehicle identifiers in a database of extracted vehicle identifiers 6181. Ideally, the computer 612 will then be able to uniquely identify the owner of the vehicle by selecting a vehicle identifier that uniquely identifies the vehicle (eg. license plate information or vehicle fingerprint) and searching one or more internal or external vehicle register databases to find a register containing a matching vehicle identifier. Unfortunately, the extraction of the vehicle identifier is an imperfect process. The extracted vehicle identifier may not correspond to the real vehicle identifier, and therefore may not uniquely identify the vehicle. An incorrectly or partially extracted vehicle identifier may not match the identifier of any vehicle, may match the identifier of the wrong vehicle, or may match the identifiers of more than one vehicle. To increase identification accuracy, system 612 computer 600 implements a multi-row identification process using two or more vehicle identifiers.

FIG. 7 is a flowchart of an example 700 double-row identification process that can be implemented to increase vehicle identification accuracy. Photographs and / or sensor data are captured for a vehicle that interacts with a toll object (hereinafter referred to as "target vehicle") and two vehicle identifiers are extracted from captured data (block 710). In one embodiment, only the image data is taken and the two vehicle identifiers are extracted from the number plate and fingerprint of the vehicle. In another embodiment, the image data and induction sensor data are taken, and the extracted vehicle identifiers are the vehicle's fingerprint and induction signature.

One of the two extracted vehicle identifiers is designated as the first vehicle identifier and used to identify a set of one or more matching candidate vehicles (block 720). Typically, the vehicle identifier considered to be the least able to accurately and / or uniquely identify the target vehicle is designated as the first vehicle identifier. For example, if the two extracted vehicle identifiers were the number plate and fingerprint of the vehicle, the number plate would be marked as the first vehicle identifier due to the lower expected accuracy of vehicle identification by matching the number plate compared to matching the fingerprint. One or more matching candidate vehicles may be determined, for example, by accessing a vehicle register database and fingerprint registers that contain vehicle identifiers that match or nearly match the first vehicle identifier.

When a set of one or more matching candidate vehicles is specified, the target vehicle is identified from the set based on the second vehicle identifier (block 730). For example, if 12 candidate vehicles were identified as matching a partially extracted license plate number, the target vehicle is identified by accessing vehicle fingerprints for each of the 12 candidate vehicles and determining which of 12 vehicle's fingerprints matches the vehicle's fingerprint. If no match is found within the specified confidence threshold, manual vehicle identification may be used. In another embodiment, one or more larger sets (e.g., super sets) of matching candidate vehicles are determined sequentially or simultaneously by change (e.g. loosening the matching criteria and additional attempts are made to identify the target vehicle from each of the one or more larger sets before resorting to manual identification.

In certain embodiments, the toll management system may be intentionally designed to identify a larger set of matching candidate vehicles during operation 720 to, for example, ensure that the expected lower accuracy of vehicle identification by the first identifier does not mistakenly exclude the target vehicle from the set of matching candidate vehicles. For example, if the first vehicle identifier is the number of the registration plate, the algorithm for reading the registration plate can be intentionally modified, for example, in two ways: (1) the criteria for matching the license plate reading algorithm can be loosened to enable the algorithm to generate a larger set of matching candidate vehicles, and (2) the registration table read algorithm can be "re-tuned" by lowering the read confidence threshold used to determine if a read result is included in the set matching candidates. For example, the license plate reading algorithm may be loosened to only require a matching vehicle to match a subset or fewer characters in the number of the registration plate extracted for the target vehicle. Additionally or alternatively, the reading certainty threshold may be lowered to allow previously suspected incorrect readings (namely, partial or low certainty readings) to include candidates in a set of matching vehicles.

The 700 double-row identification process ensures greater accuracy of identification over the single-row / single identifier identification system. By requiring that two vehicle identifiers be successfully matched for successful vehicle identification. Furthermore, the process 700 can provide a higher identification speed by limiting the matching of the second vehicle identifier to only those vehicle vehicles having registers that successfully match the first vehicle identifier. This can provide increased speed if, for example, the extracted second vehicle identifier consumes time to match with other such identifiers or if there is a large number of such other identifiers (namely, millions of identifiers for millions of vehicles in the vehicle database).

In another embodiment, two or more second identifiers are used to identify the target vehicle from among a set of matching candidate vehicles. Each of the second identifiers must match the same vehicle of the candidate at a certain level of trust for successful vehicle identification. Alternatively, the degree of matching of each of the two or more second identifiers can be weighted and a combined result of equivalent matching can be generated. If the combined result of the equivalent match is above a certain threshold, the identification is considered successful.

In one embodiment, each second vehicle identifier is assigned a matching trust level number, which is in the range of 1 to 10, where 1 corresponds to a missing match and 10 corresponds to an exact match. Each vehicle identifier also has a weight value of 1 to 10 assigned to it with higher weight values ​​assigned to vehicle identifiers considered to be more accurate in unique vehicle identification. If, for example, the second vehicle identifiers are the laser signature and license plate information, a weight of 6 can be allocated to the laser signature and a larger weight 9 can be allocated to the license plate information. If the combined result of the equivalent match 100 is necessary to consider the identification successful, and the license plate information matches the confidence level of 7 and the laser signature also matches the confidence level of 7, the combined result of the equivalent match will be 7 * 6 + 7 * 9 = 105 and identification will be considered successful.

In another embodiment, two or more first vehicle identifiers are used to identify vehicles in the set of matching candidate vehicles. Each of the first vehicle identifiers for a possible candidate vehicle must match the target vehicle with a certain level of confidence so that the possible candidate vehicle is in the set of matching candidate vehicles. Alternatively, the degree of matching of each of the two or more first identifiers can be weighed and a combined result of equivalent matching can be generated. If the combined result of the equivalent match is above the specified threshold, the possible candidate vehicle is in the set of matching candidate vehicles.

In another embodiment, the second identifier is not used to uniquely identify the target vehicle among the vehicles in the set of matching candidate vehicles. Rather, the second identifier is used to generate a new and smaller set of matching candidate vehicles as a subset of the set identified using the first identifier, and then a third identifier is used to uniquely identify the target vehicle from this set of matching candidate vehicles. In another embodiment, multiple vehicle identifiers are used to further reduce the set of matching candidate vehicles and the target vehicle is uniquely identified from the successively reduced subset by using one or more final vehicle identifiers. In yet another embodiment, each of the plurality of vehicle identifiers is used to generate its own set of matching candidate vehicles by matching or close matching techniques, and the reduced set is a cross between all specified sets. In yet another embodiment, the reduced assembly is determined using a combination of the techniques described above.

FIG. 8 is a flowchart of an exemplary double-row identification process 800 that can be implemented to increase accuracy and / or automate vehicle identification. The 800 process is an implementation of the 700 process, where the first identifier is the license plate number and the second identifier is the vehicle's fingerprint. In particular, process 800 includes operations 810-830, and associated sub-operations that correspond to and illustrate one possible implementation of operations 710-730, respectively. For convenience, reference is made to the specific elements described with reference to FIG. 6 during process 800. However, similar methodologies can be used in other embodiments where different elements are used to determine the structure of the system, or where functionality is differently distributed between the elements shown in FIG. 6.

The image acquisition module 624 captures image data for the target vehicle based on the interaction between the target vehicle and the toll object 628 (block 812). In another embodiment, the image acquisition module 624 additionally or alternatively captures sensor data including, for example, laser scanning data and / or a loop sensor. The image processing module 625 obtains license plate data, including, for example, the entire or partial number of the license plate and the state for the target vehicle from captured image data (block 814). Optionally, the photo processing module 625 may also determine the vehicle's fingerprint for the target vehicle from the photo data. In another embodiment, the image processing module 625 may determine other vehicle signature data, such as, for example, laser and / or induction signature data, from image data and / or sensor data.

Computer 612 stores captured image data in an image database 614 and stores extracted license plate data in a database of extracted identifiers 6181. When applicable, the toll management computer 612 also stores the vehicle's fingerprint and other reference data, such as, for example, induction signature and / or laser signature in the extracted identifier database 6181.

Computer 612 accesses the vehicle identification register set from the vehicle register database 6182 (block 822). Each of the vehicle identification registers combines the vehicle owner / driver with vehicle identifier data. Computer 612 compares the extracted license plate data with the license plate data in the set of vehicle identification registers (block 824) and identifies the set of candidate vehicles from vehicles having registers in the set of registers (block 826). The comparison can be made using matching or close matching techniques.

Computer 612 accesses extracted vehicle fingerprint data for the target vehicle (832). If the vehicle's fingerprint has not yet been determined / extracted from the captured image data, the computer 612 calculates the vehicle's fingerprint and stores the vehicle's fingerprint in a database of retrieved vehicle identifiers 6181.

Computer 612 accesses the fingerprint data for the vehicle in the candidate vehicle set by accessing the corresponding vehicle identification register (block 834) and compares the vehicle fingerprint data for the target vehicle with the fingerprint data for the candidate vehicle (block 836). Computer 612 identifies the candidate vehicle as a target vehicle based on the results of vehicle fingerprint data comparison (block 838). If the vehicle's fingerprint data matches a certain level of trust, the candidate vehicle is considered the target vehicle and the candidate vehicle owner / driver is considered the target vehicle owner / driver.

FIG. 9A-9C are a flow chart of an example 900 double-row identification process that can be implemented to increase vehicle identification accuracy while minimizing the need for manual vehicle identification. Process 900 is another embodiment of process 700, where the first identifier is the license plate number and the second identifier is the vehicle's fingerprint. In particular, process 900 includes operations 910-930, and associated sub-operations that correspond to and illustrate one possible implementation of operations 710-730, respectively. For convenience, reference is made to the specific elements described with reference to FIG. 6 during process 800. However, similar methodologies can be used in other embodiments where different elements are used to determine the structure of the system, or where functionality is differently distributed between the elements shown in FIG. 6.

The photo capture module 624 captures photo and sensor for target vehicle (block

911). Side road sensors, for example, run cameras that take pictures of the front and rear of the target's vehicle. Other sensors may capture additional data used for vehicle classification / identification. For example, a laser scan can be used to specify laser signature data including height, width, length, number of axles, and vehicle dimensional profile. Sensors can also be used to determine transaction related data between the target vehicle and the toll object 628 such as, for example, vehicle weight, vehicle speed, and vehicle related transponder data.

The image processing module 625 performs a license plate reading on captured image data, creates a fingerprint of the vehicle from the captured image data, and optionally specifies other vehicle signature / classification data from the captured sensor data (block 912). For example, the photo processing module 625 may use the automatic license plate reading algorithm to read one or more photos taken. The license plate reading algorithm can read the pictures taken, for example, in a priority order based on the visibility of the plate and its location in the picture. The license plate reading results may include one or more license plate numbers, license plate status, license plate style, read confidence score, plate location in the photo, and plate size. The image processing module 625 may also use a visual signature extraction algorithm to generate a vehicle fingerprint for the target vehicle. The algorithm for extracting the visual signature may be similar to that developed by JAI-PULNiX Inc. from San Jose, California and described in US Patent No. 6,747,687. Computer 612 stores the photos taken in the image database 614 and stores the results of reading license plates, vehicle fingerprint, and other vehicle reference / classification data in the database of extracted 6181 vehicle identifiers.

The photo processing module 625 determines if the photos taken have provided any partial or full reading results for the number plate and the vehicle's condition (block 913). If partial or full reading results have not been provided by the images taken, process 900 proceeds to operation 941 of the manual identification process 940.

If partial or full reading results for the registration number and condition of the vehicle of the target were provided by the pictures taken, computer 612 searches the vehicle register database 6182 and the reading error database 6183 for the exact (or partial or full) number of the registration plate (as read by the plate reader registration) (block 921)

The 6182 vehicle register database includes registers for all previously recognized vehicles and potentially includes registers for vehicles that can be expected to be seen. The 6182 vehicle register database is usually populated by a registration process during which the driver / owner of the vehicle saves the vehicle for automatic toll payment management. The driver / owner of the vehicle can register the vehicle for automatic management of the payment of road tolls by driving the vehicle through a special registration lane at the road toll facility 628 and providing his customer service representative at the object 628 his or her identity and other contact details. The image acquisition module 624 and the image processing module 625 capture the license plate number, fingerprint, and other identification / classification data (e.g. vehicle dimensions) of the user's vehicle when the vehicle exceeds object 628. The vehicle and owner identification data are stored in a new vehicle identification register associated with the newly registered vehicle and owner / driver.

Alternatively, the driver / owner can register a vehicle to automatically manage the payment of road tolls simply by passing through facility 628 without stopping. The computer 612 captures image data and sensor data for the vehicle and attempts to identify the driver / owner by reading the license plate photo and searching for the reading results in an external system 634 database (e.g. vehicle registration authorities). If the owner / driver has been identified, the computer 612 bills the owner / driver. Once the accounting relationship has been successfully established, the computer 612 officially registers the vehicle, generates as necessary vehicle fingerprint data and other reference / classification data from captured image data and sensor, and stores them in the vehicle identification register associated with the identified owner / driver.

In another embodiment, the computer 612 is configured to obtain greater accuracy in identifying the unregistered driver / owner by searching for the license plate reading results in the vehicle registration authority database (or other external system) and querying the corresponding vehicle identification number (VIN) in the registration authority vehicles (or other external system). The 612 computer uses VIN to determine the make, model and year of the vehicle. The make, model and year of the vehicle can be used to determine the length, width, and height of the vehicle. The computer 612 can then determine the successful connection of the target vehicle with the vehicle registered in the vehicle registration body not only by comparing the license plate data, but also by comparing the dimensions of the vehicle (as captured, for example, in the laser signature and / or induction signature). Locally, the computer 612 will consider the match successful if the target vehicle license plate reading results match the license plate data for the vehicle registered by the vehicle registration authority within a certain threshold and the dimensions of both vehicles fit within a given tolerance.

The make, model and year of the vehicle may be used, for example, to determine the length, width and height of the vehicle either by obtaining this information from a public database or from a third party database or, in addition or alternatively, by accessing the vehicle register database 6182 to obtain data on the length, width, and height from one or more vehicle identification registers corresponding to vehicles of the same make, model and year as the target vehicle. Due to the fact that the dimensions of the vehicle may change if the vehicle has been modified, the obtained length, width, and height from vehicle identification registers may differ for the vehicle. Therefore, the computer 612 may need to determine statistically valid dimensions for comparison by, for example, adopting the average or median dimensions of length, width, and height.

In one embodiment, the computer 612 identifies the vehicle in part by using an electronic signature, which includes a laser signature and / or an induction (namely, magnetic) signature. When a vehicle makes a transaction with a toll system, the electronic signature for the vehicle is captured. The photo and dimensions of the vehicle created by the laser (namely, the laser signature) and / or magnetic scan (namely, the induction signature) are compared against known dimensions and photos of vehicles based on the vehicle identification number (VIN), which were, for example, previously captured by toll system or through an external system. By comparing the photo and dimensions of the electronic signature to known vehicle dimensions based on VIN, the search for vehicle matching and associated VIN can be narrowed down. If, for example, the LPR for a vehicle has a low level of trust but the vehicle's electronic signature has been captured, the toll system can access the database as described above, known vehicle dimensions and photos and associated VINs, and refer to the dimensions and photos of the electronic signature relative to the database data to identify the matching VIN of the vehicle or to identify a potential match of candidate / VIN vehicles. The 6183 read error database connects previous erroneous readings to valid vehicle identification registers. For example, when automatic vehicle identification fails, but manual vehicle identification is successful, the vehicle identification data captured (e.g. the result of the license plate reading) that led to the "error" (namely, identification failures) by the automated system are stored in the error register in the 6183 read error database, which is connected to the vehicle identification register that has been manually identified for the vehicle. In this way, when the same vehicle identification data is captured again at a later time, the computer 612 can successfully identify the vehicle automatically by accessing the error register in the 6183 reading error database that identifies the correct vehicle identification register without requiring another manual vehicle identification .

An error log can also be generated and stored in the 6183 read error database when the automatic identification of the vehicle is successful based on a close match of the result of the incorrect reading of the license plate. For example, if the number plate "ABC123" is read as "ABC128" and the set of matching identified candidates is "ABC128", "ABC123", "ABG128" and "ABC128", which in turn gives the correct matching "ABC123" can be created an error register that automatically connects the result of the "ABC128" license plate reading to the vehicle having the "ABC123" license plate number.

Computer 612 determines whether any vehicle identification registers correspond to the results of the license plate reading for the target vehicle (block 922). If no vehicle identification registers match the read results, the computer 612 performs an extended search (block 923).

Computer 612 performs an extended search by changing or loosening the criteria for a successful match or re-tuning the license plate reading algorithm. For example, computer 612 may perform an advanced search by one or more of the following: (1) comparing a subset of the license plate reading result with the license plate number characters stored in the 6182 vehicle register database (e.g. the last two characters of the license plate number may be omitted so that if the license plate number is "ABC123", all vehicles having license plate numbers "ABC1 **" are considered to be matching candidates, where "*" is a variable): (2) subset comparison the result of reading the license plate in reverse order with the registration plate number mark stored in the reverse vehicle registration database 6182 (e.g. the last two characters of the number plate in reverse order may be omitted so that if the number plate is "ABC123" which in reverse order is "321CBA", all vehicles having number plates in reverse order "321C **" are considered matching candidates, where "*" is a variable; and (3) other close matching techniques involving the comparison of modified versions of the number plate read results and number plates stored in the 6182 vehicle register database in which some of one or both are replaced and / or removed to reduce the impact of unread characters. For example, if the OCR algorithm does not indicate a level of trust above a certain threshold as a result of reading a mark on the license plate, that mark may be ignored. Additionally or alternatively, if the OCR algorithm indicates that the sign on the license plate can be one of two possible different characters, both alternative characters can be used in the extended search.

Computer 612 determines whether any vehicle identification registers correspond to the read results for the target vehicle after performing the expanded search (block 924). If no vehicle identification registers are found, process 900 proceeds to operation 941 of manual identification process 940 (block 924).

With reference to FIG. 9B, if the search or expanded search results in the identification of one or more vehicle identification registers, the computer 612 obtains the vehicle's fingerprint and optionally other vehicle reference / classification data from the identification records of the identified vehicles (block 931). Computer 612 compares the obtained vehicle fingerprint and optional other vehicle signature / classification data for each matching candidate vehicle with the corresponding vehicle related data to identify one or more possible matches (block 932). The comparison of a vehicle's fingerprint can be made using a comparison algorithm identical or similar to the algorithm developed by JAI-PUI_NiX Inc. from San Jose, California and described in US Patent No. 6,747,687.

A possible match may be defined, for example, as a fingerprint of the vehicle with a confidence score greater than or equal to the specified threshold and all or some other classification / reference data within the tolerances specified for each type of data. For example, if the fingerprint matching algorithm produces a result of the order of 1 to 1000, where 1 is no match and 1000 is a perfect match, then a result greater than or equal to 900 may be required for a successful match. In addition, if other classification / reference data include the height, width, and length of the target vehicle, then it may be required that the height, width, and length of the candidate vehicle be plus or minus four inches of the extracted height, width, and length of the target vehicle for a successful fit . One or more vehicle identification registers may be considered to correspond to vehicles that match the target vehicle as closely as possible.

Computer 612 determines whether a possible match is sufficient to automatically identify the vehicle without human intervention by determining the combined result of equivalent matching for each possible match and comparing the result with the specified automatic trust threshold (block 933). Computer 612 may, for example, determine the combined result of an equivalent match for each possible match in a manner similar to that previously described for process 700. Specifically, computer 612 may allocate the number of matching confidence levels to the matching fingerprint and, optionally, matching classification / signature data, allocate weight to each type of data, and calculate the combined result of the equivalent match by combining the weighted numbers of the match confidence level. If the combined result of the equivalent match exceeds the specified automatic trust threshold, the computer 612 will consider the target vehicle to be successfully identified and the process 900 will proceed to operation 937 to record the transaction event between the identified vehicle and the object 628. If more than one possible match exceeds the automatic trust level, the automatic identification process may be faulty, and the 900 process may optionally proceed (not shown) to operation 941 of the 940 manual identification process.

If no possible match is considered sufficient to automatically identify the vehicle without human intervention, the computer 612 will determine if one or more possible matches meet the lower probable match threshold (block 934). The computer 612 may, for example, determine that a possible match meets the probable match threshold if the result of the combined equivalence match of the probable match is higher than the probable match threshold but lower than the automatic trust threshold.

If at least one possible match meets the likely match threshold, the computer 612 allows the operator to perform a visual check of the match. Visual matching check is the process by which the computer 612 presents the operator with one or more photos of the target vehicle along with one or more reference photos related to the vehicle or vehicles that are likely to match the target vehicle. The operator quickly confirms or rejects any likely match with a simple yes or no by, for example, selecting the appropriate buttons on the user interface (block 936). The operator can optionally also provide a detailed explanation to justify his or her response.

If the match exceeds the automatic trust threshold or is visually confirmed by the operator by visually checking the match, the computer 612 creates an event log (namely, the interaction log between the target vehicle and object 628 that has been positively identified) as, for example, a billable or non-profit transaction (block 937). If the match has been confirmed by a visual match check, the computer 612 may optionally update the error reading database 6183 to include extracted vehicle identification data and a link linking the extracted vehicle identification data to the appropriate vehicle identification register (block 938).

Also referring to FIG. 9C, the computer 612 is configured to allow the operator to manually identify the target vehicle (block 941) in the following situations: (1) taken pictures of the target vehicle do not provide full or partial readings for the number plate and the state of the target vehicle (block 913); (2) no vehicle identification registers matching the results of the license plate readout for the target vehicle are searched after performing an extended search (block 924); (3) one or more possible matches are found, but the level of trust in one or more possible matches, which is reflected in the results of the combined equivalent equivalence match, is below both the automatic trust threshold and the likely match threshold (block 934); and (4) one or more possible matches are found, but the human operator rejects one or more likely matches by visual matching check (block 936).

The human operator attempts to manually identify the vehicle by (1) reading the registration plate (s), and (2) observing the details of the vehicle captured by the photo capture module 624, and (3) comparing the number plate data and the vehicle details with data available from 6182 vehicle register databases, 6183 read error databases, and / or external databases of 634 systems. License plates read by a human operator can be confirmed by comparison with the results of the automatic reading of license plates and / or multiple entries by many human operators.

Manual identification can be considered successful if manually collected data weighed against identifiable criteria for a positive vehicle match exceeds the specified identification confidence threshold (block 942). This determination can be made by computer 612, the operator who provided the manual data, and / or more qualified operators.

In one embodiment, if the vehicle cannot be automatically positively identified, and no close matches are found, one or more images of the vehicle are displayed to the first human reviewer. The first human reviewer browses the photos and manually specifies the number of the license plate, which he considers corresponding to the vehicle based on the photos. Because of this manual review by the first human reviewer, it is also subject to error (e.g. error in perception or typography), the license plate read by the first human reviewer is compared to the LPR database to determine if the number plate specified by the first human reviewer exists. In addition, if there is a database register having fingerprint data corresponding to the number plate read, a fingerprint comparison can also be made. If the reading results of the first human reviewer do not match any known LPR or vehicle results, one or more vehicle photos may be displayed for the second human reviewer. A second human reviewer browses the photos and manually specifies the number of the license plate that he thinks corresponds to the vehicle based on the photos. If the reading result of the second human reviewer differs from the reading result of the first human reviewer, it may be necessary for the third human reviewer to read, who is usually a more qualified reviewer. In summary, the reading of the first human reviewer is effectively a stepping stone for attempting automatic matching again. If automatic matching still fails, many human reviewers need to match the license plate reading for the reading to be accurate.

If the vehicle is not successfully identified, computer 612 creates an event log as an unidentified or unassigned transaction (block 943). If the vehicle is successfully identified, computer 612 creates an event log as, for example, an accountable or non-profit transaction (block 937). If the vehicle has never been identified before, the 612 computer can create a new vehicle identification register for the vehicle and its owner / driver in the 6182 vehicle register database. The 612 computer can also update the 6183 reading error database to include vehicle identification data and a link linking extracted data vehicle identification with the appropriate vehicle identification register (block 938).

FIG. 10 is a block diagram of the electronic road toll management system 1000, which enables electronic management of the payment of road tolls by vehicles passing through the road toll facility without requiring direct communication between the road lane transaction system and the system's image system. The electronic toll management system 1000 is just one embodiment and various other embodiments are either described below or obvious to those of ordinary skill. The electronic toll management system 1000 includes a 1012 toll management computer. The 1012 toll management computer includes the 1014 image database, the 1016 accounting database, the 1018 vehicle identification database, the 1020 vehicle identifier database, the 1020 billing machine, the photo capture module and lane transaction data (ILDM) 1010, the image processing module 1025, and customer management module 1026. The 1012 toll management computer connects to or is integrated with the 1028 toll object that interacts with vehicle 1030 and a person associated with that vehicle, 1032. The 1012 toll management computer also connects to external 1034 systems.

Examples of each element within the toll management system 1000 of FIG. 10 are described extensively above with reference to FIG. 1. In particular, the 1012 toll management computer, 1014 image database, 1016 accounting database, 1018 vehicle identification database, 1020 vehicle identifier database, 1022 billing machine, 1025 photo processing module, 1026 customer management module, and toll object 1028 tolls usually have features comparable to and illustrate one possible embodiment of a toll management computer respectively 12, image database 14, accounting database 16, vehicle identification database 18, database of tagged vehicle identifiers 20, billing machine 22, photo processing module 25, customer management module 26, and toll object 28 of FIG. 1. Similarly, vehicle 1030, person associated with vehicle 1032, and external systems 1034 typically have features comparable to vehicle 30, person associated with vehicle 32, and external systems 34 of FIG. 1.

ILDM includes the lane 1020 transaction system, image acquisition module 1024, and video server 1030. The image acquisition module 1024 typically has features comparable to and illustrates one possible embodiment of the image acquisition module 24 of FIG. 1. The 1024 image acquisition module includes the 1024A vehicle image system (VIS) and the 1024B vehicle image acquisition computer (VIC).

The toll management system 1000 can be configured to automatically identify only vehicles without a transponder that are considered "infringers." The infringer is a vehicle that does not provide payment for the transaction with the toll object 1028 at the time of the transaction. For example, the infringer may be a vehicle without a transponder driving through the toll 1028 site without providing toll payment by, for example, stopping to pay in cash at the toll site or by having an active financial account that the toll site may have access to and which may be charged by the toll object. The 1000 toll management system is, however, still different from the conventional toll system in that the 1020 lane transaction system and 1024 image acquisition module do not need to communicate directly with each other to identify infringers. Rather, the video server 1030 is configured to match each violation transaction identified by the lane 1020 transaction system with the violation photo taken by the image acquisition module 1024 by applying the matching process described in detail below.

The 1020 lane transaction system is a system that includes one or more computers and sensors configured to capture transaction data for each 1030 vehicle that passes through a 1028 toll object. Transaction data includes all data relevant to the transaction between the 1030 vehicle and the toll object road 1028, such as, for example, identifier for the road lane used by the vehicle, type of transaction, transaction time (e.g. transaction timestamp), vehicle classification data (e.g. number of vehicle axles), transponder information, if any, vehicle, calculated fee, and an indication of whether the vehicle has committed an offense or not.

The 1020 lane transaction system is configured to periodically send a lane activity report or file to the 1030 video server. The lane activity report includes a chronologically sequential list of data entries or transaction entries. Each transaction entry includes transaction-related data for transactions between object 1028 and a single vehicle. In one embodiment, the lane transaction system 1020 sends the lane activity report to the video server once a day or multiple times a day as a file containing the same type of entries attached to an email.

FIG. 11 shows an excerpt from an exemplary report on lane 1100 activity generated by the 1020 lane transaction system. Extract 1100 includes a group of ten chronologically sequential lane transaction entries, each entry corresponds to a vehicle transaction with a toll object 1028. The first and last entries (namely, entries 1110 and 1130) in the group of transaction entries are "border transaction" entries. Borderline transactions and borderline transaction entries are further discussed below.

Entry 1110 is an example of an entry corresponding to a successful transaction (namely, no violation transaction). Entry 1110 includes various data fields that include transaction related data. Data fields include: (1) transaction data field 1110a, which includes the nature of the transaction or lane operation (e.g. an infringement transaction, a paid transaction, and an unpaid transaction that is not, however, considered an infringement transaction because, for example, the vehicle is a government vehicle; (2) location data field 1110b that identifies the place where the lane transaction took place (e.g. identification number corresponding to the specific toll location where the toll transaction took place); (3) data field with transaction date 1110c that identifies the date on which the transaction took place; (4) time data field 1110d, which identifies the time at which the transaction took place; (5) vehicle classification data field 110e, which identifies the vehicle class, number of axles, and / or vehicle dimensions; (6) data field for the due toll 1110f, which indicates the amount that the toll object charged for the transaction; (7) data field for payment of the 1110g fee, which indicates the amount paid by the vehicle for the transaction at the road toll facility; (8) data field with payment method 1110h, which indicates the method used by the vehicle to pay the fee (e.g. cash payment, credit card payment and transponder payment); (9) data field about the account issuer 1110i, which identifies the unit that issued the financial account from which the fee may be charged (e.g. bank account issuer such as "Bank of America", credit card issuer such as "Visa", transponder account issuer such as "Virginia" transponder issuing authority); and (10) field with account identifier 1110j, which identifies the financial account from which the fee can be charged (e.g. credit card number or transponder number). Entry 1120 is an example entry corresponding to a violation transaction.

With reference again to FIG. 10, VIS 1024A the image acquisition module 1024 is a system that includes both configured computers and sensors to capture image data and optional sensor data for each vehicle that passes through or transacts with the toll object 1028. VIS 1024A may include any and / or all sensors and imaging devices previously described in relation to image acquisition modules 24, 624. VIS 1024A is configured to transfer captured vehicle image data and sensor to the VIC 1024B. Photographs and vehicle sensor data typically include time stamps (time and date information) indicating when the data was captured by the VIS 1024A.

In one embodiment, VIS 1024A includes cameras, light sensors, and lasers. Light sensors constantly monitor ambient lighting and update cameras many times per second to ensure that cameras optimize image quality by regularly adjusting it whenever necessary for any change in ambient light. Lasers detect vehicles as they pass each lane of the road and trigger cameras when the vehicle leaves the lane. Each lane of the road can have one camera that takes one or more photos of the rear of the vehicle when the vehicle is passing.

The VIC 1024B is a computer system configured to receive vehicle image data and, optionally, sensor data from the VIS 1024A, to compress images and sensor data to minimize storage needs, and to store image data and sensors in image / sensor files having associated data. The data may include, for example, the unique identifier of the image / sensor file, a time stamp indicating when the image and sensor data were captured, and a location indicating where the image and sensor data were captured (e.g., lane identifier).

After receiving photo data and sensors for a passing vehicle and storing them in a photo / sensor file, the VIC 1024B can send a message to the 1030 video server informing it that the file is available for delivery. VIC 1024B can send the captured image / sensor file to the 1030 video server in response to a request received from the 1030 video server. After the 1030 video server indicates that it has securely stored the desired photo / sensor file, the VIC1024B can optionally delete the photo / sensor file from its data resources.

Fig. 12 shows an exemplary set of photo / sensor files 1200 containing image data received by video server 1030 from VIC 1024B. The image / sensor file set 1200 includes ten files, each of which is shown in Fig. 12 as a miniature of the stored image with associated unique file name. Each photo / sensor file of the set of photo / sensor files 1200 corresponds to a single transaction entry of the lane transaction of the 1100 lane activity report. For example, the image / sensor file 1210 corresponds to the border transaction 1110 entry, and the image / sensor file 1220 corresponds to the border transaction entry 1130.

VIC 1024B is also configured to send messages to and receive messages from the 1030 video server. In particular, VIC 1024B can send status messages to the 1030 video server that indicate the status of the VIC 1024B and / or various VIS 1024A components. The VIC 1024B can receive management messages from the 1030 video server, which allows administrators to interact with the 1030 video server to configure or otherwise control the operation of the VIC 1024B. The VIC 1024B may also receive clock synchronization messages from the 1030 video server that instruct the VIC 1024B to reset its internal clock.

As described in more detail below, the synchronization or matching of photo / sensor data with transaction entries is based in part on time entries associated with the photo / sensor data and transaction entries. For this reason, it is desirable to synchronize the internal clock VIC 1024B, which allocates time to given photos / sensors, with the internal clock of the 1020 lane transaction system, which allocates time to each transaction. By periodically resetting the VIC 1024B internal clock to coincide with a network clock setting (not shown) known to be synchronized with the internal clock of the 1020 lane transaction system, the 1030 video server can minimize clock shifts between image timestamps generated by the VIC 1024B and transaction timestamps generated by the 1020 lane transaction system.

The 1030 video server is usually a computer system that is configured to receive lane activity report from the 1020 lane transaction system and to receive photo / sensor files from the VIC 1024B image acquisition module 1024. In another embodiment, the 1030 video server is configured to receiving lane activity reports from more than one lane transaction system and / or receiving photo / sensor files from more than one VIC.

The 1030 video server is typically configured to process lane activity reports by parsing it and assigning unique transaction identifiers for each transaction entry in the report. After assigning transaction IDs, the video server 1030 typically synchronizes or matches the photo / sensor file with each transaction entry in the lane activity report.

FIG. 13 and 14 illustrate operations performed by, for example, video server 1030 to match transaction entries with photo / sensor files. In particular, FIG. 13 illustrates the process 1300 for selecting groups of transaction entries and the corresponding groups of photo / sensor files for each violation transaction entry, and FIG. 14 illustrates a process 1400 for identifying a photo / violation sensor file for each violation transaction entry.

With reference to FIG. 13, video server 1030 identifies transaction entries in the lane activity report that correspond to violation transactions ("violation transaction entries") (1310). The video server 1030 can identify violation transaction entries as transaction entries in the lane activity report that meet the specified set of criteria.

For example, a transaction entry in a lane activity report can be identified as an infringement transaction entry if it meets a set of validation criteria. It is noteworthy that many sets of different criteria can now be used to determine an infringement transaction entry.

After identifying one or more transaction entries based on a set of criteria (s), the video server 1030 can validate each entry of an identified infringement transaction by (1) reviewing the alleged infringement transaction entry for anomalies and (2) by examining the corresponding transaction entry anomalies transactions that occurred in a configurable time window before and / or after the alleged violation transaction (132d). If any anomalies are found, the alleged violation transaction entry cannot be validated (namely, it can be an error).

The video server 1030 may view all or a subset of the half-entry data of the alleged violation transaction to determine, for example, whether the lane of the 1020 lane transaction system may be unhealthy and, therefore, may consider non-violating vehicles as infringers. An unhealthy lane may, for example, generate conflicting transaction data such as, for example, detecting a different number of axles when entering a vehicle on a lane than detected when exiting the lane, or than as indicated by the transponder information. However, in another embodiment, the violation transaction entries may be identified simply as transaction entries having violation disposition data fields 1120n set to indicate violation. If such anomalies are found, the alleged violation transaction is probably an error, and the 1030 video server may reject the alleged violation transaction as invalid.

Video server 1030 can also investigate for anomalies transaction entries corresponding to transactions that appeared in the time window (e.g. 5 minutes) preceding the alleged violation transaction. For example, one of the pre-transaction entries may indicate that an early reading occurred within five minutes just before the alleged violation. Video server 1030 may reject the alleged violation transaction entry as invalid because early reading indicates that the transponder reading may have been incorrectly associated with the vehicle.

The video server 1030 can also investigate for anomalies transaction entries corresponding to transactions that occurred in the time window (e.g., 5 minutes following the alleged violation transaction. For example, one of the following transaction entries may indicate that the lane has reset to minimize error cascades.

After successful validation of one or more infringement transaction entries, the video server 1030 selects a group of chronologically sequential transaction entries for each validated infringement transaction entry (1330). The selected transaction entry group includes transaction entries that correspond to transactions that precede and follow the violation transaction, and therefore allows the transaction to be placed in its context. Through the matching process discussed later, the selected group of transaction entries can be used to achieve greater accuracy in identifying the photo / sensor file that matches or matches the validated transaction transaction entry.

The group of transaction entries, for a valid breach transaction entry, can be selected as all transaction entries starting from the first transaction entry corresponding to the "border transaction" (namely, the border transaction entry) that appeared before the validated transaction violation entry and ending with the first border transaction entry that appeared after the validated violation transaction entry. Therefore, the group of transaction entries selected for each validated infringement transaction entry typically includes a validated infringement transaction entry, two border transaction entries, and one or more other transaction entries, where two border transaction entries surround or close the validated transaction transaction entry and one or more. transaction entries in the group. FIG. 11 shows an example of a group of transaction entries 1100 that includes two border transaction entries 1110 and 1130 that close or surround the rest of transaction entries in a group, including violation transaction entries 1120.

A border transaction is a transaction that corresponds to a transaction entry and is easily matched or synchronized with the associated image / sensor file. A borderline transaction, for example, can be a transaction that follows a certain amount of time during which no transaction occurs (namely, "dead" time). For example, if no transaction appears between the vehicle and the 1028 toll object for 10 seconds, the transaction that appears shortly after 10 seconds is a border transaction because its transaction entry is easily matched with the corresponding image / sensor file. The border transaction is easily matched to the corresponding image / sensor file because they are easily identifiable as the first transaction entry and image / sensor file captured after a 10 second dead time. Similarly, a transaction that precedes 10 seconds 'dead' time is also a borderline transaction because its transaction entry and photo / sensor file are also similarly easily identifiable as the last transaction entry and photo / sensor file preceding 10 seconds 'dead' time. With reference to FIG. 11, entry 1110 may be, for example, a border transaction because it occurs after a 10 second "dead" time, and entry 1130 may be, for example, a border transaction because it is preceded by a 10 second "dead" time.

Other examples of borderline transactions include transactions that involve a visually unique vehicle or vehicles that have been positively identified. For example, a transaction involving a multi-axle vehicle (namely, a vehicle with 3 or more axles) such as a truck, may be a border transaction if most vehicles passing through the toll object are cars with only 2 axles. By searching for a truck photo among all the car photos, the photo / sensor file having the photo that matches the multi-axis transaction entry is easy to search. Similarly, a transaction involving a vehicle with a tranche nder may also be a border transaction if the transponder information captured in the transaction entry can be used to identify the vehicle's license plate number. If the vehicle's license plate number is successfully identified from the transponder data, the corresponding image and, therefore, the image / sensor file can then be positively identified by using LPR.

In particular, the video server 1030 may not indicate a transaction as a border transaction if it is preceded or followed by a violation transaction or other unusual type of transaction (e.g., lane reset or any transaction that does not meet the validation criteria). If two or more validated violation transaction entries appear in the time window specified by a single pair of border transaction entries, video server 1030 may use the same transaction group to match both validated violation transaction entries.

In some embodiments, the video server 1030 may limit the size of transaction entry groups by imposing settable limits on the number of transaction entries and / or the maximum time interval preceding and / or following the validated infringement transaction entry. For example, the number of transaction entries may be limited to twenty or one hundred transactions and / or the maximum time interval following the validated infringement transaction entry may be limited to one minute or five minutes. If only one border transaction appears in this adjustable limit or time interval, then only this border transaction is used in the matching process. If no transaction appears within this adjustable limit or time interval, then no border transactions are used in the matching process. If no borderline transactions are used in the matching process, then the matching process is performed manually by searching for time patterns or identification information in the images to perform confirmation related to the corresponding transactions (e.g. transponder information in transaction entries can be combined with the expected number plate numbers as shown in the pictures).

In one embodiment, the group of transaction entries for a validated violation transaction entry includes all entries corresponding to transactions immediately following the last 6-10 second pause preceding the violation transaction, and all entries corresponding to transactions that occurred up to one minute after the violation transaction. In this embodiment, only the starting transaction entry in the group is the border transaction entry (namely, the entry corresponding to the first transaction after a 6-10 second pause).

After identifying the group of transaction entries for each validated infringement transaction entry, the video server 1030 uses border transaction entries for each group of transaction entries to identify corresponding chronologically sequential groups of photo / sensor files (1340 and 1350). The group of photo / sensor files for a validated infringement transaction typically includes all photo / sensor files having time stamps between this photo / sensor file having a photo of the border transaction transaction preceding the violation transaction (namely, the photo corresponding to the border transaction transaction preceding the infringement transaction) and this photo / sensor file having a photo of the border transaction following the infringement transaction (namely, photo corresponding to the border transaction following the infringement transaction). Accordingly, a group of photo / sensor files for a validated infringement transaction is usually a group of photo / sensor files surrounded or closed by two photo / sensor files having images of border transactions. In the absence of errors, the image / sensor file group includes the image / sensor file having the image corresponding to the validated infringement transaction (namely, "image / violation file").

In the previously described embodiment having only one border transaction entry, the corresponding group of photo / sensor files may be defined as all photo / sensor files having time stamps that enter the time window between the time stamp of the photo / sensor file corresponding to the border transaction entry and time, which is close to one minute after the photo / sensor file timestamp corresponding to the border transaction entry. The video server 1030 can adjust the "time" of the border transaction and / or the period of one minute to include clock deviation or offset (see below) between the clock of the image acquisition module 1024 and the clock of the 1020 lane transaction system.

As an example of the result of surgery 1340, FIG. 11 and 12 represent respectively a group of transaction entries 1100 and the corresponding group of photo / sensor files 1200. The group of transaction entries 1100 includes transaction entries closed by, i.e., surrounded by, border transaction entries 1110 and 1130. Similarly, the photo / sensor file group 1200 includes photo / sensor files closed or surrounded by photo / sensor files 1210 and 1220 having border transaction images corresponding to border transaction entries 1110 and 1130, respectively.

After all group pairs have been identified, the video server 1030 can optionally estimate clock shifts between the internal clock of the 1020 lane transaction system and the internal clock of the image acquisition module 1024 for each pair of groups (1360). The video server 1030 can estimate the clock offset for a pair of groups, for example, as the difference between the time stamp of the border transaction entry and the time stamp of the photo / sensor file having the corresponding photo of the border transaction. If each group contains two borderline transactions, the clock offset for the group pair may be determined, for example, by calculating the average difference between the timestamps of the image acquisition module 1024 and the 1020 lane transaction system corresponding to each borderline transaction. Determining clock offsets is useful for determining one-on-one matching between lane transaction entries and photo / sensor files as described below. Clock deviation can also be calculated by estimating the rate of change in clock shifts based on differences in border / photo transaction times at any end of a significant range (e.g. hours, rather than a minute, or similar within a group).

With reference to FIG. 14, after identifying the groups of photo / sensor files, the video server 1030 may be configured to identify the photo / violation sensor file corresponding to each validated violation transaction through the 1400 matching process. The 1400 matching process identifies the photo / sensor violation file by establishing a one-on-one reference between each photo / sensor file in the photo / sensor file group and each transaction entry in the corresponding transaction entry group. In particular, by using the information contained in the transaction entries both before and after, the violation transaction entry in the matching process, the 1400 process can identify a photo / violation sensor file that matches the violation transaction entry with greater accuracy than is possible by simple comparisons timestamps (namely, by simply indicating the photo / sensor file as a photo / violation sensor file if its photo time stamp is the same or substantially the same as the time stamp of the violation transaction entry).

In particular, as shown in FIG. 14, the process for identifying the photo / violation sensor file for the validated violation transaction entry typically begins by matching the start border transaction entry in the transaction entry group with the starting photo / sensor file in the corresponding photo / sensor file group (1410). For example, if the group of border transaction entries is the group of 1100 ten entries shown in FIG. 11 and the photo / sensor file group is the 1200 file group of ten files shown in FIG. 12, the 1110 border transaction entry will be matched to the image / sensor file 1210.

Using the initial border transaction as a reference, the video server 1030 establishes a one-on-one reference between each subsequent transaction entry in the border transaction group and each subsequent photo / sensor file in the corresponding photo / sensor file group (1420). For example, the video server 1030 matches the second border transaction entry (namely, the first entry registered after the border transaction entry) in group 1100 with the second image / sensor file (namely, the first image / sensor file having data downloaded after the image / limit sensor file) in the group 1200, the third entry of the border transaction in group 1100 with the third photo / sensor file in group 1200, and so on.

If the matching process is successful, the result of the matching process is a one-on-one match between each transaction entry in the transaction entry group and each photo / sensor file in the photo / sensor file group. FIG. 15 is an example of a matched pair of 1600 groups. The matched pair is represented in the figure by lines connecting or matching a transaction entry with a photo / sensor file.

Because each transaction entry is matched to a photo / sensor file, the video server 1030 updates the record of data related to the violation transaction to indicate a match between the given transaction entry and the photo / sensor file. If the 1030 video server identifies one or more abnormal transaction entries and / or photo / sensor files in a group, the 1030 video server may update the data log to mark the corresponding entries as abnormal and to indicate why the entries / photo / sensor files were considered abnormal.

The 1030 video server can be configured to confirm if the matching process was successful (1430) by determining, for example, whether the following criteria are met: (1) the number of transaction entries in the transaction entry group is equal to the number of image files / sensors in the image file group / sensors; (2) each transaction entry in the transaction entry group is matched with each image / sensor file in the corresponding group of image / sensor files; (3) the offset-adjusted time stamp for each transaction entry falls within the expected deviation from the time stamp corresponding to the matched image / sensor files; and (4) the time interval between transactions as reflected by the time stamps is within the range of the expected deviation from the time interval between transactions as reflected by the time stamps of photo / sensor files. Each of these is explained below.

The 1030 video server can count the number of photo / sensor files and border transaction entries in groups to determine if they are equal in quantity. If the 1030 video server specifies that the number of photo / sensor files is different from the number of transaction entries, the 1030 video server may insert one or more transaction entry fillers and / or photo / sensor file filler as necessary to align the number of transaction entries with the number of photo files / sensors (1440). The video server 1030 can insert fillers in groups in place in time that minimizes the deviation between the expected time intervals between transactions and the expected differences between photo / sensor file tags and transaction entry time stamps. The video server can also determine which photo probably does not correspond to the lane transaction using LPR to obtain the license plate data for the vehicles in the photos. A photo with an absent license plate may be assigned a transaction filler rather than a lane related transaction. The number of transaction entries may differ from the number of photo / sensor files due to, for example, the false start of VIS 1024A cameras. Such false launch may create an additional photo / sensor file that has a photo, for example, of a road lane without any vehicle present. If the video server 1030 considers it necessary to add a filler to the groups associated with the violation transaction, the video server 1030 typically modifies the violation data register to indicate that fillers have been inserted and to identify fillers.

The video server 1030 may also determine the deviation between the time stamps of each transaction entry and the time stamps of each corresponding photo / sensor file. Because the internal clocks of the 1020 lane transaction system and the 1024 image acquisition module are independent, clocks, and therefore time stamps generated by the clocks, often indicate different times (e.g., due to the difference in clock resolution, difference in clock deviation, and / or fixed difference in the clock setting). The time difference between the clocks can be represented as the set offset between the clocks and the variable deviation. In one embodiment, the video server may estimate the set offset, for example, based on the difference in timestamps of the border transaction entry and the corresponding borderline photo / sensor file. After estimating the offset between two clocks, the video server 1030 can adjust each transaction entry timestamp (or photo / sensor file timestamp) by offsetting before comparing it to the corresponding photo / sensor file timestamp (or transaction entry timestamp).

If the time stamp of the offset offset transaction (or photo / sensor file timestamp) is significantly different from the photo / sensor file timestamp (or transaction entry timestamp), video server 1030 may modify the violation data log to indicate that the transaction entry and the corresponding image / sensor file are abnormal and that the matching process is not considered successful. For example, if the time stamp of the adjustable offset transaction entry is different from the time stamp of the matching photo / sensor file by one second or more, the video server 1030 may mark the transaction entry and the corresponding photo / sensor file as not having the matching time stamps and may add attention to registry data related to infringement indicating why the matching process was unsuccessful.

The video server 1030 may also determine the deviation between the time interval between transactions reflected by transaction entry time stamps and the time interval between transactions reflected by time stamps of photo / sensor files. For example, if the time interval between the first transaction and the second transaction as reflected by the time stamps of the corresponding transaction entries is eleven seconds, and the corresponding time interval between the same transactions as the time stamps of the matching photo / sensor files is five seconds, the deviation between the time intervals is 6 seconds . If the 1030 video server is configured to indicate a matching error if the transaction intervals differ by more than 4 seconds, the 1030 video server may mark both transaction entries and the corresponding photo / sensor files as not providing a consistent time interval between transactions and may add a note to the data register related to violation indicating why the matching process was not successful. Such differences in the time intervals between transactions may indicate that one or both transaction entries (or photo / sensor files) are not properly matched with the photo / sensor file (or transaction entry).

Upon completion of the matching process, the video server 1030 may identify the photo / violation sensor file as the photo / sensor file that was matched to the violation transaction entry during the matching process (1450). FIG. 15 shows the photo / violation sensor file 1510 fitted to the violation transaction entry 1120 of FIG. 11. Video server 1030 can repeat operations 1410-1450 to identify the photo / violation sensor file for each valid violation transaction entry and associated group of transaction entries and the corresponding group of photo / sensor files (1460).

In another embodiment, the video server 1030 is configured to match each transaction identified by the lane 1020 transaction system with other sensor data (e.g., magnetic signature data and laser signature data) instead of with image data. In this embodiment, the video server 1030 may identify borderline transaction sensor data that correspond to borderline transaction entries rather than identifying borderline transaction photos. The borderline transaction sensor data may be matched to the corresponding borderline transaction entries for synchronizing sensor data with transaction entries as previously described. In another embodiment, the video server 1030 may match each transaction identified by the lane 1020 system based on a combination of border sensor data and border transaction images.

After completing the matching process and associating each violation transaction entry or transaction entry with the corresponding image / sensor file, the video server typically stores identified transaction groups and corresponding image / sensor file groups, identified infringement or image / transaction sensor files, and associated lane transaction data in violation or transaction registers, and sends violation or transaction registers to the 1025 image processing module for vehicle identification and processing. The 1030 video server typically performs all or most of the functions described above (e.g., syntax analysis, identification and validation of violation entry, grouping and matching) using the package application.

Video server 1030 may also be configured to perform various other functions, including: receiving status information from VIC 1024A (and, in some embodiments, from other VICs) and forwarding status messages to a suitable monitoring system (not shown); providing a network-based interface that allows administrators to create messages and send them to VIC 1024B (and, in some embodiments, to other VICs); synchronizing your internal clock to the network time server and periodically setting the internal VIC 1024B clock (and, in some embodiments, the internal clock of other VICs) through automatically generated management messages.

FIG. 16 is a flowchart of an example process 1600 that identifies and charges tolling vehicles for tolls incurred without requiring the system of the road lane transaction system to directly communicate with the toll system image system. For convenience, reference is made to the individual components described in relation to FIG. 10 during process 1300. However, similar methodologies can be used in other embodiments where different components are used to determine the structure of the system, or where functionality is differently distributed between the components shown in FIG. 10. In one embodiment, the 1300 process is implemented by a 1000 toll management system.

The 1024 lane transaction system captures transaction data for each vehicle that deals with object 1028 (1602). The 1020 lane transaction system records transaction data in the lane activity report and sends to the 1030 video server or otherwise allows the 1030 video server to periodically access the lane activity report (e.g., once a day) (1604).

The image acquisition module 1024 captures vehicle photos and optionally other sensor data, and stores image / sensor data in image / sensor files (1606). The 1024 image acquisition module sends to the 1030 video server or otherwise allows the 1030 video server to periodically access photo / sensor files and / or when photo / sensor data is captured (1608).

The video server 1030 receives or accesses lane activity report (1610) and receives or accesses photo / sensor files (1612). Video server 1030 processes the received lane activity report and photo / sensor files to select a group of transaction entries and the corresponding group of photo / sensor files for each violation transaction entry in the lane activity report (1614). FIG. 13 shows an example process 1300 that can be used by video server 1030 to perform operation 1614.

After identifying the group of transaction entries and the corresponding group of photo / sensor files for one or more validated violation transactions, the video server 1030 identifies the photo / violation sensor file for each validated violation transaction entry by performing the transaction entry and photo / sensor file matching process for each pair of groups (1616). FIG. 14 illustrates an exemplary process 1400 that can be used by video server 1030 to perform operation 1616.

For each validated violation transaction entry, video server 1030 is configured to record one or more of the following in the data register (namely, the "violation register") (1618): (1) a group of transaction entries that correspond to and include a validated violation transaction entry; (2) the corresponding group of photo / sensor files that include the photo / sensor file of the identified violation; (3) matching data identifying matched pairs of transaction entries and photo / sensor files; (4) markings and notes indicating entries of abnormal transactions and / or photo / sensor files and an explanation of why they are abnormal; and (5) data indicating whether and why the matching process was considered successful or unsuccessful.

The video server 1030 is configured to send to the image processing module 1025 or to allow the image processing module 1025 access to one or more violation registers (1620). The image processing module 1025 receives or accesses one or more violation registers (1622) and, optionally, can provide information in the violation registers to the user for manual confirmation of the matching process and the photo / sensor file of the identified violation (1624).

The 1025 photo processing module may include a violation review application that allows the user to perform the following tasks: (1) checking the relationship or matching between lane transaction entries and photo / sensor files; (2) improving the fit when necessary (including manual insertion of fillers when necessary); (3) confirmation of the correct matching of lane transaction entries and photo / sensor files based on time adjustment and photo content; (4) manual determination of photo file / violation sensor; (5) enter identification information for the infringing vehicle or the owner or person associated with the infringing vehicle; and (6) determining the nature of the breach transaction and the reason for that nature.

FIG. 17 depicts an example user interface 1700 that lists matched transaction entries for the ten transactions corresponding to the transaction entries of FIG. 11 and the photo / sensor files of FIG. 12. The 1700 user interface includes a matched transaction entry for each matched pair of transaction entries - photo / sensor file.

An example of a matched transaction entry 1700 includes lane ID 1715 (e.g., "15 3738"), lane transaction timestamp 1720 (e.g., "4:30:39"), timestamp of a matching photo 1725 \ (e.g., "4:30 : 34 "), transaction type 1730 (e.g." STD AVI "), transaction description 1735 (e.g. empty if the transaction is not a violation); the nature of the transaction 1740 (e.g. empty if the transaction is not a violation); the time interval between the last transaction and the current transaction according to the 1745 lane transaction timestamps (e.g., "2" second), and the time interval between the last transaction and the current transaction according to the timestamps of 1750 matching images (e.g., "2" second).

The 1700 user interface can be configured to include a button, icon, or other interface element (not shown) selected to generate a graph showing the timestamp deviation between the photo / sensor file time stamp and the transaction entry time stamp for one or more matched pairs. For example, FIG. 18 shows an example of a 1800 bar chart configured to indicate the difference in time between a lane transaction time stamp for a transaction and a time stamp of the corresponding photo / sensor file for the same transaction. If the time difference is above a certain threshold, such as, for example, 1 second, the transaction may be considered poorly matched, problematic, and / or abnormal. The bar chart 1800 shows that the transaction marked as "transaction 4" has a time stamp difference of more than 1 second, and can therefore be considered abnormal.

Bar charts, such as the 1800 bar chart, are particularly useful because they allow the user to quickly, at a glance, determine the accuracy of the matching process and focus on poorly matched, problematic, and / or abnormal transactions. In some embodiments, the graph reflects the time differences after offset compensation between the internal clock of the image acquisition module 1024 and the internal clock of the road lane transaction system 1020.

The 1700 user interface can also be configured to include a button, icon, or other interface element selected to generate a graph showing the time intervals between the current and previous transaction as reflected by the time stamps of photo / sensor files and the time intervals between the current and previous transaction as reflected by the markers transaction entry times For example, FIG. 19 shows an example of a 1900 bar chart that allocates a pair of bars to each transaction, one having an altitude reflecting the time interval between the current and previous transaction as determined from time lane transaction entries and the other having an altitude reflecting the time interval between the current and previous transaction as determined from the markers time of photo / sensor files.

If the difference in the amount of two-ply charts related to the same transaction indicates a difference in time over a certain threshold, such as, for example, four seconds, the transaction may be considered poorly matched, problematic, and / or abnormal. The 1900 bar chart does not show any transaction related to two bars that differ in height by four or more seconds. Rather, it shows differences of 0 seconds, 1 second, 1 second, and 1 second for transactions 1 to 4, respectively. Thus, the 1900 bar chart indicates that all four transactions are well matched with respect to this criterion. Like the 1800 bar chart, the 1900 bar chart allows the user on the glass, at a glance, to determine the accuracy of the matching process and focus on poorly matched, problematic, and / or abnormal transactions.

With reference again to FIG. 16, image processing module 1025 identifies infringing vehicles from photo / violation sensor files and, optionally, from transaction data in the violation transaction entry (1626). The image processing module 1025 can perform this identification using any or all of the methods previously described. After identifying the infringing vehicle for each violation transaction, the photo processing module 1025 sends the identified vehicle and associated transaction data to the bill issuing machine 1022 for processing as already described (e.g. with reference to FIGS. 3 and 4) (1628).

The above applications are illustrative examples, and the disclosed techniques can be used in other applications. Furthermore, various aspects and disclosed techniques (including systems and processes) may be modified, combined in whole or in part with each other, supplemented, or removed to create additional embodiments.

The systems and techniques described here may be implemented in digital electronic circuitry assemblies, or in computer hardware, firmware, software, or a combination thereof. The systems and techniques described herein may be implemented as a computer program product, namely a computer program mandrel embodied in an information medium, e.g. in a machine-readable storage device or in a distributed signal, for performing by or for controlling an operation, a data processing device, e.g. a programmable processor, computer, or multiple computers. The computer program may be saved in any form of programming language, including compiled or interpreted languages, and may be used in any form, including as an autonomous program or as a module, component, subroutine, or other entity suitable for use in a computing environment . The computer program can be used for use on one computer or on many computers in one place or distributed in many places and connected to a communication network.

The method steps of the systems and techniques described herein may be performed by one or more programmable processors executing a computer program to perform the function of the invention by operating on input data and generating output. The method steps can also be carried out by, and the device according to the invention can be implemented as logical circuits for special purposes, e.g. FPGA (user programmable input array) or ASIC (special purpose integrated circuit).

Processors suitable for executing a computer program include, for example, general and specific application microprocessors, and any one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from read-only memory or sparse memory or both.

Typical components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, the computer will also include, or will be operably connected to receive or transfer data, or both, to one or more mass storage devices for storing data, e.g., magnetic, magnetic optical discs, or optical discs. Information media suitable to contain computer program and data instructions include all forms of non-volatile memory, including, for example, semiconductor memory devices, e.g. EPROM, EEPROM, flash memory devices; magnetic disks such as internal hard disks and removable disks; magnetic optical discs; and CD-ROMs and DVD-ROMs. The processor and memory can be supplemented by, or incorporated into, special purpose logic circuitry.

To ensure user interaction, the systems and techniques described herein can be implemented into a computer having a display such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and pointing device such as a mouse or ball manipulator, thanks to which the user can provide a batch to the computer. Other types of devices may also be provided to ensure interaction with the user; for example, the feedback provided to the user may be in any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the load from the user may be received in any form, including the load acoustically, by speech, or by touch.

The system and techniques described herein can be implemented in a computing system that includes an internal element, e.g., a data server, or includes a software and hardware element, e.g., an application server, or includes an external element, e.g., a client computer having a graphical user interface or a Web browser, by which the user can interact with an embodiment of the invention, or any combination of such internal, software-hardware, or external elements. Elements of the system can be connected through any form or medium of digital data communication, e.g. a communication network. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

The computing system may include clients and servers. The client and server are generally distant from each other and usually interact via the communication network. Client and server relationships arise from computer programs that work on the appropriate computers and have a client-server relationship.

Proxy:

"BELLEPAT" LAW AND PATENT OFFICE

Izabela Szychulska-Hawranek ul. Słowackiego 44, 37-700 Przemyśl tei. (016) 732-37-77 fax: (016) &lt;575-02-87 mobile phone (0608) 503-081 e-maii:<a href="mailto:beltepat@op.pl">beltepat@op.pl</a> NIP: 795-207-16-72 REGON: 1803505: 6, Γ,

PATENT ADVISOR mgr Izabela Szythułska-Hawranek, entry no. 3182