Surveillance System

CROSS-REFERENCE TO RELATED APPLICATIONS

0001This application claims the benefit of U.S. Provisional Patent Application No. 61/471,362, filed on 4 Apr. 2011, entitled “Unified Synchronous CCTV Camera Information Display and Storage System” (attorney docket no. 465-070us), which is incorporated by reference herein.

FIELD OF THE INVENTION

0002The present invention relates to surveillance systems in general, and, more particularly, to augmenting the video stream of a surveillance system.

BACKGROUND OF THE INVENTION

0003Surveillance systems are being deployed in increasing numbers, in both public and private venues, for security and counter-terrorism purposes. Typically, a surveillance system employs one or more closed-circuit television (CCTV) cameras and delivers the video feed (i.e., video signal) from a CCTV camera to one or both of: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0004">a display for real-time monitoring by a human operator, and</li><li id="ul0002-0002" num="0005">archival storage that enables future viewing of the video. <br /> Often, however, the operators of the surveillance system are unable to identify the people in the video. Therefore, the need exists for an improved technique for identifying the people in surveillance video. </li></ul></li></ul>

SUMMARY OF THE INVENTION

0006The operator of a surveillance system often desires four pieces of information for each person of interest: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0007">the person's identity,</li><li id="ul0004-0002" num="0008">the person's location at a given time,</li><li id="ul0004-0003" num="0009">a clear picture of the person's face, and</li><li id="ul0004-0004" num="0010">the person's mobile telephone number (so that the operator can eavesdrop on the person's voice and text communications). <br /> Often, however, the operator does not have all four pieces. The pieces that a surveillance system can ascertain depend on the circumstances. </li></ul></li></ul>

0011First, the video feed of a surveillance system is invariably able to ascertain a person's location at a given time—from the mere fact that there is a person in the picture. The video surveillance system might or might not be able to identify the person, but it can show that a person—some person—is present at a specific location at a specific time. And by using projective geometry and trigonometry, the location of the person can be ascertained precisely.

0012Second, the surveillance system might or might not capture a clear picture of the person's face. If the person is intentionally hiding his or her face or if the lighting conditions are bad, then it might be difficult to capture a clear picture of the person's face.

0013Third, the surveillance system might be able to ascertain the person's identity by using a facial recognition system if the video feed does in fact capture a clear picture of the person's face. If the person's face and identity are not in the facial recognition system, then it might be difficult to ascertain the person's identity.

0014Fourth, the surveillance system might be able to ascertain the person's mobile phone telephone number if it can ascertain his or her identity. But if the person carries a disposable mobile phone or uses an alias to register his or her mobile phone, then it might be difficult to ascertain the person's mobile phone telephone number.

0015In contrast, embodiments of the present invention augment the information available to a traditional surveillance system with the information available to a mobile phone location system. A mobile phone location system is typically used by a 911 call center to ascertain the location of a mobile phone so that police, fire, and first aid can respond to an emergency. Although a mobile phone location system is traditionally used to ascertain the unknown location of a mobile phone from a known telephone number, it can be used to correlate a mobile phone's location with its telephone number.

0016By using the information from a surveillance system in conjunction with a mobile phone location system, a number of inferences can be made that are unknowable to either system alone. For example and without limitation, a person at a given location (as determined from the video feed) and a mobile phone at the same location (as determined from the mobile phone location system) can be associated. If the phone is registered to a name, the person's identity can be tentatively established. Even if the phone is not registered to a name or is registered to an obvious alias, the operator can thereafter eavesdrop on the person's communications.

0017Some embodiments of the present invention are capable of: <ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0000"><ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0018">i. augmenting a video signal with information that is associated with persons who appear in the video—and also other information associated with persons who have a relevant relationship with persons who appear in the video;</li><li id="ul0006-0002" num="0019">ii. augmenting a video with graphics depicting communications, financial transactions, social network connections, and personal information;</li><li id="ul0006-0003" num="0020">iii. ascertaining the identity of a person in a video stream (e.g., via a facial-recognition system, via information from a wireless location system, via information from a lawful interception system, etc.);</li><li id="ul0006-0004" num="0021">iv. ascertaining when an identified person initiates, terminates, or is currently participating in a communication (e.g., places a telephone call, receives a telephone call, hangs up, sends/receives a Short Message Service (“SMS”) message, receives/sends an email, participates in an instant message (“IM”) session, fetches a webpage within a browser, sends/receives a text message, receives/sends/participates in a social network system, e.g., Twitter, Facebook, LinkedIn, etc.);</li><li id="ul0006-0005" num="0022">v. obtaining information associated with the communication (e.g., the mobile telephone number of the identified person or of the other parties involved in the communication, the type of communication, the identities of other parties involved in the communication, other information about one or more of the other parties, etc.);</li><li id="ul0006-0006" num="0023">vi. generating graphics that represent the communication and depict some or all of the associated information;</li><li id="ul0006-0007" num="0024">vii. ascertaining when an identified person initiates, terminates, or is currently executing a financial transaction (e.g., a bank deposit, a transfer from one account to another, an online purchase, etc.);</li><li id="ul0006-0008" num="0025">viii. obtaining information associated with the financial transaction (e.g., the type of transaction, the magnitude/amount of the transaction, the identity of the financial institution, the identity of another party associated with the transaction, a bank account number, a credit card number, etc.);</li><li id="ul0006-0009" num="0026">ix. generating graphics that represent the financial transaction and depict the associated information;</li><li id="ul0006-0010" num="0027">x. obtaining other types of personal information associated with the identified person (e.g., the person's social security number, the person's driver's license number, an image of the person's driver's license, an image of the person's passport, the person's birthdate and/or birthplace, an image of the person's birth certificate, border crossing information, motor vehicle records, criminal history records, credit reports, home address, telephone directory information, etc.)—even when the identified person does not appear in the video but has a relevant relationship or association with a person who appears;</li><li id="ul0006-0011" num="0028">xi. generating graphics that depict the personal information;</li><li id="ul0006-0012" num="0029">xii. combining one or more of the generated graphics with the received video signal into a resultant “composite video,” e.g., overlaying the graphics on the video stream, generating a new stream that comprises a first window with the video feed and a second window with the graphics, etc.;</li><li id="ul0006-0013" num="0030">xiii. estimating the identity of a confederate who is in communication with the mobile in the surveillance area—whether the confederate also appears in the video or not;</li><li id="ul0006-0014" num="0031">xiv. identifying relationships between the person in the video and others, including social relationships, familial relationships, financial relationships, communications past or present, etc.;</li><li id="ul0006-0015" num="0032">xv. identifying a relationship between two persons appearing in the surveillance video, even when the two persons are not actively communicating;</li><li id="ul0006-0016" num="0033">xvi. inferring that two persons appearing in the video are actively communicating, whether by voice call, text, or otherwise; and</li><li id="ul0006-0017" num="0034">xvii. retrieving information about the identified person(s) and mobile(s) from other data sources, such as criminal records databases, public records databases, surveillance records, etc.</li></ul></li></ul>

0035When facial recognition is applied to an image of a person appearing in a video and the person's identity is thus recognized, the illustrative video augmentation system generates an association between the identity according to the facial recognition and the identity of the “co-located” mobile phone. This association can be critical when there is no personal identification available for a mobile phone such as a pre-paid mobile. This association is displayed and also stored for archival purposes.

0036The composite video provides a unified view of activities and information associated with persons in the video feed, thereby enabling a human operator to view a great deal of information and manipulate it interactively in real time. Advantageously, the illustrative embodiment obtains and integrates the supplemental information automatically, without requiring the human operator to actively seek out and import these data. In addition, the illustrative embodiment archives the composite video for future viewing and analysis (e.g., for forensic analysis of a crime, etc.). As noted, the augmented information that is displayed in the composite video (and any other information, whether displayed in the composite video or not) is archived for future retrieval and analysis.

0037In accordance with the illustrative embodiment, communication, financial, social network, and personal information associated with a particular person appearing in the video stream is obtained by the system via one or more of the following techniques: <ul id="ul0007" list-style="none"><li id="ul0007-0001" num="0000"><ul id="ul0008" list-style="none"><li id="ul0008-0001" num="0038">transmitting a query to or polling a remote entity (e.g., a public records database, a call record database, a criminal records database, a credit reporting database, a news source/database, a surveillance database, etc.),</li><li id="ul0008-0002" num="0039">actively polling for pertinent events associated with the person (e.g., the initiation of a new communication, the termination of a communication, a new online transaction, etc.), and</li><li id="ul0008-0003" num="0040">registering to be notified about events associated with the person (e.g., via a publish/subscribe messaging system, via an email generated by a database trigger, etc.).</li></ul></li></ul>

0041An illustrative embodiment comprises: receiving, by a data-processing system, a first video signal that carries a first video frame, wherein the first video frame comprises an image of a person at a location; estimating the identity of a mobile phone based on the location of the person; estimating the identity of the person based on the identity of the mobile phone; and transmitting, by the data-processing system, a second video signal that carries a second video frame that is based on the first video frame, wherein the second video frame comprises: (i) the image of the person, and (ii) a depiction of the identity of the person.

BRIEF DESCRIPTION OF THE DRAWINGS

0042<figref idref="DRAWINGS">FIG. 1A</figref> depicts a block diagram of some salient components of video augmentation system <b>100</b> according to an illustrative embodiment of the present invention.

0043<figref idref="DRAWINGS">FIG. 1B</figref> depicts a block diagram of some salient components of video augmentation system <b>100</b> as connected to external systems and data sources, in accordance with an illustrative embodiment of the present invention.

0044<figref idref="DRAWINGS">FIG. 2</figref> depicts a block diagram comprising the salient elements of data-processing system <b>101</b> according to an illustrative embodiment of the present invention.

0045<figref idref="DRAWINGS">FIG. 3A</figref> depicts an illustrative frame of video signal <b>111</b> according to an illustrative embodiment of the present invention.

0046<figref idref="DRAWINGS">FIG. 3B</figref> depicts an illustrative frame of composite video signal <b>113</b> in accordance with an illustrative embodiment of the present invention.

0047<figref idref="DRAWINGS">FIG. 4</figref> depicts an illustrative frame of composite video signal <b>400</b> in accordance with an illustrative embodiment of the present invention.

0048<figref idref="DRAWINGS">FIG. 5A</figref> depicts an illustrative frame of composite video signal <b>500</b> according to an illustrative embodiment of the present invention.

0049<figref idref="DRAWINGS">FIG. 5B</figref> depicts an illustrative frame of composite video signal <b>520</b> according to an illustrative embodiment of the present invention.

0050<figref idref="DRAWINGS">FIG. 6A</figref> depicts a first illustrative frame of composite video signal <b>600</b> according to an illustrative embodiment of the present invention.

0051<figref idref="DRAWINGS">FIG. 6B</figref> depicts a second illustrative frame of composite video signal <b>600</b> according to an illustrative embodiment of the present invention.

0052<figref idref="DRAWINGS">FIG. 6C</figref> depicts a third illustrative frame of composite video signal <b>600</b> according to an illustrative embodiment of the present invention.

0053<figref idref="DRAWINGS">FIG. 7</figref> depicts a flowchart of the salient operations of illustrative method <b>700</b> in accordance with an illustrative embodiment of the present invention.

0054<figref idref="DRAWINGS">FIG. 8</figref> depicts a flowchart of the salient operations of illustrative operation <b>705</b> in accordance with an illustrative embodiment of the present invention.

0055<figref idref="DRAWINGS">FIG. 9</figref> depicts a flowchart of the salient operations of illustrative operation <b>807</b> in accordance with an illustrative embodiment of the present invention.

0056<figref idref="DRAWINGS">FIG. 10</figref> depicts a flowchart of the salient operations of illustrative operation <b>905</b> in accordance with an illustrative embodiment of the present invention.

0057<figref idref="DRAWINGS">FIG. 11</figref> depicts a flowchart of the salient operations of illustrative operation <b>707</b> in accordance with an illustrative embodiment of the present invention.

0058<figref idref="DRAWINGS">FIG. 12</figref> depicts a flowchart of the salient operations of illustrative operation <b>1109</b> in accordance with an illustrative embodiment of the present invention.

0059<figref idref="DRAWINGS">FIG. 13</figref> depicts a flowchart of the salient operations of illustrative operation <b>1207</b> in accordance with an illustrative embodiment of the present invention.

0060<figref idref="DRAWINGS">FIG. 14</figref> depicts a flowchart of the salient operations of illustrative operation <b>709</b> in accordance with an illustrative embodiment of the present invention.

DETAILED DESCRIPTION

0061For the purposes of this specification, the following terms and their inflected forms are defined as follows: <ul id="ul0009" list-style="none"><li id="ul0009-0001" num="0000"><ul id="ul0010" list-style="none"><li id="ul0010-0001" num="0062">The term “location” is defined as a zero-dimensional point, a one-dimensional line, a two-dimensional area, or a three-dimensional volume.</li><li id="ul0010-0002" num="0063">The term “composite video” is an electronic or electromagnetic signal that results from combining: (a) all or parts of the original video signal obtained from a surveillance camera or from another source of video material with (b) supplemental information that is to be displayed. Hereinafter and in the accompanying figures the terms “composite video” and “composite video signal” and “composite video stream” are used interchangeably.</li><li id="ul0010-0003" num="0064">The term “tag” is defined as a graphic that comprises information about something in the video signal, such as text, a text box, a balloon graphic, an arrow, an icon, a ticker tape strip, etc. A tag comprises information such as a name, a telephone number, an indicium of an active telephone conversation, an indicium of a data session, an indication of a relationship, a numerical designation of a person in the video, etc. The shape of the tag, the contents of the tag, and any other formatting aspects of the tag do not limit the present invention.</li></ul></li></ul>

0065<figref idref="DRAWINGS">FIG. 1A</figref> depicts a block diagram of some salient components of video augmentation system <b>100</b> according to an illustrative embodiment of the present invention. Video augmentation system <b>100</b> comprises: data-processing system <b>101</b>; video camera <b>103</b>; input device <b>105</b>; display <b>107</b>; and data store <b>109</b>. Furthermore, video signal <b>111</b> is transmitted from video camera <b>103</b> to data processing system <b>101</b>, and composite video signal <b>113</b> is transmitted from data-processing system <b>101</b> to display <b>107</b> and/or to data store <b>109</b>.

0066Data-processing system <b>101</b> is a computer that comprises non-transitory memory, processing component(s), and communication component(s) as described in more detail in <figref idref="DRAWINGS">FIG. 2</figref>. Data-processing system <b>101</b> executes and coordinates the salient operations of video augmentation system <b>100</b> according to an illustrative embodiment. For example, data-processing system <b>101</b> receives the original video signal <b>111</b> from video camera <b>103</b>, processes it according to the methods disclosed herein, generates composite video signal <b>113</b>, transmits it to display <b>107</b> and, for archiving, transmits it to data store <b>109</b> and elsewhere. Data-processing system <b>101</b>, in order to perform some of its functions, also communicates, coordinates, and electronically interacts (wired or wirelessly as appropriate) with systems outside of video augmentation system <b>100</b>, as described in further detail in <figref idref="DRAWINGS">FIG. 1B</figref>. For example, data-processing system <b>101</b> is also connected to a wireless network location system <b>131</b>, which in some embodiments is the Wireless Location Signatures (WLS™) technology from Polaris Wireless, Inc.

0067Video camera <b>103</b> is well known in the art. Video camera <b>103</b> generates an original video signal <b>111</b> that depicts the geographic area under the “eye” of video camera <b>103</b>. Video signal <b>111</b> is the video feed from the surveillance camera(s). Video camera <b>103</b> transmits video signal <b>111</b> to data-processing system <b>101</b> in well known fashion.

0068Input device <b>105</b> is well known in the art. Input device <b>105</b> enables a user of video augmentation system <b>100</b> to interact with the system, for example, enabling the user to: <ul id="ul0011" list-style="none"><li id="ul0011-0001" num="0000"><ul id="ul0012" list-style="none"><li id="ul0012-0001" num="0069">command video camera <b>103</b> to take an action, e.g., move its viewing angle, change its geographical surveillance area, zoom, etc.,</li><li id="ul0012-0002" num="0070">indicate that a person visible on display <b>107</b> is of interest to the user,</li><li id="ul0012-0003" num="0071">request supplemental information about a person visible on display <b>107</b>, including information about the person's identity and/or their mobile units,</li><li id="ul0012-0004" num="0072">request reports about a person or persons visible on display <b>107</b>,</li><li id="ul0012-0005" num="0073">request other data and/or reports from data store <b>109</b>, etc.</li></ul></li></ul>

0074Display <b>107</b> is well known in the art. Display <b>107</b> receives composite video signal <b>113</b> from data-processing system <b>101</b> and displays the composite video signal in a manner visible to a user.

0075Data store <b>109</b> is an electronic data storage element that is well known in the art. Data store <b>109</b> comprises non-transitory memory (e.g., a hard disk, etc.) that is used by video augmentation system <b>100</b> to store, archive, and retrieve information. For example, data store <b>109</b> receives video signal <b>111</b> from data-processing system <b>101</b> and archives it; data store <b>109</b> receives composite video signal <b>113</b> from data-processing system <b>101</b> and archives it; data store <b>109</b> receives supplemental information from data-processing system <b>101</b> and archives it; data store <b>109</b> transmits supplemental information data to data-processing system <b>101</b> in response to a retrieval request; data store <b>109</b> transmits archived composite video signal <b>113</b> to data-processing system <b>101</b> in response to a retrieval request.

0076It will be clear to those having ordinary skill in the art after reading the present disclosure that the system illustrated in <figref idref="DRAWINGS">FIG. 1A</figref> can be embodied in different variations that are consistent with the present invention. For example, some embodiments comprise several video cameras that cover a plurality of geographic regions. For example, some embodiments comprise several displays and input devices for a plurality of users. For example, in some embodiments, data store <b>109</b> comprises a plurality of data stores or a plurality of data storage technologies (e.g., a cloud-based storage system). For example, in some embodiments, not all depicted components are on-site. For example, in some embodiments, the depicted components are interconnected indirectly, e.g., through servers, gateways, switches, networks, the Internet, etc. It will be clear to those having ordinary skill in the art, after reading the present disclosure, how to make and use video augmentation system <b>100</b>.

0077<figref idref="DRAWINGS">FIG. 1B</figref> depicts a block diagram of some salient components of video augmentation system <b>100</b> as connected to external systems and data sources, in accordance with an illustrative embodiment of the present invention. In addition to the components depicted in <figref idref="DRAWINGS">FIG. 1A</figref>, in some embodiments, video augmentation system <b>100</b> is also connected to components that are external to the system and that provide additional sources and repositories of data. <figref idref="DRAWINGS">FIG. 1B</figref> depicts: video augmentation system <b>100</b> according to <figref idref="DRAWINGS">FIG. 1A</figref>; telecommunications network <b>120</b>; data store <b>121</b>; facial recognition system <b>123</b>; communications reporting system <b>125</b>; financial reporting system <b>127</b>; social network system <b>129</b>; and wireless network location system <b>131</b>. Some embodiments are configured with a plurality of one or more of these external systems. Some embodiments incorporate one or more of these systems within data augmentation system <b>100</b>.

0078Telecommunications network <b>120</b> is well known in the art. Telecommunications network <b>120</b> provides video augmentation system <b>100</b> with connectivity to other (external) systems that enable video augmentation system <b>100</b> to retrieve data and also to transmit/store/archive data as needed. Telecommunications network <b>120</b> has wired and wireless elements. In some embodiments, telecommunications network <b>120</b> is the Public Switched Telephone Network; in some embodiments, telecommunications network <b>120</b> is the Internet; in some embodiments, telecommunications network <b>120</b> is a private data network. It will be clear to those having ordinary skill in the art, after reading the present disclosure, that in some embodiments, telecommunications network <b>120</b> can be one or more of the above-mentioned networks, without limitation.

0079Data store <b>121</b> is analogous to, and performs the same functions as, data store <b>109</b> described above.

0080Facial recognition system <b>123</b> is well known in the art. Facial recognition system <b>123</b> is capable of receiving images extracted from video signal <b>111</b> and determining whether an image represents a known person. Facial recognition system <b>123</b> is also capable of reporting its results to data-processing system <b>101</b>, including the identity of the recognized person, several possible identities of the person, several other traits associated with the image of the person, etc.

0081Communications reporting system <b>125</b> is well known in the art. Communications reporting system <b>125</b> is capable of receiving queries and requests for communications data from data-processing system <b>101</b>. Communications data comprises: <ul id="ul0013" list-style="none"><li id="ul0013-0001" num="0000"><ul id="ul0014" list-style="none"><li id="ul0014-0001" num="0082">information about an active communication in which a given telecommunications terminal is currently engaged, e.g., an ongoing voice call, text messaging, email, etc.,</li><li id="ul0014-0002" num="0083">a call record about a past call by the given telecommunications terminal,</li><li id="ul0014-0003" num="0084">a list of voice calls to/from a certain telephone number,</li><li id="ul0014-0004" num="0085">a list of text messages to/from a certain mobile telephone number,</li><li id="ul0014-0005" num="0086">a list of emails to/from a certain email address that is associated with the given telecommunications terminal, etc.</li></ul></li></ul>

0087Communications reporting system <b>125</b> processes the requests (including retrieving information from other systems) and transmits responsive results to data-processing system <b>101</b>. Communications reporting system <b>125</b> is also capable of storing and archiving communications records, such as call records, etc.

0088Financial reporting system <b>127</b> is well known in the art. Financial reporting system <b>127</b> receives queries and requests for data from data-processing system <b>101</b>, such as requests for financial records involving a given telecommunications terminal or financial records for a given person's identity, e.g., a list of banking transactions that occurred from the telecommunications terminal, a list of banking transactions for a certain person, a list of bank accounts for a certain person, a list of credit cards issued to a certain person, etc. Financial reporting system <b>127</b> processes the requests (including retrieving information from other systems) and transmits responsive results to data-processing system <b>101</b>. Financial reporting system <b>127</b> is also capable of storing and archiving financial records.

0089Social network system <b>129</b> is well known in the art. Examples of social network system <b>129</b> include Facebook, LinkedIn®, Twitter, and other meta-systems that extract, report, and store data from these individual systems. Social network system <b>129</b> receives queries and requests for data from data-processing system <b>101</b>, such as a person's list of “friends” or “connections” on the social network system, a person's information about themselves posted on the social network system, a person's list of people they are following on the social network system, etc. Social network system <b>129</b> processes the requests (including retrieving information from other systems) and transmits responsive results to data-processing system <b>101</b>.

0090Wireless network location system <b>131</b> is well known in the art and is connected to a wireless network that covers the geographic region under surveillance by video augmentation system <b>100</b> (the wireless networks that cover the surveilled region are not shown in the present figure for the sake of simplicity). A wireless network location system <b>131</b> deployed in a wireless network analyzes data that are available from the network infrastructure and from the mobile units served by the network and estimates, to a certain resolution, the location of the mobile units in the network. In the present illustration, wireless network location system <b>131</b> is a Serving Mobile Location Center (“SMLC”) operating in a GSM wireless network. A GSM SMLC computes estimates of the locations of mobiles in the network. An example of a wireless network location system <b>131</b> that is an SMLC is the Wireless Location Signature (WLS™) service from Polaris Wireless, Inc., which performs a mass location analysis that provides relatively high-resolution location estimates for large numbers of mobile units in a given geographic area covered by the wireless network.

0091It will be clear to those having ordinary skill in the art, after reading the present disclosure, how to make and use alternative embodiments of the present invention that operate in a non-GSM wireless network. Typically, each wireless network has its own wireless network location system <b>131</b> that connects to video augmentation system <b>100</b>, but in some embodiments of the present invention, wireless network location system <b>131</b> is an aggregator of location data available from all the relevant wireless networks. Moreover, in some embodiments, data-processing system <b>101</b> comprises a data aggregation function that receives and integrates location data from a plurality of wireless network location systems <b>131</b>.

0092Other external systems can also connect to video augmentation system <b>100</b>, but are not depicted in FIG. <b>1</b>B—for example: <ul id="ul0015" list-style="none"><li id="ul0015-0001" num="0000"><ul id="ul0016" list-style="none"><li id="ul0016-0001" num="0093">a database of surveillance records;</li><li id="ul0016-0002" num="0094">a public records database, such as real estate taxation maps;</li><li id="ul0016-0003" num="0095">a criminal records database;</li><li id="ul0016-0004" num="0096">etc.</li></ul></li></ul>

0097In addition to being sources of information for video augmentation system <b>100</b>, the above-recited systems also are capable of receiving, storing, and archiving information transmitted to them by video augmentation system <b>100</b>. It will be clear to those having ordinary skill in the art after reading the present disclosure that the system illustrated in <figref idref="DRAWINGS">FIG. 1B</figref> can be embodied in many variations that are consistent with the present invention. For example, in some embodiments, video augmentation system <b>100</b> comprises one or more of components <b>121</b>-<b>131</b> and/or other above-mentioned systems. It will be clear to those having ordinary skill in the art, after reading the present disclosure, how to make and use video augmentation system <b>100</b> according to <figref idref="DRAWINGS">FIG. 1B</figref>.

0098<figref idref="DRAWINGS">FIG. 2</figref> depicts a block diagram comprising the salient elements of data-processing system <b>101</b> according to an illustrative embodiment of the present invention. Data-processing system <b>101</b> comprises: processor <b>201</b>; memory <b>202</b>; transceiver <b>203</b>; and communication paths to video camera <b>103</b>, input device <b>105</b>, display <b>107</b>, data store <b>109</b>, and telecommunications network <b>120</b>.

0099Data-processing system <b>101</b> executes and coordinates the salient operations of video augmentation system <b>100</b> as explained in more detail below and in the accompanying figures. Data processing system <b>101</b> comprises the hardware necessary to execute software, receive data, transmit data, and process data such that it can perform the operations described herein.

0100Processor <b>201</b> is a processing device such as a microprocessor that, in conjunctions with the other components in data-processing system <b>101</b>, is capable of executing the software and processing the data according to the operations described herein. Processor <b>201</b> receives user input via input device <b>105</b> and, after processing, transmits composite video signals to display <b>107</b>. Processor <b>201</b> is well known in the art.

0101Memory <b>202</b> is non-transitory memory that stores program code and data sufficient to enable the execution of software and data processing according to the operations recited herein. Memory <b>202</b> is well known in the art.

0102Transceiver <b>203</b> is a component that enables data-processing system <b>101</b> to communicate electronically (whether in a wired or wireless configuration) with other components internal and external to video augmentation system <b>100</b>, including receiving video signals from camera <b>103</b>; and transmitting to and from data store <b>109</b> and external systems via telecommunications network <b>120</b>. Transceiver <b>203</b> is well known in the art.

0103It will be clear to those having ordinary skill in the art, after reading the present disclosure, that data-processing system <b>101</b> can be embodied as a multi-processor platform, as a server, as a sub-component of a larger computing platform, or in some other computing environment—all within the scope of the present invention. It will be clear to those having ordinary skill in the art, after reading the present disclosure, how to make and use data-processing system <b>101</b>.

0104<figref idref="DRAWINGS">FIG. 3A</figref> depicts illustrative frame <b>301</b> of video signal <b>111</b> according to an illustrative embodiment of the present invention. Video signal <b>111</b> is transmitted by video camera <b>103</b> and captures a certain geographic area covered by the camera's “eye.” Video signal <b>111</b> is a video feed that is well known in the art. Video signal <b>111</b> carries a plurality of video frames, including illustrative frame <b>301</b>.

0105Frame <b>301</b> comprises: image <b>303</b>; mobile phone <b>305</b>; background landscape <b>320</b>; and foreground landscape <b>330</b>.

0106Image <b>303</b> depicts a person in a manner well known in the art. Image <b>303</b> is an image of a person in the geographic region monitored by video camera <b>103</b>.

0107Mobile phone <b>305</b> is (illustratively) a GSM cellular telephone that is carried by the person in image <b>303</b>. It will be clear to those having ordinary skill in the art, after reading the present disclosure, that mobile phone <b>305</b> is well known in the art and can be any type of wireless telecommunications terminal, including a cellular telephone (e.g., CDMA, GSM, UMTS, etc.), smartphone (e.g., iPhone, Android, Blackberry, etc.), mobile computing tablet (e.g., iTouch, Kindle Fire, iPad, etc.), or a combination thereof, or other mobile unit. Voice capabilities are not required according to the present invention, although as is evident from the illustrative examples, the mobile phones depicted herein support voice calls. Mobile phone <b>305</b> (and others discussed herein) is referred to synonymously as “mobile” or “mobile unit” or “wireless telecommunications terminal” or “user endpoint” or “user equipment.” It will be clear to those having ordinary skill in the art, after reading the present disclosure, that video augmentation system <b>100</b> operates with more than one type of wireless network, so that different kinds of mobiles <b>305</b> from different networks can operate with the system. Although mobile <b>305</b> is clearly depicted (and not to scale) in <figref idref="DRAWINGS">FIG. 3A</figref>, it is not necessary for mobile <b>305</b> to be visible in frame <b>301</b>.

0108Location <b>307</b> is a geographic area that comprises both the person in image <b>303</b> and, impliedly, the mobile phone that the person carries, i.e., mobile <b>305</b>. For purposes of the present disclosure, location <b>307</b> is the location of the person in image <b>303</b> as determined by data-processing system <b>101</b>. In other words, data-processing system <b>101</b> can resolve location <b>307</b> for image <b>303</b> within the known geographic area that is under surveillance by camera <b>103</b>.

0109Location <b>307</b> is expressed in latitude and longitude coordinates with a resolution (or confidence) factor, e.g., 10-meter radius. The precision (resolution, confidence, probability distribution) with which location <b>307</b> is known, determined, and expressed depends on the implementation of the embodiment of the surveillance subsystem of video augmentation system <b>100</b>. It should also be noted that the precision with which location <b>307</b> is expressed need not be the same as the precision of the estimated location of the mobile that is typically obtained from wireless network location system <b>131</b>. For example, the resolution of location <b>307</b> can be a lat/long with a 2-meter radius, while the estimated location of mobile <b>305</b> can be a 10-meter by 10-meter square having a certain probability. The discussion below in regard to operation <b>803</b> discusses the relationship between these parameters in more detail.

0110Background landscape <b>320</b> illustratively depicts objects that appear in frame <b>301</b> in the background relative to image <b>303</b>. Background landscape <b>320</b> illustratively depicts a plurality of buildings.

0111Foreground landscape <b>330</b> illustratively depicts objects that appear in frame <b>301</b> in the foreground relative to image <b>303</b>. Foreground landscape <b>330</b> illustratively depicts a tree and a picnic table.

0112It will be clear to those having ordinary skill in the art, after reading the present disclosure, that frame <b>301</b> is merely illustrative of a frame received from a video camera feed that covers a certain geographic region, and that in alternative embodiments, any number of images of persons could appear with any number of backgrounds and foregrounds, wherein each person could carry any number of mobile phones. Some of the later examples illustrated herein are based on a variation of frame <b>301</b> that comprises images of more than one person, but which is not shown here for the sake of simplicity.

0113<figref idref="DRAWINGS">FIG. 3B</figref> depicts illustrative frame <b>311</b> of composite video signal <b>113</b> in accordance with an illustrative embodiment of the present invention. Composite video signal <b>113</b> is generated by data-processing system <b>101</b> according to the methods and operations recited herein, and comprises a plurality of frames including illustrative frame <b>311</b>.

0114Frame <b>311</b> comprises: image <b>303</b>; mobile phone <b>305</b>; tag <b>315</b>; background landscape <b>320</b>; and foreground landscape <b>330</b>. Frame <b>311</b> is based on frame <b>301</b> depicted in <figref idref="DRAWINGS">FIG. 1A</figref>. Image <b>303</b>, mobile phone <b>305</b>, background landscape <b>320</b>, and foreground landscape <b>330</b> are the same as depicted in <figref idref="DRAWINGS">FIG. 1A</figref>. Likewise, location <b>307</b> is the location of the person in image <b>303</b>. According to the present depiction, the coordinates of location <b>307</b> are not displayed in frame <b>311</b>, but it will be clear to those having ordinary skill in the art, after reading the present disclosure, how to construct frame <b>311</b> such that it comprises the coordinates of location <b>307</b>.

0115Tag <b>315</b> comprises a depiction of: the identity of the person in image <b>303</b>, e.g., the person's name; the person's address; and the person's Social Security Number. Tag <b>315</b> is an element of frame <b>311</b> that has been generated by data-processing system <b>101</b> according to the operations recited herein. By positioning tag <b>315</b> proximate to image <b>303</b>, data-processing system <b>101</b> creates an association between the image of the person and tag <b>315</b> including its constituent information that is visible to a human operator. The system also captures the association in a data structure for storage, transmission, and archiving.

0116It will be clear to those having ordinary skill in the art, after reading the present disclosure that frame <b>311</b> and tag <b>315</b> are merely illustrations, and that many other embodiments are possible to represent, within frame <b>311</b>, various aspects of the identity of the person in image <b>303</b>. For example, other personal information might be displayed, including a nickname, a date of birth, etc.—presented in one or more tags. Many variations are possible within the scope of the present invention.

0117<figref idref="DRAWINGS">FIG. 4</figref> depicts illustrative frame <b>401</b> of composite video signal <b>400</b> in accordance with an illustrative embodiment of the present invention. Composite video signal <b>400</b> is generated by data-processing system <b>101</b> according to the methods and operations recited herein and comprises a plurality of frames including illustrative frame <b>401</b>. Frame <b>401</b> is based on frame <b>301</b> depicted in <figref idref="DRAWINGS">FIG. 1A</figref>.

0118Frame <b>401</b> comprises: image <b>303</b>; mobile phone <b>305</b>; background landscape <b>320</b>; foreground landscape <b>330</b>; tag <b>415</b>; and tag <b>417</b>. Image <b>303</b>, mobile phone <b>305</b>, background landscape <b>320</b>, and foreground landscape <b>330</b> are the same as depicted in <figref idref="DRAWINGS">FIG. 1A</figref>. Likewise, location <b>307</b> is the location of the person in image <b>303</b>.

0119Tag <b>415</b> comprises a depiction of: the identity of the person in image <b>303</b> shown here as the person's name (Name-<b>1</b>); the identity of mobile phone <b>305</b> shown here as the telephone number (xxx-xxx-xxxx); and the person's “social” relationship or social network connection to another party identified here by Name-<b>2</b>. Tag <b>415</b> is an element of frame <b>401</b> that has been generated by data-processing system <b>101</b> according to the operations recited herein. By positioning tag <b>415</b> proximate to image <b>303</b>, data-processing system <b>101</b> creates a visible association between the image of the person and tag <b>415</b> including its constituent information that is visible to a human operator. The system also captures the association in a data structure for storage, transmission, and archiving.

0120Tag <b>417</b> comprises information about another party who is communicating with the person in image <b>303</b>, but the other party is not visible in the frame. Tag <b>417</b> comprises: an indication (“communicating with”) that there is an active communication between the person in image <b>303</b> and another party; the identity of the other party's mobile (yyy-yyy-yyyy); an estimate of the location of the other party's mobile; the elapsed time of the active communication; the name that is associated with the other party's mobile, e.g., the subscriber of mobile yyy-yyy-yyyy (Name-<b>2</b>); and a known home address of the other party. Tag <b>417</b> is an element of frame <b>401</b> that has been generated by data-processing system <b>101</b> according to the operations recited herein.

0121<figref idref="DRAWINGS">FIG. 5A</figref> depicts illustrative frame <b>501</b> of composite video signal <b>500</b> according to an illustrative embodiment of the present invention. Composite video signal <b>500</b> is generated by data-processing system <b>101</b> according to the methods and operations recited herein and comprises a plurality of frames including illustrative frame <b>501</b>.

0122Frame <b>501</b> comprises: image <b>303</b>; mobile phone <b>305</b>; tag <b>515</b>; background landscape <b>320</b> (not labeled); and foreground landscape <b>330</b> (not labeled). Frame <b>501</b> is based on frame <b>301</b> depicted in <figref idref="DRAWINGS">FIG. 1A</figref>. Image <b>303</b>, mobile phone <b>305</b>, background landscape <b>320</b>, and foreground landscape <b>330</b> are the same as depicted in <figref idref="DRAWINGS">FIG. 1A</figref>. Likewise, location <b>307</b> (not labeled) is the location of the person in image <b>303</b>.

0123Tag <b>515</b> comprises: a depiction of the identity of the person in image <b>303</b> (Name-<b>1</b>); an identity of mobile phone <b>305</b> (xxx-xxx-xxxx); a familial relationship to another person identified as Name-<b>2</b> but not appearing in this frame (Brother of Name-<b>2</b>); a “communication” relationship to Name-<b>2</b>, having texted Name-<b>2</b> at a certain date and time; a “social” relationship with Name-<b>2</b> on Twitter; a financial relationship to Name-<b>2</b>, having transferred a substantial sum of money to Name-<b>2</b> on a certain date and time. Tag <b>515</b> is an element of frame <b>501</b> that has been generated by data-processing system <b>101</b> according to the operations recited herein. By positioning tag <b>515</b> proximate to image <b>303</b>, data-processing system <b>101</b> creates a visible association between the image of the person and tag <b>515</b> including its constituent information that is visible to a human operator. The system also captures the association in a data structure for storage, transmission, and archiving.

0124<figref idref="DRAWINGS">FIG. 5B</figref> depicts illustrative frame <b>521</b> of composite video signal <b>520</b> according to an illustrative embodiment of the present invention. Composite video signal <b>520</b> is generated by data-processing system <b>101</b> according to the methods and operations recited herein and comprises a plurality of frames including illustrative frame <b>521</b>.

0125Frame <b>521</b> comprises the same information as frame <b>501</b> and is presented here to show an alternative view of how data-processing system <b>101</b> prepares a composite video signal for presentation to users/operators of video augmentation system <b>100</b>. Frame <b>521</b> comprises: sub-frame <b>523</b> and tag <b>525</b>.

0126Sub-frame <b>523</b> is identical in content to frame <b>301</b>.

0127Tag <b>525</b> comprises the same information as tag <b>515</b>.

0128In frame <b>521</b>, tag <b>525</b> and sub-frame <b>523</b> are positioned side-by-side. It will be clear to those having ordinary skill in the art, after reading the present disclosure, how to devise many other variations of frame <b>501</b> and frame <b>521</b>.

0129<figref idref="DRAWINGS">FIG. 6A</figref> depicts a first illustrative frame (<b>601</b>) of composite video signal <b>600</b> according to an illustrative embodiment of the present invention. Composite video signal <b>600</b> is generated by data-processing system <b>101</b> according to the methods and operations recited herein and comprises a plurality of frames including illustrative frame <b>601</b>.

0130Frame <b>601</b> comprises: image <b>303</b>; mobile phone <b>305</b>; background landscape <b>320</b> (not labeled); foreground landscape <b>330</b> (not labeled); image <b>603</b>; mobile phone <b>605</b>; tag <b>615</b>; tag <b>617</b>; and communication icon <b>619</b>. Image <b>303</b>, mobile phone <b>305</b>, background landscape <b>320</b>, and foreground landscape <b>330</b> are the same as depicted in <figref idref="DRAWINGS">FIG. 1A</figref>. Likewise, location <b>307</b> is the location of the person in image <b>303</b>. Frame <b>601</b> is based on a variation of frame <b>301</b> that would show more than one person appearing in the frame.

0131Image <b>603</b> depicts a person in a manner well known in the art. Image <b>603</b> is an image of a person in the geographic region monitored by video camera <b>103</b>.

0132Mobile phone <b>605</b> is (illustratively) a GSM cellular telephone that is carried by the person in image <b>603</b>. Mobile phone <b>605</b> is analogous, though not necessarily identical, to mobile <b>305</b>. Thus, mobile phone <b>605</b> is well known in the art and can be any type of mobile unit, including a cellular telephone (e.g., CDMA, GSM, UMTS, etc.), smartphone (e.g., iPhone, android, Blackberry, etc.), mobile computing tablet (e.g., iTouch, Kindle Fire, iPad, etc.), or other mobile unit. In some embodiments, mobile <b>305</b> operates within a first wireless network that is different than a second wireless network in which mobile <b>605</b> operates.

0133Location <b>607</b> is the location of the person in image <b>603</b>. According to the present depiction, the coordinates of location <b>307</b> and <b>607</b> are not displayed in frame <b>601</b>, but it will be clear to those having ordinary skill in the art, after reading the present disclosure, how to generate frame <b>601</b> such that it comprises the coordinates of location <b>307</b> and/or location <b>607</b>.

0134Tag <b>615</b> comprises: a depiction of the identity of the person in image <b>303</b> (Name-<b>1</b>); an identity of mobile phone <b>305</b> (xxx-xxx-xxxx); an identity of the service provider (wireless network operator) that services mobile <b>305</b> (Service Provider-<b>1</b>); a “social” relationship on Facebook to another person identified as Name-<b>2</b>, e.g., “Facebook Friend”; and a familial relationship with Name-<b>2</b>, e.g., sibling. By positioning tag <b>615</b> proximate to image <b>303</b> as depicted here, data-processing system <b>101</b> creates an association between image <b>303</b> of the first person and tag <b>615</b> including its constituent information that is visible to a human operator. The system also captures the association in a data structure for storage, transmission, and archiving.

0135Tag <b>617</b> comprises: a depiction of the identity of the person in image <b>603</b> (Name-<b>2</b>); an identity of mobile phone <b>605</b> (yyy-yyy-yyyy); an identity of the service provider (wireless network operator) that services mobile <b>605</b> (Service Provider-<b>2</b>); a call origination time; and an elapsed time for the call. By positioning tag <b>617</b> proximate to image <b>603</b> as depicted here, data-processing system <b>101</b> creates an association between image <b>603</b> of the second person and tag <b>617</b> including its constituent information that is visible to a human operator. The system also captures the association in a data structure for storage, transmission, and archiving.

0136Communication icon <b>619</b> depicts the fact that an active communication is occurring between mobile <b>305</b> and mobile <b>605</b>. Icon <b>619</b> indicates that a voice call is connected and active. It will be clear to those having ordinary skill in the art, after reading the present disclosure, when and how to depict a communication icon between two parties that are actively texting each other; or are actively IM'ing each other; or are engaged in another collective communication such as an online application, an online game, etc. For example, communication icon <b>619</b> could be depicted when mobile <b>305</b> and mobile <b>605</b> exchange text messages within a predetermined time threshold such as within 1 minute.

0137<figref idref="DRAWINGS">FIG. 6B</figref> depicts a second illustrative frame (<b>621</b>) of composite video signal <b>600</b> according to an illustrative embodiment of the present invention. Composite video signal <b>600</b> is generated by data-processing system <b>101</b> according to the methods and operations recited herein and comprises a plurality of frames including illustrative frame <b>621</b>. Frame <b>621</b> is based on a variation of frame <b>301</b> that would show more than one person appearing in the frame.

0138Frame <b>621</b> comprises the same elements as frame <b>601</b>, but frame <b>621</b> depicts image <b>603</b> and mobile <b>605</b> at a different location, i.e., at location <b>623</b>. In other words, in frame <b>621</b>, the person in image <b>603</b> has moved (from location <b>607</b> in frame <b>601</b>) to location <b>623</b>.

0139<figref idref="DRAWINGS">FIG. 6C</figref> depicts a third illustrative frame (<b>631</b>) of composite video signal <b>600</b> according to an illustrative embodiment of the present invention. Composite video signal <b>600</b> is generated by data-processing system <b>101</b> according to the methods and operations recited herein and comprises a plurality of frames including illustrative frame <b>631</b>. Frame <b>631</b> is based on a variation of frame <b>301</b> that would show more than one person appearing in the frame.

0140Frame <b>631</b> comprises: sub-frame <b>633</b>; tag <b>635</b>; tag <b>637</b>; and tag <b>639</b>.

0141Sub-frame <b>633</b> is similar, but not identical, to frame <b>601</b> in <figref idref="DRAWINGS">FIG. 6A</figref>. Sub-frame <b>633</b> comprises: image <b>303</b>; background landscape <b>320</b> (not labeled); foreground landscape <b>330</b> (not labeled); image <b>603</b>; and communication icon <b>619</b> (not labeled). Image <b>303</b>, mobile phone <b>305</b> (not shown), background landscape <b>320</b>, foreground landscape <b>330</b>, image <b>603</b>, terminal <b>605</b> (not shown), and communication icon <b>619</b> are the same as depicted/described earlier. Likewise, location <b>307</b> (not labeled) is the location of the person in image <b>303</b>; location <b>607</b> (not labeled) is the location of the person in image <b>603</b>.

0142Tag <b>635</b> comprises: the number “1” illustratively labeling the person in image <b>303</b> as the first party; and information about the first party, including name, the identity of mobile <b>305</b> (not shown in frame <b>631</b> but associated with the first party), birthdate, birthplace, passport country and passport number, U.S. visa number, bank and checking account number, bank and credit card number.

0143Tag <b>637</b> comprises: the number “2” illustratively labeling the person in image <b>603</b> as the second party; and information about the second party, including name, the identity of mobile <b>605</b> (not shown in frame <b>631</b> but associated with the second party), bank and credit card number, a list of relationships between the second party and the first party, including a financial relationship in which the second party received $25,000 at a certain date/time, and a “communication” relationship in which the second party and the first party had a voice call at a certain date/time.

0144Tag <b>639</b> comprises: an indication that the person in image <b>303</b> and the person in image <b>603</b> are engaged in a mobile-to-mobile call initiated at a certain time and having a certain elapsed time.

0145In regard to the illustrative frames depicted in <figref idref="DRAWINGS">FIGS. 3 through 6C</figref>, it will be clear to those having ordinary skill in the art, after reading the present disclosure, that the depicted frames and their constituent elements are merely illustrative and that many other variations are possible within the scope of the present invention.

0146<figref idref="DRAWINGS">FIG. 7</figref> depicts a flowchart of the salient operations of illustrative method <b>700</b> in accordance with an illustrative embodiment of the present invention. In general, data-processing system <b>101</b> performs the operations of method <b>700</b> and its sub-operations, unless expressly recited otherwise. It will be understood that, because data-processing system <b>101</b> is a computer, the recited operations executed by data-processing system <b>101</b> (e.g., processing, determining, estimating, retrieving, generating, polling, etc.) comprise a plurality of data processing operations such as calculating, computing, encoding, decoding, transmitting, receiving, comparing, etc.—as appropriate to process data and produce a useful result according to the disclosed methods herein.

0147At operation <b>701</b>, data-processing system <b>101</b> receives a first video signal, e.g., video signal <b>111</b> from video camera <b>103</b>, in a manner well known in the art.

0148Operation <b>703</b> is a decision point asking whether facial recognition can recognize the identity of one or more persons in the first video signal, e.g., can facial recognition system <b>123</b> recognize the identity of the person in image <b>303</b>. In other words, is a given image of a person sufficient for the system to recognize and identify the person? If not, control passes to operation <b>705</b>. If yes, control passes to operation <b>707</b>.

0149At operation <b>705</b>, for the one or more persons appearing in a frame who cannot be recognized via facial recognition, data-processing system <b>101</b> performs an analysis that is based on the identity of a mobile phone that is inferred to be co-located with the person, as described in more detail below.

0150At operation <b>707</b>, for the one or more persons who can be recognized via facial recognition, data-processing system <b>101</b> performs an analysis based on the facial recognition of the image of the respective person, as described in more detail below.

0151At operation <b>709</b>, based on the preceding analysis (operation <b>705</b> and/or operation <b>707</b>) data-processing system <b>101</b> generates a second video signal that is a composite video, and further transmits, displays, and archives the composite video signal. This operation is described in more detail in <figref idref="DRAWINGS">FIG. 14</figref> below and in the accompanying text.

0152It will be clear to those having ordinary skill in the art, after reading the present disclosure, how to generate and execute method <b>700</b> according to the present invention. It will be clear to those having ordinary skill in the art that some of the operations and/or sub-operations of method <b>700</b> can be performed in parallel with each other or in a different order than those depicted herein without departing from the scope of the present invention.

0153<figref idref="DRAWINGS">FIG. 8</figref> depicts a flowchart of the salient operations of operation <b>705</b> in accordance with an illustrative embodiment of the present invention.

0154At operation <b>801</b>, data-processing system <b>101</b> processes the first video signal, which carries a first video frame, e.g., video signal <b>111</b> carrying video frame <b>301</b>. The first video frame comprises an image of a first person at a first location and an image of a second person at a second location, e.g., image <b>303</b> at location <b>307</b> and image <b>603</b> at location <b>607</b> (in a variation of frame <b>301</b> that would show more than one person appearing in the frame). As previously noted, within the geographic area under surveillance by video camera <b>103</b>, every location is known or determined by data-processing system <b>101</b>; therefore, when data-processing system <b>101</b> detects an image of a person in the frame of a video signal, data-processing system <b>101</b> determines a location for that person. Thus, illustratively, data-processing system <b>101</b> determines the location of the person in image <b>303</b>, i.e., data-processing system <b>101</b> determines location <b>307</b>.

0155At operation <b>803</b>, data-processing system <b>101</b> estimates the identity of a first mobile phone based on the location of the first person, e.g., estimating the identity of mobile <b>305</b> based on location <b>307</b> for the person in image <b>303</b>. Estimating the identity of a mobile is based in part on the estimated location of the mobile, which is available (illustratively) from wireless network location system <b>131</b>.

0156In some illustrative embodiments, data-processing system <b>101</b> performs the estimation of the mobile's identity. For example, in operation <b>801</b>, data-processing system <b>101</b> determined a location for the image of the person, i.e., data-processing system <b>101</b> determined location <b>307</b>. In the present operation, data-processing system <b>101</b> requests location data from one or more wireless network location systems <b>131</b> that serve the geographic area under surveillance by video camera <b>103</b>—particularly requesting location data that corresponds to location <b>307</b>. Equipment deployed in the polled wireless network(s), e.g., wireless network location systems <b>131</b>, analyzes data that are available from network infrastructure and from mobile units to estimate a location for each mobile unit in the network. For example, the Wireless Location Signature (WLS™) service from Polaris Wireless, Inc. performs ongoing mass location analysis that provides relatively high-resolution location data for large numbers of mobile units in a given geographic area. Location information from wireless network location systems <b>131</b> is transmitted to data-processing system <b>101</b>, which then estimates which, if any, of the mobiles in the surveillance area corresponds to the location of the person in the surveillance frame, e.g., location <b>307</b>. Thus, data-processing system <b>101</b> determines whether any mobiles can be inferred to be at location <b>307</b>. When a sufficient match or correlation exists as between mobile location estimate(s) and location <b>307</b>, data-processing system <b>101</b> infers that the person at location <b>307</b> carries the matching mobile. The identity of the mobile, e.g., the mobile telephone number, is known from the location data received from wireless network location system <b>131</b>. Because the identity of the mobile is known, data-processing system <b>101</b> associates the identity of the mobile with the person in image <b>303</b> appearing at location <b>307</b>. In this way, illustratively, data-processing system <b>101</b> estimates the identity of mobile <b>305</b> based on location <b>307</b> of the person in image <b>303</b>.

0157As noted above in describing location <b>307</b> in respect to <figref idref="DRAWINGS">FIG. 3A</figref>, the precision of location <b>307</b> can differ substantially from the precision of the estimate of the location of the mobile as received from wireless network location systems <b>131</b>. Nevertheless, data-processing system <b>101</b> comprises stored-program software comprising algorithms that compute the necessary correlations as between the estimated location of the mobile and the estimated location of the person in the image, e.g., location <b>307</b> to determine whether a given mobile corresponds to location <b>307</b>. The confidence parameters to determine when a correlation is sufficient for data-processing system <b>101</b> to infer that a given mobile is co-located with a given person are left to the discretion of the implementers of the illustrative embodiment.

0158This scenario is merely illustrative of one way to estimate the identity of a first mobile phone based on the location of the first person in the video frame, but it will be clear to those having ordinary skill in the art, after reading the present disclosure, how to make and use alternative embodiments that perform this operation within the scope of the present invention.

0159At operation <b>805</b>, as described in detail in operation <b>803</b>, the identity of a second mobile phone is estimated based on the location of the second person in the frame. Thus, illustratively, data-processing system <b>101</b> estimates the identity of mobile <b>605</b> based on location <b>607</b> of the person in image <b>603</b>.

0160At operation <b>807</b>, data-processing system <b>101</b> receives, retrieves, and estimates supplemental information based on the identity of the first and the second mobile phones. This operation is described in more detail below.

0161<figref idref="DRAWINGS">FIG. 9</figref> depicts a flowchart of the salient operations of illustrative operation <b>807</b> in accordance with an illustrative embodiment of the present invention.

0162At operation <b>901</b>, data-processing system <b>101</b> polls data sources and data systems for information about the first and second mobile phone, whose identities were estimated in earlier operations. The data sources and data systems being polled are internal and/or external to video augmentation system <b>100</b>. In an illustrative embodiment, data-processing system <b>101</b> polls each of data store <b>109</b>, data store <b>121</b>, communication reporting system <b>125</b>, financial reporting system <b>127</b>, and social network system <b>129</b> for information about mobile <b>305</b> and mobile <b>605</b>.

0163At operation <b>903</b>, data-processing system <b>101</b> receives the polling results. Illustratively, data-processing system <b>101</b> receives information comprising: <ul id="ul0017" list-style="none"><li id="ul0017-0001" num="0000"><ul id="ul0018" list-style="none"><li id="ul0018-0001" num="0164">a currently active communication involving mobile <b>305</b>—received from communication reporting system <b>125</b>;</li><li id="ul0018-0002" num="0165">a financial transaction involving mobile <b>305</b>—received from financial reporting system <b>127</b>;</li><li id="ul0018-0003" num="0166">surveillance information previously collected and previously saved about mobile <b>305</b>—received from data store <b>109</b> and data store <b>121</b>; and</li><li id="ul0018-0004" num="0167">a call record of a past call involving mobile <b>305</b>—received from communication reporting system <b>125</b>.</li></ul></li></ul>

0168At operation <b>905</b> (described in more detail in the next figure) data-processing system <b>101</b> estimates, based on the information received in the previous operation, additional information such as a relationship, an identity, and/or an association. Illustratively, data-processing system <b>101</b> estimates the following examples of “additional information:” <ul id="ul0019" list-style="none"><li id="ul0019-0001" num="0000"><ul id="ul0020" list-style="none"><li id="ul0020-0001" num="0169">an identity of the person in image <b>303</b> from wireless network records indicating the identity of the subscriber who owns mobile <b>305</b>, e.g., Name-<b>1</b>;</li><li id="ul0020-0002" num="0170">a “communication” relationship as between the person associated with mobile <b>305</b> (Name-<b>1</b>) and another person who is associated with a terminal that frequently communicates with mobile <b>305</b>;</li><li id="ul0020-0003" num="0171">a familial relationship as between the subscriber who owns mobile <b>305</b> (Name-<b>1</b>) and a known sibling of the subscriber; and</li><li id="ul0020-0004" num="0172">an association between the image of a person, e.g., image <b>303</b>, and the identity of the person.</li></ul></li></ul>

0173In some embodiments, after operation <b>905</b> is executed, control loops back to operation <b>901</b>, wherein further polling is executed based on the received information in operation <b>903</b> and the “additional information” estimated in operation <b>905</b>. In this way, more relevant data can be gathered and processed by data-processing system <b>101</b>. When sufficient data have been gathered, control passes to operation <b>907</b>.

0174At operation <b>907</b>, data-processing system <b>101</b> generates “supplemental information” from (i) the received information in operations <b>901</b>-<b>903</b> and (ii) the estimated additional information in operation <b>907</b>. In effect, this operation generates a superset of information that has been received, retrieved, inferred, estimated, and/or calculated by data-processing system <b>101</b> in reference to the person(s) and mobile unit(s) of interest.

0175<figref idref="DRAWINGS">FIG. 10</figref> depicts a flowchart of the salient operations of illustrative operation <b>905</b> in accordance with an illustrative embodiment of the present invention.

0176At operation <b>1001</b>, data-processing system <b>101</b> estimates the identity of the first person based on the identity of the first mobile phone, e.g., data-processing system <b>101</b> estimates the identity of the person in image <b>303</b> based on the identity of mobile <b>305</b>. Typically the identity of a person is expressed as a proper name, but in some embodiments it is expressed as a nickname or street name, or as some other moniker.

0177At operation <b>1003</b>, data-processing system <b>101</b> estimates the identity of the second person based on the identity of the second mobile phone, e.g., data-processing system <b>101</b> estimates the identity of the person in image <b>603</b> based on the identity of mobile <b>605</b>. Additionally, it should be noted that an image of the second person need not appear in a video frame; rather, data-processing system <b>101</b> estimates the identity of a mobile phone that is not “visible” in a frame based on its relationship to a person and/or mobile that appears in the frame. For example, when data-processing system <b>101</b> determines that mobile <b>305</b> is engaged in an active call with another terminal, e.g., mobile-3, data-processing system <b>101</b> can estimates the identity of the user of mobile-3 by retrieving information about the name of the subscriber who owns mobile-3. The image of the mobile-3 subscriber need not be visible in the surveillance area covered by video camera <b>103</b>.

0178At operation <b>1005</b>, data-processing system <b>101</b> generates an association between the image of the first person and the identity that was estimated for the first person, e.g., data-processing system <b>101</b> generates an association between image <b>303</b> and Name-<b>1</b>.

0179At operation <b>1007</b>, data-processing system <b>101</b> estimates a relationship between the first person and the second person, e.g., a familial relationship, a telecommunications event, a social network connection, a financial transaction, a common trait, etc. For example, based on information retrieved from a public records database, data-processing system <b>101</b> estimates that Name-<b>1</b> is a sibling of Name-<b>2</b>. For example, based on information received from communications reporting system <b>125</b> reporting that mobile <b>305</b> and mobile <b>605</b> frequently communicate by voice call and/or text messaging, data-processing system <b>101</b> estimates that there is a relationship between Name-<b>1</b> (subscriber of mobile <b>305</b>) and Name-<b>2</b> (subscriber of mobile <b>605</b>). For example, based on information received from social network system <b>129</b>, data-processing system <b>101</b> estimates that there is a relationship between Name-<b>1</b> and Name-<b>2</b>, because they are on each other's list of connections or “friends” on the social network system. For example, based on information received from financial reporting system <b>127</b>, data-processing system <b>101</b> estimates that there is a relationship between Name-<b>1</b> and Name-<b>2</b>, because funds have been transferred from a bank account belonging to Name-<b>1</b> to a bank account belonging to Name-<b>2</b>. For example, based on information received from a criminal records database, data-processing system <b>101</b> estimates that there is a relationship between Name-<b>1</b> and Name-<b>2</b> based on a common trait, e.g., they both served time in the same prison over a certain period of time, or they both called the same telephone number at approximately the same time on a number of occasions. It will be clear to those having ordinary skill in the art, after reading the present disclosure, that numerous other relationships can be inferred in this operation based on information that data-processing system <b>101</b> receives from a variety of data sources.

0180At operation <b>1009</b>, data-processing system <b>101</b> estimates the identity of another (third) telecommunications terminal (i.e., other than mobile <b>305</b> and mobile <b>605</b>) based on information about the first mobile, such as a communication involving mobile <b>305</b>, a transaction involving mobile <b>305</b>, etc. Importantly, the third telecommunications terminal need not be wireless, and can be any wired or mobile phone. In the illustrative embodiment, the third terminal is a wired telephone. Illustratively, data-processing system <b>101</b> estimates the identity of terminal-<b>3</b> (e.g., its telephone number) based on information indicating that mobile <b>305</b> is currently engaged in a voice call with terminal-<b>3</b>.

0181At operation <b>1011</b>, data-processing system <b>101</b> estimates the identity of another (third) person based on the identity of the third terminal (terminal-<b>3</b>). For example, data-processing system <b>101</b> estimates the identity of the user of terminal-<b>3</b> by retrieving information about the name of the subscriber who owns terminal-<b>3</b>.

0182At operation <b>1013</b>, in a manner similar to operation <b>1007</b>, data-processing system <b>101</b> estimates a relationship between the first person and the third person, e.g., a familial relationship, a telecommunications event, a social network connection, a financial transaction, a common trait, etc. As described in more detail in operation <b>1007</b>, the information that forms the basis for estimating the relationship can be based on the identity of a mobile or on the identity of its known subscriber/owner.

0183It will be clear to those having ordinary skill in the art, after reading the present disclosure, how to define and estimate additional information that is useful and appropriate according to operation <b>905</b> and the principles of the present invention.

0184<figref idref="DRAWINGS">FIG. 11</figref> depicts a flowchart of the salient operations of illustrative operation <b>707</b> in accordance with an illustrative embodiment of the present invention.

0185At operation <b>1101</b>, as described in detail in operation <b>801</b>, data-processing system <b>101</b> processes the first video signal, which carries a first video frame, e.g., video signal <b>111</b> carrying video frame <b>301</b>.

0186At operation <b>1103</b>, data-processing system <b>101</b> estimates the identity of the first person and/or the second person based on facial recognition applied to the image of the respective person. Thus, illustratively, image recognition is applied to image <b>303</b> and to image <b>603</b>. Based on information obtained from facial recognition system <b>123</b>, data-processing system <b>101</b> estimates the identity of the person in image <b>303</b> (Name-<b>1</b>) and also estimates the identity of the person in image <b>603</b> (Name-<b>2</b>). In an illustrative embodiment, the facial recognition analysis is executed at facial recognition system <b>123</b> and results are reported to data-processing system <b>101</b>. In another illustrative embodiment, the facial recognition analysis is executed by data-processing system <b>101</b> based on data and parameters received from facial recognition system <b>123</b>. When facial recognition is applied to an image of a person appearing in a video and the person's identity is thus recognized, data-processing system <b>101</b> generates an association between the identity according to the facial recognition and the identity of the co-located mobile phone. This association is displayed and also stored for archival purposes. This association can be critical when there is no personal identification available for a mobile phone such as a pre-paid mobile.

0187At operation <b>1105</b>, as described in detail in operations <b>803</b> and <b>805</b>, data-processing system <b>101</b> estimates the identity of the first and the second mobile phones based on the respective locations of the first and second persons.

0188At operation <b>1107</b>, data-processing system <b>101</b> generates an association between the identity of the first person (operation <b>1103</b>) and the identity of the first mobile phone (operation <b>1105</b>).

0189At operation <b>1109</b>, data-processing system <b>101</b> receives, retrieves, and estimates supplemental information based on the estimated identities from operations <b>1103</b> and <b>1105</b>. This operation is described in more detail below.

0190<figref idref="DRAWINGS">FIG. 12</figref> depicts a flowchart of the salient operations of illustrative operation <b>1109</b> in accordance with an illustrative embodiment of the present invention.

0191At operation <b>1201</b>, data-processing system <b>101</b> polls data sources and data systems for information about the first person and second person, whose identities (Name-<b>1</b> and Name-<b>2</b>) were estimated in earlier operations. The data sources and data systems being polled are internal and/or external to video augmentation system <b>100</b>. In an illustrative embodiment, data-processing system <b>101</b> polls each of data store <b>109</b>, data store <b>121</b>, communication reporting system <b>125</b>, financial reporting system <b>127</b>, and social network system <b>129</b> for information about Name-<b>1</b> and Name-<b>2</b>.

0192At operation <b>1203</b>, as described in more detail at operation <b>901</b>, data-processing system <b>101</b> polls data sources and data systems for information about the first and second mobile phone, whose identities were estimated in earlier operations.

0193At operation <b>1205</b>, in a manner similar to operation <b>903</b>, data-processing system <b>101</b> receives the polling results from polling operations <b>1201</b> and <b>1203</b>. The polling results comprise information such as, illustratively: <ul id="ul0021" list-style="none"><li id="ul0021-0001" num="0000"><ul id="ul0022" list-style="none"><li id="ul0022-0001" num="0194">a currently active communication involving Name-<b>2</b>—received from communication reporting system <b>125</b>;</li><li id="ul0022-0002" num="0195">a currently active communication involving mobile <b>305</b>—received from communication reporting system <b>125</b>;</li><li id="ul0022-0003" num="0196">a financial transaction involving Name-<b>1</b>—received from financial reporting system <b>127</b>;</li><li id="ul0022-0004" num="0197">a financial transaction involving mobile <b>305</b>—received from financial reporting system <b>127</b>;</li><li id="ul0022-0005" num="0198">a “social” connection involving Name-<b>1</b> and Name-<b>2</b>—received from social network system <b>129</b>;</li><li id="ul0022-0006" num="0199">personal information such as height and weight and distinguishing facial features for Name-<b>1</b>—received from a criminal records database;</li><li id="ul0022-0007" num="0200">surveillance information previously collected about mobile <b>305</b>, Name-<b>1</b>, and Name-<b>2</b>—received from data store <b>109</b> and data store <b>121</b>,</li><li id="ul0022-0008" num="0201">personal information such as birthdate, birthplace, address, etc.—received from a public records database; and</li><li id="ul0022-0009" num="0202">a call record of a past call involving mobile <b>305</b>—received from communication reporting system <b>125</b>.</li></ul></li></ul>

0203At operation <b>1207</b> (described in more detail in the next figure) data-processing system <b>101</b> estimates additional information such as a relationship, an identity, and an association, based on the information received in the previous operation. Operation <b>1207</b> is similar to operation <b>905</b> described above.

0204In some embodiments, after operation <b>1207</b> is executed, control loops back to operation <b>1201</b>, wherein further polling is executed based on the received information in operation <b>1205</b> and the “additional information” estimated in operation <b>1207</b>. In this way, data-processing system <b>101</b> gathers more relevant data. When sufficient data has been gathered, control passes to operation <b>1209</b>.

0205At operation <b>1209</b>, in a manner similar to operation <b>907</b>, data-processing system <b>101</b> generates “supplemental information” from (i) the received information in operations <b>1201</b>-<b>1205</b> and (ii) the estimated additional information in operation <b>1207</b>. In effect, this operation generates a superset of information that has been received, retrieved, inferred, estimated, and/or calculated by data-processing system <b>101</b> in reference to the person(s) and mobile unit(s) of interest, e.g., Name-<b>1</b> (image <b>303</b>), Name-<b>2</b> (image <b>603</b>), mobile <b>305</b>, and mobile <b>605</b>.

0206<figref idref="DRAWINGS">FIG. 13</figref> depicts a flowchart of the salient operations of illustrative operation <b>1207</b> in accordance with an illustrative embodiment of the present invention.

0207At operation <b>1301</b>, which is the same as operation <b>1001</b>, data-processing system <b>101</b> estimates the identity of the first person based on the identity of the first mobile phone (e.g., mobile <b>305</b>).

0208At operation <b>1303</b>, data-processing system <b>101</b> compares (i) the identity of the first person estimated on the basis of the facial recognition operation <b>1103</b> and (ii) the identity of the first person estimated on the basis of the identity of the first mobile phone (e.g., mobile <b>305</b>). When the comparison produces a match or probable match, there is additional confidence that the user of mobile <b>305</b> is actually present at the scene under surveillance. The added confidence may enable additional enforcement action to be taken against the identified person.

0209At operation <b>1305</b>, which is the same as operation <b>1003</b>, data-processing system <b>101</b> estimates the identity of the second person based on the identity of the second mobile phone (e.g., mobile <b>605</b>).

0210At operation <b>1307</b>, which is the same as operation <b>1007</b>, data-processing system <b>101</b> estimates a relationship between the first person (Name-<b>1</b>) and the second person (Name-<b>2</b>).

0211At operation <b>1309</b>, which is the same as operation <b>1009</b>, data-processing system <b>101</b> estimates the identity of another (third) telecommunications terminal, wireless or otherwise, based on information about the first terminal (e.g., mobile <b>305</b>) and/or information about the first person (e.g., Name-<b>1</b>) such as a communication, a transaction, surveillance data, etc.

0212At operation <b>1311</b>, which is similar to operation <b>1011</b>, data-processing system <b>101</b> estimates the identity of another (third) person based on the identity of the third telecommunications terminal and/or based on the identity of the first person (Name-<b>1</b>).

0213At operation <b>1313</b>, which is the same as operation <b>1013</b>, data-processing system <b>101</b> estimates a relationship between the first person and the third person.

0214<figref idref="DRAWINGS">FIG. 14</figref> depicts a flowchart of the salient operations of illustrative operation <b>709</b> in accordance with an illustrative embodiment of the present invention. Operation <b>709</b> follows operations <b>705</b> and <b>707</b>, and is generally directed at generating a composite video signal that is transmitted, displayed and archived.

0215At operation <b>1401</b>, data-processing system <b>101</b> selects the information that is to be displayed from the set of information that was received, retrieved, estimated, calculated, and generated in earlier operations. This set of information comprises the images of the persons in the video frame, the locations of the persons/mobiles in the video frames, the identities of the terminals, the identities of the persons, and other supplemental information that was inferred, estimated, calculated, and generated by the preceding operations. The selection process of operation <b>1401</b> depends on the choices of the implementers.

0216At operation <b>1403</b>, data-processing system <b>101</b> generates each composite video frame for the composite video signal that is to be transmitted by data-processing system <b>101</b>, e.g., composite video signal <b>113</b>. Examples of composite video frames that are generated in the present operation are illustrated in <figref idref="DRAWINGS">FIGS. 3B through 6C</figref>, e.g., frame <b>311</b>, frame <b>401</b>, frame <b>501</b>, frame <b>521</b>, frame <b>601</b>, frame <b>621</b>, and frame <b>631</b>. Each composite video frame is based on a first frame in the original incoming video signal, e.g., frame <b>301</b> in video signal <b>111</b>. Thus, the original plain video feed is transformed and augmented with supplemental information that is advantageous over the plain contents of the original video signal such as video signal <b>111</b>.

0217The generating process in the present operation involves compositing the incoming frame with supplemental information that augments the original. For example, in frame <b>311</b>, frame <b>301</b> has been augmented with tag <b>315</b>, which provides information about the person in image <b>303</b>. For example, in frame <b>401</b>, frame <b>301</b> has been augmented with tag <b>415</b> and tag <b>417</b>. Tag <b>415</b> provides information about the person in image <b>303</b>. Tag <b>417</b> provides information about another party who is currently communicating with the person in image <b>303</b>. Likewise, for the remaining illustrative composite frames. Selecting what information to show and formatting the information that is to be augmented into the composite video frame remains at the discretion of the implementers of an embodiment of the present invention. For example, “ticker-tape” formatting might be appropriate for some information.

0218At operation <b>1405</b>, data-processing system <b>101</b> generates a composite video signal that carries the composite frame generated in the preceding operation. Generating a video signal that comprises one or more frames is well known in the art.

0219At operation <b>1407</b>, data-processing system <b>101</b> transmits the composite video signal to a display unit, e.g., display <b>107</b>. This operation is well known in the art.

0220At operation <b>1409</b>, each composite frame is displayed as part of the composite video signal received at the display unit, e.g., display <b>107</b>. This operation is well known in the art.

0221At operation <b>1411</b>, data-processing system <b>101</b> transmits the composite video signal to a data store, e.g., data store <b>109</b>, data store <b>121</b>, for archiving and for later retrieval. In some embodiments all or some of the supplemental data is also transmitted for archiving, but not in the form of a video signal. For example, original video signal <b>111</b> is archived in data store <b>109</b>. For example, the associations generated by data-processing system <b>101</b> are archived in data store <b>109</b>, etc. In some embodiments, a database is constructed (in electronic form) from the various data that have been received, inferred, estimated, calculated, and generated by data-processing system <b>101</b>. What particular data are selected and what formats and data structures are used for archiving and/or database construction remains at the discretion of the implementers of the embodiment of the present invention. Archiving data is well known in the art. It should be noted that data can also be transmitted to and stored by other systems that are in communication with video augmentation system <b>100</b>, such as systems <b>123</b>-<b>131</b>, and other systems recited herein, such as a surveillance system, etc.

0222At operation <b>1413</b>, the receiving data store and/or system archives the data received from data-processing system <b>101</b>, e.g., data store <b>109</b>, data store <b>121</b>, systems <b>123</b>-<b>131</b>, etc. This operation is well known in the art.

0223It is to be understood that the disclosure teaches only some examples of illustrative embodiments and that many variations of the invention can easily be devised by those skilled in the art after reading this disclosure. The scope of the present invention is to be determined by the following claims.