Methods and systems for detection of anomalous motion in a video stream and for creating a video summary
Summary by NHIP
Video Anomaly Detection Method
The method obtains motion indicators for video samples and calculates anomaly states for sequential, non-overlapping time windows. It estimates parameters using data from preceding windows and determines the current state by comparing indicators against those estimates.
Claim Score by NHIP
Abstract
A computer-implemented method, comprising: obtaining motion indicators for a plurality of samples of a video stream; obtaining an anomaly state for each of a plurality of time windows of the video stream, each of the time windows spanning a subset of the samples, by (i) obtaining estimated statistical parameters for the given time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the given time window and (ii) determining the anomaly state for the given time window based on the plurality of motion indicators obtained for the samples in the given time window and the estimated statistical parameters; and processing the video stream based on the anomaly state for various ones of the time windows.
Term
13.1 yearsleft in the term
Expires 7 November 2039.
- Priority
- Filed
- Granted
- Today
- Expires
36 claims: 3 independent, 33 dependent
- 1Broadest claimClaim Score 57, broad(NHIP)A computer-implemented method, comprising:obtaining motion indicators for a plurality of samples of a video stream;obtaining an anomaly state for a given time window of a plurality of time windows of the video stream, each of the time windows spanning a subset of the samples, by: obtaining estimated statistical parameters for the given time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the given time window;and determining the anomaly state for the given time window based on the plurality of motion indicators obtained for the samples in the given time window and the estimated statistical parameters;and processing the video stream based on the anomaly state for various ones of the time windows.
- 30A non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes:obtaining motion indicators for a plurality of samples of a video stream;obtaining an anomaly state for a given time window of a plurality of time windows of the video stream, each of the time windows spanning a subset of the samples, by: obtaining estimated statistical parameters for the given time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the given time window;and determining the anomaly state for the given time window based on the plurality of motion indicators obtained for the samples in the given time window and the estimated statistical parameters;and processing the video stream based on the anomaly state for various ones of the time windows.
- 31A video management system, comprising:a memory storing computer-readable instructions;an input/output interface;and a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: obtaining a video stream from the input/output interface or the memory;obtaining motion indicators for a plurality of samples of the video stream;obtaining an anomaly state for a given time window of a plurality of time windows of the video stream, each of the time windows spanning a subset of the samples, by: obtaining estimated statistical parameters for the given time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the given time window;and determining the anomaly state for the given time window based on the plurality of motion indicators obtained for the samples in the given time window and the estimated statistical parameters;and processing the video stream based on the anomaly state for various ones of the time windows;outputting a result of the processing to the input/output interface or to the memory.
Independent claims3
193 paragraphs in 6 sections, as filed
CROSS-REFERENCE TO RELATED APPLICATIONS
The present application claims the benefit of U.S. Provisional Patent Application Ser. No. 62/756,645, filed Nov. 7, 2018; U.S. Provisional Patent Application Ser. No. 62/796,734, filed Jan. 25, 2019; and U.S. Provisional Patent Application Ser. No. 62/928,531, filed Oct. 31, 2019; all of the aforementioned applications are hereby incorporated by reference herein.
FIELD
The present disclosure relates generally to video processing and, more particularly, to methods and systems for detection of anomalous motion in a video stream and for creating a video summary from one or more video streams.
BACKGROUND
A video stream often contains some image data of greater importance and other image data of lesser importance to its viewer. This is especially true in the case of security cameras, which contain image data of lesser interest to the viewer for the vast majority of their “on” time. Reviewing live or recorded footage from one or multiple cameras can therefore be a tedious process, which may result in errors if the viewer loses concentration. A system that could assist the viewer in effectively gaining access to more relevant video footage would be welcomed by the industry.
SUMMARY
According to a broad aspect of the disclosure, there is provided a computer-implemented method, comprising: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0005">obtaining motion indicators for a plurality of samples of a video stream;</li><li id="ul0002-0002" num="0006">obtaining an anomaly state for each of a plurality of time windows of the video stream, each of the time windows spanning a subset of the samples, by: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0007">obtaining estimated statistical parameters for the given time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the given time window; and</li><li id="ul0003-0002" num="0008">determining the anomaly state for the given time window based on the plurality of motion indicators obtained for the samples in the given time window and the estimated statistical parameters; and</li></ul></li><li id="ul0002-0003" num="0009">processing the video stream based on the anomaly state for various ones of the time windows.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes: <ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0000"><ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0011">obtaining motion indicators for a plurality of samples of a video stream;</li><li id="ul0005-0002" num="0012">obtaining an anomaly state for each of a plurality of time windows of the video stream, each of the time windows spanning a subset of the samples, by: <ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0013">obtaining estimated statistical parameters for the given time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the given time window; and</li><li id="ul0006-0002" num="0014">determining the anomaly state for the given time window based on the plurality of motion indicators obtained for the samples in the given time window and the estimated statistical parameters; and</li></ul></li><li id="ul0005-0003" num="0015">processing the video stream based on the anomaly state for various ones of the time windows.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a video management system, comprising: <ul id="ul0007" list-style="none"><li id="ul0007-0001" num="0000"><ul id="ul0008" list-style="none"><li id="ul0008-0001" num="0017">a memory storing computer-readable instructions;</li><li id="ul0008-0002" num="0018">an input/output interface; and a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: <ul id="ul0009" list-style="none"><li id="ul0009-0001" num="0019">obtaining a video stream from the input/output interface or the memory;</li><li id="ul0009-0002" num="0020">obtaining motion indicators for a plurality of samples of the video stream;</li><li id="ul0009-0003" num="0021">obtaining an anomaly state for a given time window of a plurality of time windows of the video stream, each of the time windows spanning a subset of the samples, by: <ul id="ul0010" list-style="none"><li id="ul0010-0001" num="0022">obtaining estimated statistical parameters for the given time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the given time window; and</li><li id="ul0010-0002" num="0023">determining the anomaly state for the given time window based on the plurality of motion indicators obtained for the samples in the given time window and the estimated statistical parameters; and</li></ul></li><li id="ul0009-0004" num="0024">processing the video stream based on the anomaly state for various ones of the time windows;</li><li id="ul0009-0005" num="0025">outputting a result of the processing to the input/output interface or to the memory.</li></ul></li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a computer-implemented method, comprising: <ul id="ul0011" list-style="none"><li id="ul0011-0001" num="0000"><ul id="ul0012" list-style="none"><li id="ul0012-0001" num="0027">receiving a plurality of video streams from a plurality of cameras;</li><li id="ul0012-0002" num="0028">obtaining an anomaly state associated with each of the video streams, the anomaly state indicating whether an associated portion of the respective video stream exhibits a statistically abnormal change in motion; and</li><li id="ul0012-0003" num="0029">displaying a subset of the video streams on an output device as a function of the anomaly state associated with each of the video streams.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes: <ul id="ul0013" list-style="none"><li id="ul0013-0001" num="0000"><ul id="ul0014" list-style="none"><li id="ul0014-0001" num="0031">receiving a plurality of video streams from a plurality of cameras;</li><li id="ul0014-0002" num="0032">obtaining an anomaly state associated with each of the video streams, the anomaly state indicating whether an associated portion of the respective video stream exhibits a statistically abnormal change in motion; and</li><li id="ul0014-0003" num="0033">displaying a subset of the video streams on an output device as a function of the anomaly state associated with each of the video streams.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a video management system, comprising: <ul id="ul0015" list-style="none"><li id="ul0015-0001" num="0000"><ul id="ul0016" list-style="none"><li id="ul0016-0001" num="0035">a memory storing computer-readable instructions;</li><li id="ul0016-0002" num="0036">an input/output interface; and</li><li id="ul0016-0003" num="0037">a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: <ul id="ul0017" list-style="none"><li id="ul0017-0001" num="0038">obtaining at least one media stream from the input/output interface or the memory;</li><li id="ul0017-0002" num="0039">determining an anomaly state associated with each of the at least one media stream;</li><li id="ul0017-0003" num="0040">identifying segments of interest of each of the at least one media stream based on the anomaly state associated with each of the at least one media stream; and</li><li id="ul0017-0004" num="0041">creating a media summary based on the identified segments of interest of the at least one media stream.</li><li id="ul0017-0005" num="0042">outputting the media summary to the input/output interface or to the memory.</li></ul></li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a computer-implemented media processing method, comprising: <ul id="ul0018" list-style="none"><li id="ul0018-0001" num="0000"><ul id="ul0019" list-style="none"><li id="ul0019-0001" num="0044">obtaining at least one media stream divided into time windows;</li><li id="ul0019-0002" num="0045">determining an anomaly state of each time window of each of the at least one media stream;</li><li id="ul0019-0003" num="0046">identifying subsets of each of the at least one media stream based on the anomaly state of each time window of each of the at least one media stream; and</li><li id="ul0019-0004" num="0047">creating a media summary based on the identified subsets of the at least one media stream.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes: <ul id="ul0020" list-style="none"><li id="ul0020-0001" num="0000"><ul id="ul0021" list-style="none"><li id="ul0021-0001" num="0049">obtaining at least one media stream divided into time windows;</li><li id="ul0021-0002" num="0050">determining an anomaly state of each time window of each of the at least one media stream;</li><li id="ul0021-0003" num="0051">identifying subsets of each of the at least one media stream based on the anomaly state of each time window of each of the at least one media stream; and</li><li id="ul0021-0004" num="0052">creating a media summary based on the identified subsets of the at least one media stream.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a video management system, comprising: <ul id="ul0022" list-style="none"><li id="ul0022-0001" num="0000"><ul id="ul0023" list-style="none"><li id="ul0023-0001" num="0054">a memory storing computer-readable instructions;</li><li id="ul0023-0002" num="0055">an input/output interface; and</li><li id="ul0023-0003" num="0056">a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: <ul id="ul0024" list-style="none"><li id="ul0024-0001" num="0057">obtaining a plurality of video streams associated with a plurality of cameras;</li><li id="ul0024-0002" num="0058">determining an anomaly state associated with each of the video streams;</li><li id="ul0024-0003" num="0059">obtaining one or more criteria;</li><li id="ul0024-0004" num="0060">identifying occurrences in each video stream, said occurrences being indicative of the one or more criteria being met by the anomaly state; and</li><li id="ul0024-0005" num="0061">taking an action involving said occurrences.</li></ul></li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a computer-implemented media processing method, comprising: <ul id="ul0025" list-style="none"><li id="ul0025-0001" num="0000"><ul id="ul0026" list-style="none"><li id="ul0026-0001" num="0063">obtaining a media stream;</li><li id="ul0026-0002" num="0064">for each portion of a plurality of portions of the media stream, determining a state associated with that portion, the state being one of a plurality of possible states;</li><li id="ul0026-0003" num="0065">obtaining input from a user that identifies one or more criteria;</li><li id="ul0026-0004" num="0066">processing at least the state associated with each portion of the media stream to identify a subset of portions of the plurality of portions of the media stream that meet the one or more criteria; and</li><li id="ul0026-0005" num="0067">creating a media summary based on the identified subset of portions.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes: <ul id="ul0027" list-style="none"><li id="ul0027-0001" num="0000"><ul id="ul0028" list-style="none"><li id="ul0028-0001" num="0069">obtaining an original media stream;</li><li id="ul0028-0002" num="0070">for each of a plurality of portions of the media stream, determining an anomaly state associated with that portion, the anomaly state being one of a plurality of possible anomaly states; and</li><li id="ul0028-0003" num="0071">storing a composite media stream in a memory, each of a plurality of portions of the composite media stream comprising a corresponding one of the portions of the original media stream and an indication of the anomaly state associated with the corresponding one of the portions of the original media stream.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a video management system, comprising: <ul id="ul0029" list-style="none"><li id="ul0029-0001" num="0000"><ul id="ul0030" list-style="none"><li id="ul0030-0001" num="0073">a memory storing computer-readable instructions;</li><li id="ul0030-0002" num="0074">an input/output interface; and</li><li id="ul0030-0003" num="0075">a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: <ul id="ul0031" list-style="none"><li id="ul0031-0001" num="0076">obtaining an original media stream;</li><li id="ul0031-0002" num="0077">for a given portion of a plurality of portions of the media stream, determining an anomaly state associated with the given portion, the anomaly state being one of a plurality of possible anomaly states; and</li><li id="ul0031-0003" num="0078">storing a composite media stream in a memory, each of a plurality of portions of the composite media stream comprising a corresponding one of the portions of the original media stream and an indication of the anomaly state associated with the corresponding one of the portions of the original media stream.</li></ul></li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a computer-implemented media processing method, comprising: <ul id="ul0032" list-style="none"><li id="ul0032-0001" num="0000"><ul id="ul0033" list-style="none"><li id="ul0033-0001" num="0080">receiving a first plurality of media streams,</li><li id="ul0033-0002" num="0081">determining an anomaly state of each of the media streams;</li><li id="ul0033-0003" num="0082">selecting a second plurality of media streams from the first plurality of media streams based on (i) the anomaly state of each of the first plurality of media streams or (ii) transitions in the anomaly state of each of the first plurality of media streams; and</li><li id="ul0033-0004" num="0083">displaying the second plurality of media streams on a set of screens.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes: <ul id="ul0034" list-style="none"><li id="ul0034-0001" num="0000"><ul id="ul0035" list-style="none"><li id="ul0035-0001" num="0085">receiving a first plurality of media streams,</li><li id="ul0035-0002" num="0086">determining an anomaly state of each of the media streams;</li><li id="ul0035-0003" num="0087">selecting a second plurality of media streams from the first plurality of media streams based on (i) the anomaly state of each of the first plurality of media streams or (ii) transitions in the anomaly state of each of the first plurality of media streams; and</li><li id="ul0035-0004" num="0088">displaying the second plurality of media streams on a set of screens.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a video management system, comprising: <ul id="ul0036" list-style="none"><li id="ul0036-0001" num="0000"><ul id="ul0037" list-style="none"><li id="ul0037-0001" num="0090">a memory storing computer-readable instructions;</li><li id="ul0037-0002" num="0091">an input/output interface; and</li><li id="ul0037-0003" num="0092">a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: <ul id="ul0038" list-style="none"><li id="ul0038-0001" num="0093">receiving a first plurality of media streams,</li><li id="ul0038-0002" num="0094">determining an anomaly state of each of the media streams;</li><li id="ul0038-0003" num="0095">selecting a second plurality of media streams from the first plurality of media streams based on (i) the anomaly state of each of the first plurality of media streams or (ii) transitions in the anomaly state of each of the first plurality of media streams; and</li><li id="ul0038-0004" num="0096">displaying the second plurality of media streams on a set of screens.</li></ul></li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a media processing method implemented by a computer, comprising: <ul id="ul0039" list-style="none"><li id="ul0039-0001" num="0000"><ul id="ul0040" list-style="none"><li id="ul0040-0001" num="0098">obtaining a media stream;</li><li id="ul0040-0002" num="0099">obtaining a target characteristic;</li><li id="ul0040-0003" num="0100">obtaining one or more criteria;</li><li id="ul0040-0004" num="0101">creating a media summary based on subsets of the media stream that meet the one or more criteria; and</li><li id="ul0040-0005" num="0102">adapting the one or more criteria based on whether the media summary satisfies the target characteristic.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes: <ul id="ul0041" list-style="none"><li id="ul0041-0001" num="0000"><ul id="ul0042" list-style="none"><li id="ul0042-0001" num="0104">obtaining a media stream;</li><li id="ul0042-0002" num="0105">obtaining a target characteristic;</li><li id="ul0042-0003" num="0106">obtaining one or more criteria;</li><li id="ul0042-0004" num="0107">creating a media summary based on subsets of the media stream that meet the one or more criteria; and</li><li id="ul0042-0005" num="0108">adapting the one or more criteria based on whether the media summary satisfies the target characteristic.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a video management system, comprising: <ul id="ul0043" list-style="none"><li id="ul0043-0001" num="0000"><ul id="ul0044" list-style="none"><li id="ul0044-0001" num="0110">a memory storing computer-readable instructions;</li><li id="ul0044-0002" num="0111">an input/output interface; and</li><li id="ul0044-0003" num="0112">a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: <ul id="ul0045" list-style="none"><li id="ul0045-0001" num="0113">obtaining a media stream;</li><li id="ul0045-0002" num="0114">obtaining a target characteristic;</li><li id="ul0045-0003" num="0115">obtaining one or more criteria;</li><li id="ul0045-0004" num="0116">creating a media summary based on subsets of the media stream that meet the one or more criteria; and</li><li id="ul0045-0005" num="0117">adapting the one or more criteria based on whether the media summary satisfies the target characteristic.</li></ul></li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a computer-implemented media processing method, comprising: <ul id="ul0046" list-style="none"><li id="ul0046-0001" num="0000"><ul id="ul0047" list-style="none"><li id="ul0047-0001" num="0119">in an original state, displaying a map and a display window, the map corresponding to locations of a plurality of cameras and the display window displaying video feeds corresponding to video data sent by the cameras;</li><li id="ul0047-0002" num="0120">in response to a selection of a subset of said plurality of cameras, referred to as active cameras, displaying on the display window video feeds corresponding to video data sent by the active cameras, and creating summaries of the video data sent by the cameras in the plurality of cameras that are not the active cameras, referred to as passive cameras, in response to a request to return to the original state, playing back on the display window the summaries to catch up to present time; and</li><li id="ul0047-0003" num="0121">returning to the original state after the summaries have caught up to present time.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a non-transitory computer-readable medium comprising computer-readable instructions which, when executed by a computing device, configure the computing device to carry out a method that includes: <ul id="ul0048" list-style="none"><li id="ul0048-0001" num="0000"><ul id="ul0049" list-style="none"><li id="ul0049-0001" num="0123">in an original state, displaying a map and a display window, the map corresponding to locations of a plurality of cameras and the display window displaying video feeds corresponding to video data sent by the cameras;</li><li id="ul0049-0002" num="0124">in response to a selection of a subset of said plurality of cameras, referred to as active cameras, displaying on the display window video feeds corresponding to video data sent by the active cameras, and creating summaries of the video data sent by the cameras in the plurality of cameras that are not the active cameras, referred to as passive cameras,</li><li id="ul0049-0003" num="0125">in response to a request to return to the original state, playing back on the display window the summaries to catch up to present time; and</li><li id="ul0049-0004" num="0126">returning to the original state after the summaries have caught up to present time.</li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a video management system, comprising: <ul id="ul0050" list-style="none"><li id="ul0050-0001" num="0000"><ul id="ul0051" list-style="none"><li id="ul0051-0001" num="0128">a memory storing computer-readable instructions;</li><li id="ul0051-0002" num="0129">an input/output interface; and</li><li id="ul0051-0003" num="0130">a processor operatively coupled to the memory and to the input/output interface and configured for executing the computer-readable instructions stored in the memory to carry out a method that comprises: <ul id="ul0052" list-style="none"><li id="ul0052-0001" num="0131">in an original state, displaying a map and a display window, the map corresponding to locations of a plurality of cameras and the display window displaying video feeds corresponding to video data sent by the cameras;</li><li id="ul0052-0002" num="0132">in response to a selection of a subset of said plurality of cameras, referred to as active cameras, displaying on the display window video feeds corresponding to video data sent by the active cameras, and creating summaries of the video data sent by the cameras in the plurality of cameras that are not the active cameras, referred to as passive cameras,</li><li id="ul0052-0003" num="0133">in response to a request to return to the original state, playing back on the display window the summaries to catch up to present time; and</li><li id="ul0052-0004" num="0134">returning to the original state after the summaries have caught up to present time.</li></ul></li></ul></li></ul>
According to another broad aspect of the disclosure, there is provided a system, comprising: <ul id="ul0053" list-style="none"><li id="ul0053-0001" num="0000"><ul id="ul0054" list-style="none"><li id="ul0054-0001" num="0136">an input for receiving a video stream;</li><li id="ul0054-0002" num="0137">an output for releasing an output stream; and</li><li id="ul0054-0003" num="0138">a processor for producing the output stream from the video stream;</li><li id="ul0054-0004" num="0139">wherein the processor is configured such that when the video stream is characterized by a sequence of oscillating frames having a % motion change of M from one frame to the next for a non-zero value of M that is less than 100%, the output stream is characterized by first change in output value that lasts for a certain number of oscillating frames in the video stream followed by a second change in output value to a second value, which from that point on ceases to change while the video stream continues to include the first sequence of oscillating frames.</li></ul></li></ul>
BRIEF DESCRIPTION OF THE DRAWINGS
The following provides a description of certain embodiments with reference to the accompanying drawings, which are not to be considered limiting, and wherein:
<figref idref="DRAWINGS">FIGS. 1A and 1B</figref> are block diagrams of two non-limiting embodiments of a video processing environment;
<figref idref="DRAWINGS">FIG. 2</figref> illustrates a video stream made up of a plurality of time windows, each spanning a number of image samples;
<figref idref="DRAWINGS">FIG. 3</figref> is a flowchart showing steps in an anomaly detection process, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 4</figref> illustrates a sequence of image samples including indicators of inter-sample motion;
<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of a video processing environment in which a database stores a composite stream broken down into time windows each including video data and non-video data, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 6</figref> is a block diagram of an anomaly detector, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 7A</figref> is a timeline showing three video streams and <figref idref="DRAWINGS">FIG. 7B</figref> illustrates creation of a video summary, in accordance with non-limiting embodiments;
<figref idref="DRAWINGS">FIGS. 8 and 9</figref> illustrate finite state machines that may be used to represent changes in anomaly state of a video stream, in accordance with non-limiting embodiments;
<figref idref="DRAWINGS">FIG. 10</figref> conceptually illustrates an anomaly state output stream, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 11A</figref> shows a table conceptually illustrating an association between a set of cameras and a set of screens, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 11B</figref> is a flowchart of a carousel-type video display process that may be carried out by the video management system in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 12</figref> is a block diagram depicting, inter glia, a memory that stores instructions for carrying out a video summary creation tool, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 13</figref> graphical user interface presented by the video summary creation tool, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIGS. 14A to 14C</figref> conceptually show various states of a security application implemented by the video management system, in accordance with a non-limiting embodiment;
<figref idref="DRAWINGS">FIG. 15</figref> is a block diagram depicting a test bed for determining whether a device under test exhibits statistical anomaly detection capabilities, in accordance with a non-limiting embodiment; and
<figref idref="DRAWINGS">FIG. 16</figref> illustrates a finite state machine that may be used to represent state changes in the security application, in accordance with a non-limiting embodiment.
DETAILED DESCRIPTION
Reference is made to <figref idref="DRAWINGS">FIG. 1A</figref>, which is a block diagram of a first non-limiting embodiment of a video processing environment comprising an image acquisition device <b>130</b> (e.g., a camera), a video management system <b>110</b> and an output device <b>140</b> (e.g., a screen). The camera <b>130</b> supplies video data <b>135</b> to the video management system <b>110</b>, typically over a network such as an in-building network. Other cameras may similarly supply video data to the video management system <b>110</b>. The video management system <b>110</b> is configured to process the video data <b>135</b> from the camera <b>130</b> and possibly other cameras. In an embodiment, the video management system <b>110</b> is configured to carry out an anomaly detection process <b>120</b>, which will be described in further later on. The video management system <b>110</b> may be an evolved version of an existing video management system, such as Security Center®, sold by Genetec Inc., Saint-Laurent, Canada. The screen <b>140</b> may be associated with a desktop computer, laptop, network operations center or mobile device, to name a few non-limiting possibilities.
<figref idref="DRAWINGS">FIG. 1B</figref> is a block diagram of a second non-limiting embodiment of a video processing environment comprising the aforementioned image acquisition device <b>130</b> (e.g., a camera) and the aforementioned output device <b>140</b> (e.g., a screen). In this case, the video management system of <figref idref="DRAWINGS">FIG. 1A</figref> is replaced by a video management system <b>110</b>A and a separate anomaly detector <b>110</b>B. In this second embodiment, the video management system <b>110</b>A and the anomaly detector <b>110</b>B together execute certain functions of the video management system <b>110</b> of the first embodiment in <figref idref="DRAWINGS">FIG. 1A</figref>. For example, the anomaly detector <b>110</b>B is specifically configured to carry out the aforementioned anomaly detection process <b>120</b>.
Although <figref idref="DRAWINGS">FIGS. 1A and 1B</figref> show the camera <b>130</b> as being the source of the video data <b>135</b>, this does not imply that the video data <b>135</b> is a live feed. In various examples of implementation, the video data <b>135</b> may be a live feed or it may be delayed by interim passage through a network, sanitization by a network sanitizer, or storage on a disk. In other embodiments, the video data <b>135</b> may be retrieved from memory at a user-defined time that is unrelated to the time when the images in the video data <b>135</b> were recorded.
The video data <b>135</b> may be raw or parametrized. In the case where it is raw, the video data <b>135</b> may include a sequence of image samples (sometimes referred to as frames). Each image sample may be an arrangement of pixels each having an intensity value. In the case where it is parametrized, the video data <b>135</b> may include data (parameters) that represent the image samples and requiring a decoder at the receiving end. For example, such data may include a compressed representation of an image sample or information about changes in intensity between an image sample and the immediately preceding image sample (such as by way of one or more motion vectors). This information may be encoded into the video data <b>135</b> in accordance with a particular standard. In such a case, a decoder with knowledge of the standard is needed in order to reconstruct the image samples for display on a screen (such as the screen <b>140</b>). In contrast, raw video data may in some cases be directly displayed on a screen without having recourse to a decoder.
<figref idref="DRAWINGS">FIG. 2</figref> conceptually illustrates a plurality of image samples <b>210</b> associated with the video data <b>135</b>. Adjacent image samples are separated in time by a time interval. In some embodiments, the time interval between subsequent image samples may be 200 ms, whereas in other embodiments, the image samples may be produced at a rate of 24 or 30 samples per second, to name a few non-limiting possibilities. It is recalled that where the video data <b>135</b> is raw, the video data <b>135</b> carries the intensity values of the pixels of the image samples <b>210</b>, whereas where the video data <b>135</b> is parametrized (e.g., H.264 or H.265), the video data <b>135</b> carries a parametric representation the image samples <b>210</b>.
The video data <b>135</b> can be considered as divided into portions. Specifically, the image samples <b>210</b> can be grouped into “time windows” denoted <b>230</b><sub>t-3</sub>, <b>230</b><sub>t-2</sub>, <b>230</b><sub>t-1</sub>, <b>230</b><sub>t</sub>, . . . . Each of the time windows is associated with N image samples <b>220</b><sub>1 </sub>. . . <b>220</b><sub>N</sub>. In one non-limiting embodiment, the value of N may be 20, which corresponds to 4 seconds (if adjacent image samples were separated by 200 ms). In other embodiments, the value of N may be 10, 50, 100 or any other positive integer value.
A limited number of time windows (for example, 64, 100, 150, 200 or 300 of time windows, without being limited thereto) may be stored in a buffer of temporary/volatile storage or may be committed to non-volatile memory (e.g., stored on a disk). For example, a circular buffer containing 150 time windows of 20 image samples per time window and 200 ms between adjacent image samples would thus be able to store video spanning 600 seconds (=10 minutes).
In an embodiment, the time windows <b>230</b><sub>t-3</sub>, <b>230</b><sub>t-2</sub>, <b>230</b><sub>t-1</sub>, <b>230</b><sub>t</sub>, . . . are non-overlapping, i.e., the image samples associated with each time window do not include any of the image samples associated with any of the other time windows. Alternatively or in addition, at least one of the image samples associated with each of the time windows <b>230</b><sub>t-3</sub>, <b>230</b><sub>t-2</sub>, <b>230</b><sub>t-1</sub>, <b>230</b><sub>t</sub>, . . . is associated only with that time window and not with any of the other time windows. Alternatively or in addition, at least one of the image samples associated with each of the time windows <b>230</b><sub>t-3</sub>, <b>230</b><sub>t-2</sub>, <b>230</b><sub>t-1</sub>, <b>230</b><sub>t</sub>, . . . is associated with at least another one of the time windows.
The anomaly detection process <b>120</b> is used in some embodiments of the present disclosure. The anomaly detection process <b>120</b> includes receiving the video data <b>135</b>, processing it to identify an “anomaly state” of the various time windows and providing an indication of the anomaly state as an output.
In an embodiment, an “anomaly state” of a particular window reflects the anomalousness of the motion occurring in that time window. For example, in an embodiment, an “anomaly state” of a particular window reflects whether that there is a statistically abnormal increase in the motion occurring in that time window (low-motion to high-motion: this is termed “anomaly up”, or “A+”), a statistically abnormal decrease in the motion occurring in that time window (high-motion to low-motion: this is termed “anomaly down”, or “A−”), or neither (a situation that may be referred to as “no anomaly”, or “A0”). In accordance with various embodiments, it is not relevant whether the scene captured in the video data is busy or quiet (i.e., it is not merely a function of the instantaneous or average amount of motion). That is to say, it is not because a time window captures a busy scene versus a quiet scene that it will necessarily be associated with an A+ anomaly state versus an A− anomaly state; rather, what will qualify a time window as having an A+ anomaly state is if the motion in the scene becomes statistically abnormally high, which requires insight into what is normal, i.e., looking at past behavior/evolution of the video relating to the scene in question. These subtleties may be captured by the anomaly detection process <b>120</b>.
Reference is now made to <figref idref="DRAWINGS">FIG. 3</figref>, which is a flowchart showing steps forming part of the anomaly detection process <b>120</b> that could be executed by the anomaly detector <b>110</b>B/video management system <b>110</b> at a certain rate (e.g., once per image sample, for example).
Step <b>310</b><ul id="ul0055" list-style="none"><li id="ul0055-0001" num="0000"><ul id="ul0056" list-style="none"><li id="ul0056-0001" num="0169">The anomaly detection process <b>120</b> computes or otherwise obtains a “motion indicator” (denoted X(i;t)) for the “current sample” (denoted <b>220</b><sub>i</sub>) of the “current time window” (denoted <b>230</b><sub>t</sub>). The motion indicator X(i;t) could be obtained in various ways, depending on, for example, whether the video data <b>135</b> supplied to the anomaly detector <b>110</b>B/video management system <b>110</b> is raw or parametrized.</li><li id="ul0056-0002" num="0170">In a first example embodiment, it is observed that video data encoded according to certain encoding standards (such as, for example, the H.264 standard) includes motion vectors embedded as part of the video data itself for each coded sample. Therefore, obtaining the motion indicator simply requires obtaining or determining the magnitude of the motion vectors for the current sample, which gives the percentage of motion in the scene relative to the previous sample. This is an example of where the motion indicator X(i;t) can be easily derived from information explicitly present in the available video data <b>135</b>.</li><li id="ul0056-0003" num="0171">In a second example embodiment, assume that the video data <b>135</b> is raw. In this case, the anomaly detector <b>110</b>B/video management system <b>110</b> computes the sum of differences between pixel intensity values from one image sample to the next. This broad measure of how much an image changes from one sample to the next can be an example of a “motion indicator”, which in this example is not included in or trivially derived from the received video data <b>135</b> but is somewhat more complex to compute.</li><li id="ul0056-0004" num="0172">In a third example embodiment, also assume that the video data <b>135</b> is raw. Referring to <figref idref="DRAWINGS">FIG. 4</figref>, there is shown a sequence of samples (the last sample <b>220</b><sub>N </sub>of the previous time window <b>230</b><sub>t-1</sub>, the first sample <b>220</b><sub>1 </sub>of the current time window <b>230</b><sub>t</sub>, the second sample <b>220</b><sub>2 </sub>of current time window <b>230</b><sub>t </sub>and the third sample <b>220</b><sub>3 </sub>of the current time window <b>230</b><sub>t</sub>). Here, the image represented by each sample is broken down into blocks; in this case there are 8×8=64 blocks. Then, the content of correspondingly positioned blocks is compared from sample to sample. Thus, for the first sample <b>220</b><sub>1 </sub>of the current time window <b>230</b><sub>t</sub>, the number of blocks that experienced a change in the level of inter-sample movement since the previous sample is 6, which gives a motion indicator of X<sub>1</sub>=6. In the second sample <b>220</b><sub>2 </sub>of the current time window <b>230</b><sub>t</sub>, the number of blocks that experienced a change in the level of inter-sample movement since the previous sample is 8, which gives a motion indicator of X<sub>2</sub>=8, and in the third sample <b>220</b><sub>3 </sub>it was 12, which gives a motion indicator of X<sub>3</sub>=12.</li><li id="ul0056-0005" num="0173">There are of course other encoding standards (e.g., HEVC, VP9, AV1, . . . ) as well as other techniques for measuring or computing a change in motion from one image sample to the next, in order to obtain the sought-after motion indicator X(i;t) for the current sample <b>220</b><sub>i </sub>and the current time window <b>230</b><sub>t</sub>; such other techniques may be used in the context of various embodiments.</li><li id="ul0056-0006" num="0174">The next step in the anomaly detection process <b>120</b> is Step <b>315</b>A.</li></ul></li></ul>
Step <b>315</b>A <ul id="ul0057" list-style="none"><li id="ul0057-0001" num="0000"><ul id="ul0058" list-style="none"><li id="ul0058-0001" num="0176">The anomaly detection process <b>120</b> computes a moving average of the motion indicators for the Q most recent image samples. Q is an integer that can be as small as 1 or arbitrarily large, even as large as N (the number of image samples per time window), or perhaps even larger. The moving average of the motion indicators for the Q most recent image samples is denoted X<sub>A</sub>(i;t;Q). As such, X<sub>A</sub>(i;t;Q) is the moving average of X(i;t), X(i−1;t), . . . X(i−Q+1;t) although where Q>i, this would involve motion vectors for image samples of the previous time window <b>230</b><sub>t-1 </sub>as well. The anomaly detection process <b>120</b> proceeds to step <b>315</b>B.</li></ul></li></ul>
Step <b>315</b>B <ul id="ul0059" list-style="none"><li id="ul0059-0001" num="0000"><ul id="ul0060" list-style="none"><li id="ul0060-0001" num="0178">The anomaly detection process <b>120</b> determines whether the current sample <b>220</b><sub>i </sub>is the first sample in the current time window <b>230</b><sub>t </sub>(i.e., i=1, in which case the anomaly detection process <b>120</b> proceeds to Step <b>315</b>D), the last sample in the current time window <b>230</b><sub>t </sub>(i.e., i=N, in which case the anomaly detection process <b>120</b> proceeds to Step <b>315</b>C) or neither (i.e., 1<i<N, in which case the anomaly detection process <b>120</b> proceeds to Step <b>315</b>E).</li></ul></li></ul>
Step <b>315</b>C <ul id="ul0061" list-style="none"><li id="ul0061-0001" num="0000"><ul id="ul0062" list-style="none"><li id="ul0062-0001" num="0180">This step is entered when i=N, i.e., the current sample <b>220</b><sub>i </sub>is the last (Nth) sample in the current time window <b>230</b><sub>t</sub>. The anomaly detection process <b>120</b> measures certain “statistical parameters” of the various motion indicators X(i;t) for the current time window <b>230</b><sub>t </sub>to obtain certain “measured statistical parameters” for the current time window <b>230</b><sub>t</sub>. For example, the “measured statistical parameters” could be the “measured mean” (denoted M<sub>t</sub>) and the “measured variance” (denoted V<sub>t</sub>) of the N motion indicators X(i;t), i=1 . . . N, computed over the N samples <b>220</b><sub>1 </sub>. . . <b>220</b><sub>N </sub>in the current time window <b>230</b><sub>t</sub>. However, this need not be the case in all embodiments, namely other statistical parameters could be used, and they need not be measured over the entire number of N samples in the current time window <b>230</b><sub>t</sub>.</li><li id="ul0062-0002" num="0181">The measured statistical parameters for the current time window <b>230</b><sub>t </sub>(in this case, the measured mean M<sub>t </sub>and the measured variance V<sub>t</sub>) are stored for future use, that is to say, when the anomaly detection process <b>120</b> is executed for a later sample that may not be the last sample of a time window. (It follows that the measured statistical parameters for previous time windows would be available as they would already have been computed in the past. For example, this is the case with the measured mean M<sub>t-1 </sub>and the measured variance V<sub>t-1 </sub>for prior time window <b>230</b><sub>t-1</sub>, as well as the measured mean M<sub>t-2 </sub>and the measured variance V<sub>t-2 </sub>for prior time window <b>230</b><sub>t-2</sub>, and so on.) The anomaly detection process <b>120</b> proceeds to Step <b>315</b>E.</li></ul></li></ul>
Step <b>315</b>D <ul id="ul0063" list-style="none"><li id="ul0063-0001" num="0000"><ul id="ul0064" list-style="none"><li id="ul0064-0001" num="0183">This step is entered when i=1, i.e., the current sample <b>220</b><sub>i </sub>is the first sample in the current time window <b>230</b><sub>t</sub>. The anomaly detection process <b>120</b> computes certain “estimated” statistical parameters of the various motion indicators X(i;t) expected to be computed during the current time window <b>230</b><sub>t</sub>. In this non-limiting example, the estimated statistical parameters include an “estimated mean” and an “estimated variance” of the motion indicators expected to be computed during the current time window <b>230</b><sub>t</sub>. The estimated mean is denoted M*<sub>t </sub>and the estimated variance is denoted V*<sub>t</sub>. Calculation of the estimated statistical parameters can be done according to formulae that look back at the “measured mean” and the “measured variance” of the motion indicators for one, two or more previous time windows, namely M<sub>t-1</sub>, M<sub>t-2</sub>, . . . and V<sub>t-1</sub>, V<sub>t-2</sub>, . . . . It will be appreciated that these values are available due to having been computed during previous instantiations of Step <b>315</b>C of the anomaly detection process <b>120</b>.</li><li id="ul0064-0002" num="0184">In a specific non-limiting embodiment, the following formulae for the estimated mean M*<sub>t </sub>and the estimated variance V*<sub>t </sub>of the motion indicators for the current time window <b>230</b><sub>t </sub>could be used: <br /><i>M*</i><sub>t</sub><i>=αM</i><sub>t-1</sub>+(1−α)<i>M</i><sub>t-2 </sub><br /><i>V*</i><sub>t</sub><i>=αV</i><sub>t-1</sub>+(1−α)<i>V</i><sub>t-2</sub>+α(1−<i>a</i>)(<i>M</i><sub>t-2</sub><i>−M</i><sub>t-1</sub>)<sup>2 </sup></li><li id="ul0064-0003" num="0185">where the parameter α (alpha) is referred to as a “weight”. The weight could be a variable that is user-determined. It may be different in each of the above formulae, i.e., a different weight may be used for the estimated mean M*<sub>t </sub>and the estimated variance V*<sub>t</sub>. The weight is an indication of how much influence the measured mean and measured variance of the motion indicators for the immediately preceding time window <b>230</b><sub>t-1 </sub>have on the estimated mean and the estimated variance of the motion indicators for current time window <b>230</b><sub>t</sub>, relative to the measured statistical parameters of the even earlier time windows <b>230</b><sub>t-2</sub>, etc. It will be noted that the right-most term in each of the aforementioned formulae captures the influence of all previous time windows. This is a result that is expected from a mathematical assumption that the distribution of the motion indicators X(i;t), i=1 . . . N, for the N samples <b>220</b><sub>1 </sub>. . . <b>220</b><sub>N </sub>in the current time window <b>230</b><sub>t </sub>is a normal (Gaussian) distribution. In other embodiments, it may be possible to limit how far back one goes to consider past contributions of the measured mean and measured variance to the estimated mean and estimated variance for the current time window <b>230</b><sub>t</sub>. The anomaly detection process <b>120</b> proceeds to Step <b>315</b>E.</li></ul></li></ul>
Step <b>315</b>E <ul id="ul0065" list-style="none"><li id="ul0065-0001" num="0000"><ul id="ul0066" list-style="none"><li id="ul0066-0001" num="0187">Based on X<sub>A</sub>(i;t;Q), namely, the moving average of the last Q motion indicators (when considering the current sample <b>220</b><sub>i </sub>of the current time window <b>230</b><sub>t</sub>), and also based on the estimated mean M*<sub>t </sub>and the estimated variance V*<sub>t </sub>of the motion indicators for the current time window <b>230</b><sub>t</sub>, the anomaly detection process <b>120</b> computes a positive P-value P+(i;t) and a negative P-value P−(i;t) for the current sample <b>220</b><sub>i </sub>and the current time window <b>230</b><sub>t</sub>.</li><li id="ul0066-0002" num="0188">In a non-limiting example embodiment, to compute the positive P-value P+(i;t) and the negative P-value P−(i;t) for the current sample <b>220</b><sub>i </sub>and the current time window <b>230</b><sub>t</sub>, an assumption is made that the moving average motion indicators X<sub>A</sub>(i;t;Q), obey an underlying distribution (behavior) of a certain type (such as normal or beta, by way of two non-limiting examples).</li><li id="ul0066-0003" num="0189">Then, for moving average motion indicator X<sub>A</sub>(i;t;Q), the anomaly detection process <b>120</b> computes the “one-tail, positive P-value P+(i;t)”, which is the likelihood of obtaining a motion indicator greater than or equal to X<sub>A</sub>(i;t;Q), if the set of motion indicators were indeed governed by the underlying (hypothesized) probability distribution. Mathematically, the one-tail, positive p-value P+(i;t) can be calculated as follows (where P+(i;t) is denoted P<sup>+</sup> for short): <br /><i>P</i><sup>+</sup>=1−<i>F</i>(<i>X</i><sub>i</sub><i>|M*</i><sub>t</sub><i>,V*</i><sub>t</sub>),</li><li id="ul0066-0004" num="0190">where F( ) is the cumulative density function (CDF) of the one-dimensional hypothesized probability distribution which is defined as:</li></ul></li></ul>
<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>[</mo><mrow><mn>1</mn><mo>+</mo><mrow><mi>erf</mi><mo>(</mo><mfrac><mrow><msub><mi>X</mi><mi>i</mi></msub><mo>-</mo><msubsup><mi>M</mi><mi>t</mi><mo>*</mo></msubsup></mrow><msqrt><mrow><mn>2</mn><mo></mo><msubsup><mi>V</mi><mi>t</mi><mo>*</mo></msubsup></mrow></msqrt></mfrac><mo>)</mo></mrow></mrow><mo>]</mo></mrow><mo>,</mo></mrow></math></maths><img file="US10971192B2_D0001.tif" /><ul id="ul0067" list-style="none"><li id="ul0067-0001" num="0000"><ul id="ul0068" list-style="none"><li id="ul0068-0001" num="0192">in the case where the hypothesized probability distribution is a normal distribution.</li><li id="ul0068-0002" num="0193">Conversely, the anomaly detection process <b>120</b> also computes the “one-tail, negative p-value” P−(i;t), which is the likelihood of obtaining a motion indicator less than or equal to X<sub>A</sub>(i;t;Q), if the set of motion indicators were indeed governed by the underlying (hypothesized) probability distribution. This can be computed mathematically as: <br /><i>P</i><sup>−</sup><i>=F</i>(<i>X</i><sub>i</sub><i>|M*</i><sub>t</sub><i>,V*</i><sub>t</sub>)=1−<i>P</i><sup>+</sup>.</li><li id="ul0068-0003" num="0194">Generally speaking, these instantaneous positive and negative P-values will, depending on their values, be representative of either a detected instantaneous anomalous increase in motion, a detected instantaneous anomalous decrease in motion or neither. In some embodiments, what is detected in this way is a relative deviation of the statistical behavior of X<sub>A</sub>(i;t;Q), compared to what is considered “normal” based on the measured mean and measured variance of the motion indicators in past time windows. The anomaly detection process <b>120</b> proceeds to Step <b>345</b>.</li></ul></li></ul>
Step <b>345</b><ul id="ul0069" list-style="none"><li id="ul0069-0001" num="0000"><ul id="ul0070" list-style="none"><li id="ul0070-0001" num="0196">The anomaly detection process <b>120</b> determines whether i=N, i.e., whether the current sample <b>220</b><sub>i </sub>is the last sample in the current time window <b>230</b><sub>t</sub>. If yes, the anomaly detection process <b>120</b> proceeds to Step <b>350</b>, otherwise the anomaly detection process <b>120</b> proceeds to Step <b>360</b>.</li></ul></li></ul>
Step <b>350</b><ul id="ul0071" list-style="none"><li id="ul0071-0001" num="0000"><ul id="ul0072" list-style="none"><li id="ul0072-0001" num="0198">This step is entered once per time window when i=N, in other words when the current sample <b>220</b><sub>i </sub>is indeed the last sample in the current time window <b>230</b><sub>t</sub>. The anomaly detection process <b>120</b> determines an “anomaly state” for the current time window <b>230</b><sub>t</sub>. This is obtained based on the various positive and negative P-values computed for the N samples <b>220</b><sub>1 </sub>. . . <b>220</b><sub>N </sub>of the current time window <b>230</b><sub>t </sub>(i.e., P+(i;t) and P−(i;t), i=1 . . . N). Specifically, one first computes the average (e.g., mean) positive P-value (denoted P+<sub>A</sub>(t)) and the average (e.g., mean) negative P-value (denoted P−<sub>A</sub>(t)) over the N samples in the time window. Then, based on these average positive and negative P-values P+<sub>A</sub>(t) and P−<sub>A</sub>(t), the anomaly detection process <b>120</b> determines an “anomaly state” for the current time window <b>230</b><sub>t</sub>.</li><li id="ul0072-0002" num="0199">For example, if P+<sub>A</sub>(t) (i.e., the average positive P-value taken over all N samples associated with the current time window <b>230</b><sub>t</sub>) is less than or equal to a threshold A (of, say, 0.05), this means that it is highly likely (if one's assumption about the underlying statistical behavior is correct) that one is dealing with an outlier time window. As such, this associated level of motion is considered “abnormally high” (or statistically abnormally high), and an anomaly state of “A+” can be associated with the current time window <b>230</b><sub>t</sub>.</li><li id="ul0072-0003" num="0200">For its part, if P−<sub>A</sub>(t) (i.e., the average negative P-value taken over all N samples associated with the current time window <b>230</b><sub>t</sub>) is less than or equal to a threshold B (of, say, 0.05), this means that it is highly likely (if one's assumption about the underlying distribution was correct) that one is dealing with an outlier time window. In this case, the associated level of motion is considered “abnormally low” (or statistically abnormally low), and an anomaly state of “A−” can be associated with the current time window <b>230</b><sub>t</sub>.</li><li id="ul0072-0004" num="0201">In the case where the value of P+<sub>A</sub>(t) is greater than A and the value of P−<sub>A</sub>(t) is greater than B, then an anomaly state of “A0” (“no anomaly”) can be associated with the current time window <b>230</b><sub>t</sub>. In this case, the associated level of motion is considered neither abnormally low nor abnormally high. It should be noted that A and B may be the same value in some embodiments, although this is not a requirement.</li><li id="ul0072-0005" num="0202">In some embodiments, the aforementioned averages P+<sub>A</sub>(t), P−<sub>A</sub>(t) can be moving averages taken over the samples looking back over more than one time window, e.g., the current time window <b>230</b><sub>t </sub>and one or more time windows <b>230</b><sub>t-1</sub>, <b>230</b><sub>t-2</sub>, etc. that precede the current time window <b>230</b><sub>t</sub>.</li><li id="ul0072-0006" num="0203">The anomaly detection process <b>120</b> advances to Step <b>360</b>.</li></ul></li></ul>
Step <b>360</b><ul id="ul0073" list-style="none"><li id="ul0073-0001" num="0000"><ul id="ul0074" list-style="none"><li id="ul0074-0001" num="0205">The anomaly detection process <b>120</b> increments the value of i (unless i already equals to N, in which case i is reset to zero and the value of t is incremented) and returns to Step <b>310</b> to obtain a motion indicator for the next sample.</li></ul></li></ul>
In summary, an anomaly state is obtained for each time window of the video data <b>135</b>. The anomaly state is obtained by (i) obtaining estimated statistical parameters for the current time window based on measured statistical parameters characterizing the motion indicators for the samples in at least one time window of the video stream that precedes the current time window (considered “normal motion”); and (ii) determining the anomaly state for the current time window based on the estimated statistical parameters and motion indicators obtained for the samples in the current time window. In particular, statistical hypothesis testing under certain assumptions is conducted to determine if in the current time window contains motion that is normal or anomalous.
It is also noted that thresholds A and B may be kept constant throughout the processing of the video stream, yet they will still enable detection of abnormal increases or decreases in motion, even if what is considered “abnormal” changes over time as the video data <b>135</b> undergoes changes in the average amount of motion from one time window to the next. This occurs because a change in the anomaly state is triggered a function of estimated and computed statistical parameters, which may fluctuate from one time window to the next. Thus, the thresholds A and B may be considered robust in the face of fluctuations in the average amount of motion being captured.
Those skilled in the art will appreciate that in the case of a hypothesized normal distribution, the statistical parameters include the mean and the variance (or standard deviation) that are used to characterize the distribution. In the case of a beta distribution characterized by shape parameters α and β, the statistical parameters may be α and β. When the underlying hypothesized distribution is different from normal or beta (e.g., gamma), other statistical parameters may be computed and estimated, as long as at least two such statistical parameters are used.
From the above, it will be appreciated that the anomaly state produced at Step <b>350</b> can be indicative of the current time window <b>230</b><sub>t </sub>exhibiting an anomalous increase in motion (denoted A+), an anomalous decrease in motion (denoted A−) or neither (denoted A0). Transitions between anomaly states may be represented by a finite-state-machine (FSM), illustrated in <figref idref="DRAWINGS">FIG. 8</figref> where the FSM can transition from anomaly state A+ to anomaly state A0 and vice versa, as well as from anomaly state A− to anomaly state A0 and vice versa, but not directly between anomaly states A+ and A−. The notion of a FSM is not restricted to 3 anomaly states, and can be any number of anomaly states greater than 2. For example, an expansion into a higher number of anomaly states (multi-threshold) can be done where one has a “mild” increase/decrease of anomalous motion and a “severe” increase/decrease of anomalous motion, providing in this case 5 states for how anomalous the level of motion can be in a given time window.
For example, these can be associated with two values of the “A” threshold (corresponding to states A++ and A+) and two values of the “B” threshold (corresponding to states A− and A−−), as is illustrated in <figref idref="DRAWINGS">FIG. 9</figref>.
Multiple passes through Step <b>350</b> for multiple time windows of the video data <b>135</b> result in the creation of an anomaly state output stream <b>125</b>, which is shown in <figref idref="DRAWINGS">FIG. 10</figref> by way of non-limiting example. The anomaly state output stream <b>125</b> can include a stream of data sets <b>125</b>A, <b>125</b>B, <b>125</b>C, each data set including an ID (“time window ID”) of a given time window <b>135</b>A, <b>135</b>B, <b>135</b>C, of the video data <b>135</b> and the anomaly state for the given time window (as computed by Step <b>350</b> when that time window was the “current time window” during execution of the anomaly detection process <b>120</b>), and possibly also an identifier of the image acquisition device (e.g. camera) <b>130</b> that originated the given time window, which may be useful in a multi-camera environment. The anomaly state stream <b>125</b> may be stored in memory or transmitted in real-time.
In some cases it may be desirable to keep statistics on the statistics. In other words, a data structure may be created in the memory for storing an association between video streams (e.g., according to video stream ID) and the relative number of time windows spent in each of the various anomaly states (e.g., A−, A0 and A+). This data structure may be in the form of a table and may be updated after the anomaly state of each time window has been computed. The table may include a record including an ID field with the identifier of the video stream, and a plurality of fields, one for each particular anomaly state and holding a count of the number of time windows in that particular anomaly state. The table may also compute the percentage of the total time spent in each state. The contents of the table may be read by the video management system <b>110</b>/<b>110</b>A and displayed on the display <b>140</b>. As such, in a non-limiting example, the operator may be presented with at-a-glance information on multiple video streams, each with a data element indicating the percentage of time that the corresponding video stream has spent in, say, anomaly state A+. This allows the operator to compare the relative “anomalounsess” of various streams. This may lead the operator to perform various actions, such as adjust the sensitivity (e.g., thresholds A and B against which the p-values are compared) or zero in on specific video streams having, e.g., greater anomalousness. In an alternative embodiment, the record for a particular video stream may indicate the total amount of time spent in a particular anomaly state.
Test Bed
The above teachings can be used to devise a test bed for determining whether or not a particular detector (a “detector under test”—DUT) is configured to detect abnormal changes in motion. For example, a DUT may be configured to receive a video stream and produce an output stream based on the video stream. The output stream can indicate a value for each time window of the video stream. It may not be known a priori whether the DUT is configured to perform detection of abnormal changes in motion as opposed to performing mere “motion level” threshold detection. By use of the test bed, one may be able to determine whether the DUT is configured to perform detection of abnormal changes in motion as opposed to performing mere “motion level” threshold detection. Specifically, by constructing the video stream a certain way, if the DUT is an anomaly state detector implementing one of the previously described (statistics-based) techniques, then the values in the output stream will exhibit certain first behavior, and if the DUT is a standard “motion level” detector, then the values in the output stream will exhibit certain second behavior. This will now be described in further detail.
In this regard, and as shown in <figref idref="DRAWINGS">FIG. 15</figref>, a test bed <b>1500</b> comprises a test system <b>1510</b> that outputs a test video stream <b>1520</b> having certain predetermined characteristics that can be designed or selected during a preliminary phase. The test video stream <b>1520</b> is provided to a detector under test (DUT) <b>1530</b>. The DUT <b>1530</b> produces an output stream <b>1540</b> that indicates a value for each time window of the test video stream <b>1520</b>. This value may be binary or ternary, for example. For instance, the DUT <b>1530</b> may produce an output value for each time window that is selected between “motion” and “no motion”. This may suggest to an external observer that the DUT <b>1530</b> is a “standard” motion detector with an (unknown, hypothetical) internal motion level threshold M<sup>HYP</sup><sub>ON </sub>above which the output is set to “motion”. The possibility may exist of an (unknown, hypothetical) internal level threshold M<sup>HYP</sup><sub>OFF </sub>(different from M<sup>HYP</sup><sub>ON</sub>) below which the output changes from “motion” to “no motion”.
As such, to the external observer, the DUT <b>1530</b> with this type of output stream <b>1540</b> may appear to function as a standard motion level detector. However, it is conceivable that the DUT <b>1530</b>, despite its external appearance, is actually functioning as an anomaly detector that implements one of the previously described anomaly detection approaches. This is a situation where a DUT purporting to be a standard motion level detector is actually implementing an anomaly detector (detecting abnormal changes in motion as described earlier in this document). It is through careful design of the characteristics of the test video stream <b>1520</b> that this situation can be uncovered.
Accordingly, in the context of the embodiment of <figref idref="DRAWINGS">FIG. 15</figref>, the output stream <b>1540</b> is fed as an input to the test system <b>1510</b> for analysis. The test system <b>1510</b> then produces a determination output <b>1550</b>. The determination output <b>1550</b> may be as simple as a binary value (yes/no) indicative of the answer to the question “Does the DUT <b>1530</b> perform statistical anomaly detection?”
In order to answer this question and set the value of the determination output <b>1550</b>, the test system <b>1510</b> includes a processor that executes computer-readable instructions stored in a memory. When executed, the computer-readable instructions cause the processor to process the output stream <b>1540</b> as a function of the predetermined characteristics of the test video stream <b>1520</b>. By having carefully designed the test video stream <b>1520</b> to have certain predetermined characteristics, it is possible to infer, from the output stream <b>1540</b>, the inner workings of the DUT <b>1530</b> to some degree. In one embodiment, it is possible to determine whether the DUT <b>1530</b> secretly functions as an anomaly detector while suggesting to the outside world that it functions as a standard motion level detector.
More specifically, two design cases are considered, one for raw/MPEG video and one for parametrized video, each of which will be approached separately. In the raw/MPEG design case, motion estimation is done on a pixel-by-pixel basis. Consider each pixel of the test video stream <b>1520</b> to be an unsigned character ranging from (0-255). There are many formats for raw and MPEG however (e.g. YUV422, Y800, etc., more information can be found at http://www.fourcc.org/yuv.php, hereby incorporated by reference herein). In any event, the test video stream <b>1520</b> may be constructed with pixels that are varied by the same amount, which could be a certain user-definable percentage V.
Assume for the time being that the DUT <b>1530</b> purports to be a standard motion detector. In that case, it uses a hypothetical “motion on threshold”, denoted M<sup>HYP</sup><sub>ON</sub>. Since V is user-definable, it can be adjusted so as to have a value that will always cause the level of motion to exceed whatever value of M<sup>HYP</sup><sub>ON </sub>might by used by the DUT <b>1530</b> if it is a standard motion detector (e.g., such as is found in customary camera/video management systems).
As such, by choosing V, each component of the underlying video feed can be guaranteed to vary by more than M<sup>HYP</sup><sub>ON </sub>% between frames. As such, assume that the first frame is designed to have the following set of pixel values: <ul id="ul0075" list-style="none"><li id="ul0075-0001" num="0000"><ul id="ul0076" list-style="none"><li id="ul0076-0001" num="0222">Frame 1: [0, 0, 0, 0, 0, 0, 0, 0, . . . ] (this arrangement of pixel values can be referred to as “Set A”).</li></ul></li></ul>
Then, the next frame may be designed have the following set of pixel values: <ul id="ul0077" list-style="none"><li id="ul0077-0001" num="0000"><ul id="ul0078" list-style="none"><li id="ul0078-0001" num="0224">Frame 2: [255*V/100, 255*V/100, 255*V/100, 255*V/100, . . . ] (this arrangement of pixel values can be referred to as “Set B”)</li></ul></li></ul>
A sequence of frames (Frame 1, Frame 2, Frame 3, Frame 4, Frame 5, . . . ) is thus produced having the above alternating sets of pixel values (Set A, Set B, Set A, Set B, Set A, . . . ), where every pixel between adjacent frames varies by exactly V percent. The test video stream <b>1520</b> would look like “noise” if someone were to watch it directly on a screen; however, mathematically it results in an exact, measurable degree of motion.
This variation in pixel values (oscillating between Set A and Set B between frames) makes a significant difference on the output stream <b>1540</b> depending on the type of DUT <b>1530</b>, namely whether the DUT <b>1530</b> is a standard motion detector or implements a statistical algorithm to detect motion anomalies.
For instance, in the case where the DUT <b>1530</b> implements variance-based statistical anomaly detection as described herein above, there would at first be a detection of an anomaly (e.g., anomaly state becomes A+). Thereafter, a “burn-in” would set in, where after the above alternating sequence of frames has been running for a period of time, the variance decreases to at/near zero, to the point where there is considered to no longer be any anomalous motion (i.e., anomaly state becomes A0). The output stream <b>1540</b> would therefore change values at some point in the above sequence of frames.
However, in the case where the DUT <b>1530</b> implements standard level-based motion detection (for some unknown value of M<sup>HYP</sup><sub>ON</sub>), the above sequence of frames in the test video stream <b>1520</b> would trigger a “motion on” condition, as V has been selected to be high enough. This motion-on condition would remain unchanged throughout the sequence of frames, as there is no “burn-in period”, and thus the output stream <b>1540</b> would not change values throughout the above sequence of frames.
As such, based on the behavior of the output stream <b>1540</b> throughout the above sequence of frames (alternating between Set A and Set B) it is possible to infer the type of motion detection being performed by the DUT <b>1530</b>.
For added confidence in determining the type of motion detection being performed by the DUT <b>1530</b>, one can increase V to 100% after a certain number of alternating frames/amount of time (e.g., 1 minute, 10 minutes, 30 minutes, etc.) This means that at a given point in time (e.g., at frame “X”), under control of the test system <b>1510</b>, the video stream <b>1520</b> now includes frames with the following sets of pixel values: <ul id="ul0079" list-style="none"><li id="ul0079-0001" num="0000"><ul id="ul0080" list-style="none"><li id="ul0080-0001" num="0231">Frame X: [0, 0, 0, 0, 0, 0, 0, 0, . . . ] (“Set A”);</li><li id="ul0080-0002" num="0232">Frame X+1: [255, 255, 255, 255, 255, 255, 255, 255, . . . ] (this arrangement of pixel values can be referred to as “Set C”);</li><li id="ul0080-0003" num="0233">Frame X+2: [0, 0, 0, 0, 0, 0, 0, 0, . . . ] (“Set A”);</li><li id="ul0080-0004" num="0234">Frame X+1: [255, 255, 255, 255, 255, 255, 255, 255, . . . ] (Set “C”);</li><li id="ul0080-0005" num="0235">Etc.</li></ul></li></ul>
This would change the motion indicator from V % to 100%. Assuming the variance before the processing of Frame X would have been zero or close to zero, the arrival of frames X, X+1, etc. now changes the variance. This triggers an anomalous motion “high” event (i.e., anomaly state A+) if the DUT <b>1530</b> were using a statistical anomaly detection algorithm (with any probability distribution parametrized by at least 2 variables so the variance can be independent of the mean estimate). As such, the output stream <b>1540</b> would change values very quickly after Frame X.
Yet no change in the output stream <b>1540</b> would occur if a standard motion detector were being used, because the output stream <b>1540</b> continues to signal high motion from the previous set of frames (because V was pre-selected to cause a high motion level).
In summary, based on a test video stream <b>1520</b> being constructed as discussed above: <ul id="ul0081" list-style="none"><li id="ul0081-0001" num="0000"><ul id="ul0082" list-style="none"><li id="ul0082-0001" num="0239">If the DUT <b>1530</b> is not using a standard variance-based motion level detection algorithm, then one expects that the output stream <b>1540</b> signals an event which will be raised once the test video stream <b>1520</b> (with V % motion) starts playing. Then, when the change comes to V=100% motion, one expects that no further events will be raised since the DUT <b>1530</b> will already be in a (traditional) motion “on” state. In this case, the determination output <b>1550</b> may be set to signal that the DUT <b>1530</b> implements standard motion level detection.</li><li id="ul0082-0002" num="0240">If the DUT <b>1530</b> is using a statistical variance-based anomaly detection algorithm (with any probability distribution which estimates the mean separate from the variance), then one expects that the output stream <b>1540</b> signals an event which will be raised once the test stream <b>1520</b> starts playing (this is due to any possible starting values); however it would eventually go back to “normal” (again depending on the initializing values for the mean and variance), and then would rise again when the test stream <b>1520</b> changes to V=100% motion. In this case, the determination output <b>1550</b> may be set to signal that the DUT <b>1530</b> implements statistical anomaly detection.</li></ul></li></ul>
As such, the test system <b>1510</b> can be used to supply a test video stream <b>1520</b> to the DUT <b>1530</b> in order to determine the anomaly detection behavior of the DUT <b>1530</b> irrespective of external indicators (e.g., marketing, labeling) that suggests standard behavior.
In the parametrized video design case (e.g., H.264 and H.265), the video feeds are unique in that the “motion vectors” are encoded in the headers of the video feeds themselves. Traditional pixel-by-pixel motion detection may not be suitable for these types of video streams. Motion detection in this case may be achieved by reading the motion vectors and computing the “magnitude” of the vectors. As such, one need not generate pixel changes that then result in desired motion vectors, but rather one can create “fake” motion vectors themselves such that they encode V % motion for a sufficiently long time to allow the estimated variance to go towards near 0. Thereafter, the fake motion vectors can be changed to 100% to trigger a new change in anomaly state (A+). In this case, the actual image content of the resulting test video stream <b>1520</b> video; one only needs to create the “fake” motion vectors.
More generally, it will be appreciated that the following protocol may be followed in order to provide a test bed for a device under test (DUT) whose ability to detect anomalous motion in a statistical fashion is being determined: <ul id="ul0083" list-style="none"><li id="ul0083-0001" num="0244">(a) Design a test video stream having the following frame content (each “Set” represents a certain set of pixel values for the frame): <ul id="ul0084" list-style="none"><li id="ul0084-0001" num="0245">Set A, Set B, Set A, Set B, . . . (for a certain amount of time, e.g., 1 minute, 5 minutes, etc.)</li><li id="ul0084-0002" num="0246">Set A, Set C, Set A, Set C, . . . (for a certain amount of time, e.g., 1 minute, 5 minutes, etc.)</li></ul></li><li id="ul0083-0002" num="0247"> where A→B represents a V % change in motion and A→C represents a 100% (or almost 100%) change in motion.</li><li id="ul0083-0003" num="0248">(b) Apply the test video stream to the device under test that produces an output stream indicative of some value for each frame (this is the unknown information: does this value represent a statistical anomaly?);</li><li id="ul0083-0004" num="0249">(c) Monitor the output stream and fit it into either one of several, e.g., two behavior patterns (variable: e.g., “up, down, up”) or flat (constant: e.g., “up”). The detected behavior pattern is mapped to either a standard motion level detector or a statistical motion level detector associated with the device under test.</li></ul>
The above protocol may need to be conducted several times for different (e.g., increasingly higher) values of V to ensure reliable data collection, and eliminate the possibility of falsely declaring that the DUT exhibits statistical anomaly detection behavior when in fact it is a standard motion level detector with a motion level threshold M<sup>HYP</sup><sub>ON </sub>that is relatively high.
Therefore, it will be appreciated that the DUT has a processor which is configured such that when the input video stream is characterized by a sequence of oscillating frames having a % motion change of M from one frame to the next for a non-zero value of M that is less than 100%, the output stream is characterized by first change in the output stream that lasts for a certain number of oscillating frames in the video stream (the “burn-in”). This is followed by a change in the output stream to a second value, which from that point on ceases to change while the first sequence of frames in the input video stream continues to oscillate in the same way. In an example, M may be at least 25%, at least 50%, at least 90%, at least 95%, etc.
Then, when the input video stream includes a new, second sequence of oscillating frames, this time having a 100% motion change from one frame to the next, the output stream is characterized by another change in the output stream (similar to the first change) that lasts for a certain number of oscillating frames in the input video stream. This followed by a fourth change in output value back to the second value mentioned above, which from that point on ceases to change while the video stream continues to include the second sequence of oscillating frames.
Of course, modifications may be made to the above protocol to meet operational requirements.
Applications
In the embodiment of <figref idref="DRAWINGS">FIG. 1A</figref>, the anomaly state output stream <b>125</b> can be stored in memory (e.g., a volatile or non-volatile storage medium) and used by the video management system <b>110</b>. In the embodiment of <figref idref="DRAWINGS">FIG. 1B</figref>, the anomaly detector <b>110</b>B may supply the anomaly state output stream <b>125</b> to the video management system <b>110</b>A. The video management system <b>110</b>/<b>110</b>A may then process the video data <b>135</b> based on the anomaly state output stream <b>125</b> and, depending on the application, take an appropriate action. Examples of such processing and actions are now described.
Example 1: Video Summary
In this example, processing the video stream data <b>135</b> may involve identifying “video segments of interest”. In an example embodiment, a “video segment of interest” may include any time window in the video data <b>135</b> for which the anomaly state meets certain criteria. This information can be obtained from the anomaly state output stream <b>125</b> (e.g., data sets <b>125</b>A, <b>125</b>B, <b>125</b>C, . . . ).
The criteria could be, for example: <ul id="ul0085" list-style="none"><li id="ul0085-0001" num="0000"><ul id="ul0086" list-style="none"><li id="ul0086-0001" num="0258">An anomaly state of interest: <ul id="ul0087" list-style="none"><li id="ul0087-0001" num="0259">The anomaly state equals A+;</li><li id="ul0087-0002" num="0260">The anomaly state equals A+ or A−;</li></ul></li><li id="ul0086-0002" num="0261">An anomaly state transition of interest: <ul id="ul0088" list-style="none"><li id="ul0088-0001" num="0262">The anomaly state equals to A+ and is different from the anomaly state of the previous time window (e.g., the anomaly state has transitioned from A0→A+);</li><li id="ul0088-0002" num="0263">The anomaly state equals to A− and is different from the anomaly state of the previous time window (e.g., the anomaly state has transitioned from A0→A−).</li></ul></li></ul></li></ul>
As such, the video management system <b>110</b>/<b>110</b>A may be configured to identify the A0→A+ (or A0→A−) anomaly state transitions from the anomaly state output stream <b>125</b>, and to extract the corresponding time windows from the video data <b>135</b>, as well as the time windows for which the anomaly state remains A+ (or A−) and possibly one or more time windows preceding or following the transition. These time windows form the “video segment of interest”, which can then be displayed on the screen <b>140</b> for viewing by an operator <b>150</b> or stored in memory or even played back simultaneously with the video stream itself so as to allow the operator (e.g., a security guard) to see both live stream video and a video summary simultaneously, e.g., after a long break from his or her desk.
In some cases, only transitions to the A+ anomaly state may be of interest. As such, the video management system <b>100</b>/<b>110</b>A can be configured to record video continuously, monitor the anomaly state of various time windows and, in response to an anomaly state of A+ for a given time window, pull out a recording that goes back a certain number of seconds before the start of the given time window, and display this recording for a certain number of seconds after the end of the given time window and/or until the monitoring of the recorded video reveals an anomaly state of A0 (the initially anomalous behavior of the scene becomes the new normal) or A−.
The video management system <b>110</b>/<b>110</b>A can be configured to create a “video summary” of the video data <b>135</b> by concatenating the “video segments of interest”. One implementation is to summarize a video for when it is in anomaly state A+ or A− but ignoring when it is in anomaly state A0; however, other implementations are possible, depending on operational requirements.
A non-limiting example is shown in <figref idref="DRAWINGS">FIG. 7A</figref>, where a video summary <b>720</b> includes a concatenation (stitching) of individual video segments of interest of a plurality of video streams, each video segment of interest having been extracted from either a video stream V<b>1</b>, a video stream V<b>2</b> or a video stream V<b>3</b>. The video streams V<b>1</b>, V<b>2</b>, V<b>3</b> may be from the same camera during different periods or from different cameras during the same time period or different time periods. As such, the video summary <b>720</b> may include video data from a single camera or video data from multiple cameras. Headers containing specially marked data may be generated by the video management system <b>110</b>/<b>110</b>A and included in the video summary <b>720</b> so as to allow a receiver to determine the identity of the camera and its video stream (e.g., V<b>1</b>, V<b>2</b>, V<b>3</b>), as well as possibly the period of real time covered by that segment of interest. In an alternative embodiment, rather than creating a single video summary containing segments of interest from multiple video streams, individual video summaries could be created, each containing concatenated segments of interest of a corresponding one the video streams (e.g., V<b>1</b>, V<b>2</b>, V<b>3</b>).
Each video stream V<b>1</b>, V<b>2</b>, V<b>3</b> is shown as being in one of three anomaly states A+, A0 or A−, at any given point in time. Video segments of interest extracted from the video streams V<b>1</b>, V<b>2</b>, V<b>3</b> (forming part of the video summary <b>720</b>) could be those time windows during which the corresponding video stream is in anomaly state A+ or A− (or, equivalently, not in anomaly state A0). In the illustrated embodiment, an attempt has been made at placing the video segments of interest more or less in chronological order, but this is not a requirement when creating the video summary <b>720</b>. A data element <b>710</b> precedes each segment where there is a change in the source video stream for that video segment of interest, and this data element <b>710</b> may include metadata such as an indication of the source of the video stream of the subsequent video segment of interest.
In this way, the video streams V<b>1</b>, V<b>2</b>, V<b>3</b> are condensed by the video management system <b>110</b>/<b>110</b>A so as to keep only the time windows following an indication of a statistically abnormal change in motion (i.e., a statistically abnormal increase or a statistically abnormal decrease) until there has been a return to statistically non-abnormal (i.e., statistically normal) motion (anomaly state A0). Thus, the video management system <b>110</b>/<b>110</b>A provides a programmatically compiled video summary <b>720</b> that may be stored by the video management system <b>110</b>/<b>110</b>A in a separate memory. The condensed nature of the video summary <b>720</b> facilitates viewing by the operator <b>150</b> and has the potential to increase the relevance of what is presented to the operator <b>150</b>.
In another embodiment, the video segments of interest extracted from the video streams V<b>1</b>, V<b>2</b>, V<b>3</b> to form the video summary <b>720</b> could be those time windows during which the corresponding video stream has changed to anomaly state A+ or A− from anomaly state A0, including one or several time windows before and one or several time windows after the change.
In a variant, the video management system <b>110</b>/<b>110</b>A may query a database (e.g., database <b>510</b>) for a subset of time windows in a desired time range and then the video summary may be created from the time windows (and images samples) within the desired time range. The desired time range may be a user input to the video management system <b>110</b>/<b>110</b>A. In another variant, shown in <figref idref="DRAWINGS">FIG. 7B</figref>, the video management system <b>110</b>/<b>110</b>A may be configured to allow the operator <b>150</b> to enter anomaly criteria <b>750</b>, such as one or more desired anomaly states (A+, etc.) or anomaly state transitions (A0→A+, A−→A0, etc.), and to enter video stream criteria <b>760</b>, such as the identity of one or more desired video streams (e.g., V<b>1</b>, V<b>2</b>, V<b>3</b>) stored in the database <b>510</b>. The video management system <b>110</b>/<b>110</b>A is configured to access the database <b>510</b> and apply the user-defined criteria <b>750</b> to the user-selected video streams <b>760</b> in order to create a video summary (e.g., video summary <b>720</b>) which is then stored, displayed or otherwise output.
In a further variant, the video management system <b>110</b>/<b>110</b>A may be configured to create a file containing ancillary data relating to the video data <b>135</b>. Such ancillary data could include timestamps of “video segments of interest” in the video streams V<b>1</b>, V<b>2</b>, V<b>3</b>; for example, these could be segments containing time windows for which the anomaly state is indicative of a statistically abnormal change in motion (i.e., a statistically abnormal increase or a statistically abnormal decrease), along with the identity of the video stream (e.g., V<b>1</b>, V<b>2</b>, V<b>3</b>). This results in storing, as ancillary data, a video stream identifier and one or more time window identifiers indicative of the time windows (e.g., time stamps or packet IDs, etc.) in the corresponding video stream (identified by the video stream identifier) where abnormal motion can be found. In this way, the video streams V<b>1</b>, V<b>2</b>, V<b>3</b> can be archived in one memory and the ancillary-data-containing file can be stored elsewhere (e.g., in another memory) and/or organized differently, without necessarily having to create or store a video summary.
Example 2: Composite Stream
The video management system <b>100</b>/<b>110</b>A can be configured to (i) monitor the anomaly state of each time window of the video data <b>135</b>, as provided by the anomaly state output stream <b>125</b>, and (ii) insert this anomaly state into metadata for the video data for the appropriate time window. In this way, a “composite stream” can be provided, which includes metadata on top of raw video, and the composite data may be stored/archived. For example, with reference to <figref idref="DRAWINGS">FIG. 5</figref>, a database <b>510</b> stores a composite stream that is broken down into time windows, each identified by Frame ID. Each time window is associated with actual video data and metadata (such as the anomaly state). In some embodiments, additional metadata of a statistical nature can be added, such as p-values and various other statistical parameters. As such, in the aforementioned scenarios for creating a video summary, the video management system <b>110</b>/<b>110</b>A may have access to the database <b>510</b> (e.g., over a data network <b>550</b> such as the internet) rather than separately accessing the video data <b>135</b> and the anomaly state output stream <b>125</b>.
Example 3: Carousel
In some embodiments, the video management system <b>110</b>/<b>110</b>A carries out a method whereby a first plurality of media streams is received, the anomaly state of each of the media streams is determined, and then a second plurality of media streams is selected from the first plurality of media streams based on (i) the anomaly state of each of the first plurality of media streams and/or (ii) changes/transitions in the anomaly state of each of the first plurality of media streams. The second plurality of media streams is displayed on a set of screens, windows or tiles.
In a specific non-limiting, there are multiple (e.g., tens or hundreds or thousands of) cameras (such as the camera <b>130</b>) feeding into the video management system <b>110</b>/<b>110</b>A. By utilizing the A+ anomaly states and transitions into and out of this anomaly state, a video monitoring screen page can be populated with a plurality of (e.g., <b>16</b>) camera feeds which are managed (e.g., automatically carouselled (e.g. 1, 2, . . . , 16, 1, 2, . . . )) to show only “video segments of interest” associated with anomalous increases in motion (A+ anomaly states). As a new “anomaly up” state is detected, the camera on which this is occurring takes the place of the “oldest” camera on the carousel. This may facilitate the manner in which the operator <b>150</b> can monitor possibly hundreds or thousands of live video cameras, namely by emphasizing or only presenting to the operator <b>150</b> those cameras where anomalous increases motion have been found to occur (thus potentially allowing efficient identification of scenes that have escalated in activity). The same can be done for presenting or emphasizing to the operator <b>150</b> those cameras where anomalous decreases motion have been found to occur (thus potentially allowing the identification of cameras that have been tampered with, such as by covering the lens or having been oriented towards the sky).
<figref idref="DRAWINGS">FIG. 11A</figref> conceptually illustrates the aforementioned carousel in the form of a table <b>1100</b> showing an instantaneous association between cameras and screens. Specifically, the table includes a set of records. Each record includes a “screen” entry (one of, say, 16 screens of the video monitoring screen page), a camera entry (the camera shown on the screen, which may be one of, say, 256 cameras) and a time stamp entry (the time when that screen began showing the camera with which it is currently associated). The number of screens and cameras is merely used as an example. In an actual implementation, any scenario where the number of screens is less than the number of cameras would work in a similar way.
<figref idref="DRAWINGS">FIG. 11B</figref> shows a flowchart for a process that may be carried out by the video management system <b>110</b>/<b>110</b>A in the context of Example 2. The process may be carried out at every time window, e.g., each time there is a new time window and a new anomaly state for that time window is available for inspection. Three possible anomaly states are assumed for the purposes of this non-limiting example: A−, A+ and A0. Two variables are used, namely “camera” and “screen”. The camera variable is cycled through; accordingly, at step <b>1100</b>, the camera variable is initialized to its initial value, e.g., 1, to denote the “first” camera. At step <b>1120</b>, the anomaly state of the time window is inspected (e.g., from the anomaly state output stream <b>125</b>) and it is determined whether it has changed from A+ to A0 or vice versa. If neither is true, the next step is step <b>1140</b>, whereby the camera variable is incremented so that the next camera is considered at step <b>1120</b>.
If step <b>1120</b> reveals that, for a given camera, the anomaly state has changed from A+ to A0, then this means that there is in fact no longer a reason to show the video feed from the camera currently designated by the camera variable camera on the monitoring screen page. Accordingly, the next step is step <b>1150</b>, where the screen that had been showing this video feed (denoted as screen<sup>−1</sup>(camera)) is reset so as to make it “available” for the display of a video feeds from another camera. Practically speaking, this could result in clearing the camera entry from the row of the table <b>1100</b> associated with screen<sup>−1</sup>(camera). The next step is step <b>1140</b> where the camera variable is incremented.
If it is revealed that the anomaly state has changed from A0 to A+, then the next step is step <b>1130</b>, where it is verified whether there is an available screen, i.e., not all screens on the video screen monitoring page are occupied. If this is the case, the next step is step <b>1160</b>, where an association is created between the available screen and the camera currently designated by the camera variable. Practically speaking, this could result in filling the camera entry for the row of the table <b>1100</b> belonging to the available screen, and also setting the corresponding time stamp entry to the current time stamp. The next step is step <b>1140</b> where the camera variable is incremented.
If, on the other hand, step <b>1130</b> reveals that there is no available screen, then it is envisaged that the “oldest” (e.g., longest-running) screen will start showing the video feed from camera currently designated by the camera variable. Accordingly, the next step is step <b>1170</b>, where the oldest screen is identified based on the various time stamp entries in the table <b>1100</b>. Thereafter, at step <b>1180</b>, an association is created between this “oldest” screen and the camera currently designated by the camera variable. Practically speaking, this could result in filling the camera entry for the row of the table <b>1100</b> belonging to the “oldest” screen (which is now of course no longer considered the oldest or longest-running screen), and also setting the corresponding time stamp entry to the current time stamp. The next step is step <b>1140</b> where the camera variable is incremented.
In other embodiments, a different criterion (one other than age) can be used for selecting which screen should be chosen to display the video feed from a camera that exhibits a new change in anomaly state from A0 to A+.
Example 4: Media Summary Based on User-Definable Criteria
In this example, and with reference to <figref idref="DRAWINGS">FIG. 12</figref>, the video management system <b>110</b>/<b>110</b>A implements a GUI-based media summary creation tool, which can be the result of a processor <b>1220</b> of the video management system <b>110</b>/<b>110</b>A executing computer-readable instructions (media summary creation tool instructions <b>1205</b>). The GUI-based media summary creation tool receives input from the operator <b>150</b> (e.g., via a graphical user interface <b>1200</b> on the display <b>140</b>). The GUI-based media summary creation tool also receives video data and anomaly state data. The video data may include video data <b>135</b>A for one or several cameras obtained from memory (e.g., database <b>510</b> or some other memory that may be internal to the video management system <b>110</b>/<b>110</b>A) and/or video data <b>135</b>B constituting a live feed from one or multiple cameras <b>130</b>. The anomaly state data may be provided in separate anomaly state output streams (e.g., stream <b>125</b>A, <b>125</b>B) or may be conveyed together with the video data <b>135</b>A, <b>135</b>B itself (e.g., in the form of metadata).
The GUI-based media summary creation tool is configured to process the video data from one or multiple cameras based on (i) the anomaly state of each time window of the video data, (ii) the operator input and (iii) possible additional metadata associated with each time window, thereby to result in curated video and event data. The GUI-based media summary creation tool is then configured carry out playback of the curated video and event data via the GUI <b>1200</b> on the display <b>140</b>. The GUI-based media summary creation tool is also configured to receive additional input from the operator <b>150</b> via the GUI <b>1200</b> during playback of the curated video and event data and to produce a media summary <b>1290</b> based on such additional input.
In a non-limiting embodiment, and with reference to <figref idref="DRAWINGS">FIG. 13</figref>, there is shown a GUI <b>1200</b> presented by the GUI-based media summary creation tool and including a selection window <b>1310</b>, a playback window and an event list window <b>1330</b>. The GUI-based media summary creation tool will now be described in terms of selection of system parameters & event criteria, playback of curated video & event data, and creation of the media summary <b>1290</b>.
a) Selection of System Parameters & Event Criteria
The selection window <b>1310</b> allows operator interaction with the GUI-based media summary creation tool. The selection window <b>1310</b> presents one or more system parameters and event criteria that the operator <b>150</b> may select (e.g., with a mouse or by using the display <b>140</b> if implemented as a touch screen).
A wide variety of selectable system parameters and event criteria may be presented by the GUI-based media summary creation tool in the selection window <b>1310</b> of the GUI <b>1200</b>. Non-limiting examples of system parameters may include: <ul id="ul0089" list-style="none"><li id="ul0089-0001" num="0000"><ul id="ul0090" list-style="none"><li id="ul0090-0001" num="0288">A desired or minimum or maximum number of “segments of interest” to be included in the eventually created media summary <b>1290</b> covering a given period of real time (e.g., include no less than 2 and no more than 10 segments of interest in the media summary covering 12:00 PM to 3:00 PM)</li><li id="ul0090-0002" num="0289">Absolute duration parameters (e.g., a desired or minimum or maximum overall length) of the eventually created media summary <b>1290</b> (e.g., length of the media summary=2 minutes)</li><li id="ul0090-0003" num="0290">Relative duration parameters (e.g., a desired or minimum or maximum overall length) of the eventually created media summary <b>1290</b> covering a given period of real time (e.g., desired length of the media summary covering 6:00 PM to 7:00 PM=2 minutes)</li><li id="ul0090-0004" num="0291">A desired or maximum or minimum duration of a segment of interest to qualify it for inclusion in the eventually created media summary <b>1290</b></li><li id="ul0090-0005" num="0292">Threshold sensitivity (e.g., the values of A and B in the above-described calculation of p-values): this may have an impact on the frequency of anomaly state transitions;</li><li id="ul0090-0006" num="0293">Etc.</li></ul></li></ul>
Non-limiting examples of event criteria may include video-based event criteria.
Evaluating whether a video data associated with a given camera meets these criteria is determined by processing the video data. Such criteria may include, for example: <ul id="ul0091" list-style="none"><li id="ul0091-0001" num="0000"><ul id="ul0092" list-style="none"><li id="ul0092-0001" num="0296">Whether the video data associated with a particular camera is in an anomaly state(s) of interest (e.g., one or more of A+, A−, A0, . . . )</li><li id="ul0092-0002" num="0297">Whether the anomaly state of the video data associated with a particular camera undergoes a transition(s) of interest (e.g., A0→A+, A++→A+, . . . )</li><li id="ul0092-0003" num="0298">Whether the video data associated with a particular camera includes contextual metadata of interest (e.g., presence of a cat, presence of a dog, presence of a cat and dog, presence of no animal)</li><li id="ul0092-0004" num="0299">Combinations of statistical and non-statistical criteria (e.g., (((“A+” OR “A−”) AND “other criterion 1”) OR “other criterion 2”))</li><li id="ul0092-0005" num="0300">Filters (e.g., only low-light/high-light scenes, only scenes where motion lasts more/less than 30 seconds, etc.)</li><li id="ul0092-0006" num="0301">Whether the video data associated with a particular camera contains motion, including motion above or below a particular threshold level of motion</li><li id="ul0092-0007" num="0302">Whether the video data associated with a particular camera contains a face, including a recognized face from a database or absence from the database, or a minimum number of faces simultaneously in a particular image sample or time window</li><li id="ul0092-0008" num="0303">Whether the video data associated with a particular camera contains a particular object or object class;</li><li id="ul0092-0009" num="0304">Whether the video data associated with a particular camera contains a license plate, e.g., a license plate whose number is listed (or not listed) in a given database stored in memory or online; etc.</li></ul></li></ul>
Non-limiting examples of event criteria may also include non-video-based event criteria that is associated with a given camera, which may ultimately trigger display of the video data associated with the given camera, such as: <ul id="ul0093" list-style="none"><li id="ul0093-0001" num="0000"><ul id="ul0094" list-style="none"><li id="ul0094-0001" num="0306">Events based on access control inputs, for example detected badging events (such as access grants, access denials, etc.) in an area associated with a particular camera;</li><li id="ul0094-0002" num="0307">Events based on vehicle data, for example detected bus or train arriving at a station in an area associated with a particular camera</li><li id="ul0094-0003" num="0308">Events based on building conditions, for example smoke alarm, fire alarm, forced door entry, etc</li><li id="ul0094-0004" num="0309">Events based on weather conditions (e.g., temperature or humidity reaching a certain upper or lower threshold)</li><li id="ul0094-0005" num="0310">Etc.</li></ul></li></ul>
In some embodiments, the system parameters and the event criteria may be provided in the form of one or more checklists, e.g., in the selection window <b>1310</b>. In other embodiments, one or more of the system parameters or event criteria may be associated with a pre-defined set of possible values that may be presented in a certain way (e.g., drop-down menu, radio button, . . . ) and from which the operator <b>150</b> may make a selection. In other embodiments, one or more of the selectable system parameters or event criteria may be entered by the operator <b>150</b> in a free-form manner.
Other more conventional system parameters (e.g., “resolution”, “input language”) and event criteria can also be made available to the operator <b>150</b>. Of course, operator input is not restricted to be entered via the selection window <b>1310</b> and instead may be entered by another mechanism, such as via a different window using a keyboard or mouse, for example.
b) Playback of Curated Video & Event Data
In the present non-limiting embodiment, the GUI-based media summary creation tool processes (i) the video streams from the cameras <b>130</b> and/or the database <b>510</b> and (ii) the anomaly state of each video stream (which may be in the video data <b>135</b>A, <b>135</b>B or in a separate stream <b>125</b>A, <b>125</b>B) in accordance with the selected system parameters and event criteria in order to create (i) curated video streams that are played back in the playback window <b>1320</b>; and (ii) a running list of events that is presented in the event list window <b>1330</b>. This provides a rough draft video summary that the operator can then evaluate for suitability.
With reference to <figref idref="DRAWINGS">FIG. 13</figref>, the playback window <b>1320</b> is shown to include one or more tiles of curated video (i.e., video data that is in compliance with the selected system parameters and event criteria). The number of tiles may correspond to the total number of cameras being monitored by the video management system <b>110</b>/<b>110</b>A, but this need not be the case. A timestamp <b>1322</b> may also be shown for one or more of the tiles, which could be representative of the time at which the currently displayed frame of video data was captured, which may not be the current system time. A progress bar <b>1324</b> may be provided for one or more of the tiles in order to represent the time within the curated video and the corresponding time in the original video data <b>135</b>. In case the video data does not match the system parameters or the event criteria, the playback window <b>1320</b> may be left blank or dark.
In other embodiments, where there are more cameras than displayable tiles, the GUI-based media summary creation tool may provide a mechanism (e.g., a management algorithm) for displaying the curated video for each camera sequentially. For example, assume there is only one tile available. Then, for a given overall period of time X to Y, the tile may display the curated video for camera <b>1</b> (covering time X to Y), followed by the curated video for camera <b>2</b> (covering time X to Y), and so on. This cycling through the different cameras works well where, on average, the total length of all curated video (for all cameras sharing the same tile) is less than the elapsed real time between time X and time Y, otherwise more severely curated video may be required. This can be achieved by selecting more stringent system parameters or event criteria.
In another embodiment, curated video from multiple cameras can be toggled and displayed on the same tile according to order of event occurrence. In other words, as soon as a new event is detected in the video data from camera X, the tile displays video from camera X, until an event is detected in the video from camera Y, at which point the tile displays the video from camera Y, and so on.
With continued reference to <figref idref="DRAWINGS">FIG. 13</figref>, the event list window <b>1330</b> is shown to include a running list of events detected in the video from one or more cameras. This list may be made sortable, exportable and transmittable, by the GUI-based media summary creation tool. The running list may include a camera identifier, a time at which the detected event started, a duration of the event, a type of the event and a description if applicable. Generally speaking the event list conveys those occurrences where an event matching the event criteria is found in the video stream being processed.
In some embodiments, cameras may be associated with geographic locations (known ahead of time or obtainable from the metadata). In this case, the geographic location of the camera may be added to the running list of events displayed in the event list window <b>1330</b>. In another embodiment, a map window (not shown) may include a map that shows locations associated with a plurality of cameras, and each time an event that matches the event criteria is detected by the system, a location on the map in the map window is emphasized (e.g., flashes, changes color, etc.). The map may provide a link to the video data, allowing the operator to touch or click on a location and cause the video data associated with that location to be displayed in a corresponding tile in the playback window <b>1320</b>.
For example, if one of the event criteria is a transition from A0 to A+ anomaly state, and if this transition is detected on the video data from camera <b>5</b>, then an icon positioned at the location of camera <b>5</b> on the map n the map widow may be caused to flash or light up. The associated video data may be played back as curated video in the playback window <b>1320</b>.
Those skilled in the art will appreciate that non-video event data may also be processed by the GUI-based media summary creation tool. For example, non-video event data may include data obtained from access control inputs (badging IDs), weather data, vehicle data (bus or train arriving in station), smoke/fire alarm data, and the like. Some of the non-video event data may be associated with a respective geographic location. As such, the non-video event data can be associated with an icon on a map information. The GUI-based media summary creation tool may be configured to synchronize the video data with the non-video event data so as to display, in the display window <b>1320</b>, images or video from those cameras closest to the non-camera events.
In another embodiment, a plurality of video streams associated with a plurality of cameras is obtained and the anomaly state associated with each of the video streams is determined, e.g., on a time window by time window basis, either by computing it or in an accompanying anomaly state output stream. Based on the one or more criteria, occurrences in each video stream are identified. These occurrences indicate that the one or more criteria are met by the anomaly state. An action is taking involving these occurrences could be displaying a map of icons corresponding to the cameras (e.g., at their locations) and then emphasizing a particular one of the camera icons in response to determining one of the aforementioned occurrences, i.e., in the video stream from the camera corresponding to the particular one of the icons.
It is also noted that the GUI-based media summary creation tool may be configured to allow the operator <b>150</b> to change system parameters and event criteria during playback. For example, if the operator lowers the sensitivity, the video shown in the playback window <b>1320</b> would show a greater number of highlights, and then the operator can choose to return to a higher sensitivity at a later point in time. Thus, through the GUI <b>1200</b>, a user is able to dynamically change the event criteria.
c) Creation of the Media Summary <b>1290</b>
The media summary <b>1290</b> may take on many embodiments. In one example, the media summary <b>1290</b> may simply be a concatenation of the curated video shown in the tiles of the playback window <b>1320</b>, and encoded in a file.
In another example, the media summary <b>1290</b> may include frames of the curated video shown in the tiles of the playback window <b>1320</b>, together with occasional frames of video from the original video data <b>135</b> that did not meet the event criteria. These occasional frames could represent a decimated/accelerated version (e.g., every Xth frame) of the video not considered to meet the event criteria yet still useful to provide context to the operator <b>150</b>. This would allow the operator <b>150</b> to view all segments of interest at their normal rate and to view video that is between segments of interest at an accelerated rate. This would give the operator at least some contextual sense of what is happening between adjacent video segments of interest. It is also envisaged that the segments of interest could also be accelerated, but at a different rate than the video data between segments of interest.
In another example, the media summary <b>1290</b> may include the curated video shown in the tiles of the playback window <b>1320</b>, together with the associated event data as displayed in the event list window <b>1330</b>.
In another example, the media summary <b>1290</b> may include the curated video shown in the tiles of the playback window <b>1320</b>, together with, for each frame of video data, a snapshot or image of the progress bar that accompanies that frame in the playback window <b>1320</b>.
In another example, the media summary <b>1290</b> may not include any video data, but may simply include the event list.
In another example, the media summary <b>1290</b> may include a progress bar that has multiple entries along the progress bar to show the occurrence of events over time. The entries could be color coded to different types of events. This allows the viewer to gain an overall sense of the behavior of a time segment of a particular camera without having to scroll through video. Of course, the associated video data could be made available and/or linked to the data in the media summary <b>1290</b>.
Those of skill in the art will appreciate that there are other numerous other feasible examples of the media summary <b>1290</b>, which could range from data that includes only a video summary to data element that includes a video summary and other information to a data that does not include video at all.
The GUI-based media summary creation tool may include various “manual” functions for the operator <b>150</b> to have control over creation of the media summary <b>1290</b> being created for dissemination or further investigation (and stored in memory). Such manual functions may include, for example: <ul id="ul0095" list-style="none"><li id="ul0095-0001" num="0000"><ul id="ul0096" list-style="none"><li id="ul0096-0001" num="0333">Playback tool with the GUI <b>1200</b> to allow the operator <b>150</b> to select a camera view (e.g. from the split screen) either to enlarge, or to keep in the media summary, or to discard.</li><li id="ul0096-0002" num="0334">Add extra time before or after highlights forming part of the media summary.</li><li id="ul0096-0003" num="0335">Crop video and digitally zoom.</li><li id="ul0096-0004" num="0336">Other edits and tweaks.</li></ul></li></ul>
d) Variants
In some embodiments, the GUI-based media summary selection tool may automatically pre-select default values of one or more of the system parameters or event criteria, which are then modified by the operator <b>150</b>. For example, the “sensitivity” system parameter (e.g., the values of the A and B thresholds in the calculation of p-values) may be a variable parameters that is selected based on, e.g., the length of the resulting curated video summary, or on a ratio of the length of the resulting curated video summary relative to the overall duration of the video data in a given period of time. This ratio can be referred to as a compression ratio.
As such, the default values of A and B may be selected so as to give a total duration of, say between 30 seconds and 2 minutes, or a compression ratio of, say, between 5% and 10%. This would result in an initial attempt at finding the most relevant 30-120 seconds or 5%-10% of the video data. However, as optimal values of A and B may depend on the actual content of the video data (depending on the type of application), the values of A and B may be further adjusted by the operator <b>150</b> using the GUI-based media summary creation tool.
In the example case where one of the system parameters is a duration parameter (e.g., a maximum duration of the eventually created media summary <b>1290</b>), the GUI-based media summary creation tool may adjust various other system parameters or event criteria to meet the duration criteria. As discussed above, this could include adjusting the sensitivity (e.g., thresholds for p-values) until the ensuing media summary has the desired duration. Alternatively or in addition, this could include adjusting event criteria in an application-dependent fashion. For example, in a building security application, if the media summary resulting from the selected sensitivity would be too long and needs curtailing, the GUI-based media summary creation tool may adjust selection of the segments of interest to include only those that are temporally in the vicinity (a few frames before, during and a few frames after) of a badging event or, if still too long, only those segments of interest temporally in the vicinity of a denial of entry. In some embodiments, the GUI-based media summary creation tool does not determine the anomaly state of the video streams at all. As such, the aforementioned example may apply but without the notion of “sensitivity” impacting the manner in which the media summary <b>1290</b> is created. The video-based and non-video-based event criteria may be modified by the user or by the system itself in order to achieve the required absolute or relative duration parameters.
As such, there has been described a multi-step adjustment process for determining the values of the sensitivity and other event criteria, including an initial selection computed algorithmically by the system <b>110</b>/<b>110</b>A, followed by a second adjustment that is system-driven or user-based.
Those skilled in the art will appreciate that in some embodiments, creation and/or display of a media summary may be further influenced by characteristics other than operator-selected criteria. In particular, this could include the ID of the operator and his/her associated user type or security level, for example. In this case, the media summary that is created or displayed may differ depending on the user type of the operator. For example, in a corporation where the video data includes metadata indicative of the department where the video is being filmed, the media summary that is suitable for viewing by a security guard may include only video segments associated with anomalous motion, whereas the media summary suitable for viewing by the manager of the department may include all motion, anomalous or not, but restricted to having taken place in that department.
Those of skill in the art will appreciate that some embodiments may provide for a multi-camera implementation. In such a case, the same selected criteria used for creating the media summaries may be applied across multiple cameras, or individualized criteria may be selected on a per-camera basis.
Example 5: Analysis of Anomaly States
The video management system <b>110</b>/<b>110</b>A may be further configured to process the video data <b>135</b> and the metadata associated with the video data <b>135</b>.
For example, in response to operator input, the video management system <b>110</b>/<b>110</b>A may be configured to process the anomaly state output stream <b>125</b> which, it is recalled with reference to <figref idref="DRAWINGS">FIG. 10</figref>, can include a stream of data sets <b>125</b>A, <b>125</b>B, <b>125</b>C, . . . , each data set including an ID (“Time Window ID”) of a given time window of the video data <b>135</b> and the anomaly state for the given time window.
In one example, based on the anomaly state of each time window, the video management system <b>110</b>/<b>110</b>A may determine the amount of time (relative or absolute) that the video data <b>135</b> is in a particular anomaly state. This can be converted into a percentage of time during which the video in anomaly state A+ or A− (which may be referred to as a “compression ratio”). The compression ratio may give the operator <b>150</b> a quick and informative overview of the state of the detection algorithm as well as possibly information about the state of the scene that the camera is recording.
In another example, the GUI-based media summary creation tool may be configured to provide the user with an opportunity to select the compression ratio. In response, the video management system <b>110</b>/<b>110</b>A selects the A and B thresholds for the A+ and A− anomaly states so that the percentage of time during which the video was in anomaly state A+ or A− matches the user-selected compression ratio. The user may similarly be given the opportunity to select a range of compression ratios and the video management system <b>110</b>/<b>110</b>A selects the A and B thresholds for the A+ and A− anomaly states so that the percentage of time during which the video was in anomaly state A+ or A− (i.e., the compression ratio) matches the user-selected compression ratio. In another example, based on the anomaly state of each time window, the video management system <b>110</b>/<b>110</b>A may determine the number, frequency, and time of transitions between anomaly states. As such, looking at how often a video stream transitions from, say, anomaly state A0 to anomaly state A+ and similarly from anomaly state A0 to anomaly state A− would give some indication of how much “variability” is present in the recorded scene. It also could give the operator <b>150</b> an estimate on what time of day needs to be most investigated, for example. In this context, the video management system may output a histogram over the frequency of state transitions per the time period of a day.
In some embodiments, processing of the anomaly states may allow the video management system <b>110</b>/<b>110</b>A to select default values of one or more of the selectable system parameters and event criteria (see description of the GUI-based media summary creation tool in Section 4 above). For example, consider possible anomaly states of A−−, A−, A0, A+ and A++. If the ratio of the number of time windows in anomaly state A+ or A++ to the total number of time windows is X but the ratio of the number of time windows in anomaly state A++ to the total number of time windows is Y, where Y is within a pre-determined range (e.g., from 0% to 20%), then the GUI-based media summary creation tool may be configured to consider as curated video only those time windows associated with an anomaly state of A++. Of course, after such automated initial selection, the operator <b>150</b> may be free to select with greater specificity which anomaly states are monitored for the purposes of creating the media summary.
Example 6: Mappable Event Data
In another non-limiting example embodiment, shown with reference to <figref idref="DRAWINGS">FIGS. 14A-14C</figref>, the video management system <b>110</b>/<b>110</b>A implements a security application. The security application is configured to gather georeferenced event data (e.g. door openings, camera motion, 911 calls (for a city-wide system), license plate reads, license plate hit-list hits, motion detection, etc. . . . ). The georeferenced event data may be in the form of multiple event data streams from multiple sources, including cameras and non-camera sensors (e.g., 911 center, license plate readers, building motion detectors, . . . ).
As such, it will be appreciated that some of the event data streams include video data and others do not.
Each of the event data streams may be associated with a geographic location, which may be fixed or variable. In some cases, the geographic location associated with an event data stream may be a fixed geographic location where the camera or non-camera sensor for that event data stream is known to be placed. In such cases, the geographic location may be determined by consulting a database <b>510</b> that associates cameras and non-camera sensors with geographic locations. In other cases (e.g., 911 calls), the geographic location associated with an event data stream is a variable information element that is provided with the event data itself. Such may be the case with an event data stream from a 911 center which forwards data regarding 911 calls, including the nature of the call and the location where each call originated.
The security application is configured to provide a display that represents various event data streams associated with geographic locations within a region of interest. In a non-limiting embodiment, the screen may include several sections, including a map <b>1410</b> and a display window <b>1415</b>. The map <b>1410</b> and the display window <b>1415</b> share the same display screen so as to be simultaneously displayed on the display <b>140</b>.
The map <b>1410</b> may be used to represent various data, particularly non-video data. This could include icons representing event data streams from non-camera sensors. For instance, a license plate detector operating at location XYZ may be represented on the map <b>1410</b> by a license plate graphic <b>1412</b> containing the names of the owners of the last N vehicles whose license plates were detected by the license plate reader. In another example, in the case of an event data stream from a 911 center indicating a reported gunshot at a particular address within the region of interest, then this may be shown by placing a gun icon <b>1414</b> on the map <b>1410</b> in the vicinity of the particular address. In addition, the map <b>1410</b> may include icons (shown as stars <b>1416</b>) placed on the map to indicate where those cameras are physically located.
The display window <b>1415</b> may be used to simultaneously present video data from one or more cameras, including those in the region of interest. In some embodiments, portions of the display window <b>1415</b> may be activated by contacting or selecting portions of the map <b>1410</b>. In some embodiments, the display window <b>1415</b> includes a number of sub-windows <b>1420</b>, <b>1430</b>, <b>1440</b>, <b>1450</b>.
In some embodiments, the security application can be viewed as being in one of a plurality of fundamental states at any given time: State 1 (“live”), State 2 (“focused”) and State 3 (“catch-up”). Transitions can occur between states, as now described in greater detail with additional reference to <figref idref="DRAWINGS">FIG. 16</figref>.
In State 1 (“live”), the security application processes the event data streams and populates the region of interest on the map <b>1410</b> with indications of events in the event data streams and locations of cameras. In addition, video data from the cameras in the region of interest is presented in the display window <b>1415</b>. In the case where there are more cameras in the region of interest than there are cameras that may be simultaneously presented in the display window, a camera management/multiplexing algorithm <b>1460</b> (such as a carousel) for presenting a greater number of cameras on a smaller number of display screens may be carried out. Other multiplexing techniques besides casrouselling can be used.
The security application may be configured so as to provide the operator <b>150</b> with an ability to select one or more cameras for more focused viewing. This selection process may have the effect of labeling the selected cameras as “active”, with the others being labeled as “passive”, and results in the security application transitioning from State 1 (“live”) to State 2 (“focused”).
For instance, one way to transition from State 1 to State 2 is for the operator <b>150</b> to explicitly identify (e.g., by tapping the map <b>1410</b> or using a keyboard or mouse) the cameras that he or she wishes to view in the display window <b>1415</b>. The selected cameras are labeled as “active” and their presentation in the display window overrides the carouseling or other multiplexing algorithm <b>1460</b> that may have been implemented by the security application in State 1. Other (non-selected) cameras are labeled as “passive”. A resettable “focused mode” flag may be set and presented to the operator <b>150</b> to indicate that there has been an override of the cameras being shown in the display window <b>1415</b> with only active cameras. In other words, with the “focused mode” flag being set, passive cameras are not being presented in the display window <b>1415</b>. This implies that there exists a possibility that activity of interest may be occurring elsewhere in the region of interest (and caught by a passive camera) without the operator <b>150</b> being given an opportunity to see it unfold.
Another way to transition from State 1 to State 2 is for the operator <b>150</b> to zoom in on a sub-region <b>1490</b> of the region of interest. To achieve this, the operator <b>150</b> may zoom in using a pair of fingers (as with zooming into a picture on a smart phone) or the operator <b>150</b> may use a mouse, keyboard, microphone or other input device to achieve the required zoom onto the sub-region <b>1490</b>. This has the effect of automatically labeling those cameras in the sub-region <b>1490</b> of the region of interest <b>1408</b> as “active”, with the remaining ones (i.e., those in the region of interest but outside the sub-region <b>1490</b>) being labeled as “passive”. If the number of cameras in the sub-region <b>1490</b> is less than or equal to the number of sub-windows or cameras that can be shown in the display window <b>1415</b> (e.g., 3 in the sub-region <b>1490</b> versus 4 in the display window <b>1415</b>, as seen in <figref idref="DRAWINGS">FIG. 14B</figref>), then no carouselling or multiplexing is needed in order to present all active cameras in the display window <b>1415</b>.
Yet another way to transition from State 1 to State 2 is for the operator <b>150</b> to pause live camera viewing and use the security application for other functionalities. A pause functionality may be provided for this purpose. In this case, the number of active cameras is zero, and all cameras in the region of interest are considered passive.
Upon transitioning to State 2, the security application may be configured to record the time (HH:MM:SS) of the transition and store it in memory. Also, because there exists a possibility that activity of interest may be caught by a passive camera without the operator <b>150</b> being given an opportunity to see it unfold, the security application compiles a video summary of the video data associated with each of the passive cameras. The compiling of video summaries may begin with video data as of the time of the transition to State 2. It should be appreciated that video summaries can be created based not only on anomalous motion detected in the video data captured by the passive cameras but also by making links with other metadata and/or timelined events, such as detected gunshots, 911 calls about gunshots, speeding cars, etc. In fact, the operator <b>150</b> may be provided with an opportunity to specify in detail the criteria needed to be met by the video data retained as a video summary.
As such, during the time that the security application is in State 2 (“focused”), the passive cameras in the region of interest are picking up activity that may be potentially of interest (e.g., anomalous motion), and for which video summaries are being created. However, this activity of potential interest caught by the passive cameras will not be seen by the operator, as he or she is being presented the video data from the active cameras in the display window <b>1415</b>. In order to allow the operator to “catch up” on events that may have occurred during the operator's focused attention on the active cameras and caught by one or more of the passive cameras, the operator <b>150</b> may transition the security application from State 2 (“focused”) to State 3 (“catch up”).
There are numerous ways in which the security application may transition from State 2 to State 3, depending on the mechanism used to transition from State 1 to State 2. For example, in the case where a resettable “focused mode” flag had been set to indicate that the security application is in State 2, one way to transition from State 2 to State 3 is for the operator <b>150</b> to reset the “focused mode” flag. Alternatively, in the case where the operator <b>150</b> zoomed in to sub-region <b>1490</b> of the map <b>1410</b> to effect the transition from State 1 to State 2, the operator <b>150</b> may zoom out of the sub-region <b>1490</b>, thus transitioning into State 3. Also, in the case where the operator <b>150</b> may have transitioned to State 2 by pausing live camera viewing, the operator may provide a command to resume live camera viewing.
Once the security application is in State 3 (see <figref idref="DRAWINGS">FIG. 14C</figref>), the video summaries from the “passive” cameras are played back in the display window <b>1415</b>. If there are more cameras for which there is an available video summary (i.e., formerly passive cameras) than there are sub-windows in the display window <b>1415</b> for simultaneous display, then the security application may apply a management algorithm <b>1480</b> to select how to display the video summaries. In a non-limiting embodiment, this may result in the video summaries being displayed in sequence, one after the other. The video summaries may be played back from as far back as the time when the transition to State 2 occurred or was recorded. It should be appreciated that the video summaries represent time from the past and not present time. As such, during playback of the video summary from a particular camera, a time bar may show the playback time (with or without a reference to the present time) in order to inform the operator <b>150</b> as to what time of day he or she is looking at, and an icon may be emphasized on the map <b>1410</b>, corresponding to the geographic location of the particular camera, thereby to assist the operator <b>150</b> in knowing what part of the region of interest he or she is looking at.
It should be appreciated that with the security application being in State 3, the operator <b>150</b> has relinquished his or her selection of cameras for focused viewing and there are no more specifically active or passive cameras. Moreover, it is possible for action to be caught by any of the cameras while the video summaries are being presented in the display window <b>1415</b>. As such, the operator <b>150</b> may miss out on live action while catching up on earlier summaries. Accordingly, various embodiments are possible to allow the operator <b>150</b> to catch up on old video without missing out on ongoing activity.
In a first embodiment, the display window <b>1415</b> is split while the security application is in State 3. For example, rather than occupy the entire display window <b>1415</b>, the video summaries being played back may occupy a first section of the display window <b>1415</b> (e.g., sub-windows <b>1440</b>, <b>1450</b>), with live feeds from the various cameras in the region of interest being shown in a second section of the display window <b>1415</b> (e.g., sub-windows <b>1420</b>, <b>1430</b>). The security application may implement a first management algorithm <b>1480</b> to manage the display of video summaries in the first section and a second management algorithm <b>1460</b> to manage the display of live video data in the second section. The two management algorithms <b>1460</b>, <b>1480</b> may be interlinked so as to allow flexibility of control over the various sub-windows. This allows the operator <b>150</b> to view some live video data and some video summaries simultaneously. This may be advantageous in the case where the video summaries take some time to play out, and during this time while the operator <b>150</b> is “catching up” using the first section (sub-windows <b>1440</b>, <b>1450</b>), he or she continues to monitor some live camera feeds in the second section (sub-windows <b>1410</b>, <b>1420</b>), thereby allowing the operator <b>150</b> to witness live action that may require the operator's further attention.
In a second embodiment, playback of the previously created video summaries occupies the entire display window <b>1415</b>, but in the background, video summaries are being created from ongoing live action caught by the various cameras in the region of interest <b>1408</b>. The newly created video summaries (i.e., those created during playback of the previously displayed video summaries) are added to the list of video summaries that needs to be presented in the display window <b>1415</b>.
As such, the security application in State 3 (“catch-up”) allows the operator <b>150</b> to catch up with what he or she missed, in faster-than-real-time. At some point, the video summaries will catch up with the present time. Once this occurs, or at any other time based on manual input from the operator <b>150</b>, the security application can transition from State 3 (“catch up”) back to State 1 (“live”).
The above example use case has been described in order to illustrate certain basic concepts of video summary management in the context of a security application. Of course, other variants exist in order to accommodate more sophisticated usage patterns, such as where the operator wishes to vary the region of interest (and hence the cameras associated therewith) or zoom into a different sub-region of the region of interest before the security application has transitioned from State 3 back to State 1, etc. However, it is expected that those of skill in the art will be able to apply the teachings herein to develop a suitable mechanism for managing video summaries to meet operational goals.
Various operational embodiments are provided herein, including embodiments in which one or more of the operations described may correspond to computer readable instructions stored on one or more computer readable media, which if executed by a computing device, will configure the computing device to perform the operations described. Accordingly, and with reference to <figref idref="DRAWINGS">FIG. 6</figref>, such a computing device <b>610</b> may include a processing entity <b>612</b> and a memory <b>614</b> that stores computer-readable instructions <b>620</b>. The memory <b>614</b> may be implemented in a variety of ways, such as a magnetic disk, or solid state memory, and may include flash memory, SRAM, DRAM, phase-change memory and the like. The processing entity <b>612</b> is configured to execute the computer-readable instructions in the memory <b>614</b>. In doing so, the processing entity <b>612</b> of the computing device <b>610</b> causes the computing device <b>610</b> to implement a graphics pipeline. Examples of a processing entity may include electronic components such as a computer processor on a microchip. The graphics pipeline may be encoded as a subset <b>616</b> of the computer-readable instructions <b>620</b> in the memory <b>614</b>. An input/output (I/O) <b>618</b> enables the processing entity <b>612</b> to communicate externally and may include a screen (e.g., touchscreen), keyboard/mouse, network interface device/card (e.g., to support NFC, WiFi, Ethernet or cellular/GSM/LTE communications), USB port(s), etc.
Components of the computing device <b>610</b>, such as the processing entity <b>612</b> and the memory <b>614</b> and various other input and other output devices, may be connected by various interconnects, such as a bus. Such interconnects may include a Peripheral Component Interconnect (PCI), such as PCI Express, a Universal Serial Bus (USB), Firewire (IEEE 1394), an optical bus structure, and the like. In another embodiment, components of the computing device <b>610</b> may be interconnected by a network. For example, the memory <b>614</b> may be comprised of multiple physical memory units located in different physical locations interconnected by a network. Moreover, depending on the exact device configuration and type, the memory <b>614</b> may be volatile (such as RAM, for example), non-volatile (such as ROM, flash memory, etc., for example) or some combination of the two. The computer readable instructions stored in the memory <b>614</b> may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. Typically, the functionality of the computer readable instructions may be combined or distributed as desired in various environments.
Furthermore, the order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order-dependent. Alternative ordering will be appreciated by one skilled in the art having the benefit of this description. Further, it is possible that not all operations described in the context of one embodiment are necessary to permit operation of that embodiment.
It should also be appreciated that the aforementioned disclosure exposes various problems, the solution to which requires a computer.
Finally, although the disclosure has been shown and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the disclosure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application.
Contents6
23 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23
Every citation, both waysCites: the store holds 51 of 52
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US12418632B1 | Cited by | United States of America | Search report |
| US12387497B2 | Cited by | United States of America | Applicant |
| WO0161993A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2004088723A1 | Cites | United States of America | Applicant |
| JP2005094245A | Cites | Japan | Applicant |
| US2006109341A1 | Cites | United States of America | Applicant |
| US2012063641A1 | Cites | United States of America | Applicant |
| US2012189284A1 | Cites | United States of America | Applicant |
| US2012323925A1 | Cites | United States of America | Applicant |
| US2013147915A1 | Cites | United States of America | Search report |
| US2014350436A1 | Cites | United States of America | Applicant |
| US2016014482A1 | Cites | United States of America | Search report |
| US2016042621A1 | Cites | United States of America | Applicant |
| US2016248835A1 | Cites | United States of America | Search report |
| US2018075877A1 | Cites | United States of America | Search report |
| WO2018176162A1 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| US2019026882A1 | Cites | United States of America | Search report |
| US4652907A | Cites | United States of America | Applicant |
| US4979036A | Cites | United States of America | Applicant |
| US6493041B1 | Cites | United States of America | Applicant |
| US7075581B1 | Cites | United States of America | Applicant |
| US7286709B2 | Cites | United States of America | Applicant |
| US7324163B2 | Cites | United States of America | Applicant |
| US7535515B2 | Cites | United States of America | Applicant |
| US7925106B2 | Cites | United States of America | Applicant |
| US8102406B2 | Cites | United States of America | Applicant |
| US8189869B2 | Cites | United States of America | Applicant |
| US8275038B2 | Cites | United States of America | Applicant |
| US8345761B2 | Cites | United States of America | Applicant |
| US8437505B2 | Cites | United States of America | Applicant |
| US8553783B2 | Cites | United States of America | Applicant |
| US9158974B1 | Cites | United States of America | Applicant |
| US9160897B2 | Cites | United States of America | Applicant |
| US9460522B2 | Cites | United States of America | Applicant |
| US9473746B2 | Cites | United States of America | Applicant |
| US9502073B2 | Cites | United States of America | Applicant |
| US9615087B2 | Cites | United States of America | Search report |
| US9779303B2 | Cites | United States of America | Applicant |
| US20040088723A1 | Cites | United States of America | Applicant |
| US20060109341A1 | Cites | United States of America | Applicant |
| US20120063641A1 | Cites | United States of America | Applicant |
| US20120189284A1 | Cites | United States of America | Applicant |
| US20120323925A1 | Cites | United States of America | Applicant |
| US20130147915A1 | Cites | United States of America | Search report |
| US20140350436A1 | Cites | United States of America | Applicant |
| US20160014482A1 | Cites | United States of America | Search report |
| US20160042621A1 | Cites | United States of America | Applicant |
| US20160248835A1 | Cites | United States of America | Search report |
| US20180075877A1 | Cites | United States of America | Search report |
| US20190026882A1 | Cites | United States of America | Search report |
| JP2005094245 | Cites | Japan | Applicant |
| WO2001061993 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| WO2018176162 | Cites | World Intellectual Property Organization (WIPO) | Applicant |
| Fathy, M. and Siyal, Y. A window based image processing technique for quantitative and qualitative analysis of road traffic parameters, IEEE Transactions on Vehicular Technology, 1998, pp. 1342-1349. | Non-patent | – | Applicant |
| Jing G, Rajan D and Chng Eng Siong, Motion detection with adaptive background and dynamic thresholds, Information, Communications and Signal Processing, 2005,Fifth International Conference on. IEEE. | Non-patent | – | Applicant |
| Quen-Zong Wu, Hsu Yung Cheng and Bor-Shenn Jeng, Motion detection via change-point detection for cumulative histograms of ratio images, Pattern Recognition Letters 26.5 ,2005, pp. 555-563. | Non-patent | – | Applicant |
| Elhabian, El Sayed and Ahmed, Moving object detection in spatial domain using background removal techniques—state-of-art, Recent patents on computer science 1.1, 2008, pp. 32-54. | Non-patent | – | Applicant |
| Dimou, Nemethova and Rupp, Scene change detection for H. 264 using dynamic threshold techniques, 2005. | Non-patent | – | Applicant |
| Zhang and Liang, Motion human detection based on background subtraction, Education Technology and Computer Science (ETCS), 2010 Second International Workshop on. vol. 1. IEEE, 2010. | Non-patent | – | Applicant |
| Browne et al, Evaluating and combining digital video shot boundary detection algorithms, 2000. | Non-patent | – | Applicant |
| Intelligent Video Analytics Solutions—BriefCam, https://www.briefcam.com/technology/how-it-works/, 8 pages. | Non-patent | – | Applicant |
| Wisenet Wave—VMS Made Simple, https://wavevms.com/, 8 pages. | Non-patent | – | Applicant |
| Fast Finder—Apps on Google Play, 2019, 4 pages. | Non-patent | – | Applicant |
| International Search Report and Written Opinion dated Feb. 5, 2020, in connection with PCT CA2019/051588, 11 pages. | Non-patent | – | Applicant |
| Fathy, M. and Siyal, Y. A window based image processing technique for quantitative and qualitative analysis of road traffic parameters, IEEE Transactions on Vehicular Technology, 1998, pp. 1342-1349. | Non-patent | – | Applicant |
| Jing G, Rajan D and Chng Eng Siong, Motion detection with adaptive background and dynamic thresholds, Information, Communications and Signal Processing, 2005,Fifth International Conference on. IEEE. | Non-patent | – | Applicant |
| Quen-Zong Wu, Hsu Yung Cheng and Bor-Shenn Jeng, Motion detection via change-point detection for cumulative histograms of ratio images, Pattern Recognition Letters 26.5 ,2005, pp. 555-563. | Non-patent | – | Applicant |
| Elhabian, El Sayed and Ahmed, Moving object detection in spatial domain using background removal techniques—state-of-art, Recent patents on computer science 1.1, 2008, pp. 32-54. | Non-patent | – | Applicant |
| Dimou, Nemethova and Rupp, Scene change detection for H. 264 using dynamic threshold techniques, 2005. | Non-patent | – | Applicant |
| Zhang and Liang, Motion human detection based on background subtraction, Education Technology and Computer Science (ETCS), 2010 Second International Workshop on. vol. 1. IEEE, 2010. | Non-patent | – | Applicant |
| Browne et al, Evaluating and combining digital video shot boundary detection algorithms, 2000. | Non-patent | – | Applicant |
| Intelligent Video Analytics Solutions—BriefCam, https://www.briefcam.com/technology/how-it-works/, 8 pages. | Non-patent | – | Applicant |
| Wisenet Wave—VMS Made Simple, https://wavevms.com/, 8 pages. | Non-patent | – | Applicant |
| Fast Finder—Apps on Google Play, 2019, 4 pages. | Non-patent | – | Applicant |
| International Search Report and Written Opinion dated Feb. 5, 2020, in connection with PCT CA2019/051588, 11 pages. | Non-patent | – | Applicant |
10 members in 3 offices
Priority claims14
| Document | Office | Kind | Date |
|---|---|---|---|
| 201862756645 | United States of America | P | |
| 201862756645 | United States of America | P | |
| 201962796734 | United States of America | P | |
| 201962796734 | United States of America | P | |
| 201962928531 | United States of America | P | |
| 201962928531 | United States of America | P | |
| 201916677330 | United States of America | A | |
| 62756645 | – | – | – |
| 62796734 | – | – | – |
| 62928531 | – | – | – |
| US201862756645P | – | – | – |
| US201916677330 | – | – | – |
| US201962796734P | – | – | – |
| US201962928531P | – | – | – |
Members10
| Document | Office | Kind | |
|---|---|---|---|
| US2020143844A1 | United States of America | A1 | |
| WO2020093164A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US10971192B2This record | United States of America | B2 | |
| US2021225409A1 | United States of America | A1 | |
| EP3878187A1 | European Patent Office (EPO) | A1 | |
| EP3878187A4 | European Patent Office (EPO) | A4 | |
| US11893796B2 | United States of America | B2 | |
| US2024112470A1 | United States of America | A1 | |
| US12387497B2 | United States of America | B2 | |
| US2025336210A1 | United States of America | A1 |
51 transactions on the USPTO file
Allowed after 1 non-final rejection.
- Non-final rejections
- 1
- Final rejections
- 0
- RCEs
- 0
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Payment of Maintenance Fee, 4th Year, Large EntityM1551 | M1551 | |
| Recordation of Patent Grant MailedPGM/ | PGM/ | |
| Patent Issue Date Used in PTA CalculationAllowedPTAC | PTAC | |
| Email NotificationEML_NTR | EML_NTR | |
| Issue Notification MailedAllowedWPIR | WPIR | |
| Dispatch to FDCD1935 | D1935 | |
| Application Is Considered Ready for IssuePILS | PILS | |
| Issue Fee Payment VerifiedN084 | N084 | |
| Issue Fee Payment ReceivedIFEE | IFEE | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Notice of AllowanceAllowedMN/=. | MN/=. | |
| Notice of Allowance Data Verification CompletedAllowedN/=. | N/=. | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Non-Final ActionA... | A... | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Email NotificationEML_NTR | EML_NTR | |
| PG-Pub Issue NotificationPG-ISSUE | PG-ISSUE | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Email NotificationEML_NTR | EML_NTR | |
| Application Is Now CompleteCOMP | COMP | |
| Filing Receipt - UpdatedFLRCPT.U | FLRCPT.U | |
| Sent to Classification ContractorPGPC | PGPC | |
| FITF set to YES - revise initial settingFTFS | FTFS | |
| Patent Term Adjustment - Ready for ExaminationPTA.RFE | PTA.RFE | |
| Additional Application Filing FeesADDFLFEE | ADDFLFEE | |
| Applicant has submitted new drawings to correct Corrected Papers problemsCORRDRW | CORRDRW | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTR | EML_NTR | |
| Email NotificationEML_NTF | EML_NTF | |
| Application ready for PDX access by participating foreign officesCCRDY | CCRDY | |
| Filing ReceiptFLRCPT.O | FLRCPT.O | |
| Corrected PaperCPAP | CPAP | |
| Cleared by OIPE CSRL194 | L194 | |
| Preliminary AmendmentA.PE | A.PE | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| PTO/SB/69-Authorize EPO Access to Search ResultsSREXR141 | SREXR141 | |
| Applicants have given acceptable permission for participating foreignAPPERMS | APPERMS | |
| IFW Scan & PACR Auto Security ReviewSCAN | SCAN | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Entity Status Set To Undiscounted (Initial Default Setting or Status Change)BIG. | BIG. | |
| Initial Exam Team nnIEXX | IEXX |
6 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Maintenance fee paymentMAFP | MAFP | |
| Information on status: patent grantGrantedPATENTED CASESTCF | STCF | |
| Information on status: patent application and granting procedure in generalPUBLICATIONS -- ISSUE FEE PAYMENT VERIFIEDSTPP | STPP | |
| Information on status: patent application and granting procedure in generalNON FINAL ACTION MAILEDSTPP | STPP | |
| AssignmentAS | AS | |
| Fee payment procedureENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITYFEPP | FEPP |
Numbers
- Publication
- 10971192
- Publication, DOCDB
- 10971192
- Publication, EPODOC
- US10971192
- Application
- 16677330
- Application, DOCDB
- 201916677330
- Application, EPODOC
- US201916677330
Titles
- English
- Methods and systems for detection of anomalous motion in a video stream and for creating a video summary
Patent term adjustment
- Net adjustment
- 0 days
Classification
- CPC, 11
- G11B27/34
- G06V20/52
- G08B17/125
- G06K9/00711
- G06K9/6298
- G06V20/47
- G11B27/3081
- G06V10/764
- G06K2009/00738
- G06V20/40
- G06V20/44
- IPC, 6
- G11B27 34
- G06K9 00
- G11B27 30
- G06K9 62
- G08B17 12
- G06V10 764
- USPC, 1
- 348043000