System for annotating media content for automatic content understanding.
Abstract
Un sistema para anotar tramas en un flujo de medios 114 incluye un sistema de reconocimiento de patrones (PRS) 108 para generar metadatos de salida del PRS para una trama; un archivo 106 para almacenar metadatos de alta precisión (GTM); un dispositivo para unir los GTM y los metadatos de salida del PRS y de esta manera generar datos de anotación propuestos (PAD) 110; y una interfaz de usuario 109 para utilizarse por el anotador humano (HA) 118. La interfaz de usuario 104 incluye un editor 111 y un dispositivo de entrada 107 utilizado por el HA 118 para aprobar los GTM para la trama. Un sistema de optimización 105 recibe los GTM aprobados y los metadatos proporcionados por el PRS 108, y ajusta los parámetros de entrada para el PRS para minimizar una métrica de distancia correspondiente a una diferencia entre los GTM y los metadatos proporcionados por el PRS.

Term
6.6 yearsleft in the term
Expires 22 April 2033.
- Priority
- Filed
- Granted
- Today
- Expires
16 claims: 5 independent, 11 dependent
- 1NOVEDAD DE LA INVENCIÓN Habiendo descrito la presente invención como antecede, se considera como una novedad y, por lo tanto, se reclama como propiedad lo contenido en las siguientes:REIVINDICACIONES 1. Un sistema para anotar contenidos de medios, caracterizado porque comprende: un sistema de reconocimiento de patrones (PRS) (108) que tiene un conjunto inicial de parámetros de entrada que genera metadatos de salida del PRS asociados con una trama de un flujo de medios (114) ;un archivo de dispositivo de almacenamiento electrónico (106) para almacenar metadatos de alta precisión (GTM) asociados con la misma trama del flujo de medios (114);un dispositivo para fusionar los GTM y los metadatos de salida del PRS y de esta manera generar datos propuestos de anotación (PAD) (110);y una interfaz de usuario (104) para utilizarse por un anotador humano (HA) (118) que incluye: un editor (111) y un dispositivo de entrada (104) para aprobar editar los PAD (110) para la trama;y ΙΜΡΪ INSTITUTO MEXICANO 2 5 DE LA PROPIEDAD INDUSTRIAL un sistema de optimización (105) para ajustar parámetros de entrada, de acuerdo a la entrada del HA, para el PRS (108) para minimizar una métrica de distancia que corresponde a una diferencia entre los GTM y los metadatos de salida del PRS. 10 caracterizado porque un retraso de tiempo entre los metadatos de terceros (101) y el flujo de medios (114) se corrige con una alineación.
- 24. El sistema de acuerdo con la reivindicación 2, caracterizado porque una red de comunicaciones (103) habilita 15 una pluralidad de HAs (118) para interactuar con el mismo flujo de medios (114).
- 35. El sistema de acuerdo con la reivindicación 2, caracterizado porque cuando se aprueban los PAD (110) se convierten en GTM. 20
- 46. El sistema de acuerdo con la reivindicación 5, caracterizado porque cuando se aprueban los PAD (110) se sobreponen gráficamente en el flujo de medios (114).
- 57. El sistema de acuerdo con la reivindicación 1, caracterizado porque el sistema de optimización (105) ajusta IMPI el conjunto inicial de parámetros de pnt-rarla HpI prs (108) para minimizar la diferencia entre los GTM y los metadatos de salida del PRS aumentando de esta manera la precisión.
- 68. El sistema de acuerdo con la reivindicación 1 5 caracterizado porque el PRS (108) incluye un conjunto de variables de estado almacenado como un grupo temporal (112) ajustable como un grupo en respuesta a los GTM.
- 79. Un método implementado por computadora para anotar contenidos de medios, caracterizado porque el método
- 810 comprende los pasos de:recibir, mediante un sistema de computadora, datos desde un flujo de medios, donde los datos se organizan en tramas (202);procesar los datos utilizando un sistema de 15 reconocimiento de patrones (PRS) (108) ;almacenar, mediante un sistema de computadora, un estado del PRS (108) ;generar metadatos asociados con la trama con el uso del PRS (108) ;20 recibir la entrada que está caracterizada como metadatos de alta precisión (GTM), en un sistema de optimización;ajustar, mediante un sistema de computadora, parámetros de entrada para el PRS (108) para minimizar una métrica de IMPI distancia que corresponde a una diferencia entre los GTM y los metadatos de salida del PRS. 10. El método implementado por computadora de acuerdo con la reivindicación 9 caracterizado porque dicha entrada se obtiene de uno o más de los flujos de medios archivados (115), metadatos de terceros (101) y uno o más anotadores humanos (118).
- 911. El método implementado por computadora de acuerdo con la reivindicación 10, caracterizado porque después de recibir dicha entrada, dichos GTM y dichos metadatos asociados con dicho PRS se alinean temporalmente.
- 1012. El método implementado por computadora de acuerdo con la reivindicación 10, caracterizado porque dichos GTM y dichos metadatos asociados con dicho PRS (108) se almacenan continuamente en la memoria y se almacenan periódicamente en un disco, de esta manera se habilita una recuperación rápida ante una falla del sistema.
- 1113. Un método implementado por computadora para anotar contenidos de medios, caracterizado porque el método comprende los pasos de:recibir de un anotador humano (HA), a través de una interfaz de usuario de un anotador humano (HAUI), información acerca de un punto en el tiempo seleccionado por el HA en una línea de tiempo del flujo de medios;IΜ ΡI 9 β INSTITUTO MEXICANO ν DE LA PROPIEDAI' INDUSTRIAL fusionar, mediante el sistema de computadora, metadatos de alta precisión (GTM) existentes que hacen referencia a una trama de medios que corresponde al punto en el tiempo seleccionado con metadatos de salida del sistema de 5 reconocimiento de patrones (PRS) que hace referencia a dicha trama de medios, de esta manera se generan datos de anotación propuestos (PAD) (110) para la trama de medios;mostrar, por medio de una interfaz de usuario (104), la trama de medios y los PAD (110) al HA (118) ;10 recibir, mediante el sistema de computadora, la entrada desde el HA (118) que incluye la corrección y/o aprobación a los PAD (110) , en donde los PAD (110) aprobados se caracterizan como nuevos GTM relacionados con el punto en el tiempo seleccionado;15 almacenar, mediante el sistema de computadora, los nuevos GTM;comparar, mediante el sistema de computadora, los metadatos de salida del PRS y los nuevos GTM relacionados con el punto en el tiempo seleccionado;y 20 ajustar, mediante el sistema de computadora, los parámetros de entrada del PRS de modo que se minimiza una métrica de distancia que corresponde a una diferencia entre los nuevos GTM y los metadatos de salida del PRS relacionados con el punto en el tiempo seleccionado. IMPI INSTITUTO MEXICANO 9 Q ÜE LA PROPIEDAD INDUSTRIAL
- 1214. El método implementado por nnmputadnra dp amprrln con la reivindicación 13, caracterizado porque dichos GTM se obtienen de uno o más de los flujos de medios (115) , los metadatos de terceros (101), dichos anotadores humanos (118) y otros anotadores humanos.
- 1315. El método implementado por computadora de acuerdo con la reivindicación 14, caracterizado porque cuando dicho anotador humano (118) aprueba dichos PAD (110), dichos PAD (110) se sobreponen gráficamente en dicho flujo de medios.
- 1416. Un método implementado por computadora para anotar contenidos de medios, caracterizado porque el método comprende los pasos de:generar metadatos de salida asociados con una trama de un flujo de medios, la salida por medio de un sistema de reconocimiento de patrones (PRS) (108) ;almacenar, mediante el sistema de computadora, en un archivo (106) entradas de un anotador humano (HA) (118) relacionadas con la trama, caracterizadas como metadatos de alta precisión (GTM);fusionar, mediante el sistema de computadora, los GTM y los metadatos de salida del PRS (108) para generar de esta manera datos propuestos de anotación (PAD) (110);y mostrar los PAD (110) al HA (118) por medio de una interfaz de usuario 104;IMPI recibir a través de la interfaz de usuario una entrada del HA (118) que indica aprobación a los GTM de la trama;y ajustar parámetros de entrada para el PRS (108) utilizando un sistema de optimización (105), para minimizar una métrica de distancia que corresponde a una diferencia entre los GTM y los metadatos de salida del PRS.
- 1517. El método implementado por computadora de acuerdo con la reivindicación 16, caracterizado porque dichos GTM se obtienen de uno o más de los flujos de medios archivados (115), de metadatos de terceros (101), de dichos anotadores humanos (118) y de otros anotadores humanos.
- 1618. El método implementado por computadora de acuerdo con la reivindicación 17, caracterizado porque cuando dicho anotador humano (118) aprueba dichos PAD, dichos PAD se sobreponen gráficamente en dicho flujo de medios. IMPI INSTITUTO MEXICANO o - DE LA PROPIEDAD J X INDUSTRIAL
Independent claims16
204 paragraphs in 31 sections, as filed
(54) Title: ANNOTATION OF MEDIA CONTENTS FOR AUTOMATIC UNDERSTANDING OF CONTENTS. (54) Title: SYSTEM FOR ANNOTATING MEDIA CONTENT FOR AUTOMATIC CONTENT UNDERSTANDING.
(57) Summary
A system for annotating frames in a media stream 114 includes a pattern recognition system (PRS) 108 for generating output metadata from the PRS for a frame; a file 106 for storing high precision metadata (GTM); a device for merging the GTMs and the PRS output metadata and thereby generating proposed annotation data (PAD) 110; and a user interface 109 for use by the human annotator (HA) 118. User interface 104 includes an editor 111 and an input device 107 used by HA 118 to approve the GTMs for the frame. An optimization system 105 receives the approved GTMs and the metadata provided by the PRS 108, and adjusts the input parameters for the PRS to minimize a distance metric corresponding to a difference between the GTMs and the metadata provided by the PRS.
(57) Abstract
A system for annotating frames in a media stream 114 ineludes a pattern recognition system (PRS) 108 to generate PRS output metadata for a frame; an archive 106 for storing ground truth metadata (GTM); a device to merge the GTM and PRS output metadata and thereby generate proposed annotation data (PAD) 110; and a user interface 109 for use by the human annotator HA 118. The user interface 104 ineludes an editor 111 and an input device 107 used by the HA 118 to approve GTM for the frame. An optimization system 105 receives the approved GTM and metadata output by the PRS 108, and adjusts input parameters for the PRS to minimize a distance metric corresponding to a difference between the GTM and PRS output metadata.
'V
<img file="MX339009B_D0001.tif" />
_SE_ μ i tu uoww ·.
of the
Institute
Mexican
Property
Industrial
PATENT TITLE NO. 339009
Owner (s): LIVECLIPS LLC
Address: 2260 E. Imperial Highway, El Segundo, California, 90245, USA
Name: ANNOTATION OF MEDIA CONTENTS FOR AUTOMATIC UNDERSTANDING OF CONTENTS.
Classification: lnt.CI.8: G06F17 / 24; G06K9 / 00; H04N5 / 44
Inventor (s): ERIC DAVID PETAJAN; DAVID EUGENE WEITE; DOUGLAS W. VUNIC
<td></td><td>REQUEST</td><td></td>
<td>Number: MX / a / 2014/012970</td><td colspan="2"> . . International filing date: Abhi 22, 2013 PRIORITY</td>
<td>Country:</td><td>Date:</td><td>Number:</td>
<td>US</td><td>April 24, 2012</td><td> 61/637,344</td>
<td>US</td><td>March 15, 2013</td><td> 13/836,605</td>
Validity: Twenty years
Expiration Date: April 22, 2033
The reference patent is granted based on articles 1, 2nd section V, 6th section III, and 59 of the Industrial Property Law.
In accordance with article 23 of the Industrial Property Law, this patent has a non-extendable term of twenty years, counted from the filing date of the international application and will be subject to the payment of the fee to keep the rights in force. .
Whoever subscribes to this title does so based on the provisions of articles 6 sections III and 7 bis 2 of the Industrial Opportunity Law (Official Gazette of the Federation (DOF) 06/27/1991, amended 08/02/1994, 10/25/1996, 12/26/1997, 05/17/1999, 26 / eH / 2004, 06/16/2005, 01/25/2005, 09/05/2009, 06/06 01/2010, 18/00/2010, 06/28/2010, 01/27/2012 and 04/09/2012); Articles 1, 3, fraction V, subsection a), sub subsection iii), 4th and 12th fractions I and III of the Regulations of the Mexican Institute of Industrial Property (DOF 12/14/1999, rewritten on 07/07 / 2002,15 / t | / 2004, 07/28/2004 and 09/07/2007); articles 1 ·, 3 ·, 4 ·, 5 · fraction V subsection a), sub subsection II), 16 fractions I and III and 30 of the Organic Statute d * Mexican Institute of Industrial Property (DOF 12/27/1999, amended on 10/10/2002, 07/29/2004, 08/04/2004 and 08/13/2007); 1 «3rd and 5th clause a) and. AnteMBultimate paragraph of the Agreement delegated by facultg« tafe<sub>;</sub>«<sub>;</sub>..to, Deputy Directors of Sectoral Offices, Coordinator, Divisional Directors, Regional Office Titles, Deputy Directors, Departmental Coordinators and other subordinates of the Mexican Institute of Industrial Property. (DOF 12/15/1999, amended on 02/04/2000, 07/29/2004, 08/04/2004 and 09/13/2007). ; ':'
MU.
<img file="MX339009B_D0002.tif" />
Issue Date: May 5, 2016
DIVISIONAL TOR OF EXAMINATION OF THE FUND FOR PATENTS, AREAS, ELECTRICAL AND INDUSTRIAL DESIGN REGISTRIES AND
<img file="MX339009B_D0003.tif" />
PEDRMmffmmeoe ·
Arenal No. 550, Floor 1,
Coi. Sania María Tepepan village. Xochímilco, C P. 16020,
Mexico City
Tei. (55) 53 34 07 00 wwvv íi'npi.qob.mx
<img file="MX339009B_D0004.tif" />
MX / 2016/36546
<img file="MX339009B_D0005.tif" />
ANNOUNCEMENT OF MEDIA CONTENTS FOR THE
<img file="MX339009B_D0006.tif" />
AUTOMATIC CONTENT
FIELD OF THE INVENTION
The present invention relates to media presentations (eg, live sporting events), more particularly to a system for improving performance by generating annotations for the media stream.
BACKGROUND OF THE INVENTION
A media presentation, such as a broadcast of an event, can be understood as an audio / video frame stream (live media stream). It is desirable to add information to the media stream to enhance the viewer experience; this is generally called annotation in the media stream. Annotating in a media stream is a tedious and time consuming task for a human. Visual inspection of text, players, balls, and the position of the field / court is mentally draining and error prone.
Keyboard and mouse input are required to enter annotation data but they are also error prone and mentally draining. Accordingly, systems have been developed to automate at least partially the annotation process.
<img file="MX339009B_D0007.tif" />
IMPI _ INSTITUTO MEXICANO ¿DE LA FUIPILUAÍ
INDUSTRIAL
The Pattern Recognition Systems (PRS,
Pattern Recognition System), e.g. machine vision or
Automatic Speech Recognition (ASR)
Recognition) process media streams in order to generate meaningful metadata. Recognition systems operating in natural media streams always perform with less than absolute precision due to the presence of noise. Artificial Vision (CV,
Computer Vision) is notoriously error prone and ARS is only usable under restricted conditions. Measuring the accuracy of the system requires knowledge of the correct result provided by the PRS, which is referred to in this document as High Precision Metadata (GTM, Ground Truth Metadata). The development of a PRS requires the generation of GTM that must be validated by
Human Annotators (HA). GTMs can consist of positions in space or time, tagged elements, events, text, region boundaries, or any data with a single tag that allows for reference and comparison.
A compilation of the acronyms used in this document is attached to this Specification.
There is still a need for a system that can reduce the human time and effort required to create the GTMs.
IΛίΡΡ
MEXICAN INSTITUTE Μ
PROPERTY X
INDUSTRIAL
BRIEF DESCRIPTION OF THE INVENTION
We call a system for labeling elements in a given video (or audio) frame or events at a given point in time as a Media Stream Recorder (MSA, Media
Stream Annotator). If accurate enough, a given PRS automatically generates metadata from the media stream that can be used to reduce the human time and effort required to create the GTMs. According to one aspect of the disclosure, a system, and a process of an MSA, which has a Human-Computer Interface (HCI),
Interface), provides more efficient generation of GTM and PRS input parameter adjustment.
The GTMs are used to verify the accuracy of the PRS and to adjust the input parameters of the PRS or to guide the development of the algorithm for optimal recognition precision. GTMs can be generated at low levels of detail in space and time, or at high levels such as events and states with start times and durations that can be imprecise compared to the time of a low-level video frame.
Adjustments to the PRS input parameters that are designed to be static during a program should be applied to all sections of a program with associated GTM in order to maximize the accuracy of
IMPI ^
MEXICAN INSTITUTE OF THE TKUPIEDAD
INDUSTRIAL average recognition and not just the precision of the given section or video frame. If the MSA processes live media, the effect of any automated input parameter adjustment on the PRS must be measured in all sections t
with GTM (past and present) before making changes to generate final production output.
A system that represents disclosure can be applied to live and archived media programs and has the following elements:
- Random access to a given frame or section of the archived media stream and associated metadata
- Visualization or graphic superimposition in real time of metadata generated by the PRS in or near the presentation of the plot of the video.
- One-click approval of conversion
Proposed Annotation Data (PAD) to GTM
- PRS recalculates all metadata when GTMs change - Merge third party metadata with human annotations
- Graphic overlay of compressed and decoded metadata on or near low bit rate decoded videos to enable real time operation on
IMPI
<img file="MX339009B_D0008.tif" />
mobile devices and consumer grade internet connections
The foregoing broadly highlights the preferred elements of the present disclosure so that those skilled in the art can better understand the detailed description of the following disclosure. Additional elements which form the subject of the disclosure claims will be described hereinafter.
Those skilled in the art should appreciate that they can easily use the disclosed conception and modality as a basis for designing or modifying other structures to carry out the same purposes of the present disclosure and that such other structures do not depart from the spirit and scope of disclosure in its broadest form.
BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 is a schematic illustration of the Media Stream Recorder (MSA), according to one embodiment of the disclosure.
Figure 2 is a schematic illustration of the Media Annotator flowchart during ingestion of
Third Party Metadata (TPM), in accordance with a disclosure modality.
<img file="MX339009B_D0009.tif" />
IMPI
Figure 3 is a schematic illustration of the Media Annotator flowchart during Human Annotation, according to one embodiment of the disclosure.
Figure 4 is a schematic illustration of a mini American football panel, according to one embodiment of the disclosure.
DETAILED DESCRIPTION OF THE INVENTION
The precision of any PRS depends on the application of restrictions that reduce the number or range of possible results. These constraints can take the form of a priori information, physical and logical constraints, or highly reliable partial recognition results.
A priori information for sports includes the type of sport, the architecture and location of the stadium, the date and time, the teams, the players, the station, the language, and the media intake process (eg, the A / V (Audio / Video) resolution and original transcoding). Physical restrictions include camera inertia, camera mount type, lighting, and the physicality of players, balls, equipment, courts, fields, and limits. Logical restrictions include the rules of the game, sports production methods, uniform colors and patterns, and the operation of the
IMPI
<img file="MX339009B_D0010.tif" />
scoreboard board. Some of the information can be reliably extracted from the media stream with minimal a priori information and can be used to initiate the boot strap instruction for subsequent recognition processes. For example, the presence of the mini graphic panel superimposed on the game video (shown in the
Figure 4) can be detected with only knowledge of the sport and the station (eg ESPN, FOX Sports, etc.).
If a live media sporting event is processed in real time, only current and past media streams are available for pattern recognition and metadata generation. A recorded sports event can be processed with access to any frame throughout the program.
The PRS that processes a live event may become more accurate as time progresses as more information becomes available over time, while any frame of a recorded event can be repeatedly analyzed from the past or future until the maximum precision.
Annotating in a media stream is a tedious and time consuming task for a human. Visual inspection of text, players, balls, and the position of the field / court is mentally exhausting and error prone. Keyboard and mouse input are required to
<img file="MX339009B_D0011.tif" />
IMPI enter annotation data but they are also error prone and mentally draining. The productivity of human annotation (speed and accuracy) is greatly enhanced by properly displaying automatically generated available Proposed Annotation Data (PADs) and thereby minimizing the mouse and keyboard inputs required to edit and approve PADs. If the PADs are correct, the
Human Annotator (HA) can simultaneously approve the current frame and select the next frame for annotation by simply pressing a key or mouse button once. PADs are the best automatically generated current metadata that can be brought to the user without significant delay. Expecting the system to maximize the accuracy of PADs can decrease editing by using the
HA but will also delay the approval of the given plot.
Figure 1 shows a Flow Annotator system for
Media (MSA, Media Stream Annotator) in accordance with a disclosure modality. The MSA ingests the 114 and archived Live Media Streams (LMS)
AMS (Archived Media Stream) 115), and Third Party Metadata (TPM) 101 and optional HA 118 entries. PADs are derived from a combination of the metadata resulting from PRS 108 and TPM 101. Metadata output via PRS 108 is archived in a File
IMPI
Q MEXICAN INSTITUTE
FROM INDUSTRIAL PROHEDAL Metadata 109. If TPM 101 is available during live events the system can convert TPM 101 to
GTM through Metadata Mapper 102 and then use the Performance Optimization System (POS,
Performance Optimization System) 105 to adjust the
PRS Input Parameters to improve metadata precision for media ingested in the past (AMS 115) and in the present (LMS 114). The PAD Encoder 110 merges the GTMs with metadata for each media frame and encodes the PADs in a compressed form suitable for transmission on the Human Annotator User Interface (HAUI,
Human Annotator User Interface) 104 via a suitable network, eg the Internet 103. This information is subsequently decoded and displayed to the HA, in a form that the HA can edit, by a Decoder, Monitor and Flow Editor Media and PAD (MSPDE, Media Stream and PAD Decoder,
Display and Editor) 111. The HAUI also includes a Media Stream Navigator (MSN) 117 which the HA uses to select points in time in the media stream whose corresponding frames should be annotated.
A low bit rate version of the media stream is transcoded from the AMS by a Media Transcoder
116 and then it is transmitted to HAUI.
<img file="MX339009B_D0012.tif" />
IMPI
<img file="MX339009B_D0013.tif" />
As GTMs are generated by HA Π8 and rp stored in GTM File 106, POS 105 compares the output metadata of PRS 108 with the GTMs and detects significant differences between them. During the design and development of PRS 108, input parameters are established with initial estimated values that produce accurate results on a set of sample media streams and associated GTMs.
These parameter values are adjusted with POS 105 until the difference between all the GTMs and the metadata generated by the PRS is minimized.
During development (unlike live production) the POS 105 does not need to operate in real time and extensive optimization algorithms can be used. During a live program the POS 105 should operate as fast as possible to improve the performance of the PRS 108 every time the HA 118 generates new GTMs; therefore faster optimization algorithms are used during a live program. POS 105 is also invoked when new TPM 101 is converted to GTM.
The choice of metric distance between the PRS and GTM output metadata depends on the type of data and the variation that can be allowed. For example, in a presentation of a football game the scoreboard information extracted from the mini panel should be
<img file="MX339009B_D0014.tif" />
<img file="MX339009B_D0015.tif" />
absolutely accurate while the spatial position of a player on the field can vary. If a PRS input parameter affects multiple types of results, then the distance values for each type can be weighed into a linear combination of distances to calculate a single distance for a given frame or time segment of the game.
A variety of TPM 101 (eg from stats.com) is available after a delay period of live action that can be used as a GTM either during development or after the delay period during a live game. . Since the TPMs are delayed by an unspecified period of time, they must be time aligned with the program. The alignment can be done either manually, or the GTMs can be aligned with TPM 101, and / or the resulting metadata from PRS 108 can be aligned using rough match techniques.
The PRS 108 maintains a set of state variables that changes over time as environment models, players, overlay graphics, cameras, and weather update. The advent of the TPM 101 and, in turn, the GTMs may drive changes for current and past state variables. If the history of the state variables is not persistently stored, the
IMPI
<img file="MX339009B_D0016.tif" />
POS 105 would have to start the media flow from scratch in order to use PRS 108 to regenerate metadata using new PRS Input Parameters
108. The number of PRS 108 state variables can be large, and is compressed using State Code 112 in one or more Group of States (GOS) sequences such that a temporary section of PRS States is encode and decode as a group for greater compression efficiency and recovery speed. The GOS is stored in a GOS File 113. The number of media frames in a GOS can be as low as one.
If the resulting metadata from PRS 108 is persistently stored, the HA can navigate to a past point in time and immediately retrieve the associated metadata or GTMs through PAD Encoder 110, which formats and compresses the PADs to bring them to HA 118 by the network.
Figure 2 shows a flow diagram of the operation of the MSA, according to an embodiment of the disclosure in which the live media stream (LMS, Live Media) is ingested.
Stream) and TPM. All LMS are archived in AMS (step 201). At system startup, the initial or default GOS values are entered into the PRS which then initiates LMS processing in real time (step 202). If he
PRS does not have enough resources to process each frame of
IMPI
<img file="MX339009B_D0017.tif" />
LMS, the PRS will skip frames to minimize latency between a given LMS frame and its associated resulting Metadata (step 203). Periodically, the internal variable values of the PRS status are encoded in the GOS and
5. file (step 4204). Finally, the PRS generates metadata which is archived (step 205); the process returns to step 201 and the next or most recent media frame is ingested.
Processing cycle 201-205 can iterate indefinitely.
When TPMs arrive over the Internet, they merge with any of the GTMs that exist for that media frame through the Metadata Mapper (step 206). The
POS then notifies itself of new GTMs and generates new sets of PRS Input Parameters, while comparing all the resulting Metadata with any of the corresponding GTMs for each set until an optimal set of PRS Input Parameters is found that minimizes the Global distance between all the GTMs and the corresponding Metadata (step 207).
Figure 3 shows a flow chart for the operation of the ΜΞΑ while HA approves the new GTMs. This process operates in parallel with the process shown in the flowchart in Figure 2. The HA must first select a point on the timeline of the flow of
<img file="MX339009B_D0018.tif" />
IMPI means to make an annotation (step 301). The HA can find a point in time by dragging a graphic cursor on a media player while viewing a low bitrate version of the transcoded media stream from the AMS (step 302). The Metadata and any GTMs associated with the selected point in time are retrieved from their respective files 109, 106 and encoded in the PADs (step 303); they are transmitted with the Media Stream to HAUI over the Internet (step 304); and are presented to the HA via the HAUI after decoding the PADs and the Low Bit Rate Media Stream (step 305). HAUI displays PADs at or near the displayed Media Stream (step 306).
The HA compares the PADs to the Media Frame and clicks an Approval 107 button or corrects the PADs using an editor and approves the PADs (step 307). After approving the
PADs, HAUI transmits the corrected and / or approved PADs as new GTMs to be stored in the GTM Archive (step 308).
The POS is then notified about the new GTMs and generates new sets of PRS Input Parameters, while comparing all the resulting Metadata with any of the
Corresponding GTMs for each set (step 309) until an optimal set of PRS Input Parameters is found that minimizes the overall distance between all GTMs and the
Corresponding metadata (step 310).
<img file="MX339009B_D0019.tif" />
IMPI
If the MSA is operated only on the AMS (and not on the LMS), the
POS can perform more extensive and time consuming algorithms to minimize the distance between the GTMs and the
Metadata; the consequence of having incomplete or less accurate metadata is an increase in the editing time for the
HE HAS. If the MSA is operated on an LMS during a live production, the POS is restricted to not update the PRS Input Parameters for live productions until the accuracy of the Metadata is maximized.
The HA does not require any special skills other than basic knowledge of the content of the media stream (eg sport event rules) and an installation with a basic computer interface. The performance of
PRS relies on collecting large amounts of GTM to ensure that optimization by the POS will result in optimal PRS performance on new media streams.
Therefore, it is usually convenient to employ multiple HAs for a given media stream. The HAs reserve is increased if the HAUI client can communicate with the remaining system over the consumer grade internet or through mobile internet connections which have limited capacity. The primary consumer of internet capacity is the media stream that is brought to HAUI for decoding and display. Fortunately, the bit rate
<img file="MX339009B_D0020.tif" />
IMPI of the media stream can be greatly decreased to allow transportation through the consumer or mobile internet connections by transcoding the video to a lower resolution and quality. Most of the bitrate required for high-quality compression of sporting events is applied to complex regions in the video, such as views that contain large spectators at the event; however, the HA does not need high-quality video from viewers for annotations. Instead, HA needs minimal visual quality for the mini panel, player identification, ball tracking, and markings on the field which is easily accomplished with a minimum of compressed bit rate.
PADs are also transmitted to HAUI, but this information is easily compressed as text, graphic coordinates, geometric objects, color properties, or animation data. All PADs can be compressed losslessly using statistical compression techniques (eg zip), but animation data can be highly compressed using lossy animation stream codes such as those found in the standard tools Synthetic and Natural Hybrid Coding (SNHC) of MPEG-4 (eg,
Face and Body Animation and 3D Mesh Coding).
i7 IMPI ^ h
MEXICAN INSTITUTE OF THE PPOHEUAU
INDUSTRIAL --The sample of the PADs transmitted and decoded to the HA is arranged for clearer display and compression between the video and the PADs. For example, as shown in
Figure 4, the content of the PADs in the mini panel should be displayed below the video frame in its own box
402 and vertically aligned with the mini panel in the video
401. PAD content that refers to natural (non-graphic) objects in the video must be graphically overlaid in the video.
Editing the PADs done by the HA can be done either in the mini-panel text window directly for the data in the mini-panel or by dragging spatially-positioned data directly onto the video in the correct position (eg, the lines of the field or the identities of the players). The combined use of low bit rate, video with adequate quality and compressed text, graphics and animation data which are composed in the video results in a
HAUI that can be used with low bit rate internet connections.
Referring again to Figure 1, Metadata File 109 and GTM File 106 are ideally designed and developed to provide fast in-memory access to metadata while writing file contents to disk as often as necessary to
IMPI
P MEXICAN INSTITUTE
FROM INDUSTRIAL PROPERTY allow rapid recovery after a system failure (power outage, etc.). In addition to the inherent speed of memory access (vs. disk access), metadata files should ideally be outlined to provide fast search and data derivation operations. Quick search is needed to find the corresponding entries in the GTM 106 vs Metadata 109 files, and to support asynchronous writes to the GTM 106 File from the Mapping of
Metadata 102. Preferred layouts of data structures in files that support quick search include the use of linked lists and hash tables. Linked lists enable insert edit operations without the need to move data blocks to accommodate new data. Hash tables provide quick searches for sparse dataset addresses.
Intake of TPM 101 requires that the timestamps of the TPMs align with the timestamps of the GTM 106 and Metadata File 109. The alignment operation may involve multiple passes through all data sets while calculating the Accumulated distance metrics to guide alignment. Ingesting multiple overlapping / redundant TPMs requires a policy to be established to deal with conflicting or
<img file="MX339009B_D0021.tif" />
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX339009B_D0022.tif" />
inconsistent. In case of conflict between TPMs
101, the Metadata Mapper 102 should ideally compare the PRS-generated Metadata 109 with the conflicting TPM 101 in case other preceding knowledge does not resolve the conflict. If the conflict cannot be reliably resolved, then a confidence value should ideally be set for the given metadata which is also stored in the GTM 106. Alternatively, the conflicting data may be omitted from the GTM 106.
GTM 106 and Metadata Files 109 ideally should contain processes to efficiently perform common operations on files. For example, if the metadata time base needs to be adjusted, an internal archiving process could adjust each timestamp throughout the file without impacting other communication channels, or tie up other processing resources.
An example of TPM is the game clock of a live sporting event. TPM game clocks generally consist of an individual message for each tick / second of the clock that contains the clock value. The delay between the clock value live on the sports site and the carried message of the clock value can be seconds or tenths of a second with variation. PRS recognizes live video feed clock and start time
IMPI υ MEXICAN INSTITUTE
INDUSTRIAL PROPERTY of the game is published in advance. The Metadata Mapper
102 You should use all of this information to precisely align the TPM clock tick with the GTM 106 time base and Metadata Files 10 9. At the beginning of the game, there may not be enough data to determine accurately. This alignment is highly accurate, but as time progresses, more metadata accumulates and past alignments can be updated to achieve greater precision.
Another desirable element of the GTM 106 and Metadata 109 files is the ability to repopulate files virtually as an emulation of original intake repetition and TPM processing. This emulation element is useful for tuning and debugging the system.
An exemplary implementation of the system and method discussed in this document is during a football game the start clock (shown as 40/25) counts down from 40 to 25 seconds and stops or disappears when the game starts (if has not counted down to zero). If the recognition of the start clock in the PRS is not precise enough then the detection of the start of the game is less reliable. The figure shows the start clock on the mini panel with a value of
<img file="MX339009B_D0023.tif" />
IMPI
one MEXICAN INSTITUTE * "OF THE PROPERTY
INDUSTRIAL ”08 in block 401 and block 402 shows the results of the mini panel PRS that includes the start clock value of” 08. TPMs provide live data streams that contain the start clock (one value per second) plus the start clock, dialer, etc. TPMs are expected to arrive too late for direct real-time use for game processing but the POS (shown in Figure
1) Compare the TPM start clock values with the game clock output values provided by the
PRS after aligning their respective timestamps. The digit recognizer in the PRS has optional parameters that are set by default at system initialization. When the start clock data enters the POS as GTM the POS adjusts the PRS parameters and stores the digit recognition result for each parameter value. The PRS parameters are adjusted and the associated results are tested until the parameters are optimized.
An optimization technique must test a range of values for a given parameter from a minimum value to a maximum value and store the correction of the result for each parameter value. · The optimal value of the parameter is taken as the equidistant value between the minimum and maximum values that produce a correct result.
<img file="MX339009B_D0024.tif" />
<img file="MX339009B_D0025.tif" />
An alternative to using TPM and GTM, a
Human Annotator (HA) can select a plot of a video and year annotate the start clock value for that frame. This value becomes the GTMs that are entered into the POS and PRS parameter optimization procedures as described above. As multiple digits such as GTM are entered into the POS, optimal PRS parameter values may not be the same across different digits within a frame or between different frames. In this case the POS can either compute the percentage of the optimal individual PRS parameter values or search for common PRS parameter values that produce correct results for each digit in the GTM.
Since the POS operates while the PRS is processing Live Media Streams (LMS), the new optimized PRS parameters can be used immediately if the PRS is running well enough and the PRS's past output does not have to recalculate.
However, if the PRS can search its past status in the
GOS file and re-run the game from past to present faster than real-time (assuming there are enough computational resources to process LMS in real time and AMS faster than real time). When AMS processing reaches the present, the PRS will stop
IMPI
Mexican INSTITUTE
OF THE PROPERTY í
INDUSTRIAL ** · AMS processing and more accurate Metadata output from PRS will be replaced with old Metadata in the
Metadata File.
<td> 5</td><td colspan="3">Collection of Acronyms</td>
<td></td><td>AMS</td><td>Archived Media Stream</td><td></td>
<td></td><td>ASR</td><td>Automatic Voice Recognition</td><td></td>
<td></td><td>CV</td><td>Artificial vision</td><td></td>
<td></td><td>GOS</td><td>Group of States</td><td></td>
<td> 10</td><td>GTM</td><td>High Precision Metadata</td><td></td>
<td></td><td>HE HAS</td><td>Human Scorers</td><td></td>
<td></td><td>HAUI</td><td>Scorer User Interface</td><td>Human</td>
<td></td><td>HCI</td><td colspan="2">Human Computer Interface</td>
<td></td><td>LMS</td><td>Live Media Stream</td><td></td>
<td> 15</td><td>MSA</td><td>Media Flow Recorder</td><td></td>
<td></td><td>MSN</td><td>Media Stream Browser</td><td></td>
<td></td><td>MSPDE</td><td>Media Stream and Decoder ·</td><td>from PAD</td>
<td></td><td>Pad</td><td>Proposed Annotation Data</td><td></td>
<td></td><td>POS</td><td>System for the Optimization of</td><td>performance</td>
<td> 20</td><td>PRS</td><td colspan="2">Pattern Recognition System</td>
<td></td><td>TPM</td><td>Third Party Metadata</td><td></td>
<img file="MX339009B_D0026.tif" />
IMPI
Contents31
30 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30
24 members in 8 offices
Priority claims11
| Document | Office | Kind | Date |
|---|---|---|---|
| 201261637344 | United States of America | P | |
| 61637344 | United States of America | – | |
| 13836605 | United States of America | – | |
| 201313836605 | United States of America | A | |
| 2013037545 | United States of America | W | |
| 13836605 | – | – | – |
| 61637344 | – | – | – |
| US1337545 | – | – | – |
| US201261637344P | – | – | – |
| US201313836605 | – | – | – |
| WO2013US37545 | – | – | – |
Members24
| Document | Office | Kind | |
|---|---|---|---|
| US2013283143A1 | United States of America | A1 | |
| CA2870454A1 | Canada | A1 | |
| WO2013163066A2 | World Intellectual Property Organization (WIPO) | A2 | |
| US2014168517A1 | United States of America | A1 | |
| CO7121323A2 | Colombia | A2 | |
| WO2013163066A3 | World Intellectual Property Organization (WIPO) | A3 | |
| MX2014012970A | Mexico | A | |
| EP2842054A2 | European Patent Office (EPO) | A2 | |
| US2015071618A1 | United States of America | A1 | |
| WO2015126830A1 | World Intellectual Property Organization (WIPO) | A1 | |
| MX339009BThis record | Mexico | B | |
| US9367745B2 | United States of America | B2 | |
| AR099514A1 | Argentina | A1 | |
| EP2842054A4 | European Patent Office (EPO) | A4 | |
| US2016353180A1 | United States of America | A1 | |
| US9659597B2 | United States of America | B2 | |
| BR112014026589A2 | Brazil | A2 | |
| US2017221523A1 | United States of America | A1 | |
| US10056112B2 | United States of America | B2 | |
| US2018336928A1 | United States of America | A1 | |
| US10381045B2 | United States of America | B2 | |
| US2019318765A1 | United States of America | A1 | |
| US10491961B2 | United States of America | B2 | |
| US10553252B2 | United States of America | B2 |
1 legal event, as the office reported them to INPADOC
Events
| Event | Code | |
|---|---|---|
| Grant or registrationFG | FG |
Numbers
- Publication
- 339009
- Publication, DOCDB
- 339009
- Publication, EPODOC
- MX339009
- Application
- 2014012970
- Application, DOCDB
- 2014012970
- Application, EPODOC
- MX20140012970
Titles
- Spanish
- ANOTACION DE CONTENIDOS DE MEDIOS PARA LA COMPRENSION AUTOMATICA DE CONTENIDOS.
Classification
- CPC, 9
- G11B27/036
- G06F16/48
- G11B27/28
- H04N21/23418
- H04N21/84
- H04N21/854
- G06F40/169
- G11B27/19
- H04N21/23424
- IPC, 4
- G06F17 24
- G06F40 00
- G06K9 00
- H04N5 44