Method for search in an audio database
Abstract
A procedure to identify an audio sample, characterized by: for the sample, generate milestone / fingerprint pairs of the sample, where each milestone is presented at a specific temporary location within the audio sample, calculating the location with respect to the content of the audio sample, and where each fingerprint characterizes one or more features of the audio sample at, or near, the specific location; for each or more of the audio files, generate milestone / footprint pairs of the file, where each milestone appears in a specific temporary location within the audio file, calculating the location with respect to the content of the audio file, and in where each fingerprint characterizes one or more features of the audio file at, or near, the specific location; identify essentially linear correspondences between the respective milestone / fingerprint pairs of the sample and the milestone / footprint pairs of previously generated files; and identify a winning file as one that has a significant number of essentially linear correspondences.

Term
Term ended
Projected expiry passed 26 July 2021, 5.2 years ago.
- Priority
- Filed
- Published
- Projected expiry
- Today
28 claims: 2 independent, 26 dependent
- 1ES 2 266 254 T3 REIVINDICACIONES 1. Un procedimiento para identificar una muestra de audio, caracterizado por:para la muestra, generar pares de hito/huella de la muestra, en donde cada hito se presenta en una ubicación temporal específica dentro de la muestra de audio, calculándose la ubicación con respecto al contenido de la muestra de audio, y en donde cada huella caracteriza uno o más rasgos de la muestra de audio en, o cerca de, la ubicación específica;para cada uno o más de los ficheros de audio, generar pares de hito/huella del fichero, en donde cada hito aparece en una ubicación temporal específica dentro del fichero de audio, calculándose la ubicación con respecto al contenido del fichero de audio, y en donde cada huella caracteriza uno o más rasgos del fichero de audio en, o cerca de, la ubicación específica;identificar correspondencias esencialmente lineales entre los respectivos pares de hito/huella de la muestra y los pares de hito/huella de ficheros previamente generados;e identificar un fichero ganador como aquél que tiene un número significativo de correspondencias esencialmente lineales.
- 2Un procedimiento según la reivindicación 1, en el cual cada huella representa un cierto número de rasgos del audio en cada ubicación de hito, o bien desplazados levemente desde dicha ubicación.
- 3Un procedimiento según cualquier reivindicación precedente, en el cual cada huella se calcula por medio de un procedimiento tal que es invariante ante la dilatación temporal de la muestra.
- 4Un procedimiento según cualquier reivindicación precedente, en el cual cada huella se calcula como una entre:una huella de tajada espectral, una huella multitajada, un coeficiente de LPC, un coeficiente cepstral, y un componente de frecuencia de picos de espectrograma.
- 5Un procedimiento según la reivindicación 4, en el cual se calcula una huella de tajada espectral en un conjunto de desplazamientos temporales a partir de un instante de hito temporal.
- 6Un procedimiento según cualquier reivindicación precedente, en el cual la posición de cada hito se identifica utilizando un procedimiento de determinación de hitos que halla ubicaciones distinguibles y reproducibles dentro de la grabación sonora.
- 7Un procedimiento según la reivindicación 6, en el cual el procedimiento de determinación de hitos utiliza una norma Lp espectral para calcular la potencia instantánea en todo instante temporal posible en la grabación, y selecciona los máximos locales como los hitos.
- 8Un procedimiento según la reivindicación 6 o la reivindicación 7, en el cual uno o más hitos son hitos multitajada derivados de componentes espectrales a lo largo de múltiples tajadas temporales, con desplazamientos fijos o variables entre sí.
- 9Un procedimiento según cualquier reivindicación precedente, en el cual los pares de hito/huella del fichero se almacenan en una base de datos, estando indizado cada fichero dentro de la base de datos por las huellas de ese fichero.
- 10Un procedimiento según la reivindicación 9, en el cual los índices se clasifican según las huellas.
- 11Un procedimiento según la reivindicación 10, en el cual se compila una lista del índice maestro que tiene una entrada para cada huella única, y un puntero a una lista de los correspondientes hitos.
- 12Un procedimiento según cualquiera de las reivindicaciones 9 a 11, en el cual cada fichero está identificado por un IDENTIFICADOR_DE_SONIDO, y la base de datos almacena una pluralidad de tripletes de huella, hito, IDENTIFICADOR_DE_SONIDO.
- 13Un procedimiento según cualquier reivindicación precedente, en el cual el fichero con los pares de correspondencias linealmente vinculadas, estadísticamente más significativos, se selecciona como el fichero ganador.
- 14Un procedimiento según cualquier reivindicación precedente, en el cual una correspondencia lineal entre los hitos (hito n , hito* n ) de muestra y de fichero tiene lugar cuando una pareja de hito/huella de la muestra se corresponde con una pareja de hito/huella del fichero, dentro de un entorno de tolerancia.
- 15Un procedimiento según cualquier reivindicación precedente, en el cual una correspondencia lineal entre una pareja de hito/huella de una muestra y una pareja de hito/huella de un fichero tiene lugar cuando se corresponden las respectivas huellas y los respectivos hitos están vinculados por una relación lineal. ES 2 266 254 T3
- 16Un procedimiento según la reivindicación 15, en el cual las huellas se corresponden cuando son idénticas o cuando difieren dentro de una tolerancia previamente determinada.
- 17Un procedimiento según la reivindicación 15 o la reivindicación 16, en el cual una correspondencia lineal tiene lugar si el par de hitos (hito n , hito* n ) de la muestra y del fichero, dentro de la lista, están vinculados según la relación:hito* n = m*hito n + desplazamiento.
- 18Un procedimiento según cualquier reivindicación precedente, en el cual la muestra tiene la forma de ondas acústicas, ondas de radio, un flujo PCM de audio digital, un flujo de audio digital comprimido, o una transmisión continua por Internet.
- 19Un procedimiento según cualquier reivindicación precedente, en el cual las huellas de la muestra se almacenan en un almacén circular de datos.
- 20Un procedimiento según la reivindicación 19, en el cual la etapa de identificación puede llevarse a cabo periódicamente sobre el contenido del almacén circular de datos de huellas.
- 21Un procedimiento según la reivindicación 19 o la reivindicación 20, en el cual la etapa de identificación puede llevarse a cabo tan pronto como se dispone en el almacén de información suficiente para el reconocimiento.
- 22Un procedimiento según cualquier reivindicación precedente, en el cual la etapa de identificación se lleva primero a cabo sobre un subconjunto de ficheros y, si no se identifica ningún fichero ganador en el primer subconjunto, se registra un segundo subconjunto, que contiene los ficheros restantes.
- 23Un procedimiento según la reivindicación 22, en el cual el primer subconjunto contiene ficheros que tienen una probabilidad, a priori o empírica, más alta de ser identificados que los ficheros que no están en el primer subconjunto.
- 24Un procedimiento según la reivindicación 1, en el cual dicha etapa de identificar correspondencias lineales comprende la localización de una línea diagonal dentro de un gráfico de dispersión de dichas ubicaciones correspondientes, formando las diferencias entre dichas ubicaciones correspondientes y calculando un pico de un histograma de dichas diferencias.
- 25Un procedimiento según la reivindicación 1, en el cual dicha etapa de identificar un fichero ganador comprende adicionalmente proporcionar un indicador de un desplazamiento con respecto a una ubicación en dicho fichero ganador, donde tiene lugar dicho número significativo de correspondencias.
- 26Un procedimiento para identificar una muestra de audio, que comprende las etapas de:como respuesta a una solicitud desde un cliente, retransmitir al menos una porción de la muestra de audio a un servidor, realizando dicho servidor las etapas del procedimiento de la reivindicación 1;y como respuesta a la identificación de un fichero ganador por dicho servidor, responder a dicho cliente en consecuencia.
- 27Un producto de programa de ordenador que realiza todas las etapas de un procedimiento según cualquier reivindicación precedente, cuando se carga en un ordenador.
- 28Un sistema informático dispuesto para llevar a cabo un procedimiento según cualquiera de las reivindicaciones 1 a 25, que incluye un extremo cliente que envía un resumen de rasgos extraídos de la muestra de señal capturada, que contiene pares de hitos y huellas, a un extremo servidor, el cual realiza el reconocimiento.
Independent claims28
128 paragraphs in 5 sections, as filed
ES 2 266 254 T3
DESCRIPTION
Search procedure for an audio database.
Field of the invention
This invention relates, in general, to the retrieval of information based on its content. More particularly, it relates to the recognition of an audio signal, including sound or music, that is highly distorted or contains a high level of noise.
Previous technique
There is a growing need for automatic recognition of music or other audio signals, generated by various sources. For example, owners of copyrighted works, or advertisers, are interested in obtaining data about the frequency of transmission of their material. Music tracking services provide broadcast lists of major radio stations in large markets. Consumers would like to identify songs or radio broadcasts so that they can purchase interesting new music or other products and services. All kinds of sound recognition, continuous or on demand, is inefficient and labor-intensive when performed by humans. An automated method of recognizing music or sound would therefore provide a significant benefit to consumers, artists, and a wide variety of industries. As the music distribution paradigm shifts from in-store purchases to Internet downloading, it is quite feasible to directly link computer-implemented music recognition with Internet acquisition and other Internet-based services.
Traditionally, the recognition of songs played on the radio has been done by matching the radio stations and the times at which the songs were played with the broadcast lists provided either by the radio stations or by third-party sources. This method is inherently limited to only the radio stations for which the information is available. Other methods rely on the insertion of inaudible codes into the transmitted signals. The inserted signals are decoded at the receiver in order to extract identification information about the transmitted signal. The disadvantage of this procedure is that special decoding devices are required to identify signals, and only those songs with inserted codes can be identified.
All large-scale audio recognition requires some kind of content-based audio retrieval, in which an unidentified transmitted signal is compared to a database of known signals in order to identify similar or identical signals from the database. . Note that content-based audio retrieval differs from existing audio retrieval, via internet search engines, in which only the metadata text surrounding, or associated with, the files of the content is searched. Audio. Also note that while speech recognition is useful for converting speech signals to text, which can then be indexed and queried using well-known techniques, it is not applicable to the vast majority of audio signals containing music and sounds. In some ways, audio information retrieval is analogous to text-based information retrieval provided by search engines. In other cases, however, audio recognition is not analog: audio signals lack easily identifiable entities, such as words, that provide identifiers for searching and for indexing. Thus, current audio retrieval methods index audio signals by computed perceptual characteristics that represent various qualities or features of the signal.
Content-based audio retrieval is typically performed by analyzing a query signal, in order to obtain a number of representative characteristics, and then applying a similar measure to the derived characteristics in order to locate database files that are as similar as possible to the query token. The similarity of the received objects is necessarily a reflection of the selected perceptual characteristics. A number of content-based retrieval procedures are available in the art. For example, US Patent No. 5,210,820, issued to Kenyon, discloses a signal recognition procedure in which received signals are processed and sampled to obtain signal values at each sampling point. The statistical moments of the sampled values are then computed to generate a feature vector that can be compared to the identifiers of the stored signals to extract similar signals. US Patent Nos. 4,450,531 and 4,843,562, both issued to Kenyon et al, disclose similar transmitted information classification procedures, in which cross-correlations between unidentified signals and stored reference signals are calculated.
A system for retrieving audio documents based on acoustic similarity is revealed in JT Foote's article, "Content-Based Retrieval of Music and Audio," in Multimedia Storage and Archiving. Systems II, Proc. of SPIE [Storage and Archive Systems II, Annals of SPIE], by C.-CJ Kuo et al, editor, volume 3229, pages 138-147, 1997. The characteristic vectors are calculated by parameterizing each audio file in cepstral coefficients of the Mel scale, and a quantization tree is generated from the parameterization data. To perform a query, an unknown signal is parameterized to obtain feature vectors that are then classified as terminal ends of the tree. A histogram is compiled for each terminal end, thereby generating an N-dimensional vector representing the unknown signal. The distance between two such vectors gives an indication of the similarity between two sound files. In
In this procedure, the supervised quantization method recognizes the distinctive characteristics of the audio, while ignoring unimportant variations, based on classes in which the learning data is assigned by a human. According to the classification system, different acoustic features are chosen as important. Therefore, this procedure is more suitable for finding similarities between songs and for classifying music in classes than for recognizing music.
A method for content-based analysis, storage, retrieval, and segmentation of audio information is disclosed in US Patent No. 5,918,223, issued to Blum et al. In this procedure, a number of acoustic characteristics, such as volume, bass, grade, boost, bandwidth, and Mel scale frequency cepstral coefficients, are measured at periodic intervals in each file. Statistical measurements of the features are taken and combined to form a feature vector. Audio data files within a database are extracted based on the similarity of their feature vectors to the feature vector of an unidentified file.
The article "Landmark detection for distinctive feature-based speech recognition", SA Liu, JASA, 100 (5) Nov. 1996, reveals a speech recognition system that uses landmarks to guide the search for distinctive features.
A key problem with all the preceding prior art audio recognition procedures is that they tend to fail when the signals to be recognized are subject to linear and non-linear distortion, caused for example by background noise, errors, and dropouts. transmission, interference, band-limited filtering, quantization, time warping, and digital compression of voice quality. In prior art methods, when a distorted sound sample is processed to obtain acoustic characteristics, only a fraction of the derived characteristics are found for the original recording. The resulting feature vector is therefore not very similar to the feature vector from the original recording, and it is unlikely that correct recognition can be performed. There remains a need for a sound recognition system that performs well under high noise and distortion conditions.
Another problem with prior art procedures is that they are computationally burdensome, and they do not scale well. Real-time recognition, therefore, is not possible using prior technology procedures with large databases. In such systems, it is unfeasible to have a database of more than a few hundred or thousands of recordings. Search time in prior technology procedures tends to grow linearly with the size of the database, making scaling up of millions of sound recordings economically unfeasible. Kenyon's procedures also require large banks of specialized digital signal processing hardware.
Existing business procedures often have strict requirements on the input sample in order to be able to perform recognition. For example, they require the entire song to be sampled, or at least 30 seconds of the song, or they require the song to be sampled from the beginning. They also have difficulty recognizing multiple songs mixed together in a single stream. All of these disadvantages make prior art procedures unfeasible for use in many practical applications.
Objects and advantages
Accordingly, it is a primary object of the present invention to provide a method for recognizing an audio signal subjected to a high level of noise and distortion.
It is a further object of the invention to provide a recognition procedure that can be carried out in real time, based only on a few seconds of the signal to be identified.
It is another object of the invention to provide a recognition method that can recognize sounds based on samples from virtually any position within the sound, not just from the beginning.
It is a further object of the invention to provide a recognition method that does not require sound samples to be encoded or correlated with specific radio stations or broadcast lists.
It is a further object of the invention to provide a recognition method that can recognize each of the multiple sound recordings mixed together in a single stream.
It is another object of the invention to provide a sound recognition system in which unknown sound can be supplied to the system from any environment, by means of practically any known method.
Summary
These objects and advantages are achieved by a method, as claimed in claim 1, for recognizing a sample of a certain medium, such as an audio sample, given a database index of a large number of known media files. The database index contains footprints that represent characteristics in particular locations of the indexed media files. The unknown media sample is identified by a
ES 2 266 254 T3 media file in the database (the winning media file) whose relative fingerprint locations coincide as closely as possible with the relative locations of the sample fingerprints. In the case of audio files, the temporal evolution of the tracks in the winning file coincides with the temporal evolution of the tracks in the sample.
The procedure is preferably implemented in a distributed computer system, and contains the following steps: determining a set of fingerprints at specific locations in the sample; locate matching footprints in the database index; generating correspondences between locations in the sample and locations in the file that have equivalent footprints; and identifying media files for which a significant number of the matches are essentially linearly linked. The file with the highest number of linearly linked matches is considered the winning media file. One procedure for identifying files with a large number of matches is to perform the equivalent of scanning a diagonal line in a scatter plot generated from the matching pairs. In one embodiment, identifying the media files with a large number of linear matches involves searching for only a first subset of the media files. Files in the first subset have a higher probability of being identified than files that are not in the first subset. The probability of identification is preferably based on measures of the empirical frequency or recent occurrence of previous identifications, together with a priori projections of the frequency of identification. If no media files are identified in the first subset, then the second subset, which contains the remaining files, is crawled. Alternatively, the files can be sorted according to probability, and tracked according to the order of categorization. The search ends when a file is found.
Preferably, specific locations within the sample are reproducibly calculated, depending on the sample. Such reproducibly calculable locations are called "milestones." The fingerprints are preferably numerical values. In one embodiment, each footprint represents a number of characteristics of the media sample at each location, or slightly offset from the location.
The procedure is especially useful for recognizing audio samples, in which case the specific locations are instants within the audio sample. These instants take place, for example, at the local maxima of the spectral Lp norms of the audio sample. The fingerprints can be calculated by any analysis of the audio sample, and are preferably invariant with respect to the time dilation of the sample. Examples of fingerprints include spectral slice fingerprints, multislice fingerprints, LPC coefficients, cepstral coefficients, and frequency components of the spectrogram peaks.
The present invention also provides a system for implementing the above procedure, which contains a landmark generator object to calculate the specific locations, a footprint generator object to calculate the footprints, a database index that contains the file locations and the footprints for media files, and an analysis generator object. The analysis generator object implements the procedure by locating the matching footprints in the database index, generating matches, and analyzing the matches in order to select the winning media file.
Also provided is a program storage device, accessible from a computer, that tangibly realizes a program of instructions executable by the computer in order to carry out the procedural steps for the foregoing procedure.
Furthermore, the invention provides a method for creating an index of a certain number of audio files in a database, which contains the following steps: calculation of a set of fingerprints at specific locations of each file; and storing the tracks, locations and identifiers of the files in a memory. A corresponding fingerprint, location, and identifier are associated in memory to form a triplet. Preferably, the locations, which can be instants within the audio file, are calculated in correspondence with the file, and are playable. For example, the instants can occur at the local maxima of the spectral Lp norms of the audio file. In some cases, each fingerprint, which is preferably a numerical value, represents a number of features in the file near the specific location. The fingerprints can be calculated from any analysis or digital signal processing of the audio file. Examples of fingerprints include spectral slice fingerprints, multislice fingerprints, LPC coefficients, cepstral coefficients, frequency components of spectrogram peaks, and linked spectrogram peaks.
Finally, the invention provides methods for identifying audio samples, incorporating time-dilation invariant fingerprints, and various hierarchical searches.
Brief description of the figures
Fig. 1 is a flow chart of a method of the invention for recognizing a sound sample.
Fig. 2 is a block diagram of an example of a distributed computing system for implementing the procedure of Fig. 1.
Fig. 3 is a flow chart of a method for building a database index of sound files used in the method of Fig. 1.
ES 2 266 254 T3
Fig. 4 schematically illustrates the milestones and footprints calculated for a sound sample.
Fig. 5 is a chart of L4 norms for a sound sample, illustrating the selection of milestones.
Fig. 6 is a flow chart of an alternative embodiment for building a database index of sound files used in the method of Fig. 1.
Figs. 7A-7C show a spectrogram, indicating salient points and linked salient points.
Figs. 8A-8C illustrate index sets, an index list and a master index list of the procedure of Fig. 3.
Figs. 9A-9C illustrate an index list, a candidate list and a hash list of the method of Fig. 1.
Figs. 10A-10B are scatter plots illustrating correct identification and non-identification, respectively, of an unknown sound sample.
Detailed description
The present invention provides a method for recognizing a sample of exogenous media, given a database containing a large number of known media files. It also provides a method for generating a database index that allows efficient search using the recognition method of the invention. While the following discussion is primarily concerned with audio data, it is to be understood that the method of the present invention can be applied to any type of media samples and media files, including, but not limited to, text, audio, video, image, and any multimedia combinations of individual media types. In the case of audio, the present invention is especially useful for recognizing samples that contain high levels of linear and non-linear distortion, caused for example by background noise, transmission errors and stretches of silence, interference, band-limited filtering. , quantization, time warp, and voice quality digital compression. As will become apparent from the following description, the invention operates under such conditions because it can correctly recognize a distorted signal, even if only a small fraction of the calculated characteristics survive the distortion. Any type of audio, including sound, voice, music, or combinations of types, can be recognized by the present invention. Examples of audio samples include recorded music, radio broadcasts, and commercials.
As used herein, an exogenous media sample is a segment of media data of any size, obtained from a wide variety of sources, as described below. In order for recognition to take place, the sample must be a version of part of a media file indexed in a database used by the present invention. The indexed media file can be thought of as an original recording, and displayed as a distorted and / or abridged version of the original recording. Typically, the sample corresponds to only a small portion of the indexed file. For example, recognition can be performed on a ten second segment of a five minute song, indexed in the database. Although the term "file" is used to describe the indexed entity, the entity can be in any format for which the necessary values can be obtained (described below). Also, there is no need to store or access the file after the values have been obtained.
A block diagram, conceptually illustrating the general steps of a method 10 of the present invention, is shown in Fig. 1. The individual steps are described in more detail below. The procedure identifies a winning media file, a media file whose relative locations of characteristic fingerprints more closely match the relative locations of the same fingerprints in the exogenous sample. After an exogenous sample has been captured at stage 12, milestones and footprints are calculated at stage 14. Milestones appear at specific locations, eg. eg, temporary instants, within the sample. The location within the sample of the landmarks is preferably determined by the sample itself, that is, it depends on the qualities of the sample, and is reproducible. That is, the same milestones are calculated for the same signal each time the process is repeated. For each landmark, a footprint is obtained that characterizes one or more features of the sample at or near the landmark. The proximity of a feature to a landmark is defined by the fingerprint determination procedure used. In some cases, a feature is considered close to a milestone if it clearly corresponds to the milestone and not to a previous or subsequent milestone. In other cases, the features correspond to multiple adjacent landmarks. For example, textual fingerprints can be strings of words, audio fingerprints can be spectral components, and image fingerprints can be pixel RGB color values. Two general embodiments of step 14 are described below: one in which the milestones and footprints are calculated sequentially, and one in which they are calculated simultaneously.
In step 16, the sample fingerprints are used to retrieve matching fingerprint sets stored in a database index 18, in which the matching fingerprints are associated with milestones and identifiers from a set of media files. The set of retrieved file identifiers and milestone values is then used to generate the matching pairs (step 20), which contain the sample milestones (calculated in step 14) and the retrieved file milestones, in which the they calculated the same footprints. The calculated match pairs are then sorted based on the song identifier, generating match sets between milestones
ES 2 266 254 T3 of file samples and milestones for each relevant file. Each set is examined for the alignment between the file milestones and the sample milestones. That is, linear correspondences are identified in the pairs of milestones, and the set is scored according to the number of pairs that are linearly linked. Linear mapping occurs when a large number of corresponding sample locations and file locations can be described with essentially the same linear equation, within a supported tolerance. For example, if the slopes of a certain number of equations, describing a set of pairs of correspondences, vary by ± 5%, then the entire set of correspondences is considered to be linearly linked. Of course, any suitable tolerance can be selected. The identifier of the set with the highest score, that is, with the highest number of linearly linked matches, is the identifier of the winning file, which is located and generated in step 22.
As described further below, recognition can be carried out with a time component proportional to the logarithm of the number of entries in the database. Recognition can be carried out essentially in real time, even with a very large database. That is, a sample can be recognized as it is recovering, with a small time lag. The procedure can identify a sound based on segments of 5 to 10 seconds, and even up to 1 to 3 seconds. In a preferred embodiment, the milestone and footprint analysis, in step 14, is performed in real time, as the sample is being captured in step 12. Queries to the database (step 16) are carried out as sample fingerprints become available, and the results of the match are accumulated and periodically examined, looking for linear matches. In this way, all stages of the procedure take place simultaneously, and not in the sequential linear style suggested in Fig. 1. Note that the procedure is partially analogous to a text search engine: a user submits a query sample, and a corresponding file is returned, indexed in the database.
The procedure is typically implemented in the form of software running on top of a computer system, with the individual steps implemented, most efficiently, as separate software modules. Thus, a system implementing the present invention can be considered to consist of a milestone and footprint determination object, an indexed database, and a parser object for searching the database index, calculating matches, and identifying the winning file. In the case of sequential determination of milestones and tracks, the object of determination of milestones and tracks can be considered as two different objects of determination of milestones and tracks. The computer instruction code for the various objects is stored in a memory of one or more computers, and is executed by one or more computer processors. In one embodiment, the code objects are concentrated on a single computer system, such as an Intel hardware-based personal computer, or other workstations. In a preferred embodiment, the method is implemented by a networked cluster of central processing units (CPUs), in which different software objects are executed by different processors, in order to distribute the computing workload. Alternatively, each CPU can have a copy of all software objects, allowing for a homogeneous network of identically configured elements. In the latter configuration, each CPU has a subset of the database index, and is responsible for searching its own subset of media files.
Although the invention is not limited to any particular hardware system, an example of a preferred embodiment of a distributed computing system 30 is schematically illustrated in Fig. 2. System 30 contains a cluster of Linux-based processors 32a-32f, connected by a multiprocessor bus architecture 34, or by a network protocol such as the Beowulf Cluster Computing Protocol, or by a mixture of the two. In such an arrangement, the database index is preferably stored in random access memory (RAM) at at least one end 32a in the cluster, ensuring that the fingerprint search is performed very quickly. The computational endpoints for the other objects, such as the milestone end 32c and 32f, the footprint endpoint 32b and 32e, and the alignment tracking end 32d, do not require as much RAM memory in raw as the 32nd endpoint (s) that support the database index. The number of endpoints in charge of calculations, assigned to each object, can therefore be scaled according to need, such that no single object becomes a bottleneck. The computational network is therefore highly parallelizable, and can additionally process multiple simultaneous signal recognition queries that are spread across the available computing resources. Note that this enables applications in which large numbers of users can request recognition and receive results in approximately real time.
In an alternative embodiment, certain functional objects are more closely coupled to each other, while being less closely coupled to other objects. For example, the milestone and footprint determination object may reside in a physically separate location from other objects that are responsible for calculations. An example of this is a close association of the footprint and landmark determination objects with the signal capture process. In this arrangement, the milestone and fingerprint determination object can be incorporated as additional embedded hardware or software, for example, in a mobile phone, Wireless Application Protocol (WAP) browser, PDA or other terminal. remote, such as the client end of an audio search engine. In an Internet-based audio search service, such as a content identification service, the milestone and fingerprint determining object can be incorporated into the client browser application as a linked set of software instructions, or as a pluggable module of software, such as a Microsoft dynamic link library (DLL). In these embodiments, the combined signal capture, milestone, and fingerprint object constitutes the client end of the service. The client end sends a summary, extracted from the characteristics, of the captured signal sample, which contains pairs of milestones and footprints, to the end
ES 2 266 254 T3 server, which performs recognition. Sending this extracted summary of features to the server, rather than the raw captured signal, is advantageous because the magnitude of data is greatly reduced, often by a factor of 500 or more. Such information can be sent in real time over a low bandwidth side channel, in conjunction with, or instead of, for example, an audio stream transmitted to the server. This allows the invention to be carried out on public communication networks, which offer relatively low bandwidths to each user.
The procedure will now be described in detail with reference to audio samples and audio files indexed in a sound database. The procedure consists of two general components, the construction of the sound database index and the recognition of samples.
Building the database index
Before sound recognition can be performed, an index of the traceable sound database must be built. As used here, a database is any indexed collection of data, and is not limited to commercially available databases. In the database index, linked data items are associated with each other, and individual items can be used to retrieve associated data. The sound database index contains a set of indexes for each file or recording in the selected collection or in the library of recordings, which may include voice, music, announcements, sonar rubrics, or other sounds. Each recording also has a unique identifier, Sound_Identifier. The sound database itself does not necessarily store the audio files for each recording, but the Sound_Identifiers can be used to extract the audio files from other sites. The sound database index is expected to be very large, containing indexes for millions and even billions of files. New recordings are preferably added incrementally to the database index.
A block diagram of a preferred method 40 for building the trackable index of the sound database, according to a first embodiment, is shown in Fig. 3. In this embodiment, milestones are first calculated, and then milestones are calculated. footprints at, or near, the milestones. As will be apparent to the average tech savvy, alternative procedures for building the database index can be devised. In particular, many of the steps listed below are optional, but they serve to generate a database index that is crawled more efficiently. While tracking efficiency is important for real-time sound recognition from large databases; small databases can be crawled relatively quickly, even if they have not been optimally classified.
In order to index the sound database, each recording in the collection undergoes a landmark and fingerprint analysis that generates a set of indexes for each audio file. Fig. 4 schematically illustrates a segment of a sound recording for which milestones and footprints have been calculated. Milestones appear at specific times in the sound, and take values of time units shifted from the beginning of the file, while footprints characterize the sound at or near a specific landmark. Thus, in this embodiment, each landmark for a particular file is unique, while the same footprint may appear numerous times within a single file, or multiple files.
At step 42, each sound recording is assigned a milestone using procedures to find distinguishable and reproducible locations within the sound recording. A preferred landmark determination algorithm is capable of marking the same instants within a sound recording, despite the presence of noise and other linear and nonlinear distortion. Some milestone determination procedures are conceptually independent of the fingerprint determination process described below, but can be chosen to optimize the performance of the latter. The milestone determination results in a list of times (milestone<sub>k</sub>) within the sound recording in which the fingerprints are subsequently calculated. A good method of determining milestones marks between 5 and 10 milestones per second of sound recording; of course, the density of the landmarks depends on the magnitude of the activity within the sound recording.
A wide variety of techniques for calculating milestones are possible, all of which are within the scope of the present invention. The specific technical processes employed to implement the inventive landmark determination methods are known in the art, and will not be discussed in detail. A simple landmark determination technique, known as the Power Norm, is to calculate the instantaneous power at every possible instant in the recording, and select the local maxima. One way to do this is to calculate the envelope by directly rectifying and filtering the wave. Another way is to calculate the Hilbert transform (quadrature) of the signal, and use the sum of the square of the magnitudes of the Hilbert transform and the original signal.
The Power Standard procedure for determining milestones is good for finding transient components in the sound signal. The Power Norm, in effect, is a special case of the more general Lp Spectral Norm, in which p = 2. The general Lp Spectral Norm is calculated at each moment along the sound signal, calculating a time spectrum reduced, for example, by means of a Fast Fourier Transform (FFT) with Hanning windows. A preferred embodiment uses a sample rate of 8000 Hz, an FFT frame size of 1024 samples, and a stride of 64 samples for each time slice. The Lp norm for each temporal slice is then calculated as the sum of the pth power of the absolute values of the spectral components, after which, optionally, the pth root is extracted. As before, the milestones are chosen as the local maxima of the resulting values over time. An example of the Lp Spectral Norm procedure is shown in the
ES 2 266 254 T3
Fig. 5: A graph of the L4 norm as a function of time for a particular sound signal. Dotted lines at local maxima indicate the location of the chosen landmarks.
When p = ro, the norm Lro is, in effect, the maximum norm. That is, the value of the norm is the absolute value of the largest spectral component in the spectral slice. This standard results in robust milestones and good overall recognition performance, and is preferred for tonal music.
Alternatively, the "multislice" spectral milestones can be calculated by taking the sum of the worst powers of the absolute values of the spectral components over the multiple time slices, with fixed or variable offsets from each other, rather than a single slice. Finding the local maxima of this extended sum allows the optimization of the location of the multitask footprints, described below.
Once the milestones have been calculated, a footprint is calculated at each milestone instant in the recording, in step 44. The footprint is generally a value, or a set of values, that summarizes a set of characteristics in the recording in , or close to, the time instant. In a presently preferred embodiment, each fingerprint is an individual numerical value that is a multi-feature recast function. Possible types of fingerprints include spectral slice fingerprints, multislice fingerprints, LPC coefficients, and cepstral coefficients. Of course, any type of footprint that characterizes the sign, or features of the sign, near a landmark is within the scope of the present invention. The fingerprints can be calculated by any type of digital signal processing or signal frequency analysis.
To generate spectral slice fingerprints, a spectral analysis is performed in the vicinity of each time point of a landmark, in order to extract the various maximum spectral peaks. A single fingerprint value is just the single frequency value of the strongest spectral peak. Using such a simple peak results in surprisingly good recognition in the presence of noise; however, single frequency spectral slice fingerprints tend to generate more false positive values than other fingerprint determination methods, because they are not unique. The number of false positive values can be reduced by using fingerprints that consist of a function of the two or three strongest spectral peaks. However, there may be a higher susceptibility to noise if the second strongest spectral peak is not strong enough to distinguish it from its competitors in the presence of noise. That is, the calculated value of the fingerprint may not be robust enough to be reliably reproducible. Despite this, the benefits of this case are also good.
In order to take advantage of the temporal evolution of many sounds, a set of temporal slices is determined, adding a set of temporal offsets to a landmark instant. On each resulting time slice, a spectral slice footprint is calculated. The resulting set of fingerprint information is then combined to form a multitone or multitone fingerprint. Each multi-slice fingerprint is much more specific than the individual spectral slice fingerprint, because it tracks time evolution, resulting in fewer false matches in the database index lookup, described below. Experiments indicate that, due to their increased uniqueness, multislice fingerprints calculated from the strongest individual spectral peak in each of the two time slices result in a much faster calculation (about 100 times faster) in the search. subsequent index of the database, but with some degradation in the recognition percentage, in the presence of significant noise.
Alternatively, instead of using one or more fixed offsets from a given time slice, in order to calculate a multi-slice footprint, variable offsets can be employed. The variable offset from the chosen slice is the offset to the next milestone, or a milestone in a certain offset environment from the "anchor" milestone for the footprint. In this case, the time difference between milestones is also encoded in the fingerprint, along with multi-frequency information. By adding more dimensions to the footprints, they become more specific and have a lower probability of a false match.
In addition to spectral components, other spectral features such as fingerprints can be extracted and used. Linear Predictive Coding (LPC) analysis extracts linearly predictable features of a signal, such as spectral peaks, as well as spectral shape. LPC is well known in the art of digital signal processing. For the present invention, the LPC coefficients of the wave slices anchored at the landmark positions can be used as footprints by recasting the quantized LPC coefficients into an index value.
Cepstral coefficients are useful as a measure of periodicity, and can be used to characterize signals that are harmonic, such as voices or many musical instruments. Cepstral analysis is well known in the art of digital signal processing. For the present invention, a number of cepstral coefficients are merged together in an index, and used as a fingerprint.
An alternative embodiment 50, in which milestones and footprints are simultaneously calculated, is shown in Fig. 6. Steps 42 and 44 of Fig. 3 are replaced by steps 52, 54 and 56. As described below, A multidimensional function is calculated from the sound recording in step 52, and the landmarks (54) and footprints (56) of the function are extracted.
In one implementation of the embodiment of Fig. 6, landmarks and footprints are calculated from a spectrogram of the sound recording. A spectrogram is a time-frequency analysis of a sound recording in which
ES 2 266 254 T3 spectrally analyze overlapping and windowing frames of sound samples, typically using a Fast Fourier Transform (FFT). As before, a preferred embodiment uses a sampling rate of 8000 Hz, an FFT frame size of 1024 samples, and a stride of 64 samples for each time slice. An example of a spectrogram is shown in Fig. 7A. Time is on the horizontal axis, and frequency is on the vertical axis. Each sequential FFT frame is vertically stacked at corresponding, equally spaced intervals along the time axis. A graph of the spectrogram illustrates the energy density at each time frequency point; the darkest areas on the graph represent the highest energy density. Spectrograms are well known in the art of digital signal processing. For the present invention, landmarks and footprints can be obtained from salient points, such as local maxima of the spectrogram, circled in the spectrogram of Fig. 7B. For example, the time and frequency coordinates of each peak are obtained, the time to be used as a landmark is taken, and the frequency is used to calculate the corresponding footprint. This spectrogram peak landmark is similar to the I <»norm, in which the absolute maximum value of the norm determines the location of the landmark. In the spectrogram, however, the search for the local maximum is done over sections of the time-frequency plane, rather than over an entire time slice.
In this context, the set of salient points that result from the point extraction analysis of a sound recording is called a constellation. For a constellation consisting of local maxima, a preferred analysis is to select points that are energy maxima of the time-frequency plane in a neighborhood around each selected point. For example, a coordinate point (t<sub>0</sub>,F<sub>0</sub>) is selected if it is the point of maximum energy within a rectangle with vertices (t<sub>0</sub>-T, f<sub>0</sub>-F), (t<sub>0</sub>-T, f<sub>0</sub>+ F), (t<sub>0</sub>+ T, f<sub>0</sub>-F) and (t<sub>0</sub>+ T, f<sub>0</sub>+ F), that is, a rectangle with sides of length 2T and 2F, with T and F chosen to provide an adequate number of constellation points. The boundaries of the rectangle can also vary in size depending on the frequency value. Of course, a region can be used in any way. The maximum energy criterion can also be weighted in such a way that a competing energy peak in terms of time and frequency is weighted inversely with respect to a distance metric in the time-frequency plane, that is, the most distant points have a lower weight. For example, energy can be weighted as
S (t, f) + Ct (t - to)<sup>2</sup> + Cf (f - fo)<sup>2 ,</sup> where S (t, f) is the square of the value of the magnitude of the spectrogram at the point (t, f), and Ct and Cf are positive values (not necessarily constant). Other distance weighting functions are possible. Local maximum selection constraints may be applied to other salient point (non-maximum) feature extraction methods, and are within the scope of the invention.
This procedure results in pairs of values that are very similar to the single frequency spectral fingerprint described above, with many of the same properties. The spectrogram time-frequency procedure generates more milestone / fingerprint pairs than the single-frequency procedure, but it can also produce many false matches in the matching step described below. However, it provides a more robust determination of landmarks and fingerprints than the single frequency spectral fingerprint, because the dominant noise in the sound sample may not spread to all parts of the spectrum at every slice. That is, there are, most likely, some pairs of landmarks and tracks in parts of the spectrum that are not affected by the dominant noise.
This procedure for determining spectrogram landmarks and traces is a special case of feature analysis procedures that calculate a multidimensional function of the sound signal, in which one of the dimensions is time, and that locate salient points in the values. functional. The salient points can be local maxima, local minima, null ordinate values, or other distinctive features. The milestones are taken as the temporal coordinates of the salient points, and the corresponding footprints are calculated from at least one of the remaining coordinates. For example, the non-temporal coordinate (s) of the multidimensional salient point may merge with each other to form a multidimensional functional footprint.
The variable shift procedure described above for multi-slice spectral fingerprints can be applied to the spectrogram or other fingerprints of multidimensional functions. In this case, the points in a constellation are linked together to form linked points, as illustrated in the spectrogram shown in Fig. 7C. Each point in the constellation serves as an anchor point that defines the moment of the milestone, and the remaining coordinate values of the other points combine to form the linked footprint. Points that are close to each other, for example, as defined below, link together to form more complex fingerprints of compound features, which can be more easily distinguished and searched for. As with multisite spectral fingerprints, the goal of combining information from multiple linked salient points into a single fingerprint is to create a greater diversity of possible fingerprint values, thereby decreasing the probability of a false match, that is, decreasing the probability of that the same footprint describes two different musical samples.
In principle, each of the N salient points can be linked to every other point in a two-point linking method, yielding around N<sup>2</sup>/ 2 combinations. Similarly, for a link of K points, the number of possible combinations resulting from a constellation is of the order of N<sup>K</sup>. In order to avoid such a combinatorial explosion, it is desirable to restrict the neighborhood of points that link to each other. One way to achieve such a restriction is to define a "target zone" for each anchor point. An anchor point is then linked to points in its area
ES 2 266 254 T3 objective. It is possible to select a subset of points within the target zone to link to - not every point needs to be linked. For example, only the points associated with the strongest peaks in the target zone can be linked. A target zone can have a fixed shape or vary according to the characteristics of the anchor point. A simple example of an anchor point target zone (t<sub>0</sub>,F<sub>0</sub>) for a constellation of spectrogram peaks is the set of points (t, f) on the spectrogram strip, such that t is in the interval [t<sub>0</sub>+ L, t<sub>0</sub>+ L + W], where L is the time advance and W is the width of the target area. In this method, all frequencies are allowed in the target zone. L or W can be variable, for example, if a speed control mechanism is used to modulate the number of link combinations that occur. Alternatively, frequency constraints can be implemented, for example, restricting the target area such that the frequency f is in the interval [f<sub>0</sub>F, f0 + F], where F is a bounding parameter. An advantage of a frequency constraint is that in psychoacoustics it is known that melodies tend to coalesce better when sequences of notes have frequencies that are close to each other. Such a restriction may allow more "psychoacoustically realistic" recognition performances, although modeling of psychoacoustics is not necessarily an objective of this invention. It is also possible to consider the opposite rule, in which f is chosen outside the region [f0-F, f0 + F]. This forces the linkage of points that are different from each other in frequency, possibly avoiding cases in which constellation extraction devices produce jagged sequences of points of time and frequency values that are close in time and have the same frequency. . As with other locality parameters, F is not necessarily constant and may, for example, be a function of f0.
When including temporal coordinates of non-anchor salient points in the footprint values, relative temporal values must be used to allow the footprints to be time invariant. For example, the footprints can be a function of (i) non-temporal coordinate values and / or (ii) the difference (s) of the corresponding temporal coordinate values of the salient points. The time difference (s) may be taken, for example, with respect to the anchor point or as successive differences between the sequential salient points in the linked set. The coordinate and difference values can be packed into concatenated bit fields to form the recast fingerprint. As will be apparent to one of ordinary skill in the art, there are many other ways of mapping coordinate values to a fingerprint value and are within the scope of the present invention.
A concrete instantiation of this method uses N> 1 linked peaks from the spectrogram with coordinates (t<sub>k</sub>,F<sub>k</sub>), k = 1, ..., N. Then, (i) the time ti of the first peak is taken as the time of the milestone, and (ii) the time differences At<sub>k </sub>= t<sub>k</sub> -t<sub>1</sub>, k = 2, ..., N, plus the frequencies f<sub>k</sub>, k = 1, ..., N, of the linked peaks, merge together to form a fingerprint value. The footprint can be calculated from all, or a subset of, the coordinates At<sub>k</sub> and f<sub>k</sub> available. For example, some, or all, of the time difference coordinates can be omitted if desired.
Another advantage of using multiple points to form the fingerprint is that the fingerprint coding can be made invariant with respect to time dilation, e.g. For example, when a sound recording is played at a speed other than the original recording speed. This advantage applies to both the spectrogram and time slice procedures. Note that in a time-delayed signal, time differences and frequency have a reciprocal relationship (p. (e.g., decreasing the time difference between two points by a factor of two doubles the frequency). This procedure takes advantage of this fact by combining temporal differences and frequencies, in a way that excludes the temporal dilation of the footprint.
For example, in a case of peaks of an N-point spectrogram with coordinate values (t<sub>k</sub>,F<sub>k</sub>), k = 1, ..., N, the intermediate values available to be recast into a footprint are At<sub>k</sub> = t<sub>k</sub> -1<sub>1</sub>, k = 2, ..., N, and f<sub>k</sub>, k = 1, ..., N. The intermediate values can then be made invariant with respect to time dilation, taking one of the frequencies as the reference frequency, say f<sub>1</sub>, and forming (i) quotients with the remaining frequencies and (ii) products with the temporal differences. For example, intermediate values can be g<sub>k</sub> = f<sub>k</sub>/F<sub>1</sub>, k = 2, ..., N and s<sub>k</sub> = At<sub>k</sub> F<sub>1</sub> , k = 2, ..., N. If the sample is accelerated by a factor a, then the frequency f<sub>k</sub> becomes Af<sub>k</sub>, and the time difference At<sub>k</sub> becomes At<sub>k</sub>/ a, so that g<sub>k</sub> = Af<sub>k</sub>/ Af<sub>1</sub> = f<sub>k</sub>/F<sub>1</sub>, and yes<sub>k</sub> = (At<sub>k</sub>/ a) (af<sub>1</sub>) = At<sub>k</sub> F<sub>1</sub>. These new intermediate values are then combined using a function to form a recast footprint value that is independent of time dilation. For example, the gk and sk values can be merged by packing them into concatenated bit fields.
Alternatively, instead of a reference frequency, a reference time difference, e.g. eg, At<sub>2</sub>. In this case, the new intermediate values are calculated as (i) the quotients At<sub>k</sub>/ At<sub>2</sub> of the remaining temporary differences, and (ii) the products At<sub>2</sub> F<sub>k</sub> with frequencies. This case is equivalent to using a reference frequency, because the resulting values can be formed from products and quotients of the preceding values g<sub>k</sub> and yes<sub>k</sub>. The reciprocals of the frequency ratios can be used equally effectively; the sums and differences of log values of the original intermediate values can also substitute for the products and differences, respectively. Any independent fingerprint value of time dilation, obtained by means of such commutations, substitutions and permutations of mathematical operations, is within the scope of the invention. In addition, multiple reference frequencies or reference time differences can be used, which also make time differences relative. Using multiple reference frequencies or reference time differences is equivalent to using a single reference, because the same result can be achieved by arithmetic manipulation of the gk and sk values.
Turning now to Figs. 3 and 6, the footprint and milestone determination analyzes, by any of the preceding procedures, result in a set of indices for each Sound_Identifier, as shown
ES 2 266 254 T3 in Fig. 8A. A set of indices for a given sound recording is a list of value pairs (footprint, milestone). Each indexed recording typically has on the order of a thousand pairs (footprint, milestone) in its index set. In the first embodiment described above, in which the techniques for determining landmarks and footprints are essentially independent, they can be treated as separate and interchangeable modules. Depending on the system, the quality of the signal, or the type of sound to be recognized, one of several different modules for determining milestones or footprints may be used. Indeed, because the set of indices is simply composed of pairs of values, it is possible, and often preferable, to use multiple footprint and milestone determination methods simultaneously. For example, a landmark and fingerprint determination method may be good at detecting unique tonal patterns but poor at identifying percussion, while a different algorithm may have the opposite attributes. Employing multiple footprint / landmark determination strategies results in a richer and more robust range of recognition capabilities. Different fingerprint determination techniques can be used together, reserving certain ranges of fingerprint values for certain kinds of fingerprints. For example, in a 32-bit fingerprint value, the first 3 bits can be used to specify which of the 8 fingerprint determination methods are encoding the next 29 bits.
After sets of indexes have been generated for each sound recording to be indexed in the sound database, a crawlable database index is constructed in such a way as to allow fast searches (ie, logarithmic times). This is accomplished in step 46 by building a list of triplets (footprint, milestone, sound_identifier), obtained by adding the corresponding sound_identifier to each pair within each set of indices. All such triplets, for all sound recordings, are collected in a large index list, an example of which is shown in Fig. 8B. In order to optimize the subsequent search process, the list of triplets is then sorted with respect to the footprint. Quick sort algorithms are well known in the art, and are discussed in detail in DE Knuth, Reading, The Art of Computer Programming, Volume 3: Sorting and Searching. , Massachusetts: Addison Wesley, 1998, incorporated herein by reference. High performance sorting algorithms can be used to sort the list in a time equivalent to N log N, where N is the number of items in the list.
Once the index list is sorted, it is further processed in step 48, segmenting it in such a way that each unique fingerprint in the list is collected into a new master index list, an example of which is shown in Fig. 8C. Each item in the master index list contains a fingerprint value and a pointer to a list of pairs (milestone, sound_identifier). Depending on the number and character of the indexed records, a given footprint can appear hundreds of times, or more, within the entire collection. Reordering the index list into a master index list is optional, but saves memory, because each fingerprint value appears only once. It also speeds up the subsequent search in the database, since the effective number of items in the list is greatly reduced, down to a list of unique values. Alternatively, the master index list can be built by inserting each triplet into a B-tree. There are other possibilities for constructing the master index list, as is known to those of ordinary skill in the technology. The master index list is preferably kept in system memory, such as in DRAM memory, for quick access during signal recognition. The master index list can be kept in the memory of a single endpoint within the system, as illustrated in Fig. 2. Alternatively, the master index list can be decomposed into chunks distributed across multiple compute endpoints. Preferably, the above-mentioned sound database index is the master index list illustrated in Fig. 8C.
The index of the sound database is preferably built offline, and is incrementally updated as new sounds are incorporated into the recognition system. To update the list, new footprints can be inserted at the appropriate location in the master list. If the new recordings contain existing tracks, the corresponding pairs (milestone, sound_identifier) are added to the existing lists for those tracks.
Recognition system
Using the master index list generated as described above, sound recognition is carried out on an exogenous sound sample, typically provided by a user interested in identifying the sample. For example, the user hears a new song on the radio and wants to know the artist and song title. The sample can originate in any type of environment, such as a radio broadcast, a record, a pub, a submarine, a sound file, a streamed audio segment, or a stereo system, and it can contain background noise, sections of silence or voices. The user can store the audio sample on a storage device such as an answering machine, a computer file, a tape recorder, or a landline or mobile phone voicemail system, before providing it to the system for recognition. Based on the system configuration and user restrictions, the audio sample is provided to the recognition system of the present invention from any number of analog or digital sources, such as a stereo system, a television, a record player. compact, a radio transmission, an answering machine, a landline, a mobile phone, a live webcast, FTP, a computer file as an email attachment, or any other suitable means to transmit such recorded material. Depending on the source, the sample can be in the form of acoustic waves, radio waves, a PCM digital audio stream, a compressed digital audio stream (such as Dolby Digital or MP3), or a live webcast. A user interacts with the recognition system through a standard interface such as a landline phone, a mobile phone, an Internet browser, or email. The sample can be captured by the system and processed in real time, or it can be played back for processing from a previously captured sound (eg, a sound file). During capture, the audio sample is digitally sampled and sent to the system by a sampling device, such as a microphone. Depending on the procedure
ES 2 266 254 T3, the sample is likely to undergo further degradation, due to channel or sound capture device limitations.
Once the sound signal has been converted into its digital form, it is processed for recognition. As in the construction of sets of indexes for the database files, the milestones and footprints are calculated for the sample using the same algorithm that was used to process the database of sound recordings. The procedure works best if processing a highly distorted version of the original sound file produces the identical, or similar, set of pairs of landmarks and tracks that was obtained for the original recording. The resulting set of indices for the sound sample is a set of pairs of analyzed values, (footprint, milestone), shown in Fig. 9A.
Given the pairs for the sound sample, the database index is searched to locate potentially matching files. The search is carried out as follows: each pair (footprint<sub>k</sub>, milestone<sub>k</sub>) in the set of indices of the unknown sample is processed looking for the fingerprint<sub>k</sub> in the master index list. Quick search algorithms in an ordered list are well known in the art and are discussed extensively in DE Knuth's The Art of Computer Programming, Volume 3: Sorting and Searching. , Reading, Massachusetts: Addison-Wesley, 1998. If the footprint<sub>k</sub> is in the master index list, then its corresponding list of matching pairs (milestone * j, sound_identifierj) is copied and expanded with the milestone<sub>k</sub> to form a set of triplets of the form (landmark<sub>k</sub>, milestone * j, sound_identifierj). In this notation, an asterisk (*) indicates a landmark of one of the indexed files in the database, while a landmark without an asterisk refers to the sample. In some cases, it is preferable that the matching footprints are not necessarily identical, but are similar; for example, that they differ within a predetermined threshold. Matching footprints, whether identical or similar, are called equivalents. The sound_identifierj in the triplet corresponds to the file that has the milestone marked with an asterisk. In this way, each triplet contains two different milestones, one in the database index and one in the sample, in which equivalent footprints have been calculated. This process is repeated for all k that varies within the set of indices of the input sample. All the resulting triplets are collected into a large candidate list, illustrated in Fig. 9B. The candidate list is so named because it contains the sound_identifiers of the sound files which, by virtue of their matching fingerprints, are candidates for identification with the exogenous sound sample.
Once the candidate list has been compiled, it is further processed by segmenting it based on the sound_identifier. A convenient way to do this is to sort the candidate list by their sound_identifier, or insert it into a B-tree. A large number of sorting algorithms are available in the art, as discussed above. The result of this process is a list of candidate sound_identifiers, each of which has a scatter list of sample and file point time milestone pairs, with the sound_identifiers optionally removed, (milestone<sub>k</sub>, milestone * j), as shown in Fig. 9C. Each dispersion list therefore contains a set of corresponding milestones, by virtue of being characterized by the value of the equivalent footprint.
The scatter list for each candidate sound_identifier is then analyzed to determine if the sound_identifier corresponds to the sample. An optional threshold determination step may be employed first in order to eliminate a potentially high number of candidates who have very small hash lists. Obviously, candidates who have only one entry in their scatter lists, that is, only one fingerprint in common with the sample, do not correspond to the sample. Any suitable threshold number, greater than or equal to one, can be used.
Once the final number of candidates has been determined, the winning candidate is located. If the following algorithm does not locate a winning candidate, then a failure message is returned. A key concept of the matching process is that the time evolution in the matching of sounds must follow a linear correspondence, assuming that the time bases on both sides are constant. This is almost always true, unless one of the sounds has been deliberately warped non-linearly, or subjected to faulty playback equipment, such as a tape deck with a playback speed jerk problem. In this way, the correct pairs of milestones (milestone<sub>n</sub>,milestone*<sub>n</sub>) in the scatter list of a given sound_identifier must have a linear correspondence of the form milestone *<sub>n</sub> = m * milestone<sub>n</sub> + displacement, where m is the slope, which should be close to one; milestone<sub>n</sub> is the instant within the exogenous sample; milestone*<sub>n</sub> is the corresponding instant within the sound recording indexed by the sound_identifier; and the offset is the time offset within the sound recording corresponding to the beginning of the exogenous sound sample. The pairs of milestones that can satisfy the above equation for particular values of m and the displacement are said to be linearly related. Obviously, the concept of being linearly related is only valid for more than a couple of corresponding milestones. Note that this linear relationship identifies the correct sound file with high probability, while excluding pairs of external landmarks that are not significant. While it is possible that two different signals contain a number of identical tracks, it is highly unlikely that these tracks have the same relative time evolutions. The requirement for linear correspondences is a key feature of the present invention, and provides significantly better recognition than techniques that simply count the total number of features in common or that measure the similarity between features. Indeed, because of this aspect of the
In the invention, sounds can be recognized even if less than 1% of the traces of the original recording appear in the exogenous sound sample, that is, if the sound sample is very short or if it is significantly distorted.
The problem of determining whether there is a match for the exogenous sample thus reduces to the equivalent of finding a diagonal line with a slope close to one within a scatterplot of the milestone points of a given scatter list. Two examples of scatter plots are shown in Figs. 10A and 10B, with landmarks of sound files on the horizontal axis and landmarks of exogenous sound samples on the vertical axis. In Fig. 10A, a diagonal line of slope approximately equal to one is identified, which indicates that the song, indeed, corresponds to the sample, that is, that the sound file is a winning file. The intercept on the horizontal axis indicates the offset within the audio file where the sample begins. No statistically significant diagonal line is found in the scatter plot in Fig. 10B, which indicates that the sound file does not correspond to the exogenous sample.
There are many ways to find a diagonal line on a scatter plot, all of which are within the scope of the present invention. The phrase "locating a diagonal line" is to be understood to refer to all procedures that are equivalent to locating a diagonal line without explicitly producing a diagonal line. A preferred procedure begins by subtracting m * milestone<sub>n</sub> from both sides of the above equation, to obtain (milestone *<sub>n</sub> - m * milestone<sub>n</sub>) = displacement.
Assuming that m is approximately equal to one, that is, assuming there is no time dilation, we arrive at (milestone *<sub>n</sub> - milestone<sub>n</sub>) = displacement.
The problem of finding the diagonal then boils down to finding multiple pairs of landmarks for a given sound_identifier that cluster near the same offset value. This can easily be accomplished by subtracting one milestone from the other and collecting a histogram of the resulting offset values. The histogram can be prepared by ranking the resulting shift values, using a quick ranking algorithm, or by creating counter entries with counters and inserting them into a B-tree. The winning shift ark in the histogram contains the highest number of points. This ark is here called the peak of the histogram. Since the offset must be positive if the exogenous sound signal is fully contained within the correct library sound file, milestone pairs that result in a negative offset can be excluded. Similarly, offsets beyond the end of the file can also be excluded. The number of points in the winning shift ark of the histogram is noted for each supported sound_identifier. This number becomes the score for each sound recording. The sound recording on the list of candidates with the highest score is chosen as the winner. The winning sound_identifier is revealed to a user as described below to indicate the success of the identification. In order to avoid false identification, a minimum threshold score can be used to monitor the success of the identification process. If no library sounds have a score that exceeds the threshold, then there is no recognition, and the user is informed.
If the exogenous sound signal contains multiple sounds, then each individual sound can be recognized. In this case, the multiple winners are located on the lineup scan. It is not necessary to know that the sound signal contains multiple winners, because the alignment scan will locate more than one sound_identifier with a score that is much higher than the remaining scores. The fingerprint determination procedure used preferably shows a good linear overlap, so that individual fingerprints can be extracted. For example, a spectrogram fingerprint determination procedure shows linear overlap.
If the sound sample has been subjected to time dilation, then the slope is not identically equal to one. The result of assuming a slope equal to unity in a temporally dilated sample (assuming that the footprints are invariant for time dilations) is that the calculated displacement values are not equal. One way to address this and to accommodate moderate time dilation is to increase the size of the displacement chests, that is, to consider a range of displacements as equal. In general, if the points do not lie on a straight line, then the calculated displacement values are significantly different, and a slight increase in the size of the displacement arches does not produce a significant number of false positive values.
Other line search strategies are possible. For example, a Radon or Hough transformation can be employed, described in "Hough Transform for Line Recognition" of T. Risse, in Computer Vision and Image Processing, 46, .327-345, 1989, which are well known in machine vision technologies and graphic research. In the Hough transform, each point on the scatter plot is projected onto a line in pair space (slope, displacement). The set of points in the scatter plot is thus projected onto the dual line space in the Hough transform. The peaks in the Hough transform correspond to the intersections of the parameter lines. The global peak of such a transform of a given scatter plot indicates the largest number of intersecting lines in the Hough transform and hence the largest number of collinear points. To allow for a 5% speed variation, for example, the construction of the Hough transform can be restricted to the region where the slope parameter varies between 0.95 and 1.05, thus saving some computational effort.
ES 2 266 254 T3
Hierarchical search
In addition to the threshold determination step that eliminates candidates with very small hash lists, additional efficiency improvements can be made. In one such improvement, the database index is segmented into at least two parts, according to the probability of occurrence, and only the sound files with the highest probability of matching the sample are initially searched. The division can take place at various stages of the process. For example, the master index list (Fig. 8C) can be segmented into two or more parts, such that steps 16 and 20 are carried out first on one of the segments. That is, the files corresponding to the matching footprints are extracted only from a fraction of the database index, and a hash list is generated from this fraction. If a winning sound file is not found, then the process is repeated on the rest of the database index. In another implementation, all files are extracted from the database index, but the scan of the diagonal line is carried out separately on the different segments.
Using this technique, the diagonal line scan, a very expensive part of procedural calculations, is first performed on a small subset of the sound files in the database index. Because the diagonal line scan has a time component that is roughly linear with respect to the number of sound files being scanned, performing such a hierarchical search is highly convenient. For example, suppose that the sound database index contains footprints representing 1,000,000 sound files, but that only about 1,000 files correspond to high-frequency sample queries, eg. For example, 95% of the queries are for 1000 files, while only 5% of the queries are for the remaining 999,000 files. Assuming a linear dependence of the calculation cost with respect to the number of files, the cost is proportional to 1000 95% of the time, and proportional to 999,000 only 5% of the time. The average cost is therefore proportional to around 50,900. A hierarchical search, therefore, yields savings of about a factor of 20 in computational load. Of course, the database index can be segmented into more than two levels of hierarchy, eg. eg, a group of new releases, a group of recently released songs, and a group of older and less popular songs.
As described above, the search is first carried out on a first subset of sound files, the high probability files, and then, only if the first search fails, is it carried out on a second subset containing the remaining files. Diagonal line scan failure occurs if the number of points in each offset ark does not reach a predetermined threshold value. Alternatively, the two searches can be carried out in parallel (simultaneously). If the correct sound file is located in a search for the first subset, then a signal is sent to terminate the search for the second subset. If the correct sound file is not located in the first search, then the second search continues until a winning file is located. These two different implementations involve trade-offs between effort and computation time. The first implementation is more efficient in terms of computation, but introduces a slight latency if the first search fails, while the second implementation wastes computational effort if the winning file is in the first subset, but minimizes latency if it is not.
The purpose of segmenting the list is to estimate the probability that a sound file is the target of a query and to limit the search to those files that have the highest probability of corresponding to the query sample. There are several possible ways of assigning probabilities and of classifying sounds in the database, all of which are within the scope of the present invention. Preferably, the probabilities are assigned based on how recent or how often you are identified as the winning sound file. The recent identification criterion is a useful measure, particularly for popular songs, because musical interests change quite rapidly over time as new songs are released. After the probability scores have been calculated, the files are assigned categories, and the list is self-classified according to the category. The ranked list is then segmented into two or more subsets for search. The smallest subset can contain a predetermined number of files. For example, if the categorization locates a file within the first 1000 files, say, then the file is placed in the smallest and fastest search. Alternatively, the cut points for the two subsets can be dynamically adjusted. For example, all files with a score that exceeds a specific threshold value can be placed within the first subset, and thus the number of files in each subset changes continuously.
A particular way of calculating the probability is to increase the score of a sound file by one each time it is identified as corresponding to the query sample. To account for the recent identification criteria, all scores are periodically lowered, so that the most recent queries have a greater effect on the categorization than the oldest queries. For example, all scores can scale down by a constant factor for each query, resulting in an exponential decline in the score if it is not updated. Depending on the number of files in the database, which can easily be one million, this procedure may require updating a large number of scores on each query, making it potentially undesirable. Alternatively, the scores can be adjusted downward at relatively infrequent intervals, such as once per day. The ordering resulting from a less frequent fit is effectively similar, but not exactly identical, to the ordering that results from the fit in each query. However, the computational load to update the categorizations is much less.
A slight variation on this setting of the recent identification criteria, which more accurately preserves the recent identifications score, is to add an exponentially growing a 'score update to the file
ES 2 266 254 T3 of winning sound per query, where t is the time elapsed since the last global update. All scores are then adjusted downward by dividing by a<sup>T</sup> in each global update, where T is the total time elapsed since the last global update. In this variation, a is the recent identification factor, which is greater than one.
In addition to the categorization described above, some a priori knowledge can be introduced to help make the cast of sound recordings more fruitful. For example, new releases are likely to have higher numbers of queries than older songs. In this way, news can be automatically placed in the first subset, which contains songs with a higher probability of matching queries. This can be done independently of the self-categorization algorithm described above. If the self-categorization feature is also used, novelties can be assigned initial categorizations that place them somewhere within the first subset. New releases can be spread at the very top of the list, at the bottom end of the high probability song list, or somewhere in the middle. For search purposes, the starting location does not matter, because the categorization converges over time to reflect the true level of interest.
In an alternative embodiment, the search is performed in the order of recent identification categorizations and is terminated when a score of the sound_identifier exceeds a predetermined threshold value. This is equivalent to the preceding method, in which each segment contains only one sound_identifier. Experiments show that the score for a winning sound is much higher than the scores for all other sound files and therefore a suitable threshold can be chosen with minimal experimentation. One way to implement this embodiment is to rank all sound_identifiers in the database index according to how recent the identification was, with an arbitrary tiebreaker in the case of identical scores. Because each identification recentness categorization is unique, there is a one-to-one correspondence between the identification recentness score and the sound_identifier. The categorization can then be used in place of the sound_identifier when sorting by sound_identifier to form the list of candidate sound_identifiers and their associated scatter lists (FIG. 9C). Categorization numbers can be linked to the index when the triplet index list (footprint, milestone, sound_identifier) is generated, and before the index list is sorted into the master index list. The categorization then takes the place of the sound_identifier. Alternatively, a find and replace function can be used to replace the sound_identifier with the categorization. As categorizations are updated, new categorizations are mapped over old ones, assuming mapping integrity is maintained.
Alternatively, the categorizations can be linked later in the process. Once the scatter lists are created, a categorization can be associated with each sound_identifier. The sets are then classified by categorization. In this implementation, it is only necessary to modify the pointers to the hash lists; no need to repeat grouping into scatter lists. The advantage of subsequent bindings is that the entire database index does not have to be rebuilt each time the categorizations are updated.
Note that the popularity category, itself, can be of interest as an object of economic value. That is, the category reflects the desirability of consumers to obtain an identification of an unknown sound sample. In many cases, the consultation is prompted by a desire to purchase a recording of the song. Indeed, if the demographic information about the user is known, then alternative categorization methods can be implemented for each desired demographic group. A user's demographic can be obtained from profile information requested when the user registers for the recognition service. It can also be determined dynamically using standard collaborative filtering techniques.
In a real-time system, sound is supplied to the recognition system incrementally over time, allowing chained recognition. In this case, it is possible to process the incoming data in segments and incrementally update the set of indices in the sample. After each update period, the newly augmented set of indexes is used to extract candidate library sound recordings, using the preceding search and scan steps. The database index is examined for fingerprints that match the newly obtained sample fingerprints, and new triplets are generated (milestone<sub>k</sub>, milestone * j, sound_identifierj). New pairs are added to the scatter lists, and the histograms are enlarged. The advantage of this approach is that if enough data has been collected to unambiguously identify the sound recording, e.g. For example, if the number of points in a shift ark of one of the sound files exceeds a high threshold, or exceeds the next highest sound file score, then data acquisition can be terminated and the result announced.
Once the correct sound has been identified, the result is reported to the user or the system by any appropriate procedure. For example, the result can be reported by means of a computer printer, an email, a web search results page, an SMS (short message service) text message to a mobile phone, a voice message generated by computer to a landline phone, or sending the result to a headquarters or Internet account that the user can access later. The reported results can include identifying information for the sound, such as the name and artist of a song; the composer, name, and recording attributes (eg, performers, conductor, setting) of a classical piece; the company and product of an advertisement; or any other suitable identifiers. In addition, biographical information, information about concerts in the neighborhood, and other information of interest to fans can be provided; they can
ES 2 266 254 T3 hyperlinks to such data are provided. The reported results can also include the absolute score of the sound file or its score compared to the next highest scoring file.
A useful consequence of the recognition procedure is that it does not confuse two different versions of the same sound. For example, different performances of the same piece of classical music are not considered to be the same, even if a human cannot detect a difference between the two. This is because it is highly unlikely that the milestone / footprint pairs and their time evolution coincide exactly for two different interpretations. In a current embodiment, the milestone / footprint pairs must be within 10 ms of each other for a linear correspondence to be identified. As a result of this, the automatic recognition performed by the present invention makes it possible for the proper performance / soundtrack and artist / label to be credited in all cases.
Implementation example
A preferred implementation of the invention, continuous sliding window audio recognition, is described below. A microphone or other sound source is continuously sampled in a data store in order to obtain a record of the previous N seconds of sound. The content of the audio data store is periodically analyzed to verify the identity of the audio content. The sound data store may be of a fixed size or it may grow in size as the sound is sampled, referred to herein as sequentially increasing segments of the audio sample. A report is produced to indicate the presence of identified sound recordings. For example, a log file can be compiled, or a viewer can be displayed on a device that indicates information about the music, such as the title, artist, album cover image, lyrics, or information. shopping. To avoid redundancy, a report can occur only when the identity of the recognized sound changes; for example, after a program change on a phonola. Such a device can be used to create a list of music played from any sound source (radio, internet broadcast radio, hidden microphone, phone call, etc.). In addition to the identity of the music, information such as the recognition time can be recorded. If location information (eg from GPS) is available, such information can also be recorded.
To achieve identification, each data store can be re-identified each time. Alternatively, parameters can be extracted from the sound, for example in fingerprints or other intermediate forms of extracted features, and stored in a second data store. New tracks can be added to the beginning of the second store, discarding the old tracks at the end of the store. The advantage of such a circular magazine method is that it is not necessary to redundantly perform the same analysis of the old overlapping segments of the sound samples, thus saving computational effort. The identification process is carried out periodically on the contents of the circular fingerprint store. In the case of a small portable device, fingerprint analysis can be performed on the device, and the results transmitted to a recognition server using a relatively low-bandwidth data channel, since the fingerprint stream does not have much data upload. The circular fingerprint store can be maintained on the handheld device and transferred each time to the recognition server, or it can be maintained on the recognition server, in which case a continuous recognition session is cached on the server.
In such a circular magazine recognition system, new sound recordings can be recognized as soon as sufficient information is available for their recognition. Sufficient information can occupy less than the length of the warehouse. For example, if a distinguishable song can be recognized individually after one second of playback, and the system has a recognition periodicity of one second, then the song can be recognized immediately, although the data store can be between 15 and 30 seconds. Vice versa, if a less distinguishable song requires more seconds of sampling to be recognized, the system must wait for a longer period before declaring the identity of the song. In this sliding window recognition method, sounds are recognized as soon as they can be identified.
It is important to note that while the present invention has been described in the context of a fully functional recognition system and procedure, those skilled in the art will appreciate that the mechanism of the present invention is capable of being delivered in the form of a medium, computer readable, with instructions in various forms, and that the present invention applies equally, it does not matter the particular type of signal-carrying medium used to actually carry out the distribution. Examples of such computer-accessible devices include computer memory (RAM or ROM), floppy disks, and CD-ROM disks, as well as transmission-type media such as digital and analog communication links.
Contents5
13 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13
48 members in 14 offices
Priority claims10
| Document | Office | Kind | Date |
|---|---|---|---|
| 20000222023P | United States of America | – | |
| 22202300 | United States of America | P | |
| 22202300 | United States of America | P | |
| 20010839476 | United States of America | – | |
| 83947601 | United States of America | A | |
| 83947601 | United States of America | A | |
| 222023P01969535 | – | – | – |
| 839476 | – | – | – |
| US20000222023P | – | – | – |
| US20010839476 | – | – | – |
Members48
| Document | Office | Kind | |
|---|---|---|---|
| WO0211123A2 | World Intellectual Property Organization (WIPO) | A2 | |
| AU8976601A | Australia | A | |
| WO0227600A2 | World Intellectual Property Organization (WIPO) | A2 | |
| AU9298201A | Australia | A | |
| WO0211123A3 | World Intellectual Property Organization (WIPO) | A3 | |
| US2002083060A1 | United States of America | A1 | |
| EP1307833A2 | European Patent Office (EPO) | A2 | |
| BR0112901A | Brazil | A | |
| KR20030059085A | Republic of Korea | A | |
| HK1051248A | Hong Kong, China | A | |
| HK1051248A1 | Hong Kong, China | A1 | |
| WO0227600A3 | World Intellectual Property Organization (WIPO) | A3 | |
| JP2004505328A | Japan | A | |
| US2004199387A1 | United States of America | A1 | |
| CN1592906A | China | A | |
| US6990453B2 | United States of America | B2 | |
| EP1307833B1 | European Patent Office (EPO) | B1 | |
| US2006122839A1 | United States of America | A1 | |
| AT329319T | Austria | T | |
| ATE329319T1 | Austria | T1 | |
| DE60120417D1 | Germany | D1 | |
| DK1307833T3 | Denmark | T3 | |
| PT1307833E | Portugal | E | |
| DE60120417T2 | Germany | T2 | |
| ES2266254T3This record | Spain | T3 | |
| CN1996307A | China | A | |
| KR100776495B1 | Republic of Korea | B1 | |
| US7346512B2 | United States of America | B2 | |
| US2008208891A1 | United States of America | A1 | |
| CN100538701C | China | C | |
| CN1592906B | China | B | |
| US7853664B1 | United States of America | B1 | |
| US7865368B2 | United States of America | B2 | |
| US2011071838A1 | United States of America | A1 | |
| US8190435B2 | United States of America | B2 | |
| JP4945877B2 | Japan | B2 | |
| US2012221131A1 | United States of America | A1 | |
| US8386258B2 | United States of America | B2 | |
| US2013138442A1 | United States of America | A1 | |
| US8700407B2 | United States of America | B2 | |
| US8725829B2 | United States of America | B2 | |
| US2014316787A1 | United States of America | A1 | |
| BRPI0112901B1 | Brazil | B1 | |
| US9401154B2 | United States of America | B2 | |
| US2016328473A1 | United States of America | A1 | |
| US9899030B2 | United States of America | B2 | |
| US2018374491A1 | United States of America | A1 | |
| US10497378B2 | United States of America | B2 |
Numbers
- Publication
- 2266254
- Publication, DOCDB
- 2266254
- Publication, EPODOC
- ES2266254T
- Application
- 1969535
- Application, DOCDB
- 01969535
- Application, EPODOC
- ES20010969535T
Titles2
- Spanish
- PROCEDIMIENTO DE BUSQUEDA DE UNA BASE DE DATOS DE AUDIO.
- English
- PROCEDURE FOR SEARCHING AN AUDIO DATA BASE.
Classification
- CPC, 8
- G06F16/634
- G11B20/10
- G10L19/018
- G10L17/26
- G11B27/28
- G10L15/26
- G06F16/683
- G10L25/54
- IPC, 9
- G06F17 30
- G10L15 10
- G01H1 00
- G06K9 00
- G10L11 00
- G10L15 00
- G10L15 02
- G10L15 20
- G10L15 26