Systems and methods for generating audio thumbnails

The present invention generally relates to computer systems and, more specifically, for the purpose of facilitating browsing of audio files, generating mnemonic audio thumbnails or clips, or for other purposes. Relates to systems and methods of using prints to determine common or repetitive elements in an audio file.

One of the current features supported by many modern software systems is the ability to store and play audio files. Many of these systems allow users to store and manage a diverse collection of audio files. But over time, many users will inevitably become dissatisfied with the large amount of data that occupies a larger storage space. Also, as the collection grows, it becomes more difficult and time consuming to retrieve and reproduce the desired audio information. Many systems provide software that helps users manage this ever-growing amount of audio information. For example, these systems include MP3, Ogg Vorbis (OGG), and Windows Media. May include audio managers that support popular audio file formats, including Audio® (WMA), MPC, MP + files, and more. This allows users to catalog their entire collection of audio files, quickly find their favorite songs, use the album cover as thumbnails, browse albums, report and other useful features. Or create.

In addition to organizing audio files, these systems manage files by, for example, editing tags, renaming, editing lyrics, burning CDs, and looking up artist information1 A set of tools is provided. Users can work with audio files stored on hard disks, CD-ROMs, network drives, ZIP drives or any other removable media. It includes tools that allow the user to play multiple playlists and view the image associated with each title. Additional features include auto-generated database statistics, personal ratings, genre / mood / year sorting, and custom database queries.

Audio fingerprinting (AFP) has emerged in recent years as a powerful way to identify audio in streams or files. Several companies are currently offering music services based on audio fingerprinting. These services require extracting one or more fingerprints from the audio to be identified and collating those fingerprints against a large database of fingerprints calculated so far.

<p> However, managing large audio collections is difficult (unlike images with thumbnails), as it is not currently possible to parse audio files quickly. Users generally have to rely on labeling, but this help is also limited. Labeling is often inaccurate, and even if labeling is accurate, the user may not remember a given song until he or she listens to it. If the user can't remember what a song is, he usually has to play it and stop playing when he knows the song. In addition, some scenarios require a "hands-off" approach to song selection, for example, wanting to browse an audio collection to select a song while driving. There is.</p><p> Previous efforts have attempted to summarize songs in order to solve some of the problems of browsing songs. However, these previous efforts have focused on calculating features from a single frame of audio. These frames are typically 16-30 ms long. Previous efforts have calculated similarities between such frames. This similarity is inevitably coarse due to the inadequate information available for the similarity metric.</p>

<p> The following is a simplified summary of the invention to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the present invention. It is not intended to identify the important / significant elements of the invention or to define the scope of the invention. The sole purpose is to present some concepts of the invention in a simplified form as a prelude to a more detailed description below.</p><p> The present invention relates to systems and methods for generating audio thumbnails. The invention of interest deals with the problem of presenting a mnemonic "audio thumbnail" to the user to facilitate browsing or to summarize audio for other purposes.</p><p> Thumbnails are short (usually less than about 15 seconds), but are extracted from the part of the song or audio file that the user is most likely to remember. Therefore, the present invention works by, in part, determining the portion of the audio that is nearly repeated within the audio clip. For example, if a song has a chorus and the reproduction of that chorus is similar enough, the system can identify the chorus and build a highly effective segment of audio that is reminiscent of the original. To find similar iterations, the present invention uses a (partially) based fingerprinting component whose output is based on multiple frames of transformed audio data.</p><p> In addition to the fingerprinting component, the system can also use a measure of spectral flatness and a measure of spectral energy to determine different parts of the audio that are repeated. The system can also use these measures to identify mnemonic sections of audio, even if the audio does not contain repeating sections. When the system identifies a mnemonic section, it extracts a segment (in some embodiments, 15 seconds) around that position in the file. This extracted section (or, equivalently, a pointer into the audio file that determines where the identified segment is in the audio file) is used as the "audio thumbnail".</p><p> Certain exemplary embodiments of the invention are described herein in connection with the following description and accompanying drawings in order to achieve the above and related objects. These aspects suggest various methods in which the present invention can be practiced, the invention comprising all of them. Other advantages and novel features of the invention will become apparent from the following detailed description of the invention when considered in conjunction with the drawings.</p>

The present invention relates to a system and methodology that facilitates the automatic generation of mnemonic audio parts or segments called audio thumbnails. The present invention replaces traditional music summarization techniques by calculating fingerprints (partially) based on information contained in multiple frames. Therefore, fingerprints have much more information and the similarities between them are much less noisy. A system for summarizing audio information is provided. The system includes an analyzer that converts audio to frames and a fingerprinting component that converts frames to fingerprints, each fingerprint being partially based on multiple frames. The similarity detector calculates the similarity between fingerprints, and the heuristic module generates thumbnails of audio files based in part on the similarity between fingerprints. The system has an analysis component that determines common features in an audio file to generate thumbnails of the audio file, and a mnemonic detector that extracts the fingerprint portion of the audio file based in part on the common features. including. The generated thumbnails can then be used to facilitate browsing or exploration of audio files so that parts or segments of such files do not have to be listened to for long periods of time.

As used herein, the terms "component" and "objects", "Generator" "system" hardware, hardware and Seo any any combination of software, software, software in execution, intended to refer to a computer-related entity And. For example, components can be, but are not limited to, processes, processors, objects, executables, threads of execution, programs, and / or computers running on the processor. .. As an example, an application running on a server and its server can both be components. One or more components may be in processes and / or threads of execution, one component may be localized on one computer, and / or distributed among two or more computers. Good. In addition, these components can be executed from various computer-readable media that store various data structures. A component is a signal that has one or more data packets (for example, from one component that interacts with another component across a network such as the Internet with local systems, distributed systems, and / or other systems). It can communicate via local and / or remote processes that follow the data).

First, referring to FIG. 1, the audio thumbnail generator system 100 is shown according to one aspect of the present invention. System 100 also includes a database 110 of audio files, which is also processed by the summarizer 120, which summarizer is also referred to as the audio thumbnail generator. The generator 120 includes an analyzer 130 that processes the audio file to determine the components, segments, or parts of the audio file 110 that are suitable as audio thumbnails 140. The audio thumbnail 140 is generally a short clip or segment of audio that is likely to remind the user of the contents of the audio file 110 (eg, a chorus of lyrics "Goodbye Yellow Brick Road" when played as a thumbnail. Reminds the user of a song by Elton John of the same name).

The mnemonic detector 150 works with the analyzer 130 to determine which part of the audio file 110 should be used as the audio thumbnail 140. As illustrated, the analyzer 130 has a fingerprint component for analyzing a stream of audio information, an energy component that further processes the audio file to determine a suitable segment of audio for the thumbnail 140, and / or a flatness component. including. Note that the components within the analyzer 130 can be used in various combinations and degrees to determine the thumbnail 140.

In general, system 100 uses audio fingerprinting to identify repeating sections of audio. One idea is that similar sections of a song produce similar fingerprints. Therefore, by using fingerprints rather than using the original audio, the present invention provides fingerprints that are very similar with slightly different variants, and therefore the fingerprints use the original audio. It offers the advantage of being more robust than. In addition, fingerprints have the advantage of integrating information extracted from windows for much longer times than previously used in the art, and are therefore robust. Fingerprints also have the advantage of being very low dimensional representations of the original song, and therefore the processing of these entities is more efficient in terms of memory and CPU usage. Further details of the fingerprint processing according to the present invention will be provided in the discussions relating to FIGS. 3-5.

Various techniques are possible to identify the audio section that can be used as the audio thumbnail 140 (see Figures 2-3). The following description provides details of an implemented system, but it should be understood that it is just one example of such a system. For example, this implemented system uses 3 seconds (or other time) for fingerprinting and 186 milliseconds (or other time) for steps to and from the start of subsequent fingerprints. doing. While another system uses a 6 second fingerprint, the fingerprinting system can generate fingerprints of any length, and 3 seconds is a good balance for chorus detection.

In System 100, there are three basic objects involved in the calculation of audio thumbnails, which are contained in Analyzer 130. That is, the fingerprint and associated normalization (A), the fingerprint-calculated energy scale in audio (B), and the fingerprint-calculated audio spectrum flatness scale (C). One aspect is to use these features to allow the system to select a voice chorus in preference to the repetitive phrases of a pure musical instrument performance. This is because the chorus of voice seems to have a higher recall effect (mnemonic) than the repetition of pure instrumental performance. In addition, features (B) and (C) can be used when a suitable chorus cannot be found due to the features of (A). The current system calculates fingerprints about 3 seconds in length by concatenating 16 time windows of 372 ms, each overlapping in half (186 ms). All three quantities (A, B, C) can be calculated using these 372 ms frames (or other time frames). Before calculating these features with Analyzer 130, it should be noted that the silence at the beginning and end of the clip can be removed using a simple energy-based threshold.

Here, with reference to FIG. 2, the feature calculation 200 and related modes of processing are shown according to the present invention. In this aspect, the quantities A, B, and C described above with respect to the analyzer component will be described in more detail. At 210, the fingerprints are calculated, for example, as described for FIGS. 4-6. In one example, the fingerprint is calculated (or other sampling rate) in 186 ms steps for each 3-second window in the audio clip. For each fingerprint, calculate the normalization so that the average Euclidean distance from that fingerprint to the other fingerprints for that audio clip is 1. Again, this is different from the usual way normalization is calculated for systems that use fingerprinting for search tasks. That is, only the audio in that clip is used here. This is because the fingerprint will usually be compared to other fingerprints extracted from the same clip.

At 220, it processes the spectral energy of the audio. Fingerprint calculations generally require the calculation of a set of spectral magnitudes per frame. The spectral magnitude is, for example, MCLT (modulated complex lapped). It can be calculated by the transform) operator. The spectral energy 220 and spectral flatness 230 described below use the magnitude of the average spectrum as a normalization factor (so that the features produced by 220 and 230 do not depend on the overall volume level of the audio). For each frame, the average normalized energy E is calculated by dividing the average energy per frequency component within the frame by the average of that amount across the frames in the clip. The average energy is averaged over all frames (16 in this example) that contribute to a given fingerprint. This amount can be calculated efficiently by using the moving average. Therefore, spectral energy 220 is a measure of spectral energy per fingerprint.

At 230, the amount of spectral flatness can be determined. For example, first consider the calculation of this amount for a given frame. In this case, a very small number (eg 10)^-10) Is added to the spectral magnitude of each frequency component to alleviate the numerical problem when taking logarithms. This calculated frame quantity is the lognormalized geometric mean of the spectral magnitude. It is calculated as the logarithmic geometric mean of the spectral magnitude minus the logarithmic arithmetic mean of the spectral magnitude. Note that the geometric mean is less than or equal to the arithmetic mean, so this limits the quantity to greater than or equal to 0. Therefore, if the spectral energy is spread uniformly over the entire spectrum, this amount will be much larger than if it were concentrated over a small number of frequency components.

For some types of audio, high values of this amount have been found to indicate a "full" sound (for example, in audio where vocals dominate the sound when singing, the song's While this amount is large). With respect to the spectral energy 220, this quantity 230 is calculated by averaging over all the frames that contribute to that fingerprint per fingerprint. Therefore, 230 is a measure of spectral flatness per fingerprint.

FIG. 3 is a flow chart showing audio thumbnail processing according to one aspect of the present invention. For simplicity of explanation, the methodology is presented and described as a series of acts, some of which are presented and described herein in accordance with the present invention, in a different order than those shown and described. It should be understood and recognized that the present invention is not limited by the order of actions, as it may occur at the same time as / or other actions. Those skilled in the art will appreciate and recognize that a methodology can also be represented as a series of interrelated states or events, such as a phase diagram. Moreover, all of the illustrated actions may not be required to implement the method according to the invention.

Go to 310 and think about cluster calculations. A "cluster" is a number of fingerprints that are clustered in time and can be defined as representing a contiguous section of a piece of music that repeats somewhere in an audio clip. To explain the cluster calculation, we introduce the concept of "cluster set S" and "multiplicity M" of cluster set S. Each set S can contain an integer greater than or equal to 0 that indexes the fingerprint (where the starting point corresponds to the beginning of the audio clip, the first fingerprint to be calculated has index 1 and the starting point is The second fingerprint, which corresponds to the beginning of the audio clip with a half frame added, has an index of 2 and so on).

By "adding a fingerprint to a set", this involves adding the index of that fingerprint to the set. The multiplicity M of a given set is the number of clusters contained in that set. For example, if a set contains the integers 1, 2, 3, 100, 101, 102, then this set contains two clusters (one corresponding to the fingerprint indexes 0, 1, 2 and the other one. One corresponds to the fingerprint indexes 100, 101, 102), so the multiplicity can be 2. Each fingerprint has a Boolean flag called "AccountedFor" associated with it, the default value of which is "false".

In general, all sets are empty. Then the first fingerprint F₁Set 1 (S) (that is, the fingerprint corresponding to the first 3 seconds of the audio clip)₁). Then all the remaining fingerprints are inspected. Each remaining fingerprint F_iAbout F₁And F_iThis is also S only if the following conditions are met₁Put in. That is, (1) F₁And F_iThe normalized Euclidean distance between is below the initial threshold T (where the normalized Euclidean distance is F the Euclidean distance).₁(Divided by the normalization factor of) and (2) F₁At the beginning of the corresponding points in the audio and F_iThe time required to and from the corresponding point in the audio at the beginning of is to exceed the second fixed threshold Z (eg, Z = 6 seconds). Condition (2) is usually required because adjacent fingerprints may have a normalized Euclidean distance less than T and sound the same, but should be determined for pieces of audio that are temporally separated. Is. F the rest of the fingerprints in this way₁Compared to, it executes a second loop and recursively adds to the set all fingerprints that meet the above conditions compared to any fingerprint already in the set.

When adding a fingerprint to a set, set its "AccountedFor" flag to true and remove it from the set of fingerprints you are considering adding to any set. The above phase is then repeated to create a new set for the first fingerprint that has not yet been set with the "AccountedFor" flag and add the fingerprint as described above. Continue this until all fingerprints are members of a (and only one) set (and all "AccountedFor" flags are true). Therefore, the assignment of fingerprints to a set forms a set partition of all fingerprints. For each set, calculate the corresponding multiplicity. This is achieved by arranging the fingerprints in sequence and then searching for a gap corresponding to at least Z seconds in the array. The number of clusters is then the number of gaps with one fingerprint at each end plus one.

Do all of the above for the initial value of threshold T. At this point, check the maximum multiplicity for at least 3 values (ie, there is at least one cluster set containing at least 3 clusters). If this is not true, T is incremented by a small value and the cluster set is recalculated. Continue this process until a set with at least 3 clusters is found or T reaches the upper limit. In this way, in finding at least three clusters, the conditions required to become a member of the cluster are gradually relaxed. This process results in a cluster set where all sets contain only two clusters (in this case, the sets will be used in the process described below), or to a cluster set that contains only one cluster. (In this case, the audio thumbnail will be calculated using the energy scale described below).

At 320, the optimum cluster set is determined. At this point, assuming that the clustering 310 above results in one or more cluster sets containing at least two clusters, the remaining task is to select the appropriate cluster set (here, "appropriate". "Nail" means "likely to contain a fingerprint index for chorus or repetitive instrumental performances") and use that fingerprint to select the appropriate 15 seconds from the audio clip. That (where these 15 seconds are the audio thumbnails).

To calculate a suitable cluster set, combine the scales (B) and (C) above (Figure 1) with a third scale that measures how uniformly the clusters are spread over a song (this scale is used). Called (D)). For example, if three clusters were found, but all were in the first 20 seconds of the song, then these clusters are unlikely to be choruses, whereas three clusters were found and they were If they are evenly distributed throughout the song, it is likely that these clusters are choruses. The quantity (D) is measured for each set found. For a given set, (D) is measured as follows. Consider the case of a given cluster set in which N clusters are found. First, normalize the entire audio file so that the time required is equal to 1. t the time position of the i-th cluster_iAnd define as follows.

<maths num="3"><img file="JP4878437B2_D0001.tif" /></maths>

At this time, the quantity (D) is

<maths num="2"><img file="JP4878437B2_D0002.tif" /></maths>

Is calculated as.

The quantity (D) has the following properties. First,

<maths num="3"><img file="JP4878437B2_D0003.tif" /></maths>

And t_i t_i-1i, so the difference t_i-t_i-1Can be interpreted as a probability, therefore (D) is proportional to the Renyi entropy for the corresponding distribution (with an additive offset). Therefore, in this sense, choosing clustering with a larger (D) value corresponds to choosing clustering that spreads more evenly (for any discrete distribution, when all probabilities have the same value). Because it is known to have maximum entropy). t_i-t_i-1It should be emphasized that maximizing (D) was only interpreted as a probability to show that it was equivalent to choosing the most uniformly spread cluster. This stochastic interpretation has not been used elsewhere. Second, the offset and scaling factors are chosen so that (D) has a maximum value of 1 and a minimum value of 0 for any N. This allows the quality of the spread of a set of clusters to be compared between cluster sets, even if these sets contain different numbers of clusters. In addition, this allows prior knowledge to be easily applied as to which multiplicity is prioritized (eg, a multiplicity of 3, 4, or 5 is greater than any other multiplicity). Choose by weighting, because choruses are likely to occur this number of times).

The geometric mean feature, (C) above, predicts the section of audio that contains audio in some cases, but in other cases (eg, if the singing does not acoustically protrude from the rest of the song). May not be. However, in the latter case, the amount (C) tends to remain unchanged throughout most of the audio clips, whereas in the former case it changes significantly throughout the audio clips. To clarify this, for the middle third of the set of validation songs (for fingerprints where the energy scale (B) was above the threshold to avoid bias due to silence in the song), fingers. The standard deviation of the log geometric mean for each print was calculated. The middle third is used to reduce the bias caused by the beginning and end of songs with low (C) values due to musically quiet preludes and fades.

At this point, s_maxAnd s_minLet be the maximum and minimum standard deviations of the frame-by-frame feature (C) found in the validation set. Linear map (a, b), as_min+ b = 0 and as_maxDefined by + b = 1 (hence parameters a and b are fixed by the validation set). Suppose you want to calculate an audio thumbnail for a new song. Calculate the standard deviation s of the value (C) for each frame and apply the linear map y = as + b. If y&gt; 1, replace y with 1, and if y &lt;0, replace y with 0. Next, for that song, the value (C) of that song is linearly mapped to the interval [0, y]. So each set is attributed to the average spectral quality, which is the average of the scaled values (C) for the fingerprints in that set. As a result of this scaling, when combined with the Cluster Quality Scale (D) (which takes a maximum of 1), for example, when choosing a thumbnail of a song where the feature (C) does not change so much that it can be recognized across the song, the feature (C). ) Will be reduced.

Therefore, each set has two numbers associated with it. One measures the quality of the spread of the cluster and varies from 0 to 1, the other measures the quality of the spread of the spectrum and changes from 0 to y, where y is at most 1 and of those Y is large for songs with a large dispersion of spectral spread. For the "best" or optimal set, choose the one with the largest sum of the squares of these two numbers. For songs where the amount of spectral spread (geometric mean) has a small variance (compared to the validation set), y is small and therefore its value has a smaller weight when combined with the quality of cluster spread. It will be. For songs where the amount of spectral spread (geometric mean) has a large variance (compared to the validation set), y is approximately 1, and therefore its value is approximately this when combined with the quality of cluster spread. Will have the same weight.

You can go to 330 and consider alternative cluster options. In this aspect, clustering can be performed by finding the longest section of audio in the clip that is repeated somewhere in the clip. When the cluster set is calculated as above, fill a vector with a size equal to the number of fingerprints with 0, then replace 0 with 1 for each fingerprint that occurs in a set with a multiplicity of at least 2, and finally, This can be achieved efficiently by performing run-length coding on this vector and finding the longest string of 1. Then, make these corresponding fingerprints correspond to the best cluster.

At 340, the optimal fingerprint can be determined from the cluster set determined above. Therefore, the task remains to find the best cluster in the set, and to find the best fingerprint in the cluster, and to extract the audio around that fingerprint as audio thumbnails. At this point, various heuristics can be used. One example is a cluster whose energy (scale (B) above) is below the threshold for any fingerprint in a 6 second time window around that cluster, eliminating clusters that are too close to the beginning or end of the song. Eliminate and finally select the fingerprint (from the fingerprints that survived the above test) that maximizes the measure of mean spectral flatness (C) at 15 seconds around that fingerprint.

If the above process fails (for example, if no cluster set with a multiplicity greater than 1 is found), the best fingerprint is as follows using the above two energy scales (B) and (C): Calculate to. To avoid the quiet part of the song, consider only the fingerprints where the energy scale (B) is in the upper third of the value of (B) for the whole song (the quiet part of the song is still the spectrum). The flatness scale (C) is large, because white noise has the largest possible spectral flatness scale, and the very quiet parts of a song can be close to white noise). For fingerprints that survive this test, the fingerprint with the largest peripheral 15-second mean spectral flatness scale (C) is selected as the best fingerprint.

At 350, extract audio from the fingerprint selected at 340. Use a fixed period audio section around the fingerprint position as a thumbnail. It turned out to be advantageous to put the fingerprint near the beginning of this section. This is because the system may identify a passage in a repetitive instrumental performance just before the actual chorus. This "audio thumbnail" (eg, a 15-second clip) can then be saved to disk, for example as a separate audio file, or a file with a time offset that defines the position of the thumbnail within the entire audio file. It can be saved to (for example, a playlist .ASX file). If desired, fading can be applied automatically at the beginning and end of the audio using standard techniques to provide a more pleasing effect.

FIG. 4 shows a distortion discriminant (DDA) according to one aspect of the present invention. analysis) Shows system 400. Audio processing techniques, such as the technique of extracting features from speech, often use a frame period of about 20 milliseconds. However, it is desirable to generate fingerprints from the stream 2-3 times per second to reduce computational overhead for fingerprinting applications. At a 20ms input frame, the step size used in the last DDA layer must be sampled below the initial 100Hz sampling rate, which can cause aliasing and is a source of additional distortion. It will work. The system 400 shown in FIG. 4 avoids this problem. There is generally no aliasing because there is no intermediate layer with a low sampling rate. This requirement, and the requirement that fingerprints be generated on a time scale of about half a second, severely constrains the possible duration of the first layer frame. Also, the time-wide first layer gives the DDA greater flexibility in choosing important directions in frequency space.

FIG. 5 shows generalized eigenvalues 500 according to one aspect of the invention. The selection of the output dimensions of 64 in the first layer of the system 400 described above is guided by the measured generalized eigen spectrum of the training data shown in FIG. Most of the useful information from the first layer is captured during the first 100 projections. The spectrum on the second layer has a less steep fall. However, to speed up database lookup, we only considered the top 64 projections for the second tier. Database lookup speed could be doubled by simply sampling the output every 372 ms instead of every 186 ms.

The stream audio fingerprinting system described above first converts the stereo audio signal to monaural and then downsamples it to 11025 Hz. Divide the signal into half-overlapping fixed-length 372 ms frames. MCLT (Fourier Transform with Overlapping Windows) is then applied to each frame. A logarithmic spectrum is generated by taking the log modulus of each MCLT coefficient. This stream audio fingerprinting system performs two frame-by-frame preprocessing steps that suppress specific, easily identifiable distortions.

The first pretreatment step removes the distortion caused by frequency equalization and volume adjustment. This "de-equalization thresholding" step takes the DCT of the logarithmic spectrum, multiplies each DCT coefficient by a weight that linearly ramps the 1st to 6th and higher components 0 of the first component, and then Apply a lowpass filter to the logarithmic spectrum by performing an inverse DCT. As a result, a smooth approximation A to the logarithmic spectrum is obtained. Then A is uniformly lowered by 6 dB and clipped at -70 dB. Then, the output vector of the first preprocessing step is the difference if the difference in the component units from the logarithmic spectrum is positive, otherwise it is 0.

The second processing step removes distortion in the signal that the human listener cannot hear. This step exponentiates the logarithmic spectrum from the first step and then algorithmically generates a frequency-dependent perceptual threshold. Then, the final preprocessed signal is the difference if the difference between the log spectrum expressed in dB and the log perception threshold is positive, otherwise it is 0. The final preprocessed data consists of 2048 real coefficients per frame (hence 2048 bands).

Referring to FIG. 6, an exemplary environment 710 for implementing various aspects of the invention includes a computer 712. The computer 712 includes a processing unit 714, system memory 716, and system bus 718. System bus 718 binds system components to processing unit 714, including but not limited to system memory 716. The processing unit 714 can be any of a variety of available processors. Dual microprocessors and other multiprocessor architectures can also be used as processing unit 714.

The system bus 718 is a 16-bit bus, ISA (Industrial Standard Architecture), MSA (Micro-Channel Architecture), EISA (Extended ISA), IDE (Intelligent Drive Electronics), VLB (VESA Local Bus), PCI (Peripheral Component Interconnect). , USB (Universal Serial Bus), AGP (Advanced Graphics Port), PCMCIA (Personal Computer Memory Card International Association) Bus, and SCSI (Small Computer Systems) Any of several types of bus structures, including but not limited to Interface), memory buses or memory controllers with any of the various available bus architectures, peripheral or external buses, and / or local buses. It can be done.

System memory 716 includes volatile memory 720 and non-volatile memory 722. The BIOS (basic input / output system), which contains basic routines for transferring information between elements in computer 712, such as at startup, is stored in non-volatile memory 722. As an example, but not limited to, the non-volatile memory 722 can include ROM (read only memory), PROM (programmable ROM), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), or flash memory. The volatile memory 720 includes RAM (random access memory) that acts as an external cache memory. As an example, not limited to, RAM is SRAM (synchronous RAM), DRAM (dynamic RAM), SDRAM (synchronous DRAM), DDR SDRAM (double data rate). It is available in many forms such as SDRAM), ESDRAM (enhanced SDRAM), SLDRAM (SyncLink DRAM), and DRRAM (direct Rambus RAM).

Computer 712 also includes removable / non-removable, volatile / non-volatile computer storage media. FIG. 6 shows, for example, a disk storage 724. Disk storage 724 includes, but is not limited to, devices such as magnetic disk drives, floppy (registered trademark) disk drives, tape drives, Jaz drives, Zip drives, LS-100 drives, flash memory cards, or memory sticks. Further, the disk storage 724 includes a CD-ROM (compact disk ROM device), a CD-R drive (CD recordable drive), a CD-RW drive (CD rewritable drive), and a DVD-ROM (digital versatile disk ROM). Drive) is included, but is not limited to, and storage media may be included separately or in combination with other storage media. Removable or non-removable interfaces, such as interface 726, are typically used to facilitate the connection of disk storage device 724 to system bus 718.

It should be understood that Figure 6 describes software that acts as an intermediary between the user and the basic computer resources described in the appropriate operating environment 710. Such software includes operating system 728. Operating system 728 can be stored on disk storage 724, but acts to control and allocate resources for computer system 712. System application 730 utilizes the management of resources by operating system 728 through program module 732 and program data 734 stored in system memory 716 or on disk storage 724. It should be understood that the present invention can be implemented in various operating systems or combinations of operating systems.

The user inputs a command or information into the computer 712 via the input device 736. The input device 736 includes a pointing device such as a mouse, a trackball, a stylus, a touchpad, a keyboard, a microphone, a joystick, a game pad, a satellite antenna, a scanner, a TV tuner card, a digital camera, a digital video camera, and a Web camera. Not limited to these. These and other input devices connect to the processing unit 714 through system bus 718 via interface port 738. Interface port 738 includes, for example, a serial port, a parallel port, a game port, and a USB (Universal Serial Bus). The output device 740 uses some of the ports of the same type as the input device 736. Therefore, for example, a USB port can be used to provide input to computer 712 and output information from computer 712 to output device 740. The output adapter 742 illustrates that, among other output devices 740, there are some output devices 740 that require special adapters such as monitors, speakers, and printers. The output adapter 742 includes, by example, but not limited to, video and sound cards that provide a means of connecting the output device 740 to the system bus 718. Note that other devices and / or systems of devices provide both input and output capabilities, such as the remote computer 744.

Computer 712 can operate in a networked environment with a logical connection to one or more remote computers, such as remote computer 744. The remote computer 744 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device or other common network node, etc., and usually has many of the elements mentioned with respect to the computer 712 or Including everything. For brevity, only the memory storage device 746 is shown with the remote computer 744. The remote computer 744 is logically connected to computer 712 through network interface 748 and then physically connected via communication connection 750. Network interface 748 includes communication networks such as LAN (local-area network) and WAN (wide-area network). LAN technology is FDDI (Fiber Distributed Data Interface), CDDI (Copper) Distributed Data Interface), Ethernet (registered trademark) / IEEE802.3, Token Ring / IEEE802.5, etc. are included. WAN technologies include, but are not limited to, point-to-point links, circuit-switched networks such as ISDN (Integrated Services Digital Network) and variants thereof, packet-switched networks, and DSL (Digital Subscriber Line).

Communication connection 750 refers to the hardware / software used to connect network interface 748 to bus 718. The communication connection 750 is shown inside computer 712 for clarity of explanation, but it can also be external to computer 712. The hardware / software required to connect to network interface 748 is for illustrative purposes only, such as regular phone grade modems, modems including cable and DSL modems, ISDN adapters, and Ethernet® cards. Includes internal and external technologies.

FIG. 7 is a schematic block diagram of a computing environment 800 as an example that the present invention can interact with. System 800 includes one or more clients 810. Client 810 can be hardware and / or software (eg, threads, processes, computing devices). System 800 also includes one or more servers 830. Server 830 can also be hardware and / or software (eg, threads, processes, computing devices). The server 830 can accommodate, for example, a thread that performs the conversion by using the present invention. One possible communication between client 810 and server 830 can be in the form of data packets adapted to be transmitted between two or more computer processes. System 800 includes a communication framework 850 that can be used to facilitate communication between client 810 and server 830. Client 810 is operably connected to one or more client datastores 860 that can be used to store information local to client 810. Similarly, the server 830 is operably connected to one or more server datastores 840 that can be used to store information local to the server 830.

What has been described above includes an example of the present invention. It will of course not be possible to describe any possible combination of components or methodologies for the purposes of describing the invention, but it will be appreciated by those skilled in the art that many further combinations and rearrangements of the invention are possible. Accordingly, the present invention includes all such alternative, modified and modified forms that fall within the spirit and scope of the appended claims. Further, as long as the term "include" is used in the detailed description or claims, such term is used when "comprising" is used as a transitional term in the claims. As interpreted, it shall be as inclusive as the term "prepare".

<figref num="1">It is a schematic block diagram of the audio thumbnail generator system by one aspect of this invention.</figref><figref num="2">It is a figure which illustrates the feature calculation by this invention.</figref><figref num="3">It is a flow chart which illustrates the audio thumbnail processing by this invention.</figref><figref num="4">It is a figure which illustrates the strain discriminant analysis by one aspect of this invention.</figref><figref num="5">It is a figure which illustrates the generalized eigenvalue by one aspect of this invention.</figref><figref num="6">It is a schematic block diagram which illustrates the appropriate operating environment by one aspect of this invention.</figref><figref num="7">It is a schematic block diagram of the computing environment as an example which the present invention can exchange.</figref>

Code description

100 Audio Thumbnail Generator System 110 audio files 120 Summarizer / Thumbnail Generator 130 analyzer 140 audio thumbnails 150 mnemonic detector 200 Feature calculation 210 fingerprint 220 Spectral energy 230 spectral flatness 710 Operating environment 712 computer 714 processing unit 716 system memory 718 bus 720 volatile 722 Non-volatile 724 disk storage 726 interface 728 operating system 730 application 732 module 734 data 736 Input device 738 interface port 740 output device 742 output adapter 744 remote computer 746 memory storage 748 Network interface 750 communication connection 800 Computing environment 810 client 830 server 840 server data store 850 communication framework 860 client data store