Processing of audio signals during high frequency reconstruction

Technical field [0001] This application relates to the reconstruction / regeneration of High Frequency Reconstruction / Regeneration (HFR) audio signals. The application relates in particular to a method and system for performing HFR audio signals having a large variation in energy level in the low frequency range, which is used to reconstruct the high frequencies of the audio signal.

Background of the invention [0002] HFR technologies, such as Spectral Band Replication (SBR) technology, can significantly improve the coding efficiency of traditional perceptual audio codecs. In combination with the MPEG-4 Advanced Audio Coding (AAC), HFR creates a very efficient audio codec that is already used in the XM Satellite Radio and DRM transmission systems. Digital Radio Mondiale) and is standardized in 3GPP, DVD Forum and others. The combination of AAC and SBR is referred to as aacPlus. It is part of the MPEG-4 standard, in which it is referred to as the High Efficiency AAC Profile (HE-AAC). Typically, HFR technology can be combined with any perceptual audio codec in a backward and forward compatible manner, thus enabling the modernization of already created broadcast systems, such as MPEG Layer-2 used in the Eureka DAB system. HFR methods can also be combined with speech codecs to enable broadband speech at ultra-low transmission speeds.

[0003] The basic essence of HFR is the observation that there is usually a strong correlation between high frequency range characteristics of a signal and low frequency range characteristics of the same signal. Thus, a good approximation of the representation of the original input high frequency range of the signal can be achieved by transposing the signal from the low frequency range into the high frequency range.

[0004] This transposition concept was established in WO 98/57436 as a method of reproducing the high frequency band from the low frequency band of an audio signal. Significant savings in transmission speed can be obtained by using this concept in audio coding and / or speech coding. In the following, reference is made to audio coding, but it should be noted that the described methods and systems can also be used for speech coding and for unified speech and audio coding. unified speech and audio coding, USAC).

[0005] WO 02/41301 A1 discloses an audio decoder with linear-prediction based spectral whitening after high frequency reconstruction and after envelope adjustment.

[0006] High frequency reconstruction can be performed in the time domain or in the frequency domain using a selected filter bank or transform. The process usually involves several stages in which the two main operations are, first, to generate a high frequency excitation signal, and then to shape the high frequency excitation signal to approximate the spectral envelope of the original high frequency spectrum. The step of producing a high frequency excitation signal may be, for example, based on single-band modulation (SSB), where the sine wave frequency is mapped to a sine wave frequency ω + Δω, where Δω is a constant frequency shift. In other words, a high frequency signal can be generated from a low frequency signal by a "copy-up" operation of low frequency subbands to high frequency subbands. Another approach to generating a high frequency excitation signal may involve transposing the harmonics of the low frequency subbands. The harmonic transposition of the order T is generally designed to map a sinusoid with a frequency ω of low frequency signal to a sinusoid with frequency Τω, with T> 1, a high frequency signal.

[0007] HFR technology can be used as part of source coding systems in which various control information for directing the HFR process is transmitted from the encoder to the decoder, together with the representation of the narrowband / low frequency signal. For systems in which an additional control signal cannot be transmitted, the process may be used on the decoder side with the corresponding control data estimated from the available information on the decoder side.

[0008] Said regulation of the envelope of the high-frequency excitation signal aims at obtaining a spectral shape that resembles the spectral shape of the original high band. To obtain this, the spectral shape of the high frequency signal must be modified. In other words, the high bandwidth adjustment is a function of the existing spectral envelope and the desired target spectral envelope.

[0009] For systems that operate in the frequency domain, for example HFR systems implemented in a pseudo-QMF filter bank, prior art methods are suboptimal in this sense, since the creation of the high band signal by combining multiple inputs in the source frequency range introduces an artificial envelope spectral bandwidth to be adjusted in terms of the envelope. In other words, the high band or high frequency signal generated from the low frequency signal during the HFR process usually has an artificial spectral envelope (generally containing spectral discontinuities). This introduces difficulties for the spectral envelope regulator, because the regulator must not only be able to use the desired spectral envelope with a specified time and frequency resolution, but also the ability to undo the artificially introduced spectral characteristics by the signal generator

-3HFR. This introduces difficult design constraints on the envelope regulator. As a result, these difficulties generally lead to perceived high frequency energy losses and audible spectral discontinuities in the high band signal, particularly for speech signals. In other words, conventional HFR signal generators generally introduce discontinuities and level changes into the high band signal for signals that have high level variation in the low band range, for example, wheezing. When the envelope regulator is then directed to this high band signal, the envelope regulator cannot reasonably and consistently separate the newly introduced discontinuity from any natural spectral characteristics of the low band signal.

[0010] This document provides a solution to the above problem that results in improved perceived audio quality. In particular, this document describes a solution to the problem of generating a high band signal from a low band signal, wherein the envelope of the high band signal is effectively adjusted so that it resembles the original high band spectral envelope without introducing undesirable artifacts.

Summary of the Invention [0011] According to the invention, there is provided a system as set forth in claim 1, an encoder as set forth in claim 11, methods as set out in claims 12 and 14 and a computer program as set forth in claim 15. Preferred embodiments are set out in the dependent claims.

[0012] This document proposes an additional correction step as part of the high frequency reconstruction signal generation. As a result of the additional correction step, the audio quality of the high frequency component or high band signal is improved. An additional correction step can be applied to all source coding systems that use high frequency reconstruction techniques as well as to any one-way post-processing or system that aims to reproduce high frequencies of the audio signal.

[0013] According to the subject of the invention, a system configured to generate a plurality of high frequency subband signals comprising a high frequency range is described. The system may be configured to generate a plurality of high frequency subband signals from a plurality of low frequency subband signals. Many low frequency subband signals may be subband signals of a low or narrowband audio signal that can be determined using an analysis or transform filter bank. In particular, many low frequency subband signals can be determined from a time domain low band signal using QMF (mirror quadrature filter) analysis or FFT (fast Fourier transform) filter banks. Many of the generated high frequency subband signals may correspond to the approximation of the high frequency subband signals of the original audio signal from which multiple low frequency subband signals were obtained. Especially, many low frequency subband signals and many (re) generated

The high frequency subband signals may correspond to the subbands of the QMF filter bank and / or the FFT transform.

[0014] The system may include means for receiving a plurality of low frequency subband signals. As such, the system can be placed behind an analysis or transform filter bank that generates multiple low frequency subband signals from the low band signal. The low band signal may be an audio signal that has been decoded at the core decoder from the received bit stream. The bit stream may be stored on a storage medium, for example a compact disc or DVD, or the bit stream may be received in a decoder via a transmission medium, for example an optical medium or a radio transmission medium.

[0015] The system may comprise means for receiving a set of target energies, which may also be referred to as scaling factor energies. Each target energy can cover a different target range, which can also be referred to as a scaling factor band, in the high frequency range. In general, a set of target bands that corresponds to a set of target energies covers the entire high frequency band. The target energy of the target energy set is usually an indicator of the desired energy of one or more high frequency subband signals falling within the respective target range. In particular, the target energy may correspond to the average desired energy of one or more high frequency subband signals falling within the respective target range. The target energy of the target range is generally obtained from the signal energy of the high bandwidth primary audio signal within the target range. In other words, the set of target energies generally describes the spectral envelope of the high band portion of the original audio signal.

[0016] The system may include means for generating a plurality of high frequency subband signals from the plurality of low frequency subband signals. For this, the means for generating multiple high frequency subband signals may be configured to perform copy-up transposition of multiple low frequency subband signals and / or to perform harmonic transposition of multiple low frequency subband signals.

[0017] In addition, the means for generating the plurality of high frequency subband signals may include a plurality of spectral gain factors during the process of generating the plurality of high frequency subband signals. Multiple spectral gain factors can be appropriately associated with multiple low frequency subband signals. In other words, each low frequency subband signal among a plurality of low frequency subband signals may have a corresponding spectral gain factor among the multiple spectral gain factors. The spectral gain factor among the many spectral gain factors can be applied to the corresponding low frequency subband signal.

[0018] A plurality of spectral gain factors may be appropriately associated with the energy of a plurality of low frequency subband signals, respectively. In particular, each spectral gain factor may be associated with the energy of the respective low frequency subband signal. In an embodiment of the invention, the spectral gain factor is determined based on the energy of the respective low frequency subband signal. For this purpose, a frequency-dependent curve can be determined based on multiple energy values ​​of many low-frequency subband signals. In this case, the method for determining multiple gain factors may rely on a frequency-dependent curve that is determined from (e.g. logarithmic) energy representation of multiple low frequency subband signals.

[0019] In other words, many spectral gain factors can be determined from a frequency-dependent curve matched to the energy of many low frequency subband signals. In particular, the frequency-dependent curve can be a polynomial of a predetermined order / degree. Alternatively or additionally, the frequency-dependent curve may comprise different curve segments, wherein the different curve segments are matched to the energy of many low frequency subband signals in different frequency ranges. Different curve segments can be different polynomials of a predetermined order. In an embodiment of the invention, the different curve segments are zero order polynomials, so that the curve segments represent the average energy values ​​of many low frequency subband signals in the respective frequency range. In another embodiment, the frequency dependent curve is matched to the energy of many low frequency subband signals by performing a moving average filtering operation at different frequency ranges.

[0020] In an embodiment of the invention, the gain coefficient among the multiple gain coefficients is determined from the difference in the energy average of many low frequency subband signals and the corresponding frequency-dependent curve value. The corresponding frequency-dependent curve value may be the curve value at a frequency in the frequency range of the low frequency subband signal to which the gain factor corresponds.

[0021] Generally, the energy of many low frequency subband signals is determined on a certain time grid, for example, frame by frame, i.e., the energy of the low frequency subband signal in the time interval determined by the time grid corresponds to the average energy of low frequency subband signal samples in the interval time, for example, in the frame. As such, many different spectral gain factors can be determined on a selected time grid, for example many different spectral gain factors can be determined for each audio signal frame. In an embodiment of the invention, a plurality of spectral gain factors can be determined sample by sample, for example by determining the energy of multiple low frequency subbands using a free window on samples of each low frequency subband signal. It should be noted that the system may include means for determining multiple spectral gain factors from multiple low frequency subband signals. These measures can be

Configured to perform the abovementioned methods of determining multiple spectral gain factors.

[0022] Means for generating multiple high frequency subband signals may be configured to amplify multiple low frequency subband signals using a plurality of spectral gain factors respectively. Although it refers to "strengthening" or "extension" in the following, the "strengthening" operation may be replaced by other operations such as "multiplication" operation, "rescaling" operation, or "adjustment" operation. The gain can be made by multiplying the sample of the low frequency subband signal by its corresponding spectral gain factor. In particular, the means for generating multiple high frequency subband signals may be configured to determine a high frequency subband signal sample at a given time from low frequency subband signal samples at a given time and at least one preceding time moment. Furthermore, the samples of the low frequency subband signal can be enhanced by a corresponding spectral gain factor among a plurality of spectral gain factors. In an embodiment of the invention, the means for generating multiple high frequency subband signals are configured to generate multiple high frequency subband signals from multiple low frequency subband signals in accordance with the "copy-up" algorithm described in MPEG-4 SBR. Many of the low frequency subband signals used in this "copy-up" algorithm could be amplified using multiple spectral gain factors, with the "gain" operation being performed as described above.

[0023] The system may include means for regulating the energy of multiple high frequency subband signals using a set of target energies. This operation is generally referred to as spectral envelope adjustment. The adjustment of the spectral envelope may be made by adjusting the energy of many high frequency subband signals, such that the average energy of many high frequency subband signals being in the target range corresponds to the respective target energy. This can be achieved by determining the envelope adjustment value from the energy values ​​of many high frequency subband signals being in the target range and the respective target energy. In particular, the envelope adjustment value may be determined from the ratio of target energy and energy values ​​of many high frequency subband signals being in the target range. This envelope adjustment can be used to regulate the energy of many high frequency subband signals.

[0024] In an embodiment of the invention, the energy regulating means comprises means for limiting the energy regulation of the high frequency subband signals being in the limiter compartment. Generally, the limiter compartment covers more than one target compartment. Limiting means are typically used to avoid undesirable noise gain in specific high frequency subband signals. For example, the limiting means may be configured to determine the average value of the envelope adjustment from among the envelope adjustment values ​​corresponding to the intervals

-7 target covered by or within the limiter range. In addition, the limiting means may be configured to limit the energy regulation of the high frequency subband signals being in the limiter range to a value that is proportional to the average value of the envelope adjustment.

[0025] Alternatively or in addition, the energy regulation means of the plurality of high-frequency subband signals may include means for ensuring that the regulated high-frequency subband signals being in the specified target range have the same energy. The latter are often referred to as "interpolation" measures. In other words, the "interpolation" means ensure that the energy of each of the high frequency subband signals being within a specific target range corresponds to the target energy. The "interpolation" means can be implemented by separately adjusting each high frequency subband signal in a specific target range such that the energy of the regulated high frequency subband signal corresponds to the target energy associated with the specific target range. This can be achieved by specifying a different envelope adjustment value for each high frequency subband signal in the specified target range. The different value of the envelope adjustment may be determined based on the energy of the specified high frequency subband signal and the target energy corresponding to the specified target range. In an embodiment of the invention, the envelope adjustment value for a specific high frequency subband signal is determined based on the ratio of target energy and energy of the specific high frequency subband signal.

[0026] The system may further comprise means for receiving control data. The control data may indicate whether to use multiple spectral gain factors to generate multiple high frequency subband signals. In other words, the control data may indicate whether additional gain adjustment of the low frequency subband signals should be made or not. Alternatively or in addition, control data may indicate a method for determining multiple spectral gain factors. By way of example, control data may indicate a predetermined order of polynomials that will be used to determine a frequency-dependent curve matched to the energy of many low frequency subband signals. Control data is generally received from a suitable encoder that analyzes the original audio signal and informs the respective decoder or HFR system how to decode the bit stream.

[0027] According to another object, an audio decoder configured to decode a bit stream comprising a low frequency audio signal and comprising a set of target energies describing the spectral envelope of the high frequency audio signal is described. In other words, an audio decoder configured to decode a bit stream representing a low frequency audio signal and representing a set of target energies describing the spectral envelope of the high frequency audio signal is described. The audio decoder may include a core decoder and / or a transform unit configured to determine the plurality of low frequency subband signals associated with the low frequency audio signal from the bit stream. Alternatively or

In addition, the audio decoder may include a high frequency generating unit according to the system described in this document, wherein the system may be configured to determine a plurality of high frequency subband signals from a plurality of low frequency subband signals and a set of target energy. Alternatively or in addition, the decoder may include a merge and / or inverse transform unit configured to generate an audio signal from multiple low frequency subband signals and multiple high frequency subband signals. The merge and inverse transform unit may include a synthesis filter bank or transforms, for example a reverse QMF filter bank or an inverse FFT filter bank.

[0028] According to another object, an encoder configured to generate control data from an audio signal is described. The audio encoder may comprise means for analyzing the spectral shape of the audio signal and determining the degree of discontinuity of the spectral envelope introduced during regeneration of the high frequency component of the audio signal from the low frequency component of the audio signal. As such, the encoder may contain specific elements of the respective decoder. In particular, the encoder may include an HFR system as described in this document. This would allow the encoder to determine the degree of discontinuities in the spectral envelope that may have been introduced into the high frequency component of the audio signal on the decoder side. Alternatively or additionally, the encoder may include means for generating control data to control regeneration of the high frequency component based on the degree of discontinuity. In particular, control data may correspond to control data received by the respective decoder or HFR system. Control data may indicate whether to use multiple spectral gain factors during the HFR process and / or which predetermined order of the polynomial to use to determine multiple spectral gain factors. To determine this information, the ratio of selected parts of the low frequency range can be determined, i.e. frequency range covered by many low frequency subband signals. This ratio information can be determined by, for example, testing the lowest low band frequencies and the highest low band frequencies to evaluate the spectral variation of the low band signal, which will then be used in the decoder for high frequency reconstruction. A high ratio may indicate an increased degree of discontinuity. Control data can also be determined using signal type detectors. By way of example, the detection of speech signals may indicate an increased degree of discontinuity. On the other hand, detection of prominent sinusoids in the original audio signal may lead to control data indicating that many spectral gain factors should not be used during the HFR process.

[0029] According to another object, a method has been described for generating a plurality of high frequency subband signals covering a high frequency range from a plurality of low frequency subband signals. The method may include the steps of receiving a plurality of low frequency subband signals and / or receiving a set of target energies. Each target energy can cover a different target range in the high frequency range. In addition, each target energy may be an indicator of the desired energy of one or more high frequency subband signals being in

-9 target range. The method may include the step of generating a plurality of high frequency subband signals from the plurality of low frequency subband signals and from the plurality of spectral gain factors associated with the plurality of low frequency subband signals, respectively. Alternatively or additionally, the method may include the step of regulating the energy of multiple high frequency subband signals using a set of target energies. The energy regulating step may include the step of limiting the energy regulation of the high frequency subband signals being in the limiter compartment. Generally, the limiter compartment covers more than one target compartment.

[0030] According to another object, a method for decoding a bit stream representing or comprising a low frequency audio signal and a set of target energies describing a spectral envelope of the corresponding high frequency audio signal is described. Generally, low frequency and high frequency audio signals correspond to the low frequency and high frequency components of the same original audio signal. The method may include the step of determining a plurality of low frequency subband signals associated with the low frequency audio signal from the bit stream. Alternatively or additionally, the method may include the step of determining a plurality of high frequency subband signals from the plurality of low frequency subband signals and the set of target energy. This step is generally performed according to the HFR methods described in this document. Then, the method may include the step of generating an audio signal from the plurality of low frequency subband signals and the plurality of high frequency subband signals.

[0031] According to another object, a method for generating control data from an audio signal has been described. The method may include the step of analyzing the spectral shape of the audio signal to determine the degree of discontinuities introduced during regeneration of the high frequency component of the audio signal from the low frequency component of the audio signal. In addition, the method may include the step of generating control data for controlling regeneration of the high frequency component based on the degree of discontinuity.

[0032] According to another object, a computer program has been described. The computer program may be adapted to perform on the processor and to perform the method steps described in this document while operating on a computing device.

[0033] A storage medium has been described. The storage medium may include a computer program adapted to be executed on the processor and to perform the method steps described in this document, operating on a computing device. A computer product-program has been described. The computer program may contain executable instructions for performing the method steps described in this document when executed on a computer.

[0034] It should be appreciated that methods and systems including their embodiments as described in this patent application may be used alone or in combination with other methods and systems disclosed in this document. Also,

-10 all subjects of methods and systems described in this patent application may be combined at their discretion. In particular, the features in the claims may be combined as desired.

Brief Description of the Drawings [0035] The invention is explained below by examples with reference to the drawings, at

<td>which</td><td></td>

<td>Fig. 1a</td><td>Shows the absolute spectrum of an example high band signal before adjusting the spectral envelope;</td>

<td>Fig. 1b</td><td>Shows an example relationship between time frames of audio data and time boundaries of spectral envelopes;</td>

<td>Fig. 1c</td><td>Shows the absolute spectrum of an example high signal before spectral envelope adjustment and the corresponding scaling factor bands, limiter bands and HF (high frequency) fragments;</td>

<td>Fig. 2</td><td>Shows an embodiment of the HFR system in which the copy-up process is supplemented with an additional stage of profit regulation;</td>

<td>Fig. 3</td><td>Shows the approximation of the coarse spectral envelope of an exemplary low band signal;</td>

<td>Fig. 4</td><td>Shows an embodiment of an additional gain regulator operating on optional control data, QMF subband samples, and returning a gain curve;</td>

<td>Fig. 5</td><td>Shows a more accurate embodiment of the additional gain controller of Fig. 4;</td>

<td>Fig. 6</td><td>Shows an embodiment of the HFR system with a narrowband signal at the input and a wideband signal at the output;</td>

<td>Fig. 7</td><td>Shows an embodiment of the HFR system included in the SBR module of the audio decoder;</td>

<td>Fig. 8</td><td>Shows an embodiment of a high frequency reconstruction module of an exemplary audio decoder;</td>

<td>Fig. 9</td><td>Shows an embodiment of an example encoder;</td>

<td>Fig. 10a</td><td>Shows the spectrogram of an example voice segment that has been decoded using a conventional decoder;</td>

<td>Fig. 10b</td><td>Shows the spectrogram of the voice segment of Fig. 10a that has been decoded using a decoder using additional gain control processing, and</td>

<td>Fig. 10c</td><td>Shows the voice segment spectrogram of Fig. 10a for the original uncoded signal.</td>

Description of Preferred Embodiments of the Invention [0036] The embodiments described below are merely illustrative of the nature of the invention. Processing of audio signals during high frequency reconstruction. It is understood that modifications and variations of the arrangements and details described herein will be clear to others skilled in the art. It is therefore intended that the limitation be made only by the scope of the claims and not by the specific details described for the illustration and explanation of the embodiments.

[0037] As described above, audio decoders using HFR techniques generally include an HFR unit for generating the high frequency audio signal and the following spectral envelope adjustment unit for adjusting the spectral envelope of the high frequency audio signal. When adjusting the spectral envelope of an audio signal, this is usually done by implementing a filter bank, or by using time-domain filtering. The adjustment can either try to make an absolute correction of the spectral envelope, or it can be performed using filtering, which also improves the phase characteristics. In one way or another, the adjustment is generally a combination of two steps, removing the current spectral envelope and applying the target spectral envelope.

[0038] It should be noted that the methods and systems described in this document are not only intended to remove the spectral envelope of the audio signal. Methods and systems attempt to perform the appropriate correction of the spectral envelope of the low band signal as part of the high frequency regeneration step to avoid introducing a discontinuity of the spectral envelope of the high frequency spectrum generated by combining different low band segments, i.e. the low frequency signal shifted or transposed to different frequency ranges high bandwidth, this is a high frequency signal.

[0039] Fig. 1a shows a stylistically drawn spectrum 100, 110 of the output of the HFR unit, before entering the envelope controller. On the top panel, a copyup (two-fragment) method was used to generate a high band signal from a low band signal 101, for example the copy-up method used in MPEG-4 SBR (Spectral Band Replication), which has been described in "ISO / IEC

14496-3 Information Technology - Coding of audio-visual objects - Part 3: Audio. " The copy-up method translates parts of the lower frequencies 101 into the higher frequencies 105. On the bottom panel, the harmonic transposition method (with two fragments) is used to generate the 115 high band signal from the low band 111 signal, for example the harmonic transposition method from MPEG-D USAC , which is described in "MPEG-D USAC: ISO / IEC 23003-3- Unified Speech and Audio Coding".

[0040] In the next step of the envelope adjustment, the target spectral envelope is applied to the high frequency components 105, 115. As can be seen in the spectrum 105, 115 entering the envelope regulator, discontinuities (especially at the fragment boundaries) can be observed in the spectral shape of the signal 105, 115

-12 high, that is, the high band signal entering the envelope regulator. These discontinuities arose from the fact that many low frequency cartridges 101, 111 are used to generate the high band 105, 115. As can be seen, the spectral shape of the high band signal 105, 115 is associated with the spectral shape of the low band signal 101, 111. As a result, the specific spectral shapes of the low band signal 101, 111, for example, the gradient shape shown in Fig. 1a may lead to discontinuities in the general spectrum 100, 110.

[0041] In addition to spectrum 100, 110, Fig. 1a shows exemplary spectral envelope data frequency bands 130 representing the target spectral envelope. These frequency bands 130 are referred to as scaling factor bands or target bands. In general, the target energy value, i.e. the energy of the scaling factor, is defined for each target range, i.e. the band of scaling factors. In other words, the scaling factor bands define the effective frequency resolution of the target spectral envelope as there is only a single target energy value per target range. Using scaling factors or target energies specified for the scaling factor bands, another envelope regulator attempts to regulate the high band signal so that the high band signal energy in the scaling factor bands equals the energy of the received spectral envelope data, i.e. the target energy, for the respective scaling factor bands.

[0042] Fig. 1c shows a more detailed description using the exemplary audio signal. The graph shows the spectrum of the real 121 audio signal entering the envelope controller, as well as the corresponding original signal 120. In this particular example, the SBR range, i.e. the high frequency signal range starts at 6.4 kHz and consists of three different range replications low band frequencies. Frequency ranges of different replications are shown by "fragment 1", "fragment 2" and "fragment 3". It is seen on the spectrogram that fragmentation introduces discontinuities in the spectral envelope at approximately 6.4 kHz, 7.4 kHz and 10.8 kHz In this example, these frequencies correspond to the limits of the fragments.

[0043] Fig. 1c further shows the scaling factor bands 130 as well as the limiter bands 135 whose function will be described in more detail below. In the embodiment of the invention shown, an envelope regulator with MPEG-4 SBR was used. This envelope regulator operates using a QMF filter bank. The main aspects of the operation of such an envelope regulator are:

• calculating the average energy in the band of 130 scaling factors of the input signal to the envelope controller, ie the signal output from the HFR unit; in other words, the average energy of the regenerated highband signal is calculated in each scaling band / target range 130;

• determining the value of profit, also referred to as the value of the envelope adjustment, for each band of 130 scaling factors, where the value of the envelope adjustment is the square root of the target energy ratio (that is, the energy target

13 received from the encoder) and average energy of the regenerated high band signal 121 in the appropriate band of 130 scaling factors;

• applying the appropriate envelope adjustment value to the frequency band of the regenerated high band signal 121, the frequency band corresponding to the corresponding band of 130 scaling factors.

[0044] In addition, the envelope regulator may include additional steps and variations, in particular:

• limiter functionality that limits the maximum allowable envelope adjustment value to be applied to a specific frequency band, that is, to the limiter band 135. The maximum allowable envelope adjustment value is a function of the envelope adjustment values ​​specified for different bands of 130 scaling factors that fall within the limiter band 135. In particular, the maximum allowable envelope adjustment value is a function of the average envelope adjustment value determined for the various limiter bands 130 which fall within the limiter band 135. By way of example, the maximum allowable envelope adjustment value may be the average value of the corresponding envelope adjustment values ​​multiplied by a limiter factor (such as 1.5). The limiter functionality is generally used to limit the introduction of noise to the regenerated highband signal 121. This is particularly important for audio signals containing prominent sine waves, i.e. audio signals having a spectrum with discriminating peaks at specific frequencies. Without the limiter functionality, significant envelope adjustment values ​​would be determined for 130 scaling factors bands for which the original audio signal contains such distinguishable peaks. As a result, the spectrum of the full band of 130 scaling factors (and not just the distinguishable peak) would be regulated, thereby introducing noise.

• interpolation functionality that allows calculating the envelope adjustment value for each individual QMF subband in the scaling factor band, instead of calculating the single envelope adjustment value for the entire scaling factor band. Because the scaling factor bands generally contain more than one QMF subband, the envelope adjustment value can be calculated as the ratio of the energy of a specific QMF subband in the scaling factor band and the target energy received from the encoder, instead of calculating the average energy ratio of all QMF subbands in the scaling and target band energy received from the encoder. As such, a different envelope adjustment value may be specified for each QMF subband in the scaling factor band. Note that the received energy target for the band of scaling factors generally corresponds to the average energy of this frequency range in the original signal. The decoder's role is to apply the received average target energy to the appropriate frequency band of the regenerated high band signal. This can be done by applying the overall value of the envelope adjustment to

-14 QMF subbands in the band of scaling factors of the regenerated high band signal or by applying a single envelope adjustment value to each QMF subband. The second approach can be understood as if the received envelope information (that is, one target energy per scaling factor band) was "interpolated" by the QMF subbands in the scaling factor band to provide greater frequency resolution. Thus, this approach is referred to as "interpolation" in MPEG-4 SBR.

[0045] Returning to Fig. 1c, it can be seen that the envelope regulator would have to use large values ​​of envelope adjustment to match the spectrum 121 of the signal entering the envelope regulator with the spectrum 120 of the original signal. It can also be seen that due to discontinuities, significant variations in the envelope adjustment values ​​occur in the limiter bands 135. As a result of such large variations, envelope adjustment values ​​that correspond to the local minima of the regenerated spectrum 121 will be limited by the functionality of the envelope regulator limiter. As a result, discontinuities in the regenerated spectrum 121 will remain, even after the envelope adjustment operation has been performed. On the other hand, if no limiter functionality is used, unwanted noise may be introduced as described above.

[0046] Thus, the problem of regenerating the high band signal occurs for any signal that has large horizontal variations in the low band range. This problem is caused by discontinuities introduced during high frequency regeneration of the high band. When the envelope regulator is then directed at this regenerated signal, it cannot reasonably and consistently separate the newly introduced discontinuity from any natural spectral characteristics of the low band signal. The effects of this problem are twofold. First, the spectral shapes are introduced in a high band signal that the envelope regulator cannot compensate for. Thus, the output has an incorrect spectral shape. Secondly, the effect of instability is perceived due to the fact that this effect appears and disappears as a function of the spectral characteristics of the low band.

[0047] This document addresses the aforementioned problem by describing a method and system that provides a high HFR band signal at the envelope regulator input that does not exhibit spectral discontinuities. For this reason, it is proposed to remove or reduce the spectral envelope of the low band signal when performing high frequency regeneration. By doing this, you will avoid entering spectral discontinuities in the highband signal before adjusting the envelope. As a result, the envelope regulator will not have to deal with such spectral discontinuities. In particular, a conventional envelope regulator may be used, wherein the envelope regulator limiter functionality is used to avoid introducing noise into the regenerated high band signal. In other words, the described method and system can be used to regenerate a high HFR band signal having little or no spectral discontinuities and low noise.

[0048] It should be noted that the temporal resolution of the envelope regulator may be different from the temporal resolution of the proposed spectral envelope processing during high band signal generation. As indicated above, the processing of the spectral envelope during regeneration of the high band signal is intended to modify the spectral envelope of the low band signal to soften processing in the next envelope controller. This processing, i.e. modification of the spectral envelope of the low band signal, may be carried out, for example, once per audio frame, the envelope controller may adjust the spectral envelope over several time intervals, i.e. using several received spectral envelopes. This is described in Fig. 1b, wherein the spectral envelope data time grid 150 is shown on the top panel and the time grid 155 for processing the low band signal spectral envelope during regeneration of the high band signal is shown on the bottom panel. As can be seen in the example of Fig. 1b, the time boundaries of the spectral envelope data change over time, while the processing of the spectral envelope of the low band signal operates on a fixed time grid. It can also be seen that several envelope adjustment cycles (represented by time limits) can be performed during one processing cycle of the spectral envelope of the low band signal. In the example shown, the processing of the spectral envelope of the low band signal operates in a frame by frame mode, that is, many different spectral gain factors are determined for each signal frame. It should be noted that the low band signal processing can operate on any time grid and that the time grid of such processing need not coincide with the time grid of the spectral envelope data.

[0049] In Fig. 2, a 200 HFR system based on a filter bank is shown. The 200 HFR system operates using a pseudo-QMF filter bank, and the 200 system can be used to produce the high band and low band signals 100 shown in the top panel of Fig. 1a. However, an additional profit adjustment step has been added as part of the High Frequency Generation process, which in the example shown is a copy-up process. The low frequency input signal is analyzed by 32 subband QMF 201 to generate multiple low frequency subband signals. Some or all of the low frequency subband signals are fragmented to a higher frequency location according to the HF (high frequency) generation algorithm. In addition, many low frequency subbands enter directly into the synthesis filter bank 202. Said synthesis filter bank 202 is a 64 inverse QMF 202 subband. For the specific implementation shown in FIG. 2, the use of 32 subband QMF analysis filter bank 201 and the use of 64 subband QMF synthesis filter bank 202 will give twice the output sampling frequency of the output signal from the input sampling frequency input signal. Note, however, that the systems described in this document are not limited to systems with different input and output sample rates. Many different sampling frequency relationships can be imagined by those skilled in the art.

[0050] As described in Fig. 2, lower frequency subbands are mapped to higher frequency subbands. Profit adjustment step 204 is introduced as part of this process

-16 copy-up type. The generated high frequency signal, i.e., many high frequency subband signals generated enters the envelope controller 203 (possibly including the limiter and / or interpolation functionality) before being combined with multiple low frequency subband signals in the synthesis filter bank 202. By using the 200 HFR system, and in particular by applying the gain control step 204, the introduction of a spectral envelope discontinuity as shown in Figure 1 can be avoided. Therefore, gain control step 204 modifies the spectral envelope of the low band signal, i.e., the spectral envelope of many low frequency subband signals, such that the modified low band signal can be used to generate a high band signal, i.e. many high frequency subband signals that are not they show discontinuities, especially discontinuities at the borders of the fragments. Referring to Fig. 1c, an additional gain adjustment step 204 ensures that the spectral envelope 101, 111 of the low band signal is modified such that discontinuities in the generated high band signal 105, 115 are not present or are limited.

[0051] Modification of the spectral envelope of the low band signal may be achieved by applying a gain curve to the spectral envelope of the low band signal. Such a gain curve can be determined by the profit curve determination unit 400 shown in Fig. 4. The module 400 receives QMF data 402 as input corresponding to the frequency range of the low band signal used to reproduce the high band signal. In other words, many low frequency subband signals enter the gain curve determination unit 400. As already indicated, only a subset of the available QMF subbands of the low band signal can be used to generate the high band signal, i.e. only a subset of the available QMF subbands can enter the gain curve determination unit 400. In addition, module 400 may receive optional control data 404, e.g., control data sent from an appropriate encoder. Module 400 returns a gain curve 403 that is used during the high frequency regeneration process. In an embodiment, the gain curve 403 is applied to the QMF subbands of the low band signal, which are used to generate the high band signal. That is, a profit curve 403 may be used in the HFR process copy-up process.

[0052] Optional control data 404 may include information about the coarse resolution of the spectral envelope to be estimated in the module 400, and / or information about the validity of using the gain adjustment process. As such, control data 404 may control the amount of additional processing involved during the gain adjustment process. Control data 404 may also trigger bypassing of additional gain control processing if there are signals that are not suitable for rough estimation of the spectral envelope, for example signals containing single sine waves.

[0053] Fig. 5 shows a more detailed view of the module u 400 of Fig. 4. Low band signal data 402 QMF enters the envelope estimation unit 501 that estimates the spectral envelope, for example on a log scale of energy. The spectral envelope then enters module 502, which estimates the coarse spectral envelope from the spectral envelope

-17 high resolution (frequency) received from the envelope estimation 501 unit. In one embodiment, this is accomplished by fitting a low order polynomial to the spectral envelope data, i.e., a polynomial of the order in the range of, for example, 1, 2, 3 or 4. A coarse spectral envelope can also be determined by performing a mean operation of a moving high resolution spectral envelope along the frequency axis. Determining the coarse spectral envelope 301 of the low band signal is shown in Fig. 3. It can be seen that the absolute spectrum 302 of the low band signal, i.e. the energy of 302 QMF bands, is approximated by the coarse spectral envelope 301, i.e. the frequency-dependent curve spectral of many low frequency subband signals. In addition, it has been shown that only QMF subband signals are used to generate the high band signal, i.e. only part of the QMF subband signals is used in the HFR process.

[0054] The method used to determine the coarse spectral envelope from the high resolution spectral envelope, and in particular the order of the polynomial that matches the high resolution spectral envelope, can be controlled by optional control data. The order of the polynomial may be a function of the size of the low band signal frequency range 302 for which the rough spectral envelope 301 will be determined, and / or may be a function of other parameters suitable for the general rare spectral shape of the respective low band signal range 302. Polynomial matching calculates a polynomial that approximates data using the least squares method. The preferred embodiment using the Matlab code is described below:

function GainVec = calculateGainVec (LowEnv) %% function GainVec = calculateGainVec (LowEnv)% Input: low band envelope energy in dB% Output: gain vector to apply to the low band before HF% generation

% The function performs the low order polynomial matching of the low band spectral envelope,% as a representation of the overall low band spectral slope. The overall slope as per% above is then translated into a gain vector that can be used before the HF generation to remove the% overall slope (or sparse spectral shape).

%% This prevents the introduction of discontinuities by the HF generation into the spectral shape, which will be "confusing" for% of the next envelope adjustment and limiter process. "Confusion" occurs when the envelope regulator and% limiter must handle a large discontinuity, and thus a large profit. It is very difficult to fine-tune the% and correct operation of these modules when they are to accept both "natural" variations in the high band as well as "artificial" variations introduced by the HF generation process.

polyOrderWhite = 3; x_lowBand = 1: length (LowEnv); p = polyfit (x_lowBand, Lowenva, polyOrderWhite); lowBandEnvSlope = zeros (size (x_lowBand)); for k = polyOrderWhite: -1: 0 tmp = (x_lowBand.'k). * p (polyOrderWhite - k + 1);

lowBandEnvSlope = lowBandEnvSlope + tmp;

end

GainVec = 10. '((Mean (LowEnv) - lowBandEnvSlope) ./ 20);

[0055] In the above code, the input is the spectral envelope (LowEnv) of the low band signal obtained by averaging the QMF subband samples for each subband in the time interval corresponding to the current time frame of the data on which it operates

-18 next envelope controller. As indicated above, low band signal gain control processing can be performed on various other time grids. In the above example, the estimated absolute spectral envelope is presented in the logarithmic domain. A low order polynomial, in the example above, a 3 polynomial is matched to the data. With a given polynomial, a gain curve (GainVec) is calculated from the difference between the average energy of the low band signal and the curve (lowBandEnvSlope) obtained from the polynomial fitted to the data. In the above example, the operation of determining the profit curve is performed in the logarithmic domain.

[0056] The calculation of the profit curve is performed by the profit curve calculation unit 503. As indicated above, the gain curve may be determined from the average energy of the low band signal part used to regenerate the high band signal, and from the spectral envelope of the low band signal part used to regenerate the high band signal. In particular, the gain curve can be determined from the difference of the average energy and the rough spectral envelope, represented for example by a polynomial. That is, the calculated polynomial can be used to determine a gain curve that includes a separate gain value, also referred to as a spectral gain factor, for each respective QMF subband of the low band signal. This profit curve containing profit values ​​is then used in the HFR process.

[0057] As an example, the process of generating HFR according to MPEG-4 SBR is further described. The generated HF signal can be determined by the following formula (see MPEG-4 Part 3 (ISO / IEC 14496-3), sub-part 4, section 4.6.18.6.2): ​​(7¹ + Ufmj) = ^XFROM_IN (p, l + vol_HFAdj) + bw Array (g (ł)) · α₀(ρ) · (p, l -1 +) + [bwArray (g (ł))]² · A_x (/?) · X_iow (P, l <a name="caption2"></a>^ ^{+ t}HFAdj}>

where p is the subband index of the low band signal, i.e. p identifies one of many low frequency subband signals. The above HF generation pattern may be replaced by the following formula which performs combined profit control and HF generation:

^x High 7, and + t_HFAdj) = preGain (p) (X_and)"(Ρ, l + vol_HFAdJ)) + bwArray (g (ł)) · a₀ (p) · X_hunting (p, l -1 + t_HFAdJ ) + [bwArray (g (ł))]² · A_x (p) X_Bovine", (P, ł- 2 + vol_HFAdJ) where the profit curve is marked as preGairr (p).

[0058] Further details of the copy-up process, for example with respect to the relationship between the peaks, are defined in the above-mentioned MPEG-4, Part 3. In the above formula, XLow (p, 1) indicates the sample at the time and time of the subband signal low frequency having a p subband index. This sample in combination with previous samples is used to generate a XHigh (k, 1) signal sample of a high frequency subband having the k subband index.

[0059] It should be noted that the object of gain control can be used in any high frequency reconstruction system based on a filter bank. Fig. 6 shows the invention being part of a standalone HFR unit 601 that operates on signal 602

-19 narrowband or lowband and returns a 604 wideband or highband signal. Module 601 may receive additional control data 603 as input, wherein control data 603 may define, amongst other things, the amount of processing used for the described gain control, as well as, for example, information about the target spectral envelope of the high band signal. However, these parameters are only examples of optional control data 603. In an embodiment, the corresponding information may also be determined from narrowband signal 602 entering the module 601, or by other means. That is, control data 603 may be determined in module 601 based on information available in module 601. It should be noted that the stand alone HFR unit 601 may receive multiple low frequency subband signals and may return multiple high frequency subband signals, i.e., analysis / synthesis filter banks or transforms may be placed outside the HFR 601 unit.

[0060] As indicated above, it may be advantageous to signal activation of gain control processing in the bit stream from encoder to decoder. For some types of signals, for example a single sine wave, the gain control processing may not be suitable and thus it may be beneficial to allow the encoder / decoder system to disable additional processing to avoid introducing undesirable behavior for such signals in extreme cases. For this purpose, the encoder may be configured to analyze audio signals and to generate control data that enable and disable the gain control processing in the decoder.

[0061] In Fig. 7, the proposed gain control step is included in the high frequency reconstruction unit 703, which is part of the audio codec. One example of such a 703 HFR unit is the MPEG-4 Spectral Band Replication tool used as part of the High Performance AAC codec or MPEG-D USAC (Unified Speech and Audio Codec). In this embodiment, the 704 bit stream is received at the audio decoder 700. The 704 bit stream is demultiplexed in the demultiplexer 701. The corresponding SBR portion of the 708 bit stream is fed to the SBR module or 703 HFR unit, and the 707 bit stream of the corresponding core encoder, for example, AAC data or USAC core decoder data, is sent to the core encoder module 702. Additionally, the low band or narrowband band signal 706 is passed from the core decoder 702 to the HFR unit 703. The invention is part of the SBR process in the HFR unit 703, for example according to the system described in Fig. 2. The HFR unit 703 returns a wideband or highband signal 705 using the processing described in this document.

[0062] Fig. 8 describes in detail an embodiment of the high frequency reconstruction module 703. Fig. 8 shows that HF ​​(high frequency) signal generation can be determined from different HF generation modules at different times. The HF generation may be based on either a QMF-based copy-up relay 803 or the HF generation may be based on a FFT-based 804 relay. For both HF signal generation modules, the low band signal is processed 801, 802 as part of the generation

-20HF for determining the gain curve that is used in the 803 copy-up process or harmonic transposition process 804. Outputs from two relays selectively enter the envelope 805 controller. The decision on which relay to use is controlled by a 704 or 708 bit stream. It should be noted that due to the nature of the copy-up type of the QMF based relay, the shape of the spectral envelope of the low band signal is maintained more clearly than when a harmonic relay is used. This will usually result in more distinct spectral discontinuities in the high band signal when using copy-up relays. This is shown in the upper and lower panel of Fig. 1a. Thus, it may be sufficient to enable gain control for the QMF-based copy-up method performed in module 803. However, the use of gain control for harmonic transposition performed in module 804 may also be beneficial.

[0063] Fig. 9 describes the corresponding encoder module. Encoder 901 can be configured to analyze a specific input signal 903 and determine the amount of gain control processing that is suitable for a particular type of input signal 903. In particular, the encoder 901 can determine the degree of discontinuity in the high frequency subband signal that will be caused by the 703 HFR unit in the decoder. For this purpose, the encoder 901 may comprise an HFR 703 unit or at least the corresponding parts of the HFR 703 unit. Based on the analysis of the input signal 903, control data 905 can be generated for the appropriate decoder. Information 905, which relates to gain control to be performed in the decoder, is combined in multiplexer 902 with a stream of 906 bits of audio, thereby forming a full stream of 904 bits, which is transmitted from the respective decoder.

[0064] Fig. 10 shows the output spectra of a real signal. Fig. 10a shows the MPEG USAC decoder output decoding a 12 kbps mono bit stream. The real signal section is the voice part of the a capella recording. The abscissa corresponds to the time axis, while the ordinate corresponds to the frequency axis. Comparing the spectrogram of Fig. 10a to Fig. 10c, which shows the corresponding primary signal spectrogram, it is clear that there are holes (see reference symbols 1001, 1002) appearing in the spectrum for the slotted portions of the voice segment. Fig. 10b describes the spectrogram output of the MPEG USAC decoder containing the solution of the invention. It can be seen on the spectrogram that the holes in the spectrum have disappeared (see reference symbols 1003, 1004 corresponding to reference symbols 1001, 1002).

[0065] The complexity of the proposed profit adjustment algorithm has been calculated as weighted MOPS, where functions such as POW / DIV / TRIG have a weight of 25 operations and all other operations have a weight of one operation. Assuming these assumptions, the calculated complexity is approximately 0.1 WMOPS and RAM / ROM is insignificantly used. In other words, the proposed processing of profit regulation requires little processing and memory resources.

[0066] This document describes a method and system for generating a high band signal from a low band signal. The method and system are adapted to generate a high band signal with little or no discontinuity, thus

- improving the perceived performance of high frequency reconstruction methods and systems. The method and system can be easily integrated into existing audio coding / decoding systems. In particular, the method and system can be enabled without the need for modification of the envelope adjustment processing of existing audio coding / decoding systems. This particularly applies to limiter functionality and interpolation of envelope adjustment processing that can perform their tasks. As such, the described method and system can be used to regenerate high band signals having few or no spectral discontinuities and low noise. In addition, the use of control data has been described, wherein the control data may be used to adapt the parameters of the described method and system (and computational complexity) to the type of audio signal.

[0067] The methods and systems described in this document may be implemented as software, firmware and / or hardware. Certain components may, for example, be implemented as software running on a digital signal processor or microprocessor. Other components can, for example, be implemented as hardware and / or as specialized integrated circuits. The signals occurring in the described methods and systems may be stored on media such as random access memory or optical media. They can be transmitted through networks such as radio networks, satellite networks, wireless networks or wired networks, for example the Internet. Typical devices using the methods and systems described in this document are portable electronic devices or other consumer devices that are used to store and / or process audio signals. The methods and systems can also be used on computer systems, for example, web web servers that store and provide audio signals, e.g., music signals, for download.

Prepared and verified

Anna Stenzel

Patent Attorney