Apparatus and method for combining multiple parametrically coded audio sources

ES 2 396 072 T3

DESCRIPTION

Device for combining multiple parametrically encoded audio sources

Field of Invention

[0001] The present invention relates to multi-channel audio coding and, in particular, to a concept for combining parametrically encoded audio streams in a flexible and efficient manner.

BACKGROUND OF THE INVENTION AND PRIOR TECHNIQUE

[0002] Recent development in the area of ​​audio coding has given rise to various parametric audio coding techniques to co-encode a multi-channel (eg, 5.1 channel) audio signal into one (or more) sub-mix channels. plus a stream of lateral information. In general, the side information stream has parameters related to the properties of the original channels of the multichannel signal, either with respect to other original channels of the multichannel signal, or with respect to the downmix channel. The particular definition of the reference channel parameters to which these parameters refer depends on the specific implementation. Some of the techniques known in the art are "binaural cue coding", "spatial audio coding" and "parametric stereo".

[0003] For details of these particular implementations, reference is made herein to related publications. For example, binaural indication encoding is detailed in:

C. Faller and F. Baumgarte, "Efficient representation of spatial audio using perceptual parametrization", IRRR WASPAA, Mohonk, NY, October 2001; F. Baumgarte and C Faller, “Estimation of auditory spatial cues for binaural cue coding”, ICASSP, Orlando, FL, May 2002; C. Faller and F. Baumgarte, “Binaural cue coding: a novel and efficient representation of spatial audio”, ICASSP, Orlando, FL, May 2002; C. Faller and F. Baumgarte, “Binaural cue coding applied to audio compression with flexible rendering”, AES 113^th Convention, Los Angeles, Prepress 5686, October 2002; C. Faller and F. Baumgarte, “Binaural cue coding - Part II: Schemes and applications,” IEEE Trans. On Speech and Audio Proc., Vol., 11, no. 6, November 2003, and J. Herre, C. Faller et al., “Spatial Audio Coding: Next-generation efficient and compatible coding of multi-channel audio”, Audio Engineering Society Convention Paper, October 28 , 2004, San Francisco, CA. USA.

[0004] Although binaural indication coding uses multiple original channels, parametric stereo is a related technique for parametric coding of a two-channel stereo signal resulting in a mono transmitted signal and side parameter information, for example, as is reviewed in the following publications: J. Breebaart, S. van de Par, A. Kohlrausch, E. Schuijers, “High-Quality Parametric Spatial Audio Coding at Low Bit rates”, AES 116^th Convention, Berlin, Prepress 6072, May 2004; E. Schuijers, J. Breebaart, H. Purnhagen, J. Engdegard, “Low Complexity Parametric Stereo Coding”, AES 116^th Convention, Berlin, Prepress 6073, May 2004.

[0005] Other technologies are based on the multiplexing of arbitrary numbers of audio sources or objects in a single audio transmission channel. Multiplexing-based schemes, for example, are introduced as "flexible representation" in publications related to BCC (binaural indication coding), or, more recently, by a scheme called "junction source coding" (JSC). Related publications are, for example: C. Faller, “Parametric Joint Coding of Audio Sources”, Convention Paper 6752, 120^th AES Convention, Paris, May 2006. Similar to binaural and stereo parametric indication coding schemes, these techniques are intended to encode multiple original audio objects (channels) for transmission using fewer channels of submix. By further derivation of the object-based parameters for each input channel, which can be encoded to a very low data rate and are also transmitted to a receiver, these objects can be separated on the receiving side and represented (mixed) for a number of output devices, such as headphones, two-channel stereo speakers, or multi-channel speaker installations. This procedure allows for level adjustment and redistribution (panning) of different audio objects to different locations in the playback facility, ie, on the receiving side.

[0006] Basically, such techniques operate as a MkN transmitter, where M is the number of audio objects at the input, where k is the number of transmitted sub-mix channels, typically k is <2. N is the number of audio channels in the representation output, ie, for example, the number of speakers. That is, N = 2 for a

ES 2 396 072 T3 stereo representation or N = 6 for a 5.1 multichannel installation. In terms of compression efficiency, typical values ​​are, eg, 64 kbps or less for a perceptually encoded submix channel (consisting of k audio channels) and about 3 kbps for audio object-by-object parameters. transmitted.

The application scenarios for the above techniques are, for example, encoding the spatial audio scenes related to cinema-movie-productions to allow the spatial reproduction of the sound in a home theater system. Common examples are the widely known 5.1 and 7.1 surround sound tracks on a movie medium such as DVD and the like. Film productions have become more and more complex with respect to audio scenes, intended to provide a spatial auditory perception experience and therefore have to be mixed with great care. Different sound engineers may be commissioned to mix the different surround sources or sound effects, and consequently the transmission of parametrically encoded multi-channel stages between individual sound engineers to carry the audio streams of individual sound engineers is desirable. efficiently.

[0008] Another application scenario for such technology is teleconferencing with multiple talkers at either end of a point-to-point connection as described in US 2005 / 0062843. To save bandwidth, most teleconferencing facilities operate with monophonic transmission. For example, it can be achieved using stitched source coding or one of the other multichannel coding techniques for the transmission, redistribution and level alignment of the different speakers at the receiving end (each end) and thus the intelligibility and balance of The speakers are improved by spending a marginally increased transmission speed compared to a monophonic system. The advantage of increased intelligibility is particularly evident in the special case of assigning each individual participant in the conference to a single channel (and thus the speaker) of a multi-channel service facility at a receiving end. However, this is a special case. In general, the number of participants will not equal the number of speakers at the receiving end. However, using the existing service facility it is possible to represent the signal associated with each participant so that it appears to originate from any desired position. That is, the individual participant is recognized not only by his different voice, but also by the location of the audio source related to the speaking participant.

[0009] Although the state of the art implements concepts about how to efficiently encode multiple audio channels or objects, all currently known techniques lack the possibility of combining two or more of these transmitted audio streams efficiently to derive an output stream (output signal), which is a representation of all input audio streams (input audio signals).

[0010] The problem arises, for example, when considering a teleconferencing scenario with more than two locations, each location having one or more speakers. Then, an intermediate instance is required to receive the audio input signals from the individual sources and to generate an audio output signal for each teleconference location having only the information from the remaining teleconference locations. That is, the intermediate instance has to generate an output signal, which is derived from a combination of two or more audio input signals and which allows a reproduction of the individual audio channels or audio objects of the two or more signals. input.

[0011] A similar scenario can arise when two audio engineers in a film-film production wish to combine their spatial audio signals to verify the auditory perception impression generated by both signals. Thus, it may be desirable to directly combine two coded multichannel signals to verify the combined auditory perception impression. That is, a combined signal needs to be such that it resembles all the audio objects (sources) of the two audio engineers.

However, according to prior art techniques, such a combination is only possible by decoding the audio signals (streams). The decoded audio signals can then be re-encoded by prior art multi-channel encoders to generate a combined signal in which all original audio channels or audio objects are appropriately represented.

[0013] This has the disadvantage of high computational complexity, thus wasting a lot of energy and sometimes making it impossible to apply the concept, especially in real-time scenarios. Furthermore, a combination by subsequent decoding and re-encoding of the audio can cause considerable delay due to the two processing steps, which is unacceptable for certain applications, such as teleconference / telecommunication.

Summary of the Invention

[0014] An object of the present invention is to provide a concept for efficiently combining multiple parametrically encoded audio signals.

According to a first aspect of the present invention, this objective is achieved by an audio signal generator according to claim 1 for generating an audio output signal, the audio signal generator comprising: a receiver of audio signal to receive a first audio signal comprising a first sub-mix channel having information about two or more original first channels and comprising a

ES 2 396 072 T3 original parameter associated with one of the first original channels that describes a property of one of the first original channels with respect to a reference channel; and a second audio signal comprising a second submix channel having information on at least one second original channel; a channel combiner for deriving a combined sub-mix channel by combining the first sub-mix channel and the second sub-mix channel; a parameter calculator to derive a first combined parameter that describes the property of one of the first original channels with respect to a common reference channel, and a second combined parameter that describes the property of another of the first original channels or of the at least a second original channel with respect to the common reference channel; and an output interface for producing the audio output signal comprising the combined sub-mix channel and the combined first and second parameters.

According to a second aspect of the present invention, this objective is achieved by a method for generating an audio output signal according to claim 10, the method comprising: receiving a first audio signal comprising a first submix channel having information about two or more first original channels and comprising an original parameter associated with one of the first original channels that describes a property of one of the first original channels with respect to a reference channel; and a second audio signal comprising a second sub-mix channel having information about at least one second original channel; derive a combined sub-mix channel by combining the first sub-mix channel and the second sub-mix channel; derive a first combined parameter that describes the property of one of the first original channels with respect to a common reference channel, and a second combined parameter that describes the property of another of the first original channels or of the at least one second original channel with with respect to the common reference channel; and producing the audio output signal comprising the combined submix channel and the combined first and second parameters.

[0017] According to a third aspect of the present invention, this objective is achieved by a data stream of a representation of three or more audio channels as defined in claim 11, comprising: a sub-channel combined mix that is a combination of a first sub-mix channel that has information about at least two original first channels and a second sub-mix channel that has information about at least one second original channel; a first parameter describing a property of one of the at least two original first channels with respect to a reference channel; and a second parameter that describes the property of another channel of the first original channels or the property of the at least one second original channel with respect to the reference channel.

According to a fourth aspect of the present invention, this objective is achieved by a computer program that implements a method for generating an audio output signal as defined in claim 13, the method comprising: receiving a first audio signal comprising a first submix channel having information about two or more first original channels and comprising an original parameter associated with one of the first original channels that describes a property of one of the first original channels with respect to a reference channel; and a second audio signal comprising a second sub-mix channel having information about at least one second original channel; derive a combined sub-mix channel by combining the first sub-mix channel and the second sub-mix channel; derive a first combined parameter that describes the property of one of the first original channels with respect to a common reference channel, and a second combined parameter that describes the property of another of the first original channels or of the at least one second original channel with with respect to the common reference channel; and producing the audio output signal comprising the combined submix channel and the combined first and second parameters.

According to a fifth aspect of the present invention, this objective is achieved by a conference system having an audio signal generator for generating an audio output signal, comprising: an audio signal receiver receiving a first audio signal comprising a first submix channel having information about two or more first original channels and comprising an original parameter associated with one of the first original channels describing a property of one of the first original channels with respect to a reference channel; and a second audio signal comprising a second submix channel having information about at least one second original channel; a channel combiner for deriving a combined sub-mix channel by combining the first sub-mix channel and the second sub-mix channel; a parameter calculator to derive a first combined parameter that describes the property of one of the first original channels with respect to a common reference channel, and a second combined parameter that describes the property of another of the first original channels or of the at least a second original channel with respect to the common reference channel; and an output interface for producing the audio output signal comprising the combined sub-mix channel and the combined first and second parameters.

According to some embodiments of the present invention, the parameter calculator is operative to use a predetermined channel of the first original channels or the at least one second original channel as a common reference channel.

According to some embodiments of the present invention, the common reference channel is chosen to be the reference channel of the first audio signal.

According to further embodiments of the present invention, the common reference channel is chosen to be the combined sub-mix channel.

ES 2 396 072 T3

[0023] According to further embodiments of the present invention, the common reference channel is chosen to be the original channel that has the highest energy.

[0024] According to further embodiments of the present invention, the audio signal receiver is operative to receive audio signals comprising submix channels represented by sampling parameters sampled at a predetermined sampling rate.

The present invention is based on the discovery that multiple parametrically encoded audio signals can be efficiently combined using an audio signal generator or an audio signal combiner, which generates an audio output signal by combining the audio channels. submix and associated parameters of the audio input signals directly within the parameter domain, ie, without reconstructing or decoding the individual audio input signals before generating the audio output signal. To be more specific, this is achieved by directly mixing the associated submix channels of the individual input signals, for example by adding or forming a linear combination thereof. A key feature of the present invention is that the combining of the submix channels is achieved by simple computationally inexpensive arithmetic operations, such as addition.

The same applies to the combination of the parameters that are associated with the submix channels. Since at least a sub-set of the associated parameters will have to be altered during the combination of the input audio signals, the most important thing is that the calculations carried out to alter the parameters are simple and therefore do not require power. computationally significant or incur additional delays, eg, using filter banks or other operations involving memory.

According to an embodiment of the present invention, an audio signal generator is implemented to generate an audio output signal to combine a first and a second audio signal, both being parametrically encoded. To generate the audio output signal, the audio signal generator of the invention extracts the sub-mix channels from the input audio signals and generates a combined sub-mix channel forming a linear combination of the two input channels. sub-mix. That is, individual channels are added by applying additional weights.

In a preferred embodiment of the present invention, the applied weights are derived by extremely simple arithmetic operations, for example, using the number of channels represented by the first audio signal and the second audio signal as a basis for the calculation. .

In a further preferred embodiment, the weight calculation is carried out assuming that each original audio channel of the input signals contributes the same amount to the total signal energy. That is, the weights applied are simple proportions of the channel numbers of the input signals and the total number of channels.

[0030] In a further preferred embodiment of the present invention, the weights of the individual submix channels are calculated based on the energy contained within the submix channels in order to allow more authentic reproduction of the submix channel. Combined sub-mix included in the generated output audio signal.

[0031] In a further preferred embodiment of the present invention, the computational effort is further decreased in that only the parameters associated with one of the two audio signals are altered. That is, the parameters of the other audio signal are transmitted unaltered, consequently without causing any computation and therefore, minimizing the load on the audio signal generator of the invention.

In the following paragraphs, the concept of the invention will be detailed mainly for an encoding scheme using join source encoding (JSC). In that sense, the present invention extends this technology to connect multiple monophonic or JSC-enabled transceivers to remote stations, mixing the JSC submix signals and object information within the parameter domain. As the above considerations have shown, the concept of the invention is not restricted in any way to the use of JSC encoding, but could also be implemented with BCC encoding or other multi-channel encoding schemes, such as MPEG spatial audio encoding (MPEG surround ) and the like.

[0033] Since the concept of the invention will be detailed mainly using JSC encoding, the JSC encoding will be briefly reviewed within the following paragraphs in order to more clearly point out the flexibility of the concept of the invention and the improvements that can be achieved over the technique. above when applying the concept of the invention to existing multichannel audio coding schemes.

Brief Description of Drawings

[0034]

Figure 1 shows an example of a JSC encoding scheme;

Figure 2 shows an example of a JSC renderer;

Figure 3 shows a teleconferencing scenario with two locations;

ES 2 396 072 T3

Figure 4 shows a teleconferencing scenario with three locations;

Figure 5 shows an example of teleconferencing using an audio signal generator of the invention;

Figure 6 shows a further example of teleconferencing using an audio signal generator of the invention;

Figure 6b shows the reverse compatibility of the concept of the invention; Y

Figure 7 shows an example for an audio signal generator of the invention.

[0035] For explanation of JSC encoding, reference will be made in the following to Figures 1 and 2. Within the following figures, functionally identical components share the same reference marks, indicating that individual components providing the same functionality can be interchanged between individual embodiments of the present invention without losing or restricting functionality and without limiting the scope of the present invention.

[0036] Figure 1 shows a block diagram of the encoding scheme of the junction source, a corresponding encoder 2 and a corresponding decoder 4.

[0037] Encoder 2 receives discrete audio inputs Si (N), 6a, 6b, and 6c, and creates a submix signal s (n) 8, for example, by summing the waveforms.

Additionally, a parameter extractor 10 within encoder 2 extracts the parametric side information for each individual object (signal 6a, 6b and 6c). Although not shown in Figure 1, the downmix signal 8 can be further compressed by a speech or audio encoder and transmitted with the parametric side information to the JSC decoder 4. A synthesis module 12, inside the decoder 4, regenerates the estimates 14a, 14b and 14c (si (n)) of the input objects (channels 6a, 6b and 6c).

In order to reconstruct the estimates 14a, 14b and 14c, being perceptually similar to the discrete input objects (input channels) 6a, 6b and 6c, the appropriate parametric side information has to be extracted for each channel. Since the individual channels are added together for the generation of the downmix signal 8, the energy ratios between the channels are such suitable amounts. Consequently, the parametric information for the different objects or channels consists of delta p energy ratios from each object to the first object (reference object).

[0040] This information is derived in the frequency domain into non-uniformly spaced frequency bands (sub-bands) that correspond to the critical band resolution of human auditory perception. This is a concept described in greater detail, for example, in: J. Blauert, "Spatial Hearing: The Psychophysics of Human Sound Localization", The MIT Press, Cambridge, MA, revised edition 1997.

That is, the wideband input audio channels are filtered into several finite bandwidth frequency bands and for each of the individual frequency bands the following calculations are carried out. As already mentioned, the band energy of the first object (reference object or reference channel) acts as a reference value.

Ap, (n) = 101og_it

Effin)} e ^ w} 'i = 2 ... M

Equation 1

[0042] To avoid the additional introduction of artifacts, for example introduced by means of a division by zero, these energy ratios (in the logarithmic representation) can be further limited to a maximum of, for example, 24 dB in each sub- band. The power ratio can further be quantified before undergoing to further save transmission bandwidth.

[0043] It is not necessary to explicitly transmit the energy of the first object. Rather, this value can be derived from the assumption that, for statistically independent objects, the sum of the energies of the synthesized signals si (n) equals the energy of the submix signal s (n). In terms of a mathematical expression, this means:

M

It is²(n)} = ZE {Í («)} Equation 2

[0044] Based on this assumption and equation, the sub-band energies for the first object (reference object or 6

ES 2 396 072 T3 reference channel) can be reconstructed, as will be further described below when detailing the concept of the invention.

To summarize, an audio signal or audio stream, according to JSC, comprises a submix channel and associated parameters, the parameters describing energy ratios of the original channels with respect to an original reference channel. It can be noted that this scenario can be easily altered in that other channels are selected to be the reference channel. For example, the reference channel itself may be the reference channel, requiring the transmission of an additional parameter, relating the energy of the first reference channel above to the energy of the submix channel. Also, the reference channel can be selected to be variable, in that the only channel having the highest energy is selected to be the reference channel. Hence, since the energy within individual channels can change with time, the reference channel can also change with time. Also, due to the fact that all processing is typically carried out in a frequency selective manner, the reference channel may be different for different frequency bands.

[0046] Figure 2 shows a further improved JSC coding scheme, based on the scheme of Figure 1. The detailed characteristics with respect to Figure 1, are included with the storage or transmission box 20, which receives the input channels 6 to be encoded and the estimates 14 that are output from input channels 6. The scheme of Figure 2 is improved in that it also comprises a mixer 22 that receives the estimates. That is, the synthesized objects 14 are not output directly as single audio signals, but represented to the N output channels in the mixer module. Such mixing can be implemented in different ways, for example, by receiving the additional mixing parameters 24 as input, to direct the mixing of the synthesized objects 14. As an example only, a teleconferencing scenario can be considered, in which each of the output channels 26 is allocated to a conference participant. Consequently, a participant at the receiving end has the possibility to virtually separate the other participants by assigning their voices to individual positions. Thus, not only voice can serve as a criterion to distinguish between different participants in a conference call, but also the direction from which the listener receives a participant's voice. In addition, the listener can arrange the output channel in such a way that all participants from the same teleconferencing location are grouped in the same direction, further enhancing the perceptual experience.

[0047] As shown in Figure 2, sl (n) ... sM (n) denotes the discrete audio objects at the input of the JSC encoder. At the JSC decoder output sl (n) ... sM (n) represents the virtually separate audio objects that are fed into the mixer. The mixing parameters 24 can be interactively modified on the receiver side to place the different objects in a sound stage that is reproduced by the output channels sl (n) ... sN (n).

[0048] Figure 3 shows the application of multi-channel audio coding schemes to a basic teleconferencing scenario, taking place between two locations. Here, a first location 40 communicates with a second location 42. The first location may have A participants, ie, A audio objects, the second location has B participants or audio objects. For point-to-point teleconferencing, the described JSC coding technology can be applied directly to transmit the audio signals from multiple objects at each location to the corresponding remote station. That is, parameters (A-1) ai and an associated submix are transferred to location 42. In the opposite direction, parameters (B-1) bl are transmitted together with a submix associated with location 40.

[0049] For teleconferencing with more than two end points, the situation is completely different as illustrated in Figure 4.

[0050] Figure 4 shows, in addition to locations 40 and 42, a third location 44. As can be seen in Figure 4, such a scenario requires a central distributor for the associated audio signals, generally called a multi-point control unit. , MCU. Each of the locations (sites) 40, 42, and 44 is connected to the MCU 46. For each site 40, 42, and 44, there is a unique upstream for the MCU that contains the signal from the site. Since each individual site needs to receive the signals from the remaining sites, the downdraft for each site 40, 42, and 44 is a mix of the signals from the other sites, excluding the site's own signal, which is also referred to as the signal. (N-1). Generally, to meet installation requirements and to keep transmission bandwidth reasonably low, it is not possible to transmit the JSC N-1 encoded streams from the MCU to each site. This, of course, would be the direct option.

The state of the art aimed at deriving the individual downdrafts is to synthesize all the incoming currents (objects) within the MCU 46 using a JSC decoder. The resynthesized audio objects could then be regrouped and re-encoded in order to provide each site with audio streams comprising the desired audio objects or audio channels. Even within this simple scenario, this would mean three decoding and three encoding tasks, which must be carried out simultaneously within the MCU 46. Despite significant computational demands, additional audible artifacts can be expected by this parametric process of " serial encoding ”(repeated encoding / decoding). Increasing the number of sites would further increase the number of streams and, therefore, the number of encoding or decoding processes required, making direct procedures for real-time scenarios impossible.

According to the present invention, accordingly, a scheme is developed to mix different

ES 2 396 072 T3 parametrically encoded streams (JSC streams in this particular example) directly within the submix domain and object for such an MCU-like scenario, creating the desired output signals (output audio streams) with a minimum of computational effort and loss of quality.

Within the following paragraphs, the inventive concept of directly mixing parametrically encoded multi-channel audio streams within the parameter domain is detailed for JSC encoded audio streams.

[0054] The concept of the invention is explained by combining two original (current) audio signals into one output signal. By joining three or more streams together they can easily be derived in the case of combining two streams. The following mathematical considerations are illustrated by Figure 5, which shows a case where three audio channels from Site A have to be combined with four audio channels from Site B. This is, of course, only an example to visualize the concept of the invention. By using JSC encoding, site 50 (A) having three conference participants (speakers) 52a to 52c, generating sax signals, transmits an audio stream or audio signal 54. The audio signal 54 has one channel of sub-mix sa and parameters a2 ya₃, which relate the energy of channels 52b and 52c to the energy of channel 52a. Equivalently, site 56 (B) transmits an audio signal 58 that has a sub-mix channel sb and three parameters b₂, b3 and b4, which are the JSC-encoded representation of four speakers 60a through 60d. MCU 46 combines the audio signals 54 and 58 to derive an output signal 62 that has a combined sub-mix channel sy and six parameters and₂, ... and 7.

On the receiving side, the receiver 64 decodes the output signal 62 to derive representations of the 7 audio objects or audio channels from sites 50 and 56.

[0056] In general terms, the objective is to form a single combined representation 62 of two JSC streams 54 and 58, each representing a number of objects by a common submix signal sy and a set of object parameters that characterize the objects. Ideally, the combined JSC representation should be identical to that which would be obtained by encoding the entire set of original source signals joining both JSC streams into a single JSC stream in one stage.

[0057] To keep the following equations simple, we assume that the relative energy ratios in Equation 1 are not available in the logarithmic domain, but only as energy ratios. Each object parameter r, (n) of a certain object i can be derived as

E {s>)}

ER (n) j

Equation 3

[0058] Logarithmic domain transposition can subsequently be applied to each parameter to allow quantization using a logarithmic energy scale.

[0059] All the following signals are assumed decomposed into a sub-band representation, therefore, each of the calculations is applied for each sub-band separately.

[0060] We have the current A, with its sub-mixing signal sa and the parameters (relative energy ratios) for the objects U a₂... au. Stream B consists of the submix signal sb and the parameters for the objects V b₂... bv.

The combined submix signal sy can be formed as a linear combination of both the submix signals sa and sb. To ensure correct volume leveling of the different object contributions, the gain factors g can be applied_TO and g_B.

^SAnd Sa ' ^STO ^{+ g}B ' ^SB ^{with 8a} ~ $ + vy^gB ~ (u + f)

[0062] This type of scale can be meaningful if individual sound sources of equal average energy have been added and if they have been normalized to the full scale of the sub-mix path.

Alternatively, an energy conservation procedure could be used for gain factors with ^ (u + y) '^gB + k)

ES 2 396 072 T3

Another possibility is to select the gain factor such that both sub-mix signals contribute the same average energy for the combined sub-mix, ie, selecting g = E {?₄(n)}

Sa] ^AND{4⁽«⁾}

The object parameters y¡ for the combined current s_Y must represent all objects U + V.

[0066] Since the parameters associated with the submix channels are relative energy ratios, the parameters a2, ..., a_or can be used as found (unaltered) and the parameters for the objects in B can be concatenated to the parameters a2, ..., a_or. Once the first signal object A is selected to be the reference object or reference channel, the original parameters b, have to be transformed to relate to that reference channel. It can be noted that only the parameters of one current have to be recalculated, further decreasing the computational load within an MCU 46.

[0067] It can further be noted that it is by no means necessary to use the reference channel of one of the original audio streams as a new reference channel. The inventive concept of combining parametrically encoded audio streams within the parameter domain can also very well be implemented with other reference channels, selected from the number of original channels at sites A or B. An additional possibility would be to use the combined submix channel as a new reference channel.

Following this procedure of using the original reference channel of site A as the new reference channel (combined reference channel), the power energy) of the first object (channel) of each signal A and B has to be calculated first, given these are only implicitly available.

[0069] The energy representation for the sub-mix signal A, assuming the statistically independent sources, provides:

<img file="ES2396072T3_D0001.tif" />

<img file="ES2396072T3_D0002.tif" />

<img file="ES2396072T3_D0003.tif" />

[0070] The signal energies

<img file="ES2396072T3_D0004.tif" />

<img file="ES2396072T3_D0005.tif" />

(«) J are defined with their relative energies α₂... αυ until

[0071]

<img file="ES2396072T3_D0006.tif" />

<img file="ES2396072T3_D0007.tif" />

<img file="ES2396072T3_D0008.tif" />

[0072] This leads to sai energy as:

ES 2 396 072 T3 w^{&gt; =} / X

E \ S (n)

TO ^{v 7}

X.

l

By applying the same for the submix signal sb, the energy of the object sbi can be calculated as:

()

()

B ^{V 7}

X.

J ⁽7 + Óg + ... + Z&gt;_JZ⁾

[0074] Now the new set of parameters can be constructed for all objects of the sy signal:

yi: (not transmitted, reference object, implicitly available) y2 = a2 y3 = a3 yu = aU yu + i

<img file="ES2396072T3_D0009.tif" />

<img file="ES2396072T3_D0010.tif" />

(energy ratio of the first object of signal B with respect to the reference object A1) yu + 2

<img file="ES2396072T3_D0011.tif" />

(n)&gt;

(ratio of the energy of the second object of the renormalized signal B to the energy of the reference object A1) yu + 3

<img file="ES2396072T3_D0012.tif" />

yn + v

<img file="ES2396072T3_D0013.tif" />

(n)&gt;

As the previous paragraphs have shown, the concept of the invention allows the generation of a combined audio stream using only simple arithmetic operations, being, therefore, computationally

ES 2 396 072 T3 extremely efficient. Therefore, the combination of multiple parametrically encoded audio streams can be carried out in real time.

[0076] To further emphasize the great flexibility of the concept of the invention, Figure 6 shows how a monophonic signal 70, caused by a single speaker at site 56, can be combined according to the invention, with two or more signals encoded by JSC from speakers on site 50. That is, due to the flexibility of the concept of the invention, monophonic signals from arbitrary teleconferencing systems can be combined according to the invention, with parametrically encoded multi-channel sources (multi-objects) to generate an audio signal encoded by JSC representing all original audio channels (objects).

[0077] By extending the compatibility also with remote stations that are not capable of transmitting JSC objects, but traditional monophonic signals, this technique is also applicable to insert a monophonic object, eg, from a legacy conferencing device into the stream based on the object.

The previous example with the current JSC A (sub-mix sa, parameters a2 ... au) and a monophonic object C (sub-mix sC) leads to a combined signal Z with the sub-mix signal

Sz = gA · Sa + ge · Sc with gain factors as previously discussed and their object parameters:

yy. not transmitted (reference channel, implicitly available) y2 = a2 y3 = a3 yu = a_or

Yu + i

<img file="ES2396072T3_D0014.tif" />

(ratio of energy of signal C to reference object A1)

[0079] The aforementioned example to transcode / merge two JSC streams, depends on the representation of the energy of the objects as provided in Equation 1. However, the same scheme of the invention can also be applied to other ways of representing this information.

[0080] Figure 6b emphasizes again the great flexibility of the concept of the invention incorporating a monophonic audio source. Figure 6b is based on the multi-channel scenario of Figure 4 and further shows how easily a prior art mono audio encoder can be integrated into an audio source c (44), in a multi-channel audio conference using the MCU 46 of the invention.

As mentioned previously, the concept of the invention is not restricted to JSC coding having a fixed predetermined reference channel. Accordingly, in an alternative example, the energy ratio can be computed with respect to a reference channel, which is variable with time, the reference channel being one that has the highest energy within a predetermined time interval.

Instead of normalizing the energy values ​​of the band-shaped signal with the energy of the corresponding band of a fixed reference channel (object) and transposing the result to the logarithmic domain (dB) as noted in the Equation 1, normalization can take place in relation to the maximum energy over all objects in a certain frequency band:

pnornii (n) = ---- γ max (i

UA ¿. Ί

E &lt;S (n)&gt;

X.

, i = 1 ... M

Equation 4

These normalized energy values ​​(which are provided in a linear representation) do not need any additional limitation for a certain upper limit, since innately they can only take the values ​​between 0 and 1.

ES 2 396 072 T3

This advantage outweighs the disadvantage of having to transmit an additional parameter for the reference channel not already known a priori.

[0084] The mixing process for this scenario would include the following steps (which again have to be carried out for each sub-band separately):

We have a current A with its sub-mix signal sa and the parameters (normalized energy values,

Equation 3, Equation 1) for the objects U ai ... aU.

Stream B consists of the submix signal sb and the parameters for the objects V bi ... bV.

[0085] A combined submix signal can be formed according to one of the options already shown:

^sy = Sa ' ^sto ⁺ Sb ^sb

[0086] All normalized energy values ​​for the combined representation y¡ have to be adjusted relative to the object with the highest energy of all objects in the Y signal. There are no two candidates to be this "maximum object" of Y Both the maximum object of A and the maximum object of B can be identified by having a normalized energy ratio of "1".

[0087] This decision can be made by comparing the absolute energy of both candidates. Again the relation for the energy of the submix signals (Equation 2) can be used to obtain:

í ²

THAT ( ⁿgirl: t

()

Γ 2 ¡\

E ^ S _λ («H

- y e &lt;s

Bmax

() í ² / \

The S (i)&gt;

[0088] Now, we can compare the weighted maximum object energies with the gain factors of the sub-mixing process:

<img file="ES2396072T3_D0015.tif" />

E &lt;S

TO

Ήηαχ

<img file="ES2396072T3_D0016.tif" />

E &lt;S τ&gt;

ⁿmax

<img file="ES2396072T3_D0017.tif" />

[0089] Whatever the highest energy of the object, this object will serve as the "maximum object" for the combined parameters y¡.

As an example, let a2 be the total maximum energy object amax of both signals A and B, then all other parameters can be combined as:

^Y1 = 31

Y2 = 32 yu = 3u yu + 1 =

<img file="ES2396072T3_D0018.tif" />

IT IS

5max

4’

ES 2 396 072 T3 (energy ratio of the first object of signal B with respect to the “maximum object”, here a₂) ⁸B

Y_{ü + 2}= b₂· —E &lt;S

B max

Yu + V

<img file="ES2396072T3_D0019.tif" />

max max

[0091] For this example, all parameters for objects A can remain unchanged, since signal A contained the total maximum object.

[0092] Also in this representation, the insertion of a monophonic object can therefore be effected, eg, assuming that V = 1.

[0093] Generally, the transcoding process is carried out in such a way that its result achieves the result that would have been obtained if all the original objects for both streams had been encoded in a single JSC stream in the first place.

[0094] Figure 7 shows an example for an audio signal generator of the invention to generate an audio output signal, as it can be used within the MCU 46 to implement the concept of the invention.

The audio signal generator 100 comprises an audio signal receiver 102, a channel combiner 104, a parameter calculator 106, and an output interface 108.

The audio signal receiver 103 receives a first audio signal 110 comprising a first submix channel 110a having information about two or more of the first original channels and comprising an original parameter 110b associated with one of the first original channels describing a property of one of the first original channels relative to a reference channel. The audio signal receiver 102 further receives a second audio signal 112 comprising a second sub-mix channel 112a having information about at least one second original channel.

The audio signal receiver outputs the first sub-mix channel 110a and the second sub-mix channel 112a to an input of the channel combiner 104 and to the first sub-mix channel 110a, the second sub-mix channel submix 112a and original parameter 110b to parameter calculator 106.

[0098] Channel combiner 104 derives a combined submix channel 114 by combining the first submix channel 110a and a second submix channel 112b, ie, by combining the submix channels directly without reconstructing the channels. underlying original audio files.

The parameter calculator 106 derives a first combined parameter 116a that describes the property of one of the first original channels with respect to a common reference channel and a second combined parameter 116b that describes the property of another of the first original channels or of the at least one second original channel with respect to the same common reference channel. The first and second combined parameters are input to output interface 108, which further receives combined submix channel 114 from channel combiner 104. Finally, the output interface outputs an output signal 120 comprising the combined submix channel 114 and combined first and second parameters 116a and 116b.

[0100] The audio output signal has therefore been derived without full reconstruction of the input audio signals and therefore without computationally costly operations.

[0101] Within the preceding paragraphs, the general concept of mixing two or more signals, each based on a JSC parametric procedure, has been shown. In particular, the above equations show how to apply these techniques for a case, where the parametric information consists of relative energy ratios. However, this technique is not restricted to a specific representation of the object parameters. Consequently, parameters describing amplitude measurements or other properties of individual audio channels, such as correlations, can also be used. Power ratios can also be computed with respect to the combined submix channel, at the cost of transmitting an additional parameter. On the other hand, one benefits from this alternative scenario because of the reduced computational complexity during mixing of the audio streams, since the energy reconstruction of the reference channel, which is not explicitly transmitted in "generic" JSC, is obsolete. .

[0102] Furthermore, the invention is not limited to a teleconferencing scenario, but can be applied when desired

ES 2 396 072 T3 multiplexing parametric objects into a single stream. This, for example, may be the case in BCC coding schemes, MPEG spatial envelopes, and others.

[0103] As shown, the concept of the invention even allows seamlessly to include legacy remote stations that provide a single mono signal on stage based on the object. In addition to combining different streams of objects, the concept of the invention also shows how many different ways of representing parametric data can be generated, so that they are suitable to allow computationally efficient combining processes. As such, it is an advantageous feature of a parametric bitstream syntax of the invention to express the properties of the object in such a way that two streams can be combined by merely performing simple operations.

[0104] Accordingly, the concept of the invention also shows how to create appropriate bit streams or bit stream formats to parametrically encode multiple audio channels (audio objects), adhering to the following criteria:

• The combined sub-mix signal is simply formed from the sub-mix signals.

• The combined parametric lateral information is formed from the combination of the individual parametric lateral information and some simple ones to compute the characteristics of the sub-mixing signals (eg, energy).

• In no case does a complex operation such as a decoding / encoding step have to be performed for the audio objects.

[0105] Consequently, the parametric representation describing the objects must be selected in such a way that a combination ("addition") of two or more object streams is possible using only bit stream fields that are available as part of the parametric side information, and possibly simple to compute the metrics of the sub-mix signals (eg, energy, peak value).

[0106] An example for such a representation could be to use normalized energy values ​​(Equation 4) for each object. These could be transformed into a logarithmic (dB) representation and then quantized to a number of quantizer stages or to their representative quantizer indices. The bitstream syntax should allow easy incrementing (or decrementing) the number of object parameters in a stream, eg, simply by concatenating, inserting, or removing the parameters.

[0107] In summary, the concept of the invention allows a more flexible and computationally efficient combination of parametrically encoded audio streams. Due to the high computational efficiency, the concept of the invention is not restricted to a maximum number of channels to be combined. Mainly, the channels, which can be combined in real time, can be provided to an audio signal generator of the invention in arbitrary numbers. Also, the precise parametric representation (JSC) used to illustrate the concept of the invention is not mandatory. Furthermore, as already mentioned, other parametric coding schemes, such as the commonly known wrap-around schemes, can be the basis for the application of the concept of the invention.

[0108] Furthermore, the necessary computations do not necessarily have to be applied in software. Hardware implementations using, for example, DSPs, ASICs and other integrated circuits, can also be used to carry out calculations, which will further increase the speed of the concept of the invention, allowing the application of the concept of the invention in scenarios in real time.

[0109] Due to the flexibility of the inventive concept, the inventive audio streams can be based on different parametric representations. The parameters to be transmitted could, for example, also be amplitude measurements, time differences between the original audio channels, coherence measurements and others.

[0110] Therefore, the general concept of mixing two or more signals each based on a JSC-type parametric procedure has been shown.

[0111] The above equations show how to apply this technique for a case, where the parametric information consists of relative energy proportions. However, this technique is not restricted to a specific representation of the object parameters.

[0112] Furthermore, the invention is not limited to a teleconferencing scenario, but can be applied in any case where multiplexing of parametric objects in a single JSC stream is advantageous.

[0113] Furthermore, this technique allows seamlessly including legacy remote stations that provide a single mono signal on stage based on the object.

[0114] In addition to the actual process for combining different object streams, the invention also shows how many different ways of representing parametric data are suitable to enable this combining process. Since not all possible parametric representations allow such described combining process without full decoding / recoding of the objects, it is an advantageous feature of the bitstream syntax

Parametric ES 2 396 072 T3 express the properties of the object in such a way that two streams can be combined by merely performing simple operations.

[0115] Depending on certain implementation requirements of the methods of the invention, the methods of the invention can be implemented in hardware or software. The implementation can be carried out using a digital storage medium, in particular a disk, DVD or CD having electronically readable control signals stored therein, which cooperate with a programmable computerized system in such a way that they are carried out. carry out the processes of the invention. Generally, the present invention is, accordingly, a computer program product with a program code stored in a machine-readable vehicle, the code of the operating program for carrying out the procedures of the invention being, when the product of computer program runs on a computer. In other words, the methods of the invention, therefore, are a computer program that has a program code to carry out at least one of the procedures of the invention when the computer program runs on a computer.

[0116] Although the foregoing has been particularly shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various other changes in form and details can be made without departing from the scope thereof. . It will be understood that various changes can be made to accommodate different embodiments without departing from the broader concepts described herein and encompassed by the following claims.