EP3704875B1

Virtual rendering of object based audio over an arbitrary set of loudspeakers

Abstract

This record has no abstract on file.

EP3704875B1, drawing sheet 1 — Sheet 1 of 23

Term

12.1 yearsleft in the term

Expires 24 October 2038.

14 claims: 10 independent, 4 dependent

1
A method (400) of rendering audio, the method comprising:deriving (402) an optimal set of a plurality of filters, wherein each of the plurality of filters is associated with a corresponding one of a plurality of loudspeakers, wherein deriving the optimal set of a plurality of filters includes: defining (404) a binaural error for an audio object as a function of the plurality of filters, wherein the binaural error is a difference between desired binaural signals related to at least one listener position and modeled binaural signals related to the at least one listener position, wherein the desired binaural signals are defined based on the audio object and the desired perceived position of the audio object, from the perspective of at least one listener position, and the modeled binaural signals are defined by modeling a playback of a plurality of rendered signals, through the plurality of loudspeakers having a plurality of loudspeaker positions, based on the at least one listener position, defining (406) an activation penalty for the audio object using the plurality of filters, wherein the activation penalty is a distance penalty which has the property that for the same amount of overall energy of the plurality of rendered signals, where overall means combining across all loudspeakers, the penalty increases when more of that energy is concentrated in loudspeakers of the plurality of loudspeakers whose nominal position is more distant from the desired perceived position of the audio object, and minimizing (408) a cost function with respect to the plurality of filters, wherein the cost function is a combination of the binaural error and the activation penalty for the plurality of filters;rendering (410) the audio object using the derived optimal set of a plurality of filters to generate a plurality of rendered signals;and outputting (412), by the plurality of loudspeakers, the plurality of rendered signals.
4
The method of any one of claims 1-3, wherein the cost function is a combination function that is monotonically increasing in both A and B, wherein A corresponds to the binaural error and B corresponds to the activation penalty.
6
The method of any one of claims 1-5, wherein the audio object is one of a plurality of audio objects, wherein the plurality of audio objects is rendered using the plurality of filters, and wherein each of the plurality of audio objects has an associated desired perceived position.
7
The method of any one of claims 1-6, wherein the plurality of loudspeakers includes a first loudspeaker and a second loudspeaker, wherein the first loudspeaker has a nominal position that is at a first distance from the desired perceived position of the audio object, and wherein the second loudspeaker has a nominal position that is at a second distance from the desired perceived position of the audio object, wherein the first distance is greater than the second distance.
8
The method of any one of claims 1-7, wherein the plurality of loudspeakers has a plurality of nominal loudspeaker positions, wherein each of the plurality of nominal loudspeaker positions is one of a first position and a second position, wherein the first position is an actual loudspeaker position of a corresponding one of the plurality of loudspeakers, and wherein the second position is other than the actual loudspeaker position.
9
The method of any one of claims 1-8, wherein one of the plurality of loudspeakers has a nominal loudspeaker position, wherein the nominal loudspeaker position is derived by expanding one or more physical positions of the plurality of loudspeakers.
10
The method of any one of claims 1-9, wherein the plurality of filters is independent of the audio object.
12
The method of any one of claims 1-11, wherein the plurality of loudspeakers has a plurality of physical positions, wherein the plurality of physical positions are determined in a setup phase.
13
A non-transitory computer readable medium storing a computer program that, when executed by a processor, controls an apparatus to execute processing including the method of any one of claims 1-11.
14
An apparatus (300) for rendering audio, the apparatus comprising:a plurality of loudspeakers;and at least one processor, wherein the at least one processor is configured to derive an optimal set of a plurality of filters, wherein each of the plurality of filters is associated with a corresponding one of the plurality of loudspeakers, wherein deriving the optimal set of a plurality of filters includes: defining a binaural error for an audio object as function of the plurality of filters, wherein the binaural error is a difference between desired binaural signals related to at least one listener position and modeled binaural signals related to the at least one listener position, wherein the desired binaural signals are defined based on the audio object and the desired perceived position of the audio object, from the perspective of at least one listener position, and the modeled binaural signals are defined by modeling a playback of a plurality of rendered signals, through the plurality of loudspeakers having a plurality of loudspeaker positions, based on the at least one listener position, defining an activation penalty for the audio object using the plurality of filters, wherein the activation penalty is a distance penalty which has the property that for the same amount of overall energy of the plurality of rendered signals, where overall means combining across all loudspeakers, the penalty increases when more of that energy is concentrated in loudspeakers of the plurality of loudspeakers whose nominal position is more distant from the desired perceived position of the audio object, and minimizing a cost function with respect to the plurality of filters, wherein the cost function is a combination of the binaural error and the activation penalty for the plurality of filters;wherein the at least one processor is configured to render the audio object using the derived optimal set of a plurality of filters to generate a plurality of rendered signals, and wherein the plurality of loudspeakers is configured to output the plurality of rendered signals.

Independent claims10

129 paragraphs in 3 sections, as filed

BACKGROUND

0001The present invention relates to audio processing, and in particular, to rendering object based audio over an arbitrary set of loudspeakers.

0002Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

0003Object based audio generally refers to generating loudspeaker feeds based on audio objects. Object based audio may generally be contrasted with channel based audio. In channel based audio, each channel corresponds to a loudspeaker. For example, 5.1 surround sound is channel based, with the "5" referring to left, right, center, left surround and right surround loudspeakers and their five corresponding channels, and the "1" referring to a low-frequency effects speaker and its corresponding channel. On the other hand, object based audio renders audio objects for output by loudspeakers whose numbers and arrangements need not be defined by the audio objects; instead, each audio object may include location metadata that is used during the rendering process so that the audio for that audio object is output by the loudspeakers such that the audio object is perceived to originate at the desired location.

0004Binaural audio generally refers to audio that is recorded, or played back, in such a way that accounts for the natural ear spacing and head shadow of the ears and head of a listener. The listener thus perceives the sounds to originate in one or more spatial locations. Binaural audio may be recorded by using two microphones placed at the two ear locations of a dummy head. Binaural audio may be rendered from audio that was recorded non-binaurally by using a head-related transfer function (HRTF) or a binaural room impulse response (BRIR). Binaural audio may be played back using headphones. Binaural audio generally includes a left signal (to be output by the left headphone or left loudspeaker), and a right signal (to be output by the right headphone or right loudspeaker). Binaural audio differs from stereo in that stereo audio may involve loudspeaker crosstalk between the loudspeakers.

0005The so-called "virtual" rendering of spatial audio over a pair of loudspeakers commonly involves the creation of a stereo binaural signal which is then fed through a cross-talk canceller to generate left and right speaker signals. The binaural signal represents the desired sound arriving at the listener's left and right ears and is synthesized to simulate a particular audio scene in 3D space, containing possibly a multitude of sources at different locations. The crosstalk canceller attempts to eliminate or reduce the natural crosstalk inherent in stereo loudspeaker playback so that the left channel of the binaural signal is delivered substantially to the left ear only of the listener and the right channel to the right ear only, thereby preserving the intention of the binaural signal. Through such rendering, audio objects are placed "virtually" in 3D space since a loudspeaker is not necessarily physically located at the point from which a rendered sound appears to emanate. The theory and history of such rendering is discussed extensively by <nplcit id="ncit0001" npl-type="b"><text>W. Gardner, "3-D Audio Using Loudspeakers" (Kluwer Academic, 1998</text></nplcit>).

0006<patcit id="pcit0001" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157 </text></patcit>discusses virtual rendering of object based audio through binaural rendering of each object followed by panning of the resulting stereo binaural signal between a plurality of cross-talk cancellation circuits feeding a corresponding plurality of speaker pairs.

0007<figref idref="f0001">FIG. 1</figref> is a block diagram of a loudspeaker system 100. The loudspeaker system 100 is used to illustrate the design of a cross-talk canceller, which is based on a model of audio transmission from the loudspeakers 102 and 104 to a listener's ears 106 and 108. Signals sL and sR represent the signals sent from the left and right loudspeakers 102 and 104, and signals eL and eR represent the signals arriving at the left and right ears 106 and 108 of the listener. Each ear signal is modeled as the sum of the left and right loudspeaker signals each filtered by a separate linear time-invariant transfer function H modeling the acoustic transmission from each speaker to that ear. These four transfer functions may be modeled using head related transfer functions (HRTFs) selected as a function of an assumed speaker placement with respect to the listener.

0008The model depicted in <figref idref="f0001">FIG. 1</figref> can be written in matrix equation form as follows: <maths id="math0001" num="(1)"><math display="block"><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>e</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>e</mi><mi>R</mi></msub></mtd></mtr></mtable></mfenced><mo>=</mo><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>H</mi><mi mathvariant="italic">LL</mi></msub></mtd><mtd><msub><mi>H</mi><mi mathvariant="italic">RL</mi></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mi mathvariant="italic">LR</mi></msub></mtd><mtd><msub><mi>H</mi><mi mathvariant="italic">RR</mi></msub></mtd></mtr></mtable></mfenced><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>s</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>s</mi><mi>R</mi></msub></mtd></mtr></mtable></mfenced><mspace width="1ex" /><mi>or</mi><mspace width="1ex" /><mi mathvariant="bold">e</mi><mo>=</mo><mi mathvariant="bold">Hs</mi></math><img file="EP3704875B1_D0001.tif" /></maths>

0009Equation 1 reflects the relationship between signals at one particular frequency and is meant to apply to the entire frequency range of interest, and the same applies to all subsequent related equations. A crosstalk canceller matrix C may be realized by inverting the matrix H:<maths id="math0002" num="(2)"><math display="block"><mi mathvariant="bold">C</mi><mo>=</mo><msup><mi mathvariant="bold">H</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mi>H</mi><mi mathvariant="italic">LL</mi></msub><msub><mi>H</mi><mi mathvariant="italic">RR</mi></msub><mo>−</mo><msub><mi>H</mi><mi mathvariant="italic">LR</mi></msub><msub><mi>H</mi><mi mathvariant="italic">RL</mi></msub></mrow></mfrac><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>H</mi><mi mathvariant="italic">RR</mi></msub></mtd><mtd><mrow><mo>−</mo><msub><mi>H</mi><mi mathvariant="italic">RL</mi></msub></mrow></mtd></mtr><mtr><mtd><mrow><mo>−</mo><msub><mi>H</mi><mi mathvariant="italic">LR</mi></msub></mrow></mtd><mtd><msub><mi>H</mi><mi mathvariant="italic">LL</mi></msub></mtd></mtr></mtable></mfenced></math><img file="EP3704875B1_D0002.tif" /></maths>

0010Given left and right binaural signals bL and bR, the speaker signals sL and sR are computed as the binaural signals multiplied by the crosstalk canceller matrix: <maths id="math0003" num="(3)"><math display="block"><mi mathvariant="bold">s</mi><mo>=</mo><mi mathvariant="bold">Cb</mi><mspace width="1ex" /><mi>where</mi><mspace width="1ex" /><mi mathvariant="bold">b</mi><mo>=</mo><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>b</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>b</mi><mi>R</mi></msub></mtd></mtr></mtable></mfenced></math><img file="EP3704875B1_D0003.tif" /></maths>

0011Substituting Equation 3 into Equation 1 and noting that C=H-1 yields: <maths id="math0004" num="(4)"><math display="block"><mi mathvariant="bold">e</mi><mo>=</mo><mi mathvariant="bold">HCb</mi><mo>=</mo><mi mathvariant="bold">b</mi></math><img file="EP3704875B1_D0004.tif" /></maths>

0012In other words, generating speaker signals by applying the crosstalk canceller to the binaural signal yields signals at the ears of the listener equal to the binaural signal. This assumes that the matrix H perfectly models the physical acoustic transmission of audio from the speakers to the listener's ears. In reality, this will not be the case, so Equation 4 will in general be approximated. In practice, however, this approximation is close enough that a listener will substantially perceive the spatial impression intended by the binaural signal b.

0013Oftentimes, the binaural signal b is synthesized from a monaural audio object signal o through the application of binaural rendering filters BL and BR:<maths id="math0005" num="(5)"><math display="block"><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>b</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>b</mi><mi>R</mi></msub></mtd></mtr></mtable></mfenced><mo>=</mo><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>B</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>B</mi><mi>R</mi></msub></mtd></mtr></mtable></mfenced><mi>o</mi><mspace width="1ex" /><mi>or</mi><mspace width="1ex" /><mi mathvariant="bold">b</mi><mo>=</mo><mi mathvariant="bold">B</mi><mi>o</mi></math><img file="EP3704875B1_D0005.tif" /></maths>

0014The rendering filter pair B is most often given by a pair of HRTFs chosen to impart the impression of the object signal o emanating from an associated position in space relative to the listener. In equation form, this relationship may be represented as: <maths id="math0006" num="(6)"><math display="block"><mi mathvariant="bold">B</mi><mo>=</mo><mi mathvariant="italic">HRTF</mi><mfenced open="{" close="}"><mi mathvariant="italic">pos</mi><mfenced><mi>o</mi></mfenced></mfenced></math><img file="EP3704875B1_D0006.tif" /></maths>

0015Here pos(o) represents the desired position of object signal o in 3D space relative to the listener. This position may be represented in Cartesian (x,y,z) coordinates (e.g., Cartesian distance) or any other equivalent coordinate system such as polar (e.g., angular distance including a distance and a direction). This position might also varying in time to simulate movement of the object through space. The function HRTF{ } is meant to represent a set of HRTFs addressable by position. Many such sets measured from human subjects in a laboratory exist, such as the University of California Davis' Center for Image Processing and Integrated Computing (CIPIC) database, described at <interface.cipic.ucdavis.edu>. Alternatively, the set might be comprised of a parametric model such as the spherical head model described in <nplcit id="ncit0002" npl-type="s"><text>P. Brown and R. Duda, "A Structural Model for Binaural Sound Synthesis", IEEE Transactions on Speech and Audio Processing, September 1998, Vol. 6, No. 5, pp. 476-478</text></nplcit>. In a practical implementation, the HRTFs used for constructing the crosstalk canceller are often chosen from the same set used to generate the binaural signal, though this is not a requirement.

0016In many applications, a multitude of objects at various positions in space are simultaneously rendered. In such a case, the binaural signal is given by a sum of object signals with their associated HRTFs applied: <maths id="math0007" num="(7)"><math display="block"><mi mathvariant="bold">b</mi><mo>=</mo><mstyle displaystyle="true"><munderover><mo>∑</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><mrow><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub></mrow></mstyle><mspace width="1ex" /><mi>where</mi><mspace width="1ex" /><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub><mo>=</mo><mi mathvariant="italic">HRTF</mi><mfenced open="{" close="}"><mi mathvariant="italic">pos</mi><mfenced><msub><mi>o</mi><mi>k</mi></msub></mfenced></mfenced></math><img file="EP3704875B1_D0007.tif" /></maths>

0017With this multi-object binaural signal, the entire rendering chain to generate the speaker signals is given by: <maths id="math0008" num="(8)"><math display="block"><mi mathvariant="bold">s</mi><mo>=</mo><mi mathvariant="bold">C</mi><mstyle displaystyle="true"><munderover><mo>∑</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><mrow><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub></mrow></mstyle></math><img file="EP3704875B1_D0008.tif" /></maths>

0018In many applications, the object signals ok are given by the individual channels of a multichannel signal, such as a 5.1 signal comprised of left, center, right, left surround, and right surround. In this case, the HRTFs associated with each object may be chosen to correspond to the fixed speaker positions associated with each channel. In this way, a 5.1 surround system may be virtualized over a set of stereo loudspeakers. In other applications the objects may be sources allowed to move freely anywhere in 3D space. In the case of a next generation spatial audio format, as described in <nplcit id="ncit0003" npl-type="s"><text>C. Q. Robinson, S. Mehta, and N. Tsingos, "Scalable Format and Tools to Extend the Possibilities of Cinema Audio," SMPTE Motion Imaging Journal, vol. 121, no. 8, pp. 63-69, Nov. 2012</text></nplcit>, the set of objects in Equation 8 may consist of both freely moving objects and fixed channels.

0019The two speaker / one listener cross-talk canceller can be generalized to an arbitrary number of speakers located at arbitrary positions with respect to an arbitrary number of listeners also at arbitrary positions. This is achieved by extending Equation 1 from two speakers and one listener to M speakers and N listeners: <maths id="math0009" num="(9)"><math display="block"><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>e</mi><mrow><mi>L</mi><mn>1</mn></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>e</mi><mrow><mi>R</mi><mn>1</mn></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>e</mi><mrow><mi>L</mi><mn>2</mn></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>e</mi><mrow><mi>R</mi><mn>2</mn></mrow></msub></mtd></mtr><mtr><mtd><mi mathvariant="normal">M</mi></mtd></mtr><mtr><mtd><msub><mi>e</mi><mi mathvariant="italic">LN</mi></msub></mtd></mtr><mtr><mtd><msub><mi>e</mi><mi mathvariant="italic">RN</mi></msub></mtd></mtr></mtable></mfenced><mo>=</mo><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>11</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>12</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>1</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>11</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>12</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>1</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>21</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>22</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>2</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>21</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>22</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>2</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><mi mathvariant="normal">M</mi></mtd><mtd><mi mathvariant="normal">M</mi></mtd><mtd><mi mathvariant="normal">M</mi></mtd><mtd><mi mathvariant="normal">M</mi></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">LN</mi><mn>1</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">LN</mi><mn>2</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mi mathvariant="italic">LNM</mi></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">RN</mi><mn>1</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">RN</mi><mn>2</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mi mathvariant="italic">RNM</mi></msub></mtd></mtr></mtable></mfenced><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>s</mi><mn>1</mn></msub></mtd></mtr><mtr><mtd><msub><mi>s</mi><mn>2</mn></msub></mtd></mtr><mtr><mtd><mi mathvariant="normal">M</mi></mtd></mtr><mtr><mtd><msub><mi>s</mi><mi>M</mi></msub></mtd></mtr></mtable></mfenced><mspace width="1ex" /><mi>or</mi><mspace width="1ex" /><mi mathvariant="bold">e</mi><mo>=</mo><mi mathvariant="bold">Hs</mi></math><img file="EP3704875B1_D0009.tif" /></maths>

0020This extension is discussed in <nplcit id="ncit0004" npl-type="s"><text>J. Bauck and D. Cooper, "Generalized Transaural Stereo and Applications", Journal of the Audio Engineering Society, September 1996, Vol. 44, No. 9, pp. 683-705</text></nplcit> along with a proposed solution. In general, M, the number of speakers, and 2N, the number of ears, are not equal, and therefore the 2NxM acoustic transmission matrix H is not invertible. As such, Bauck and Cooper propose using the pseudo inverse of H, denoted H+, to generate the speaker signals s according to: <maths id="math0010" num="(10)"><math display="block"><mi mathvariant="bold">s</mi><mo>=</mo><msup><mi mathvariant="bold">H</mi><mo>+</mo></msup><mi mathvariant="bold">b</mi></math><img file="EP3704875B1_D0010.tif" /></maths> where b is the vector of desired left and right binaural signals for each of the N listeners.

0021There are two general cases to obtain a solution for s. In one case, if the number of ears is larger than the number of speakers, 2N>M, then in general no solution for s exists such that the desired binaural signal b is achieved exactly at the ears of the N listeners. In this case, the solution for s in Equation 10 minimizes the squared error between the signal at the ears e and the desired binaural signal b: <maths id="math0011" num="(11)"><math display="block"><msup><mfenced><mi mathvariant="bold">e</mi><mo>−</mo><mi mathvariant="bold">b</mi></mfenced><mo>∗</mo></msup><mfenced><mi mathvariant="bold">e</mi><mo>−</mo><mi mathvariant="bold">b</mi></mfenced><mo>=</mo><msup><mfenced><mi mathvariant="bold">Hs</mi><mo>−</mo><mi mathvariant="bold">b</mi></mfenced><mo>∗</mo></msup><mfenced><mi mathvariant="bold">Hs</mi><mo>−</mo><mi mathvariant="bold">b</mi></mfenced></math><img file="EP3704875B1_D0011.tif" /></maths> where * denotes the Hermitian transpose.

0022In another case, if the number of ears is smaller than the number of speakers, 2N<M, then in general an infinite number of solutions can be found which all result in the error of Equation 11 being zero. In this case, the particular solution defined by Equation 10 achieves the minimum signal energy over this infinite set of solutions.

0023However, in either of these cases above, the solution given by Equation 10 will in general yield a speaker vector s for which all of the individual speaker signals sm contain perceptually significant amounts of energy. In other words, the solution is not sparse across the set of loudspeakers. This lack of sparsity is problematic because the assumed acoustic transmission matrix H is in practice always an approximation to reality, particularly with respect to the listener positions (e.g., listeners tend to move). If this mismatch between model and reality becomes large, then the listeners may hear the perceived location of an audio object ok far from its intended spatial position, particularly if speakers distant from the intended position of the object contain significant amounts of energy.

0024Other spatial audio rendering techniques avoid this problem by, for each audio object being rendered, activating only loudspeakers physically closest to the intended spatial position of that object. Such systems include amplitude panners, and these systems are relatively robust to listener movement. See, e.g., <nplcit id="ncit0005" npl-type="s"><text>V. Pulkki, "Virtual sound source positioning using vector base amplitude panning," Journal of the Audio Engineering Society, vol. 45, no. 6, pp. 456-466, 1997</text></nplcit>; and <patcit id="pcit0002" dnum="US20160212559"><text>U.S. Application Pub. No. 2016/0212559</text></patcit>.

0025<patcit id="pcit0003" dnum="US5862227A"><text>US patent 5862227</text></patcit> describes a method of recording sound for reproduction by a plurality of loudspeakers, or for processing sound for reproduction by a plurality of loudspeakers. In this method some of the reproduced sound appears to a listener to emanate from a virtual source which is spaced from the loudspeakers. A filter means (H) is used either in creating the recording, or in processing the recorded signals for supply to loudspeakers, the filter means (H) being created in a filter design step in which: a) a technique is employed to minimise error between the signals (w) reproduced at the intended position of a listener on playing the recording through the loudspeakers, and desired signals (d) at the intended position, wherein: b) said desired signals (d) to be produced at the listener are defined by signals (or an estimate of the signals) that would be produced at the ears of (or in the region of) the listener in said intended position by a source at the desired position of the virtual source.

0026However, the amplitude panners discussed above do not provide the same flexibility in perceived placement of audio sources afforded by cross-talk cancellation, particularly for speaker setups that do not fully encircle a listener. Given the above problems and lack of solutions, embodiments are directed toward combining the benefits of generalized virtual spatial rendering described by Equation 9 and perceptually beneficial sparsity of speaker activation. There is provided a method of rendering audio, an apparatus for rendering audio and a non-transitory computer readable medium according to the independent claims. The dependent claims refer to preferred embodiments.

0027According to an embodiment, a method of rendering audio is defined in claim 1.

0028The binaural error is a difference between desired binaural signals related to at least one listener position and modeled binaural signals related to the at least one listener position. The binaural error may be zero. The desired binaural signals are defined based on the audio object and the desired perceived position of the audio object. The desired binaural signals may be defined using one of a database of head-related transfer functions (HRTFs) and a parametric model of HRTFs. The modeled binaural signals are defined by modeling a playback of the plurality of rendered signals, through the plurality of loudspeakers having a plurality of nominal loudspeaker positions, based on the at least one listener position. The modeled binaural signals may be defined using one of a database of head-related transfer functions (HRTFs) and a parametric model of HRTFs.

0029The activation penalty associates a cost with assigning signal energy among the plurality of loudspeakers. The activation penalty is a distance penalty, wherein the distance penalty is defined based on the plurality of rendered signals, a plurality of nominal loudspeaker positions for the plurality of loudspeakers, and the desired perceived position of the audio object. The distance penalty may be defined using one of a Cartesian distance and an angular distance.

0030The cost function may be a combination function that is monotonically increasing in both A and B, wherein A corresponds to the binaural error and B corresponds to the activation penalty. The cost function may be one of A+B, AB, eA+B, and eAB.

0031The audio object may be one of a plurality of audio objects, wherein the plurality of audio objects is rendered using the plurality of filters, and wherein each of the plurality of audio objects has an associated desired perceived position.

0032The plurality of loudspeakers may include a first loudspeaker and a second loudspeaker, wherein the first loudspeaker has a nominal position that is a first distance from the desired perceived position of the audio object, and wherein the second loudspeaker has a nominal position that is a second distance from the desired perceived position of the audio object, wherein the first distance is greater than the second distance. The activation penalty is a distance penalty, wherein the distance penalty becomes larger when, for a given overall level of the plurality of rendered signals, more of the given overall level is associated with the first loudspeaker than is associated with the second loudspeaker.

0033The plurality of loudspeakers may have a plurality of nominal loudspeaker positions, wherein each of the plurality of nominal loudspeaker positions is one of a first position and a second position, wherein the first position is an actual loudspeaker position of a corresponding one of the plurality of loudspeakers, and wherein the second position is other than the actual loudspeaker position.

0034One of the plurality of loudspeakers may have a nominal loudspeaker position, wherein the nominal loudspeaker position is derived by expanding one or more physical positions of the plurality of loudspeakers.

0035The plurality of filters may be independent of the audio object. (For example, the filters may be calculated based on one or more potential positions for the audio object, independently of the content of the audio object.) The plurality of filters may be stored as a lookup table indexed by the desired perceived position of the audio object.

0036The plurality of loudspeakers may have a plurality of physical positions, wherein the plurality of physical positions are determined in a setup phase.

0037According to another embodiment, a non-transitory computer readable medium is defined in claim 13.

0038According to another embodiment, an apparatus is defined in claim 14.

0039The apparatus may include similar details to those discussed above regarding the method.

0040The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations.

BRIEF DESCRIPTION OF THE DRAWINGS

0041<ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">FIG. 1</figref> is a block diagram of a loudspeaker system 100.</li><li><figref idref="f0002">FIG. 2A</figref> is a top view of an arrangement 250 of loudspeakers.</li><li><figref idref="f0003">FIG. 2B</figref> is a top view of a loudspeaker system 200.</li><li><figref idref="f0004">FIG. 3</figref> is a block diagram of a rendering system 300.</li><li><figref idref="f0005">FIG. 4A</figref> is a flowchart of a method 400 of rendering audio.</li><li><figref idref="f0006">FIG. 4B</figref> is a block diagram of a rendering system 450.</li><li><figref idref="f0007">FIG. 5</figref> is a top view of a loudspeaker system 500.</li><li><figref idref="f0008">FIG. 6</figref> is a top view of a loudspeaker system 600.</li><li><figref idref="f0009">FIGS. 7A-7B</figref> are top views of loudspeaker arrangements 700 and 702.</li><li><figref idref="f0010">FIG. 8</figref> is a flowchart of a method 800 of determining filters for a loudspeaker arrangement.</li></ul>

DETAILED DESCRIPTION

0042Described herein are techniques for rendering audio. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present invention.

0043In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

0044In this document, the terms "and", "or" and "and/or" are used. Such terms are to be read as having an inclusive meaning. For example, "A and B" may mean at least the following: "both A and B", "at least both A and B". As another example, "A or B" may mean at least the following: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and/or B" may mean at least the following: "A and B", "A or B". When an exclusive-or is intended, such will be specifically noted (e.g., "either A or B", "at most one of A and B").

0045The following description uses the term sweet spot. In general, a sweet spot in acoustics refers to the listening position with respect to two or more loudspeakers, where a listener is capable of hearing the audio mix the way it was intended to be heard by the mixer. For example, the sweet spot for a standard stereo layout is a point equidistant from the two loudspeakers. In general, however, a spatial audio rendering system may be configured through appropriate filtering at the loudspeakers to place the sweet spot at an arbitrary point with respect to a particular configuration of loudspeakers. The sweet spot may be conceptualized as a point, and may be perceived as an area; a listener's perception of the sound is generally the same within the area, and the listener's perception of the sound degrades outside of the area.

0046<figref idref="f0002">FIG. 2A</figref> is a top view of an arrangement 250 of loudspeakers. The arrangement 250 includes an arbitrary number of loudspeakers (shown are three loudspeakers 252, 254 and 256) that are placed in arbitrary positions. Here "arbitrary" means that their numbers or positions need not necessarily be defined by the audio signals to be output. The arrangement 250 may be contrasted with channel-based systems or with rendering systems with defined filters. For example, a 5.1-channel surround system uses six loudspeakers, five of which have defined positions; changing those positions results in changes to the sweet spot of the audio output. As another example, a rendering system with defined filters has filters that are defined according to the positions of the loudspeakers; if the speakers are re-arranged, the filters need to be re-defined, otherwise the sweet spot of the audio output changes.

0047In contrast to many existing systems, embodiments are useful for outputting audio from arbitrary loudspeaker arrangements such as the arrangement 250. However, before discussing a full arbitrary arrangement (see, e.g., <figref idref="f0009">FIGS. 7A-7B</figref>), a more fixed arrangement of <figref idref="f0003">FIG. 2B</figref> is discussed.

0048<figref idref="f0003">FIG. 2B</figref> is a top view of a loudspeaker system 200. The loudspeaker system 200 is in the form factor of a sound bar and includes seven loudspeakers: a center loudspeaker 202, a left front loudspeaker 204, a right front loudspeaker 206, a left side loudspeaker 208, a right side loudspeaker 210, a left upward loudspeaker 212, and a right upward loudspeaker 214. The left front loudspeaker 204 and the right front loudspeaker 206 may be referred to as the front pair; the left side loudspeaker 208 and the right side loudspeaker 210 may be referred to as the side pair; and the left upward loudspeaker 212 and the right upward loudspeaker 214 may be referred to as the upward pair. <patcit id="pcit0004" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit> discusses a similar form factor for virtual rendering of object based audio through binaural rendering of each object followed by panning of the resulting stereo binaural signal between a plurality of cross-talk cancellation circuits feeding a corresponding plurality of speaker pairs. More specifically in <patcit id="pcit0005" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit>, a cross-talk canceller (see <figref idref="f0001">FIG. 1</figref>) is associated with each of the three pairs, and objects meant to be in front of the listener are panned to the front pair, objects meant to be behind the listener are panned to the side pair, and objects meant to be above the listener are panned to the upward pair. (The center loudspeaker 202 is unassociated with a cross-talk canceller.) However, unlike the system described in <patcit id="pcit0006" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit>, the loudspeaker system 200 derives its filters in a different way and is not constrained to operate on a set of one or more loudspeaker pairs, as further detailed below.

0049<figref idref="f0004">FIG. 3</figref> is a block diagram of a rendering system 300. The rendering system 300 may be a component of the loudspeaker system 200 (see <figref idref="f0003">FIG. 2B</figref>). In general, the rendering system 300 receives an input audio signal 302 and generates one or more rendered audio signals 304. (For example, when the rendering system 300 is implemented in the loudspeaker system 200, the rendering system 300 generates seven rendered audio signals 304.) The input audio signal 302 may include audio objects. Each of the rendered audio signals 304 is provided to other components (not shown), such as an amplifier for output by a loudspeaker. The rendering system 300 includes a processor 310 and a memory 312.

0050The processor 310 receives the input audio signal 302 and applies one or more filters to generate the rendered audio signals 304. The processor 310 may execute a computer program that controls its operation. The memory 312 may store the computer program and the filters. The processor 310 may include a digital signal processor (DSP), and the processor 310 and the memory 312 may be implemented as components of a programmable logic device (PLD). The rendering system 300 may include other components that (for brevity) are not shown.

0051As discussed above, each filter is associated with a corresponding one of the rendered audio signals 304. Further details of the filters are provided below.

0052<figref idref="f0005">FIG. 4A</figref> is a flowchart of a method 400 of rendering audio. The method 400 may be implemented by the rendering system 300 (see <figref idref="f0004">FIG. 3</figref>), for example as controlled by one or more computer programs that implement the method. The method 400 may be performed by a device such as the loudspeaker system 200 (see <figref idref="f0003">FIG. 2B</figref>).

0053At 402, a plurality of filters are derived. Each of the filters is associated with a corresponding one of a plurality of loudspeakers. For example, for the loudspeaker system 200, each of the filters may be derived for a corresponding one of the six loudspeakers 204, 206, 208, 210, 212 and 214. The center loudspeaker 202 may also be associated with a filter derived by this method. Deriving the filters includes the sub-steps 404, 406 and 408.

0054At 404, a binaural error for a desired perceived position of an audio object is defined as a function of the filters to be computed. The desired perceived position may be indicated in the metadata of the audio object. (This position is referred to as the "desired perceived position" because the system may not actually achieve this goal precisely.) The binaural error is a difference between desired binaural signals related to at least one listener position and modeled binaural signals related to the at least one listener position. The desired binaural signals are defined based on the audio object and the desired perceived position of the audio object, from the perspective of the at least one listener position. The modeled binaural signals are defined by modeling a playback of the plurality of rendered signals, through the plurality of loudspeakers having a plurality of loudspeaker positions, based on the at least one listener position.

0055At 406, an activation penalty for the audio object is defined based on the plurality of rendered signals. The activation penalty may be based on the desired perceived position of the audio object or on other components, as discussed below. In general, the activation penalty associates a cost with assigning signal energy to the various loudspeakers and imparts a degree of sparsity to the filter derivation process. One example implementation of the activation penalty is a distance penalty. The distance penalty for the audio object is defined based on the plurality of rendered signals, a plurality of nominal loudspeaker positions for the plurality of loudspeakers, and the desired perceived position of the audio object. The distance penalty is defined such that it becomes larger when, for a given overall level of the plurality of rendered signals, more of the given overall level is associated with a first loudspeaker whose nominal position is further, than a second loudspeaker, from the desired perceived position. (The "nominal" positions of the loudspeakers are further discussed below; unless otherwise noted, the nominal position of a loudspeaker may be considered to relate to its physical position.) For example, using the loudspeaker system 250 (see <figref idref="f0002">FIG. 2A</figref>), when point 270 corresponds to the desired perceived position of the audio object, the loudspeaker 256 is closest, the loudspeaker 254 is next closest, and the loudspeaker 252 is furthest. Thus, the distance penalty is larger when more of the overall level of the rendered signal at the point 270 is associated with the loudspeaker 252 than with the loudspeaker 256. Furthermore, the loudspeaker 254 may have a distance penalty less than that of the loudspeaker 252 and greater than that of the loudspeaker 256.

0056Another example component of the activation penalty is an audibility penalty. In general, the audibility penalty applies a higher cost to nominal loudspeaker positions based on their relation to a defined position. For example, if the loudspeakers are in one room that is adjacent to a baby's room, the audibility penalty may apply a higher cost to the loudspeakers nearby the baby's room.

0057At 408, a cost function that is a combination of the binaural error and the activation penalty for the plurality of filters is minimized. The cost function is a combination function that is monotonically increasing in both A and B, wherein A corresponds to the binaural error and B corresponds to the activation penalty. Examples of such a cost function include A+B, AB, eA+B, and eAB.

0058(Often, the minimization of the cost function may be implemented using a closed-form mathematical solution, as further discussed below. Thus, the binaural error and the activation penalty are discussed above as being "defined" and not "calculated". However, when a closed-form solution is not available, the cost function may be minimized using iteration of the binaural error and the activation penalty, which may involve the explicit calculation thereof.)

0059As an example, the processor 310 (see <figref idref="f0004">FIG. 3</figref>) may derive the filters (see 402) by defining the binaural error of the desired perceived position of an audio object in the input audio signal 302 (see 404), defining the activation penalty for the audio object (see 406), and minimizing the cost function (see 408).

0060At 410, the audio object is rendered using the plurality of filters to generate a plurality of rendered signals. For example, the processor 310 (see <figref idref="f0004">FIG. 3</figref>) may generate the rendered signals 304 by rendering the audio object using the filters.

0061At 412, the plurality of rendered signals are output by the plurality of loudspeakers. For example, the loudspeaker system 200 (see <figref idref="f0003">FIG. 2B</figref>) may output the rendered signals 304 (see <figref idref="f0004">FIG. 3</figref>) using the loudspeakers 204, 206, 208, 210, 212 and 214. The output from each loudspeaker is generally an audible sound.

0062The filter derivation (see 402) may be performed using dynamic filter derivation, precomputed filter derivation, or a combination of the two.

0063In the dynamic case, the processor (see 310 in <figref idref="f0004">FIG. 3</figref>) receives an audio object that includes the desired perceived position information, then derives the filter based on the received desired perceived position information. In the precomputed case, the processor derives a number of filters for a variety of different perceived positions, and stores the filters in the memory (see 312 in <figref idref="f0004">FIG. 3</figref>, for example in a lookup table); when an audio object is received, the processor uses the desired perceived position information in the audio object to select the appropriate filter to use for that audio object. In the combination case, the processor selectively operates as per the dynamic case or the precomputed case based on various criteria, such as the closeness of the desired perceived position information in the audio object to that in the precomputed filters, the availability of computational resources, etc. The choice between the three cases may be made depending upon design criteria. For example, when the system has computational resources available, the system implements the dynamic case.

0064The filter derivation (see 402) may be performed locally, remotely, or a combination of the two. For local filter derivation, the rendering system (e.g., the rendering system 300 of <figref idref="f0004">FIG. 3</figref>) itself derives the filters. For remote filter derivation, the rendering system communicates with remote components (e.g., a cloud-based filter derivation machine) to derive the filters. For example, the local rendering system may run a calibration script and may send the raw data (e.g., relating to speaker positions) to the cloud machine. In the cloud, the position of the speakers is determined and subsequently the rendering filters as well. The lookup table of rendering filters is then sent back down to the rendering system, where they are applied during real-time playback.

0065Although one audio object is discussed above in relation to <figref idref="f0005">FIG. 4A</figref>, the method 400 may also be used for a plurality of audio objects that are received (e.g., via the input audio signal 302 of <figref idref="f0004">FIG. 3</figref>. <figref idref="f0006">FIG. 4B</figref> provides more details for the multiple audio objects case.

0066<figref idref="f0006">FIG. 4B</figref> is a block diagram of a rendering system 450. The rendering system 450 generally performs the method 400 (see <figref idref="f0005">FIG. 4A</figref>), and may be implemented by a processor and a memory (e.g., as in the rendering system 300 of <figref idref="f0004">FIG. 3</figref>). The rendering system 450 includes a number of renderers 452 (two shown, 452a and 452b) and a combiner 454.

0067The number of renderers 452 generally corresponds to the number of audio objects to be rendered at a given time. Here, two renderers 452 are shown; the renderer 452a receives an audio object 460a, and the renderer 452b receives an audio object 460b. Each of the renderers 452 renders the audio object using the appropriate filters (e.g., as derived according to 402 in <figref idref="f0005">FIG. 4A</figref>) to generate one or more rendered signals 462. Here, the renderer 452a renders the audio object 460a to generate the one or more rendered signals 462a, and the renderer 452b renders the audio object 460b to generate the one or more rendered signals 462b. Each of the rendered signals 462 corresponds to one of the loudspeakers (not shown) that are to output the rendered signals 462. For example, when the rendering system 405 is implemented in the loudspeaker system 200 (see <figref idref="f0002 f0003">FIG. 2</figref>), the rendered signals (e.g., 462a) correspond to each of the signals to be output from the six loudspeakers.

0068The combiner 454 receives the rendered signals 462 from the renderers 452 and combines the respective rendered signal for each loudspeaker, to result in one or more rendered signals 464. Generally, the combiner 454 sums the contribution of each of the renderers 452 for each respective one of the rendered signals 462 for a given one of the loudspeakers. For example, if the audio object 460a is rendered to be output by the loudspeakers 208 and 204 (see <figref idref="f0002 f0003">FIG. 2</figref>), and the audio object 460b is rendered to be output by the loudspeakers 204 and 206, then the combiner combines the rendered signals 462a and 462b such that the component signals corresponding to the loudspeaker 204 are summed.

0069The rendered signals 464 may then be output (see 412 in <figref idref="f0005">FIG. 4A</figref>).

0070Further details of the filters (see 402), including the binaural error (see 404), the activation penalty (see 406), and the cost function (see 408) are provided below.

Detailed Embodiments

0071In general, embodiments are directed toward rendering a set of one or more audio object signals, each with an associated and possibly time-varying desired perceived position, for intended playback over a set of two or more loudspeakers located at assumed physical positions. The rendering for each audio object signal is achieved through filtering the audio object signal with one or more filters, where each filter is associated with one of the set of loudspeakers. The filters are derived, at least in part, by minimizing a combination of two components. The first component is an error between (a) desired binaural signals at a set of assumed one or more physical listening positions, said desired signals derived from said audio object signal and its associated desired perceived position and (b) a model of binaural signals generated at the set of one or more listening positions by the set of loudspeakers. The model of binaural signals is derived from the rendered signals (also referred to as the set of filtered audio object signals). The second component is an activation penalty that is a function of the filtered audio signals. A specific example of the activation penalty is a distance penalty that is a function of (a) the filtered audio object signals, (b) the desired perceived audio object signal position, and (c) a set of nominal speaker positions associated with the set of speakers. The distance penalty becomes larger when, for the same amount of overall filtered object audio signal level, more signal level is present in speakers whose nominal position is further from the desired perceived audio object position.

0072For the purposes of the remaining description, the following terms are defined: <tables id="tabl0001" num="0001"><table frame="all"><title>TABLE 1</title><tgroup cols="2"><colspec colnum="1" colname="col1" colwidth="20mm" align="center" /><colspec colnum="2" colname="col2" colwidth="143mm" /><thead valign="top"><row><entry>Term</entry><entry align="center">Definition</entry></row></thead><tbody><row><entry>K</entry><entry>number of audio object signals, where K ≥ 1</entry></row><row><entry>M</entry><entry>number of loudspeakers, where M ≥ 2</entry></row><row><entry>N</entry><entry>number of listeners, where N ≥ 1</entry></row><row><entry>ok</entry><entry>the kth audio object signal out of K</entry></row><row><entry>sm</entry><entry>the mth loudspeaker signal out of M</entry></row><row><entry>eLn</entry><entry>the modelled signal at the left ear of nth listener out of N</entry></row><row><entry>eRn</entry><entry>the modelled signal at the right ear of the nth listener out of N</entry></row><row><entry>pos(ok)</entry><entry>desired perceived position of the kth audio object signal</entry></row><row><entry>pos(sm)</entry><entry>assumed physical position of the mth loudspeaker</entry></row><row><entry>npos(sm)</entry><entry>nominal position of the mth loudspeaker</entry></row><row><entry>pos(en)</entry><entry>assumed physical position of the nth listener</entry></row><row><entry>sk</entry><entry>the Mx1 vector of loudspeaker signals sm associated with the kth audio object</entry></row><row><entry>ek</entry><entry>the 2Nx1 vector of modelled listener binaural signals eLn and eRn associated with the kth audio object</entry></row><row><entry>bk</entry><entry>the 2Nx1 vector of desired listener binaural signals associated with the kth audio object</entry></row><row><entry>Rk</entry><entry>the Mx1 vector of rendering filters associated with the kth audio object</entry></row></tbody></tgroup></table></tables>

0073The loudspeaker signals associated with the kth audio object are given by the rendering filters applied to the object: <maths id="math0012" num="(12)"><math display="block"><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub><mo>=</mo><msub><mi mathvariant="bold">R</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0012.tif" /></maths>

0074The output of the renderer is given by the sum of all the individual object speaker signals <maths id="math0013" num="(13)"><math display="block"><mi mathvariant="bold">s</mi><mo>=</mo><mstyle displaystyle="true"><munderover><mo>∑</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><mrow><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub><mo>=</mo><mstyle displaystyle="true"><munderover><mo>∑</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><mrow><msub><mi mathvariant="bold">R</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub></mrow></mstyle></mrow></mstyle></math><img file="EP3704875B1_D0013.tif" /></maths>

0075For example, Equation 13 corresponds to the one or more rendered signals 464 (see <figref idref="f0006">FIG. 4B</figref>), which is the sum of the rendered signals 462 for all of the individually rendered objects 460.

0076One goal of embodiments is to compute the set of rendering filters Rk for each audio object such that a desired binaural signal bk is approximately produced at the set of L listeners while at the same time ensuring that the set of speaker signals associated with that object, the filtered audio object signals Rkok, is sparse. In particular, the solution should favor the activation of speakers whose nominal positions npos(sm) are close to the desired position of the audio object signal pos(ok).

0077The optimal set of rendering filters R̂k is achieved by minimizing, with respect to Rk, a cost function E consisting of a combination of a binaural error and an activation penalty: <maths id="math0014" num="(14a)"><math display="block"><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>k</mi></msub><mo>=</mo><munder><mi>min</mi><msub><mi mathvariant="bold">R</mi><mi>k</mi></msub></munder><mfenced open="{" close="}"><mi>E</mi><mfenced><msub><mi mathvariant="bold">R</mi><mi>k</mi></msub></mfenced></mfenced><mo>,</mo><mi>where</mi></math><img file="EP3704875B1_D0014.tif" /></maths><maths id="math0015" num="(14b)"><math display="block"><mi>E</mi><mfenced><msub><mi mathvariant="bold">R</mi><mi>k</mi></msub></mfenced><mo>=</mo><mi mathvariant="italic">comb</mi><mfenced open="{" close="}"><msub><mi>E</mi><mi mathvariant="italic">binaural</mi></msub><mfenced><msub><mi mathvariant="bold">b</mi><mi>k</mi></msub><msub><mi mathvariant="bold">e</mi><mi>k</mi></msub></mfenced><mo>,</mo><msub><mi>E</mi><mi mathvariant="italic">activation</mi></msub><mfenced><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub></mfenced></mfenced></math><img file="EP3704875B1_D0015.tif" /></maths>

0078The function comb{A, B} is meant to represent a generic combination function which is monotonically increasing in both A and B. Examples of such a function include A + B, AB, eA+B , eAB, etc.

0079The binaural error function Ebinaural(bk,ek) computes an error between desired binaural signals bk at the listeners' ears and modelled binaural signals ek at the listeners' ears. The desired binaural signals bk are computed from the object signal ok and its associated desired perceived position pos(ok). The modelled binaural signals ek are computed by modeling the playback of the filtered audio object signals Rkok through the M loudspeakers from their assumed physical positions pos(sm) to the N listeners at their assumed physical positions pos(en).

0080The activation penalty Eactivation(sk) computes a penalty based on the filtered object signals sk. It is defined such that the function becomes large when significant amounts of signal level exists in speakers that are deemed undesirable for playback. The notion of "undesirable" may be defined in a variety of ways and may involve the combination of a variety of different criteria. For example, the activation penalty might be defined so that speakers distant from the desired position of the audio object being rendered are considered undesirably (e.g., a distance penalty), while at the same time speakers audible at a particular physical location, such as a baby's room, are undesirable (e.g., an audibility penalty).

0081One particularly useful embodiment of the activation penalty is a distance penalty Edis tan ce(sk,npos(sm),pos(ok)) that defines a combined measure of the filtered object signals sk, the nominal position of each speaker npos(sm), and the desired audio object position pos(ok). The distance penalty has the property that for the same amount of overall filtered object signal level, where overall means combining across all speakers, the penalty increases when more of that energy is concentrated in speakers whose nominal position is more distant from the desired audio object position. In other words, the penalty is small when the majority of signal level is concentrated in speakers closer to the desired object position. The penalty is large when signal energy is concentrated in speakers further from the desired object position. The exact measure of "level" is not critical, but in general should correlate roughly to perceived loudness. Examples include root mean square (rms) level, weighted rms level, etc. Similarly, the exact measure of distance used to specify "closer" and "further" is not critical but should correlate roughly to spatial discrimination of audio. Examples include Cartesian distance and angular distance. The nominal positions of the loudspeakers npos(sm) used in the distance penalty may be set equal to the actual assumed physical locations of the speakers pos(sm), but this is not a requirement. In some cases, as will be discussed later, it is useful to derive alternative nominal positions from the physical positions in order to affect the activation of speakers in a more diverse manner. Maintaining this separation allows such flexibility.

0082In summary of the general relation described by Equations 14, it is the addition of the activation penalty to the binaural error term which yields solutions to the generalized virtual spatial rendering system that are sparse in a perceptually beneficial manner and differentiate embodiments from the existing solutions discussed in the Background.

0083Similar to what is presented in the Background, the desired binaural signals bk may be generated by applying a set of binaural filters to the object signal ok : <maths id="math0016" num="(15)"><math display="block"><msub><mi mathvariant="bold">b</mi><mi>k</mi></msub><mo>=</mo><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub><mo>,</mo></math><img file="EP3704875B1_D0016.tif" /></maths>

0084In the above equation, Bk is a 2Nxl vector of left and right binaural filter pairs. Though not required, it is convenient to set the filter pairs the same for all N listeners: <maths id="math0017" num="(16)"><math display="block"><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub><mo>=</mo><mfenced open="[" close="]"><mtable columnalign="right"><mtr><mtd><msub><mi>B</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>B</mi><mi>R</mi></msub></mtd></mtr><mtr><mtd><msub><mi>B</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>B</mi><mi>R</mi></msub></mtd></mtr><mtr><mtd><mi mathvariant="normal">M</mi></mtd></mtr><mtr><mtd><msub><mi>B</mi><mi>L</mi></msub></mtd></mtr><mtr><mtd><msub><mi>B</mi><mi>R</mi></msub></mtd></mtr></mtable></mfenced></math><img file="EP3704875B1_D0017.tif" /></maths>

0085This implies that we desire each of the N listeners to perceive the same binauralized version of ok. The binaural filter pair may be chosen from an HRTF set indexed by the desired position of the audio object: <maths id="math0018" num="(17)"><math display="block"><mfenced><msub><mi>B</mi><mi>L</mi></msub><msub><mi>B</mi><mi>R</mi></msub></mfenced><mo>=</mo><mi mathvariant="italic">HRTF</mi><mfenced open="{" close="}"><mi mathvariant="italic">pos</mi><mfenced><msub><mi>o</mi><mi>k</mi></msub></mfenced></mfenced></math><img file="EP3704875B1_D0018.tif" /></maths>

0086The modelled binaural signal at the ears may be computed using the generalized acoustic transmission matrix defined in Equation 9: <maths id="math0019" num="(18)"><math display="block"><mtable><mtr><mtd><mrow><msub><mi mathvariant="bold">e</mi><mi>k</mi></msub><mo>=</mo><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>11</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>12</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>1</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>11</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>12</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>1</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>21</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>22</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>L</mi><mn>2</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>21</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>22</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mrow><mi>R</mi><mn>2</mn><mi>M</mi></mrow></msub></mtd></mtr><mtr><mtd><mi mathvariant="normal">M</mi></mtd><mtd><mi mathvariant="normal">M</mi></mtd><mtd><mi mathvariant="normal">M</mi></mtd><mtd><mi mathvariant="normal">M</mi></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">LN</mi><mn>1</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">LN</mi><mn>2</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mi mathvariant="italic">LNM</mi></msub></mtd></mtr><mtr><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">RN</mi><mn>1</mn></mrow></msub></mtd><mtd><msub><mi>H</mi><mrow><mi mathvariant="italic">RN</mi><mn>2</mn></mrow></msub></mtd><mtd><mi mathvariant="normal">Λ</mi></mtd><mtd><msub><mi>H</mi><mi mathvariant="italic">RNM</mi></msub></mtd></mtr></mtable></mfenced><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub></mrow></mtd><mtd><mi>or</mi></mtd><mtd><mrow><msub><mi mathvariant="bold">e</mi><mi>k</mi></msub><mo>=</mo><msub><mi mathvariant="bold">Hs</mi><mi>k</mi></msub><mo>=</mo><msub><mi mathvariant="bold">HR</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub></mrow></mtd></mtr></mtable></math><img file="EP3704875B1_D0019.tif" /></maths>

0087Though not required, the elements of the matrix H may be chosen from the same HRTF set used to create the desired binaural signal, but now indexed by both the assumed physical listener position and the assumed physical speaker position: <maths id="math0020" num="(19)"><math display="block"><mfenced><msub><mi>H</mi><mi mathvariant="italic">Lnm</mi></msub><msub><mi>H</mi><mi mathvariant="italic">Rnm</mi></msub></mfenced><mo>=</mo><mi mathvariant="italic">HRTF</mi><mfenced open="{" close="}"><mi mathvariant="italic">pos</mi><mfenced><msub><mi>e</mi><mi>n</mi></msub></mfenced><mo>,</mo><mi mathvariant="italic">pos</mi><mfenced><msub><mi>s</mi><mi>m</mi></msub></mfenced></mfenced></math><img file="EP3704875B1_D0020.tif" /></maths>

0088In many cases, an HRTF set will be listener-centered, and therefore the position of the speaker may be computed relative to that of the listener in order to compute a single index into the set, as in Equation 17.

0089With the desired binaural signal and the modeled binaural signal now specified, it is convenient to define the binaural error term of the cost function in Equation 14b as the squared error between desired and modeled signals: <maths id="math0021" num="(20)"><math display="block"><msub><mi>E</mi><mi mathvariant="italic">binaural</mi></msub><mfenced><msub><mi>b</mi><mi>k</mi></msub><msub><mi>e</mi><mi>k</mi></msub></mfenced><mo>=</mo><mfenced><msub><mi mathvariant="bold">e</mi><mi>k</mi></msub><mo>−</mo><msub><mi>b</mi><mi>k</mi></msub></mfenced><mo>*</mo><mfenced><msub><mi>e</mi><mi>k</mi></msub><mo>−</mo><msub><mi>b</mi><mi>k</mi></msub></mfenced><mo>=</mo><mfenced><msub><mi mathvariant="italic">Hs</mi><mi>k</mi></msub><mo>−</mo><msub><mi>b</mi><mi>k</mi></msub></mfenced><mo>*</mo><mfenced><msub><mi mathvariant="italic">Hs</mi><mi>k</mi></msub><mo>−</mo><msub><mi>b</mi><mi>k</mi></msub></mfenced></math><img file="EP3704875B1_D0021.tif" /></maths>

0090A convenient, yet still very flexible, definition of the activation penalty is a weighted sum of the power of the filtered object audio signal: <maths id="math0022" num="(21a)"><math display="block"><msub><mi>E</mi><mi mathvariant="italic">activation</mi></msub><mfenced><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub></mfenced><mo>=</mo><msub><mi>s</mi><mi>k</mi></msub><mo>*</mo><msub><mi>W</mi><mi>k</mi></msub><msub><mi>s</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0022.tif" /></maths> where <maths id="math0023" num="(21b)"><math display="block"><msub><mi>W</mi><mi>k</mi></msub><mo>=</mo><mfenced open="[" close="]"><mtable><mtr><mtd><msub><mi>w</mi><mn>1</mn></msub></mtd><mtd><mrow /></mtd><mtd><mrow /></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mrow /></mtd><mtd><msub><mi>w</mi><mn>2</mn></msub></mtd><mtd><mrow /></mtd><mtd><mrow /></mtd></mtr><mtr><mtd><mrow /></mtd><mtd><mrow /></mtd><mtd><mi mathvariant="normal">O</mi></mtd><mtd><mrow /></mtd></mtr><mtr><mtd><mn>0</mn></mtd><mtd><mrow /></mtd><mtd><mrow /></mtd><mtd><msub><mi>w</mi><mi>M</mi></msub></mtd></mtr></mtable></mfenced><mo>,</mo><mspace width="1ex" /><mspace width="1ex" /><msub><mi>w</mi><mi>m</mi></msub><mo>=</mo><mi>Penalty</mi><mfenced open="{" close="}"><msub><mi>o</mi><mi>k</mi></msub><msub><mi>s</mi><mi>m</mi></msub></mfenced></math><img file="EP3704875B1_D0023.tif" /></maths>

0091The weight wm = Penalty{ok,sm} defines the penalty of activating speaker m with signal from audio object k. In general, this penalty may be the combination of a variety of different terms, each aimed at achieving a different perceptual goal. For the distance penalty described above, the weight wm may be defined as: <maths id="math0024" num="(21c)"><math display="block"><msub><mi>w</mi><mi>m</mi></msub><mo>=</mo><mi>Distance</mi><mfenced open="{" close="}"><mi mathvariant="italic">pos</mi><mfenced><msub><mi>o</mi><mi>k</mi></msub></mfenced><mo>,</mo><mi mathvariant="italic">npos</mi><mfenced><msub><mi>s</mi><mi>m</mi></msub></mfenced></mfenced></math><img file="EP3704875B1_D0024.tif" /></maths>

0092In the above equation, Distance{pos(ok),npos(sm)} is the distance between the desired object position and the nominal position of the speaker. A variety of functions for distance may be used. Cartesian distance, assuming an (x,y,z) positional representation of the object and speaker positions, produces reasonable results. However, given that HRTF sets are more often represented with polar coordinates, an angular distance may be more appropriate in some embodiments.

0093In the case where we simultaneously wish to penalize speakers audible in the baby's room (as discussed above regarding the audibility penalty), the weight wm may be defined to include an additional term: <maths id="math0025" num="(21d)"><math display="block"><msub><mi>w</mi><mi>m</mi></msub><mo>=</mo><mi>Distance</mi><mfenced open="{" close="}"><mi mathvariant="italic">pos</mi><mfenced><msub><mi>o</mi><mi>k</mi></msub></mfenced><mo>,</mo><mi mathvariant="italic">npos</mi><mfenced><msub><mi>s</mi><mi>m</mi></msub></mfenced></mfenced><mo>+</mo><mi>Aud</mi><mfenced open="{" close="}"><mi mathvariant="italic">baby</mi><msub><mi>s</mi><mi>m</mi></msub></mfenced></math><img file="EP3704875B1_D0025.tif" /></maths>

0094Here, Aud{baby,sm} defines some measure of audibility of speaker m in the baby's room. For example, the inverse of the distance of speaker m to the baby's room could be used as a proxy for audibility.

0095The virtualization techniques described herein may break down and become perceptually unstable at higher frequencies where the audio wavelength becomes very small in comparison to the physical spacing between speakers. As such, it is typical to band-limit systems using cross-talk cancellation and employ some other rendering technique, such as amplitude panning, above the cutoff. In such a hybrid approach for the present invention it is desirable to harmonize the activation of speakers between the high and low frequencies. One way to achieve this is to define the activation penalty in terms of the panning gains derived by the amplitude panner operating in the higher frequency range. In other words, penalize the activation of speakers that have not been activated by the amplitude panner. In such a system, the activation penalty weights may be defined as <maths id="math0026" num="(21e)"><math display="block"><msub><mi>w</mi><mi>m</mi></msub><mo>=</mo><mfrac><mn>1</mn><mrow><mi>Pan</mi><mfenced open="{" close="}"><msub><mi>o</mi><mi>k</mi></msub><msub><mi>s</mi><mi>m</mi></msub></mfenced><mo>+</mo><mi>ε</mi></mrow></mfrac></math><img file="EP3704875B1_D0026.tif" /></maths> where Pan{ok,sk} is the panning gain at higher frequencies for object k into speaker m, and epsilon is a small regularization term to prevent dividing by zero. <patcit id="pcit0007" dnum="US9712939B"><text>U.S. Patent No. 9,712,939</text></patcit> describes an amplitude panning technique called Center of Mass Amplitude (CMAP), which utilizes a distance penalty similar to Equations 21a-c. As such, the gains of the CMAP panner may be utilized in Equation 21e as another embodiment of the distance penalty defined herein.

0096With both elements of the cost function defined, it is convenient to define their combination as a simple sum: <maths id="math0027" num="(22)"><math display="block"><mi>E</mi><mfenced><msub><mi>R</mi><mi>k</mi></msub></mfenced><mo>=</mo><msub><mi>E</mi><mi mathvariant="italic">binaural</mi></msub><mfenced /><mo>+</mo><msub><mi>E</mi><mi mathvariant="italic">activation</mi></msub><mfenced /><mo>=</mo><mfenced><mi>H</mi><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub><mo>−</mo><msub><mi>b</mi><mi>k</mi></msub></mfenced><mo>*</mo><mfenced><msub><mi mathvariant="italic">Hs</mi><mi>k</mi></msub><mo>−</mo><msub><mi>b</mi><mi>k</mi></msub></mfenced><mo>+</mo><msub><mi>s</mi><mi>k</mi></msub><mo>*</mo><msub><mi>W</mi><mi>k</mi></msub><msub><mi>s</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0027.tif" /></maths>

0097With the overall cost function thusly defined, the goal is to next find the optimal rendering filters R̂k which minimize the function. Realizing that sk = Rkok, one may differentiate the expression in Equation 22 with respect to sk and set to zero. Doing so results in the following solution for sk<maths id="math0028" num="(23)"><math display="block"><mfrac><mrow><mo>∂</mo><mi>E</mi></mrow><mrow><mo>∂</mo><msub><mi>s</mi><mi>k</mi></msub></mrow></mfrac><mo>=</mo><mn>0</mn><mo>⇒</mo><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub><mo>=</mo><msup><mfenced><mi>H</mi><mo>*</mo><mi>H</mi><mo>+</mo><mi>W</mi></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi>H</mi><mo>*</mo><msub><mi>b</mi><mi>k</mi></msub><mo>=</mo><msup><mfenced><mi>H</mi><mo>*</mo><mi>H</mi><mo>+</mo><mi>W</mi></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi>H</mi><mo>*</mo><msub><mi>B</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0028.tif" /></maths>

0098Given that sk = Rkok, the result in Equation 23 implies that the optimal filters are given by <maths id="math0029" num="(24)"><math display="block"><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>k</mi></msub><mo>=</mo><msup><mfenced><mi>H</mi><mo>*</mo><mi mathvariant="bold">H</mi><mo>+</mo><mi>W</mi></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi>H</mi><mo>*</mo><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0029.tif" /></maths>

0099In practice, this solution yields reasonable results, but it has the drawback that, in general, it does not result in the binaural error being set to zero when conditions allow it. For example, when 2N ≤ M , there do exist solutions, such as the pseudo-inverse, that will guarantee zero binaural error. However, the addition of the activation penalty in the particular formulation of the cost function in Equation 22 prevents this from happening. In reality, the activation penalty should be scaled carefully in order to minimize the binaural error to a reasonable level while still maintaining meaningful sparsity.

0100For the case where zero binaural error is achievable, 2N ≤ M , an alternate formulation of the cost function based on the theory of Lagrange multipliers may be utilized so that zero binaural error is achieved precisely. At the same time, sparsity is enforced without having to worry about the absolute scaling of the activation penalty. In this formulation, the activation penalty remains the same as in Equations 21, but the binaural error is changed to the difference between the desired and modeled binaural signals pre-multiplied with an unknown vector Lagrange multiplier λ. <maths id="math0030" num="(25)"><math display="block"><msub><mi>E</mi><mi mathvariant="italic">binaural</mi></msub><mfenced /><mo>=</mo><mi>λ∗</mi><mfenced><mi>H</mi><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub><mo>−</mo><msub><mi>b</mi><mi>k</mi></msub></mfenced></math><img file="EP3704875B1_D0030.tif" /></maths>

0101The binaural error and activation penalty are again combined through simple addition to formulate the overall cost function <maths id="math0031" num="(26)"><math display="block"><mi>E</mi><mfenced /><mo>=</mo><mi mathvariant="normal">λ</mi><mo>*</mo><mfenced><msub><mi mathvariant="bold">Hs</mi><mi>k</mi></msub><mo>−</mo><msub><mi mathvariant="bold">b</mi><mi>k</mi></msub></mfenced><mo>+</mo><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub><mo>*</mo><msub><mi mathvariant="bold">W</mi><mi>k</mi></msub><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0031.tif" /></maths>

0102Setting the partial derivatives of the cost function with respect to both sk and λ to zero yields the unique solution for sk that minimizes the activation penalty subject to zero binaural error <maths id="math0032" num="(27)"><math display="block"><mrow><mtable><mtr><mtd><mrow><mfrac><mrow><mo>∂</mo><mi>E</mi></mrow><mrow><mo>∂</mo><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub></mrow></mfrac><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr><mtd><mrow><mfrac><mrow><mo>∂</mo><mi>E</mi></mrow><mrow><mo>∂</mo><mi mathvariant="normal">λ</mi></mrow></mfrac><mo>=</mo><mn>0</mn></mrow></mtd></mtr></mtable><mo>}</mo></mrow><mo>⇒</mo><msub><mi mathvariant="bold">s</mi><mi>k</mi></msub><mo>=</mo><msup><msub><mi mathvariant="bold">W</mi><mi>k</mi></msub><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="bold">H</mi><mo>*</mo><msup><mfenced><msup><msub><mi mathvariant="bold">HW</mi><mi>k</mi></msub><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="bold">H</mi><mo>*</mo></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup><msub><mi mathvariant="bold">b</mi><mi>k</mi></msub><mo>=</mo><msup><msub><mi mathvariant="bold">W</mi><mi>k</mi></msub><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="bold">H</mi><mo>*</mo><msup><mfenced><msup><msub><mi mathvariant="bold">HW</mi><mi>k</mi></msub><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="bold">H</mi><mo>*</mo></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub><msub><mi>o</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0032.tif" /></maths>

0103Given that sk = Rkok, the result in Equation 27 implies that the optimal filters are given by <maths id="math0033" num="(28)"><math display="block"><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>k</mi></msub><mo>=</mo><msup><msub><mi mathvariant="bold">W</mi><mi>k</mi></msub><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="bold">H</mi><mo>*</mo><msup><mfenced><msup><msub><mi mathvariant="bold">HW</mi><mi>k</mi></msub><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="bold">H</mi><mo>*</mo></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup><msub><mi mathvariant="bold">B</mi><mi>k</mi></msub></math><img file="EP3704875B1_D0033.tif" /></maths>

0104In practice it has been found that designing the disclosed system for more than one listener yields diminishing returns. A good tradeoff for performance and complexity appears to be achieved by assuming a single listener, N=1, and then relying on the sparsity constraint to make the system work reasonably well for listeners who may be located at positions other than the one assumed in the formulation. Since a single listener guarantees 2N ≤ M for M ≥ 2, the solution in Equation 28 can be used and is therefore preferred since it guarantees zero binaural error. It also has the nice property of simplifying exactly to the solution of the standard two speaker cross-talk canceller when M=2 and N=1.

0105As discussed above, <figref idref="f0002">FIG. 2A</figref> shows an arbitrary arrangement 250 of loudspeakers. Embodiments described herein are beneficial for such arbitrary arrangements by virtue of the process of deriving the filters by minimizing the cost function (see 402 in <figref idref="f0005">FIG. 4A</figref>).

0106Also as discussed above, <patcit id="pcit0008" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit> describes a system for virtual audio rendering of object based audio is described wherein a single audio object is panned between multiple sets of traditional 2-speaker / 1-listener crosstalk cancellers as a function of the object's position. The goal of the system in <patcit id="pcit0009" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit> is similar to that of the presently disclosed embodiments in that the panning is designed to provide a more robust spatial presentation for listeners located out of the sweet spot. However, the system of <patcit id="pcit0010" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit> is restricted to multiple pairs of loudspeakers, and the panning function must be hand tailored to the particular layout of these pairs.

0107Embodiments described herein achieve similar behavior in a much more flexible and elegant manner by simply assigning nominal positions to loudspeakers that are different from their physical positions, as shown with reference to <figref idref="f0007">FIG. 5</figref>.

0108<figref idref="f0007">FIG. 5</figref> is a top view of a loudspeaker system 500. The loudspeaker system 500 is similar to the loudspeaker system 200 (see <figref idref="f0003">FIG. 2B</figref>), and includes the rendering system 300 (see <figref idref="f0004">FIG. 3</figref>) that implements the method 400 (see <figref idref="f0005">FIG. 4A</figref>), as described above. The loudspeaker system 500 also includes a center loudspeaker 502, a left front loudspeaker 504, a right front loudspeaker 506, a left side loudspeaker 508, a right side loudspeaker 510, a left upward loudspeaker 512, and a right upward loudspeaker 514. Differently from the loudspeaker system 200, the loudspeaker system 500 assigns the left side loudspeaker 508 to a nominal position 528 and the right side loudspeaker 510 to a nominal position 530, both behind the listener. Similarly, nominal positions for the top pair may be assigned to locations above the listener. Nominal positions for the front pair may be set equal to their physical positions. Using this configuration, the activation penalty (e.g., the distance penalty) of the embodiments described herein will result in speaker activations similar to those described in <patcit id="pcit0011" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit>, but without the crafting of any rules specific to the layout. Instead, loudspeakers will automatically be activated when the position of an object is close to the loudspeakers' nominal positions. In addition, because the embodiments described herein are not restricted to multiple pairs of cross-talk cancellers (as described above regarding <patcit id="pcit0012" dnum="US20150245157"><text>U.S. Application Pub. No. 2015/0245157</text></patcit>), the center channel may be integrated directly into the task of designing the optimal rendering filters, and no special consideration is required.

0109The nominal position of a loudspeaker may be derived by expanding one or more physical positions of the loudspeakers into an arrangement around an assumed physical set of listening positions.

0110<figref idref="f0008">FIG. 6</figref> is a top view of a loudspeaker system 600. The loudspeaker system 600 is similar to the loudspeaker system 500 (see <figref idref="f0007">FIG. 5</figref>), and includes the rendering system 300 (see <figref idref="f0004">FIG. 3</figref>) that implements the method 400 (see <figref idref="f0005">FIG. 4A</figref>), as described above. The loudspeaker system 600 also includes a center loudspeaker 602, a left front loudspeaker 604, a right front loudspeaker 606, a left side loudspeaker 608, a right side loudspeaker 610, a left upward loudspeaker 612, and a right upward loudspeaker 614 in a soundbar form factor. The loudspeaker system 600 also includes a left rear loudspeaker 640 and a right rear loudspeaker 642. The sound bar component of the loudspeaker system 600 may communicate with the rear loudspeakers 640 and 642 via a wired or wireless connection, e.g. to provide the corresponding rendered audio signals 304 (see <figref idref="f0004">FIG. 3</figref>). Similarly to the loudspeaker system 500, the loudspeaker system 600 assigns the left side loudspeaker 608 to a nominal position 628 to the left of the listener, and assigns the right side loudspeaker 610 to a nominal position 630 to the right of the listener.

0111The loudspeaker system 600 illustrates how the embodiments disclosed herein may easily adapt to the presence of additional loudspeakers. Taking the physical positions of the additional loudspeakers 640 and 642 into account, the nominal positions of the side loudspeakers 608 and 610 on the soundbar may be moved to the locations 628 and 630 shown, halfway between the soundbar and the physical rear speakers. In this configuration, as an audio object travels from front to rear, the system will automatically pan its perceived position between the front speakers, the side speakers, and then the rear speakers, all as a consequence of the activation penalty (e.g., the distance penalty) utilized in the optimization of the rendering filters.

0112<figref idref="f0009">FIGS. 7A-7B</figref> are top views of loudspeaker arrangements 700 and 702. Both of the arrangements 700 and 702 include five loudspeakers 710, 712, 714, 716 and 718. The loudspeakers 710, 712, 714, 716 and 718 may also each include a microphone, as described in International Publication No. <patcit id="pcit0013" dnum="WO2018064410A1"><text>WO 2018/064410 A1</text></patcit>. The microphone enables each loudspeaker to determine the positions of the other loudspeakers by detecting the audio output from the other loudspeakers, and to determine the position of listeners by detecting the sounds made by the listeners. Alternatively, the microphones may be discrete devices, separate from the loudspeakers.

0113The difference between <figref idref="f0009">FIG. 7A and 7B</figref> is the different arrangements 700 and 702 for the loudspeakers 710, 712, 714, 716 and 718. For example, the loudspeakers may initially be arranged in the arrangement 700 of <figref idref="f0009">FIG. 7A</figref>, then may be re-arranged into the arrangement 702 of <figref idref="f0009">FIG. 7B</figref>. The embodiments described herein facilitate the arbitrary placement, and arbitrary rearrangement, of the loudspeaker arrangements, as described with reference to <figref idref="f0010">FIG. 8</figref>.

0114<figref idref="f0010">FIG. 8</figref> is a flowchart of a method 800 of determining filters for a loudspeaker arrangement. The method 800 may be implemented by the loudspeakers 710, 712, 714, 716 and 718 (see <figref idref="f0009">FIG.7A and FIG. 7B</figref>), for example by executing one or more computer programs.

0115For the two solutions given by Equations 24 and 28, one notes that the solution for the filters is completely independent of the object signal ok itself. Both solutions depend on the transmission matrix H, the weight matrix Wk, and the binaural filter vector Bk. Combined, these terms are in turn dependent on the desired position of the object pos(ok), the physical position of the listeners pos(en), the physical position of the speakers pos(sm), and the nominal position on the speakers npos(sm). The method 800 operates based on these observations.

0116At 802, the positions of a plurality of loudspeakers are determined. For example, given the arrangement 700 (see <figref idref="f0009">FIG. 7A</figref>), the loudspeakers 710, 712, 714, 716 and 718 may determine their positions by outputting audio and by detecting the outputs received from each other loudspeaker (e.g., by using a microphone). The positions may be relative positions, e.g. based on the position of one of the loudspeakers as a reference position.

0117At 804, the position(s) of one or more listeners is determined. For example, given the arrangement 700 (see <figref idref="f0009">FIG. 7A</figref>), the loudspeakers 710, 712, 714, 716 and 718 may determine the position of the listener by using their microphones. If the loudspeakers detect multiple listeners, they may average their positions into a single listener position, so that the N=1 assumption may be used as discussed above with reference to Equation 28. Alternatively, 804 may be omitted.

0118At 806, a plurality of filters are generated. In general, these filters are generated according to 402 (see <figref idref="f0005">FIG. 4A</figref>), using the loudspeaker positions (see 802) and the listener positions (see 804) as the inputs for the filter equations discussed above. For example, given the arrangement 700 (see <figref idref="f0009">FIG. 7A</figref>), the loudspeakers 710, 712, 714, 716 and 718 may generate the filters using the process 402 (see <figref idref="f0005">FIG. 4A</figref>) and equations described above. When 804 is omitted, the filters may be generated based only on the loudspeaker position information (see 802).

0119At this point, the system may assume that the loudspeaker positions and the listener positions may remain stationary, and may generate the filters as a lookup table of optimal rendering filters indexed by desired position of the audio object. Since these filters are not dependent on the actual object signal being rendered, only its desired position, each of the K object signals may be rendered using this same lookup table.

0120The steps 802, 804 and 806 may be referred to as a configuration phase or a setup phase. The configuration phase may be initiated by the listener, e.g. by pushing a configuration button on one of the loudspeakers, or by providing an audible command that is received by the microphones. After the configuration phase, the process continues with steps 808, 810 and 812, which may be referred to as an operational phase.

0121At 808, an audio object is rendered using the plurality of filters to generate a plurality of rendered signals. This step is generally similar to the step 410 (see <figref idref="f0005">FIG. 4A</figref>) discussed above. For example, given the arrangement 700 (see <figref idref="f0009">FIG. 7A</figref>), the loudspeakers 710, 712, 714, 716 and 718 may receive one or more audio objects and may render the audio object using the filters to generate the plurality of rendered signals.

0122At 810, the plurality of rendered signals is output by the plurality of loudspeakers. This step is generally similar to the step 412 (see <figref idref="f0005">FIG. 4A</figref>) discussed above. For example, given the arrangement 700 (see <figref idref="f0009">FIG. 7A</figref>), the loudspeakers 710, 712, 714, 716 and 718 may each output its respective rendered signal as audible sound.

0123At 812, it is evaluated whether the loudspeaker arrangement is changed. The step 812 may be initiated by a user (e.g., the listener pushes a reconfiguration button, provides a voice command, etc.), may be initiated periodically by the system itself (e.g., performing the evaluation periodically, performing the evaluation continuously by using the microphones to detect the sound output from each other loudspeaker, etc.), etc. If the arrangement has changed, the method returns to 802 and re-determines the positions of the loudspeakers. If the arrangement has not changed, the method continues with the operational phase as per 808. For example, the loudspeakers 710, 712, 714, 716 and 718 may have been in the arrangement 700 (see <figref idref="f0009">FIG. 7A</figref>), may have been changed to the arrangement 702 (see <figref idref="f0009">FIG. 7B</figref>), and may have received a voice command to re-generate the filters; the method then returns to 802.

0124Although the method 800 has been described in the context of rearranging the loudspeakers (e.g., from the arrangement 700 of <figref idref="f0009">FIG. 7A</figref> to the arrangement 702 of <figref idref="f0009">FIG. 7B</figref>), the method 800 may also include adding an additional loudspeaker to the arrangement (which may also include, or not include, rearranging the existing loudspeakers); removing one of the loudspeakers from the arrangement (which may also include, or not include, rearranging the remaining loudspeakers); and re-generating the filters according to changing the listener positions (see 804) without rearranging the loudspeakers (see 802).

Implementation Details

0125An embodiment may be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). Unless otherwise specified, the steps executed by embodiments need not inherently be related to any particular computer or other apparatus, although they may be in certain embodiments. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.

0126Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein. (Software per se and intangible or transitory signals are excluded to the extent that they are unpatentable subject matter.)

0127The above description illustrates various embodiments of the present invention along with examples of how aspects of the present invention may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present invention as defined by the following claims. The matter for which protection is sought is defined uniquely in the appended claims.

Contents3

Every citation, both ways

Citations
Document	Relation	Office
WO2012068174A2	Cites	World Intellectual Property Organization (WIPO)
US5862227A	Cites	United States of America
NELSON P A ET AL: "MULTICHANNEL SIGNAL PROCESSING TECHNIQUES IN THE REPRODUCTION OF SOUND", JOURNAL OF THE AUDIO ENGINEERING SOCIETY, AUDIO ENGINEERING SOCIETY, NEW YORK, NY, US, vol. 44, no. 11, 1 November 1996 (1996-11-01), pages 973-989, XP000687887, ISSN: 1549-4950	Non-patent	–
PARODI YESENIA LACOUTURE ET AL: "Analysis of Design Parameters for Crosstalk Cancellation Filters Applied to Different Loudspeaker Configurations", JAES, AES, 60 EAST 42ND STREET, ROOM 2520 NEW YORK 10165-2520, USA, vol. 59, no. 5, 1 May 2011 (2011-05-01), pages 304-320, XP040567479,	Non-patent	–
LEE JUNHO ET AL: "Robust Crosstalk Cancellation Based on Energy-Based Control", CONFERENCE: 34TH INTERNATIONAL CONFERENCE: NEW TRENDS IN AUDIO FOR MOBILE AND HANDHELD DEVICES; AUGUST 2008, AES, 60 EAST 42ND STREET, ROOM 2520 NEW YORK 10165-2520, USA, 1 August 2008 (2008-08-01), XP040508509,	Non-patent	–

13 members in 4 offices

This recordMember, by publication datePriority claim

Priority claims5

Priority claims
Document	Office	Kind	Date
201762578854P	United States of America	–	2017-10-30
201762578854	United States of America	P	2017-10-30
201862743275P	United States of America	–	2018-10-09
201862743275	United States of America	P	2018-10-09
2018057357	United States of America	W	2018-10-24

Members13

Family members, oldest first
Document	Office	Kind
WO2019089322A1	World Intellectual Property Organization (WIPO)	A1	2019-05-09
CN111295896A	China	A	2020-06-16
EP3704875A1	European Patent Office (EPO)	A1	2020-09-09
US2020351606A1	United States of America	A1	2020-11-05
CN111295896B	China	B	2021-05-18
CN113207078A	China	A	2021-08-03
US11172318B2	United States of America	B2	2021-11-09
US2022070605A1	United States of America	A1	2022-03-03
CN113207078B	China	B	2022-11-22
EP3704875B1This record	European Patent Office (EPO)	B1	2023-05-31
EP4228288A1	European Patent Office (EPO)	A1	2023-08-16
US12035124B2	United States of America	B2	2024-07-09
EP4228288B1	European Patent Office (EPO)	B1	2025-08-06

74 legal events, as 9 offices reported them to INPADOC

Over the term

Point at a mark for the event

Events

Legal events, newest first
	Event	Code	Office
2026-01-08	Annual fee paid to national office [announced via postgrant information from national office to epo]GrantedPGFP	PGFP	EP
2025-12-12	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2025-10-20	Annual fee paid to national office [announced via postgrant information from national office to epo]GrantedPGFP	PGFP	EP
2025-10-16	Annual fee paid to national office [announced via postgrant information from national office to epo]GrantedPGFP	PGFP	EP
2025-08-18	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2025-07-25	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2025-01-10	Annual fee paid to national office [announced via postgrant information from national office to epo]GrantedPGFP	PGFP	EP
2024-11-29	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-11-08	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-10-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-10-10	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-08-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-07-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-07-18	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-06-28	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-06-19	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-06-17	Lapsed because of non-payment of the annual feeLapsedMM	MM	BE
2024-05-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-05-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-05-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-05-31	Patent ceasedCeasedPL	PL	CH
2024-05-08	No opposition filedOpposition26N	26N	EP
2024-05-03	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-04-05	No opposition filed within time limitOppositionORIGINAL CODE: 0009261PLBE	PLBE	EP
2024-04-05	Information on the status of an ep patent application or granted ep patentGrantedSTATUS: NO OPPOSITION FILED WITHIN TIME LIMITSTAA	STAA	EP
2024-03-01	No opposition filed against granted patent, or epo opposition proceedings concluded without decisionGrantedR097	R097	DE
2024-01-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-01-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-01-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-01-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-01-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-01-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-01-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2024-01-16	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-12-29	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-11-30	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-10-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-10-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-10-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-10-31	Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25	PG25	EP
2023-10-15	Deletion acc. to par. 5 (withdrawal of the translation of the ep patent)MK05	MK05	AT
2023-10-04	Patent invalid in the netherlands as no translation has been filedMP	MP	NL
2023-09-25	Invalidation of extension of european patentsMG9D	MG9D	LT
2023-06-21	European patents granted designating irelandGrantedFG4D	FG4D	IE
2023-06-15	Dpma publication of mentioned ep patent grantGrantedR096	R096	DE
2023-06-15	Reference to at number (ep patent validated in austria)REF	REF	AT
2023-06-14	Opt-out of the competence of the unified patent court (upc) registeredP01	P01	EP
2023-05-31	Designated contracting statesAK	AK	EP
2023-05-31	European patent takes effect as a national patent in ch/liEP	EP	CH
2023-05-31	European patent grantedGrantedFG4D	FG4D	GB
2023-04-28	(expected) grantORIGINAL CODE: 0009210GRAA	GRAA	EP
2023-04-28	Information on the status of an ep patent application or granted ep patentGrantedSTATUS: THE PATENT HAS BEEN GRANTEDSTAA	STAA	EP
2023-04-18	Grant fee paidORIGINAL CODE: EPIDOSNIGR3GRAS	GRAS	EP
2023-03-15	Intention to grant announcedINTG	INTG	EP
2023-02-12	Despatch of communication of intention to grant a patentORIGINAL CODE: EPIDOSNIGR1GRAP	GRAP	EP
2023-02-12	Information on the status of an ep patent application or granted ep patentGrantedSTATUS: GRANT OF PATENT IS INTENDEDSTAA	STAA	EP
2021-09-22	First examination report despatched17Q	17Q	EP
2021-08-20	Information on the status of an ep patent application or granted ep patentGrantedSTATUS: EXAMINATION IS IN PROGRESSSTAA	STAA	EP
2021-02-17	Request for validation of the european patent (deleted)DAV	DAV	EP
2021-02-17	Request for extension of the european patent (deleted)DAX	DAX	EP
2020-09-09	Request for examination filed17P	17P	EP
2020-09-09	Designated contracting statesAK	AK	EP
2020-09-09	Request for extension of the european patentAX	AX	EP
2020-08-07	Public reference made under article 153(3) epc to a published international application that has entered the european phaseORIGINAL CODE: 0009012PUAI	PUAI	EP
2020-08-07	Information on the status of an ep patent application or granted ep patentGrantedSTATUS: REQUEST FOR EXAMINATION WAS MADESTAA	STAA	EP
2019-05-11	Information on the status of an ep patent application or granted ep patentGrantedSTATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADESTAA	STAA	EP
2018-11-20	Information on the status of an ep patent application or granted ep patentGrantedSTATUS: UNKNOWNSTAA	STAA	EP