A virtual surround sound device

0001The present invention relates to a virtual surround sound system comprising mixdown means for converting surround sound signal into binaural signals.

0002A known virtual sound reproduction system provides a surround sound effect similar to a Dolby (RTM) 5.1 channel system. However, the virtual sound reproduction system only uses two speakers.

0003Technology related to the virtual sound reproduction system is disclosed in WO-A-99/49574 and WO-A-97/30566.

0004In the known virtual sound reproduction system, a multi-channel audio signal is down mixed to a 2-channel audio signal. The down mixing is done using a far-field head related transfer function (HRTF). The 2-channel audio signal is then digitally filtered using left and right ear transfer functions H1(z) and H2(z) to which a crosstalk cancellation algorithm is applied. The filtered audio signal is then converted into an analog audio signal by a digital-to-analogue converter (DAC). The analogue audio signal is amplified by an amplifier and output to left and right channels, i.e., 2-channel speakers. As the 2-channel audio signal includes 3 dimensional (3D) audio data, a surround sound effect is achieved.

0005However, the known method of reproducing 2-channel virtual sound using a far-field HRTF uses an HRTF that is measured at a point at least 1 m from the center of a users head. Accordingly, known virtual sound technology provides exact sound information for the location where a sound source is placed, however, it cannot determine the sound for locations away from the sound source.

0006Also, since the known technology is developed under the assumption that each speaker has a flat frequency response, when a speaker not having a flat frequency response is used (for example, if the speaker is old), or when the effective frequency response of a speaker is not flat due to the acoustics in the room where the speaker is installed, the virtual sound quality is dramatically reduced. Moreover, in the known technique, if a listener moves away, even slightly, from a "sweet spot zone" located directly between the two speakers, the virtual sound quality is dramatically reduced. Finally, in the known technology, since a crosstalk cancellation algorithm is suited only for a specific speaker arrangement, the crosstalk cancellation in other speaker arrangements is not as effective.

0007The present invention relates to a virtual surround sound system comprising mixdown means for converting surround sound signal into binaural signals.

0008A virtual surround system, according to the present invention, is characterised by cross-talk cancellation means for modifying the output of the mixdown means to cancel acoustic cross-talk between the two channels of the binaural output thereof.

0009Additional preferred and optional features are set forth in claims 2 to 5 appended hereto.

0010An embodiment of the present invention will now be described, by way of example only, and with reference to the accompanying drawings, in which: <ul id="ul0001" list-style="none" compact="compact"><li>Figure 1 illustrates an audio reproduction system according to an embodiment of the invention;</li><li>Figure 2 illustrates a down mixing unit of Figure 1;</li><li>Figure 3 illustrates a method of realizing a transaural filter of a crosstalk cancellation unit of Figure 1;</li><li>Figure 4 illustrates a spatial compensator of Figure 1;</li><li>Figure 5 illustrates a method of spatial compensation performed by the spatial compensation unit of Figure 4;</li><li>Figure 6 illustrates a method of reproducing virtual sounds in an audio reproduction system according to an embodiment of the present general inventive concept;</li><li>Figure 7 illustrates the frequency quality in accordance with turning a room equalizer on/off; and</li><li>Figure 8 illustrates different speaker arrangements.</li></ul>

0011Referring to Figure 1, an audio reproduction system includes a virtual sound reproduction apparatus 100, left and right amplifiers 170 and 175, left and right speakers 180 and 185, and left and right microphones 190 and 195. The virtual sound reproduction apparatus 100 includes a Dolby prologic (RTM) decoder 110, an audio decoder 120, a down mixing unit 130, a crosstalk cancellation unit 140, a spatial compensator 150, and a digital-to-analogue converter (DAC) 160.

0012The Dolby prologic (RTM) decoder 110 decodes an input 2-channel Dolby prologic (RTM) audio signal into 5.1 channel digital audio signals (a left-front channel, a right-front channel, a centre-front channel, a left-surround channel, a right-surround channel, and a low frequency effect channel).

0013The audio decoder 120 decodes an input multi-channel audio bit stream into the 5.1 channel digital audio signals.

0014The down mixing unit 130 down mixes the 5.1 channel digital audio signals into two channel audio signals. The down mixing is achieved by adding directional information using an HRTF to the 5.1 channel digital audio signals output from either the Dolby prologic (RTM) decoder 110 or the audio decoder 120. Here, the direction information is a combination of the HRTFs measured in the near-field and far-field.

0015Referring to Figure 2, 5.1 channel audio signals are input to the down mixing unit 130. The 5.1 channels are the left-front channel 2, the right-front channel, the centre-front channel, the left-surround channel, the right-surround channel, and the low frequency effect channel 13. Left and right impulse response functions are passed on the respective 5.1 channels. Therefore, from the left-front channel 2, a left-front left (LF_L) impulse response function 4 is convolved with a left-front signal in convolver 6. The left-front impulse left (LF_L) response function 4 is an impulse response which is subsequently output from a left-front channel speaker. The left-front channel speaker is placed at a position ideally to be received by the left ear of a user, and is a mixture of the HRTFs measured in the near-field and the far-field. Here, the near-field HRTF is a transfer function measured at a location less than 1m from the centre of a head and the far-field HRTF is a transfer function measured at a location more than 1m from the centre of the head. Convolver 6 generates an output signal 7 which is added to a left channel signal 10 for outputting to the left channel. Similarly, a left-front right (LF_R) impulse response function 5 is convolved with the left-front signal 3 in a further convolver 8. The output convolved signal 9 is added to a right channel signal 11 for outputting from the left-front channel speaker placed at the ideal position for the right ear of the user. The remaining channels of the 5.1 channel audio signal may be similarly convolved and output to the left and right channel signals 10 and 11. Therefore, 12 convolution steps are required to be carried out on the 5.1 channel signals in the down mixing unit 130. Accordingly, even if the 5.1 channel signals are produced as 2 channel signals by merging and down mixing the 5.1 channel signals and the near- and far-field HRTFs, a surround effect similar to when the 5.1 channel signals are reproduced as multi-channel signals is generated.

0016The crosstalk cancellation unit 140 digitally filters the down mixed 2 channel audio signals by applying a crosstalk cancellation algorithm using transaural filter coefficients H₁₁(Z), H₂₁(Z), H₁₂(Z), and H₂₂(Z). The transaural filter coefficients H₁₁(Z), H₂₁(Z), H₁₂(Z), and H₂₂(Z) are set for crosstalk cancellation by using acoustic transfer coefficients C₁₁(Z), C₂₁(Z), C₁₂(Z), and C₂₂(Z) generated by spectrum analysis in the spatial compensator 150.

0017The spatial compensator 150 receives broadband signals output from the left and right speakers 180 and 185 via the left and right microphones 190 and 195. The left and right microphones 190 and 195 are worn by the user on a headset. The user then sits at the location where he would normally sit. The spatial compensator 150 generates the transaural filter coefficients H₁₁(Z), H_d1(Z), H₁₂(Z), and H₂₂(Z) which represent frequency characteristics of the received signals by using the frequency bands. Also, the acoustic transfer coefficients C₁₁(Z), C₂₁(Z), C₁₂(Z), and C₂₂(Z) are generated using spectrum analysis. The spatial compensator 150 thus compensates for the frequency characteristics, such as a signal delay and the signal level between the respective left and right speakers 180 and 185 and a listener, of the 2 channel audio signals output by the crosstalk cancellation unit 140. This is done using the compensation filter coefficients H₁₁(Z), H₂₁(Z), H₁₂(Z), H₂₂(Z). The compensation filter can be an infinite impulse response (IIR) filter or a finite impulse response (FIR) filter.

0018The DAC 160 converts the spatially compensated left and right audio signals into analogue audio signals.

0019The left and right amplifiers 170 and 175 amplify the analogue audio signals converted by the DAC 160 and output these signals to the left and right speakers 180 and 185, respectively.

0020Referring to Figure 3, sound waves y₁(n) and y₂(n) are reproduced at a left ear and a right ear of a listener via two speakers. Sound signals s₁(n) and s₂(n) are input to the two speakers. The acoustic transfer coefficients C₁₁(Z), C₂₁(Z), C₁₂(Z), and C₂₂(Z) are calculated through spectrum analysis performed on the broadband signals.

0021When the listener listens to the sound wave y₁(n) and y₂(n), the listener hears a virtual stereo sound. Since four acoustic paths exist between the two speakers and the two ears, although the two speakers generate the sound waves y₁(n) and y₂(n), respectively, these waves are not the waves at each ear. This is because the part of the sound wave from the left speaker is heard by the right ear and some of the wave from the right speaker is heard by the left ear. Therefore, crosstalk cancellation needs to be performed so that the listener cannot hear a signal reproduced in a left speaker using the right ear and vice versa.

0022A stereophonic reproduction system 320 calculates the acoustic transfer functions C₁₁(Z), C₂₁(Z), C₁₂(Z), and C₂₂(Z) between the two speakers and the two ears of the listener using sound waves received via the two microphones. In the transaural filter 310, transaural filter coefficients H₁₁(Z), H₂₁(Z), H₁₂(Z), and H₂₂(Z) are determined based on these acoustic transfer functions.

0023In the crosstalk cancellation algorithm, the sound waves y₁(n) and y₂(n) are given by Equation 1 and the sound values s₁(n) and s₂(n) are given by Equation 2 below.<maths id="math0001" num=""><math display="block"><mrow><mtext>[Equation 1]</mtext><mspace linebreak="newline" /><msub><mrow><mtext>y</mtext></mrow><mrow><mtext>1</mtext></mrow></msub><msub><mrow><mtext>(n) = C</mtext></mrow><mrow><mtext>11</mtext></mrow></msub><msub><mrow><mtext>(Z)s</mtext></mrow><mrow><mtext>1</mtext></mrow></msub><msub><mrow><mtext>(n) + C</mtext></mrow><mrow><mtext>12</mtext></mrow></msub><msub><mrow><mtext>(Z)s</mtext></mrow><mrow><mtext>2</mtext></mrow></msub><mtext>(n)</mtext><mspace linebreak="newline" /><msub><mrow><mtext>y</mtext></mrow><mrow><mtext>2</mtext></mrow></msub><msub><mrow><mtext>(n) = C</mtext></mrow><mrow><mtext>21</mtext></mrow></msub><msub><mrow><mtext>(Z)</mtext></mrow><mrow><mtext>s1</mtext></mrow></msub><msub><mrow><mtext>(n) + C</mtext></mrow><mrow><mtext>22</mtext></mrow></msub><msub><mrow><mtext>(Z)</mtext></mrow><mrow><mtext>s2</mtext></mrow></msub><mtext>(n)</mtext></mrow></math><img file="EP1545154A2_D0001.tif" /></maths><maths id="math0002" num=""><math display="block"><mrow><mtext>[Equation 2]</mtext><mspace linebreak="newline" /><msub><mrow><mtext> si(n) = H</mtext></mrow><mrow><mtext>11</mtext></mrow></msub><msub><mrow><mtext>(Z)x</mtext></mrow><mrow><mtext>1</mtext></mrow></msub><msub><mrow><mtext>(n) + H</mtext></mrow><mrow><mtext>12</mtext></mrow></msub><msub><mrow><mtext>(Z)x</mtext></mrow><mrow><mtext>2</mtext></mrow></msub><mtext>(n)</mtext><mspace linebreak="newline" /><msub><mrow><mtext> s</mtext></mrow><mrow><mtext>2</mtext></mrow></msub><msub><mrow><mtext>(n) = H</mtext></mrow><mrow><mtext>21</mtext></mrow></msub><msub><mrow><mtext>(Z)</mtext></mrow><mrow><mtext>x1</mtext></mrow></msub><msub><mrow><mtext>(n) + H</mtext></mrow><mrow><mtext>22</mtext></mrow></msub><msub><mrow><mtext>(Z)x</mtext></mrow><mrow><mtext>2</mtext></mrow></msub><mtext>(n)</mtext></mrow></math><img file="EP1545154A2_D0002.tif" /></maths>

0024If a matrix H(Z), given in Equation 4 below, of the transaural filter 310 is the inverse matrix of a matrix C(Z), given by Equation 3 below, of the acoustic transfer functions, the sound waves y₁(n) and y₂(n) are input sound values x₁(n) and x₂(n), respectively. Therefore, if the input sound values x₁(n) and x₂(n) are substituted for the sound values y₁(n) and y₂(n), the sound values s₁(n) and s₂(n) input to the two speakers are as shown in Equation 2, and the listener hears the sound values y₁(n) and y₂(n).<maths id="math0003" num=""><img file="EP1545154A2_D0003.tif" /></maths><maths id="math0004" num=""><img file="EP1545154A2_D0004.tif" /></maths>

0025Referring to Figure 4, a noise generator 412 generates broadband signals or impulse signals. Band pass filters 434, 436, and 438 band pass the broadband signals output from the left and right speakers 180 and 185, into N bands. These broadband signals are received by the left and right microphones 180 185. Level and phase compensators 424, 426, and 428 generate compensation filter coefficients to compensate the levels and phases of the broadband signals band pass filtered by the band pass filters 434, 436, and 438. Boost filters 414, 416, ..., and 418 also compensate the input audio signals so that a flat frequency response across the frequency range is achieved. This is achieved by applying band compensation filter coefficients generated by the level and phase compensators 424, 426, and 428 to the input audio signal. Also, a spectrum analyzer 440 analyzes the spectra of the broadband signals output from the left and right speakers 180 and 185 which is received using the left and right microphones 190 and 195. The transfer functions C₁₁(Z), C₂₁(Z), C₁₂(Z), and C₂₂(Z) between the two speakers 180 and 185 and the two ears of the listener is then calculated.

0026Speaker response characteristics are measured using broadband signals or impulse signals in operation 510.

0027Left and right speaker impulse response characteristics are measured in operation 520.

0028Band pass filtering of the broadband speaker response characteristics for each of the N bands is performed in operation 530.

0029An average energy level of each band is calculated in operation 540.

0030The compensation level of each band is calculated using the calculated average energy levels in operation 550.

0031A boost filter coefficient for each band is set using the calculated band compensation levels in operation 560.

0032Boost filters 414, 416 and 418 are applied to the speaker impulse responses using the set band boost filter coefficients in operation 570.

0033Delays between the left and right channels are measured using the speaker impulse response characteristics in operation 580.

0034Phase compensation coefficients are set using the delays between the left and right channels in operation 590. In other words, delays caused by timing differences between the left and right speakers are compensated for by controlling the delays between the left and right channels.

0035Referring to Figure 6, in operation 610, broadband signals or impulse signals are generated by left and right speakers, i.e., 180 and 185 of Figure 4. The broadband signals or impulse signals are received by left and right microphones, i.e., 190 and 195. Volume levels and signal delays between the left and right speakers 180 and 185 are controlled so that the digital filter coefficients which produce a flat frequency response are set.. Also, optimal transaural filter coefficients H₁₁(Z), H₂₁(Z), H₁₂(Z), and H₂₂(Z) for crosstalk cancellation are set by calculating stereophonic transfer functions between the speakers, 180 and 185 and ears of a listener using the signals picked up by the microphones, 190 and 195.

0036A multi-channel audio signal is down mixed into 2 channel audio signals using near and far-field HRTFs in operation 620.

0037The down mixed audio signals are digitally filtered on the basis of the optimal transaural filter coefficients H₁₁(Z), H₂₁(Z), H₁₂(Z), and H₂₂(Z) and are used for crosstalk cancellation in operation 630.

0038The crosstalk cancelled audio signals are spatially compensated by using reflection level and phase compensation filter coefficients in operation 640.

0039Accordingly, the 2 channel audio signals provide an optimal surround sound effect at the current position of the listener using crosstalk cancellation and spatial compensation.

0040Referring to Figure 7, when a room equalizer is turned on, the frequency response of the speakers is flat.

0041The present general inventive concept can also be embodied as computer readable codes on a computer readable recording medium. The computer readable recording medium may be any data storage device that can store data which can be thereafter read by a computer system. Examples of the computer readable recording medium may include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and carrier waves (such as data transmission through the Internet). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code can be stored and executed in a distributed fashion.

0042As described above, in known technologies, while a surround effect provided by two 5.1 channel speakers is optimal in a "sweet spot" zone, the virtual surround sound effect is dramatically decreased anywhere outside this sweet spot zone. However, since the position of a sweet spot is dynamically controlled to the location of the listener, an optimal 2 channel virtual sound surround effect is provided to the listener. Also, through spatial compensation, a virtual sound effect may be made much better by having a flat frequency response as shown in Figure 7. Also, as shown in Figure 8, the virtual sound effect can be improved dramatically by compensating for changes in speaker arrangement and/or a listener position through crosstalk cancellation using two microphones 190 and 195.

0043Also, the broadband or impulse signals may be output sequentially i.e. left speaker then right speaker or vice versa.