Illumination compensation method and apparatus and video encoding and decoding method and apparatus using the illumination compensation method

CROSS-REFERENCE TO RELATED PATENT APPLICATION

p-0002This application claims priority from Korean Patent Application No. 10-2006-0120949, filed on Dec. 1, 2006, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

BACKGROUND OF THE INVENTION

p-00031. Field of the Invention

p-0004Methods and apparatuses consistent with the present invention relate to illumination compensation, and more particularly, to illumination compensation for video encoding and decoding.

p-00052. Description of the Related Art

p-0006When multi-view coding (MVC) for three-dimensional (3D) display applications is performed between adjacent views during prediction between the adjacent views, changes in illumination may occur due to an incompletely calibrated camera, different perspective projection directions, different reflection effects, and the like, causing degradation in coding efficiency. Singe-view coding may also lead to degradation in coding efficiency during scene switching due to an illumination change.

p-0007To solve this problem, H.264 adopts weighted prediction. A weighted prediction scheme is applied to motion compensation at a slice level and illumination is compensated for according to an appropriate weighted factor W and an additional offset 0. Illumination change-adaptive motion estimation/motion compensation (ICA ME/MC) is an improvement on the weighted prediction scheme.

p-0008ICA ME/MC is performed in units of a 16×16 block for Y components and a differential value of illumination change (DVIC) is obtained for each macroblock.

p-0009ICA ME/MC has two modes: one is an IC-inter 16×16 mode that uses ICA ME/MC and is used for predictive (P) or bidirectional predictive (B) slices and the other is an IC-direct 16×16 mode that does not use ICA ME and is used only for B slices. In order to compensate for a local illumination change, a 1-bit flag, i.e., mb_ic_flag, is required for each of the IC-inter 16×16 mode and the IC-direct 16×16 mode.

p-0010Since DVIC for the current block has a high correlation with DVIC for neighboring blocks that are adjacent to the current block, DVIC for the current block is performed by encoding differences between DVIC for the current block and DVIC for the neighboring blocks.

p-0011Hereinafter, ICA ME/MC performed in units of a macroblock in the IC-inter 16×16 mode will be described.

p-0012For ICA ME/MC, a new sum of absolute differences (SAD) has to be defined. When a pixel at (i, j) in the current frame is referred to as f(i, j) and a pixel at (i, j) in the reference frame is referred to as r(i, j), an SAD for S×T blocks is calculated as follows, in which S×T may be 16×16, 16×8, 8×16, 8×8, 8×4, 4×8, or 4×4.

p-0013<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mrow><mi>SAD</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mi>m</mi></mrow><mrow><mi>m</mi><mo>+</mo><mi>S</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mi>n</mi></mrow><mrow><mi>n</mi><mo>+</mo><mi>T</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><mo></mo><mrow><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>i</mi><mo>+</mo><mi>x</mi></mrow><mo>,</mo><mrow><mi>j</mi><mo>+</mo><mi>y</mi></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo></mo></mrow></mrow></mrow></mrow><mo>,</mo></mrow></mtd><mtd><mrow><mo>(</mo><mn>1</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><br /> where (x, y) indicates a candidate motion vector and (i, j) indicates the position of the current block.

p-0014To compensate for an illumination change, a new SAD is required as follows:

p-0015<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msub><mi>M</mi><mi>cur</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mrow><mi>S</mi><mo>×</mo><mi>T</mi></mrow></mfrac><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mi>m</mi></mrow><mrow><mi>m</mi><mo>+</mo><mi>S</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mi>n</mi></mrow><mrow><mi>n</mi><mo>+</mo><mi>T</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><mrow><mrow><msub><mi>M</mi><mi>ref</mi></msub><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>p</mi><mo>,</mo><mi>q</mi></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mfrac><mn>1</mn><mrow><mi>S</mi><mo>×</mo><mi>T</mi></mrow></mfrac><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mi>p</mi></mrow><mrow><mi>p</mi><mo>+</mo><mi>S</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mi>q</mi></mrow><mrow><mi>q</mi><mo>+</mo><mi>T</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo>,</mo></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>2</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><br /> where M_curindicates an average of pixels values in the current block, M_refindicates an average of pixels values in the reference block, and (p, q) indicates the position of the reference block. A new SAD, i.e., NewSAD (x, y), is given as follows:

p-0016<maths id="MATH-US-00003" num="00003"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mi>NewSAD</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mi>m</mi></mrow><mrow><mi>m</mi><mo>+</mo><mi>S</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mi>n</mi></mrow><mrow><mi>n</mi><mo>+</mo><mi>T</mi><mo>-</mo><mn>1</mn></mrow></munderover><mo>⁢</mo><mrow><mo></mo><mrow><mrow><mo>{</mo><mrow><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><msub><mi>M</mi><mi>cur</mi></msub></mrow><mo>}</mo></mrow><mo>-</mo><mrow><mo>{</mo><mrow><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>i</mi><mo>+</mo><mi>x</mi></mrow><mo>,</mo><mrow><mi>j</mi><mo>+</mo><mi>y</mi></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msub><mi>M</mi><mi>ref</mi></msub><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>m</mi><mo>+</mo><mi>x</mi></mrow><mo>,</mo><mrow><mi>n</mi><mo>+</mo><mi>y</mi></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>}</mo></mrow></mrow><mo></mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>3</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

p-0017A block, e.g., a 16×16 block, that minimizes NewSAD (x, y) based on Equation 3 is searched for and a motion vector corresponding to the found block is also searched for.

p-0018Once a motion vector MV(x′, y′) that minimizes NewSAD (x, y) is found, an illumination compensation residual signal NewR(i, j) is determined as follows:

p-0019<maths id="MATH-US-00004" num="00004"><math overflow="scroll"><mtable><mtr><mtd><mtable><mtr><mtd><mrow><mrow><mi>NewR</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mi /><mo>⁢</mo><mrow><mrow><mo>{</mo><mrow><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><msub><mi>M</mi><mi>cur</mi></msub></mrow><mo>}</mo></mrow><mo>-</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>i</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>j</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo></mrow></mtd></mtr><mtr><mtd><mrow><msub><mi>M</mi><mi>ref</mi></msub><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>m</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>n</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow></mtd></mtr></mtable><mo>}</mo></mrow></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mi /><mo>⁢</mo><mrow><mrow><mo>{</mo><mrow><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>i</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>j</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>}</mo></mrow><mo>-</mo></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mi /><mo>⁢</mo><mrow><mo>{</mo><mrow><msub><mi>M</mi><mi>cur</mi></msub><mo>-</mo><mrow><msub><mi>M</mi><mi>ref</mi></msub><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>m</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>n</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>}</mo></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mi /><mo>⁢</mo><mrow><mrow><mo>{</mo><mrow><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>i</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>j</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>}</mo></mrow><mo>-</mo><mi>DVIC</mi></mrow></mrow></mtd></mtr></mtable></mtd><mtd><mrow><mo>(</mo><mn>4</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

p-0020In order to indicate whether ICA ME/MC is used, the 1-bit flag, mb_ic_flag, is stored in a syntax. A differential pulse code modulation (DPCM) value for DVIC is also included in the syntax. If mb_ic_flag is ‘0’, it indicates that ICA MC is not performed for the current block. If mb_ic_flag is ‘1’, it indicates that ICA MC is performed for the current block.

p-0021When mb_ic_flag is ‘1’, an ICA ME/MC unit of a decoder obtains a reconstructed pixel as follows:

p-0022<maths id="MATH-US-00005" num="00005"><math overflow="scroll"><mtable><mtr><mtd><mtable><mtr><mtd><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mi /><mo>⁢</mo><mrow><mrow><mo>{</mo><mrow><mrow><msup><mi>NewR</mi><mi>″</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><msup><mi>x</mi><mi>′</mi></msup><mo>,</mo><msup><mi>y</mi><mi>′</mi></msup><mo>,</mo><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>i</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>j</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>}</mo></mrow><mo>+</mo></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mi /><mo>⁢</mo><mrow><mo>{</mo><mrow><msub><mi>M</mi><mi>cur</mi></msub><mo>-</mo><mrow><msub><mi>M</mi><mi>ref</mi></msub><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>m</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>n</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>}</mo></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mo>=</mo><mi /><mo>⁢</mo><mrow><mrow><mo>{</mo><mrow><mrow><msup><mi>NewR</mi><mi>″</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><msup><mi>x</mi><mi>′</mi></msup><mo>,</mo><msup><mi>y</mi><mi>′</mi></msup><mo>,</mo><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mi>r</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mi>i</mi><mo>+</mo><msup><mi>x</mi><mi>′</mi></msup></mrow><mo>,</mo><mrow><mi>j</mi><mo>+</mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>}</mo></mrow><mo>+</mo><mi>DVIC</mi></mrow></mrow><mo>,</mo></mrow></mtd></mtr></mtable></mtd><mtd><mrow><mo>(</mo><mn>5</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><br /> where NewR″(i, j) indicates a reconstructed illumination compensation residual signal and f′(i, j) indicates a reconstructed pixel in the current frame.

p-0023Since DVIC information has to be transmitted according to conventional ICA ME/MC, coding efficiency is degraded.

SUMMARY OF THE INVENTION

p-0024The present invention provides an illumination compensation method and apparatus, in which transmission of differential value of illumination change (DVIC) information is not required, and a video encoding/decoding method and apparatus using said illumination compensation method.

p-0025According to an aspect of the present invention, there is provided an illumination compensation method for a reference block used for motion estimation. The illumination compensation method includes receiving pixel values of reconstructed neighboring pixels around a current block to be encoded and pixel values of reconstructed neighboring pixels around the reference block, and performing illumination compensation with respect to the reference block based on the input pixel values of reconstructed neighboring pixels around the current block and the input pixel values of reconstructed neighboring pixels around the reference block.

p-0026The performing of the illumination compensation may include calculating illumination compensation parameters with respect to the reference block based on a correlation between the pixel values of the reconstructed neighboring pixels around the current block and the pixel values of the reconstructed neighboring pixels around the reference block and generating an illumination-compensated reference block based on the calculated illumination compensation parameters.

p-0027The performing of the illumination compensation may also include determining values of a_x,yand b_x,ythat minimize J in the equation

p-0028<maths id="MATH-US-00006" num="00006"><math overflow="scroll"><mrow><mi>J</mi><mo>=</mo><mrow><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><msup><mrow><mo>[</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msub><mi>a</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>×</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>-</mo><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub></mrow><mo>]</mo></mrow><mn>2</mn></msup></mrow><mo>+</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><msup><mrow><mo>[</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msub><mi>a</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>×</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow><mo>-</mo><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub></mrow><mo>]</mo></mrow><mn>2</mn></msup></mrow></mrow></mrow></math></maths><br /> and generating the illumination-compensated reference block based on <o>f</o>(i, j)=a_x,y·r′_x,y(i, j)+b_x,yusing the determined values of a_x,yand b_x,y, where a_x,yand b_x,ydepend on a motion vector (x, y) and are constants with respect to the motion vector (x, y), f′(i,−1) and f′(−1,j) indicate the pixel values of the reconstructed neighboring pixels around the current block, r′_x,y(i,−1) and r′_x,y(−1,j) indicate the pixel values of the reconstructed neighboring pixels around the reference block, r′_x,y(i,j) indicates a motion-compensated reference block, and <o>f</o>(i, j) indicates the illumination-compensated reference block.

p-0029The performing of the illumination compensation may also include generating the illumination-compensated reference block using an average of differences between the pixel values of the reconstructed neighboring pixels around the current block and the pixel values of the reconstructed neighboring pixels around the reference block.

p-0030The performing of the illumination compensation may also include generating the illumination-compensated reference block using a difference between an average of the pixel values of the reconstructed neighboring pixels around the current block and an average of the pixel values of the reconstructed neighboring pixels around the reference block.

p-0031The reference block may be a block included in a reference block among reconstructed frames of an adjacent view in multi-view coding.

p-0032According to another aspect of the present invention, there is provided an illumination compensation apparatus for a reference block used for motion estimation. The illumination compensation apparatus includes an illumination compensation unit that performs illumination compensation, with respect to the reference block, based on pixel values of reconstructed neighboring pixels around a current block to be encoded and pixel values of reconstructed neighboring pixels around the reference block.

p-0033The illumination compensation unit may include a calculation unit that calculates illumination compensation parameters for the illumination compensation, with respect to the reference block, based on a correlation between the pixel values of the reconstructed neighboring pixels around the current block and the pixel values of the reconstructed neighboring pixels around the reference block and an illumination-compensated reference block generation unit that generates an illumination-compensated reference block based on the calculated illumination compensation parameters.

p-0034According to another aspect of the present invention, there is provided a video encoding method based on illumination compensation. The video encoding method includes performing illumination compensation, with respect to a reference block, used for motion estimation based on pixel values of reconstructed neighboring pixels around a current block to be encoded and pixel values of reconstructed neighboring pixels around the reference block and performing motion estimation based on the illumination-compensated reference block.

p-0035According to another aspect of the present invention, there is provided a video encoding apparatus based on illumination compensation. The video encoding apparatus includes an illumination compensation unit that performs illumination compensation, with respect to a reference block, used for motion estimation based on pixel values of reconstructed neighboring pixels around a current block to be encoded and pixel values of reconstructed neighboring pixels around the reference block and a motion estimation unit that performs motion estimation based on the illumination-compensated reference block.

p-0036According to another aspect of the present invention, there is provided a video decoding method based on illumination compensation. The video decoding method includes performing illumination compensation, with respect to a reference block, used for motion compensation based on pixel values of reconstructed neighboring pixels around a current block to be decoded and pixel values of reconstructed neighboring pixels around the reference block and performing motion compensation based on the illumination-compensated reference block.

p-0037According to another aspect of the present invention, there is provided a video decoding apparatus based on illumination compensation. The video decoding apparatus includes an illumination compensation unit that performs illumination compensation, with respect to a reference, block used for motion compensation based on pixel values of reconstructed neighboring pixels around a current block to be decoded and pixel values of reconstructed neighboring pixels around the reference block and a motion compensation unit that performs motion compensation based on the illumination-compensated reference block.

p-0038According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for implementing an illumination compensation method for a reference block used for motion estimation. The illumination compensation method includes receiving pixel values of reconstructed neighboring pixels around a current block to be encoded and pixel values of reconstructed neighboring pixels around the reference block and performing illumination compensation, with respect to the reference block, based on the input pixel values of reconstructed neighboring pixels around the current block and the input pixel values of reconstructed neighboring pixels around the reference block.

p-0039According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for implementing a video encoding method based on illumination compensation. The video encoding method includes performing illumination compensation, with respect to a reference block, used for motion estimation based on pixel values of reconstructed neighboring pixels around a current block to be encoded and pixel values of reconstructed neighboring pixels around the reference block and performing motion estimation based on the illumination-compensated reference block.

p-0040According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for implementing a video decoding method based on illumination compensation. The video decoding method includes performing illumination compensation, with respect to a reference block, used for motion compensation based on pixel values of reconstructed neighboring pixels around a current block to be decoded and pixel values of reconstructed neighboring pixels around the reference block and performing motion compensation based on the illumination-compensated reference block.

BRIEF DESCRIPTION OF THE EXEMPLARY EMBODIMENTS OF THE DRAWINGS

The above and other aspects of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

<figref idrefs="DRAWINGS">FIG. 1</figref> is a block diagram of an illumination compensation apparatus according to an exemplary embodiment of the present invention;

<figref idrefs="DRAWINGS">FIGS. 2A and 2B</figref> are views for explaining calculation of an illumination compensation parameter according to an exemplary embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 3</figref> is a flowchart illustrating an illumination compensation method according to an exemplary embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 4</figref> is a block diagram of a video encoding apparatus using an illumination compensation method according to an exemplary embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 5</figref> is a flowchart illustrating a video encoding method using an illumination compensation method according to an exemplary embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 6</figref> is a block diagram of a video decoding apparatus using an illumination compensation method according to an exemplary embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 7</figref> is a flowchart illustrating a video decoding method using an illumination compensation method according to an exemplary embodiment of the present invention;

<figref idrefs="DRAWINGS">FIG. 8</figref> is a view for explaining an encoding order in an 8×8 mode; and

<figref idrefs="DRAWINGS">FIG. 9</figref> is a view for explaining an encoding order in a 4×4 mode.

DETAILED DESCRIPTION OF THE INVENTION

p-0051Hereinafter, an exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings. Like reference numerals refer to like elements throughout the figures.

p-0052<figref idrefs="DRAWINGS">FIG. 1</figref> is a block diagram of an illumination compensation apparatus <b>100</b> according to an exemplary embodiment of the present invention.

p-0053Referring to <figref idrefs="DRAWINGS">FIG. 1</figref>, the illumination compensation apparatus <b>100</b> includes an illumination compensation parameter calculation unit <b>120</b> and an illumination-compensated reference block generation unit <b>140</b>.

p-0054The illumination parameter calculation unit <b>120</b> calculates illumination compensation parameters based on pixel values of reconstructed neighboring pixels around the current block and pixel values of reconstructed neighboring pixels around a reference block and outputs the calculated illumination compensation parameters to the illumination-compensated reference block generation unit <b>140</b>. In single-view coding, the reference block may be a block in a reconstructed previous frame. In multi-view coding, the reference block may be a block in a reference block among reconstructed frames of an adjacent view.

p-0055Calculation of the illumination compensation parameter according to the present invention will be described with reference to <figref idrefs="DRAWINGS">FIGS. 2A and 2B</figref>. Descriptions will be made separately in relation to a 16×16 mode, an 8×8 mode, a 4×4 mode, and an adaptive mode.

p-0056Calculation of the illumination compensation parameter in the 16×16 mode will now be described with reference to <figref idrefs="DRAWINGS">FIGS. 2A and 2B</figref>. The calculation in the other modes will be described later.

p-0057In <figref idrefs="DRAWINGS">FIG. 2A</figref>, a block marked with a bold line is a 16×16 Y block, which is a reconstructed reference block corresponding to a motion vector (x, y) in a reference frame. A reconstructed pixel at (i, j) in the reference block corresponding to the motion vector (x, y) is referred to as r′_x,y(i, j). Here, the apostrophe ' means a reconstructed value. Dotted pixels are reconstructed neighboring pixels around the current block. A reference frame is a previous frame in the case of single-view coding and is a reconstructed frame of an adjacent view in multi-view coding.

p-0058In <figref idrefs="DRAWINGS">FIG. 2B</figref>, a block marked with bold lines is a 16×16 current Y block. The original pixel at (i, j) in the current block is referred to as f(i, j). Dotted neighboring pixels are used for motion estimation compensation. The neighboring pixels used for motion estimation compensation have already been reconstructed. As in <figref idrefs="DRAWINGS">FIG. 2A</figref>, the apostrophe ' means a reconstructed value.

p-0059A predictor of the pixel f(i, j) in the current block can be expressed as a linear function of r′_x,y(i, j), as follows: <br /><i><o>f</o></i>(<i>i,j</i>)=a_x,y<i>·r′</i>_x,y(<i>i,j</i>)+<i>b</i>_x,y  (6)

p-0060Predictors obtained using Equation 6 are clipped to be in the range [0, 255] in the case of an 8-bit image.

p-0061As in Equation 6, once a_x,yand b_x,yare determined, the predictor can be calculated. In the current exemplary embodiment of the present invention, the illumination compensation parameters a_x,yand b_x,ycorresponding to the motion vector (x, y) are calculated using the neighboring pixels illustrated in <figref idrefs="DRAWINGS">FIGS. 2A and 2B</figref> based on the fact that the illumination of pixels in the current block is not much different from that of neighboring pixels around the current block.

p-0062In other words, since a_x,yand b_x,ycan be determined using neighboring reconstructed pixels of the current block and the reference block, a decoder can calculate a_x,yand b_x,ywithout a need for an encoder to transmit a_x,yand b_x,yto the decoder using a predetermined syntax, thereby reducing transmission data.

p-0063Three exemplary embodiments for determining a_x,yand b_x,yaccording to the present invention, i.e., linear regression, an average-of-difference based prediction (ADP), and a difference-of-average based prediction (DAP) will be now described.

p-0064Linear regression will be first described with reference to Equation 7 and Equation 8.

p-0065In the linear regression method, values of a_x,yand b_x,ythat minimize a value of Equation 7 can be determined.

p-0066<maths id="MATH-US-00007" num="00007"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mi>J</mi><mo>=</mo><mrow><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>-</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><msup><mrow><mo>[</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msub><mi>a</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>×</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>-</mo><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub></mrow><mo>]</mo></mrow><mn>2</mn></msup></mrow><mo>+</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>-</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><msup><mrow><mo>[</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msub><mi>a</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>×</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow><mo>-</mo><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub></mrow><mo>]</mo></mrow><mn>2</mn></msup></mrow></mrow></mrow><mo>,</mo></mrow></mtd><mtd><mrow><mo>(</mo><mn>7</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><br /> where f′(i,−1) and f′(−1,j) indicate pixel values of neighboring pixels around the current block, r′_x,y(i,−1) and r′_x,y(−1,j) indicate pixel values of neighboring pixels around the motion-compensated reference block, and a_x,yand b_x,ydepend on the motion vector (x, y) and are constants with respect to the motion vector (x, y).

p-0067Using a partial differential method, values of a_x,yand b_x,ythat minimize J of Equation 7 can be given as follows:

p-0068<maths id="MATH-US-00008" num="00008"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msub><mi>a</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>=</mo><mfrac><mrow><mrow><mi>N</mi><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>⁢</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow><mo>·</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow></mrow></mrow></mrow><mo>-</mo><mrow><munderover><mo>∑</mo><mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>⁢</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>⁢</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mrow><mrow><mi>N</mi><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>⁢</mo><msup><mrow><mo>(</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow><mn>2</mn></msup></mrow></mrow><mo>-</mo><msup><mrow><mo>(</mo><mrow><munderover><mo>∑</mo><mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>⁢</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow><mn>2</mn></msup></mrow></mfrac></mrow><mo>,</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>=</mo><mfrac><mrow><mrow><munderover><mo>∑</mo><mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>⁢</mo><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow></mrow><mo>-</mo><mrow><msub><mi>a</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>·</mo><mrow><munderover><mo>∑</mo><mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>⁢</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mi>n</mi><mo>)</mo></mrow></mrow></mrow></mrow></mrow><mi>N</mi></mfrac></mrow><mo>,</mo></mrow></mtd><mtd><mrow><mo>(</mo><mn>8</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><br /> where, when each of the current block and the reference block is 16×16 in size, N is equal to 32, f′(n) indicates a reconstructed neighboring pixel around the current block and is one of f′(i,−1) and f′(−1,j), and r′_x,y(n) is one of r′_x,y(i,−1) and r′_x,y(−1,j). Here, i and j may be in the range between 0 and 15.

p-0069Calculation of a_x,yand b_x,yusing ADP will now be described with reference to Equation 9.

p-0070According to ADP, only b_x,yis calculated while fixing a_x,yto 1. In other words, b_x,yis determined as an average of differences between pixel values of reconstructed neighboring pixels as follows:

p-0071<maths id="MATH-US-00009" num="00009"><math overflow="scroll"><mtable><mtr><mtd><mrow><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>=</mo><mfrac><mrow><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mn>32</mn></mfrac></mrow></mtd><mtd><mrow><mo>(</mo><mn>9</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

p-0072Hereinafter, calculation of a_x,yand b_x,yusing DAP will be described with reference to Equation 10.

p-0073In DAP, only b_x,yis calculated while fixing a_x,yto 1. In other words, b_x,yis determined as a difference between averages of pixel values of reconstructed neighboring pixels as follows:

p-0074<maths id="MATH-US-00010" num="00010"><math overflow="scroll"><mtable><mtr><mtd><mrow><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>=</mo><mrow><mfrac><mrow><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>+</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><msup><mi>f</mi><mi>′</mi></msup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow></mrow><mn>32</mn></mfrac><mo>-</mo><mfrac><mrow><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁢</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mrow><mo>-</mo><mn>1</mn></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>+</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mrow><mo>-</mo><mn>1</mn></mrow><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow></mrow><mn>32</mn></mfrac></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>10</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

p-0075As discussed above, the illumination compensation parameter calculation unit <b>120</b> calculates the illumination compensation parameters a_x,yand b_x,yusing linear regression, ADP, and DAP and outputs the calculated a_x,yand b_x,yto the illumination-compensated reference block generation unit <b>140</b>.

p-0076The illumination-compensated reference block generation unit <b>140</b> generates an illumination-compensated reference block composed of the predictors for the pixel f(i, j) obtained using Equation 6 and outputs the generated illumination-compensated reference block to a motion estimation unit (not shown).

p-0077<figref idrefs="DRAWINGS">FIG. 3</figref> is a flowchart illustrating an illumination compensation method implemented by the illumination compensation apparatus <b>100</b>.

p-0078In operation <b>310</b>, pixel values of reconstructed neighboring pixels around the current block and pixel values of reconstructed neighboring pixels around a reference block are input. In single-view coding, the reference block may be a block in a reconstructed previous frame. In multi-view coding, the reference block may be a block in a reference block among reconstructed frames of an adjacent view.

p-0079In operation <b>320</b>, illumination compensation is performed with respect to the reference block based on the input pixel values of the reconstructed neighboring pixels around the current block and the input pixel values of the reconstructed neighboring pixels around the reference block. In operation <b>320</b>, illumination compensation parameters are generated based on a correlation between the input pixel values of the reconstructed neighboring pixels around the current block and the input pixel values of the reconstructed neighboring pixels around the reference block in the current frame, as in Equations 7 through 10, and an illumination-compensated reference block is generated using the generated illumination compensation parameters using Equation 6.

p-0080<figref idrefs="DRAWINGS">FIG. 4</figref> is a block diagram of a video encoding apparatus using the illumination compensation method according to an exemplary embodiment of the present invention.

p-0081The video encoding apparatus according to the present invention includes a transformation/quantization unit <b>410</b>, an inverse transformation/inverse quantization unit <b>420</b>, a frame storing unit <b>430</b>, an illumination compensation unit <b>440</b>, an ME/MC unit <b>450</b>, a first addition unit <b>460</b>, a second addition unit <b>462</b>, and an entropy-coding unit <b>470</b>.

p-0082The transformation/quantization unit <b>410</b> transforms input video data to remove the spatial redundancy of the input video data. The transformation/quantization unit <b>410</b> quantizes transform coefficients obtained by performing transform encoding using a predetermined quantization step, thereby obtaining two-dimensional (2D) N×M data composed of the quantized transform coefficients. A DCT may be used as the transform. The quantization is performed using a predetermined quantization step.

p-0083An inverse quantization/inverse transform unit <b>420</b> inversely quantizes the video data that is quantized by the transformation/quantization unit <b>410</b> and inversely transforms the inversely quantized video data using, for example, an inverse DCT (IDCT).

p-0084The second addition unit <b>462</b> adds reconstructed video data obtained by the inverse transform/inverse quantization unit <b>420</b> to prediction video data output from the ME/MC unit <b>450</b>, thereby generating reconstructed video data.

p-0085The frame storing unit <b>430</b> stores the reconstructed video data obtained by the second addition unit <b>462</b> in frame units.

p-0086The illumination compensation unit <b>440</b> receives pixel values of reconstructed neighboring pixels around the current block and pixel values of reconstructed neighboring pixels around the reference block, which are input from the frame storing unit <b>430</b>, in order to generate an illumination-compensated reference block, and outputs the generated illumination-compensated reference block to the ME/MC unit <b>450</b>. When the input video data is based on MVC, the reconstructed neighboring pixels around the reference block may be those located in a frame of an adjacent view. In this case, the reconstructed neighboring pixels around the reference block may be input from an MVC based reference frame storing unit (not shown).

p-0087The illumination compensation unit <b>440</b> functions in the same way as the illumination compensation apparatus <b>100</b> illustrated in <figref idrefs="DRAWINGS">FIG. 1</figref> and thus will not be described in detail for simplicity of explanation.

p-0088The ME/MC unit <b>450</b> estimates a motion vector MV for each macroblock based on the input video data of the current frame and the illumination-compensated reference block output from the illumination compensation unit <b>440</b>. The ME/MC unit <b>450</b> also generates a motion-compensated prediction area P based on the estimated motion vector, e.g., a 16×16 area selected by motion estimation, and outputs the motion-compensated prediction area P to the first addition unit <b>460</b>.

p-0089In other words, the ME/MC unit <b>450</b> obtains an illumination-compensated (IC) SAD corresponding to the current block based on the illumination-compensated reference block obtained by the illumination compensation unit <b>440</b> as follows. The ME/MC unit <b>450</b> also compares IC SADs and searches for a final motion vector.

p-0090<maths id="MATH-US-00011" num="00011"><math overflow="scroll"><mtable><mtr><mtd><mtable><mtr><mtd><mrow><mrow><mi>IC</mi><mo>-</mo><mrow><mi>SAD</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></mrow></mrow><mo>=</mo><mi /><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><mo></mo><mrow><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><mover><mi>f</mi><mi>_</mi></mover><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow><mo></mo></mrow></mrow></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mo>=</mo><mi /><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mn>15</mn></munderover><mo>⁢</mo><mrow><mo></mo><mrow><mrow><mi>f</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msub><mi>a</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub><mo>·</mo><mrow><msubsup><mi>r</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mi>′</mi></msubsup><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow></mrow><mo>-</mo><msub><mi>b</mi><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow></msub></mrow><mo></mo></mrow></mrow></mrow></mrow><mo>,</mo></mrow></mtd></mtr></mtable></mtd><mtd><mrow><mo>(</mo><mn>11</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths>

p-0091where a_x,yand b_x,ydepend on the motion vector (x, y) and are constants with respect to the motion vector (x, y).

p-0092The ME/MC unit <b>450</b> may also include a comparison unit (not shown).

p-0093The comparison unit compares the efficiency of coding based on the input video data of the current frame and the illumination-compensated reference block output from the illumination compensation unit <b>440</b> with the efficiency of coding based on the input video data of the current frame and a non-illumination-compensated reference block output from the frame storing unit <b>430</b>, and selects one having the higher efficiency. At this time, the ME/MC unit <b>450</b> generates the motion-compensated prediction area P obtained using selected ME/MC and outputs the motion-compensated prediction area P to the first addition unit <b>460</b>.

p-0094Additionally, 1-bit flag information indicating whether illumination compensation is applied may be transmitted to a decoder using a predetermined syntax in units of a macroblock. Also, the performance of illumination compensation may be evaluated in units of a group of pictures (GOP) or a slice and flag information indicating whether illumination compensation is applied may be transmitted in units of a GOP or a slice.

p-0095The first addition unit <b>460</b> inputs a difference between the original video data and a predictor output from the ME/MC unit <b>450</b> to the transformation/quantization unit <b>410</b> in units of a predetermined block. When the final motion vector, (p, q), for the current block is determined by a motion vector search, the first addition unit <b>460</b> calculates a residue (p, q) for the current block as given below and outputs the residue (p, q) to the transformation/quantization unit <b>410</b>. <br />residue(<i>p,q</i>)={<i>f</i>(<i>i,j</i>)−<i>a</i>_p,q<i>·r′</i>_p,q(<i>i,j</i>)−<i>b</i>_p,q|0≦<i>i,j≦</i>15}  (12),

p-0096where a_p,qand b_p,qdiffer with the motion vector (p, q) and are constants with respect to the motion vector (p, q).

p-0097The entropy-coding unit <b>470</b> receives the quantized transform coefficients output from the transformation/quantization unit <b>410</b> and the motion vector output from the ME/MC unit <b>450</b> and performs entropy-coding on the received transform coefficients and motion vector, thereby outputting a final encoded bitstream.

p-0098<figref idrefs="DRAWINGS">FIG. 5</figref> is a flowchart illustrating a video encoding method implemented by the video encoding apparatus using the illumination compensation method according to an exemplary embodiment of the present invention.

p-0099In operation <b>510</b>, transformation and quantization are performed.

p-0100In operation <b>520</b>, inverse transformation and inverse quantization are performed on the transformed and quantized data in order to generate reconstructed video data.

p-0101In operation <b>530</b>, illumination compensation is performed on a reference block based on pixel values of reconstructed neighboring pixels around the current block to be encoded and pixel values of reconstructed neighboring pixels around the reference block. In the current exemplary embodiment of the present invention, the reference block is a block included in a previous frame. However, when input video data is based on MVC, the reference block may be a block located in a frame of an adjacent view.

p-0102In operation <b>540</b>, ME and MC are performed based on the illumination-compensated reference block in order to generate a prediction block.

p-0103In operation <b>550</b>, residual video data is generated based on the original video data and the generated prediction block. Transformation and quantization are performed on the generated residual video data. The transformed and quantized residual video data is entropy-coded with a motion vector obtained by ME.

p-0104<figref idrefs="DRAWINGS">FIG. 6</figref> is a block diagram of a video decoding apparatus using the illumination compensation method according to an exemplary embodiment of the present invention.

p-0105Referring to <figref idrefs="DRAWINGS">FIG. 6</figref>, the video decoding apparatus includes an entropy-decoding unit <b>610</b>, an inverse quantization/inverse transformation unit <b>620</b>, a frame storing unit <b>630</b>, an illumination compensation unit <b>640</b>, an ME/MC unit <b>650</b>, and an addition unit <b>660</b>.

p-0106The entropy-decoding unit <b>610</b> performs entropy-decoding on an input encoded bitstream in order to extract video data, motion vector information, and the like. The entropy-decoded video data is input to the inverse quantization/inverse transformation unit <b>620</b> and the motion vector information is input to the ME/MC unit <b>650</b>.

p-0107The inverse transformation/inverse quantization unit <b>620</b> performs inverse transformation/inverse quantization on the video data that is extracted by the entropy-decoding unit <b>610</b>.

p-0108The frame storing unit <b>630</b> stores the data that is inversely quantized and inversely transformed by the inverse transformation/inverse quantization unit <b>620</b> in frame units.

p-0109The illumination compensation unit <b>640</b> receives pixel values of reconstructed neighboring pixels around the current block and pixel values of reconstructed neighboring pixels around a reference block in order to generate an illumination-compensated reference block and outputs the generated illumination-compensated reference block to the ME/MC unit <b>650</b>. The illumination compensation unit <b>640</b> functions in the same manner as the illumination compensation apparatus <b>100</b> illustrated in <figref idrefs="DRAWINGS">FIG. 1</figref> and thus will not be described in detail for simplicity of explanation.

p-0110Optionally, when the video decoding apparatus illustrated in <figref idrefs="DRAWINGS">FIG. 6</figref> is based on MVC, the reconstructed neighboring pixels around the reference block may be those located in a frame of an adjacent view. In this case, the reconstructed neighboring pixels around the reference block may be input from an MVC based reference frame storing unit (not shown).

p-0111The ME/MC unit <b>650</b> estimates a motion vector MV for each macroblock based on video data of the current frame and the illumination-compensated reference block output from the illumination compensation unit <b>640</b>. The ME/MC unit <b>650</b> also generates a motion-compensated prediction area P based on the estimated motion vector, e.g., a 16×16 area selected by motion estimation, and outputs the motion-compensated prediction area P to the addition unit <b>660</b>.

p-0112In other words, the ME/MC unit <b>650</b> obtains an illumination-compensated (IC) SAD corresponding to the current block based on the illumination-compensated reference block obtained by the illumination compensation unit <b>640</b>. The ME/MC unit <b>650</b> also compares IC SADs and searches for a final motion vector.

p-0113The ME/MC unit <b>650</b> may also include a comparison unit (not shown). The comparison unit compares the efficiency of coding based on the input video data of the current frame and the illumination-compensated reference block output from the illumination compensation unit <b>640</b> with the efficiency of coding based on the input video data of the current frame and a non-illumination-compensated reference block output from the frame storing unit <b>630</b>, and selects one having the higher efficiency. At this time, the ME/MC unit <b>650</b> generates the motion-compensated prediction area P obtained using selected ME/MC and outputs the motion-compensated prediction area P to the first addition unit <b>660</b>.

p-0114The entropy decoding unit <b>610</b> may also extract flag information indicating whether illumination compensation is applied and the ME/MC unit <b>650</b> outputs a motion-estimated and motion-compensated prediction block based on an illumination-compensated reference block or based on a non-illumination-compensated reference block, which is determined by the extracted flag information to the addition unit <b>660</b>.

p-0115The addition unit <b>660</b> adds video data reconstructed by the inverse transformation/inverse quantization unit <b>620</b> to a predictor output from the ME/MC unit <b>650</b> and outputs the addition result to a display unit (not shown) and the frame storing unit <b>630</b>.

p-0116In the current exemplary embodiment of the present invention, the addition unit <b>660</b> adds a reconstructed residual signal residue′(p, q) for the residual signal residue(p,q) input from the inverse quantization/inverse transformation unit <b>620</b> to pixel values of the prediction block input from the ME/MC unit <b>650</b>, thereby calculating a final reconstructed pixel f′(x, y) in the current block as follows: <br /><i>f</i>′(<i>i,j</i>)=residue′_p,q(<i>i,j</i>)+<i>a</i>_p,q<i>·r</i>′(<i>i,j</i>)+<i>b</i>_p,q  (13),

p-0117where a_p,qand b_p,qdiffer with the motion vector (p, q) and are constants with respect to the motion vector (p, q).

p-0118<figref idrefs="DRAWINGS">FIG. 7</figref> is a flowchart illustrating a video decoding method implemented by the video decoding apparatus using the illumination compensation method according to an exemplary embodiment of the present invention.

p-0119In operation <b>710</b>, an input encoded bitstream is entropy-decoded in order to extract video data and motion vector information.

p-0120In operation <b>720</b>, inverse transformation and inverse quantization are performed on the extracted video data.

p-0121In operation <b>730</b>, reconstructed video data obtained by inverse transformation and inverse quantization is stored in frame units.

p-0122In operation <b>740</b>, illumination compensation is performed on a reference block based on pixel values of reconstructed neighboring pixels around the current block to be decoded and pixel values of reconstructed neighboring pixels around the reference block. When the input bitstream is based on MVC, the reconstructed neighboring pixels around the reference block are those located in a frame of an adjacent block.

p-0123In operation <b>750</b>, ME and MC are performed based on the illumination-compensated reference block and the extracted motion vector information, thereby generating a prediction block.

p-0124In operation <b>760</b>, reconstructed video data is generated using the generated prediction block. For example, the reconstructed video data is generated by adding the prediction block to the inversely transformed and inversely quantized video data.

p-0125Illumination compensation and MC in an 8×8 mode will now be described with reference to <figref idrefs="DRAWINGS">FIG. 8</figref>.

p-0126In the 8×8 mode, a 16×16 block includes a total of four 8×8 sub blocks which have different motion vectors. In this case, a processing order is as illustrated in <figref idrefs="DRAWINGS">FIG. 8</figref>. Although the 8×8 mode uses an 8×8 sub block as a basic unit to be illumination-compensated, it operates in the same manner as a 16×16 mode. In other words, illumination compensation parameters a_x,yand b_x,yare obtained using linear regression, ADP, and DAP. The MC process in the 8×8 mode in the video encoding apparatus and the video decoding apparatus is the same as that in the 16×16 mode except that it uses an 8×8 block as a unit to be motion compensated.

p-0127Illumination compensation and MC in a 4×4 mode will now be described with reference to <figref idrefs="DRAWINGS">FIG. 9</figref>.

p-0128In the 4×4 mode, a 16×16 block includes a total of sixteen 4×4 sub blocks which have different motion vectors. In this case, a processing order is as illustrated in <figref idrefs="DRAWINGS">FIG. 9</figref>. Although the 4×4 mode uses a 4×4 sub block as a basic unit to be illumination-compensated, it operates in the same manner as a 16×16 mode. In other words, illumination compensation parameters a_x,yand b_x,yare obtained using linear regression, ADP, and DAP. The MC process in the 4×4 mode in the video encoding apparatus and the video decoding apparatus is the same as that in the 16×16 mode except that it uses a 4×4 block as a unit to be motion compensated.

p-0129Illumination compensation and MC in an adaptive mode will now be described.

p-0130In the adaptive mode, the video encoding apparatus may perform illumination compensation and encoding in units of a macroblock in the 16×16 mode, the 8×8 mode, and the 4×4 mode and select one of the 16×16 mode, the 8×8 mode, and the 4×4 mode. At this time, a residual block corresponding to each of the 16×16 mode, the 8×8 mode, and the 4×4 mode is encoded and transmitted. Mode information indicating a mode may be transmitted to the video decoding apparatus. Optionally, the mode information may not be transmitted if the video decoding apparatus can perform illumination compensation and ME in each of the 16×16 mode, the 8×8 mode, and the 4×4 mode.

p-0131Although the 16×16 mode, the 8×8 mode, and the 4×4 mode are taken as examples in the exemplary embodiments of the present invention, the illumination compensation method according to the present invention may also be applied to MC based on a block unit of other sizes or shapes.

p-0132The present invention may also be embodied as computer-readable information on a computer-readable recording medium. The computer-readable recording medium is any data storage device that can store data which can be thereafter read by a computer system. Examples of computer-readable recording media include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium can also be distributed over networks of coupled computer systems so that the computer-readable code is stored and executed in a decentralized fashion.

p-0133As described above, according to exemplary embodiments of the present invention, it is not necessary to transmit illumination compensation parameters, thereby improving coding efficiency.

p-0134While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention as defined by the following claims.