Image processing apparatus, image processing method and program

The present invention relates to a technique for suppressing image quality deterioration due to foreign matter adhering to the surface of an optical low-pass filter or the like arranged in front of an image sensor in an image pickup device using an image sensor such as a CCD or CMOS sensor, particularly during moving image shooting. It relates to a technique for suppressing deterioration of image quality due to foreign matter.

In recent years, in order to handle moving image information as digital data and use it for storage and transmission, a technique for encoding with a high compression rate and high image quality has been desired. For compression of image information, a method such as MPEG that compresses and encodes image information by orthogonal transform such as discrete cosine transform and motion prediction / motion compensation, utilizing the redundancy peculiar to moving image information, has been widely proposed. It is widespread.

Each manufacturer has developed and commercialized an imaging device such as a digital camera or a digital video camera or a DVD recorder that can record an image using these coding methods. Then, the user can easily view the image using these devices, a personal computer, a DVD player, or the like.

In recent years, there is H.264 (MPEG4-Part10 AVC) as a coding method aiming at higher compression rate and higher image quality. Compared with conventional coding methods such as MPEG2 and MPEG4, this H.264 requires a large amount of calculation for its coding and decoding, but it is known that higher coding efficiency is realized. (See Non-Patent Document 1).

FIG. 1 is a diagram showing a configuration of an image processing device that compresses image data by the H.264 method.

In FIG. 1, the input image data is divided into macro blocks and sent to the subtraction unit 101. FIG. 2 shows a schematic diagram when the input image data is divided into macro blocks. Figure 3 shows a typical macroblock partition. In H.264, the block size can be selected from 16x16 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, and in the case of 8x8 pixels, it is further 8x8 pixels, 8x4 pixels, 4x8 pixels and 4x4 pixels. You can choose either one.

The subtraction unit 101 obtains the difference between the image data and the predicted value, and outputs the difference to the integer DCT (Discrete Cosine Transform) transform unit 102. The integer DCT transform unit 102 converts the input data into an integer DCT and outputs it to the quantization unit 103. The quantization unit 103 quantizes the input data. One of the quantized data is sent to the entropy coding unit 115 as the difference image data. The other is inversely quantized by the inverse quantization unit 104 and then subjected to the inverse integer DCT transform by the inverse integer DCT transform unit 105. A predicted value is added to the data converted to the inverse integer DCT by the addition unit 106. This restores the image.

One of the restored images is sent to frame memory 107 for intra (in-frame) prediction. The other is sent to the frame memory 110 for inter (inter-frame) prediction after the deblocking filter processing is performed by the deblocking filter 109. The image in the frame memory 107 for intra-prediction is used for intra-prediction by the intra-prediction unit 108. In this intra prediction, the value of the adjacent pixel of the already encoded block in the same picture is used as the prediction value.

The image in the frame memory 110 for inter-prediction is composed of a plurality of pictures as described later. These multiple pictures are divided into two lists, "List 0" and "List 1". The plurality of pictures divided into two lists are used in the inter-prediction by the inter-prediction unit 111. After the inter-prediction is made, the memory controller 113 updates the internal image. In the inter-prediction performed by the inter-prediction unit 111, the predicted image is determined by using the optimum motion vector based on the result of the motion detection performed by the motion detection unit 112 for the image data having different frames.

As a result of the intra prediction and the inter prediction, the optimum prediction is selected by the selection unit 114. Further, the motion vector is sent to the entropy coding unit 115 and encoded together with the difference image data. The output bitstream is thus formed.

Here, the H.264 method inter-prediction will be described in detail with reference to FIGS. 4 to 7.

In the H.264 inter-prediction, a plurality of pictures can be used for the prediction. Therefore, two lists (List 0 and List 1) are prepared to identify the reference picture. Up to 5 reference pictures can be assigned to each list.

In the P picture, only "List 0" is used to mainly perform forward prediction. In the B picture, "List 0" and "List 1" are used to perform bidirectional prediction (or prediction only forward or backward). That is, "List 0" is mainly assigned a picture for forward prediction, and "List 1" mainly contains a picture for backward prediction.

Figure 4 shows an example of a reference list used for coding. Here, a case where the ratios of the I picture, the P picture, and the B picture are standard will be described as an example. That is, the case where the I picture has an interval of 15 frames, the P picture has an interval of 3 frames, and the B picture in between is 2 frames will be described. In FIG. 4, 401 shows the image data arranged in the display order. A number indicating the type and display order of the pictures is entered in the square of the image data 401. For example, picture I15 is the 15th I picture in the display order, and only intra-prediction is performed. Picture P18 is the 18th P picture in the display order, and only forward prediction is performed. Picture B16 is the 16th B picture in the display order, and performs bidirectional prediction.

The order of coding is different from the display order, and is the order of prediction. That is, in FIG. 4, the coding order is "I15, P18, B16, B17, P21, B19, B20, ...".

Further, in FIG. 4, 402 shows a reference list (List 0). This reference list (List 0) 402 contains pictures that have been once encoded and decoded. For example, when performing inter-prediction with picture P21 (P picture in the 21st display order), the already encoded and decoded picture in the reference list (List 0) 402 is referred to. In the example shown in FIG. 4, pictures P06, P09, P12, I15, P18 are included in reference list 402.

In the inter-prediction, the motion vector having the optimum predicted value is obtained and encoded from the reference pictures in the reference list (List 0) 402 for each macroblock. The pictures in the reference list (List 0) 402 are given reference picture numbers in order and distinguished (given separately from the numbers shown).

When the coding of the picture P21 is completed, the picture P21 is newly decoded and added to the reference list (List 0) 402. The oldest referenced picture (here picture P06) is removed from the reference list (List 0) 402. After this, coding is performed with pictures B19 and B20, and continues to picture P24. The state of the reference list (List 0) 402 at this time is shown in FIG.

Figure 6 shows how the reference list changes for each picture.

In FIG. 6, the pictures are encoded in order from the top. It also shows the picture being coded and the contents of the reference list (List 0 and List 1) for the picture being coded. As shown in Figure 6, when the P-picture (or I-picture) is encoded, the reference list (List 0 and List 1) is updated and the oldest picture in the reference list (List 0 and List 1) is removed. Will be done. In this example, the reference list (List 1) has only one picture. This is because if the number of pictures referenced for backward prediction is increased, the amount of buffer until decoding increases. That is, the reference to the rear picture that is too far from the coded picture is avoided.

In the example given here, the pictures used for reference are I picture and P picture, and all of the I picture and P picture are sequentially added to the reference list (List 0 and List 1). In addition, the only picture used in the reference list (List 1) for backward prediction is the P picture. The reason for the above is that it is usually the most commonly used picture structure. However, the composition of pictures in such a reference list is only one example that will be most often used, and H.264 itself has a higher degree of freedom in the composition of the reference list.

For example, it is not necessary to add all I pictures and P pictures to the reference list, and it is possible to add B pictures to the reference list. It also defines a long-term reference list that stays in the reference list until explicitly indicated. Figure 7 shows how the reference list changes when a B picture is added to the reference list. When adding a B-picture to the reference list, it is usually conceivable to add the encoded picture to the reference list each time all B-pictures are encoded.

Next, a file format for recording the moving image data compressed in this way will be described.

As described above, the MP4 (MPEG-4) file format is used as a general-purpose format for recording MPEG (MPEG-2 or MPEG-4 format) image data taken by a digital video camera, digital still camera, etc., and MP4 files. By recording as, compatibility is guaranteed such as playback on other digital devices.

An MP4 file basically consists of an mdat box containing encoded stream image data and a moov box containing information related to the stream image data, as shown in Figure 8 (a). ing. The mdat box is further composed of multiple chunks (chunk cN) as shown in Fig. 8 (b), and each chunk is composed of multiple samples (sample sM) as shown in Fig. 8 (d). .. Each sample is encoded with I0, B-2, B-1, P3, ... For sample s1, sample s2, sample s3, samples4, ..., for example, as shown in FIG. 8 (e). Corresponds to MPEG image data.

Here, I0, I1, I2, ..., In are intra-coded (in-frame coding) frame image data, and B0, B1, B2, ..., Bn are referenced from both directions. It is frame image data that is coded (coded between frames). Further, P0, P1, P2, ..., Pn are frame image data that are encoded (inter-frame coding) by reference from one direction (forward direction), and all of them are variable length code data.

As shown in Fig. 8 (c), the moov box consists of an mvhd box consisting of header information in which the creation date and time are recorded, and a trak box containing information related to the stream image data stored in the mdat box. The information stored in the trak box includes the stco box that stores the offset value information for each chunk of the mdat box as shown in Fig. 8 (h), and the inside of each chunk as shown in Fig. 8 (g). There is a stsc box that stores information on the number of samples in, and a stsz box that stores information on the size of each sample as shown in Fig. 8 (f).

Therefore, the amount of data stored in the above stco box, stsc box, and stsz box increases with the amount of recorded image data, that is, the recording time. For example, if an image of 30 frames per second is stored in one chunk every 15 frames and recorded as an MP4 file, the data will be about 1 Mbytes in 2 hours, and a moov box with a capacity of 1 Mbytes is required. ..

When playing this MP4 file, read the moov box of the MP4 file from the recording medium, analyze the above stco box, stsc box, stsz box from the moov box so that you can access each chunk in the mdat box. become.

When recording in the MP4 file format, the stream data grows over time and its size is very large, so it is necessary to write the stream data to a file during recording. However, as mentioned above, since the size of the moov box also increases according to the recording time, the size of the MP4 header is unknown until the recording is completed, so it is not possible to determine the write offset position of the stream data in the file. .. Therefore, when recording with a general video processing device, the flexibility of the MP4 file format is used to support as follows. (1) Place the mdat box at the beginning of the file, and place the moov box after the mdat box at the end of recording (Fig. 9 (a)). (2) As proposed in Patent Document 1, the size of the moov box is determined in advance, the offset position of the mdat box is determined, and recording is performed (Fig. 9 (b)). Even if the recording time is short and the header area is left over, that area is free. Leave as a box. When recording beyond the header size, the header size is maintained at a predetermined size by thinning out the frame number information of the I picture at any time. (3) The pair of moov box and mdat box is divided into multiple parts and arranged (Fig. 9 (c). The second and subsequent header areas are called moof boxes.

The above is the structure of a general MP4 file.

The general playback method of the above-mentioned MP4 file will be described below.

FIG. 10 shows a basic configuration example of a moving image playback device that reproduces a moving image compressed and encoded by the H.264 method.

In FIG. 10, the moving image playback device includes a recording medium 1001, a playback circuit 1002 for reproducing data from the recording medium, a buffer circuit 1003, a variable length decoding circuit 1004, an inverse quantization circuit 1005, an inverse DCT circuit 1006, an adder circuit 1007, and a memory. It has 1008, motion compensation circuit 1009, switch circuit 1010, rearrangement circuit 1011, output terminal 1012, header information analysis circuit 1013, reproduction control circuit 1014, and control signal input terminal 1015.

Next, the flow of the playback process of the moving image playback device in FIG. 10 will be described.

Upon receiving the instruction of the reproduction control circuit 1014, the reproduction circuit 1002 reproduces the MP4 file recorded on the recording medium 1001 and starts supplying the MP4 file to the buffer circuit 1003. At the same time, the reproduction control circuit 1014 controls the header information analysis circuit 1013 to analyze the offset, chunk, and sample information from the stco box, stsc box, and stsz box that indicate the storage status in the mdat of the moov box. In addition, the playback circuit 1002 is controlled so that the stream image data in the mdat box is started to be played back from the recording medium 1001.

The reproduction circuit 1002 reproduces the stream image data of the mdat box of the file recorded on the recording medium 1001 from the start address thereof, and supplies the stream image data to the buffer circuit 1003. The stream image data stored in the buffer circuit 1003 is read out by observing the occupied state of the buffer circuit 1003 and the like, and the stream image data is supplied to the variable length decoding circuit 1004. The variable-length decoding circuit 1004 performs variable-length decoding of the reproduced stream image data supplied from the buffer circuit 1003 and supplies it to the inverse quantization circuit 1005.

The dequantization circuit 1005 dequantizes the variable-length decoded stream image data supplied from the variable-length decoding circuit 1004 and supplies it to the reverse DCT circuit 1006. The inverse DCT circuit 1006 applies the inverse DCT to the inverse quantized data supplied from the inverse quantization circuit 1005 and supplies it to the adder circuit 1007. The adder circuit 1007 adds the reverse DCT data supplied from the reverse DCT circuit 1006 and the data supplied from the switch circuit 1010.

Here, as the stream image data reproduced from the recording medium 1001, first, as shown in FIG. 11, the intra-frame encoded I0 of GOP0 (Group Of Picture) is reproduced. Therefore, the reproduction control circuit 1014 controls to select the terminal a of the switch circuit 1010, and the switch circuit 1010 supplies the data "0" to the adder circuit 1007. The adder circuit 1007 adds the "0" data supplied from the switch circuit 1010 and the reverse DCT data supplied from the reverse DCT circuit 1006, and supplies the reproduced frame F0 to the memory 1008 and the rearrangement circuit 1011. The memory 1008 stores the addition data supplied from the addition circuit 1007.

Next to the in-frame coded data I0 of GOP0, the bidirectional predictive coded picture data B-2 and B-1 are reproduced, and the reproduction procedure up to the inverse DCT circuit 1006 is the above-mentioned in-frame coded data. Since it is the same as the reproduction procedure explained in I0, it is omitted.

The inverse DCT bidirectional predictive coded image data is supplied from the inverse DCT circuit 1006 to the adder circuit 1007. At this time, the reproduction control circuit 1014 controls the switch circuit 1010 so that the movable terminal c of the switch circuit 1010 selects the fixed terminal b, and supplies the data from the motion compensation circuit 1009 to the adder circuit 1007.

The motion compensation circuit 1009 detects the motion vector generated at the time of encoding from the stream image data to be reproduced and recorded in the stream image data, and reproduces the data of the reference block (in this case, since it is the start of recording, it is reproduced. (Only the data from the in-frame coded data F0) is read from the memory 1008 and supplied to the movable terminal c of the switch circuit 1010.

The adder circuit 1007 adds the inverse DCT data supplied from the inverse DCT circuit 1006 and the motion-compensated data supplied from the switch circuit 1010, and rearranges them as the reproduced frames F-2 and F-1. Supply to.

Next, the one-way predictive coded picture data P3 is reproduced, but the reproduction procedure up to the inverse DCT circuit 1006 is the same as the reproduction procedure described in the in-frame coded data I0, and thus is omitted.

The one-way predictive encoded picture data inversely DCTed from the inverse DCT circuit 1006 is supplied to the adder circuit 1007. At this time, the reproduction control circuit 1014 controls the switch circuit 1010 so that the movable terminal c of the switch circuit 1010 selects the fixed terminal b, and supplies the data from the motion compensation circuit 1009 to the adder circuit 1007.

The motion compensation circuit 1009 detects the motion vector generated at the time of coding and recorded in the stream image data from the reproduced stream image data, and the data of the reference block (from the reproduced in-frame encoded data F0). Data) is read from the memory 1008 and supplied to the movable terminal c of the switch circuit 1010.

The adder circuit 1007 adds the reverse DCT data supplied from the reverse DCT circuit 1006 and the motion-compensated data supplied from the switch circuit 1010, and supplies the reproduced frame F3 to the memory 1008 and the rearrangement circuit 1011. The memory 1008 stores the addition data supplied from the addition circuit 1007.

Next, pictures B1 and B2 are played back, but since it is not the frame at the start of recording, it is explained in B-2 and B-1 above except that it is played back from frames F0 and F3 as bidirectional prediction. It is regenerated by following the same procedure. Thus, as explained above, P6, B4, B5, ..., and so on are reproduced in sequence.

The rearrangement circuit 1011 uses frames F0, F-2, F-1, F3, F1, F2, F6, F4, F5, ..., which are sequentially reproduced, as F-2, F-1, F0, F1, Sort in the order of F2, F3, F4, F5, F6, ..., and output to the output terminal 1012.

At the start of file playback, the header information analysis circuit 1013 analyzes offset, chunk, and sample information from the stco box, stsc box, and stsz box that indicate the storage status of the MP4 file moov box in mdat. 1014 works to skip to GOP1 and start the next playback from GOP1.

By the way, in a digital camera with an interchangeable lens, when the lens is removed from the camera body, dust or the like floating in the air may enter the inside of the camera body. In addition, various mechanically operating mechanical parts such as a shutter mechanism are arranged inside the camera, and when these mechanical parts operate, dust such as metal pieces is generated in the camera body. In some cases.

When such foreign matter such as dust and dirt adheres to the surface of an optical low-pass filter or the like arranged in front of the image sensor constituting the image pickup unit of the digital camera, the foreign matter is reflected in the captured image as a shadow. This will reduce the quality of the captured image.

In order to solve such a problem, it is conceivable to correct the shadow of a foreign substance. As a technique applicable to the correction, for example, Patent Document 1 describes image defect correction for correcting a pixel defect of an image sensor. A method has been proposed.

Further, in Patent Document 2, in order to simplify the setting of the position information of the pixel defect, the dust information image is automatically set on the PC side by making the extension of the image file taken in the dust acquisition mode different from the normal image. A method has been proposed in which the image is discriminated and the image to be corrected is corrected using the information. Further, there is also a product in which the above-mentioned dust information is recorded as shooting information in a shooting image file, and the correction target image is corrected later by using the information.<patcit num="1"><text>Japanese Unexamined Patent Publication No. 2003-289495</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 6-105241</text></patcit><patcit num="3"><text>Japanese Unexamined Patent Publication No. 2004-242158</text></patcit><nplcit num="1"><text>ISO / IEC 14496-10, "Advanced Video Coding"</text></nplcit>

<p> However, playing a video file such as the MP4 file described above while correcting the image to be corrected based on the dust information described above has a problem of increasing the amount of memory used for dust correction. is there. In addition, it also leads to a problem that the quality of moving image reproduction is deteriorated due to a decrease in operating speed.</p><p> This is because in the still image reproduction, the still image after the dust correction is reproduced, so that the dust correction processing may be performed once for each image. Even if the dust correction process takes time due to memory restrictions or the dust correction process itself takes time, there is little adverse effect of waiting until the dust correction process is completed because it is a still image playback. ..</p><p> However, in moving image playback, the movement of an image is expressed by continuously playing back a plurality of still images such as 15 frames and 30 frames per second. Therefore, in addition to the general playback processing, 15 frames per second must be processed to correct 15 times of dust for 1 frame, and 30 frames must be processed to correct 30 times of dust for 1 frame. It doesn't become. Further, in the process of correcting the dust for one frame, the process of correcting the dust for the number of dusts is performed.</p><p> In other words, the number of dust correction processes that must be performed per second is 15 frames. Number of dust correction processes = (15 frames) x (number of dust) Will be.</p><p> Also, at 30 frames, Number of dust correction processes = (30 frames) x (number of dust) Will be.</p><p> Therefore, natural moving image playback cannot be performed unless these series of processes are completed within a limited time of one second.</p><p> Therefore, the present invention has been made in view of the above-mentioned problems, and an object of the present invention is to play a moving image file such as an MP4 file while performing dust correction processing using dust information, and to reflect dust or the like on the user. Is to provide high quality video playback corrected for.</p>

<p> In order to solve the above-mentioned problems and achieve the object, the image processing apparatus according to the present invention is an optical device arranged in front of an imaging element in an imaging apparatus that has imaged the moving image data from the input moving image data. A foreign matter information acquisition means for acquiring foreign matter information including the position and size of a foreign matter attached to an element, and a motion vector information acquisition means for acquiring a motion vector of a macro block obtained by dividing the moving image data from the moving image data. In the correction reproduction means for correcting the deterioration of the image quality due to the foreign matter and reproducing the moving image data based on the foreign matter information acquired by the foreign matter information acquisition means, and the moving image data reproduced by the correction reproduction means. It is characterized by comprising a determination means for determining whether or not to perform foreign matter correction reproduction of the macroblock based on a motion vector of the macroblock that matches the position of the foreign matter.</p><p> Further, the image processing method according to the present invention includes, from the input moving image data, the position and size of foreign matter adhering to the optical element arranged in front of the imaging element in the imaging device that has imaged the moving image data. The foreign matter information acquisition step of acquiring foreign matter information, the motion vector information acquisition step of acquiring the motion vector of the macroblock obtained by dividing the moving image data from the moving image data, and the foreign matter acquired in the foreign matter information acquisition step. Based on the information, the correction reproduction step of correcting the deterioration of the image quality due to the foreign matter and reproducing the moving image data, and the movement of the macroblock matching the position of the foreign matter in the moving image data reproduced in the correction reproduction step. It is characterized by including a determination step of determining whether or not to perform foreign matter correction reproduction of the macroblock based on a vector.</p>

<p> According to the present invention, it is possible to play a moving image file such as an MP4 file while performing dust correction processing using dust information, and provide the user with playback of a high-quality moving image in which dust and the like are corrected. It will be possible.</p>

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment) First, the configuration of the image pickup apparatus according to the first embodiment of the present invention will be described with reference to FIG. In the present embodiment, a single-lens reflex type digital still camera will be described as an example of the imaging device. As the image pickup apparatus, the present invention can be applied to other digital video cameras and the like having interchangeable lenses.

As shown in FIG. 12, the image pickup apparatus of this embodiment is mainly composed of a camera body 100 and an interchangeable lens type lens unit 300.

In the lens unit 300, 310 is an imaging lens composed of a plurality of lenses, 312 is an aperture, and 306 is a lens mount that mechanically connects the lens unit 300 to the camera body 100. The lens mount 306 includes various functions for electrically connecting the lens unit 300 to the camera body 100. Reference numeral 320 is an interface for connecting the lens unit 300 to the camera body 100 in the lens mount 306, and 322 is a connector for electrically connecting the lens unit 300 to the camera body 100.

The connector 322 has a function of transmitting control signals, status signals, data signals, etc. between the camera body 100 and the lens unit 300, and also having a function of supplying currents of various voltages. Further, the connector 322 may be configured to transmit not only telecommunications but also optical communication, voice communication and the like.

The 340 is an aperture control unit that controls the aperture 312 based on the photometric information from the photometric control unit 46 in cooperation with the shutter control unit 40 that controls the sitter 12 of the camera body 100, which will be described later. Reference numeral 342 denotes a focus control unit that controls focusing of the image pickup lens 310, and reference numeral 344 is a zoom control unit that controls zooming of the image pickup lens 310.

The 350 is a lens system control circuit that controls the entire lens unit 300. The lens system control circuit 350 includes a memory for storing constants, variables, programs, and the like for operation. Furthermore, it also has a non-volatile memory that holds identification information such as a number unique to the lens unit 300, management information, functional information such as open aperture value and minimum aperture value, focal length, and current and past setting values.

Next, the configuration of the camera body 100 will be described.

106 is a lens mount that mechanically connects the camera body 100 and the lens unit 300, and 130 and 132 are mirrors that guide the light rays incident on the image pickup lens 310 to the optical viewfinder 104 by a single-lens reflex method. The mirror 130 may have either a quick return mirror configuration or a half mirror configuration. Reference numeral 12 is a shutter, and 14 is an image sensor that photoelectrically converts the subject image. The light beam incident on the image pickup lens 310 is guided by the single-lens reflex method through the diaphragm 312, the lens mounts 306 and 106, the mirror 130, and the shutter 12 which are light amount limiting means, and is imaged on the image pickup element 14 as an optical image. An optical element 14a such as an optical low-pass filter is arranged in front of the image sensor 14, and foreign matter such as dust adhering to the surface of the optical element 14a is reflected in the image generated by the image sensor 14. Deteriorates image quality. The present embodiment relates to a technique for suppressing this deterioration in image quality.

Reference numeral 16 denotes an A / D converter that converts an analog signal output from the image sensor 14 into a digital signal. Reference numeral 18 denotes a timing generation circuit that supplies a clock signal and a control signal to the image pickup element 14, the A / D converter 16, and the D / A converter 26, respectively, and is controlled by the memory control circuit 22 and the system control circuit 50.

Reference numeral 20 denotes an image processing circuit, which performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 16 or the data from the memory control circuit 22. Further, the image processing circuit 20 performs predetermined arithmetic processing using the image data output from the A / D converter 16 as needed. TTL (Through the Lens) type autofocus (AF) processing and automatic exposure (AE) for the system control circuit 50 to control the shutter control unit 40 and focus adjustment unit 42 based on the obtained calculation results. Processing and flash pre-emission (EF) processing can be performed. Further, the image processing circuit 20 performs predetermined arithmetic processing using the image data output from the A / D converter 16, and also performs TTL-type auto white balance (AWB) processing based on the obtained arithmetic result. ing.

In this embodiment, the focus adjustment unit 42 and the photometric control unit 46 are exclusively provided. Therefore, the focus adjustment unit 42 and the photometric control unit 46 are used to perform AF processing, AE processing, and EF processing, and the image processing circuit 20 is not used to perform AF processing, AE processing, and EF processing. It doesn't matter. Further, the focus adjustment unit 42 and the photometric control unit 46 are used to perform AF processing, AE processing, and EF processing, and further, AF processing, AE processing, and EF processing are performed using the image processing circuit 20. It may be configured.

Reference numeral 22 denotes a memory control circuit, which controls an A / D converter 16, a timing generation circuit 18, an image processing circuit 20, an image display memory 24, a D / A converter 26, a memory 30, and a compression / decompression circuit 32. The image data output from the A / D converter 16 is written to the image display memory 24 or the memory 30 via the image processing circuit 20, the memory control circuit 22, or only the memory control circuit 22.

24 is an image display memory, 26 is a D / A converter, 28 is an image display unit consisting of a TFT type LCD, etc., and the image data for display written in the image display memory 24 is the D / A converter 26. It is displayed by the image display unit 28 via the image display unit 28. The electronic viewfinder (EVF) function can be realized by sequentially displaying the captured image data using the image display unit 28. Further, the image display unit 28 can arbitrarily turn on / off the display according to the instruction of the system control circuit 50, and when the display is turned off, the power consumption of the camera body 100 can be significantly reduced. it can.

Reference numeral 30 denotes a memory for storing captured still images or moving images, which has a sufficient storage capacity for storing a predetermined number of still images or a predetermined amount of moving images. As a result, even in the case of continuous shooting or panoramic shooting in which a plurality of still images are continuously shot, it is possible to write a large amount of images at high speed to the memory 30. Further, when shooting a moving image, it is used as a frame buffer for images continuously written at a predetermined rate. Further, the memory 30 can also be used as a work area of the system control circuit 50.

Reference numeral 31 denotes a dust removing circuit for removing dust contained in the image data by image processing by using the dust information stored in the non-volatile memory 56 described later and the optical information obtained from the lens unit 300.

Reference numeral 32 denotes a compression / decompression circuit that compresses / decompresses image data using a known compression method. The compression / decompression circuit 32 reads the image stored in the memory 30 and performs compression processing or decompression processing, and writes the processed data to the memory 30 again.

Reference numeral 40 denotes a shutter control unit, which controls the shutter 12 in cooperation with the aperture control unit 340 that controls the aperture 312 based on the photometric information from the photometric control unit 46. 42 is a focus adjustment unit for performing AF (autofocus) processing. An image is formed as an optical image by incident a light beam incident on the image pickup lens 310 in the lens unit 300 through a diaphragm 312, lens mounts 306 and 106, a mirror 130, and a focus adjustment sub-mirror (not shown) in a single-lens reflex manner. The in-focus state of the image is measured.

Reference numeral 46 denotes a photometric control unit for performing AE (automatic exposure) processing. A light beam incident on the image pickup lens 310 in the lens unit 300 is incident as an optical image through a diaphragm 312, lens mounts 306 and 106, a mirror 130, and a photometric sub-mirror (not shown) in a single-lens reflex manner. Measure the exposure of the imaged image. 48 is a flash, which also has an AF auxiliary light projection function and a flash dimming function. The metering control unit 46 also has an EF (flash dimming) processing function in cooperation with the flash 48.

Further, AF control may be performed using the measurement result by the focus adjusting unit 42 and the calculation result obtained by calculating the image data from the A / D converter 16 by the image processing circuit 20. Further, the exposure control may be performed using the measurement result by the photometric control unit 46 and the calculation result obtained by calculating the image data from the A / D converter 16 by the image processing circuit 20.

50 is a system control circuit that controls the entire camera body 100, and incorporates a well-known CPU and the like. Reference numeral 52 denotes a memory for storing constants, variables, programs, etc. for the operation of the system control circuit 50.

Reference numeral 54 denotes a notification unit for notifying the outside of an operating state or a message using characters, images, voices, etc. according to the execution of a program in the system control circuit 50. As the notification unit 54, for example, a display unit that visually displays by LCD or LED, a sounding element that performs voice notification, or the like is used, and is composed of a combination of one or more of these. In particular, in the case of the display unit, it is installed in one or more places near the operation unit 70 of the camera body 100, which is easy to see. Further, a part of the function of the notification unit 54 is installed in the optical finder 104.

Among the display contents of the notification unit 54, the following are displayed on the image display unit 28 such as the LCD. First, there are displays related to the shooting mode, such as single shooting / continuous shooting display and self-timer display. In addition, there are displays related to recording such as a compression rate display, a recording pixel number display, a recording number display, and a remaining shooting possible number display. In addition, there are displays related to shooting conditions such as shutter speed display, aperture value display, exposure compensation display, dimming compensation display, external flash emission amount display, and red-eye mitigation display. In addition, there are macro shooting display, buzzer setting display, battery level display, error display, information display by multi-digit numbers, and attachment / detachment status display of recording medium 200 and PC210. Further, the attachment / detachment state display of the lens unit 300, the communication I / F operation display, the date / time display, the display showing the connection state with the external computer, and the like are also performed.

Further, among the display contents of the notification unit 54, those to be displayed in the optical finder 104 include, for example, the following. Focusing display, shooting preparation completion display, camera shake warning display, flash charging display, flash charging completion display, shutter speed display, aperture value display, exposure compensation display, recording medium writing operation display, and the like.

Reference numeral 56 denotes a non-volatile memory in which a program or the like described later is stored and which can be electrically erased and recorded. For example, EEPROM or the like is used.

60,62,64,66,68,70 are operation means for inputting various operation instructions of the system control circuit 50, and are a single switch, dial, touch panel, pointing by line-of-sight detection, voice recognition device, etc. It consists of multiple combinations.

Here, a specific description of these operating means will be given.

60 is a mode dial switch, and each function shooting mode such as automatic shooting mode, program shooting mode, shutter speed priority shooting mode, aperture priority shooting mode, manual shooting mode, depth of focus priority (depth) shooting mode can be switched and set. it can. In addition, each function shooting mode such as portrait shooting mode, landscape shooting mode, close-up shooting mode, sports shooting mode, night view shooting mode, and panoramic shooting mode can be switched and set. In addition, the transition to the moving image shooting mode, which is also a feature of the present embodiment, can be switched with this mode dial.

62 is a shutter switch SW1 that turns on during the operation of a shutter button (for example, half-pressed) (not shown), and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing.

Reference numeral 64 denotes a shutter switch SW2, which is turned on when the operation of a shutter button (for example, fully pressed) (not shown) is completed, and instructs the start of a series of processes including exposure processing, development processing, and recording processing. First, in the exposure processing, the signal read from the image sensor 14 was written to the memory 30 via the A / D converter 16 and the memory control circuit 22, and further, the calculation in the image processing circuit 20 and the memory control circuit 22 was used. Development processing is performed. Further, in the recording process, the image data is read from the memory 30, compressed by the compression / decompression circuit 32, and written to or transmitted to the recording medium 200 or the PC 210.

Reference numeral 66 denotes a playback switch, which instructs the start of the playback operation of reading the image shot in the shooting mode from the memory 30, the recording medium 200, or the PC 210 and displaying it on the image display unit 28. The playback switch 66 can also set each function mode such as a playback mode, a multi-screen playback / erase mode, and a PC connection mode.

68 is a single shooting / continuous shooting switch. When the shutter switch SW2 (64) is pressed, the single shooting mode that shoots one frame and puts it in the standby state and the continuous shooting mode while the shutter switch SW2 (64) is pressed are continuous. You can set the continuous shooting mode to continue shooting.

Reference numeral 70 denotes an operation unit composed of various buttons and a touch panel. As an example, live view start / stop button, video recording start / stop button, menu button, set button, multi-screen playback page break button, flash setting button, single shot / continuous shooting / self-timer switching button, menu move + (plus) ) Button, menu move-including (minus) button. Furthermore, it also includes a playback image move + (plus) button, a playback image move- (minus) button, a shooting image quality selection button, an exposure compensation button, a dimming compensation button, an external flash emission amount setting button, a date / time setting button, and the like. By providing a rotary dial switch for each function of the plus button and the minus button, it is possible to select a numerical value or a function more easily.

In addition, there is an image display ON / OFF switch that sets ON / OFF of the image display unit 28, and a quick review ON / OFF switch that sets a quick review function that automatically plays back the image data taken immediately after shooting. Further, there is a compression mode switch which is a switch for selecting a compression rate of JPEG compression or for selecting a RAW mode in which a signal of an image sensor is digitized as it is and recorded on a recording medium. In addition, there is an AF mode setting switch that can set one-shot AF mode and servo AF mode. In the one-shot AF mode, the autofocus operation is started when the shutter switch SW1 (62) is pressed, and once it is in focus, the in-focus state is maintained. In servo AF mode, the autofocus operation continues continuously while the shutter switch SW1 (62) is pressed. Further, as described later, it includes a setting switch capable of setting a dust information acquisition mode that captures a dust detection image and acquires dust information.

72 is a power switch, and the power on and power off modes of the camera body 100 can be switched and set. In addition, the power on / off settings of various accessory devices such as the lens unit 300 connected to the camera body 100, the external flash 112, the recording medium 200, and the PC 210 can be switched and set.

Reference numeral 80 denotes a power supply control unit, which is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching a block to be energized, and the like. The power supply control unit 80 detects whether or not a battery is installed, the type of battery, and the remaining battery level, controls the DC-DC converter based on the detection result and the instruction of the system control circuit 50, and requires the required voltage. Supply to each part including the recording medium for the period.

82 and 84 are connectors, 86 are primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, Li-ion batteries and Li polymer batteries, and power supply units consisting of AC adapters and the like.

90 and 94 are interfaces with recording media such as memory cards and hard disks and PCs, and 92 and 96 are connectors for connecting with recording media such as memory cards and hard disks and PCs. Reference numeral 98 denotes a recording medium attachment / detachment detection circuit that detects whether or not the recording medium 200 or PC 210 is attached to the connector 92 and / or 96.

In the present embodiment, the interface and the connector to which the recording medium is attached are described as having two systems, but the interface and the connector to which the recording medium is attached may be configured to have any number of systems, one or more. Further, the configuration may include a combination of interfaces and connectors of different standards.

As the interface and the connector, those conforming to the standards of various storage media can be used. For example, PCMCIA (Personal Computer Memory Card International Association) cards, CF (Compact Flash (registered trademark)) cards, SD cards, etc. When interfaces 90 and 94 and connectors 92 and 96 are configured using standards such as PCMCIA cards and CF cards, various communication cards can be connected. Communication cards include LAN cards, modem cards, USB (Universal Serial Bus) cards, and IEEE (Institute of Electrical and Electronic Engineers) 1394 cards. Besides, P1284 card, SCSI (Small Computer System) Interface) Card, PHS, etc. By connecting these various communication cards, it is possible to transfer image data and management information attached to the image data to and from other peripheral devices such as a computer and a printer.

Reference numeral 104 denotes an optical finder, which can guide a light beam incident on the image pickup lens 310 through a diaphragm 312, lens mounts 306, 106, mirrors 130, and 132 by a single-lens reflex method, and image and display it as an optical image. it can. As a result, it is possible to take a picture using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28. Further, in the optical viewfinder 104, some functions of the notification unit 54, such as focusing state, camera shake warning, flash charging, shutter speed, aperture value, exposure compensation, and the like are displayed.

Reference numeral 112 denotes an external flash device mounted via the accessory shoe 110.

Reference numeral 120 denotes an interface for connecting the camera body 100 to the lens unit 300 in the lens mount 106.

Reference numeral 122 denotes a connector that electrically connects the camera body 100 to the lens unit 300. Further, whether or not the lens unit 300 is attached to the lens mount 106 and the connector 122 is detected by a lens attachment / detachment detection unit (not shown). The connector 122 transmits control signals, status signals, data signals, etc. between the camera body 100 and the lens unit 300, and also has a function of supplying currents of various voltages. Further, the connector 122 may be configured to transmit not only telecommunications but also optical communication, voice communication and the like.

Reference numeral 200 denotes a recording medium such as a memory card or a hard disk. The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, or the like, an interface 204 with the camera body 100, and a connector 206 for connecting to the camera body 100.

As the recording medium 200, a PCMCIA card, a memory card such as CompactFlash (registered trademark), a hard disk, or the like can be used. Of course, it may be composed of a micro DAT, a magneto-optical disk, an optical disk such as a CD-R or a CD-RW, a phase change type optical disk such as a DVD, or the like.

The 210 is a PC, and has a recording unit 212 composed of a magnetic disk (HD) or the like, an interface 214 with the camera body 100, and a connector 216 for connecting to the camera body 100. Interface 94 includes USB, IEEE1394, etc., but is not particularly limited.

Next, a reproduction process for correcting the influence of dust on the optical element 14a such as the low-pass filter and the cover glass arranged in front of the image pickup element in the image pickup apparatus having the above configuration will be described.

In the present embodiment, first, a dust detection image for obtaining dust information (foreign matter information) is taken, dust data is extracted, and dust data is added to a normal image taken thereafter. The method of playing back while correcting dust on a PC or the like is shown. Here, the dust detection image is preferably an image in which the brightness surface is as uniform as possible, but it is desirable that the image can be easily photographed in a familiar place, and therefore strict uniformity is not required. For example, it is assumed that a blue sky or a white wall surface will be photographed. In addition, in order to describe the features of this embodiment, the operation in the MP4 file format, which is mainly a moving image file, will be described in detail.

FIG. 13 is a flowchart showing processing in the image pickup apparatus when acquiring dust information in the present embodiment.

First, in step S1301, it is determined by the operation unit 70 whether or not the dust information acquisition mode is selected. The determination in step S1301 is repeated until the dust information acquisition mode is selected, and when the dust information acquisition mode is selected, the process proceeds to step S1302 to determine whether the shutter switch SW1 (62) is turned on. If it is OFF, the process returns to step S1301 and the above process is repeated.

On the other hand, if it is ON, proceed to step S1303 and set the aperture, ISO value, shutter speed, and other shooting-related parameters.

The parameters set here are shown in FIG. The aperture should be set to a narrow aperture such as F22. The lens unit 300 connected to the lens mount 306 may be used for shooting in the narrowest position within the settable range. The reason why the aperture is stopped down in this way is that dust is not normally attached to the surface of the image sensor 14, but is attached to the protective glass that protects the image sensor 14 and the optical filter that is placed closer to the subject than the image sensor. This is because the image pickup state differs depending on the aperture value of the lens unit 300. Therefore, if the aperture is close to the open value, the image will be blurred and an appropriate image for detecting dust cannot be obtained. Therefore, it is preferable to shoot with the aperture stopped down as much as possible.

Returning to the explanation of the flowchart of FIG. 13, by this time, the photographer will aim the image pickup device at a uniform brightness surface such as a white wall as much as possible and operate the shutter switch SW2 (64).

In step S1304, it is determined whether or not the shutter switch SW2 (64) is turned on. If it is OFF, the process returns to step S1302 and the shutter switch SW1 (62) is judged. If it is ON, the process proceeds to step S1305.

In step S1305, a dust detection image (photographing a uniform brightness surface) is performed, and the image data is captured in the memory 30. Next, in step S1306, dust information is acquired from the image data stored in the memory 30.

Here, the acquisition of garbage information will be described. Specifically, the position (coordinates) and size of the dust area are obtained from the captured dust detection image. First, the area of the captured dust detection image is divided into a plurality of blocks, the maximum brightness Lmax and the average brightness Lave in the block are calculated, and the threshold value T1 in the block is calculated using the following equation.

T1 = Lave × 0.6 + Lmax × 0.4 Next, the pixels that do not exceed the threshold value T1 are designated as dust pixels, and the isolated regions formed by the dust pixels are designated as one dust region di (i = 0,1, ..., n).

FIG. 15 is a diagram showing an outline of the dust area size calculation. As shown in FIG. 15, for each dust area, the maximum value Xmax and the minimum value Xmin of the horizontal coordinates of the pixels constituting the dust area and the maximum value Ymax and the minimum value Ymin of the vertical coordinates are obtained, and the dust area di The radius ri representing the size of is calculated by the following formula.

ri = [ {(Xmax-Xmin)²+ (Ymax-Ymin)²}] / 2 In addition, the center coordinates (Xdi, Ydi) at this time are approximately Xdi = (Xmax + Xmin) / 2 Ydi = (Ymax + Ymin) / 2 It shall be sought in. The position (coordinates) and radius obtained in this way are recorded as a dust information profile.

This garbage information profile has a structure as shown in FIG. As shown in FIG. 16, the dust information profile stores lens information, dust position, and size information at the time of taking an image for dust detection. More specifically, the actual aperture value (F value) at the time of capturing the detected image and the lens pupil position at that time are stored as the lens information at the time of capturing the detected image. Subsequently, the number of detected dust areas (integer value) is stored in the storage area, and subsequently, the parameters of each specific dust area are repeatedly stored as many as the number of dust areas. The dust area parameters are a set of three numbers: the dust radius (eg 2 bytes), the x-coordinate of the center in the effective image area (eg 2 bytes), and the same y-coordinate of the center (eg 2 bytes).

Especially for video files such as the MP4 file mentioned above, the garbage information profile is stored in the mvhd box consisting of the header information in the moov box or the mvhd box in the moof box.

In the present embodiment, the position and size of the detected dust are converted into data as dust information and described in the image file, but the present invention is not limited to this. For example, it may be configured to record the bitmap data indicating the dust area for the entire image as it is. Further, a garbage information file in which the above garbage information profile is filed may be separately created, and the image file may have only the link information for this garbage information file. Therefore, the data format of garbage information is not particularly limited. Further, as a method for detecting dust, when dust adheres to the optical element 14a in front of the image sensor 14, the amount of light incident on the pixel corresponding to the dust adhesion position may be reduced. For example, by comparing each pixel data with a preset brightness value, it is possible to detect a pixel in which the amount of incident light is reduced. If it is not expected to capture a uniform luminance surface, the determination may be made not only by the threshold value of the luminance but also by the difference with the pixel output in the vicinity. The dust detection method described above is an example, and the present invention does not particularly limit the method for detecting dust.

The acquired dust information is stored in the non-volatile memory 56 in step S1307, and the process for acquiring the dust information is completed. Here, the dust information is stored in the non-volatile memory 56 in the normal shooting performed from the acquisition of the dust information to the next acquisition of the dust information, and the dust information is continuously added to the obtained image data. Because. Therefore, it does not have to be a non-volatile memory if the configuration is such that the photographer is requested to acquire dust information every time the power is turned on.

Further, since the shooting operation in the dust information acquisition mode aims to acquire dust information, in the present embodiment, the captured image itself is not compressed and recorded on the recording medium 200. This is to prevent unnecessary image data for the photographer from wasting the capacity of the recording medium 200, but the image data may be compressed and then stored in the recording medium 200 in the same manner as a normal image. At that time, some modification such as changing the extension may be performed.

Next, the flow of dust correction reproduction (foreign matter correction reproduction) processing will be described. In the following, a case where the dust correction reproduction processing is performed not in the digital camera main body but on a separately prepared image processing device will be described.

FIG. 17 is a diagram showing an outline of the system configuration of the image processing apparatus. The CPU1701 controls the operation of the entire system and executes programs stored in the primary storage unit 1702. The primary storage unit 1702 is mainly a memory, and reads and stores programs and the like stored in the secondary storage unit 1703. The secondary storage unit 1703 corresponds to, for example, a hard disk. Generally, the capacity of the primary storage unit is smaller than the capacity of the secondary storage unit, and programs and data that cannot be stored in the primary storage unit are stored in the secondary storage unit. In addition, data that must be stored for a long time is also stored in the secondary storage unit.

In the present embodiment, the program is stored in the secondary storage unit 1703, read into the primary storage unit 1702 when the program is executed, and executed by the CPU 1701. The input device 1704 corresponds to, for example, a mouse and a keyboard used for controlling a system, a card reader, a scanner, a film scanner, and the like necessary for inputting image data. The output device 1705 may be, for example, a monitor or a printer. Although various other forms of the configuration of this device can be considered, the description thereof will be omitted because it is not the main purpose of the present invention.

The image processing device is equipped with an operating system that can execute multiple programs in parallel, and the operator can operate the programs running on this device using the GUI.

FIG. 18 is a diagram showing a GUI (Graphical User Interface) of an image editing program in an image processing device. The window has a close button 1800 and a title bar 1801, and you can exit the program by pressing the close button. Specify the image to be corrected and played by dragging and dropping the file into the image display area 1802. When the correction playback target image is determined, the file name is displayed in the title bar 1801, and then the target image is Fit-displayed in the image display area 1802. When the correction playback execution button 1803 is pressed, the dust correction playback processing described later is executed and displayed in the image display area 1802.

FIG. 19 shows the flow of dust correction reproduction processing in the image processing apparatus.

First, in step S1901, a moving image file to which dust position correction data is added is taken into an image processing device from a recording medium 200 in or removed from the digital camera. Then, the stream image data to be displayed is stored in the primary storage unit 1702 or the secondary storage unit 1703. As mentioned above, the method of acquiring stream image data is to read the moov box of the MP4 file from the recording medium, analyze the above stco box, stsc box, and stsz box from the moov box, and then transfer to each chunk in the mdat box. It is done by accessing.

Next, in step S1902, a dust correction block selection process is performed to select a block to be dust-corrected and reproduced from each frame of the moving image displayed in the image display area 1802 from the moving image file. The dust correction block selection process will be described later with reference to FIG.

Next, in step S1903, display processing is performed. The display process is to display an image in the image display area 1802, but since this process is a known technique described in the background technique, detailed description thereof will be omitted.

Next, in step S1904, it is determined whether or not the processing of all streams has been completed. If it is determined that the processing of all streams has been completed, the dust correction reproduction processing is terminated. If the processing of all streams has not been completed, the process returns to the dust correction frame selection process, and these are repeated until all streams are completed.

In the above flow, the dust correction reproduction process is performed.

Next, FIG. 20 shows the flow of the above-mentioned dust correction block selection process.

First, in step S2001, dust position correction data is extracted from moov or moof (foreign matter information acquisition). As described above, in the present embodiment, the garbage information profile is stored in the mvhd box composed of the header information in the moov box or the mvhd box in the moof box, but the data format of the garbage information is not particularly limited. The details of the other acquisition methods will not be described in particular, but it is important that the processing for extracting the dust position correction data is performed.

Next, from the dust position correction data extracted in step S2001, first, the dust correction data is extracted, and the coordinate sequence Di (i = 1,2, ... n) and the radius sequence Ri (i = 1,2, ... n) are extracted. ., n), obtain the aperture value f1 and the lens pupil position L1 (step S2002). Furthermore, the aperture value f2 at the time of shooting and the pupil position L2 of the lens are acquired. Here, Ri is the size of dust at the coordinate Di calculated in step S1306 of FIG. In step S2002, Di is converted by the following equation. Here, d is the distance from the center of the image to the coordinates Di, and H is the distance between the surface of the image sensor 14 and dust. The coordinate Di'after conversion and the radius Ri'after conversion are defined by, for example, the following equation.

Di'(x, y) = (L2 × (L1-H) × d / ((L2-H) × L1)) × Di (x, y) Ri'= (Ri × f1 / f2 + 3) (1) The unit here is a pixel, and "+3" for Ri'is a margin amount.

As a result, the position and size of dust in the image can be known from the dust position correction data.

Next, the stream image data acquired in step S1901 is decoded (reproduced) (step S2003). In this method, stream image data is decoded (reproduced) according to the basic configuration and procedure as described above, but in this configuration, the CPU 1701 and the primary storage unit 1702 mainly correspond to this part.

During the decoding performed in step S2003, the motion vector generated at the time of encoding and recorded in the stream image data is detected from the played stream image data, and the reference block data is read from the memory 1008. Processing is done. Therefore, in the dust position macroblock / motion vector information acquisition process in step S2004, the macroblock / motion vector information that includes (matches) dust from the dust position and size obtained in step S2002 is associated with the dust and is primary. It is stored in the storage unit 1702.

Next, in step S2005, it is determined from which macroblock the motion vector has the motion vector of what size from the acquired dust information and the macroblock / motion vector information associated with the dust information. If it is determined that the motion vector of the macroblock that matches the dust position is smaller than the threshold value (Mv), dust in the area indicated by the coordinates Di'and radius Ri'is detected in step S2006, and if necessary. And apply the interpolation process. The details of the interpolation process will be described later. On the other hand, if it is determined to be equal to or higher than the threshold value (Mv), the interpolation process of step S2006 is not performed and the process proceeds to step S2007. Here, the threshold value (Mv) may be a statically determined value. It may be something that can be dynamically selected and changed depending on the operating environment of playback. Of course, the value may be arbitrarily determined by the user.

To make such a judgment, the stream image data determines the predicted image using the optimum motion vector based on the motion detection result for the image data of different frames, and the motion vector is entropy together with the difference image data. Since the coded output bit stream is formed, it can be determined that the larger the motion vector, the larger the macroblock portion between the frames and the more violent the motion. In other words, the threshold value (Mv) can be said to be a number that serves as a measure of the magnitude of movement of a macroblock between frames.

FIG. 21 schematically shows the relationship between the motion vector and the reference frame. The A block of the frame 2100 refers to the a block from the reference frame 2101, and shows the case where the value of the motion vector from the a block to the A block is Mv = left 10 and there is no prediction error.

Further, the B block of the frame 2100 refers to the reference frames 2101 to the b block, and the value of the motion vector from the b block to the B block is set to Mv = 0, and there is no prediction error.

This means that what was on the a block moved to the A block. It also shows that what was on the b block stayed on the B block. It can be seen that the garbage 2102 on the A block is garbage that is confused by movement. In addition, it can be seen that the dust 2103 on the B block is the dust in the non-moving part.

If there is dust on a relatively uniform and static screen such as a blue sky or a white wall, the reflection of dust is conspicuous, whereas even if there is dust on a moving screen, the reflection is conspicuous. It is hard to become a thing. This is because it is difficult for the human eye to catch small dust on a screen that changes rapidly.

Further, in the present embodiment, by selecting and determining only the motion vector of the macroblock that matches the position of the dust, it is not necessary to make the determination by the motion vector of all the macroblocks.

From the above, by determining whether the size of the motion vector of the macroblock that matches the dust position is smaller or larger than a certain threshold value (Mv), the macroblock and dust correction processing that should be dust-corrected can be determined. It is possible to determine a macroblock that does not have to be performed.

Next, in step S2007, it is determined whether or not the dust removal process has been applied to all the coordinates, and if the process is completed for all the coordinates, the process ends, and if not, the process returns to step S2005.

The above is the flow of the dust correction block selection process.

Next, the details of the interpolation process of the dust area will be described. FIG. 22 shows a flowchart showing the flow of the interpolation routine. First, in step S2201, the dust area is determined. The garbage area is an area that satisfies all of the following conditions. (1) Using the average brightness Yave and maximum brightness Ymax of the pixels included in the center coordinates Di'and radius Ri'(Di', Ri' obtained by equation (1)) calculated in step S2002 in Fig. 20, the following Area darker than the threshold value T2 obtained by the equation.

T2 = Yave × 0.6 + Ymax × 0.4 (2) The area that does not touch the circle with the above center coordinates Di'and radius Ri'. (3) An region in which the radius value calculated by the same method as in step S1306 in FIG. 13 is X1 pixel or more and less than X2 pixel with respect to the isolated region composed of the low-luminance pixels selected in (1). (4) Area containing the center coordinate Di of the circle.

In this embodiment, X1 has 3 pixels and X2 has 30 pixels. By doing so, it becomes possible to treat only a small isolated area as a garbage area. Further, when the lens pupil position cannot be accurately acquired, the condition (4) may have a width. For example, if the region of interest includes the coordinates in the range of ± 3 pixels in the X and Y directions from the coordinates Di, it can be determined that the region is a dust region.

In step S2202, if there is such an area, the process proceeds to step S2203, dust area interpolation is performed, and if it does not exist, the process ends. The dust area interpolation process executed in step S2203 is performed by a known missing area interpolation method. Known defective region interpolation methods include, for example, pattern substitution disclosed in JP-A-2001-223894. In Japanese Patent Application Laid-Open No. 2001-223894, the defective region is specified by using infrared light, but in the present embodiment, the dust region detected in step S2201 is treated as the defective region, and the dust region is treated as the surrounding normal pixel by pattern substitution. Interpolate with. For pixels that cannot be filled by pattern replacement, p normal pixels are selected in the order closest to the pixels to be interpolated from the image data after pattern interpolation, and the average color is used for interpolation.

The above is the flow of the interpolation process of the dust area.

As described above, according to the present embodiment, resources such as memory by improving the operation speed when playing back a moving image file such as an MP4 file while performing dust correction processing using dust information, and memory by dust correction processing. It is possible to reduce the amount of usage and provide high-quality video reproduction in which the reflection of conspicuous dust and the like is corrected for the user.

(Second embodiment) Since the main components of the image processing apparatus according to the second embodiment of the present invention are basically the same as those in FIG. 17, the description of the same components will be omitted, and only the different components will be scratched and described.

FIG. 23 shows the dust correction block selection process in the second embodiment, which is different from the first embodiment.

First, dust position correction data is extracted from the moov or moof containing the selected frame (step S2301), and the dust size is obtained in the same manner as in the first embodiment (step S2302). Next, in step S2303, the decoding process is performed in the same manner as in the first embodiment, and in step S2304, the dust position macroblock / motion vector information acquisition process is performed.

In step S2305, it is determined whether the size of the dust acquired in steps S2301 and S2302 is smaller than the specified threshold value (Y). Here, the threshold value (Y) may be a statically determined value or a value that can be dynamically selected depending on the operating environment of reproduction. Of course, the value may be arbitrarily determined by the user.

If the dust is large, the reflection of the dust will be noticeable. Therefore, by judging whether the size of the dust is smaller or larger than a certain threshold value (Y), it is necessary to correct the dust. You can make a judgment.

If it is determined that the dust size is smaller than the threshold value (Y), the process proceeds to step S2306, and as in the first embodiment, it is determined not to correct the dust from the motion vector of the macroblock that matches the dust position. , The details are the same as those in the first embodiment, and thus are omitted.

On the other hand, if the dust size is equal to or larger than the threshold value (Y) in step S2305 and it is determined that the dust should be corrected, the dust on the macroblock is required in step S2307 regardless of the size of the motion vector. Interpolation processing is applied accordingly. The interpolation process has already been described in the first embodiment and will be omitted.

As described above, when the dust is larger than a certain size, at least the dust is corrected regardless of the motion vector of the macroblock that matches the dust position. As a result, video files such as MP4 files can be played back while performing dust correction processing using dust information, improving the operating speed, reducing resource usage such as memory by dust correction processing, and for users. It is possible to provide high-quality video reproduction in which the reflection of conspicuous dust and the like is corrected.

(Third embodiment) Since the main components of the image processing apparatus according to the third embodiment of the present invention are basically the same as those in FIG. 17, the description of the same components will be omitted, and only the different components will be scratched and described.

Further, it is very similar to the dust correction block selection process described in the second embodiment, and different parts will be scraped and described with reference to FIG. 24.

FIG. 24 shows the dust correction block selection process of the third embodiment, which is different from the first and second embodiments.

In the second embodiment, the size of dust is acquired in steps S2301 and S2302, but in the third embodiment, the position of dust is acquired in steps S2401 and S2402 by the same method.

Next, in step S2403, decoding processing is performed in the same manner as in the first and second embodiments, and in step S2404, dust position macroblock / motion vector information acquisition processing is performed.

Next, in step S2405, the position of the dust acquired in steps S2401 and S2402 is compared with the specified threshold value (Z). Here, the threshold value (Z) is a value that expresses an area with position information, and the determination is made as to whether or not the position of the acquired dust is within the area defined by the threshold value (Z). Here, the threshold value (Z) may be a statically determined value or a value that can be dynamically selected depending on the operating environment of reproduction. Of course, the value may be arbitrarily determined by the user.

When the dust position is in the center or the position that the user pays attention to, the reflection of dust is conspicuous. Therefore, if the value of the threshold value (Z) is set to the information of the center of the screen or the position of interest of the user, the dust correction can be performed by determining whether or not the dust position is within the threshold value (Z). You can judge whether it is garbage that should be done. The threshold value (Z) is also effective when used in the enlarged display.

If it is determined that there is no dust within the threshold value (Z), the process proceeds to step S2406, and the dust correction is determined from the motion vector of the macro block that matches the dust position as in the first and second embodiments. However, since the details have already been described, they will be omitted.

On the other hand, if there is dust within the threshold value (Z) in step S2405 and it is determined that dust should be corrected, the dust on the macroblock is required in step S2407 regardless of the size of the motion vector. Interpolation processing is applied accordingly. The interpolation process has already been described in the first and second embodiments, and will be omitted.

As described above, if the dust is within the specified area, at least the dust is corrected regardless of the frame data size. As a result, video files such as MP4 files can be played back while performing dust correction processing using dust information, improving the operating speed, reducing resource usage such as memory by dust correction processing, and for users. It is possible to provide high-quality video reproduction in which the reflection of conspicuous dust and the like is corrected.

(Fourth Embodiment) Since the main components of the image processing apparatus according to the fourth embodiment of the present invention are basically the same as those in FIG. 17, the description of the same components will be omitted, and only the different components will be scratched and described.

FIG. 25 shows the dust correction block selection process of the fourth embodiment, which is different from the first to third embodiments.

In the second embodiment, the size of dust is acquired in steps S2301 and S2302, but in the fourth embodiment, the number of dust is acquired in steps S2501 and S2502 by the same method.

Next, in step S2503, decoding processing is performed in the same manner as in the first to third embodiments, and in step S2504, dust position macroblock / motion vector information acquisition processing is performed.

Next, in step S2505, the number of dusts acquired in steps S2501 and S2502 is compared with the specified threshold value (N). Here, the threshold value (N) may be a statically determined value or a value that can be dynamically selected depending on the operating environment of reproduction. Of course, the value may be arbitrarily determined by the user.

If it is equal to or less than the threshold value (N), the process proceeds to step S2507 as it is, and the dust correction is determined from the motion vector of the macroblock that matches the dust position as in the first to third embodiments. Since it has already been described, it will be omitted.

On the other hand, if it is determined that the threshold value (N) is greater than the threshold value (N), a process of lowering the threshold value (Mv) (process of changing the determination standard) is performed in step S2506. The value to be reduced is arbitrary, but it is also good to manage it with something like a table. In other words, if the number is 100 or more and 200 or less, the reduction value is Mv5, and if the number is 200 or more and 300 or less, the reduction value is Mv10. Since the subsequent processing is the same as that of the first embodiment, it will be omitted.

As described above, the threshold value (Mv) is changed according to the number of dusts in consideration of the increase in the memory usage due to the dust correction processing and the dust correction processing due to the large number of dusts. As a result, it is possible to improve the operation speed, reduce the resource usage such as memory by the dust correction processing, and provide the user with the reproduction of a high-quality moving image in which the reflection of dust and the like that is conspicuous is corrected.

(Fifth Embodiment) In the first to fourth embodiments, a case where dust correction reproduction processing is performed on a separately prepared image processing device has been described. However, by arranging a similar image processing device in the image pickup device and performing the same processing, a moving image file such as an MP4 file can be played back on the image pickup device while performing dust correction processing using dust information. You can also. In that case, the effect is that it is possible to improve the operation speed, reduce the resource usage such as memory by the dust correction processing, and provide the user with the reproduction of high-quality video in which the reflection of the dust that is conspicuous is corrected. Is obtained.

(Other embodiments) The object of each embodiment is also achieved by the following methods. That is, a storage medium (or recording medium) on which the program code of the software that realizes the functions of the above-described embodiment is recorded is supplied to the system or the device. Then, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following cases are included in the present invention. That is, based on the instruction of the program code, the operating system (OS) or the like running on the computer performs a part or all of the actual processing, and the processing realizes the function of the above-described embodiment.

Further, the following cases are also included in the present invention. That is, the program code read from the storage medium is written in the memory provided in the function expansion card inserted in the computer or the function expansion unit connected to the computer. After that, based on the instruction of the program code, the function expansion card, the CPU provided in the function expansion unit, or the like performs a part or all of the actual processing, and the function of the above-described embodiment is realized by the processing.

When the present invention is applied to the storage medium, the program code corresponding to the procedure described above will be stored in the storage medium.

<figref num="1">It is a figure which shows the prior art and shows the structure of the image processing apparatus.</figref><figref num="2">It is a schematic diagram which shows the appearance that the image data was divided into macroblocks.</figref><figref num="3">It is a figure which shows the general macroblock partition.</figref><figref num="4">It is a figure which shows the prior art and shows an example of the reference list at the time of encoding a picture P21.</figref><figref num="5">It is a figure which shows the prior art and shows an example of the reference list at the time of encoding a picture P24.</figref><figref num="6">It is a figure which shows the conventional technique and shows the state of change of a reference list for each picture.</figref><figref num="7">It is a figure which shows the state of the change of the reference list when the B picture is added to the reference list, showing the conventional technique.</figref><figref num="8">It is a figure for demonstrating the structure of an MP4 file.</figref><figref num="9">It is a figure which shows the structure example of the MP4 file.</figref><figref num="10">It is a block diagram for showing a prior art and explaining a reproduction circuit.</figref><figref num="11">It is a figure for demonstrating the prior art, and explaining the frame order of coding.</figref><figref num="12">It is a block diagram which shows the structure of the image pickup apparatus which concerns on 1st Embodiment of this invention.</figref><figref num="13">It is a flowchart which shows the process in the image pickup apparatus at the time of acquiring dust information in 1st Embodiment.</figref><figref num="14">It is a figure which shows the setting example of the parameter related to photography at the time of acquiring dust information in 1st Embodiment.</figref><figref num="15">It is a figure which shows the outline of the dust area size calculation performed in step S1306 of FIG. 13 in 1st Embodiment.</figref><figref num="16">It is a figure which shows the data format example of the dust correction data in 1st Embodiment.</figref><figref num="17">It is a figure which showed the outline of the system structure of the image processing apparatus in 1st Embodiment.</figref><figref num="18">It is a figure which shows the example of GUI in an image processing apparatus.</figref><figref num="19">It is a flowchart explaining the dust correction reproduction processing in 1st Embodiment.</figref><figref num="20">It is a flowchart explaining the detail of the dust correction block selection process in 1st Embodiment.</figref><figref num="21">It is the figure which showed typically the relationship between a motion vector and a reference frame.</figref><figref num="22">It is a flowchart explaining the detail of the interpolation routine in 1st Embodiment.</figref><figref num="23">It is a flowchart explaining the detail of the dust correction block selection process in 2nd Embodiment of this invention.</figref><figref num="24">It is a flowchart explaining the detail of the dust correction block selection process in 3rd Embodiment of this invention.</figref><figref num="25">It is a flowchart explaining the detail of the dust correction block selection process in 4th Embodiment of this invention.</figref>