Content based adjustment of an image

The present invention relates to adjusting a video signal. More specifically, it relates to adjusting image display settings based on signal characteristics, image characteristics, and image content.

Copyright notice Some of the disclosures in this patent document include elements that are subject to copyright protection. As can be seen in the files or records of the Patents and Trademark Office's patents, the copyright holder has no objection to this patent document or any copy reproduced by anyone regarding the disclosure of this patent. However, in other respects, all copyrights are reserved and the following warnings apply. Copyright holder Sony Electronics Co., Ltd. All rights reserved.

Modern televisions and monitors typically have a number of different image adjustment mechanisms. The user can search the display menu and manually adjust image display settings such as brightness, contrast, sharpness, and hue. No matter how you adjust the image display settings, the images will be displayed differently on your TV or monitor.

In addition, some televisions and monitors have one or more preset display modes. Each of its modes includes preset hues, sharpness, contrast, brightness, and more. The user can select a preset display mode by searching the display menu. In one display mode, a given image can be displayed more accurately than in other display modes. For example, the sport display mode can include image display settings that are generally considered optimal for viewing sporting events. Similarly, cinema display modes can include image display settings that are generally considered optimal for viewing moving images. However, to take advantage of such preset display modes, the user must manually change the display mode each time he or she changes what he or she is looking at. In addition, preset display modes are applied throughout the video (for all scenes), such as during sporting events. However, certain preset display modes cannot include optimal image display settings for a program, video, or entire sporting event.

The video input signal is analyzed to detect image characteristics and image content. Detection of image content involves automatically extracting image features. The content group for the video input signal is determined based on the content of the detected image. Each content group contains predetermined image characteristics. The image display settings are adjusted based on the difference between the detected image characteristics and the predetermined image characteristics.

<figref num="1A">An embodiment of the image adjusting device is shown.</figref><figref num="1B">A typical signal analyzer according to an embodiment of the present invention is shown.</figref><figref num="1C">A typical group determiner according to an embodiment of the present invention is shown.</figref><figref num="2A">A typical multimedia system according to an embodiment of the present invention is shown.</figref><figref num="2B">A typical multimedia system according to another embodiment of the present invention is shown.</figref><figref num="2C">A typical multimedia system according to still another embodiment of the present invention is shown.</figref><figref num="3">A method of modifying an image according to an embodiment of the present invention is shown.</figref><figref num="4">A method of modifying an image according to another embodiment of the present invention is shown.</figref><figref num="5">A method of modifying an image according to still another embodiment of the present invention is shown.</figref><figref num="6">A method of modifying an input signal according to an embodiment of the present invention is shown.</figref><figref num="7">A block diagram of a machine in a typical embodiment of an operable computer system of the present invention is shown.</figref>

In the following detailed description of the embodiments of the present invention, the reference symbols are constructed according to the accompanying drawings, and the same reference symbols indicate the same elements. In addition, those drawings are shown for illustration of specific embodiments in which the invention is feasible. These examples are described in sufficient detail for those skilled in the art to practice the present invention. Moreover, other embodiments may be utilized without departing from the scope of the invention, and, of course, logical changes, mechanical changes, electrical changes, functional changes, and other changes. Can also be configured. Therefore, the following detailed description should not be taken in a limited sense, and the scope of the present invention is defined only by the appended claims.

The terms are defined below for the purpose of clarifying the explanation.

The term "signal characteristic" means the characteristic of a video signal or the characteristic of a still image signal. Signal characteristics include metadata (eg, for title, time, date, GPS coordinates, etc.), signal structure (eg, interlaced or sequential manipulation (progressive scan)), encoding scheme (eg, Empeg 7), signal noise. And so on.

The term "image characteristic" means the characteristic of an image that can be displayed by a signal. Image characteristics include image size, image resolution, frame rate, dynamic range, color cast (predominance by a particular color), and histogram (pixels at each grayscale value in the image for each color channel). Number), blur, image noise, etc. can be included.

The term "image content" means objects and scenes that are recognizable when the viewer views the image (eg, objects and scenes obtained by a camera), as well as characteristics of the objects and scenes. Image content includes high-level image features (eg, faces, buildings, people, animals, landscapes, etc.) and low-level image features (eg, textures, actual colors of objects, shapes, etc.).

The term "image display setting" means a setting for display. The settings are adjusted to modify the way the image is displayed on the display. When the signal characteristic, the image characteristic, and the image content are the characteristic of the signal, the image display setting is the characteristic of the display device. Image display settings can include screen resolution (as opposed to image resolution), refreshment rate, brightness, contrast, hue, saturation, white balance, sharpness, and more.

The term "image characteristics to be displayed" means characteristics of an image after the display device (or image adjustment device) has processed the image. The image characteristics to be displayed can include the same parameters as the "image characteristics", but the values of the parameters may be different. Generally, the image display setting affects the image characteristics of the display target.

The operation of the present invention will be described from the outline. FIG. 1 shows an embodiment of the image adjusting device 100. The image adjuster 100 can modify one or more image display settings (eg, contrast, brightness, hue, etc.) with the image characteristics to be displayed (eg, dynamic range, blur, image noise, etc.). The image characteristics of the input signal can be different. Alternatively, the image adjuster 100 can modify the input signal and generate an output signal that includes the modified image characteristics. To modify the image display settings, the image adjuster 100 can analyze the video input to detect image characteristics and image content. The image adjuster 100 can determine the content group for the video input signal (and apply the content group to the video input signal) based on the detected image content, and further apply the appropriate content group. be able to. The content group can include a predetermined image characteristic that is a comparison and contrast with the detected image characteristic in order to determine how to modify the image display setting. By modifying the image display settings, it is possible to generate an image that is easier to see, an image that is more aesthetically pleasing, an image that more accurately depicts the recorded image, and the like.

The image adjuster 100 can be used to intercept and modify the video signal at any point between the source and the final display of the video signal. As an example, the image adjusting device 100 can be provided in a television, a monitor, a set-top box, a server-side router, a client-side router, or the like. A simplified image adjustment device 100 has been described for clarity of description, while the present invention relates to specific logic regarding content detection, content group assignment, or adjustment of image display settings. Or, it is not limited to the algorithm.

In one embodiment, the image adjuster 100 includes logic executed by a microcontroller, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another dedicated processing unit. In another embodiment, the image adjuster 100 may include logic executed by a central processing unit. In addition, the image adjuster 100 includes a series of state machines (eg, internal logic that determines how to perform a series of operations), logic circuits (eg, logic that examines a series of events in time). Alternatively, it can be implemented as a logic that changes the output at any time by changing the input), or as a combination of a state machine and a logic circuit.

In one embodiment, the image adjuster 100 includes an input terminal 105, a signal analyzer 110, a frame rate adjuster 112, a group determiner 115, an image adjuster 120, an audio adjuster 125, and an output terminal 130. Alternatively, one or more of the frame rate adjuster 112, the group determination device 115, the image adjuster 120, and the audio adjuster 125 may be omitted.

The input terminal 105 receives the input signal. The input signal can be a video input signal or a still image input signal. Examples of video input signals to be received include digital video signals (eg, Advanced Television Systems Committee (ATSC) signals, Digital Video Broadcasting (DVB) signals, etc.) Alternatively, analog video signals (eg, National Television System Committee (NTSC) signals, Phase Alternating Line (PAL) signals, sequential color memory (sequential color with)). memory, SECAM) signals, PAL / SECAM signals, etc.) are included. The video signal to be received and / or the still image signal may be compressed (for example, Motion Picture Expert Group 1 (MPEG-1), MPEG-2, MPEG-4, MPEG-7, Video Codec 1). It does not have to be) or compressed using (VC-1) or the like. The video input signal to be received may be interlaced or non-interlaced.

The input terminal 105 is connected to the signal analyzer 110, and can transmit the input signal to be received to the signal analyzer 110. The signal analyzer 110 can analyze the input signal to determine signal characteristics, image content, and / or image characteristics. Examples of image characteristics to be determined include image size, image resolution, dynamic range, color cast and the like. Examples of signal characteristics to be determined include encoding schemes, signal noise, signal structure (interlaced or progressive scan), and the like. In one embodiment, determining the image content includes extracting image features for the input signal. Both high-level image features (eg, faces, buildings, people, animals, landscapes, etc.) and low-level image features (eg, textures, colors, shapes, etc.) can be extracted. The signal analyzer 119 can automatically extract both low-level image features and high-level image features. Then, the signal analyzer 110 can determine the content based on the combination of the extracted image features. The signal analyzer 110 will be described in more detail with reference to FIG. 1B below.

Referring to FIG. 1A, in one embodiment, the image adjuster 100 includes a frame rate adjuster 112. In one embodiment, the frame rate regulator 112 has an input connected to the output of the signal analyzer 110 (as shown). In another embodiment, the frame rate regulator 112 has an input connected to the output of the group determiner 115, the image regulator 120, and / or the audio regulator 125.

The frame rate adjuster 112 can adjust the frame rate of the video input signal based on the signal characteristics, image content, and / or motion detected by the signal analyzer 110. In one embodiment, the frame rate regulator 112 increases the frame rate when an image characteristic that is in a state of motion greater than a certain threshold is detected. In one embodiment, the frame rate threshold is determined based on the speed of the moving object. For example, the threshold indicates that the object must not move more than 3 pixels between frames. Therefore, if it is detected that an object moves more than 3 pixels between two frames, the frame rate can be increased until the movement of the object between frames is less than 3 pixels.

Also, the amount of increase in frame rate can depend on the difference between the threshold and the detected motion. Alternatively, a lookup table or other data structure can contain multiple entries. Each entry associates a motion vector or amount of blur with a particular frame rate. In one embodiment, the average of a wide range of movements can be used to adjust or determine the frame rate. Alternatively, the frame rate adjustment may be based on the fastest moving image feature.

In one embodiment, the group determiner 115 has an input connected to the output of the frame rate adjuster 112. In another embodiment, the group determiner 115 has an input connected to the output of the signal analyzer 110. The group determiner 115 receives the low-level features and the high-level features extracted by the signal analyzer 110. Based on low-level features and / or high-level features, the group determiner 115 can determine the content group (also referred to as the content mode) for the input signal, and further determine the determined content group. Can be assigned to a signal. For example, the extracted image features include sky and natural landscapes, and in the absence of faces, landscape groups are determined and assigned. On the other hand, when multiple faces with no movement are detected, a portrait group is determined and assigned.

In one embodiment, the group determiner 115 assigns a single content group to the image. Alternatively, the group determiner 115 can divide the image into a plurality of partial images. The image is divided into multiple image portions based on the spatial arrangement (eg, the upper half of the image versus the lower half of the image), based on the boundaries of the extracted features, or based on other criteria. Can be done. Then, the group determination device 115 can determine the first content group for the first image portion including the first image feature, and further, for the second image portion including the second image feature. A second content group can be determined. In addition, the input signal may be split into more than two image portions. A different content group is assigned to each image portion. The group determination device 115 will be described in more detail with reference to FIG. 1C below.

Referring to FIG. 1A, in one embodiment, the image adjuster 100 includes an image adjuster 120. The image regulator 120 has an input connected to the output of the group determiner 115. As mentioned above, each content group contains a collection of predetermined image characteristics. The image regulator 120 sets one or more image display settings based on a comparison of the detected image characteristics (as detected by the signal analyzer 110) with the predetermined image characteristics of the assigned content group. Can be adjusted. The difference between the detected image characteristic and the predetermined image characteristic can determine the magnitude of the adjustment for the image adjustment setting. Alternatively, the image regulator 120 can modify the input signal so that when displayed on a display with known image display settings, the displayed image will have certain image characteristics. ..

In one embodiment, the image adjuster 100 comprises an audio adjuster 125 having an input connected to the output of the group determiner 115. The voice regulator 125 has one or more voice settings (eg, as detected by the signal analyzer 110) based on a comparison of the detected voice characteristics with the predetermined voice characteristics of the assigned content group. , Volume, equalizer settings, audio playback speed, etc.) can be adjusted. When multiple content groups are assigned to an input signal, the audio settings are based on the difference between the detected audio characteristics and a given audio characteristic in one of the content groups, or the audio of the assigned content group. Modified based on averages of properties or other combinations.

The output terminal 130 may have inputs connected to one or more outputs of the frame rate regulator 112, the image regulator 120, and the audio regulator 125. In one embodiment, the output terminal 130 comprises a display that displays an input signal using adjusted image display settings and / or adjusted frame rates. In addition, the output terminal 130 can include a speaker that emits an audio portion of the input signal using the adjusted audio settings.

In another embodiment, the output terminal 130 transmits an output signal to a device having a display and / or speaker. The output terminal 130 can generate an output signal based on the adjustment instructions received from one or more of the frame rate regulator 112, the image regulator 120, and the audio regulator 125. The output signal can have certain image characteristics of the determined content group when displayed on a display device with known image display settings. In addition, the output signal can have adjusted frame rates and / or predetermined audio characteristics of the determined content group when displayed on the display device.

In yet another embodiment, the output terminal 130 transmits an input signal along with a correction signal for image display setting to a device having a display and / or a speaker. The correction signal for the image display setting instructs the display device to adjust the image display setting and displays the image. This indicates the predetermined image characteristics of the determined content group. The correction signal for image display setting can also be metadata added to the input signal.

FIG. 1B shows a typical signal analyzer 110 according to an embodiment of the present invention. The signal analyzer 110 can analyze the input signal to determine signal characteristics, image content, and / or image characteristics. In one embodiment, determining the image content includes extracting image features for the input signal. Both high-level image features (eg, faces, buildings, people, animals, landscapes, etc.) and low-level image features (eg, textures, colors, shapes, etc.) are extracted. Then, the signal analyzer 110 can determine the content based on the combination of the extracted image features.

When compressing a received signal, the characteristics, image characteristics, and / or image content of one or more signals remain compressed. The signal can then be extended and then additional signal characteristics, image characteristics, and / or image content can be determined. Some signal characteristics, image characteristics, and / or image content are easy to detect while the image is compressed, while other signal characteristics, image characteristics, and / or image content are decompressed. It is easy to decide afterwards. For example, noise detection can be performed on compressed data, while face detection can be performed on decompressed data.

The signal analyzer 110 receives the signal frame by frame (eg, analyzed frame by frame, every other frame, etc.) or in a series of frames (eg, three frames are analyzed simultaneously). At the same time, the signal can be analyzed dynamically. When a series of frames are analyzed at the same time, the image content appearing in one frame may be applied to each frame in the series of sequences. In another example, if the image content does not appear in each frame in the sequence, the image content does not apply to those frames.

In one embodiment, the signal analyzer 110 includes a signal characteristic detector 135 and an image characteristic detector 140. Further, in another embodiment, the signal analyzer 110 includes a motion detector 150 and a voice characteristic detector 155.

The signal characteristic detector 135 analyzes the input signal and determines the signal characteristics (eg, encoding scheme, signal noise, signal structure, etc.). For example, the signal characteristic detector 135 can analyze the input signal and read any metadata attached to the signal. Such metadata can define information about image content and / or image characteristics. For example, an MPEG-7 encoded video signal contains image content, audio information, and metadata that can describe other information about the signal. As an example, the signal characteristic detector 135 can determine the signal-to-noise ratio (SN ratio) and can also determine whether the signal is interlaced or non-interlaced.

In one embodiment, the image characteristic detector 140 determines the image characteristics in the input signal. Examples of image characteristics include image size, image resolution, and color cast. In one embodiment, the image characteristic detector 140 extracts image characteristics for the input signal. Then, the image feature is used to determine the image content of the input signal. The image property detector 140 extracts both high-level image features (eg, faces, buildings, people, animals, landscapes, etc.) and low-level image features (eg, textures, colors, shapes, etc.) from the input signal. .. The image characteristic detector 140 can also automatically extract both low-level image features and high-level image features. Typical techniques for extracting possible image features known to those of skill in the art in content-based image retrieval are described below. However, other known techniques can be used to extract the same or different image features.

In one embodiment, the image property detector 140 analyzes the input signal and extracts the colors contained in the signal. To determine high-level features and / or image content, the extracted colors and their distribution within the image may be used alone or with other low-level features. For example, in one image, if the first image portion is dominated by blue and the other image portion is dominated by yellow, the image may be a beach landscape image. Colors can be extracted by mapping the color histogram. Then, such a color histogram can be compared with the stored color histogram. The color histogram is associated with certain high-level features. Based on the comparison, the information of the image content can be determined. In addition, other techniques (eg, Gaussian distribution models, Bayesian classifiers, etc.) can be used to determine and compare colors.

In one embodiment, the image characteristic detector 140 uses elliptical color models to determine a high level of features and / or image content. Pixel mapping in a set of images with high-level features in the Hue, Saturation, and Lightness (HSV) color space produces an elliptical color model for the high-level features. The set of images can then be manipulated using a training set of first images with high levels of characteristics and a training set of second images lacking high levels of characteristics. The optimal elliptical color model is determined by maximizing the statistical gap between the first training set for images and the second training set for images.

The image (eg, the image of the input signal) is mapped over the HSV color space and compared to an elliptical model for high-level features. The percentage of pixels in the image that fits the pixels in the elliptical model is used to predict the likelihood that the image will have a high level of image features. This comparison is determined using the following formula.<img file="JP2010541009A_D0001.tif" />here,<img file="JP2010541009A_D0002.tif" />Is the amount of pixels in the color ellipse model,<img file="JP2010541009A_D0003.tif" />Is the amount of pixels in the image in the color ellipse model. Amount of pixels in the color ellipse model<img file="JP2010541009A_D0004.tif" />Amount of pixels that fits<img file="JP2010541009A_D0005.tif" />For images with a large, the gap d (I, T) tends to be small. And it can be seen that a high level of features are present in the image.

If the high-level features contain several different colors (eg, red flowers, green leaves, etc.), the high-level features can be segmented. An independent elliptical color model can be made to be generated for each segment. The images mapped to the HSV color space can be compared with each of the color ellipse models. For both color ellipse models, if the gap d (I, T) is small, it can be seen that high levels of features are present in the image.

In one embodiment, the image property detector 140 analyzes the input signal and extracts the texture contained in the signal. The texture is determined by finding the visual pattern of the image contained in the input signal. In addition, textures can contain information about how images and visual patterns are spatially defined. Textures can be represented as texture elements (texels). Texels can be classified into many sets (eg, arrays), depending on how many textures are detected in the image, how the texels are arranged, and so on. The set can define the texture and, in addition, where the texture is located in the image. The texels and sets can be compared to the stored texels and sets associated with certain high-level features to determine the information in the image content.

In one embodiment, the image property detector 140 uses wavelet-based texture feature extraction to determine a high level of features and / or image content. The image property detector 140 can include a plurality of texture models for each particular image feature or image content. Each texture model can include a wavelet coefficient and various values associated with the wavelet coefficient, as described below. The wavelet coefficients and related values are compared with the wavelet coefficients and related values extracted from the input signal to determine if any of the high level features and / or image content is included in the video input signal.

For example, the wavelet coefficient can be calculated by performing a Haar wavelet transformation on the image. The initial low-level coefficients can be calculated by combining adjacent pixel values as shown in the formula below.<img file="JP2010541009A_D0006.tif" />Here, L is the low-frequency coefficient, i is the index number of the wavelet coefficient, and P is the pixel value of the image. The initial high level coefficients can be calculated by combining adjacent pixel values as shown in the formula below.<img file="JP2010541009A_D0007.tif" />Here, H is a high frequency coefficient, i is the index number of the wavelet coefficient, and P is the pixel value of the image.

The Haar wavelet transform can be applied recursively. This is performed on each of the initial low level coefficients and each of the early high level coefficients to generate a second level coefficient. The Haar wavelet transform can then be applied to the second level coefficients, if desired, to generate the third level coefficients. In one embodiment, a fourth level coefficient is used (for example, the wavelet transform repeated four times is calculated).

The wavelet transform with multiple levels is used to calculate the average absolute value of the coefficients for the various subbands of multiple levels. For example, the average absolute value μ of the coefficients for a given subband (eg, LH subband) is calculated using the following formula.<img file="JP2010541009A_D0008.tif" />Where LH (i) is the LH subband at level i, W is the wavelet coefficient, m is the row of coefficients, n is the column of coefficients, M is equal to the total number of rows of coefficients, and N is the coefficient. Is equal to the total number for the column. Also, for other subbands for multiple levels (eg, HH, HL, and LL), the average absolute value of the coefficients can be determined.

Wavelet coefficients for multiple levels can be used to calculate the amount of change in the coefficients. The amount of change is a value for various subbands at multiple levels. For example, the amount of change in the coefficient σ²Is calculated for a given subband LH using the following formula.<img file="JP2010541009A_D0009.tif" />Where LH (i) is the LH subband at level i, W is the wavelet coefficient, m is the coefficient row, n is the coefficient column, M is equal to the total number of coefficient rows, and N is the coefficient column. Equal to the total number of, and μ corresponds to the average absolute value of the coefficients. Also, the amount of change in the coefficients can be calculated for other subbands at multiple levels (eg, HH, HL, and LL).

Further, the mean absolute texture angles are calculated using the average absolute value of the coefficients as shown in the following formula.<img file="JP2010541009A_D0010.tif" />Where θ_{μ (i)}Is the average absolute texture angle, μ is the average absolute value of the coefficient, i is the subband level, LH is the LH subband, and HL is the HL subband.

The texture angle of the amount of change is calculated using the amount of change of the coefficient as shown in the following formula.<img file="JP2010541009A_D0011.tif" />Where θ_{σ (i)}Is the texture angle of the amount of change, σ is the amount of change in the coefficient, i is the subband level, LH is the LH subband, and HL is the HL subband. Both the average absolute texture angle and the amount of change texture angle indicate how the texture is adapted in the image.

The total of the average absolute values can be calculated by the following formula.<img file="JP2010541009A_D0012.tif" />

Where μ_(i)Is the sum of the average absolute values, i is the wavelet level, μ_{LH (i)}Is the average absolute value of the coefficients for the LH subband, μ_{HH (i)}Is the average absolute value of the coefficients for the HH subband, μ_{HL (i)}Is the average absolute value of the coefficients for the HL subband.

The total amount of change can be calculated according to the following formula.<img file="JP2010541009A_D0013.tif" />Where σ_(i)Is the total amount of change, i is the wavelet level, σ_{LH (i)}Is the amount of change in the coefficient for the LH subband, σ_{HH (i)}Is the amount of change in the coefficient for the HH subband, σ_{HL (i)}Is the amount of change in the coefficient for the HL subband.

The sum of the average absolute values and the sum of the amounts of change are used to calculate the value of the distance between the texture model and the image for the input signal. The above values are calculated for an image or reference image with a high level of features to generate a texture model. The value can then be calculated for the input signal. The value for the video input signal is compared to the value for the texture model. If the gap D (representing texture similarity) between the two values is small, the input signal is determined to contain a high level of features. The distance D is calculated using the following formula.<img file="JP2010541009A_D0014.tif" />Where D is the value for the gap, T is the texture model, I is the test image, i is the wavelet level, μ is the sum of the average absolute values, σ is the sum of the changes, θ_σIs the texture angle of the amount of change, θ_{μ μ}Is the average absolute texture angle.

In one embodiment, the image property detector 140 determines the shape of a region (eg, part of an image) within an image. The shape is determined by applying segmentation techniques, blob extraction techniques, contour detection techniques, and / or other known shape detection techniques to the image. Then, the determined shape is compared with the database about the stored shape, and the information about the image content is determined. The image characteristic detector 140 can determine the image content based on the detected image characteristic and / or the image characteristic. In one embodiment, the image characteristic detector 140 uses a combination of extracted image features to determine the image content.

In one embodiment, the image property detector 145 uses the extracted low-level image features to determine high-level image features. For example, the extracted low-level image features are used to determine one or more people, landscapes, buildings, and the like. The image characteristic detector 145 can also combine other technically known low-level image features as well as the low-level image features described above to determine high-level image features. For example, the first shape, texture, and color correspond to the face, while the second shape, texture, and color correspond to wood.

In one embodiment, the signal analyzer 110 comprises a motion detector 150. The motion detector 150 detects the motion of the video input signal in the image. As an example, the motion detector 150 determines a moving image portion by using a motion vector. For the motion vector, the total movement (total movement amount in the scene) and the whereabouts movement (movement amount in one or more parts of the image, and / or the movement of one or more features were detected. Amount) is used to determine. Further, the motion detector 150 can detect a moving image portion by analyzing image characteristics such as blur. Blur with directionality can also be determined. A directional blur is detected and indicates the movement of the object (or camera) in the direction of the blur. The degree of blur is used to predict the speed of a moving object.

In one embodiment, the signal analyzer 110 comprises a voice characteristic detector 155. The voice characteristic detector 155 determines the voice characteristic (voice characteristic) of the input signal. Often, audio characteristics show a strong interrelationship with image features. Therefore, in one embodiment, such audio is used to more optimally determine the image content. Examples of audio characteristics detected and used to determine image features include cepstrum flux, multi-channel co-clear decomposition, cepstrum flux, multi-channel vector, low energy fraction, spectral flux, spectral roll-off point, zero crossing rate, Zero crossing rate changes, energies, and energy changes are included.

FIG. 1C shows a typical group determiner 115 according to an embodiment of the present invention. The group determination device 115 includes a plurality of content groups 160. The group detector 115 receives the low-level features and the high-level features extracted by the signal analyzer 110. The group detector 115 can determine a content group (also referred to as a content mode) for an input signal from a plurality of content groups 160 based on low-level features and / or high-level features, and further determines. Assign content groups to signals. For example, if the extracted image features include a sky or natural landscape and do not include a face, a group for the landscape is determined and assigned. On the other hand, when multiple faces with no movement are detected, a portrait group is determined and assigned. In one embodiment, determining a content group includes automatically assigning the content group to a signal.

In one embodiment, the group detector 115 assigns a single content group from a plurality of content groups 160 to an image. Alternatively, the group detector 115 divides the image into a plurality of image portions. Images are assigned to multiple image portions based on spatial arrangement (eg, the top half of the image and the bottom half of the image), based on the boundaries of the extracted features, or based on other criteria. .. Then, the group detector 115 determines the first content group for the first image portion including the first image feature, and further determines the second content group for the second image portion including the second image feature. Content groups can be determined. Further, the input signal may be divided into more than two image portions. Different content groups are assigned to each part.

Each content group 160 may include predetermined image characteristics for image restoration in a particular environment (eg, cinematic environment, landscape environment, etc.). Each content group can also include predetermined audio characteristics for audio restoration. Content group 60 preferably has optimal image and / or audio characteristics for a particular environment. The predetermined image characteristics are realized by changing the image display setting of the display device. Alternatively, certain image characteristics are achieved by changing the input signal. Thereby, when displayed by a display having known image display settings, the image is displayed with predetermined image characteristics.

A typical content group 160 will be described later. These content groups 160 are for illustrative purposes only, and of course additional content groups are envisioned.

In one embodiment, content group 160 includes a default content group. The default content group is automatically applied to all images until a different content group is determined. The default content group can include predetermined image characteristics that provide a satisfactory image in a series of many environments. In one embodiment, the default content group enhances sharpness, brightens contrast, and sharpens brightness for optimal display of television programs.

In one embodiment, the content group 160 includes a portrait content group. The portrait content group includes certain image characteristics that capture an image that optimally reproduces the skin. Portrait content groups can also include predetermined image characteristics that more appropriately display still images. In one embodiment, the portrait content group provides predetermined color settings for magenta, red, and yellow tones. Its color settings enable the reproduction of healthy skin tones with minimal color bias. In addition, this portrait content group reduces sharpness to remove unwanted material on the skin texture.

In one embodiment, the content group 160 includes a landscape content group. Landscape content groups can include predetermined image characteristics. Its predetermined image characteristics allow for vivid reproduction of images in the green to blue range (eg, displaying blue sky, blue sea, green leaves, etc. more appropriately). Landscape content groups can also include predetermined image characteristics for improved sharpness.

In one embodiment, the content group 160 includes a sports content group. Sports content groups can reproduce accurate color depth and light-dark gradations, and can sharpen images using gamma curves and saturation adjustments, as well as green (eg, football competitions). You can adjust the color settings to make the field more vivid.

In one embodiment, the content group 160 includes a cinema content group. The Cinema Content Group uses gamma curve correction to enable color correction to vividly display unclear scenes, while removing unwanted material for image movement.

Other possible examples of content group 160 are text content groups (text should have a high contrast background and sharp boundaries), content groups for news broadcasts, and still photos. Includes content groups, game content groups, concert content groups, and home video content groups.

FIG. 2 shows a typical multimedia system 200 according to an embodiment of the present invention. The multimedia system 200 includes a signal source 205 connected to the display device 240. The signal source 205 can generate one or more of a video signal, a still image signal, and an audio signal. In one embodiment, the signal source is a broadcaster, a cable or satellite television provider, or another remote multimedia service provider. Alternatively, the signal source may be a video cassette recorder (VCR), digital video disc (DVD) player, high-detail digital versatile disc (HD-DVD) player, Blu-ray player, or other media player. be able to.

The means for connecting the signal source 205 to the display device 240 may depend on the attributes of the signal source 205 and / or the display device 240. For example, when the signal source 205 is a television broadcasting station, the signal source 205 can be connected to the display device 240 via radio waves that carry a video signal. On the other hand, when the signal source 205 is a DVD player, the signal source 205 can be connected to the display device via, for example, an RCA cable, a DVI cable, an HDMI cable, or the like.

The display device 240 can be a television, a monitor, a projector, or any other device capable of displaying still images and / or video images. The display device 240 can include an image adjusting device 210 connected to the display 215. In one embodiment, the image adjuster 210 corresponds to the image adjuster 100 of FIG. 1A.

The display device 240 receives the input signal 220 from the signal source 205. The image adjusting device 210 can change the image display setting. Thereby, the output signal displayed on the display 215 includes predetermined image characteristics of one or more determined content groups.

FIG. 2B shows a typical multimedia system 250 according to another embodiment of the present invention. The multimedia system 250 includes a media player 255 connected to the display device 260. The media player 255 can be a VCR, a DVD player, an HD-DVD player, a Blu-ray player, a digital video recorder (DVR), or other media player. Further, the media player 255 can generate any one or more of a video signal, a still image signal, and an audio signal. In one embodiment, the media player 255 includes an image adjuster 210. The image adjusting device 210 can correspond to the image display device 100 of FIG. 1A.

The image adjuster 210 can change the input signal generated during playback of the media in order to generate the output signal 225. The output signal 225 is transmitted to a display device 260 capable of displaying the signal 225. In one embodiment, the signal regulator 210 adjusts the input signal. Thereby, when displaying on the display device 260, the output signal indicates an image having predetermined image characteristics corresponding to one or more image content groups. The signal adjustment device 210 can receive the image setting 227 from the display device 260 to determine how the input signal should be changed to display the predetermined image characteristics on the display device 260. Obtaining a particular set of image settings 227 produces an output signal 225, which results in the display of predetermined image characteristics.

In other embodiments, the output signal includes an additional signal (or metadata added to the input signal) in addition to the unchanged input signal. The additional signal tells the display device to use a particular image display setting.

FIG. 2C shows a typical multimedia system 270 according to yet another embodiment of the present invention. The multimedia system 270 includes a signal source 205, an image adjuster 275, and a display device 260.

As an example, the image adjusting device 275 is arranged between the signal source 205 and the display device 260, and is further connected to the signal source 205 and the display device 260. The image adjusting device 275 can also be made to correspond to the image adjusting device 100 of FIG. 1A. In one embodiment, the image adjuster 275 is a stand-alone device and is used to change the signal transmitted to the display device 260. In another embodiment, the image adjuster 275, which is an example, may be a component such as a set-top box, a digital video recorder (DVR), a cable box, a satellite broadcast receiver, and the like.

The image adjuster 275 can receive the input signal 220 generated by the signal source 205. Then, the image adjusting device 275 can change the input signal 220 and generate the output signal 225. Further, the output signal 225 is transmitted to the display device 260. The display device 260 can display the output signal 225. The signal regulator 275 can receive the display setting 227 from the display device 260 prior to generating the output signal 225. The received display settings are used to determine how the input signal should be modified to generate the output signal.

Specific methods of the present invention are described with respect to computer software conforming to a series of flow diagrams shown in FIGS. 3-6. The flow charts shown in FIGS. 3-6 show how to modify the image and may be performed by the image adjuster 100 of FIG. 1A. The method configures a computer program. The computer program is formed by instructions that can be executed by a computer, and the instructions are shown, for example, as blocks (operations) 305 to 335 in FIG. Explaining the method by reference to a flow diagram allows one of ordinary skill in the art to create such a program. The program contains instructions to perform the method on a properly configured computer or computer device, such as a computer processor that performs instructions from a computer-readable medium, including memory. Instructions that can be executed by a computer can be written in a programming language for the computer. The instruction can also be realized by the logic of the firmware. Written in a program language that conforms to approved standards, such instructions can be run on different hardware platforms and can be run for different operating system interfaces.

The present invention is not expressed for a particular programming language. Of course, various programming languages can be used to implement the disclosure of the invention as described herein. Moreover, being in one form or another form (eg, a program, procedure, process, application, module, logic, etc.) to perform an action or obtain a result is common in software technology. .. This expression is simply a concise expression that the execution of software by a computer causes the processor of that computer to perform an action or obtain a result. Of course, more or less processes can be incorporated into the methods shown in FIGS. 3-6 without departing from the scope of the invention, and in addition, the block arrangements shown and described herein will be obtained. A particular order is not always used.

First, referring to Figure 3, shows the actions performed by the computer that implement Method 300 for image modification.

In a particular embodiment of the invention, method 300 receives a video input signal (block 305) (eg, by input terminal 105 in FIG. 1). At block 310, the video input signal is analyzed (eg, by the signal analyzer 110 of FIG. 1A) to detect image characteristics and image content. In other embodiments, the video input signal may be analyzed and signal characteristics may be detected (eg, whether the signal is interlaced or non-interlaced, whether the signal is compressed or uncompressed, Or, determine the noise level of the signal, etc.). If the signal is compressed, the signal needs to be decompressed before determining one or more image characteristics and / or image content.

The video input signal is analyzed frame by frame. In another embodiment, a series of frames (sequence of frames) of the video input signal is analyzed simultaneously. This causes the change to occur at each frame in the sequence. Image characteristics can include image size, image resolution, dynamic range, color cast, blur, and the like. Image content is detected by extracting low-level image features (eg, textures, colors, shapes, etc.) and high-level image features (eg, faces, people, buildings, sky, etc.). Such image features are automatically extracted by receiving the video input signal.

At block 315, a content group for the video input signal is determined (eg, by the group determiner 115 in FIG. 1A). The content group is determined based on the detected image content. In another embodiment, the content group is determined using the metadata contained in the video signal. Metadata can have information about image content and / or image characteristics. For example, the metadata can identify the current scene of the video as an action scene. At block 318, the content group is applied to the input video signal.

In block 320, it is determined whether or not the detected image characteristic matches a predetermined image characteristic associated with the determined content group. If the detected image characteristics match the predetermined image characteristics, the method proceeds to block 330. If the detected image characteristics do not match the predetermined image characteristics, the method remains in block 325.

At block 325, the image display settings and / or the video input signal are adjusted (eg, by the image adjuster 120 in FIG. 1A). The image display settings and / or the video input signal are adjusted based on the difference between the detected image characteristics and the predetermined image characteristics. This allows the input signal to be displayed with certain image characteristics. At block 330, the video input signal is displayed. This is the end of the method.

Although method 300 has been shown to have specific blocks in which different tasks are performed, it would be expected that some tasks performed by some blocks could be split and / or integrated. .. For example, block 310 indicates that the video input signal is analyzed to determine the image content and image characteristics together. However, it is also possible to analyze the video signal and detect the image content without determining the image characteristics. Then, after analyzing the signal and determining the image characteristics, the content group can be determined. In other embodiments, block 315 (determining a content group) and block 318 (assigning a content group) can be integrated into a single block (eg, a single action). Other changes to the described method can be predicted.

With reference to FIG. 4, the actions performed by the computer that implement Method 400 for image modification are shown.

In certain embodiments of the invention, method 400 comprises receiving a video input signal (block 405) (eg, by input terminal 105 in FIG. 1A). At block 410, image characteristics, signal characteristics, and / or image content are detected (eg, by the signal analyzer 110 in FIG. 1). In block 412, the image is divided into a first image portion and a second image portion. In one embodiment, the image is divided into a first image portion and a second image portion based on the boundaries of the image features. In other embodiments, the division is based on the separation of equal or unequal parts of the image. For example, the image is divided into a first half (eg, upper or right half) and a second half (eg, lower or left). The first image portion can include the first image feature and the second image portion can include the second image feature.

In block 415, the first content group for the first image portion is determined (eg, by the group determiner 115 in FIG. 1A) and the second content group for the second image portion is determined. Determining the first content group and the second content group can include applying the first content group and the second content group, respectively. The first content group can include a first predetermined image characteristic. In addition, the second content group can include a second predetermined image characteristic. The first content group and the second content group are each determined based on the detected image content of each image portion.

In block 420, it is determined whether or not the detected image characteristic conforms to a predetermined image characteristic. The predetermined image property is associated with the determined content group and / or the assigned content group. If the detected image characteristics match the predetermined image characteristics, the method proceeds to block 430. If the detected image characteristics do not match the predetermined image characteristics, the method remains in block 425.

At block 425, the image display settings and / or the video input signal are adjusted (eg, by the image adjuster 120 in FIG. 1A). As a result, the detected image characteristics match the predetermined image characteristics of the first image portion and the second image portion. The image display setting and / or the video input signal for the first image portion is adjusted based on the difference between the detected first image characteristic and the first predetermined image characteristic of the first content group. The image display setting and / or the video input signal for the second image portion is adjusted based on the difference between the detected second image characteristic and the second predetermined image characteristic of the second content group. At block 430, the video input signal is displayed. This is the end of the method.

Referencing Figure 5 shows the actions performed by a computer that implements Method 500 for changing audio output.

In certain embodiments of the invention, method 500 comprises receiving a video input signal (block 505) (eg, by input terminal 105 in FIG. 1A). At block 510, the video input signal is analyzed (eg, by the signal analyzer 110 of FIG. 1A) to detect audio characteristics and image content. Speech characteristics include cepstrum flux, multi-channel co-clear decomposition, multi-channel vector, low energy fraction, spectral flux, spectral roll-off point, zero crossing rate, zero crossing rate change, energy, and energy change. be able to. At block 515, a content group is determined for the video input signal (eg, by the group determiner 115 in FIG. 1A). The content group is determined based on the detected image content. In addition, the content group can include certain audio characteristics.

At block 520, it is determined whether the detected audio characteristic matches a predetermined audio characteristic associated with the determined content group. If the detected voice characteristics match the predetermined voice characteristics, the method proceeds to block 530. If the detected voice characteristics do not match the given voice characteristics, the method remains at block 525.

At block 525, the audio settings and / or audio portion of the video input signal is adjusted (eg, by the audio adjuster 125 in FIG. 1A). The audio setting and / or audio portion of the video input signal is adjusted based on the difference between the detected audio characteristics and the predetermined audio characteristics. Examples of adjustable audio settings include volume, playback speed, and equalizer settings. At block 530, a video input signal is output (eg, transmitted and / or played). This is the end of the method.

With reference to FIG. 6, the actions performed by the computer that implement Method 600 for image modification are shown.

In certain embodiments of the invention, method 600 comprises receiving a video input signal (eg, by input terminal 105 in FIG. 1A) (block 606). At block 610, the video input signal is analyzed (eg, by the signal analyzer 110 of FIG. 1A) and the image content is detected. In block 615, it is determined whether or not the image content is moving. An example of such a determination is made using a motion vector. Alternatively, it is performed by detecting the degree and direction of blurring. In block 620, it is determined how fast the movement of the image content is.

At block 630, the frame rate of the video signal is adjusted based on how fast the image content is (eg, by the frame rate adjuster 112 in FIG. 1A). For example, detecting that the image content moves faster will increase the frame rate. At block 636, the sharpness of the moving object is adjusted (eg, by the image adjuster 120 in Figure 1A). Alternatively, the sharpness of all signals may be adjusted. The sharpness adjustment can be based on the speed at which the image content moves.

At block 640, a video input signal is output (eg, transmitted and / or displayed) with adjusted frame rate and adjusted sharpness. The method ends here.

FIG. 7 shows a block diagram of a machine in a typical form of computer system 700. A set of instructions is performed within the computer system 700 to cause the machine to perform one or more of the methods shown herein. A typical computer system 700 includes a processor 705, a memory 710 (eg, read-only memory (ROM), storage device, static memory, etc.), and an input / output 715. The processor 705, the memory 710, and the input / output 715 are connected to each other via the bus 720. The embodiments of the present invention are performed by the computer system 700 and / or additional hardware elements (not shown). Moreover, the embodiment of the present invention may be realized by a machine-executable instruction. It should be noted that when the instruction is programmed, the instruction is used to cause the processor 705 to perform the method described above. In other embodiments, the method is performed by a combination of hardware and software.

Processor 705 refers to one or more general purpose processing units such as microprocessors, central processing units, and the like. More specifically, the processor 705 is a microprocessor that implements a composite instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, an ultra-long instruction word (VLIW) microprocessor, or another instruction set implementation. , Can be a processor that realizes a combination of instruction sets. The processor 705 can also be an application specific integrated circuit (ASIC) or one or more dedicated processing devices such as field programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, and the like. ..

The present invention may be provided as a computer program product or software that can be stored in memory 710. The memory 710 can include a machine-readable medium for storing instructions. The instructions can be used to program a typical computer system 700 (or other electronic device) to perform the processing according to the invention. Other machine-readable media include, but are not limited to, flexible disks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAM, EPROMs, EEPROMs, magnetic cards, optical cards, flash memory, and electrical instructions. These include other types of media suitable for storage, such as other types of machine-readable media, which store instructions programmed for a typical computer system 700 (or other electronic device). can do.

Input / output 715 can communicate with additional devices and / or components. As a result, the input / output 715 can transmit data to, for example, a computer, a server, a mobile device, etc. connected to the network, and further receive data from them.

The description in FIG. 7 is intended to give an overview of the appropriate computer hardware and other components for operation to achieve the present invention, but is intended to limit the environment used for application. Not a thing. The computer system 700 in Figure 7 will be understood as an example of many possible computer systems. The possible computer systems are realized with different architectures. Those skilled in the art will understand that the present invention is realized in the configuration of other computer systems. Other computer systems include microprocessor systems, microcomputers, mainframe computers, and the like. The present invention can also be realized in a distributed computer environment. In that case, the task is performed by a remote processor connected via a communication network.

The device and method for adjusting the signal have been described. Although specific embodiments have been illustrated and described, those skilled in the art can substitute any arrangement in which calculations are performed to achieve the same objectives instead of the specific embodiments shown herein. It will be understandable. The application is also intended to address modifications or modifications of the invention. For example, it is possible for those skilled in the art to use an image content detection algorithm and its technique other than those described.

With respect to signal modification, the terminology used in this application is intended to include all devices and environments in which the signal can be altered. Therefore, the present invention is limited only by the following claims and their equivalents.