Photographed image display system, image receiving apparatus, and control method thereof and server

The present invention is a photographed image display system and an image receiving device that receives and displays a photographed image.<u style="single">And its control method</u>Regarding.

Conventionally, a portable telephone device having a videophone function capable of viewing a video of the other party in addition to a normal conversation has been known (see, for example, Patent Document 1). In the case of a videophone, since the shooting side and the viewing side are different, the blurred image is displayed on the viewing side due to the influence of the shooting side blur and the viewing side blur, and it was difficult to clearly display especially small characters. .. Further, as a conventional image blur prevention technology, when the projector device projects an image on the screen, the projector device, the screen, or the viewer's blur is detected, and at least one of the positions is physically moved. A technique for correcting blur is known (see, for example, Patent Document 2).<patcit num="1"><text>Japanese Patent Application Laid-Open No. 2002-27050</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 7-36423</text></patcit>

<p> However, the conventional image blur prevention technology requires a projector device, a screen, or a mechanism for physically moving the viewer, resulting in a large-scale device configuration. Therefore, it has been difficult to apply it to mobile devices such as videophones. The present invention has been made in view of the above-mentioned circumstances, and is a photographed image display system and an image receiving device capable of displaying an image with reduced blurring on the viewing side while avoiding complication of configuration.<u style="single">And its control method</u>Is to provide.</p>

<p> In order to solve the above-mentioned problems, the present invention relates to a photographing device that transmits a photographed image and a blur amount at the time of photographing in a photographed image display system, and an image that receives a photographed image and a blur amount transmitted from the photographing device. The image receiving device includes a receiving device.<u style="single">The difference between the blur amount detecting means for detecting the blur amount of the device, the received blur amount, and the blur amount detected by the blur amount detecting means is obtained, and based on this difference.</u>It is characterized by including a blur correction means for performing blur correction processing on a received captured image and a display means for displaying the captured image corrected by the blur correction means. According to this invention<u style="single">For example, send</u>It is possible to display an image with reduced blurring on the viewing side while reducing the processing load on the credit side. In addition, it is possible to avoid complication of the configuration as compared with the conventional image blur prevention device that requires a mechanism for physically moving.<u style="single">Since the captured image received is subjected to blur correction processing based on the difference between the received blur amount and the blur amount of the own device, blurring occurs on the viewing side regardless of whether the photographing device or the image receiving device is blurred. It is possible to display an image with reduced.</u></p><p> Further, the present invention comprises a receiving means for receiving a photographed image and a blur amount at the time of photographing from a communication partner in an image receiving device.<u style="single">The difference between the blur amount detecting means for detecting the blur amount of the device, the received blur amount, and the blur amount detected by the blur amount detecting means is obtained, and based on this difference.</u>It is characterized by including a blur correction means for performing blur correction processing on a received captured image and a display means for displaying the captured image corrected by the blur correction means. According to this invention<u style="single">For example</u>It is possible to display an image with reduced blurring on the viewing side while reducing the processing load of the other party. In addition, it is possible to avoid complication of the configuration as compared with the conventional image blur prevention device that requires a mechanism for physically moving.<u style="single">Since the captured image received is subjected to blur correction processing based on the difference between the received blur amount and the blur amount of the own device, blurring occurs on the viewing side regardless of whether the communication partner or the image receiving device is blurred. It is possible to display an image with reduced.</u></p><p> Further, in the above configuration, the shake amount detecting means is preferably a gyro sensor or an acceleration sensor. Further, in the above configuration, the image receiving device is a portable device including a photographing means for photographing, a transmitting means for transmitting the photographed image to the communication partner, and a communication means for making a voice call with the communication partner. It is preferably configured as a type videophone device. According to this configuration, in the case of a videophone, it is possible to display an image with reduced blurring on the viewing side.</p><p> Further, the present invention relates to a method for controlling an image receiving device that receives and displays a photographed image and a blur amount at the time of photographing from a communication partner.<u style="single">The amount of blurring of the device is detected, the difference between the amount of blurring received and the amount of blurring detected is obtained, and based on this difference.</u>It is characterized in that the received photographed image is subjected to blur correction processing and the corrected photographed image is displayed. According to this invention<u style="single">For example</u>It is possible to display an image with reduced blurring on the viewing side while reducing the processing load of the other party. In addition, it is possible to avoid complication of the configuration as compared with the conventional image blur prevention device that requires a mechanism for physically moving.<u style="single">To. Vedanā</u>Since the captured image received is subjected to blur correction processing based on the difference between the trusted blur amount and the blur amount of the own device, blurring occurs on the viewing side regardless of whether the communication partner or the image receiving device is blurred. It is possible to display an image with a reduced amount of.</p><p> Photographed image display system and image receiving device according to the present invention<u style="single">And its control method</u>According to the above, it is possible to display an image with reduced blurring on the viewing side while avoiding the complexity of the configuration.</p>

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a configuration of a captured image display system of the present invention. This captured image display system 1 is a system that enables a telephone or a videophone between mobile phones 2 via a mobile communication network NW, and mainly shows the configuration of the mobile phone 2. The mobile phone 2 can also make a call to a general telephone connected to a fixed-line telephone communication network (not shown) via the mobile communication network NW, similarly to the conventional mobile phone. Each of the above mobile phones 2 is configured to be substantially the same, and for convenience of explanation below, one mobile phone 2 in the case of making a one-to-one videophone call is referred to as a transmitting side device (corresponding to a photographing device) 2A. However, the other mobile phone is referred to as a receiving device (corresponding to an image receiving device) 2B, and when it is not necessary to distinguish between them, it is referred to as a mobile phone 2.

The mobile phone 2 includes an operation unit 10, a communication unit 20, a communication unit 30, an angular velocity detection unit 40, a shooting unit 50, and a control unit 60, and a removable media 70 is detachably attached. The operation unit 10 notifies the control unit 60 of the operation of a plurality of controls included in the mobile phone 2. The call unit 20 functions as a means of making a voice call with another mobile phone 2 or a general telephone, and includes a voice input unit 21 and a voice output unit 22. The voice input unit 21 includes a microphone that collects the voice of the user of the mobile phone 2, and outputs the voice input through the microphone to the control unit 60. Further, the voice output unit 22 is provided with a speaker that emits a voice toward the user of the mobile phone 2, and emits a synthetic voice such as a ringtone or a voice transmitted from a telephone partner via the speaker.

The communication unit 30 performs wireless communication with another mobile phone 2 or the like via the antenna 31 according to a communication method defined in the mobile communication network NW, and includes an RF unit, a baseband processing unit, and the like. The angular velocity detection unit 40 functions as a shake amount detecting means for detecting the shake amount of the mobile phone 2. As shown in FIG. 2, the angular velocity detection unit 40 includes an X-axis gyro sensor 41 that detects the angular velocity in the vertical direction (hereinafter, defined as the X-axis) of the frame FL to be photographed, and a left-right direction (hereinafter, the Y-axis). It has a Y-axis gyro sensor 42 that detects the angular velocity of (defined as). The angular velocity detection unit 40 outputs an angular velocity detection signal SV of a voltage value corresponding to each angular velocity of the X-axis and the Y-axis to the control unit 60 by two gyro sensors 41 and 42.

The shooting unit 50 functions as a shooting means for shooting a moving image or a still image, and includes a camera control circuit 51, a shooting camera 52, a shooting unit RAM 53, and a display panel 54. The camera control circuit 51 controls each part of the photographing unit 50 under the control of the control unit 60. Further, the photographing camera 52 sequentially outputs the photographing data D1 to the camera control circuit 41 at a predetermined sampling rate, and is an image sensor in which photoelectric conversion elements such as CCD and CMOS are arranged in a matrix or honeycomb shape. An optical lens system having a plurality of optical lenses, a lens driving device for driving this optical lens system to realize zoom focus, aperture, etc., and an image of an analog signal acquired by an image sensor are digital signals. It is configured with an A / D conversion circuit that converts to and outputs image data.

The shooting unit RAM 53 functions as a buffer for temporarily storing shooting data. Further, the display panel 54 functions as a display means for displaying various information such as a captured image and a setting screen, and is composed of, for example, a flat display panel such as a liquid crystal display panel or an organic EL panel. The removable media 70 stores shooting data and the like. Under such a configuration, the shooting data D1 output from the shooting camera 52 is temporarily stored in the shooting unit RAM53 after being subjected to predetermined processing by the camera control circuit 51, and when the shot image is saved. Is stored in the removable media 70. Further, the shooting data stored in the shooting unit RAM 53 is used when displaying on the display panel 54, and at the time of a videophone call, the shooting data is transmitted to the communication destination mobile phone 2 (2B) via the communication unit 20. Is transmitted wirelessly to. Further, the shooting data stored in the removable media 70 is used when displaying (reproducing) the shooting image.

FIG. 3 is a block diagram of the camera control circuit 51. The camera control circuit 51 includes a blur data calculation unit 80, a captured image processing unit 81, a corrected image creating unit 82, and a corrected image output unit 83. The blur data calculation unit 80 performs arithmetic processing on the angular velocity detection signal SV output from the angular velocity detection unit 40, acquires blur data B1 corresponding to the amount of blur in the sampling cycle of the shooting data D1, and corrects the image creation unit 82 or Selectively output to the control unit 60. That is, the blur data calculation unit 80 acquires the blur data B1 for each frame of the shooting data D1. The blur data B1 may be information on the angular velocity, but information on the amount of deviation (for example, the number of dots) on the frame calculated from the focal length of the photographing camera 52 and the distance to the subject may be applied. When the angular velocity information is transmitted as blur data B1, it is preferable that the blur data B1 includes basic data indicating the shooting conditions such as the focal length of the shooting camera 52 and the distance to the subject.

The captured image processing unit 81 inputs the captured data D1 sent from the captured camera 52 or the captured data D2 received from another mobile phone 2 (2A) via the communication unit 30 and stores the captured data D2 in the photographing unit RAM 53. The corrected image creation unit 82 functions as a blur correction means for performing blur correction processing on the shooting data stored in the shooting unit RAM 53, and inputs the blur data B1 acquired by the blur data calculation unit 80, or a television. At the time of a call, in addition to the blur data B1, the blur data B2 received from another mobile phone 2 (2A) is input via the communication unit 30, and is stored in the photographing unit RAM based on these blur data B1 and B2. Performs blur correction processing on the shooting data. Further, the corrected image output unit 83 outputs the corrected shooting data when the blur correction processing is performed, and the uncorrected shooting data when the blur correction processing is not performed to the display panel 54 or the like. ..

The control unit 60 functions as a computer that controls each part of the mobile phone 2. As shown in FIG. 1, a CPU 61 that executes various programs and performs arithmetic processing, and a control program 100 and various types that are executed by the CPU 61. It is equipped with a rewritable flash ROM (hereinafter, simply referred to as "ROM") 62 for storing data, and a RAM 63 that functions as a work area for temporarily storing the calculation results of the CPU 61 and various data. Further, the control program 100 stored in the ROM 62 includes a blur correction program for a videophone, and by executing this blur correction program, a moving image display with the blur corrected is realized.

The control program 100 can be recorded and distributed on a computer-readable recording medium 110 (see FIG. 1) such as a magnetic recording medium, an optical recording medium, or a semiconductor recording medium. Further, by connecting the personal computer and the mobile phone 2 with a cable or the like so as to be able to communicate and outputting the control program 100 of the recording medium 110 read by the personal computer to the mobile phone 2, the control program 100 is output to the ROM 62. It is also possible to store it.

Next, the operation when the blur correction program for a videophone is executed will be described. This blur correction program is executed at the time of videophone call. Here, the operation of the mobile phone 2 during a videophone is that the blur data indicating the amount of blur at the time of shooting of the own device 2 is transmitted to the mobile phone 2 of the communication partner together with the shooting data, and the shooting transmitted from the communication partner. The major difference is that the data and the blur data indicating the amount of blur at the time of shooting are received and the camera shake of the captured image is corrected. Other than that, the voice and the moving image (photographed image) are transmitted and received to each other to realize a videophone. The operation is substantially the same as that of a known mobile phone. These operations are performed by both mobile phones 2 during the videophone call, but in order to make the explanation easier to understand, the shooting data B2 and the blur data B2 from the transmitting device 2A are transmitted via the mobile communication network NW. The case where the data is transmitted to the receiving side device 2B and the receiving side device 2B receives the shooting data B2 and the blur data B2 will be described in detail.

FIG. 4 is a flowchart showing the operation when the blur correction program is executed in the receiving side device 2B. As a premise, it is assumed that the shooting data B2 of the moving image without blur correction and the blur data B2 at the time of shooting are transmitted from the transmitting side device 2A to the receiving side device 2B in substantially real time. In the receiving device 2B, first, the control unit 60 sets the blur correction level at the start of the videophone call (step S1). This blur correction level is a level set by the user, and the control unit 60 changes the content of the blur correction processing performed by the camera control circuit 51 according to the blur correction level. More specifically, in the present embodiment, the blur correction level is divided into three stages of "low", "medium", and "high", and when it is set to "high", the image of a plurality of frames in the shooting data is displayed. Refer to, analyze the movement of the frame and subject, apply image correction processing such as image shift to eliminate this movement, and apply blur correction processing to correct camera shake, and if it is set to "Medium", When a simple blur correction process such as lowering the frame rate of the display or lowering the resolution is applied and set to "Low", blur correction is not performed. That is, the lower the blur correction level, the less the processing load due to the blur correction.

Next, the control unit 60 starts the process of receiving the shooting data D2 and the blur data B2 from the transmission side device 2A of the communication destination (steps S2 and S3), and also starts the blur amount acquisition process (steps S2 and S3). Step S4), the blur data B1 of the receiving side device 2B is acquired from the angular velocity detection signal SV of the angular velocity detection unit 40. In this case, the control unit 60 has also started to acquire the shooting data D1 in the receiving side device 2B by the shooting camera 52, and the shooting data D1 and the blur data B1 are transferred to the transmitting side device 2A via the communication unit 30. Is sending to. Subsequently, when the control unit 60 receives the shooting data D2 and the blur data B2 from the transmitting side device 2A, the control unit 60 obtains the difference obtained by subtracting the blur data B1 from the blur data B2 to obtain the difference between the transmitting side device 2A and the receiving side device 2B. Acquire the difference data BD indicating the relative amount of deviation from and (step S5). In this case, if the blur data B2 of the transmitting side device 2A is data including angular velocity information and basic data (focal length at the time of shooting and distance to the subject), the amount of deviation on the frame is calculated from this information. However, it is preferable that the difference between this deviation amount and the deviation amount on the frame calculated from the deviation data B1 of the receiving side device 2B is used as the difference data BD.

Then, the control unit 60 performs blur correction processing on the shooting data D1 received based on the difference data BD acquired by the camera control circuit 51, and blur correction is performed based on the relative blur amount between the transmitting side device 2A and the receiving side device 2B. (Step S6). That is, in the present embodiment, the control unit 60 and the camera control circuit 51 function as blur correction means. In this case, as described above, the content of the blur correction process is changed depending on the blur correction level. Therefore, when the blur correction level is "high", the relative blur amount between the transmitting side device 2A and the receiving side device 2B is determined. A blur correction process that can be accurately eliminated is executed, and a captured image with the relative blur amount eliminated can be acquired. On the other hand, when the blur correction level is "medium", the frame rate is lowered to the extent that this blur is not noticeable, or the resolution is lowered, depending on the relative blur amount between the transmitting side device 2A and the receiving side device 2B. The simple blur correction processing of the above is executed, and it is possible to reduce the processing load and acquire a captured image in which crushing is inconspicuous. Further, when the blur correction level is "low", the blur correction process is not performed, and therefore, when there is almost no blur in both the transmitting side device 2A and the receiving side device 2B, or the user is concerned about the blurring. If this is not the case, it is possible to avoid the case where the blur correction process is performed and an unnecessary processing load is applied.

Next, the control unit 60 determines whether or not the data reception (reception of the shooting data D2 and the blur data B2) from the transmission side device 2A of the communication destination is completed (step S7), that is, whether or not the videophone is terminated. If it is not finished (step S7: NO), the process proceeds to step S2 and blur correction processing is performed for the next frame of shooting data D2, while if it is finished (step S7: NO). YES), end the process.

With the above configuration, the processes of steps S2 to S6 are sequentially and cyclically repeated during the videophone, and the transmitting side device 2A is used for each frame of the shooting data D2 sent from the transmitting side device 2A of the communication destination. The blur correction process is executed to eliminate the relative blur amount between the image and the receiving device 2B, and the image of the captured data D2 after this correction is displayed on the display panel 54. As a result, when the blurring occurs only in the transmitting side device 2A, the image corrected for the blurring of the transmitting side device 2A is displayed, and when the blurring occurs only in the receiving side device 2B, the image is received. An image that hardly shakes even when the side device 2B is shaken is displayed, and even when both the transmitting side device 2A and the receiving side device 2B are shaken, even when the receiving side device 2B is shaken, almost no blurring occurs. An image that does not blur is displayed. Therefore, the user of the receiving side device 2B can visually recognize the image without blurring regardless of whether the transmitting side device 2A or the receiving side device 2B is blurred.

As described above, according to the present embodiment, the shooting data D2 and the blur data B2 are received from the transmitting side device 2A of the communication destination, and the difference between the blur data B2 and the blur data B1 of the receiving side device 2B. Since blur correction processing is performed on the shooting data D2 based on the above, there is no blur on the user side (viewing side) of the receiving side device 2B regardless of whether the transmitting side device 2A or the receiving side device 2B is blurred. Images can be displayed. As a result, the apparent resolution is improved, and even if the captured image contains small characters, the characters can be clearly displayed.

Moreover, in this configuration, since the blur correction processing is performed only by the receiving side device 2A, the processing load of the transmitting side device 2A can be reduced. In other words, in the case of a videophone that transmits and receives images in both directions and displays them, there is no need for processing to correct the blurring of the shooting data to be transmitted, the processing load on the transmitting side can be reduced, and the real-time image display is improved. To do. Further, in this configuration, since blur correction of the captured image is performed by image processing, a conventional mechanism for detecting the amount of blur in the projector device or screen or a mechanism for physically moving the projector device or screen is required. Compared to the image blur prevention device, the number of parts is small, and the complexity of the configuration can be avoided. Therefore, it is possible to easily reduce the size and thickness of the mobile phone, and if the small gyro sensors 41 and 42 are used, the size and thickness of the mobile phone can be further reduced.

Further, in this configuration, a blur correction level that can be set by the user is provided, and the lower the blur correction level is, the less the processing load is. Therefore, both the transmitting side device 2A and the receiving side device 2B are performed. When there is almost no blurring, it is possible to avoid a case where the blurring correction processing is performed and a wasteful processing load is applied. Further, if the processing load required for blur correction is large, the frame rate of the captured image after correction may decrease, which may cause so-called frame dropping. However, in this configuration, the user sets the blur correction level low. This makes it possible to reduce the processing load of blur correction and display a moving image without dropped frames or a moving image with reduced dropped frames. The blur correction level may be automatically set based on the past history set by the user, the relative blur amount (difference data BD) of the transmitting side device 2A and the receiving side device 2B, and the like. .. In this case, for example, referring to the basic data transmitted from the transmitting side device 2A, when the photographing camera 52 of the transmitting side device 2A is wide-angle and the distance to the subject is long, the image is taken even if the relative blur amount is relatively large. Since the blur in the image is hardly known, the blur correction level may be set to "low" so that the blur correction is not performed.

(Application example) In the above-described embodiment, the receiving side device 2B detects the amount of blurring of the own device 2B, and based on the difference between the amount of blurring of the own device 2A and the amount of blurring of the own device 2B, the captured image from the communication destination is obtained. The case of performing blur correction has been described, but the present invention is not limited to this, and the process of detecting the amount of blur on the receiving side device 2B is omitted, and the receiving side device 2B receives from the communication destination based only on the amount of blur of the transmitting side device 2A. It is also possible to perform blur correction on the captured image of. In this case, it is possible to reduce the processing load on the receiving side.

Further, in the above-described embodiment, the transmitting side device 2A corrects the photographed image of the own device 2A based on the amount of blur detected by the own device 2A, and then transmits the photographed image after the blur correction to the receiving side device 2B. You may try to do it. In this case, the receiving device 2A may execute a blur correction process for correcting the amount of blur of the own device 2A on the received captured image, or the received captured image is displayed as it is without performing the blur correction process. You may. As a result, it is possible to display an image in which the blurring of the transmitting side device 2A is corrected while reducing the processing load of the receiving side device 2A.

Further, in each of the above-described embodiments, the case where the receiving side device 2B or the transmitting side device 2A performs the blur correction processing has been described. However, as shown in FIG. 5, a server 90 connected to the mobile communication network NW is provided. This server 90 receives the captured image and the amount of blur transmitted from the transmitting side device 2A, executes a blur correction process for correcting the amount of blur on the received captured image, and then receives the corrected captured image on the receiving side. It may be transmitted to the device 2B. In this case, the receiving device 2A may execute a blur correction process for correcting the amount of blur of the own device 2A on the received captured image, or the received captured image is displayed as it is without performing the blur correction process. You may. As a result, the processing load on the mobile phone 2 side can be reduced by the amount that the server 90 performs the blur correction processing, and the blur correction processing is performed with the high processing capacity of the server 90, so that the blur correction without dropping frames is performed. It becomes possible to obtain an image.

Further, in the above-described embodiment, the case where the vertical (X-axis) and horizontal (Y-axis) shakes are detected by the gyro sensor has been described, but the present invention is not limited to this, and the depth direction (Z-axis) shakes are further described. May be detected, or camera shake in any one of the above directions may be detected. Further, not limited to the gyro sensor, other shake amount detecting means such as an acceleration sensor may be applied. Further, in each of the above-described embodiments, the case where the present invention is applied to a mobile phone that makes a one-to-one videophone call has been described, but the present invention is applied to a mobile phone that can make a videophone call with a plurality of mobile phones at the same time. Is also possible. In this case, the above-mentioned blur correction processing is executed for each of the captured images from a plurality of communication partners, and each captured image after correction is divided into screens by the number of communication partners on the display screen of the display panel 24, and the divided screens are displayed. Each may be displayed. Further, in this case, if the blur correction level can be set for each of a plurality of communication partners, the processing load can be reduced by setting the blur correction level appropriately by the user.

Further, in each of the above-described embodiments, the case where the present invention is applied to a portable telephone device having a videophone function has been described, but the present invention is not limited to this, and the present invention includes a function of receiving and displaying a captured image in both directions. It can be applied to a portable electronic device such as a PDA or a notebook personal computer, or a photographed image display system that receives and displays a photographed image in one direction. For example, the captured image display system can be widely applied to a system including a photographing device that transmits a captured image and an image receiving device that displays a captured image transmitted from the captured image, and the captured image is attached by e-mail. It can also be applied to a photographed image display system that does not require real-time performance, such as a mail system that plays back on the mail receiving side.

<figref num="1">It is a figure which shows the structure of the photographed image display system of this invention.</figref><figref num="2">It is a figure for demonstrating the relationship between a frame and an axis of blurring.</figref><figref num="3">It is a block diagram of a camera control circuit.</figref><figref num="4">It is a flowchart which shows the operation of the receiving side apparatus.</figref><figref num="5">It is a figure which shows the structure of the photographed image display system which concerns on a modification.</figref>

Code description

1 ... Captured image display system, 2 ... Mobile phone, 2A ... Transmitter device (shooting device), 2B ... Receiver device (image receiver), 10 ... Operation unit, 20. .. Calling unit (calling means), 30 ... Communication section (transmitting means, receiving means), 40 ... Angle speed detecting section (shake amount detecting means), 41, 42 ... Gyro sensor, 50 ... Shooting unit (shooting means), 54 ... Display panel (display means), 60 ... Control unit, 100 ... Control program, 110 ... Recording medium, D1, D2 ... Shooting data, B1, B2 ... Blurred data, NW ... Mobile communication network (communication network).