Image processing device, method, and program

The present invention relates to indexing of images and the like.

Japanese Patent Application No. 9-33490 discloses a technique related to indexing processing for searching recorded images and the like for images, sounds, and the like in a specific channel unit.

<p num="0003"><patcit num="1"><text>Japanese Patent Application No. 9-33490</text></patcit></p>

<p num="0004"><nplcit num="1"><text>C. Harris and M. Stephens (1988). "A combined corner and edge detector". Proceedings of the 4th Alvey Vision Conference: pp 147--151.</text></nplcit></p>

<p num="0005"> However, with the technology disclosed in Japanese Patent Application No. 9-33490, it is not possible to perform indexing processing at the same time while receiving a video or the like being broadcast.</p><p num="0006"> The present invention has been made in view of the above background, and an object of the present invention is to perform indexing processing at the same time while receiving a video or the like being broadcast.</p>

<p num="0007"> The image processing device according to one aspect of the present invention includes a receiving unit, a registration unit, a determination unit, and a control unit. The receiving unit receives a broadcast wave including signals of a plurality of channels. The registration department registers the recognition target. The determination unit determines whether or not the recognition target registered by the registration unit exists in the signals of a plurality of channels included in the broadcast wave received by the reception unit.<u style="single">The ratio of processing time allocated to channels whose similarity to the previous image frame of the same channel is continuously less than the threshold value is smaller than that of other channels.</u>To do. The control unit sequentially switches a plurality of channels received by the receiving unit according to the determination result determined by the determination unit.</p>

<p num="0008"> According to the present invention, the indexing process can be performed at the same time while receiving the video or the like being broadcast.</p>

<figref num="1">The block diagram which shows the structure of the image processing apparatus which concerns on 1st Embodiment.</figref><figref num="2">The figure which shows the relationship between the tuner and the signal of the receiving channel.</figref><figref num="3">The figure which shows the flowchart which shows the operation of 1st Embodiment.</figref><figref num="4">The block diagram which shows the structure of the image processing apparatus which concerns on 2nd Embodiment.</figref><figref num="5">The figure which shows the flowchart which shows the operation of the 2nd Embodiment.</figref><figref num="6">The block diagram which shows the structure of the image processing apparatus which concerns on 3rd Embodiment.</figref>

(First Embodiment) FIG. 1 is a block diagram of the image processing apparatus 1 according to the first embodiment of the present invention. The image processing device 1 is received by a receiving unit 2 for receiving a broadcast wave including a plurality of channels Ch1 to ChN (N is an integer of 2 or more), a registration unit 3 for registering a recognition target, and a receiving unit 2. The determination unit 6 that determines whether or not the recognition target registered by the registration unit 3 exists in the signal of the channel, and the channel that is received by the reception unit 2 in sequence according to the determination result determined by the determination unit 6. A control unit 8 for switching is provided. Generally, in a broadcast wave, signals of a plurality of channels are assigned to frequency bands corresponding to each channel.

The receiving unit 2 receives the broadcast wave. Here, among a plurality of channels Ch1 to ChN included in the broadcast wave, a case where the user is viewing channel Ch1 using an input unit such as a remote controller is shown as an example. The receiving unit 2 visualizes the signal of the received channel with the tuner unit 9 and sends it to the feature amount extracting unit 4 for each frame.

The registration unit 3 registers the input recognition target. The recognition target is an arbitrary shape such as a person or an object, and includes at least one of a still image and a moving image. The registration unit 3 sends at least one or more image frames including the recognition target to the feature amount extraction unit 4 for each image frame. The registration unit 3 stores the feature amount sent from the feature amount extraction unit 4 in a storage unit (not shown) as described later. The registration unit 3 also sends the stored feature amount to the similarity calculation unit 5. When the feature amount to be recognized is extracted in advance, the registration unit 3 stores the feature amount and sends it to the similarity calculation unit 5.

The feature amount extraction unit 4 extracts the feature amount to be recognized in each image frame from the image frame sent from the receiving unit 2 and the image frame sent from the registration unit 3. The feature amount extraction unit 4 sends the feature amount extracted from the image frame sent from the registration unit 3 to the registration unit 3. The feature amount sent to the registration unit 3 is stored in the storage unit in the registration unit 3. The feature amount extraction unit 4 may store the feature amount in a storage unit outside the registration unit 3. Further, the feature amount extraction unit 4 sends the feature amount extracted from the image frame sent from the reception unit 2 to the similarity calculation unit 5. Here, it is assumed that the feature amount extracted by the feature amount extraction unit 4 is a set of local image patterns based on the feature points obtained by corner point detection (see Non-Patent Document 1). Further, it is assumed that the recognition target area of the image frame sent from the registration unit 3 is known. In addition, the local pattern shall be labeled in the area to be recognized and in the other areas.

The similarity calculation unit 5 calculates the similarity between the feature amount stored in the storage unit in the registration unit 3 and the feature amount extracted from one image frame of the signal by the feature amount extraction unit 4. The similarity calculation unit 5 searches for the most similar local pattern in the registered local pattern for each local pattern extracted from the current image frame, and labels from the identification results of all the local patterns in the image frame. The histogram of is obtained, and the degree of similarity with each recognition target is obtained. The similarity calculation unit 5 sends the calculated similarity to the determination unit 6.

The determination unit 6 determines whether or not there is a recognition target in one image frame of the signal sent from the reception unit 2 based on the similarity calculated by the similarity calculation unit 5 as described later. If the similarity calculated by the similarity calculation unit 5 is equal to or higher than a preset threshold value, the determination unit 6 determines that the recognition target exists in the image frame. If the similarity calculated by the similarity calculation unit 5 is less than a preset threshold value, the determination unit 6 determines that the recognition target does not exist in the image frame. The determination unit 6 sends the determination result to the notification unit 7 and the control unit 8, respectively.

The notification unit 7 notifies the user of the determination result sent from the determination unit 6. As a method of notification, for example, when a speaker is connected to the image processing device 1, a sound can be notified, and when a display is connected, an image can be notified. However, the notification method is not limited to these, and both sound and image may be used, or the user may be notified by another method perceptible to the user. When the image processing device 1 is connected to the network, an e-mail may be sent to a preset address to notify the user. Further, it may be set not to notify the user by the specification from the user.

The control unit 8 sequentially switches a plurality of channels included in the broadcast wave based on the determination result sent from the determination unit 6. In general, the number of channels that can be simultaneously visualized while receiving the video being broadcast is equal to the number of tuners used in the receiver 2. If the number of tuners is smaller than the number of channels included in the broadcast wave, all channels cannot be received. Therefore, it was not possible to perform indexing processing at the same time while receiving the video being broadcast. On the other hand, in the present embodiment, it is possible to simultaneously perform indexing processing while receiving a video or the like being broadcast on more channels than the number of tuners. For example, when the user watches channel Ch1 with the tuner 9a, the control unit 8 sequentially switches channels Ch2 to ChN not watched by the user and controls the tuner 9b to receive the signal of each channel. As a result, the signal of the channel Ch1 that the user is watching is continuously received unless the user changes the channel, but the signals of the channels Ch2 to ChN that the user is not watching are sequentially switched and received. ..

FIG. 2 is a diagram showing the relationship between the tuner and the signal of the receiving channel. In FIG. 2, the user is watching channel Ch1 and is receiving channel Ch2 among the remaining channels Ch2 to ChN. The control unit 8 instructs the receiver unit 2 to switch channels every second period. The receiving unit 2 sequentially switches channels Ch2 to ChN according to the instruction of the control unit 8 to receive a signal of any of channels Ch2 to ChN. Since the user is watching channel Ch1, the signal of channel Ch1 is continuously visualized by the tuner 9a. The control unit 8 sequentially switches the channels Ch2 to ChN received by the reception unit 2, visualizes the signal sequentially with the tuner 9b, switches to the channel ChN, and then returns to the channel Ch2 to repeat the channel switching. When the channel that the user is watching is changed, the channels other than the changed channel may be sequentially switched.

For the channel determined by the determination unit 6 that the recognition target exists in the current image frame, the channel is not switched, and the channel is not switched until the recognition target does not exist in the image frame in the first period. Continues to receive the signal of. When the determination unit 6 determines that the recognition target does not exist in the current image frame, the control unit 8 sequentially switches channels that the user is not watching in each second period. The control unit 8 sequentially sends the information of the channel to be switched to the receiving unit 2. The first period and the second period may have the same time interval or different time intervals.

In the above description, the channel is switched by a tuner other than the tuner used for viewing by the user, but the present invention is not limited to this. For example, the signal of the channel received by the tuner used for the user's viewing may be read from the display storage unit and transferred to the processing storage unit to determine whether or not the recognition target exists. As a result, it is possible to determine whether or not a recognition target exists even for the signal of the channel received by the tuner used for viewing by the user. As a result, it is possible to acquire the signal of the channel in which the recognition target exists by using all the tuners used in the receiving unit 2. Further, although the case where the user is watching one channel is shown, the present invention is not limited to this. When the user watches a plurality of channels at the same time, the channels other than the plurality of channels that the user is watching may be sequentially switched. Further, as described above, by reading the signals of the plurality of channels that the user is viewing from the storage unit for display and transferring them to the storage unit for processing, the channel that the user is viewing and feeding can also be obtained. , It may be determined whether or not the recognition target exists.

FIG. 3 is a diagram showing a flowchart showing the operation of the image processing device 1.

In step S301, the control unit 8 sequentially switches a plurality of channels included in the broadcast wave at preset time intervals. By sequentially changing the channels, it becomes possible to receive all receivable channels.

In step S302, the receiving unit 2 receives the signal of the channel instructed by the control unit 8. The receiving unit 2 visualizes the received signal with a corresponding tuner and sends it to the feature amount extracting unit 4 for each image frame. The feature amount extraction unit 4 extracts the feature amount to be recognized from the image frame sent from the receiving unit 2 and the image frame sent from the registration unit 3, respectively. The feature amount extraction unit 4 sends the feature amount extracted from the image frame sent from the registration unit 3 to the registration unit 3.

In step S303, the determination unit 6 determines whether or not the signal received by the reception unit 2 has a recognition target in one image frame. The determination unit 6 determines whether or not there is a recognition target based on the similarity calculated by the similarity calculation unit 5 as described above. This similarity corresponds to the similarity between the feature amount of the recognition target stored in the registration unit 3 and the feature amount extracted by the feature amount extraction unit 4. If the recognition target exists (YES in step S303), the determination unit 6 proceeds to step S304. If the recognition target does not exist (NO in step S303), the determination unit 6 proceeds to step S305.

In step S304, the control unit 8 fixes the receiving channel without switching the channel. As a result, the image frame to be recognized existing in the fixed channel can be continuously acquired (recorded, etc.).

In step S305, the control unit 8 determines whether or not to end the reception of the channel. For example, the reception time of the channel may be set in advance, and the end of reception may be determined based on this.

As described above, in the present embodiment, the signals of a plurality of channels included in the broadcast wave are visualized, and the similarity between the feature amount extracted from each image frame and the feature amount of the recognition target extracted in advance is evaluated. By sequentially switching the channels to be visualized using the evaluation result at this time, it is possible to search for images, sounds, etc. (indexing process) at the same time while receiving the video or the like being broadcast.

(Second embodiment) FIG. 4 is a block diagram of the image processing device 101 according to the second embodiment of the present invention. The image processing device 101 is different from the image processing device 1 in that it further includes a recording unit 109 and a tracking unit 110. Of the image processing apparatus 101, the same configuration as that of the image processing apparatus 1 will not be described.

The receiving unit 2 receives signals of a plurality of channels included in the broadcast wave. The receiving unit 2 visualizes the signal of the channel instructed by the control unit 8 among the received signals by using two or more tuners. When the determination unit 6 determines that the tracking is successful, the reception unit 2 sends the visualized signal to the feature amount extraction unit 4 for each image frame, as will be described later.

In the image frame sent from the receiving unit 2, the feature amount extracting unit 4 learns a model based on the face image area of the recognition target learned in advance, and detects the face area based on the model. The feature amount extraction unit 4 extracts the feature amount for each detected face region. This feature amount is a normalized image obtained by detecting a face part and normalizing the image according to the position of the face part. The normalized image can also be preprocessed with a filter or the like that suppresses illumination fluctuations.

The similarity calculation unit 5 calculates the similarity between the registered feature amount and the feature amount of the face region detected from the current image frame. The similarity calculation unit 5 sends the calculated similarity to the determination unit 6.

If the similarity calculated by the similarity calculation unit 5 is equal to or higher than a preset threshold value, the determination unit 6 determines that the recognition target exists in the image frame. If the similarity calculated by the similarity calculation unit 5 is less than a preset threshold value, the determination unit 6 determines that the recognition target does not exist in the image frame. The determination unit 6 also succeeds in tracking if the tracking likelihood calculated by the tracking unit 110 as described later is equal to or higher than a threshold value different from the above, and fails in tracking if the tracking likelihood is less than the threshold value. Is determined. The determination unit 6 sends the determination result to the notification unit 7, the control unit 8, and the recording unit 109. Further, when it is determined that the registered recognition target exists or the tracking is successful, the determination unit 6 transfers the position information of the registered recognition target face area in the current image frame to the tracking unit 110. send.

The control unit 8 continuously receives the channel determined by the determination unit 6 that the recognition target exists in the current image frame without switching the channel until the tracking of the registered recognition target fails.

When the determination unit 6 determines that the recognition target exists, the recording unit 109 starts recording from the image frame in which the recognition target is recognized. In the tracking unit 110, which will be described later, if the image frame for which tracking has failed continues for a certain period of time, recording is stopped. The recording end processing may be performed based on the end of the current program program obtained from the electronic program guide (EPG) or the like.

In the image frame sent from the receiving unit 2, the tracking unit 110 limits the range and the size of the face area by the position information of the sent face area, and uses the model learned from the face image area of the recognition target learned in advance. Detects the face area based on. The tracking unit 110 also calculates the similarity between the detected face region and the face region of the previous image frame, and sends the calculated similarity as the tracking likelihood to the determination unit 6.

FIG. 5 is a diagram showing a flowchart showing the operation of the image processing device 101. Of the operations of the image processing apparatus 101, steps S501 to S503, S505, and S506 are the same as steps S301 to S305 in FIG. 3, and thus description thereof will be omitted.

In step S504, the tracking unit 110 calculates the similarity between the detected face area and the face area of the previous image frame, and sends the calculated similarity as the tracking likelihood to the determination unit 6. If the tracking unit 110 determines that the tracking is successful (YES in step S504), the tracking unit 110 proceeds to step S505. If it is determined that the tracking is not successful (NO in step S504), the tracking unit 110 proceeds to step S506.

As described above, in the present embodiment, for the channel determined that the recognition target exists in the current image frame, the recognition target is not recognized from the image frame in which the registered recognition target is recognized for a certain period of time, or the recognition target is not recognized. Receive the signal until the recognition target is no longer tracked. Then, the channels are sequentially switched using the evaluation result at this time, and when the registered recognition target is recognized, the subsequent processing is switched to tracking, and tracking is continued until the recognition target cannot be tracked.

(Third embodiment) FIG. 6 is a block diagram of the image processing device 201 according to the third embodiment of the present invention. Of the image processing devices 201, the description of the same configuration as that of the image processing device 101 will be omitted.

The receiving unit 2 sends an image frame to the inter-frame similarity calculation unit 211 in addition to the feature extraction unit 4 and the tracking unit 110.

The inter-frame similarity calculation unit 211 calculates the similarity between image frames in temporally continuous image frames on the same channel. The inter-frame similarity calculation unit 211 resizes the entire image frame to a constant size, then obtains a cosine correlation value as a single vector and uses it as the similarity. The inter-frame similarity calculation unit 211 sends the calculated inter-frame similarity to the determination unit 6.

The determination unit 6 reduces the processing time allocated to the control of channels other than the channel that recognizes the recognition target or the channel for which tracking is successful. Specifically, the determination unit 6 reduces the ratio of the processing time allocated to the channels whose inter-frame similarity is continuously less than the threshold value sent from the inter-frame similarity calculation unit 211 to be smaller than that of the other channels. ..

As described above, in the present embodiment, the video is stored from the recognized image frame of the registered person or the like until the person is not recognized or tracked for a certain period of time. By lowering the priority of the channel having a small difference between the frames at this time and performing the processing, it is possible to preferentially increase the processing time of the channel in which the recognition target appears.

Each function of each part of the image processing apparatus according to the embodiment of the present invention can also be realized by a program stored in a computer. Further, the present invention is not limited to the above-described embodiment, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments. In addition, components across different embodiments may be combined as appropriate.

1: Image processing device, 2: Receiver 2: Registration, 4: Feature extraction, 5: Similarity calculation, 6: Judgment, 7: Notification, 8: Control