Video synthesis device and program

Video synthesis device and program

Technical field

The present invention relates to an apparatus and a video synthesis program which, when a first and a second video is introduced, displays a synthesized video combining the second video with the first video.

Prior art

With the development of high-speed network infrastructure and with the development of high-capacity recording media, the absolute amount of video data that can be handled at the same time by the user or by the computer team has increased dramatically. User video With this trend, various functions have been achieved using a series of video data sets and high-performance applications based on those functions have emerged. As one of these functions, there is a function called "image in image".

"Image in image" is a function to display two videos at the same time, superimposing a small child frame on the screen (parent frame). This function is used, for example, for "multi-angle representation" for the purpose of displaying, in the child frame, a video taken from a different viewing angle with respect to the video for the parent frame, or for "viewing comments" to effects of displaying additional information in the style of comments related to the video of the parent box (for example, displaying in the child box a video of comments from the director or similar, recorded with unpublished stories during filming).

Image in image is achieved, for example, by decoding two sets of video data on two different layers and superimposing these decoded videos, as shown in Figure 17. In this case, the video for the child frame is adjusted in size. of visualization and in the position displayed, to be arranged on a parent frame. Also, the child frame during image in image may have an arbitrarily video, different from the rectangular video shown in Figure 17. The image-in-image function and the method to achieve it are described, for example, in patent document 1 .

Patent document 1: Japanese patent application for public inspection 2005-123775.

Document 2004/0098753 discloses a system that digitally decodes and combines parts of two or more video broadcast signals in a memory of a decoder, in the manner described by a description of the presentation. The description of the presentation can be transferred as part of a video broadcast signal

or it can be accessed through a network. Different descriptions of the presentation can be sent to different decoders, depending on the type of decoder or user preferences. The description of the presentation can be modified by user entries or by memorized user preferences. Combination methods include substitution, logical and mathematical operations or a combination thereof. The description of the presentation can include dynamic variables that specify the combination mode for a series of frames or a specified display period.

WO 2004030351 discloses an apparatus for receiving a first and a second video signal. The first video signal represents a sequence of images that have a first size, and the second video signal represents a sequence of images that have a second size, the second size being smaller than the first size of the images. The apparatus combines said first and second video signals, so that it obtains a composite video signal, suitable for viewing in a display unit.

Description of the invention

Problems to be solved by the invention

In conventional image imaging, the displayed position of the child frame would be determined in advance, and the child frame would be displayed in that position.

In image in image, the video of the child frame is superimposed and visualized on the video of the parent frame, and therefore part of the video of the parent frame will be covered by the video of the child frame when the child frame is displayed. For this reason, it is preferred that the displayed position of the child frame in the parent frame can be changed based on the changing content of the video of the parent frame.

In addition, as an image-to-image application, there is a possible configuration in which the video of the child frame is provided so that it can begin to play, pause and restart, freely

in arbitrary moments within a specific period. The child frame is displayed only when the video is playing. This setting is used when, for example, the video of the child frame is one that is added as a privilege video to the video of the parent frame and, therefore, it is not necessary that the video of the child frame be synchronized exactly with the frame father, but the video of the child picture (privilege video) should be allowed to be played alone for a specific period, in the video of the parent frame. Also in this case, it is preferred that the position displayed in which the child frame should be displayed in the parent frame, is properly provided based on the video content of the parent frame, each time the child frame is displayed.

However, despite said demand there has been no conventional method to provide a modifiable visualized position for the child frame based on the change in the video of the parent frame, as described above. Therefore, it has been impossible to make the application described above feasible.

The present invention has been conceived in view of the above problem, and therefore has the objective of providing data for the display that specifies the display period, the zone to be displayed or the display zone at any time, in relation to the position display of the child frame during image to image playback. It is also an objective to publicize an apparatus and a video synthesis program that, based on the data for the visualization, can provide the appropriate viewing position of the child frame, even when the playing time and the time are freely modified. Stop time of the video of the child frame, as described above.

Means to solve the problems

The problems are solved by the appended claims.

Advantage of the invention

The invention discloses data for visualization on the displayed position of a child frame during image-to-image reproduction, providing the time that can be displayed and the area that is displayed or the area that is visible. The data for display may be included in the video data of the video of the child frame or the video of the parent frame, or they may be stored in management data that are independent of the video data and are handled with the video data during the transmission or distribution of the video. In the apparatus and the video display method, the data for the display is used to determine the displayed position of the child frame in correspondence with the playing time of the video of the parent frame each time it is read. These configurations, when a child frame video is synthesized for display in the parent frame video in image in image, allow the child frame video to be displayed for playback by displaying it in the appropriate displayed position. As a result, the video of the child frame can be freely switched between display and non-display, within the range of time that can be viewed. In addition, each time the video of the child frame is freely switched between display and non-display, the video of the child frame can be synthesized for viewing in a suitable position. Therefore, it is possible to perform the display during image in image in the manner provided by the distributor.

Brief description of the drawings

[Figure 1] Figure 1 is a functional block diagram showing a schematic configuration of the video display apparatus according to the first, second and third embodiments of the present invention.

[Figure 2] Figure 2 is a diagram showing an example of data for display, used in the video display apparatus according to the first embodiment of the present invention.

[Figure 3] Figure 3 is a diagram showing another example of data for display, used in the video display apparatus according to the first embodiment of the present invention.

[Figure 4] Figure 4 is a diagram showing a variation of the data for viewing, used in the video display apparatus according to the first embodiment of the present invention.

[Figure 5] Figure 5 is a diagram showing another example of visualization data, used in the video display apparatus according to the first embodiment of the present invention.

[Figure 6] Figure 6 is a flow chart showing the process when a video is displayed on the video display apparatus according to the first, second or third embodiment of the present invention.

[Figure 7] Figure 7 is an illustrative diagram showing a first display state when a video is displayed on the video display apparatus according to the first embodiment of the present invention.

[Figure 8] Figure 8 is an illustrative diagram showing a second display status when viewed

a video in the video display apparatus according to the first embodiment of the present invention. [Figure 9] Figure 9 is an illustrative diagram showing a third viewing state when a video is displayed on the video display apparatus according to the first embodiment of the present invention.

[Figure 10] Figure 10 is an illustrative diagram showing a fourth display state when viewed

a video in the video display apparatus according to the first embodiment of the present invention. [Figure 11] Figure 11 is a diagram showing an example of display data used in the video display apparatus according to the second embodiment of the present invention.

[Figure 12] Figure 12 is an illustrative diagram showing a first display state, when a video is displayed on the video display apparatus according to the second embodiment of the present invention.

[Figure 13] Figure 13 is an illustrative diagram showing a second display state, when a video is displayed on the video display apparatus according to the second embodiment of the present invention.

[Figure 14] Figure 14 is an illustrative diagram showing a third viewing state, when a video is displayed on the video display apparatus according to the second embodiment of the present invention.

[Figure 15] Figure 15 is an illustrative diagram showing a fourth viewing state, when a video is displayed on the video display apparatus according to the second embodiment of the present invention.

[Figure 16] Figure 16 is a diagram that shows a process when a video is displayed on the device

video display in accordance with the third embodiment of the present invention. [Figure 17] Figure 17 is an illustrative diagram showing a method of performing an image function in conventional image.

Description of reference numerals 1, 2, 3 video display devices 101, 103 decoders 102, 104 buffer units 105 synthesizer 106 adjuster 107 display unit 108 input unit 109 process controller 110, 210, 310 position designation device Best mode for carrying out the invention Next, referring to the drawings, a video synthesis apparatus according to

the present invention, when applied to a video display apparatus that displays a synthesized video. (First realization)

The apparatus, the video display method and the data for the display according to the first embodiment of the present invention will be described with reference to Figures 1 to 10.

Figure 1 is the functional block diagram showing a schematic configuration of a video display apparatus 1 according to the first embodiment of the present invention. The video display apparatus 1 receives two sets of video data (encoded video stream), decodes them and combines them for display in a so-called "image-in-picture representation" state. In the following description, the video displayed in the parent frame and the video displayed in the child frame in the image-to-image representation will be called, respectively, "main video" and "secondary video" to differentiate them.

The video display apparatus includes: a decoder 101 and a buffer storage unit 102 for decoding the video data of the main video and controlling its output; a decoder 103 and a buffer storage unit 104 to decode the video data of the secondary video and control its output; a synthesizer 105 with an adjuster 106 incorporated therein, to combine the secondary video with the main video; and a display unit 107 to display the video output. In addition, the apparatus also includes: an input unit 108 for receiving a user switching instruction to display / not display the secondary video (child frame); a process controller 109 for controlling the process of decoder 103 and / or buffer storage unit 104 in accordance with the switching; and a position designation device 110 to designate the displayed position of the secondary video (child frame) from the data for display and the temporary information during playback relative to the secondary video, which are entered separately. In this description, this data for display, to be used to designate the displayed position of the secondary video (child frame), is called "metadata", in contrast to the video data.

In this case, the video display apparatus 1 has been described including decoders 101 and 103, but this is not essential. For example, if the input video data is video data that has not been encoded, the video display apparatus 1 need not include decoders 101 and 103. Also, the video display apparatus 1 of Figure 1 is constructed only with functional blocks related to the processing of the video data (data related to the video signals). However, in practice the video data includes, in addition to the data that involves video signals, audio data and management data (information necessary to decode encoded data, such as the coding system, etc., and necessary information to play video, such as a display list to designate cutting and joining the video, and the like), and the video display device itself is built including the functional blocks to process this data. In this case, the configuration of Figure 1 is installed as an internal structure of the video display apparatus itself.

To begin with, a description of the process of the video display apparatus 1 will be made when no secondary video is displayed (child frame). On this occasion, no secondary video data is introduced or, even if they are introduced, the video data is subjected to a non-viewing process.

The input video data of the main video is decoded by the decoder 101, and the decoded video is set for timing by the buffer unit 102, and delivered. Since no secondary video is displayed, the decoded video delivered from the buffer unit 102 passes through synthesizer 105 without being processed therein, and is supplied to the display unit 107. In this way, the main video is displayed as is.

Next, a description of the process will be made in the video display apparatus 1 when a secondary video (child frame) is displayed.

The video data of the inserted secondary video is decoded in the decoder 103, and the decoded video is set for timing by means of the buffer unit 104, and delivered. The decoded image of the secondary video is introduced into the adjuster 106 comprised in the synthesizer 105.

Adjuster 106, as a preprocess to synthesize the secondary video with the main video, transforms and adjusts the image size and the position displayed on the screen of the decoded image of the secondary video. Next, the secondary video (child frame) is set to be synthesized in the displayed position that is designated by the designation device 110 of the position described below, within the main video (parent frame). Then, after adjustment, the secondary video is synthesized with the decoded image of the main video inserted, so that the synthesized video is delivered and displayed by the display unit 107. In addition, it is also possible to make the main video look at through the secondary video synthesized, establishing a transmittance when both are synthesized.

The video display apparatus 1 includes the input unit 108, which receives from the user a switching instruction to display / not display the secondary video (child frame). Next, the input unit 108, based on the switching instruction introduced, generates a display status information that indicates whether the secondary video (child frame) should be displayed or not displayed at the current time, and transfers it to the process controller 109 and the position designation device 110.

The process controller 109 receives the display status information from the input unit 108 and controls the process of the decoder 103 and / or the buffer unit 104, based thereon. For example, when the display status information has gone to "status not displayed", the controller stops the decoding process in the decoder 103 and / or the output of the buffer unit 104, and restarts these processes when the display status information goes to "displayed status", to pause the secondary video during the non-display period.

The position designation device 110 receives from the input unit 108 the display status information, and when the secondary video (child frame) is in the displayed state, it determines using the mentioned metadata the position displayed of the secondary video (frame son), to be displayed in the main video (parent frame), and notifies the result to adjuster 106.

Since the main video changes over time, the position displayed on the main video, where you want the secondary video to be displayed, or can be viewed, changes temporarily when you change the main video. Therefore, if playback and display of the secondary video is restarted some time after the display of the secondary video has stopped and has been paused, by process controller 109 and decoder 103 and / or unit 104 of buffer storage, which are controlled by process controller 109 as indicated above, It is not always desirable that the secondary video be displayed in the same position as before, when its display was stopped. The data for viewing for the secondary video provided in the present invention, namely the metadata, is the data that is provided with information on where the secondary video should, or may, be displayed in the main video, with each temporary position in The main video The position designation device 110, using the metadata that is entered with the video data of the secondary video, delivers the displayed position of the secondary video (child frame) corresponding to the temporary position indicated by the temporary information during playback.

Referring to Figures 2 to 5, the metadata for video viewing, handled in the present embodiment, will be described in greater detail.

Figures 2 and 3 show specific examples of metadata, in accordance with the secondary video display provided in the present invention. The video stream (secondary video stream) contained in the video data consists of a header part and a video data part. The header part includes several kinds of information related to the flow, and this header part includes the metadata.

Figures 2 and 3 show, each, a specific metadata structure (Figure 2 (a), Figure 3 (a)) and a diagram showing the displayed area or viewable area, designated by metadata (Figure 2 (b) , figure 3 (b)). Likewise, a diagram is added (figure 2 (c), figure 3 (c)) that schematically shows the displayed area or viewable area in one dimension, to allow an easy understanding of the temporal variation of the displayed area or viewable area. That is, the vertical axis in Figures 2 (c) and 3 (c) represents the two-dimensional position on the screen, and the vertical width of the band shown corresponds to the size of the area displayed or area viewable.

Figure 2 (a) shows an example of a metadata structure. Metadata consists of: total playback time 200 of a secondary video; 201 information of the display time, which represents the time interval of the main video that allows the secondary video to be displayed based on the playback time of the main video (the playing time based on "00:00:00", the initial position Of reproduction); and information 202 of the displayed area, which shows the position of the main video in which the secondary video is displayed at any time in the display time interval. In this case, the information 202 of the displayed area of ​​Figure 2 provides the upper left vertex of the child frame, assuming that the secondary video (child frame) has a predetermined fixed display size. For example, the secondary video is displayed from the instant "00:00:10" with its upper left vertex located at (x1, y1). Of course, it should be understood that the coordinates of the vertex are not limited to the coordinates of the upper left point, but that the coordinates of the center of the secondary video can be used, for example.

Figure 2 (b) shows a two-dimensional representation of the displayed area, where the secondary video is displayed at individual time points of the main video. For example, from the instant "00:00:15" to the instant "00:00:30", the secondary video is synthesized and displayed in the main video area, with its upper left vertex set at the coordinates (x2, y2 ).

Figure 2 (c) shows a one-dimensional representation of the displayed area, in which the secondary video is displayed. The vertical direction shows the spatial position (area) in the main video and the horizontal direction shows the time (time position of the main video). For example, the upper left vertex of the secondary video moves from the coordinates (x1, y1) to (x2, y2) at the instant "00:00:15". In Figure 2 (c), the displayed area of ​​the secondary video in the main video is provided by a band zone that changes its position at the instant "00:00:15" and "00:00:30"

Figure 3 (a) also shows an example of a metadata structure. The metadata shown in Figure 3 (a) consists of: total playback time 300 of a secondary video; information 301 of the display time, which represents the time interval of the main video that allows the secondary video to be displayed, based on the playing time of the main video; information 302 of the viewable area, which shows the area of ​​the main video in which the secondary video can be viewed (viewing allowed), at any time in the viewable time interval. In this case, the information 302 of the viewable area shown in Figure 3 provides the coordinates of two points, the upper left and lower right vertices, for the area in which the child frame can be displayed. For example, referring to Figure 3 (b), it is shown that from the instant "00:00:10" the secondary video (child frame) can be displayed in a rectangular area that has the upper left vertex in the coordinates ( x1, y1) and the lower right corner of the coordinates (x1 ', y1'). If the secondary video (child frame) has a predetermined fixed display size, and when the viewable area designated by the information 302 of the viewable area of ​​Figure 3 is larger than the display size of the child frame, the secondary video may be displayed in any arbitrary position within the viewable area, when it is displayed. In addition, the secondary video (child frame) that is being viewed can be moved or enlarged within the range of the viewable area. For example, in Figure 3c, the area in which the secondary video can be viewed in the main video is specified by a band zone that changes its position and width in the instants "00:00:15" and "00:00 : 30 ".

Although in the two examples shown in Figures 2 and 3, the displayed (visible) area specified by metadata has been described under the hypothesis that the secondary video (child frame) has a fixed size, the size of the secondary video is not limit this; The information in the displayed area is adapted to provide the display size of the secondary video itself. That is, similar to Figure 3, the displayed area is specified by the two-point coordinates, the upper left and lower right vertices, and the secondary video can be displayed so that the secondary video is enlarged or reduced to suit the size of the displayed area.

The table in Figure 4 shows variations to configure the time intervals in which the visualized area (visualizable) is specified and for the descriptive format of the visualized area (visualizable), in relation to the metadata provided by the present invention. In this case, Figure 4 shows the cases in which the visualized (visualizable) areas are limited to rectangular shapes.

There are different ways to configure the time intervals; one is to specify arbitrary sections and another is to provide a display area (viewable) for each section of a fixed unit. In this case, when arbitrary sections are specified, if it is assumed that there is no temporary separation or overlap within a continuous duration, one of the start and end moments of a section can be omitted. In addition, in the table of Figure 4, a commonly used temporary notation, "hours: minutes: seconds", is used as an example. However, temporary notation is not limited to this; for example, the total time can be provided in a "seconds" or "milliseconds" format. On the other hand, when a visualized (visualizable) area is provided for each section of the fixed unit, a visualized (visualizable) area is provided to each unit of an arbitrary time, for example, every second, every 250 milliseconds, every minute or similar, different every five seconds, as in the example in Figure 4. In addition, during the encoding of the video it is possible to use a unit other than time, such as a frame unit and a GOP unit (Group Of Picture, group of images). The duration of a unit section is properly adjusted based on the properties of the stored video.

Descriptive formats of the displayed area (viewable) can be specified in several ways, including a single set of coordinates, two sets of coordinates, and a set of coordinates plus one size. Of these, the case in which the area can be determined with a single set of coordinates, is one in which the display size of the secondary video has been previously determined. When the area is specified with two sets of coordinates or with a set of coordinates with a given size there are two possible cases where the display size of the secondary video is smaller than the specified area or that a so-called specified viewable area, and where the video Secondary is resized (enlarged or reduced in size) to the specified area or to the displayed area. As a viewable area, it is also possible to designate a band type area from top to bottom, or from left to right in the main video (for example, an area that covers the upper half or the lower half of the screen). Although in Fig. 4 the examples of the displayed area (viewable) are specified as rectangular areas, the displayed area (viewable) can be provided differently as a shape other than a rectangle, such as a polygon or an oval, or it can be created With an arbitrary profile. For example, an area can be created arbitrarily using an image of

mask for the form. In this case, the description of a specific descriptive format for an arbitrary form is omitted.

In addition, it is also possible to specify a visualized (visualizable) area that moves continuously over time, as shown in Figure 5, instead of the cases shown in Figures 2 and 3 in which the position of the displayed area ( viewable) changes discreetly at certain times. In this case, a visualized (visualizable) area included in the metadata can be provided (Figure 5 (a)), for example by a combination of a time section, a position of the displayed area (visualizable) at the initial temporal position of the time section, and a position of the area displayed (viewable) at the final time position of the time section, as shown in Figure 5. As an example, Figure 5 (b) shows a displayed area of ​​a child frame. In this case, at the instant "00:00:10" a child box is displayed in a displayed area with its upper left corner in the coordinates (x1, y1). Next, the displayed area is continuously displaced, so that the child frame is displayed, at the instant "00:00:20", in the area displayed with its upper left corner at (x2, y2). In addition, the displayed area moves continuously, so that the child frame is instantly displayed "00:00:40" in the area displayed with its upper left corner at (x3, y3). Figure 5 (c) shows the previous case in a schematic way, in which the displayed area or viewable area is represented in one dimension.

It should be noted that the method for specifying a continuously changing area is not limited to this. It is also possible to specify a displayed area (viewable) by providing its position in the initial time position, together with a unit of variation (motion vector).

Furthermore, in the present invention, an area specified by the metadata is treated as a visualized area (an area in which the visualization is performed) or a visualizable area (an area in which visualization is allowed). On the contrary, this can also be understood as that areas other than the above are specified as prohibited viewing areas (areas where viewing is not allowed). That is, the present invention can be similarly applied to metadata that specify display time and prohibited display areas.

Next, referring to Figures 6 to 10, a description of the specific operation will be made when the secondary video that is synthesized with the main video is reproduced and displayed using the metadata for visualization described hereinafter.

Figure 6 is a flow chart showing a process when a secondary video is displayed, which includes the display / non-display switching of the secondary video (child frame). This flowchart mainly shows the operations of the position designation device 110, the process controller 109 and the synthesizer 105, of the apparatus components of the video display apparatus 1 shown in Figure 1. Figures 7 to 10 show an example of an operating result when a secondary video is synthesized and visualized in the video display apparatus 1 of Figure 1, in accordance with the flow chart of Figure

6. In Figures 7 to 10, a compact black part indicates the time during which the secondary video is displayed, and the position displayed at that time.

In the description that follows, the reproduction and visualization process will be described by taking an example of the metadata shown in Figure 2, in which the size of the displayed area is equal to the display size of the secondary video. However, even if metadata describing a so-called viewable area is used, that is, a case in which the size of the displayed area is, therefore, larger than the display size of the secondary video, the basic operation does not change except in that an appropriate display position is selected by the designation device 110 from the viewable area, and delivered.

The position designation device 110 reads metadata (step S1), and then determines whether the current playing time of the main video is within the viewable time based on the display time information (201 in Figure 2) included in the metadata (stages S2 and S3). If it is earlier than the start time of the viewable time, the secondary video is not displayed and is expected at the beginning of the viewable time (step S2; no).

If the current playing time in the main video is within the viewable time (step S2; yes -> step S3; no), the designation device 110 of the position takes input unit 108 a switching instruction between the states visualized and not visualized of the secondary video. In this case, when the instruction to view the secondary video is received, so that the secondary video is in the displayed state (step S4; yes), a process of decoding the secondary video is implemented, so that an image is delivered decoded (step S5). In addition, the position designation device 110 acquires the temporary information relative to the current position of the viewing time in the main video (step S6) and determines the displayed position of the secondary video, corresponding to the current position of the viewing time, based on metadata (step S7). Next, synthesizer 105 synthesizes and displays the secondary video in the position

displayed designated in the main video (step S8). When the secondary video data has not been completed (step S9; no), the operation passes step S3 to continue the process.

On the other hand, when the secondary video is ordered to go to the undisplayed state by the use, by the user, of the switching instruction between the displayed and unseen states of the secondary video (S4; no), the processes of decoding and broadcast of the secondary video (step S10) to temporarily stop the display of the secondary video itself.

When the playback of the secondary video has ended (S9; yes), or when the display time of the main video has exceeded the final time of the display period of the secondary video (S3; yes), the secondary video display process is terminated.

Figures 7 to 10 are diagrams that schematically show the positional relationships between the main video and the secondary video. The vertical direction shows the spatial position in the main video and the horizontal direction shows the time. Then the main video begins to be broadcast in the instant "00:00:00". The drawings also show the status of the secondary video displayed when the metadata structure shown in Figure 2 (a) is used.

First, Figure 7 is a diagram showing the situation so far "00:00:13". Referring to the metadata structure of Figure 2 (a), the display time of the secondary video starts from the instant "00:00:10". Then, when a user control is performed in the time "00:00:13" to display the secondary video (step S2; yes -> stage S3; no -> stage S4; yes, figure 6), the video Secondary is decoded (step S5). The secondary video is synthesized with the main video, and the visualization (compact black part in figure 7) of the secondary video is started in the position displayed corresponding to the instant "00:00:13", designated by metadata.

Second, Figure 8 is a diagram showing the situation so far "00:00:20". Referring to the metadata structure of Figure 2 (a), the process of changing the displayed area of ​​the secondary video at the instant "00:00:15" is described. In this sense, the position designation device 110 modifies the displayed position of the secondary video, in accordance with the information 202 of the displayed area contained in the metadata (step S7). Then, when a signal is input to configure the secondary video status in the non-displayed state, through the input unit 108 at the instant "00:00:20" (step S4; no), it is delivered from the designation device 110 from position to synthesizer 105 a signal to stop secondary video output. As a result, synthesizer 105 stops the secondary video output (step S10).

Next, figure 9 is a diagram showing the situation up to the instant "00:00:28", in which the display has been switched back to the visualized state of the secondary video (child frame). On this occasion, the secondary video is returned to the playback state from the pause state, and the secondary video that was displayed at the instant "00:00:20" is then displayed. At that time, the secondary video (child frame) is displayed in a displayed position corresponding to the time "00:00:28" designated by the metadata.

Next, Figure 10 is a diagram showing the situation up to the instant "00:00:36", in which the reproduction of the secondary video is finished with a total playing time of "15 seconds". Referring first to the metadata described in Figure 2 (a), the displayed area of ​​the secondary video is modified at the instant "00:00:30" (step S7). Then, at the instant "00:00:36", in which the total playing time of "15 seconds" has elapsed, the secondary video output is stopped (step S9; yes).

As described so far, in the video display apparatus 1 according to the first embodiment of the present invention, when the secondary video is synthesized and visualized with the main video by using the metadata that provide the visual area and the viewable area or the displayed area of ​​the secondary video, The displayed position of the secondary video within the main video can be appropriately designated in correspondence with the viewing time. As a result, the secondary video can be freely switched between the displayed and non-displayed states, within the viewable time interval. In addition, even if the switching between the displayed and non-displayed states is freely performed, it is possible to prevent the secondary video from being synthesized and displayed in an unfavorable position for the main video.

The metadata described above is shown in Figure 1 of the present embodiment, so that they are entered independently of each video data set. For example, when the management data for the management of video data (information such as an encoding system and the like, required to decode encoded data, and information required to view the video, such as a display list that specifies cutting and union of the video) are provided as a separate stream regarding the video data, The data can be stored in the management data in order to be provided to the video display apparatus 1. Alternatively, as already shown in Figure 2 (a) or in Figure 3 (a), the metadata described above may be supplied being stored in the video stream that includes the video data of the

secondary video In this case, it is necessary to provide a process for separating the metadata from the video stream of the secondary video, before the stream is introduced into the video display apparatus 1.

In addition, since the metadata mentioned above is used when the secondary video is played, it is generally considered that the metadata is provided in a unique correspondence with the secondary video. However, it is also possible, for example, to consider a method of use such that the main video has a set of metadata that are applied in common to a series of secondary video sets. In this case, the metadata can be stored in the video data (video data stream) of the main video. In addition, although in Figures 2 (a) and 3 (a) the mentioned metadata are stored in the header position of the video stream, the storage position is not limited to this. For example, when video data is transmitted by dividing it into a series of packets, the metadata can be incorporated as a new packet between one video packet and the next, or stored in the packet header of each video packet.

The distributor can display a secondary video in image-to-image mode, in a displayed position that is intended by the distributor, providing metadata along with the video data in the manner described above.

In addition, although the synthesizer 105 of the video display apparatus 1 shown in Figure 1 adjusts only the secondary video and makes no adjustment for the main video (ie, the main video is displayed in full screen), it is possible to use a synthesizer 105a which also includes an adjuster 106a (as an additional adjuster, independent of the adjuster 106 of the secondary video) on the input side of the decoded video of the main video, so that it produces an output by adjusting both the main and secondary videos (synthesizer 105a and adjuster 106a are not illustrated in particular). However, in this case, since the metadata represents the area displayed (viewable) in the main video over which the secondary video is synthesized and displayed, when the main video is adjusted using the said adjuster 106a, it is necessary to adjust the area displayed (viewable) of the secondary video that is provided by metadata, in accordance with the previous setting. For example, illustratively, if the main video is reduced by half for viewing, both vertically and horizontally, the area displayed (viewable) of the secondary video that is synthesized and displayed with the main video is also compressed in half , vertically and horizontally. Although this reference is not particularly mentioned in the other embodiments, this issue should apply exactly the same to the other embodiments.

(Second embodiment)

Next, a video display apparatus, a method and data for display will be described, in accordance with the second embodiment of the present invention, using Figures 1, 6, and 11 to 15.

The schematic configuration of the video display apparatus 2 according to the second embodiment of the present invention can be provided by the functional block diagram of Figure 1, similar to the first embodiment. However, in the second embodiment, the metadata handled in this case are different from those of the first embodiment. With respect to the operation of the display device, only the operation of the position designation device is different between the video display device 1 (position designation device 110) and the video display device 2 (device 210 of designation position). Therefore, in the following, the metadata used in the video display apparatus 2 of the second embodiment, and the specific operation during playback using these metadata, will be described primarily by focusing on the difference from the first embodiment.

Figure 11 shows an example of metadata handled in the second embodiment. The metadata (figures 2 and 3) exemplified in the first embodiment are to provide a visualized area of ​​a secondary video (child frame) in the main video, which is favorable to the main video when a secondary video is viewed within the viewable time. Therefore, in the metadata shown in Figure 2 and Figure 3, the displayed area of ​​the secondary video is provided so that it corresponds to each instant of the main video, based on the axis of the playback time relative to the main video By contrast, the metadata according to the second embodiment shown in Figure 11 is to provide a preferable visualized area in which, when a secondary video is viewed, the secondary video itself can preferably be displayed, depending on the content of the video. secondary and creative intention. Therefore, in metadata according to the second embodiment, the displayed area of ​​the secondary video is provided to be in correspondence with each instant of the playback time of the secondary video, based on the axis of the playback time relative to the secondary video.

In this case, the preferable displayed position depending on the content of the secondary video is applied to a use case in which, for example, when a secondary video of 10 seconds is composed of a scene of a person A facing to the right during the first five seconds, and from another scene of a person B looking to the left in the remaining five seconds, the secondary video image is displayed on the side

left of the screen in the first five seconds, and the secondary video image is displayed on the right side of the screen in the remaining five seconds, so that both people A and B look towards the center of the screen. Of course, this is only an example, and it is not always desirable that both people look to the center, but that the positions in which the images of the secondary video are displayed depend on the intention of the video director. In summary, it is understood that metadata according to the second embodiment shown in Figure 11, constitutes additional information to indicate the creative intention of the director of the secondary video, in the reproduction of the secondary video itself.

Similar to Figure 2 for the first embodiment, Figure 11 (a) shows a specific metadata structure, Figure 11 (b) shows the displayed area designated by the metadata and Figure 11 (c) schematically shows the change of the visualized area, in a one-dimensional representation for easy understanding. As described above, the horizontal axes in Figures 11 (b) and 11 (c) represent the temporal position of reproduction of the secondary video. The vertical axis in Figure 11 (c) represents the two-dimensional spatial position on the screen, and the vertical width of the band shown corresponds to the size of the area displayed.

The metadata shown in Figure 11 (a) consists of: information 1101 of the viewable time, which represents the time interval of the main video that allows the secondary video to be displayed; and information 1102 of the displayed area, which shows the position of the main video in which the secondary video should be displayed at each temporary point of view of the secondary video, based on the temporal axis of reproduction in the secondary video. However, the information 1101 of the display time is not essential and can be omitted. If omitted, all part of the main video is understood as the viewable time of the secondary video.

In Fig. 11, the case in which the displayed area is simply specified with the coordinates of the upper left vertex of the child frame (or the center of the child frame) has been described, as an example of the information 1102 of the displayed area, in the hypothesis that the display size of the secondary video (child frame) has a predetermined fixed size. However, the information of the displayed area is not limited to this. Similar to the first embodiment, two sets of coordinates may be provided to designate a viewable area (see Figure 3), or two sets of coordinates may be provided to designate a displayed area in which the secondary video is displayed with enlargement or reduction. In Figure 11 (c), the displayed area in which the secondary video should be displayed is represented as a band type area, which changes its position at the time "00:00:05" of secondary video playback (ie , in total, five seconds after the start of secondary video playback) and "00:00:10" (that is, in total, ten seconds after the start of secondary video playback).

Next, referring to Figures 6, and 12 to 15, a description of a specific operation will be made when the secondary video is synthesized with the main video, reproduced and visualized using the metadata shown in Figure 11.

Similar to the first embodiment, the process in which the video display apparatus 2 according to the present embodiment carries out the visualization of the secondary video including the display switching / is shown by the flow chart of Fig. 6 no display of the secondary video (child frame). This flowchart shows the operations of the position designation device 210, the process controller 109 and the synthesizer 105, of the apparatus components of the video display apparatus 2 shown in Figure 1.

Similar to the description of the first embodiment, the following description will describe the reproduction and visualization processes using the metadata that provide the visualized area. However, even if metadata describing the viewable area is used, the basic operation does not change except that a suitable displayed position is selected from the viewable area, by means of the position designation device 210, to be delivered.

The position designation device 210 reads the input metadata (step S1), and then, based on the information 1101 of the display time included in the metadata, determines whether the current playing time in the main video falls within the time viewable (stages S2 and S3). If the current playback time is earlier than the initial moment of the viewable time, no secondary video is displayed and is expected at the beginning of the viewable time (step S2; no).

If the current playing time in the main video is within the viewable time (step S2; yes -> step S3; no), the designation device 210 of the position takes input unit 108 a switching instruction between the states visualized and not visualized of the secondary video. In this case, when the instruction to view a secondary video is received and the secondary video is in the displayed state (step S4; yes), a process of decoding the secondary video is implemented, so that a decoded image is delivered (stage S5). In addition, the position designation device 210 acquires the temporary information

relative to the current position of the playing time in the secondary video (step S6) and determines the position displayed corresponding to the current position of the playing time of the secondary video, based on the metadata (step S7). Next, the synthesizer 105 synthesizes and displays the secondary video in the displayed position designated in the main video (step S8). In the previous way, there are two different points with respect to the first embodiment: in step S6, the temporary position of total reproduction of the secondary video itself is acquired as the time information; and in step S7 the position displayed corresponding to the temporary playback position of the secondary video is determined using the metadata.

Figures 12 to 15 are diagrams schematically showing an example of the operational result when a secondary video is synthesized and displayed in the video display apparatus 2. In this case, in the video display apparatus 2 of this embodiment, the metadata is controlled based on the playing time of the secondary video, which represents the position in which the secondary video has been played and displayed, regardless of the time of reproduction of the main video, so that each of figures 12 to 15, (a) show how the area displayed is designated by metadata based on the playing time of the secondary video and (b) show how the secondary video is synthesized and displayed with the main video based on the playing time of the main video. In the drawings (b) of Figures 12 to 15, the compact black part designates a period during which the secondary video is displayed, and the position displayed at that time.

First, Figure 12 is a diagram showing the situation so far "00:00:13". Referring to the metadata structure of Figure 11 (a), the viewable time of the secondary video starts from the instant "00:00:10". Then, when a user control is performed in the time "00:00:13" to display the secondary video (step S2; yes -> stage S3; no -> stage S4; yes, figure 6), the video Secondary is decoded (step S5). This secondary video is synthesized with the main video, and the secondary video is displayed in the position displayed corresponding to the instant "00:00:13", designated by the metadata. In this case, Figure 12 (a) shows the situation in which the video begins to be emitted from "00:00:00" in the secondary video's playing time. Figure 12 (b) shows the situation in which the secondary video begins to be broadcast when the playback time of the main video is "00:00:13".

Second, Figure 13 is a diagram showing the situation so far "00:00:20". Referring to information 1102 of the displayed area of ​​the metadata structure of Figure 11 (a), the displayed area of ​​the secondary video is modified by "00:00:05" at the time of secondary video playback. Therefore, as shown in Figure 13 (a), the area displayed changes to "00:00:05" in the secondary video's playing time. As a result, in the synthesized video the position displayed is modified at the instant "00:00:18", which corresponds to a moment five seconds after the reproduction (visualization) of the secondary video has begun, as shown in the figure 13 (b). Then, when the secondary video enters the status not displayed in "00:00:20" at the time of the main video playback, the secondary video display in the main video stops. In this condition, the secondary video has been played until "00:00:07".

Next, Figure 14 is a diagram showing the situation up to the instant "00:00:28", in which the display has been switched back to the visualized state of the secondary video (child frame). On this occasion, the secondary video is returned to the playback state from the pause state, and then the secondary video that was played in "00:00:20" in time in the main video, in other words, the secondary video begins to be played from the time position "00:00:07" in the time of the secondary video (the time position corresponding to the total playing time of 7 seconds). The displayed position of the secondary video (child frame) is provided by metadata, so that the secondary video is displayed in a displayed position corresponding to "00:00:07" in the time position of the secondary video (in the position of time corresponding to the total playback time of 7 seconds).

Next, Figure 15 is a diagram showing the situation until "00:00:36" in the time of the main video, in which the playback of the secondary video has ended with a total playing time of "15 seconds". Referring to the information 1102 of the displayed area, included in the metadata described in Figure 11 (a), the position of the secondary video is modified at "00:00:10" at the time of the secondary video (at the position of time corresponding to the total playing time of 10 seconds). Therefore, in "00:00:10" in the time of the secondary video, or in "00:00:31" in the time of the main video, the position of the secondary video is modified.

As described hereinabove, in the video display apparatus 2 according to the present embodiment, when a secondary video is synthesized and visualized with the main video by using the metadata that provide the displayed area (or viewable area) of the secondary video, it is possible to synthesize and visualize the secondary video with the main video specifying the position in which the secondary video should be viewed, which has been previously determined based on the content of the secondary video and / or the creative intention. As a result, the secondary video can be freely switched between the displayed and non-displayed states. Likewise, if the switching between the displayed and non-displayed states is carried out freely, it is

It is possible to synthesize and visualize the secondary video in the visualized position, in accordance with the content of the secondary video and / or with the creative intention.

Also, with respect to the metadata of the present embodiment, the metadata can be supplied in a manner, either by storing them in a similar manner to the first embodiment, for example, in the data flow of the management data that are independent of the data. video, or by storing them in the video stream that includes the video data of the secondary video, as shown in Figure 11 (a). When stored in the video stream, it is necessary to provide a process of separating the metadata from the video stream of the secondary video, prior to its entry into the video display apparatus 2. It should be noted that, since the metadata according to the second embodiment is provided in univocal correspondence with the secondary video, the metadata is normally added to the video data of the secondary video or to the management data related to the secondary video. In addition, although in Figure 11 (a) the metadata is stored in the header position of the video stream, the storage position is not limited to this. For example, when video data is transmitted by dividing it into a series of packets, the metadata can be incorporated as a new packet between one video packet and the next, or stored in the packet header of each video packet.

(Third embodiment).

Next, a video display apparatus, a method and the data for display will be described, in accordance with the third embodiment of the present invention, using Figures 1, 6 and 16.

The schematic configuration of a video display apparatus 3 according to the third embodiment of the present invention can be provided by the functional block diagram of Figure 1, similar to the first and second embodiments. However, since the individual operation of the position designation device 110 is different, in the present embodiment the position designation device is designated 310. The process of carrying out the visualization of the secondary video in the video display apparatus 3 according to the third embodiment is also provided by the flow chart shown in Figure 6, similar to the first and second embodiments. . In the following, the operation of the video display apparatus 3 according to the third embodiment will be described, focusing on the difference with respect to the video display apparatus 1 of the first embodiment.

In the video display apparatus 3 of the present embodiment, two kinds of metadata, described in the first and second embodiments, are introduced as metadata for the display of a secondary video, so that the displayed area of ​​the secondary video is determined based on the combination of these two sets of metadata. Accordingly, the designation device 310 of the position of the video display apparatus 3 receives two kinds of metadata and two sets of temporal information (positional information with the playing time of the main video and positional information with the playing time of the video secondary) (step S6 in the flowchart), and determines an appropriate viewing area for the secondary video (step S7 in the flowchart).

Figure 16 is a schematic diagram showing the states of the main and secondary videos. Figure 16 (a) shows the viewable area of ​​the designated secondary video in relation to the main video, provided by the metadata described in the first embodiment, while Figure 16 (b) shows the designated displayed area in relation to the video itself secondary, provided by the metadata described in the second embodiment. Figure 16 (c) is a diagram showing how the displayed area of ​​the secondary video during playback is designated by the metadata of Figure 16 (b). Figure 16 (c) is a diagram showing a situation in which the main video and the secondary video are synthesized and visualized using the metadata of Figures 16 (a) and 16 (b).

Figures 16 (c) and 16 (d), similar to the first and second embodiments, show the visualized position of a secondary video using the mentioned two kinds of metadata, when the secondary video begins to be displayed instantly " 00:00:13 ", stop displaying at the instant" 00:00:20 ", restart the display at the instant" 00:00:28 "and end the display at the instant" 00:00:36 ". In addition, a visualized area 16B corresponding to the secondary video shown in (b) is provided in Figure 16 (c), and a visible area 16A of the secondary video in the main video shown in (a) is shown in Figure 16 (d). ). The shaded areas or the compact black areas in Figure 16D show the periods in which the secondary video is displayed, and the positions displayed in those periods.

It is common for a secondary video to be provided to the main video as extra value-added content. Therefore, it is generally desirable that reproduction be performed avoiding as much as possible harm the main video. Therefore, when the two mentioned classes of metadata are provided, the displayed area is determined by giving priority to the viewable area 16A of the secondary video that is provided in relation to the main video, with respect to the displayed area 16B that is provided in relation to the own secondary video.

In Figure 16d, in a time interval 1601 ("00:00:13" to "00:00:15"), the viewable area 16A and the displayed area 16B exactly overlap each other, so that the displayed area The secondary video is determined based on both sets of metadata.

In a time interval 1602 ("00:00:15" to "00:00:20", and "00:00:28" to "00:00:30") the displayed area 16B is completely included in the 16A viewable area. Therefore, in the interval 1602 the secondary video is displayed in the displayed area that is provided to the secondary video itself based on the metadata, in a manner similar to that shown in the second embodiment.

In a time interval 1603 ("00:00:30" to "00:00:36"), the viewable area 16A of the secondary video provided to the main video and the displayed area 16B of the secondary video designated in accordance with the content itself from the secondary video, they are separated into different zones. In this case, priority is given to the viewable area 16A of the secondary video that is provided to the main video. That is, in the time interval 1603, the secondary video is displayed in the viewable area of ​​the secondary video that is provided to the main video based on the metadata, similar to that shown in the first embodiment.

Although not shown, when the viewable area shown in Figure 16 (a) and the displayed area shown in Figure 16 (b) to specify the position of the secondary video displayed, are located in different areas, and when the displayed area shown in figure 16 (a) it is larger than the display size of the secondary video (child frame), a configuration process of a visualized area of ​​the secondary video can be added, determining an area that is included in the viewable area of ​​Figure 16 (a) and is as close as possible to the area displayed in Figure 16 (b). Of course, conversely, if the creative intention of the secondary video is especially important, it is possible to forcefully set the visualized position of the secondary video, based on the displayed area of ​​Figure 16 (b), providing a high priority to the area shown in figure 16 (b).

In this case, in each of the embodiments of the present invention described above, the video data (and the management data) and the metadata introduced in the video display apparatus, can be supplied through such transmission paths as broadcasting and / or communication, or they can be recorded in advance on a recording medium, so that the vision data (and the management data) and the metadata recorded on the recording medium are read sequentially in order to reproduce them for viewing. The situation is the same for the case in which the data is recorded once on a recording medium through a transmission path, and then the recorded video data (and the management data) with the metadata are read for reproduction. That is, the video display apparatus, the method and the data for the display of the present invention can be applied as a component of a broadcasting video receiver, a video communication receiver and a recording and playback apparatus having a recording medium, and can also be applied to a recording medium on which the metadata described in each embodiment have been recorded.

Likewise, it is possible to control the metadata shown in each embodiment of the present invention separately with respect to video (and management) data. From this feature, when metadata is generated on the reproduction side, it is also possible to use the metadata generated on the reproduction side, in combination with the video data that is entered separately by broadcasting, communication or by means recording, when the video is played back in picture in picture. In this case, for example, metadata can be formed with a process such as setting user preferences, for areas of the main video that does not care about it being hidden and those that you do not want to be hidden, during playback of a secondary video. The generation of metadata on the playback side is carried out when the video data (and the management data) entered through a transmission path such as broadcasting, communication, etc., are recorded on the recording medium. , or immediately before the video data (and the management data) are reproduced after reading from the recording medium. This generation process can be done by direct user input, or dynamically using a program such as Java (registered trademark) or similar. That is, the present invention can be applied to a method and a video display apparatus using the metadata described in each embodiment, regardless of where the metadata has finally been configured.

In this case, the embodiments disclosed herein are, for all intents and purposes, illustrative and not restrictive. The scope of the present invention is defined by the scope of the claims and not by that of the above description, and it is anticipated that all changes that fall within the meaning and scope equivalent to the scope of the claims are covered by the claims.