Video production and compaction with collage picture frame user interface
Abstract
This record has no abstract on file.
Term
Term ended
Expired 13 November 2022, 3.9 years ago.
- Priority
- Filed
- Granted
- Expired
- Today
17 claims: 6 independent, 11 dependent
- 1ビデオを複数のビデオセグメントに分割するステップと、 少なくとも1つの個別のビデオフレームを有するビデオコラージュ・テンプレートを提供するステップと、 前記複数のビデオセグメントの中の1つのビデオセグメントを、前記ビデオコラージュ・テンプレートの前記個別のビデオフレームに関連付けるステップと、 重要度を示す値を前記関連付けられたビデオセグメントに割り当てるステップと、 特性ベクトルを前記関連付けられたビデオセグメントに割り当てるステップと、 前記重要度を示す値と前記特性ベクトルとに基づいて、前記関連付けられたビデオセグメントの再生時間が短縮されるように前記関連付けられたビデオセグメントをコンパクト化するステップと、 前記ビデオコラージュ・テンプレートと、前記関連付けられたビデオセグメントまたは前記コンパクト化されたビデオセグメントとから、ビデオコラージュを作成するステップと、を含む、 ビデオコラージュの作成方法。
- 2前記複数のビデオセグメントの中の1つのビデオセグメントを関連付けるステップが、 前記複数のビデオセグメントの中から、複数のビデオセグメントを選択するステップと、 前記選択された複数のビデオセグメントの各々を、前記ビデオコラージュ・テンプレートの個別のフレームと関連付けるステップと、を含む、請求項1に記載の方法。
- 3前記複数のビデオセグメントの中の1つのビデオセグメントを関連付けるステップが、 各々が前記複数のビデオセグメントのうちの1つを代表する、複数の代表画像を提供するステップと、 前記複数の代表画像の中から1つの代表画像を選択するステップと、 前記代表画像を、前記ビデオコラージュ・テンプレートの前記個別のビデオフレームと関連付けるステップと、を含む、請求項1に記載の方法。
- 4ビデオセグメント・テンプレートを提供するステップをさらに含み、 前記ビデオセグメント・テンプレートが複数の代表画像を含み、各代表画像が前記複数のビデオセグメントのうちの1つと関連付けられており、 前記ビデオセグメントを関連付けるステップが、前記複数の代表画像の中の1つの代表画像を、前記ビデオコラージュ・テンプレートの前記個別のビデオフレームと関連付けるステップを含む、請求項1に記載の方法。
- 5前記ビデオをセグメントに分割する前記ステップが、前記ビデオを選択された数のセグメントに分割するステップを含む、請求項1に記載の方法。
- 6前記ビデオをセグメントに分割する前記ステップが、ジェネティック分割アルゴリズム(GSA)を利用して前記ビデオをセグメントに分割するステップを含む、請求項1に記載の方法。
- 7前記重要度を示す値が、前記ビデオセグメントが関連付けられた前記個別のビデオフレームの表示サイズに関連している、請求項1に記載のビデオコラージュの作成方法。
- 8前記特性ベクトルが、前記ビデオセグメントの内容のアクティビティに関連している、請求項1に記載のビデオコラージュの作成方法。
- 9少なくとも1つの個別のビデオフレームを有するビデオコラージュ・テンプレートと、 前記個別のビデオフレームのうちの少なくとも1つにはめ込まれた、ビデオが分割された複数のビデオセグメントの中の1つのビデオセグメントの代表画像であり、重要度を示す値及び特性ベクトルが割り当てられた代表画像と、 を表示し、 前記代表画像 が選択手段により選択される ことにより、前記代表画像の前記重要度を示す値と前記特性ベクトルとに基づいて再生時間が 予め 短縮された、前記代表画像に関連する前記ビデオセグメントを 再生する 、 ビデオコラージュ表示装置。
- 10前記ビデオコラージュ表示装置は、複数の個別のビデオフレームを表示し、前記複数の個別のビデオフレームの各々に代表画像がはめ込まれ、各代表画像は1つのビデオセグメントと関連付けられている、請求項9に記載のビデオコラージュ表示装置。
- 11前記代表画像がはめ込まれた前記個別のビデオフレームの表示サイズに応じて、重要度を示す値が前記代表画像に割り当てられている、請求項9に記載のビデオコラージュ表示装置。
- 12前記個別のビデオフレームの表示サイズの比と、前記関連するビデオセグメントの内容のアクティビティとに基づいて前記特性ベクトルの長さの比が決定される、請求項9に記載のビデオコラージュ表示装置。
- 13ビデオを複数のビデオセグメントに分割する分割手段と、 少なくとも1つの個別のビデオフレームを有するビデオコラージュ・テンプレートを提供するテンプレート提供手段と、 前記複数のビデオセグメントの中の1つのビデオセグメントを前記ビデオコラージュ・テンプレートの前記個別のビデオフレームに関連付ける関連付手段と、 重要度を示す値を前記関連付けられたビデオセグメントに割り当てる重要度割当手段、 特性ベクトルを前記関連付けられたビデオセグメントに割り当てる特性ベクトル割当手段と、 前記重要度を示す値と前記特性ベクトルとに基づいて、前記関連付けられたビデオセグメントの再生時間が短縮されるように前記関連付けられたビデオセグメントをコンパクト化するコンパクト化手段と、 前記ビデオコラージュ・テンプレートと、前記関連付けられたビデオセグメントまたは前記コンパクト化されたビデオセグメントとから、ビデオコラージュを作成するビデオコラージュ作成手段と、を含む、 ビデオコラージュ作成装置。
- 14前記関連付手段が、前記複数のビデオセグメントの中から、複数のビデオセグメントを選択し、前記選択された複数のビデオセグメントを、前記ビデオコラージュ・テンプレートの個別のフレームと関連付ける、請求項13に記載のビデオコラージュ作成装置。
- 15前記分割手段が、前記ビデオを選択された数のセグメントに分割する、請求項13に記載のビデオコラージュ作成装置。
- 16前記分割手段が、ジェネティック分割アルゴリズム(GSA)を利用して前記ビデオをセグメントに分割する、請求項13に記載のビデオコラージュ作成装置。
- 17コンピュータに、 ビデオを複数のビデオセグメントに分割する手順と、 少なくとも1つの個別のビデオフレームを有するビデオコラージュ・テンプレートを提供する手順と、 前記複数のビデオセグメントの中の1つのビデオセグメントを前記ビデオコラージュ・テンプレートの前記個別のビデオフレームに関連付ける手順と、 重要度を示す値を前記関連付けられたビデオセグメントに割り当てる手順と、 特性ベクトルを前記関連付けられたビデオセグメントに割り当てる手順と、 前記重要度を示す値と前記特性ベクトルとに基づいて、前記関連付けられたビデオセグメントの再生時間が短縮されるように前記関連付けられたビデオセグメントをコンパクト化する手順と、 前記ビデオコラージュ・テンプレートと、前記関連付けられたビデオセグメントまたは前記コンパクト化されたビデオセグメントとから、ビデオコラージュを作成する手順と、を実行させるための、 ビデオコラージュ作成プログラム。
Independent claims17
1 paragraph, as filed
[0001] [Technical field to which the invention belongs] The present invention relates to the creation and editing of video. [0002] [Conventional technology] Video cameras continue to be popular in homes, offices, and other uses. For example, video cameras are often used to record family events such as vacations, weddings, and graduation ceremonies. In the office, video cameras are often used to record presentations, meetings, user surveys, and fieldwork, and are often carried on business trips to record people, places, and activities. [0003] In recent years, the development of a system for connecting a personal computer and a video camera as an interface and storing the video in the memory of the personal computer is progressing. As videos can be transferred from videotapes, capture cards and memory sticks to personal computers in a variety of formats, there is a growing demand for storing, viewing, editing and transferring videos. ing. [0004] Unedited videos are usually a mix of interesting and long, boring parts that can be watched once or twice, but then stowed away. Also, unedited video is of poor quality due to sudden camera movements and parts where the shooting time is too long or too short, so watch it before the next part of interest appears. It often makes people bored. [0005] Unedited videos are not only long and tedious, but often require significant storage capacity to store information. The storage capacity required to store unedited video files is approximately a few bytes to a few hundred megabytes. Unedited video consumes a lot of storage capacity, is difficult to transmit video using email or other electronic transmission systems, and is generally slow. Adobe Existing video editing systems, such as Premiere®, are aimed at professional video producers, so it can take some time to assemble and mix video and audio tracks to produce edited video. There are many. Another system is the co-pending U.S. Patent Application No. 09 / 618,533 "Interactive Custom Video Production System" filed by the same applicant as this application (filed July 18, 2000, see this patent application) Systems such as those described in (incorporated herein) are listed, which are relatively easy to use but require a significant number of user interactions to produce an edited video. [0006] [Non-Patent Document 1] Adobe Premiere, [online], Adobe Systems Incorporated, Internet <URL: http://www.adobe.com/products/premiere/> [Non-Patent Document 2] Apple iMovie, [online], Apple Computer, Inc., Internet <URL: http://www.apple.com/imovie/> [Non-Patent Document 3] Patrick Chiu et al., A Generic Algorithm for Video Segmentation and Summarization, IEEE International Conference on Multimedia and Expo (ICME), (USA), 2000, Volume 3, p.1329-1332 [Non-Patent Document 4] Andreas Girgensohn et al., A Semi-Automatic Approach to Home Video Editing, User Interface Software and Technology 2000 (UIST 2000), (USA), 2000, ACM Press, p.81-89 [Non-Patent Document 5] In-Sync Speed Razor, [online], in-sync corporation, internet <URL: http://www.in-sync.com/> [Non-Patent Document 6] Javu Technologies JavuNetwork, [online], Internet <URL: http://www.javu.com/> [Non-Patent Document 7] Allan Kuchinsky et al. FotoFile; A Consumer Multimedia Organization and Retrieval System , Conference on Human Factors in Computing Systems 1999 (CHI 1999), (USA), 1999, ACM Press, p.496-503 [Non-Patent Document 8] Shingo Uchihashi et al. Summarizing Video Using a Shot Importance Measure and a Frame Packing Algorithm, IEEE International Conference on Acoustics, Speech, and Signal Processing 1999 (ICASSP 1999), (USA), 1999, Volume 6, p.3041- 3044 [Non-Patent Document 9] Shingo Uchihashi et al. Video Manga: Generating Semantically Meaningful Video Summaries , ACM Multimedia 1999, (USA), 1999, ACM Press, p.383-392 [0007] [Problems to be Solved by the Invention] On existing systems, when trying to send a video electronically, the user has the option of sending the entire video unedited or spending hours producing the edited video and then sending the edited video. There was only. In some circumstances it may be necessary to entrust video editing to an expert, which is often undesirable and time consuming. [0008] From the above, it is possible to devise a system and method that divides a video with a relatively small number of user inputs and allows the user to easily select a portion of the video for viewing, saving or transferring. desirable. [0009] [Means for solving problems] The present invention includes a method of creating a video collage. This method divides a video into multiple video segments, provides a video collage template with at least one individual video frame, and one video segment of the plurality of video segments. With the steps associated with the individual video frames in the video collage template,<u style="single">The association is based on a step of assigning a value indicating importance to the associated video segment, a step of assigning a characteristic vector to the associated video segment, and a value indicating the importance and the characteristic vector. With the step of compacting the associated video segment so that the playback time of the video segment is shortened,</u>May be included. When a video segment is associated with a video collage template, the system will use the video collage template and the associated video segment.<u style="single">Or the compacted video segment</u>And then create a video collage. In the present invention, "dividing a video into a plurality of video segments" means dividing a video on the time axis. [0010] In some embodiments, the present invention comprises at least one individual video frame and<u style="single">The video is a representative image of one of the divided video segments, and is assigned a value indicating importance and a characteristic vector.</u>Representative image and<u style="single">To display,</u>Video collage<u style="single">Display device</u>including. The representative image is fitted within the at least one individual video frame. [0012] In yet another aspect, the invention may include a device for creating a video collage. This device<u style="single">A partitioning means for dividing a video into a plurality of video segments, a template providing means for providing a video collage template having at least one individual video frame, and a video collage of one video segment in the plurality of video segments. An associative means associated with the individual video frame of the template, an importance assigning means for assigning a value indicating importance to the associated video segment, and a characteristic vector assigning means for assigning a characteristic vector to the associated video segment. A compacting means for compacting the associated video segment so that the playback time of the associated video segment is shortened based on the value indicating the importance and the characteristic vector, and the video collage. A video collage creation means for creating a video collage from a template and the associated video segment or the compacted video segment.</u>May include<u style="single">。</u>[0013] Next, specific aspects of the present invention will be described below. A first aspect of the present invention includes a step of dividing a video into a plurality of video segments, a step of providing a video collage template having at least one individual video frame, and one of the plurality of video segments. A step of associating a video segment with the individual video frame of the video collage template, a step of assigning a value indicating importance to the associated video segment, and a step of assigning a characteristic vector to the associated video segment. A step of compacting the associated video segment so that the playback time of the associated video segment is shortened based on the importance value and the characteristic vector, and the video collage template. A method of creating a video collage, comprising the step of creating a video collage from the associated video segment or the compacted video segment. In the second aspect, in the first aspect, the step of associating one video segment among the plurality of video segments is a step of selecting a plurality of video segments from the plurality of video segments, and the selection. Includes a step of associating each of the plurality of video segments created with individual frames of the video collage template. In the third aspect, in the first aspect, a plurality of representative images in which the step of associating one video segment among the plurality of video segments represents one of the plurality of video segments, respectively. It includes a step of providing, a step of selecting one representative image from the plurality of representative images, and a step of associating the representative image with the individual video frame of the video collage template. A fourth aspect further comprises the step of providing a video segment template in the first aspect, wherein the video segment template is a plurality of representative images. Each representative image is associated with one of the plurality of video segments, the step of associating the video segment with one representative image of the plurality of representative images of the video collage template. It includes a step of associating with the individual video frame. A fifth aspect comprises, in the first aspect, the step of dividing the video into segments comprises the step of dividing the video into a selected number of segments. A sixth aspect, in the first aspect, comprises the step of dividing the video into segments using a genetic division algorithm (GSA) to divide the video into segments. In a seventh aspect, in the first aspect, the value indicating the importance is related to the display size of the individual video frame with which the video segment is associated. In the eighth aspect, in the first aspect, the characteristic vector is related to the activity of the content of the video segment. A ninth aspect is a video collage template having at least one individual video frame and one of a plurality of video segments in which the video is divided, fitted into at least one of the individual video frames. A representative image of one video segment, a representative image to which a value indicating importance and a characteristic vector are assigned, and the representative image are displayed. Includes In a seventh aspect, in the first aspect, the value indicating the importance is related to the display size of the individual video frame with which the video segment is associated. In the eighth aspect, in the first aspect, the characteristic vector is related to the activity of the content of the video segment. A ninth aspect is a video collage template having at least one individual video frame and one of a plurality of video segments in which the video is divided, fitted into at least one of the individual video frames. A representative image of one video segment, a representative image to which a value indicating importance and a characteristic vector are assigned, and the representative image are displayed. Includes In a seventh aspect, in the first aspect, the value indicating the importance is related to the display size of the individual video frame with which the video segment is associated. In the eighth aspect, in the first aspect, the characteristic vector is related to the activity of the content of the video segment. A ninth aspect is a video collage template having at least one individual video frame and one of a plurality of video segments in which the video is divided, fitted into at least one of the individual video frames. A representative image of one video segment, a representative image to which a value indicating importance and a characteristic vector are assigned, and the representative image are displayed.<u style="single">Is selected by the selection means</u>As a result, the playback time is based on the value indicating the importance of the representative image and the characteristic vector.<u style="single">In advance</u>The shortened video segment associated with the representative image<u style="single">Reproduce</u>, A video collage display device. In a tenth aspect, in the ninth aspect, the video collage display device displays a plurality of individual video frames, a representative image is fitted in each of the plurality of individual video frames, and each representative image is 1 Associated with one video segment. In the eleventh aspect, in the ninth aspect, a value indicating the importance is assigned to the representative image according to the display size of the individual video frame in which the representative image is fitted. In the twelfth aspect, in the ninth aspect, the ratio of the lengths of the characteristic vectors is determined based on the ratio of the display size of the individual video frames to the activity of the contents of the related video segment. A thirteenth aspect is a dividing means for dividing a video into a plurality of video segments, a template providing means for providing a video collage template having at least one individual video frame, and one of the plurality of video segments. An association means that associates a video segment with the individual video frame of the video collage template, an importance assignment means that assigns a value indicating importance to the associated video segment, and a characteristic vector to the associated video segment. Distributing characteristic vector A compacting means for compacting the associated video segment so that the playback time of the associated video segment is shortened based on the assigning means and the value indicating the importance and the characteristic vector. A video collage creation device comprising the video collage template and a video collage creation means for creating a video collage from the associated video segment or the compacted video segment. In the fourteenth aspect, in the thirteenth aspect, the related means selects a plurality of video segments from the plurality of video segments, and the selected video segments are combined with the video collage template. Associate with individual frames of. The fifteenth aspect is , A thirteenth aspect, the dividing means divides the video into a selected number of segments. In the sixteenth aspect, in the thirteenth aspect, the dividing means divides the video into segments using a genetic division algorithm (GSA). A seventeenth aspect is a procedure for dividing a video into a plurality of video segments, a procedure for providing a video collage template having at least one individual video frame, and one of the plurality of video segments. A procedure for associating a video segment with the individual video frame of the video collage template, a procedure for assigning a value indicating importance to the associated video segment, a procedure for assigning a characteristic vector to the associated video segment, and a procedure for assigning a characteristic vector to the associated video segment. Based on the value indicating the importance and the characteristic vector, the procedure for compacting the associated video segment so as to shorten the playback time of the associated video segment, the video collage template, and the video collage template. A program for executing a procedure for creating a video collage from the associated video segment or the compacted video segment. [0014] BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described below with respect to specific embodiments. Other objects, features, and advantages of the present invention will become apparent with reference to the specification and drawings of the present application. [0015] FIG. 1 shows an example of four video collage templates 100, 110, 120, 130 for inserting and arranging video segments according to an embodiment of the present invention. The video collage template contains each individual video frame that fits the video segment within it. For example, video collage template 100 includes individual video frames 101, 103, 105, 107. The video collage template 110 includes individual video frames 111, 113, 115. Video collage template 120 includes individual video frames 121, 123, 125, 127, 129. Video collage template 130 includes individual video frames 131, 132, 133, 134, 135, 136, 137, 138. Although the size of individual video frames is limited by the size of the video collage template, the shape and size of the video frames can be set arbitrarily. The shape of the individual video frames can be, for example, an ellipse such as the video frame 123, a circular shape such as the video frame 134, or any other shape. In addition, the number of video frames can be set arbitrarily. The number of individual video frames included in the video collage user interface can be 1, 2, 3 or any other number. [0016] The video collage template may be provided by the system or created by the user. Video collage templates can be any drawing, such as Microsoft PowerPoint®, Microsoft Visio®, Microsoft Paint®, Corel Draw®, Deneva Systems Canvas®, etc. It can be created using an application and saved as an electronic file in a storage device. Once created, a video collage template can be shared with other users or saved as an electronic file. [0017] FIG. 2 shows a video segment template 200 according to an embodiment of the present invention. The video segment template 200 includes representative images of the associated video segment (eg, representative images 201, 203, 205, 207, 209). The video is divided into a predetermined number of segments, a representative image is selected for each video segment, and this representative image is associated with a particular segment. The video segment template 200 may be used to present a representative image to the user. [0018] The representative image for each video segment can be selected by various means. For example, the image included in the first frame of the related video segment may be the representative image 201. Alternatively, an image analysis algorithm can be applied to each frame of the video segment to select a frame that simply represents the video segment, and the image of this frame can be used as a representative image. Alternatively, the user may view the video segment, select a scene or frame in the video segment, and use the image in the selected scene or frame as a representative image. [0019] FIG. 3 shows a video collage user interface 300 according to an embodiment of the present invention. The video collage user interface 300 includes a user-selected video collage template 310 and a video segment template 320. The video segment template 320 includes representative images of the video divided into segments (eg, representative images 321, 323, 325, 327, 329). The video collage user interface 300 may include a set of controller mechanisms 301, 303, 305, 307, 309 used to perform various functions. The user, for example, uses the arrow buttons 301, 303, 305, 307 to align the selection box 340 with the representative image 323 and activates the selection mechanism 309 to activate the representative image (eg, representative) from the video segment template 320. Image 323) can be selected. Further, the selection mechanism 309 may be activated again to play the video segment associated with the representative image 323. The user may also select a representative image and then insert the selected representative image into the video frame of the video collage template 310 using controller mechanisms 301, 303, 305, 307, 309. [0020] When the user inserts the representative image into individual frames of the video collage template 310, the representative image is displayed in the video frame in which the image is embedded. If you select a representative image, you can see the video segment associated with the representative image. Playback of the video segment associated with the representative image is displayed within a separate video frame containing the associated representative image, within an individual "window", or across a viewing device such as the display monitor 504. Can be done. [0021] [0021] The controller mechanisms 301, 303, 305, 307, 309, and selection box 350 are for illustrative purposes only, and the representative image and the video segment associated with the representative image can be selected, viewed, and viewed by other means. It is also possible to arrange. For example, the user can use any pointing device, such as the cursor control device 505 (see Figure 5), or select a representative image using the touch-sensitive display screen or the keystrokes of the keyboard 506. Can perform functions. [0022] FIG. 4 shows an example of a video collage 400 according to an embodiment of the present invention. Video Collage 400 is created using Video Collage User Interface 300 (see Figure 3). In creating the video collage 400, the user selects a representative image 321 and fits it into the video frame 311. As a result, the representative image 421 fitted in the video frame 411 of the video collage 400 is obtained. Next, the user selects the representative image 323 and fits it into the video frame 313. As a result, the representative image 423 fitted in the video frame 413 can be obtained. Next, the user selects the representative image 325 and fits it into the video frame 315. As a result, the representative image 425 fitted in the video frame 415 is obtained. The user then selects the representative image 327 and fits it into the video frame 317. As a result, the representative image 427 fitted in the video frame 417 is obtained. Next, the user selects the representative image 329 and fits it into the video frame 319. As a result, the representative image 429 fitted in the video frame 419 is obtained. [0023] Once created, the Video Collage 400 can be saved as a separate electronic file containing information about frame placement and display of the Video Collage 400, including selected representative images. The file may also include a video segment associated with the selected representative image. [0024] Also, the video collage or selected representative image may include a link to the location where the associated video segment is stored. For example, if the video is available via the Internet, the electronic file will only contain information about the display of the Video Collage 400, including the selected representative image, and a link to the location of each video segment associated with the representative image. It may be included. In this case, when a representative image is selected from the video collage 400, the video segment associated with the representative image flows on the Internet and is presented to the user. By including a link to each associated video segment in the electronic file, it is possible to store and transfer the electronic file even if the size of the electronic file of Video Collage 400 is very small. [0025] The video file format of the electronic file of Video Collage 400 can be any format as long as it can be viewed, saved and transferred by the user. For example, save Video Collage 400 electronic files in MPEG (Moving Picture Experts Group) files, AVI (Audio Video Interleave) files, DV (Digital Video) files, QuickTime files, or any other video file format. Can be done. [0026] Once the Video Collage 400 is created, users can publish the Video Collage 400 on a website, transfer it to other users using a transmission device such as e-mail, save it as an electronic file, or video. You can print a copy of the collage and create a DVD (Digital Video Disc), VCD (Video Compact Disc), CD (Compact Disc), etc. [0027] The scope of application of the present invention is not limited to the use of the Internet, but the Internet is taken up for the purpose of exemplifying embodiments of the present invention. The embodiments of the present invention can also be implemented on any type of end-to-end communication channel. For example, embodiments of the present invention may be implemented to work with an instant messaging system or an internal network such as a local area network (LAN), or over a peer-to-peer network. [0028] FIG. 5 shows a computing device architecture 500 suitable for implementing embodiments of the present invention. Computational device architecture 500 includes processor 502, storage device 503, Internet access device 510 such as a modem, and display monitor 504. Further, the computing device architecture 500 may be configured to include a cursor control device 505, a random access memory (RAM) 507, a read-only memory (ROM) 508, a keyboard 506, and a graphics coprocessor 509. All elements of Computational Device Architecture 500 are interconnected by a common bus 501 to transfer data between different elements. Bus 501 often contains data, addresses, and control signals. [0029] Although the architecture 500 of FIG. 5 has been taken up as a computing device architecture capable of executing the embodiment of the present invention, any other computing device architecture can be used for the implementation of the embodiment of the present invention. [0030] In one embodiment of the invention, the storage device 503 can be a manufacturing component, such as a computer-readable medium. Examples of storage device 503 include magnetic hard disks, optical disks, floppy disks, CD-ROMs, RAM, ROMs, or other readable and writable data storage technologies, which may be used alone or in combination. Can be used. [0031] The storage device 503 may include an operating system 511 capable of running a program or application using the computing device architecture 500. Examples of operating systems 511 include Microsoft Windows®, Apple Macintosh OS®, and Unix®. One embodiment of the present invention is implemented as the video collage software program 512 and is stored in the storage device 503. [0032] Embodiments of the invention, such as the video collage software program 512, include software programs, software objects, software functions, software subroutines, software methods, software instances, code fragments, hardware operations, and users. -It can take the form of operations, which can be used alone or in combination. The video collage software program 512 can be implemented using one, two or any number of computing devices 500. [0033] Those skilled in the art will appreciate that Figures 6, 7, and 8 show logic boxes for performing specific functions. The number of logic boxes can be increased or decreased depending on the embodiment. In one embodiment of the invention, the logic box provides software programs, software objects, software functions, software subroutines, software methods, software instances, code fragments, hardware operations, user operations. They may be represented, and these may be used alone or in combination. For example, the logic box can represent a portion of the video collage software program 512 shown in FIG. 5 or any other embodiment of the present invention. [0034] FIG. 6 is a flowchart 600 illustrating an outline of the steps for creating a layout of a video collage according to an embodiment of the present invention. [0035] The user starts the system in Logic Box 601 and provides the video they want to use when creating the layout of the video collage. Upon receiving the video, control goes to Logic Box 603. [0036] In Logic Box 603, the system generates a set of video segments from a given video. This set of video segments contains a predetermined number of segments in the video. In one embodiment, the user can select the number of video segments. Alternatively, the video can be divided into a preset number of segments. In addition, the system may determine the length of a given video and select the number of segments based on this length. [0037] In Logic Box 604, the user is presented with a set of predetermined video collage templates, such as video collage templates 100, 110, 120, 130 (see Figure 1), and the user selects one of them. .. Alternatively, the user may create their own video collage template and use it. [0038] Logic Box 605 presents the user with a video collage user interface. A video collage user interface, such as the video collage user interface 300 shown in Figure 3, is associated with the video collage template selected in logic box 604 and the video segment generated in logic box 603, respectively. Includes a video segment template 200 (see Figure 2) containing the representative images. [0039] As shown in Logic Box 607, the user selects a segment from the video segment template and fits the selected segment into the desired frame of the selected video collage template. Control proceeds to Logic Box 609 when the user fits an image into each of the individual frames of the video collage template, or when the user tells the system that the image fits into a separate frame of the video collage template. .. In Logic Box 609, a video collage is created from the selected video collage template and the selected representative image. [0040] FIG. 7 is a flowchart showing the steps of generating a video segment, shown as Logic Box 603 (see FIG. 6), according to an embodiment of the invention. [0041] The video split begins in Logic Box 701 in response to the video serving. Illustrates a 14-minute original video of an ice cream party, explaining the division of the video as shown in Flowchart 700. [0042] Once the video is provided, the first video split can be performed as shown in Logic Box 703. The initial video splitting can be done by a variety of well known video splitting means. For example, the video can be split by "shots". A "shot" is defined as the time from when a video camera starts recording to when it stops recording. Alternatively, the video may be split using an algorithm that detects scene switching. Shots and scenes may be manually split by the user. [0043] When the first video split occurs, control goes to logic box 704. The logic box 704 determines whether the number of segments generated by the logic box 703 is less than or equal to the number of segments (N) presented to the user as an option. The number of segments N presented to the user as an option may be set in advance, or may be set by the user. If it is determined that the number of video segments generated by logic box 703 is equal to or less than N, control proceeds to logic box 707. In Logic Box 707, the video segment is used as the video segment presented to the user. On the other hand, if the number of video segments is greater than N, control goes to logic box 705. [0044] See video example of an ice cream party. The video is first divided into 179 segments using a common color difference algorithm. Such an algorithm divides the video based on the color difference between frames. Other common split algorithms may be used for the initial video split. The number of segments of 179 is too much for the user to process quickly and easily, and N is set to 24, so control goes to logic box 705. [0045] Logic Box 705 selects N video segments from the set of video segments generated by Logic Box 703. This set of N video segments will be referred to as S (N) for convenience. A set of S (N) will contain a segment that simply shows the whole video. [0046] Video segment selection, as described in Logic Box 705, can be performed by a variety of video segmentation means. For example, N video segments may be selected using the Genetic Division Algorithm (GSA). For GSA, US Patent Application No. 09 / 611,389, "Genetic Division Method for Data such as Image Data Streams," filed by the same applicant as this application, pending at the same time (filed July 7, 2000. This patent application. Incorporated herein by reference). As another segment selection means, there is a method of focusing on the histogram difference of adjacent frames and setting the threshold value so that the point where the histogram difference exceeds the threshold value becomes the boundary of the segment. [0047] When N segments are selected, control goes to logic box 707. In Logic Box 707, a set of N video segments is used as a set of video segments presented to the user. [0048] In another embodiment, logic boxes 703, 704, 705 may be combined. In this case, the first video division is performed on the video having a predetermined number N of video segments. The division may be performed using GSA. In this case GSA is given N. The GSA divides the entire video into N segments, which are presented to the user in Logic Box 707. [0049] FIG. 8 schematically illustrates the steps of extracting and modifying selected video segments and creating a video collage, as shown in Logic Boxes 607 and 609 (see Figure 6), according to an embodiment of the invention. It is a flowchart which shows. [0050] The process starts in logic box 800 and control goes to logic box 801. In Logic Box 801, the user selects a representative image of the associated video segment for each frame of the video collage template by any of the means described above. When a representative image of the associated video is selected and this representative image is fitted into a separate video frame in the video collage template, control proceeds to Logic Box 803. [0051] In Logic Box 803, the video segments associated with the selected representative image are each extracted from the video by any video extraction means known in the art. Extracted video segment (s<sub>0</sub>, s<sub>1</sub>, s<sub>2</sub>, ..., s<sub>k-1</sub>) Is called U (k) for convenience. U (k) = {s<sub>0</sub>, s<sub>1</sub>, s<sub>2</sub>, ..., s<sub>k-1</sub>}, Where k is the number of segments selected by the user. Once each video segment is extracted, control goes to logic box 805. [0052] In Logic Box 805, each extracted video segment is assigned a value that indicates its importance. In one embodiment, in a video collage, segments assigned to large video frames are assigned high importance values and segments assigned to small video frames are assigned low importance values. The task of assigning a value indicating importance to the selected video segment may be performed by the user. [0053] In logic box 806, each segment is associated with a "characteristic vector" (represented by a (s)) to characterize the activity of the contents of each segment. Examples of content activities include (1) segment length, (2) time-dependent changes (visual changes, audio changes, text changes) in the medium. Assigning a characteristic vector to each video segment gives another characteristic that adjusts the video segment. This other property solves the problem of linearly mapping the length of video segments to the size of individual video frames. Linear mapping the length of a video segment to the size of an individual video frame creates the problem that when a short segment is assigned to a large frame, the other video segments become too short. [0054] An example of defining a characteristic vector will be explained by taking up the visual change between frames of one segment. The visual change can be determined by obtaining the difference in the color histograms and using the number of frames (F ) in which the color histogram difference is equal to or greater than a predetermined value in the equation f (s) = segment s. s) is the value of the visual change. [0055] For panning of a video camera, f (s) is large, for zooming, f (s) is small, and when the still camera is pointed at a stationary object, f (s) is zero. If the still camera is pointed at an object moving on a uniform background, f (s) may be smaller, so in order to adjust f (s) properly, edge detection of the image may be performed. Good. Since the edge detecting means is known in the art, detailed description thereof will be omitted. [0056] The characteristic vector may take into account the normalized visual change and the length of the video segment by using the following equation when defining the characteristic vector. [0057] [Number 1]<img file="JP4200741B2_D0001.tif" />[0058] [0058] The characteristic vector can be modified to include another element by adding another characteristic. For example, audio characteristics can be added to the characteristic vector by sampling the audio in the video segment (eg, every 10 milliseconds) and determining the energy difference. If you have text extracted from transcripts (eg, subtitles) or Optical Character Recognition (OCR) on presentation slides, it is useful to take advantage of text characteristics. To add text characteristics, you can sample the text stream and determine changes in the text rate. [0059] Each of the selected video segments is assigned a value indicating importance (logic box 805), and when a characteristic vector considering the value indicating importance and the activity of the content is associated with the video segment (logic box 806). ), Control proceeds to Logic Box 807. [0060] Logic Box 807 allows you to make changes to selected video segments to reduce the size of the electronic files created or to remove uninteresting parts of the video segment. To illustrate the changes in the video, we will focus here on a characteristic vector that has been normalized to fit within a sphere with a radius bounded by the longest characteristic vector. [0061] Each characteristic vector has the largest ratio of the length of each characteristic vector to the boundary radius and has the largest size in the video collage template.<sub>max</sub>It is shortened so that it is the ratio of the size A (s) of each video frame to. Therefore, by solving the following equation, it is determined how compact each video segment should be for the other selected video segments for the video segment, the associated characteristic vector, and the corresponding individual video frame. can do. [0062] [Number 2]<img file="JP4200741B2_D0002.tif" />[0063] Each video segment s that meets this constraint is left alone, and video segments that do not meet this constraint are adjusted until the constraint is met. In one embodiment, the video segment is adjusted by cutting a portion of the video from the end to the beginning of the segment to meet the constraints. | a (s) | is an increasing function of t that can be parameterized as | a (s (t)) | over time. t varies from 0 to T, where T is the length of the segment. From the above, there is a minimum point t * that satisfies the constraint. One way to determine t * is to recursively find 1/2 of the term interval [0, T]. [0064] Segment s (A (s) = A) assigned to the largest individual frame<sub>max</sub>) Always satisfies the constraint expressed by the following equation, so no adjustment (cutting a part) is necessary. [0065] [Number 3]<img file="JP4200741B2_D0003.tif" />[0066] Therefore, the resulting video collage will have the complete segment that the user has indicated to be the most important (eg, by fitting the video segment into the largest frame). In order to reduce the size of the video segment, video compactification means other than the method of cutting a part of the video can be used. Another example of compactification means is to time warp the segment (for example, set the playback parameter to increase the playback speed by a coefficient p that takes a value greater than 1). [0067] When the segment is adjusted as described above, control goes to Logic Box 809 and a video collage is created in Logic Box 809. [0068] [0068] In another embodiment, the idea of information retrieval may be included in the definition of the characteristic vector a (s). The idea of information retrieval can be the frequency of use of terms and the frequency of use of reverse documents (tf * idf). An example of this information retrieval idea for video applications is Uchihashi, It is described in "Summarizing video using a shot importance measure and a frame-packing algorithm" (Non-Patent Document 8) by S et al. This method relies on hierarchical clustering of features such as color. Another example is also described in Chiu, P et al., "A genetic algorithm for video segmentation and summarization" (Non-Patent Document 3). In these treatises, the values contained in the characteristic vector are called "importance". [0069] In such a modification, first the set C should have elements similar to the segment s in U (k).<sub>s</sub>Is defined as the following equation. [0070] [Number 4]<img file="JP4200741B2_D0004.tif" />[0071] Although this equation uses only visual features, it may include other features and media (such as audio). From the above, the following equation can be defined. [0072] [Number 5]<img file="JP4200741B2_D0005.tif" />[0073] idf (s) is the coefficient C<sub>idf</sub>It can be normalized between 0 and 1 by dividing by = log ((1 + k) / 2). Next, by multiplying this normalized equation by the term of the length of each segment, the variation of the characteristic vector is obtained as in the following equation. [0074] [Number 6]<img file="JP4200741B2_D0006.tif" />[0075] In another embodiment, the constraints can be changed by determining the radius of each corresponding video frame in order to reduce the amount of cut of a portion of the split video. This can be done by obtaining the square root of the region or by obtaining the square of the vector length of each video segment. In such an embodiment, the following equation can be used in determining the adjustment of the segment. [0076] [Number 7]<img file="JP4200741B2_D0007.tif" />[0077] FIG. 9 shows an example of the amount of compactification by cutting a portion of a video using the video adjusting means described above according to an embodiment of the present invention. In detail, the table in Figure 9 relates to an example video of an ice cream party. In this example, the system divides the ice cream party video into 24 segments and displays the divided segments through the video segment template included in the video collage user interface (see Figure 3). When the user completes the selection, the resulting video collage is shown as Video Collage 400 (see Figure 4). The segments 921, 923, 925, 927, 929 correspond to the video segments associated with the representative images 421, 423, 425, 427, 429, respectively. [0078] Table 900 in Figure 9 contains eight columns 901, 902, 903, 904, 905, 906, 907, 908. Column 901 identifies the selected video segments 921, 923, 925, 927, 929. Column 902 identifies the length (seconds) of each selected video segment 921, 923, 925, 927, 929. In column 903, the respective regions of the corresponding individual video frames 411, 413, 415, 417, 419 (see FIG. 4) are identified. Column 904 identifies the length of the characteristic vector calculated for each selected video segment 921, 923, 925, 927, 929. Column 905 identifies the rate of compactification when each selected video segment 921, 923, 925, 927, 929 is compactified using equation (2). Column 906 identifies the length (seconds) after compactification of each selected video segment 921, 923, 925, 927, 929 when compactified using equation (2). Column 907 identifies the rate of compactification when each selected video segment 921, 923, 925, 927, 929 is compactified using Eq. (3). Column 908 identifies the length (seconds) after compactification of each selected video segment 921, 923, 925, 927, 929 when compactified using equation (3). [0079] As can be seen from Table 900, the maximum value of the vector a (a)<sub>max</sub>) Is 1.18, which is related to the video segment 921 and is the maximum value of region A (A).<sub>max</sub>) Is 6.88, which is related to video frame 415, which contains the representative image 425 of video segment 925. In this example, the fully uncompacted length of the five selected video segments is 83.00 seconds, but when compactified according to constraint equation (2), the compactified length is 42.58 seconds. In seconds, the compactification rate is 49%. Looking at column 905, the video segment 925, which corresponds to the largest video frame, is not compact because it is considered to be the most important. However, the video segment 921 associated with the largest characteristic vector is assigned to the relatively small frame 411, so the compactification rate is 74%, and the segment length is constrained (2). ) Is used to reduce the time from 28.00 seconds to 7.28 seconds. [0080] [0080] Moreover, when the same five video segments are made compact by using the constraint equation (3), the length after being made compact is 63.05 seconds. Looking at column 907, the video segment 925, which corresponds to the largest video frame, is not compact because it is considered to be the most important. However, the video segment 921 associated with the largest characteristic vector is assigned to the relatively small frame 411 (see Figure 4), resulting in a compactification rate of 45% and a segment length of 45%. , Using the constraint equation (3), it is shortened from 28.00 seconds to 15.30 seconds. [0081] The above-mentioned specific embodiments merely illustrate the principles of the present invention, and those skilled in the art can make various changes to the present invention without departing from the scope and gist of the present invention. It is possible. Therefore, the scope of the present invention is limited only by the claims. [Simple explanation of drawings] FIG. 1 shows an example of four video collage templates for fitting and arranging video segments according to an embodiment of the present invention. FIG. 2 shows a video segment template according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of a user interface for creating a video collage according to an embodiment of the present invention. FIG. 4 shows an example of a video collage containing five selected representative images of video segments according to an embodiment of the present invention. FIG. 5 is a block diagram that comprehensively shows a computing device architecture suitable for implementing various embodiments of the present invention. FIG. 6 is a flowchart showing a step of creating a layout of a video collage according to an embodiment of the present invention. FIG. 7 is a flowchart showing a step of generating a video segment according to an embodiment of the present invention. FIG. 8 is a flow chart showing comprehensive steps to extract and modify selected video segments. FIG. 9 is a table showing an example of making a video compact according to an embodiment of the present invention. [Explanation of symbols] 100, 110, 120, 130 video collage templates 200 video segment template 300 Video collage user interface 400 video collage 500 Computational Device Architecture
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP09293144A | Cites | Japan |
| JP2000308008A | Cites | Japan |
| JP2000299829A | Cites | Japan |
4 members in 2 offices
Priority claims4
| Document | Office | Kind | Date |
|---|---|---|---|
| 992617 | United States of America | – | |
| 99261701 | United States of America | A | |
| 2001992617 | – | – | – |
| US20010992617 | – | – | – |
Members4
| Document | Office | Kind | |
|---|---|---|---|
| US2003095720A1 | United States of America | A1 | |
| JP2003179849A | Japan | A | |
| US7203380B2 | United States of America | B2 | |
| JP4200741B2This record | Japan | B2 |
19 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Cancellation because of no payment of annual feesLAPS | LAPS | |
| Written notification of registration of transferJAPANESE INTERMEDIATE CODE: R350R350 | R350 | |
| Written request for registration of change of nameJAPANESE INTERMEDIATE CODE: R313533S533 | S533 | |
| Renewal fee payment (event date is renewal date of database)FPAY | FPAY | |
| Renewal fee payment (event date is renewal date of database)FPAY | FPAY | |
| Renewal fee payment (event date is renewal date of database)FPAY | FPAY | |
| Renewal fee payment (event date is renewal date of database)FPAY | FPAY | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 4200741
- Publication, DOCDB
- 4200741
- Publication, EPODOC
- JP4200741B
- Application
- 329093
- Application, DOCDB
- 2002329093
- Application, EPODOC
- JP20020329093
Titles2
- Japanese
- ビデオコラージュの作成方法および装置、ビデオコラージュ表示装置、ビデオコラージュ作成プログラム
- English
- Video collage creation method and equipment, video collage display device, video collage creation program
Classification
- CPC, 1
- H04N5/262
- IPC, 2
- H04N5 91
- H04N5 262