Method for creating compound electronic expressive article, computer program and data processing system

The present invention relates to the art of accessing recorded information, and more particularly to the art of creating electronic representations that include inserted information about the recorded information.

Recording information during presentations has gained a lot of popularity in recent years. Schools and universities are starting to program lessons and lectures, companies are starting to record meetings and conferences, and so on. One or more capture devices may record information during the presentation. The recorded information may include various types and streams of information, including audio information, video information, and the like.

The present application is related to the following applications, and all the contents thereof are incorporated in the present application.

US application number 09 / 728,560, filed November 30, 2000, entitled "TECHNIQUES FOR CAPTURING INFORMATION DURING MULTIMEDIA PRESENTATIONS"; filed November 30, 2000, "TECHNIQUES FOR RECEIVING INFORMATION DURING" US Application No. 09 / 728,453 entitled "MULTIMEDIA PRESENTATIONS & COMMUNICATING THE INFORMATION"; filed on March 8, 2000, entitled "METHOD & SYSTEM FOR INFORMATION MANAGEMENT TO FACILITATE THE EXCHANGE OF IDEAS DURING A COLLABORATIVE EFFORT" US Application No. 09 / 521,252; US application number 10 / 001,895, filed November 19, 2001, entitled "PAPER-BASED INTERFACE FOR MULTIMEDIA INFORMATION"; filed September 12, 2003, "THE CHNIQUES FOR STORING MULTIMEDIA INFORMATION" US Application No. 10 / 660,985 entitled "WITH SOURCE DOCUMENTS"; US Application No. 10 / 661,052 entitled "THCHNIQUES FOR PERFORMING OPERATIONS ON A SOURCE SYMBOLIC DOCUMENT" filed on September 12, 2003; Filing filed on September 12, 2003, entitled "THCHNIQUES FOR ACCESSING INFORMATION CAPTURED DURING A PRESENTATION USING A PAPER DOCUMENT FOR THE PRESENTATION"; The US application number 10 / 696,735; and "AUTOMATED TECHNIQUES FOR COMPARING CONTENTS OF IMAGES" was filed on April 11, 2003, entitled "TECHNIQUES FOR USING A CAPTURED ELECTRONIC REPRESENTATION FOR THE RETRIEVAL OF RECORDED INFORMATION". US Application No. 10 / 412,757, entitled.

<p> After the presentation, the recorded information will be available to the user. Users may review their notes or want to see a record of their presentation. The traditional way to access these records has been to look at the records in sequence. More efficient techniques for accessing or retrieving recorded information or indexing recorded information are desired.</p>

<p> In general, the examples of the present invention relate to techniques for creating complex electronic representations. The technique involves receiving an electronic representation of a document. Features in the electronic representation are extracted and compared with the recorded information to determine matching information. For example, the matching information may be an expression in the presentation and / or recorded information. The information to be inserted is determined based on the matching information and the received electronic representation. Then, using the determined information, a complex electronic representation is created.</p><p> In one embodiment, a method of creating a complex electronic representation is provided. The method is: the step of receiving the electronic representation of the document; the step of extracting the feature from the electronic representation of the document; comparing the feature with the recorded information and determining the information in the recorded information that matches the feature. Step; Determine the information to be inserted based on the information in the recorded information that matches the characteristics and the electronic representation received in the document; and form a complex electronic representation consisting of the determined information. Have steps;</p><p> Other embodiments provide a method of creating a complex electronic representation of a document using information recorded during a presentation. The method is: a step of receiving an electronic representation of a document for the presentation, wherein the electronic representation contains features presented during the presentation; extracting the features from the electronic representation. Step; Compare the information recorded during the presentation with the feature and determine the information in the recorded information that matches one or more features; receive the information and documents in the recorded information that match the feature. It has a step of determining the information to be inserted based on the electronic representation obtained; and a step of forming a complex electronic representation consisting of the determined information.</p><p> Examples having other features and the advantages of the present invention will be further clarified by referring to the following specification, drawings and claims.</p>

<p> According to the present invention, it is possible to access or retrieve the recorded information, or to index the recorded information more efficiently.</p>

In the following description, for purposes of explanation, specific details will be given to give a full understanding of the present invention. However, it is clear that the present invention may be realized without such specific details.

FIG. 1 is a simplified block diagram of a system 100 in which an embodiment of the present invention may be incorporated. The system 100 shown in FIG. 1 merely exemplifies an embodiment incorporating the present invention, and does not limit the scope of the invention in the claims. Those skilled in the art will recognize other modifications, modifications and alternatives.

System 100 includes computer system 102, which may be used by the user to prepare the content presented in the presentation. Specific examples of presentations include meetings, conferences, classes, speeches, demonstrations, and the like. The content (material) of the presentation may include slides, photographs, voice messages, video clips, textual information, web pages, and the like. The user outputs the content of the presentation using one or more applications 104 executed by the computer 102. A specific example of an application commonly used to prepare slides presented in a presentation is PowerPoint® from Microsoft®. For example, as shown in FIG. 1, the user may create a "presentation.ppt" file (* .ppt file) using PowerPoint® application 104. * .Ppt files created using the PowerPoint® application can contain one or more pages, each page consisting of one or more slides. The * .ppt file may contain information about the order in which slides are presented in the presentation and how the slides are presented.

In addition to PowerPoint® presentation files consisting of slides, other types of files containing other presentation content may be created using another application running on computer 102. These files may be commonly referred to as "symbolic presentation files." A symbolic presentation file is any file created using an application or program and has at least some content that is presented or output during the presentation. Symbolic presentation files may consist of various types of content such as slides, photographs, voice messages, video clips, text, web pages, images and the like. The * .ppt file created using the PowerPoint® application is an example of a symbolic presentation file consisting of slides. To create a document, the user may print a portion of the presentation content on paper (also referred to as the document), which is typically distributed in the presentation. The term "paper medium" is intended to refer to any printable and touchable medium. The term "printing" or "printing" is intended to include writing, imprinting, drawing, embossing, and the like. Each of the documents may contain one or more pages of paper. A large number of documents may be printed, depending on the number of people attending the presentation.

An electronic representation of the document is received. As shown in FIG. 1, the scanner 108 may be used to scan the document 110. Documents may be scanned using a variety of other devices capable of scanning information in paper media. Specific examples of such devices include facsimile machines, copiers, scanners and the like.

Documents may have various types of characteristics. In general, documentary features relate to the information presented or discussed during the presentation in which the document is created. The feature may include part or other content of the presentation content. Specific examples of printable features include slides, photographs, web pages, textual information (eg, a list of features on the agenda discussed at the meeting), and the like. For example, the user may print one or more slides from a * .ppt file on a document. The PowerPoint® application provides a means (tool) to print one or more slides from a * .ppt file to create a document. Each page of the document may contain one or more slides printed on it. Specific examples of document pages using slides on document pages are shown in FIGS. 3 (A) and 3 (B) and are described in more detail below.

The electronic representation may be an electronic image of a document. For example, a * .ppt file may be converted to an image and used as an electronic representation of a document. Documents are used, but electronic representations of any document may be received. Documents printed on paper do not need to be used to produce electronic representations.

The capture device 118 is constructed to capture the information presented in the presentation. Various different types of information output during the presentation may be captured or recorded by the capture device 118, such as audio information, video information, slide and photographic images, whiteboard information, text information, and the like. With respect to the present application, the word "presented" is intended to include being displayed, output, spoken, and the like. With respect to the present application, the term "capture device" is intended to refer to any device, system, device or application constructed to capture or record one or more types of information. Specific examples of capture device 118 include microphones, video cameras, cameras (both digital and analog can be applicable), scanners, presentation recorders, screen capture devices (eg, whiteboard information capture devices), and symbolic (symbolic). ) Includes information acquisition devices, etc. In addition to capturing information, the capture device 118 may be able to capture temporal information related to the captured information.

A presentation recorder is a device that can capture the information presented during a presentation (eg, by acquiring and capturing an information stream from an information source). For example, if a computer running a PowerPoint® application is used to display slides from a * .ppt file, the presentation recorder will get the computer's video output and the video keys on the displayed slides. It may be configured to capture keyframes each time a significant change is detected in the frame. The presentation recorder can also capture heterogeneous information such as audio information, video information, slide information streams, and the like. Time information about the capture information, which indicates the time the information was output or captured, is used to synchronize various types of capture information. Specific examples of presentation recorders include screen capture software applications, PowerPoint® applications (which allow recording of slides and elapsed time for each slide during a presentation) and the presentation recorders described in the following US application (2000). US Application No. 09 / 738,560 filed November 30, 2000, US Application No. 09 / 728,453 filed November 30, 2000, and US Application No. 09 / 521,252 filed March 8, 2000. No.), and the content of those US applications is incorporated into this application.

The symbolic information capture device can capture the information recorded in the symbolic presentation document that may be output during the presentation. For example, a symbolic information capture device can record slides presented in a presentation as a series of images (eg, JPEGs, BMPs, etc.). The symbolic information capture device may be configured to extract the text content of the slide. For example, during a PowerPoint slide presentation, a symbolic information capture device captures slide transitions (eg, by capturing keyboard commands) and extracts the presentation information based on those transitions. You may record the slides by doing so. The whiteboard device may include a camera-like device appropriately provided to capture the contents of the whiteboard, screen, chart, and the like.

The information captured or recorded by the capture device 118 during the presentation may be stored in the repository or database 150 as record information 120. The recorded information 120 may be stored in various formats. For example, a hierarchy (directory) for storing the recorded information 120 may be formed in the repository 115, and various types of information (for example, audio information, image information, images, etc.) included in the recorded information 120 may be formed. It may be stored according to the hierarchy. In other embodiments, the recorded information 120 may be stored as a file. Various other techniques well known to those of skill in the art may be used to store the recorded information.

Images of slides found in the document will be displayed during the presentation. In one embodiment, the presentation recorder may capture slide images so that they are displayed. In addition, relevant information that may be used to index the record information 120 may be stored. For example, time information indicating the time when the slide was displayed may be stored. The time information may be used to determine a portion of the recorded information associated with the time the slide was displayed.

In addition to the time information, source information that distinguishes where the recorded information was stored during the presentation may be identified. This storage location information for the record information 120 may be updated when the record information is moved to a new storage location. Thus, in one embodiment of the present invention, the storage location of the recorded information 120 can be changed with time.

According to the embodiments of the present invention, the relevant information is stored in an XML structure. For example, the relevant information may include time and source information determined in step 206 for the presentation and / or page. The source information may be an identifier used to access the presentation. For example, the source information may be a location name and a file name. Time information is used to index parts of the presentation. The presentation may be the presentation determined in step 204 or the presentation related to the information determined in step 204 (eg, a presentation where a slide image was captured using a presentation recorder). Part of the presentation may include information that matches the extracted features determined in step 204. For example, part of the presentation may include slides that were displayed during the presentation.

Server 112 creates a complex electronic representation 122. In one embodiment, features are extracted from the electronic representation of the received document. The feature is compared with the recorded information 120 to determine matching information. In one embodiment, matching information may be determined using a variety of techniques. It will be appreciated that the slide image does not have to exactly match the slide image. For example, the text on the slide may be compared to the text to determine a substantially matching text.

The information to be inserted is determined based on the matching information and the electronic representation of the document. The complex electronic representation 122 is created based on the inserted information. The complex electronic representation 122 may include extracted features. The composite electronic representation 122 may also include matching information and information determined based on the electronic representation of the document.

The user may select the insert information in the complex electronic representation 122 to access and / or reproduce the recorded information 120. For example, an electronic representation of a slide in a document may be used to determine information related to record information 120. The information inserted may include objects that represent a portion of the presentation. When an object is selected, the recorded information at the time the slide was displayed in the presentation is accessed and / or played. Therefore, if the user desires additional information related to the slide in the document, the insert information in the composite electronic representation 122 may be used to display and / or discuss the slide. You may search for recorded information 120 (of the presentation).

FIG. 2 shows a schematic flowchart 200 of a method relating to a complex electronic representation according to an embodiment of the present invention. The method shown in FIG. 2 may be performed by software, hardware modules or a combination thereof running on the processor. The flowchart 200 shown in FIG. 2 merely shows one aspect of the present invention and is not intended to limit the scope of the present invention. Other modifications, modifications and alternatives are also within the scope of the invention.

In step 202, the electronic representation of the document is received. In one embodiment, the document is printed as a paper document. The user may take notes on the document. The memo is generally written on a paper document during the presentation. Other methods of taking notes, such as typing on electronic documents, may be envisioned. The memo may be documented, but the presence of any memo is not required.

Although it is envisioned that an electronic representation of the document will be received, it will be understood that an electronic copy of the document may be received. One or more pages in a document may be received as an electronic representation. For convenience of explanation, it is assumed that the electronic representation contains one page, but it should be understood that the electronic representation may contain any number of pages. As mentioned above, the electronic representation of the document may be received from the document being scanned.

In other embodiments, the electronic form (version) of the document may be used. For example, the electronic version may be an image of a slide found in a * .ppt file. * .ppt files may be converted to images. For example, there are known techniques for converting * .ppt slides into a.pdf or images in flash files. These images may be used as images of the document. For example, an electronic representation of a document may be received using the following process. The user may open the electronic representation in an application such as a pdf reader. For example, an electronic representation from a scanned document or an electronic version thereof may be opened in the application. Input to start the following steps may be provided in the pdf reader.

In step 204, features are extracted from the electronic representation of the document. The document may include images of one or more slides presented in the presentation. The slide image is extracted from the electronic representation of the document. The process is described as extracting slides, but it will be understood that other features may be extracted. For example, images, characters, and the like may be extracted. For example, instead of using slides in a presentation, the user may display an image of a feature. The image is then extracted.

In one embodiment, segmentation may be performed when there are more than one slide on one page. Segmentation makes the divisions and determines the individual slides in the electronic representation of the document. This may be desirable if the individual slides should relate to different parts of the record information 120. In certain cases, such as when the only slide is on the page, segmentation may not be needed.

Techniques for dividing documents and images are well known in the art. Many current division methods are used to separate individual slide areas from the electronic representation of a document. Specific examples of the classification will be described in more detail below. Although segmentation is described, other techniques may be used to discriminate slide images in electronic representations of documents. For example, the content of the electronic representation may be analyzed to determine the slide image. In one embodiment, a quadrangular frame (box) may be certified and the information within that frame may be used as a slide image.

In step 206, the extracted features are compared with the recorded information 120, and matching information is determined among the recorded information 120. For example, a portion of recorded information that matches the extracted information may be determined. The portion may include a presentation page, video, audio, and the like. In one embodiment, the slide image extracted from the electronic representation of the document is compared to the slide image in the recorded information 120 for another presentation. The recorded information 120 for the presentation may be displayed at different times and may include all matching information (eg, one slide may occur in different recorded information during the presentation).

In one embodiment, the extracted features may include multiple slides. The slides may be compared to the slide images to determine the presentation that contains the matching information. In one embodiment, each of the slides is compared to the slides in the recorded information 120 about the presentation. The portion of the recorded information 120 including the information matching the slide is determined.

In another embodiment, multiple slides are processed as a set and all slides are used to compare slides during the entire presentation. Therefore, in order to identify a presentation that contains matching information, the presentation should include slides that match each of the slides that are processed as a set.

Various techniques may be used to determine the matching information in the recorded information 120. In one example, a matching image (ie, an image from recorded information with extracted features) is found using the techniques described in the following US application and other techniques known in the art. You may. The US application is US Patent Application No. 10 / 412,757 entitled "AUTOMATED TECHNIQUES FOR COMPARING CONTENTS OF IMAGES" filed April 11, 2003; "TECHNIQUES FOR STORING MULTIMEDIA INFORMATION" filed September 12, 2003. U.S. Patent Application No. 10 / 660,985 entitled "WITH SOURCE DOCUMENTS"; U.S. Patent Application No. 10 / 661,052 entitled "TECHNIQUES FOR PERFORMING OPERATIONS ON A SOURCE SYMBOLIC DOCUMENT" filed September 12, 2003; 2003 TECHNIQUES FOR filed on September 12 ACCESSING INFORMATION CAPTURED DURING A PRESENTATION USING A PAPER DOCUMENT FOR THE PRESENTATION "US Patent Application No. 10 / 660,867;" TECHNIQUES FOR USING A CAPTURED ELECTRONIC REPRESENTATION FOR THE RETRIEVAL OF RECORDED INFORMATION "filed on September 12, 2003. Is US Patent Application No. 10 / 696,735.

In one embodiment, the extracted features may be used to determine recorded information 120 for a presentation that includes information that matches the extracted information. For example, this is done by a first extracted image from record information 120. The images extracted from the recorded information 120 include images captured during the presentation by various electronic expression capture devices, images captured by the presentation recorder, keyframe images acquired from the video information captured during the presentation, and the like. It may be included.

The extracted image is compared with the extracted features determined in step 204. The extracted image may be pre-processed to determine time information indicating a time point during the presentation in which the slide was displayed or presented. Time information about a slide also identifies one or more periods during the presentation in which the slide was presented or displayed. The period may be discontinuous.

In one embodiment, matching information may be determined using the techniques described in the "matching method" section described below. Matching information may be determined using presentation level matching. The document matching algorithm takes the electronic representation of the document Ii as input and compares it to the database of presentation recorder documents. This may be referred to as the presentation matching step. It identifies all presentation recording sessions that could have been used to give that presentation. The next step, called slide matching, is the split electronic representation within the identified presentation recorder session._{j, k}Each is mapped to a slide image on the document. The technique is described in more detail below. This provides a mapping function from each slide in the document to the source and the time stamp in the audio and video tracks.

In step 207, the information to be inserted is determined based on the matching information determined in step 206 and the electronic representation of the document received in step 202. In one embodiment, the determined information may be based on slide images that match the extracted features. The relevant information determined for the matching slide image may be used to index the presentation. The image extracted from the presentation record at the time indicated by the relevant information may be subsequently determined.

In step 208, the complex electronic representation 122 is created using the information determined in step 207. The complex electronic representation may contain many types of information. For example, the extracted features may be contained within the complex electronic representation 122.

Further, the information may include metadata, and the metadata is determined based on the recorded information 120. The metadata may be derived from the matching information determined in step 206. For example, the metadata may be determined and inserted into the created electronic representation. Recorded information 120 may be post-processed (created using the document as a template) to extract the metadata contained in the complex electronic representation 122. For example, slides are calculated and inserted for how long they have been discussed. There are no restrictions on what kind of metadata is extracted, or in the form in which they can be contained within the complex electronic representation 122, and some examples of metadata extraction are exemplary. What is given for the purpose should be understood. Techniques for determining metadata are described in more detail below.

Also, selectable objects such as icons and links may be inserted. When selected, the object accesses the record information 120 using the relevant information associated with the matching information determined in step 206. For example, the information accessed may be a presentation record at the time the slide was displayed. When the image extracted from the recording information 120 is selected, the recorded presentation is accessed accordingly at the time specified in the relevant information about the image. In another embodiment, the recorded information 120 is embedded or stored with the image. When an object is selected, the embedded information is accessed and played. Therefore, a central database for access is not essential. For example, the image reproduction object may be embedded in the image. When playback is selected, the recorded information 120 is automatically played.

In one embodiment, the electronic representation received in step 202 is used to create the complex electronic representation 122. For example, information is inserted into the received electronic representation. In addition, a new document containing the received electronic representation and insertion information may be created. In both cases, the complex electronic representation 122 is created with information related to the recording information 120 inserted therein. For example, the document may be printed and distributed during the presentation. The user may take notes on the document. The document is then scanned to create an electronic representation of the document. Then, the information is inserted into the scanned electronic representation.

In another embodiment, a document different from the received electronic representation is created. Different documents may include the features and insertion information extracted in step 204. For example, the very extracted image of the slide in the document and the determined information may be included in the complex electronic representation 122.

The complex electronic representation 122 created in step 210 may then be emailed, reprinted, stored on a server for later access, copied to a storage medium such as a CD, and displayed. You may. When the user needs to revisit that particular presentation, the user can review the notes taken within the composite electronic representation 122 (composite electronic representation 122 is a paper document memo). Is assumed to be included.). If more information is needed, the information to be inserted into the complex electronic representation 122 may be selected and the recorded information 120 corresponding to the relevant information of the inserted information may be accessed and displayed. For example, the playback interface of the presentation may be activated and playback may start from the time stored in the relevant information. In other embodiments, the insert information may be used to add information to the complex electronic representation 122. For example, the metadata may indicate how long the slides have been discussed.

FIG. 3 (A) shows schematic page 300 in writing according to an embodiment of the present invention. Page 300 shown in FIG. 3 merely exemplifies an embodiment incorporating the present invention, and does not limit the scope of the invention described in the claims. Those skilled in the art will recognize other modifications, modifications and alternatives.

Information 302 indicating the presentation and presenter is printed on page 300, as shown in FIG. 3 (A). Other information such as the time the presentation will take place, the length of the presentation, etc. may also be included in the information 302. In the example shown in FIG. 3 (A), three slides 304-1, 304-2 and 304-3 are printed on page 300. In addition, a margin (space) 308 is provided so that the user can take notes about each slide during the presentation.

FIG. 3B shows page 300 of FIG. 3A with user markings according to an embodiment of the present invention. As shown, the user is taking notes on the document in spaces 308-1 and 308-2.

FIG. 3C shows a composite electronic representation 122 according to an embodiment of the present invention. The complex electronic representation 122 may be viewed using an interface such as a pdf reader, web browser, word processing interface, and the like. The composite electronic representation 122 includes information inserted in connection with the recorded information 120 according to an embodiment of the present invention. As shown in interface 2310, the complex electronic representation 122 includes at least a portion of page 300. The composite image of FIG. 3 (C) may be created using page 300 as a basis. Other information may be superimposed on the composite image.

The complex electronic representation 122 also includes information 314. As illustrated, an image of recorded information 120 is included in information 314. In one embodiment, this image corresponds to a portion of the recorded information 120 relating to the representation. This portion may be the time when the slide matching the slide 304 was output. For example, information 314-1 includes information extracted from the recorded information 120 from which the image of slide 304-1 was output.

In one embodiment, information 314 may, when selected, cause an action to be performed. Each image in information 314 may be associated with relevant information (used to access recorded information 120) such as time and source information. Although not shown, information 314 may include non-image information such as hypertext links, icons, metadata, and the like.

The composite electronic representation 122 may include the received electronic representation of the paper document 300. In this case, information 314 is inserted into the scanned electronic representation. Therefore, the user who took the memo on the paper document can see the paper document together with the information 314 inserted in addition to the memo. In one embodiment, the electronic representation of the user's paper document is a template containing insert information 314 for the media document. Thus, the user may view the media document, or if more information is desired, the insert information may be selected and the associated record information 120 may be accessed and shown.

The composite electronic representation 122 may be a document different from the received electronic representation. The different document may include any or all features of the received electronic representation. For example, a user may want a format other than the electronic representation of a paper document. The slide image or memo in the electronic representation of the paper document may be removed or moved to another location on the page. For example, a complex electronic representation 122 with exactly the user's notes and the inserted information 314 may be generated.

The complex electronic representation 122 may be stored in various formats. For example, the complex electronic representation 122 may be a document in a format such as PDF (PDF), Hypertext Transfer Language (HTML), Flash, MS Word. In one embodiment, the format supports the insertion of information that may be used to link to record information 120.

As mentioned above, the extracted features are used to determine matching information. In another embodiment, other types of information may be used to determine matching information. For example, a part of the recorded information 120 may be identified by using the barcode on the document. For example, in one embodiment, the document may be printed with the barcode associated with each slide. Barcodes may be used to form links between slides and recorded information. Such techniques are described, for example, in the following literature: US application number 10 filed September 12, 2003, entitled "TECHNIQUES FOR ACCESSING INFORMATION CAPTURED DURING A PRESENTATION USING A PAPAER DOCUMENT FOR THE PRESENTATION". / 660,867.

In another embodiment, barcodes or some other marking may be used to represent the signature information on each slide. This signature information may include text from slides, image characteristic vectors, and the like. The signature is generated when the document is created (eg, printing). The signature may include information about the location of the document, provided that information related to record information 120 has been inserted. After the document has been captured (eg, after scanning), these printed markings are identified (extracted and decoded) and used for matching, access and insertion of information related to recorded information.

FIG. 4 shows possible output after information 314 is selected according to an embodiment of the invention. Information 314 includes images, but it should be understood that information other than images may be selected, such as icons, links, text, pictures, video, audio, and the like.

As shown, interface 502 may be displayed when the image in information 314 is selected. The interface 502 includes a window 504 that displays the recording information 120 and a window 506 that contains the image of slide 304. For convenience of explanation, it is assumed that the user has selected the image in information 314-3. After selection, relevant information about the image is used to access recorded information 120. For example, the relevant information may be source and time information. Source information may be used to access the presentation, and time information is used to determine the portion of the presentation at that time. For example, the time information may be the start time on which slide 304-3 is displayed.

Window 504 includes a media player and may be used to display a portion of the accessed recording information 120. As shown, the recording information 120 is displayed in the media player window 508. In this case, the start image corresponds to the image displayed in information 314-3. The user then selects playback on the media player and that portion of the presentation is played. Alternatively, the accessed recording information 120 may be automatically reproduced.

The image on slide 304-3 may be displayed in window 506. Also, the original slide in the * .ppt file may be displayed in window 506. That is, the user may view slides 304-3 to be viewed in addition to the presentation in window 504. Therefore, the user may not need to see a paper copy of the document. Window 506 may also contain other information, such as notes taken by the user, as shown in FIG. 3 (B). Further, the determined metadata may be displayed in window 506.

FIG. 5 is a schematic block diagram 600 of a module that may be used to implement embodiments of the present invention. Modules may be implemented in software, hardware or a combination thereof. The module shown in FIG. 5 is merely an example of an embodiment incorporating the present invention, and does not limit the scope of the invention described in the claims. Those skilled in the art will recognize other modifications, modifications and alternatives.

The electronic representation receiver 602 receives the electronic representation of a paper document. In one embodiment, the electronic representation may be received from a scanner that has scanned the document to generate an image. Also, an electronic copy of the document may be received.

The feature extraction unit 604 is configured to receive an electronic representation and extract features from the image. For example, a slide image is extracted from the electronic representation of a paper document. As mentioned above, various techniques may be used to discriminate between individual slide images. The comparator 606 is configured to receive the extracted features and determine information (matching information) that matches the extracted features. In one embodiment, database 608, which stores recorded information 120 and related information 609, receives an inquiry. The extracted features are compared with the recorded information 120 to determine matching information. Further, the related information about the matching information in the recorded information 120 may be determined. The relevant information may be used to access part of the presentation.

The information insertion unit 610 receives matching information, related information, and an image. In addition, recorded information 120 (for example, audio and video information), metadata, and the like may be received. As described above, the information insertion unit 610 is configured to determine the information to be inserted and create the complex electronic representation 122. For example, the information related to the matching information in the recorded information 120 is inserted into the complex electronic representation 122. Information 314 may be associated with relevant information. In this case, if insert information 314 is selected, the relevant information is used to access a portion of the recorded information 120. In addition, recorded information 120 (for example, audio and video information), metadata, and the like may be inserted into the electronic representation 310. Therefore, the database 608 does not need to be accessed when the recorded information 120 is regenerated.

application: I. Theater program Although the examples of the present invention have been described with reference to presentation records, it will be appreciated that the described examples may be used with recording information other than the presentation records. The paper document may be in any medium containing extractable and recognizable features and in various formats including it. For example, paper documents may be a theatrical program. The theater program is distributed to the audience before the theater begins. The program may include some scenes in the play, images of actors, or any text in the play. The play is then recorded. After the play, the complex electronic representation 122 of the play program is received. For example, the user may scan or capture the theatrical program with a digital camera. By the processing described above, the complex electronic representation 122 contains information related to the recorded play. The process is to compare the theatrical surface in the program with the captured video frame of the recorded play, to compare the image of the actor in the program with the facial recognition from the theatrical record, and the text in the program. Includes determining matching information by comparing (speech recognition) with the captured voice.

The information related to the matching information may be inserted into the composite electronic image. The complex electronic representation 122 of a play program may have objects inserted in connection with a part of the play. Users may store the complex electronic representation 122 in their digital camera. Alternatively, the complex electronic representation 122 may be notified by e-mail to the user or to others for shared purposes. Thus, as an example, if the user is interested in a feature in the program, the inserted information may be selected and part of the recorded play may be accessed and played.

II. Symphony The following is an example of another application, in which a music memo is printed and distributed as a paper document to the performer before the symphony is practiced. During the symphony, the music is recorded and the performer may take notes on the paperwork. After practice, an electronic representation of the document will be received. The association between the recorded voices is determined by performing OCR (optical character reader) processing on the captured music memos, automatically extracting the memos from the voices, and collating them. The user then receives a complex electronic representation 122, such as a PDF document, which may include scanned music memos, personal memos and insert information, which insert information is being practiced. Information is associated with the voice recorded in (or the voice played in another symphony). Also, the complex electronic representation 122 may help others who made mistakes in the initial practice practice.

Techniques for performing segmentation, matching techniques, and determining metadata are described below.

(Segmentation) In the embodiment of the present invention, the electronic representation of the document may be divided using the following process. Horizontal and vertical projection (projection) of the electronic representation is performed first. Prior to this step, it should be understood that some pretreatment of the electronic representation may be required, such as distortion or skew compensation, downsampling, smearing. , Connection of element analysis, etc.

The distance between the extracted projection and the projection obtained from the set of document templates is calculated. Figure 6 shows possible templates that may be included in a set of document templates. Document template 702 includes possible layouts that may be used to create the page. The layout contains different images. For example, documents 702-1 and 702-2 include a layout for slide images. Document template 702-1 contains two rows of slide images 704. Document template 702-2 contains a row of slide images 706. Document template 702-3 includes a left column containing three slide images 708 and a right column containing three areas 710 where the user may have entered notes. The extracted projection is compared with the document template 702. In one embodiment, these templates are used to create a paper document. For example, the illustrated image does not have to be a slide image. Rather, a window showing the slides should be provided at some point where the slide image shown in FIG. 6 may be used.

Then, the document template 702 having the shortest distance to the document image is determined. For example, a page with three slide images in the left column and a blank in the right column might match document template 702-3 substantially. Using the slide placement information of the matched document template 702, the document image is then divided into a plurality of rectangular areas. For example, since the slides are located in the left column with some space, the document may be divided into multiple parts, each of which may contain a slide.

<u style="single">Matching method</u> In one embodiment, the following techniques may be used to determine relevant information using the electronic representation of the document and the presentation record document. The document collation algorithm takes image Ii as input and compares it to a database of recorded presentation recorder documents. This may be referred to as the presentation matching step. It locates all presentation recorder documents captured during the presentation in which the slides in the document are presented. The next step, called slide matching, maps each slide image in the divided document image to the slide image captured by the presentation recorder in one presentation session.

The presentation matching algorithm applies OCR to the presentation recorder document, saves the text, and outputs it with an indication of the page where the text originated. The problem is that there is a potentially large amount of both duplicate and unwanted images in the presentation recorder document, moving their PowerPoint files back and forth while people are speaking, a few in the captured images. This is due to the use of custom animations that only make a difference, the use of video clips that allow the presentation recorder to capture hundreds of frames, and so on. A pseudo-code statement based on an example of the presentation matching algorithm is shown below.

<maths num="1"><img file="JP2005293589A_D0001.tif" /></maths> The first step is to examine all presentation recorder files containing each word n-gram in Document Ii and increase the score for that document. The second step considers the presentation recorder file with the lowest percentage of words in Ii as specified by t1 and determines the percentage of pages with more than t2 for n-grams in Ii. When it exceeds t3, this Pj is called a matching presentation recorder document.

Presentation matching algorithms are resistant to OCR errors in presentation recorder documents. Since it contains a color jpeg image, it is expected that its OCR will make many mistakes. However, in general, presentations use carefully selected fonts and short text phrases, so errors may not be a serious problem. The condition that the document contains a percentage of the pages in the presentation recorder file takes into account duplicate and unwanted images. These factors allow the threshold to be set freely.

A further consideration in building a presentation matching algorithm is to utilize an existing full-text index. This is achieved by using the word n-grams as a query term and suggesting PowerPoint files that may match. This is supported by almost all full-text indexes, such as Google.

The slide collation algorithm determines the images in the document Ii, and the images match each slide in the presentation recorder file identified by the presentation collation algorithm. The algorithm utilizes a combination of string matching against OCR results for the image containing text and edge histogram matching for the image without text. Specific examples of slide matching techniques are described in detail in the following article: US Patent Application No. 10 / 412,757, filed April 11, 2003, entitled "AUTOMATED TECHNIQUES FOR COMPARING CONTENTS OF IMAGES".

<u style="single">Metadata</u> (Text and keywords) Text is extracted from each of the captured screen images. Localization and binarization of each electronic representation is achieved by a method that utilizes not only the luminance component of the captured image but also its color component. The reason is that, unlike a normal document image, the background / foreground contrast in the electronic representation of a slide may be obtained by color contrast in addition to luminance contrast. Details of this technique are described in the following article: US Application No. 10 / 412,757, filed April 11, 2003, entitled "AUTOMATE TECHNIQUES FOR COMPARING CONTENTS OF IMAGES". Commercial OCR packages may be used to extract text from the binarized text area. The extracted text is indexed in XML format with captured images and line numbers. Keywords are discovered by performing a TF-IDF analysis on the text extracted from the entire screen image captured during the presentation session.

(Electronic representation characteristics) A large number of image characteristic vectors, namely the edge histogram and the color layout [ID-RII-311], are calculated for the screen-captured image. These features will be used later to detect duplicate slides and to associate (link) screen images with the original presentation slides.

(Symbolic Presentation Slides) Presenters can submit their original presentation documents to the server prior to their conversation. After the submission, the presentation file will have ID, ID_kIs assigned and text and slide titles are extracted from each slide by analyzing the presentation file. JPEG electronic representation of slides S_iIs extracted and used to calculate the edge histogram and color layout characteristic vector. Presentation files, JPEG images, text, titles and characteristic vectors are indexed and placed in the hierarchy of unresolved presentations.

After the presentation session is over, the image characteristics and text extracted from the screen-captured image are matched against the image characteristics and text extracted from the undecomposed presentation file. If a match is found, the file is removed from the non-decomposition presentation hierarchy and linked to the recorded presentation. After presentation level matching, each presentation slide S utilizing electronic representation and text characteristics_iA group of screen capture images {C_l, ..., C_n}. The electronic representation matching process has 98% accuracy.

(Keyframe extraction) The conference room is equipped with two cameras, a pan-zoom-tilt (PTZ) camera and an omni-directional camera that can capture at 360 °. The PTZ camera focuses on either the presenter or the entire conference room. The position of the PTZ camera is controlled by the portal of the meeting room, and keyframes are extracted from the video sequence each time the camera position changes. The omni-directional camera is installed in the center of the conference room and captures a panoramic image of the room. The camera is equipped with four microphones, and the sound source locarization (SSL) method is executed in real time on four voice channels. Each time the sound source direction changes, a keyframe showing the image only from the sound direction is extracted from the panoramic video. Figure 7 shows these keyframes 802, which are useful in navigating meeting / presentation recordings so that they match speaker changes. All extracted keyframes are indexed by their source and timestamp.

(Time spent on each slide) Amount of time spent on presentation slides sT_iCan be a good indicator of the importance of that particular slide and is calculated by:

<maths num="2"><img file="JP2005293589A_D0002.tif" /></maths>Where pT is the total presentation time and S_iIs a presentation slide, C_nIs S_iThe nth captured screen electronic representation that matches T (C)_n) Is C expressed in seconds_nIt is a time stamp of.

(Questions and Answers) The amount of discussions, questions, and comments about a particular presentation slide can be an indicator of interest in the topic being discussed. On the other hand, speaker splitting and Q &amp; A sessions are challenging but very difficult and often require pre-training of the speaker splitting system. Our method of identifying speech segments along with Q &amp; A behavior (activity) is somewhat different. It is based on SSL and works extremely robustly in practice. During the presentation, our experimental results show that at 93% of the time, the speaker is within the same 20 ° azimuth of the SSL device. Obviously, the range of orientation changes based on the settings of the conference room. Nevertheless, in most conference room settings it is often a reasonable assumption that the presenter has a limited platform to move around. This range of this platform [α_s1α_s2] Will be shown by SSL. There is no listener between the presenter and the SSL device, and sounds coming from directions other than the presenter can be interpreted as comments or questions from spectator members. Given presentation slide S_iThe question-and-answer action about is defined as the number of changes in sound source direction between the audience and the presenter:

<maths num="3"><img file="JP2005293589A_D0003.tif" /></maths>Where C_nIs S_iThe nth captured screen image that matches T (C)_n) Is expressed in seconds C_nTimestamp, pD is the QA behavior for the entire presentation, ie

<maths num="4"><img file="JP2005293589A_D0004.tif" /></maths>And pT is the total presentation time, and D (t) is

<maths num="5"><img file="JP2005293589A_D0005.tif" /></maths>Is a function defined as, where SSL (t) is the orientation of the sound direction at t.

(Notes) The amount of notes taken on a topic in the seminar is almost time-related directly to the audience's interest in that topic. The measured value of the memo writing operation is calculated as follows,

<maths num="6"><img file="JP2005293589A_D0006.tif" /></maths>Where η (t)₁, t₂) Is the time frame [t₁t₂] Related memo input count is returned. The electronic memo writing interface allows you to associate notes not only at the current time point but also at past time points, and the function η returns memo input based on the associated time rather than the input time.

FIG. 8 is a schematic block diagram of the data processing system 900, which may be used to perform processing according to an embodiment of the present invention. As shown in FIG. 8, the data processing system 900 includes at least one processor 902, which communicates with a number of peripherals by the bus system 904. Peripherals include a storage subsystem 906 with a memory subsystem 908 and a file storage subsystem 910, a user interface input device 912, a user interface output device 914, and a network interface subsystem 916. Input and output devices allow the user to interact with the data processing system 902.

The network interface subsystem 916 provides interfaces to other computer systems, networks and stored resources. The network may include the Internet, local area networks (LANs), wide area networks (WANs), wireless networks, intranets, private networks, public networks, switching networks and any other suitable network. The network interface subsystem 916 acts as an interface that receives data from other sources and sends data from the data processing system 900 to other sources. For example, the data document processing system 900 may access the stored record information and XML data structure for the presentation through the network interface subsystem 916. Specific examples of the network interface subsystem 916 include Ethernet cards, modems (telephones, satellites, cables, ISDN, etc.), (asymmetric) digital subscriber line (DSL) devices, and the like.

The user interface input device 912 is a keyboard, a pointing device (such as a mouse, trackball, touchpad or graphics tablet), a scanner, a bar code scanner, a touch screen built into the display, a voice recognition system hand-like voice. Input devices, microphones and other types of input devices may be included. Generally, when the term "input device" is used, it is intended to include all possible types of devices and methods for inputting information into the data processing system 900.

User interface output device 914 may include non-visual devices such as display subsystems, printers, facsimile machines or audio output devices. The display subsystem may be a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD) or a projection device (projector). Generally, when the term "output device" is used, it is intended to include all types of devices and methods capable of outputting information from the data processing system 900.

The storage subsystem 906 is constructed to store the basic programming and data structures that provide the functionality according to the invention. For example, according to an embodiment of the present invention, software modules that perform the functions of the present invention may be stored within the storage subsystem 906. These software modules may be executed by processor 902. The storage subsystem may provide a repository for storing the data used by the present invention. The storage subsystem 906 may consist of a memory subsystem 908 and a file / disk subsystem 910.

The memory subsystem 908 includes a large number of memories including a main random access memory (RAM) 918 for storing instructions and data during program execution and a read-only memory (ROM) 920 for storing immutable instructions. But it may be. The file storage subsystem 910 provides permanent (non-volatile) storage for programs and data, including hard disk drives, floppy disks with removable media, compact disc read-only memory (CD-ROM) drives, It may include an optical drive, a removable media cartridge and other storage media.

The bus subsystem 904 provides a mechanism for the various elements and subsystems of the data processing system 902 to communicate with each other as intended. The bus subsystem is schematically shown as a single bus, but other examples of bus systems may utilize multiple buses.

Data processing systems can be of various formats, including personal computers, portable computers, workstations, network computers, mainframes, kiosks or any other data processing system. Due to the ever-changing nature of computers and networks, the description of the data processing system shown in FIG. 8 is merely intended to illustrate preferred embodiments of computer systems. Many other configurations are possible with more or less elements than the system shown in Figure 8.

Although specific embodiments of the present invention have been described above, various modifications, substitutions, alternative configurations and equivalents are also included in the scope of the present invention. The described invention is not limited to operation within a particular data processing environment and may operate within a plurality of data processing environments. Moreover, although the invention has been described with a particular set of transactions and steps, it will be apparent to those skilled in the art that the scope of the invention is not limited to the set of transactions and steps described. It will be appreciated that the above formula is merely an example according to an embodiment of the present invention and may vary in other embodiments of the present invention.

Further, although the present invention has been described with particular combinations of hardware and software, it should be understood that other combinations of hardware and software are also within the scope of the present invention. The present invention may be realized by hardware alone, software alone, or a combination thereof.

Therefore, the specification and drawings are considered as examples, not in a limited sense. However, it will be clear that additions, substitutions, deletions and other modifications and modifications may be made to those examples without departing from the spirit and scope of the invention described in the claims. ..

<figref num="1">It is a schematic block diagram of the system which may incorporate one Example of this invention.</figref><figref num="2">A schematic flowchart of a method according to an embodiment of the present invention for creating an electronic representation having insertion information related to recorded information using the electronic representation of a document is shown.</figref><figref num="3">(A) is a page of a document according to an embodiment of the present invention, (B) is a page of (A) with a user marking according to an embodiment of the present invention, and (C) is a recorded information according to an embodiment of the present invention. Indicates an interface that contains relevant insert information.</figref><figref num="4">The possible output after the information is selected by one embodiment of the present invention is shown.</figref><figref num="5">FIG. 6 is a schematic block diagram of a module that may be used to realize an embodiment of the present invention.</figref><figref num="6">A document template according to an embodiment of the present invention is shown.</figref><figref num="7">A keyframe according to an embodiment of the present invention is shown.</figref><figref num="8">FIG. 6 is a schematic block diagram of a data processing system that may be used to perform processing according to an embodiment of the present invention.</figref>

Code description

100 System 102 Computer 104 Application 106 File 108 Scanner 110 Paper Document 112 Server 114 XML Structure 115 Repository 116 Output Device 118 Capture Device 120 Recorded Information 122 Complex Electronic Representation 300 Page 302,314 Information 304 Slide 308 Space 502 Interface 504,506 Window 508 Media Player window 602 Electronic representation receiver 604 Feature extraction section 606 Comparison section 608 Database 609 Related information 610 Information insertion section 702 Document template 704,706,708 Slide image 710 Space 802 Keyframe 900 Data processing system 902 Processor 904 Bus system 906 Storage subsystem 908 Memory sub System 910 File storage subsystem 912 User interface Input device 914 User interface Output device 916 Network interface 918 RAM 920 ROM