Data structure for multimedia applications

Data structure for multimedia applications

DATA STRUCTURE FOR MULTIMEDIA APPLICATIONS

This application generally relates to data structures for implementing a plurality of multimedia applications, and to a controller for implementing such data structures.

Multimedia experiences (such as imaging and video applications) utilize a container format (eg, a metafile, a file format) that allows multiple data streams to be embedded into a single file. These container formats contain metadata that identifies the data stream and details how to implement the data stream's functionality. Containers include: the High Efficiency Image File Format (HEIF), which uses a video codec to encode images using in-frame encoding; and the Joint Photographic Experts Group (JPEG) format (also known as EXIF and JFIF), which uses application partitioning Segment tag (APP tag) to store information. However, there is a growing need for a container format for high dynamic range (HDR) image quality for a variety of multimedia applications.

The embodiments described herein provide a unified container format for delivering different multimedia experiences. Multimedia experiences include, for example, still photo applications, video applications, live photo applications, and the like. Additionally, different multimedia experiences can be different versions of the same application (eg, an original photo application and an updated photo application). The unified container format is backwards compatible with existing formats. Multiple experiences are encapsulated in a single bit stream. Using a unified container format, a playback system can decide which application to run based on computing resources, device capabilities and/or user preferences. In addition, there needs to be minimal interference and dependency between different multimedia applications.

In an illustrative aspect of the present disclosure, a data structure for implementing a plurality of multimedia applications is provided. The data structure includes a first metadata level that contains low-level metadata for performing operations associated with media data in a bit stream. The data structure includes a second metadata level that includes mid-level metadata for applying the low-level metadata to render the media data. The data structure includes a third metadata level that includes high-level metadata for delivering the plurality of multimedia applications using the low-level metadata and the mid-level metadata. The first metadata level further includes synchronized metadata for integrating the media data, the low-level metadata, the mid-level metadata and the high-level metadata from the plurality of multimedia applications. A first multimedia application is converted into a second multimedia application among the plurality of multimedia applications.

In this manner, aspects of the present disclosure provide implementation solutions for multimedia applications such as photo applications and video applications with a high dynamic range and high resolution or a standard resolution, and at least in image projection, signal processing, Improvements in image display and other technical fields.

Cross-references to related applications

This application claims priority over the following priority applications: U.S. Provisional Application No. 63/301,467, filed on January 20, 2022, and U.S. Provisional Application No. 63/399,871, filed on August 22, 2022, and 2022 European application 22155345.6 filed on February 7, 2018.

This disclosure and its aspects may be embodied in various forms, including computer-implemented methods, computer program products, computer systems and networks, user interfaces and application development interfaces, and hardware-implemented methods, signals Processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) and the like control hardware, devices or circuits. The foregoing is intended only to give a general idea of the various aspects of the disclosure and does not limit the scope of the disclosure in any way.

Furthermore, while this disclosure focuses primarily on examples in which various circuits are used in digital projection systems, it should be understood that these are examples only. The disclosed systems and methods may be implemented in a display device, such as with an OLED display, an LCD display, a quantum dot display, or the like. It should be further understood that the disclosed systems and methods may be used in any device in which light is required to be projected, such as, for example, movie theaters, consumer and other commercial projection systems, heads-up displays, virtual reality displays, and the like. &lt;br/&gt;<b>three-level imaging format</b>&lt;b /&gt;

Figure 1 provides a three-level data structure 100 (eg, a metadata structure, a container, and a file structure) for implementing a multimedia experience (eg, a multimedia application). The data structure 100 includes low-level metadata 102 , mid-level metadata 104 and high-level metadata 106 . Low-level metadata 102 and mid-level metadata 104 serve as building blocks implemented by high-level metadata 106 that includes the multimedia experience.

Low-level metadata 102 consists of basic metadata that describes how to perform operations on the uppermost layer of standard media data in a bit stream. Low-level metadata 102 includes predictive metadata (PM) 110, display management metadata (DM) 112, photo metadata (OM) 114, immersive metadata (IM) 116, and interchangeable image file format (EXIF)/International Color Consortium (ICC) metadata (EM) 118, synchronized metadata (SM) 120, and null metadata (NM) 122. In some implementations, low-level metadata 102 may include additional metadata for implementing multimedia experiences.

The prediction metadata 110 includes polynomial predictions, multivariate multiple regression (MMR) predictions, and tensor product B-spline (TPB) predictions. The display management metadata 112 includes different levels of metadata syntax ranging from L0 to L255. Photo metadata 114 is for new still images and may have the same L1 syntax as display management metadata 112 . EXIF/ICC metadata 118 includes metadata related to camera images, such as thumbnail information. The immersive metadata 116 includes target maps, depth maps, and response curves. Empty metadata 122 is reserved and is used when base media data is required without any additional metadata. Several metadata groups may contain several identical or similar syntaxes. In addition, at a decoder, a dispatcher component can convert one syntax in one metadata group to another syntax in another metadata group, thereby reusing existing code and hardware and reducing program size. code size.

The synchronization metadata 120 has a similar structure to the display management metadata 112 . Each level of the synchronization metadata 120 specifies source experiences, sink experiences, and designed operations. The following table provides example syntax. However, other syntaxes can also be used. In some implementations, synchronization metadata 120 is updated and edited using various user-accessible tools.

The payload header indicates the number of extension blocks included in the sync metadata 120. An example of this payload header is shown in Table 1. <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>sm_data_payload() { &lt;br/&gt;</td><td><b>C</b>&lt;br/&gt;</td><td><b>Descriptor</b>&lt;br/&gt;</td></tr><tr><td><b>num_ext_blocks</b>&lt;br/&gt;</td><td>0 &lt;br/&gt;</td><td>ue(v) &lt;br/&gt;</td></tr><tr><td>if(num_ext_blocks) { &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>while(!byte_aligned() ) &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td><b>dm_alignment_zero_bit</b>&lt;br/&gt;</td><td>0 &lt;br/&gt;</td><td>f(1) &lt;br/&gt;</td></tr><tr><td>for( i = 0; i &lt; num_ext_blocks; i ++ ) { &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>ext_metadata_block() &lt;br/&gt;</td><td>0 &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>} &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>} &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>} &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr></tbody></table></table></tables>Table 1: Instance payload headers

Each extension block contains a simple header that indicates the length, level, and container to contain its own payload. An example of this simple header is shown in Table 2. <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>ext_metadata_block() { &lt;br/&gt;</td><td><b>C</b>&lt;br/&gt;</td><td><b>Descriptor</b>&lt;br/&gt;</td></tr><tr><td><b>ext_block_length</b>&lt;br/&gt;</td><td><b>0</b>&lt;br/&gt;</td><td><b>ue(v)</b>&lt;br/&gt;</td></tr><tr><td><b>ext_block_level</b>&lt;br/&gt;</td><td><b>0</b>&lt;br/&gt;</td><td><b>u(8)</b>&lt;br/&gt;</td></tr><tr><td>ext_block_payload(ext_block_length,ext_block_level) &lt;br/&gt;</td><td><b>0</b>&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>} &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr></tbody></table></table></tables>Table 2: Instance extension block header

An example of one of the payload formats themselves is provided in Table 3 below. <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>ext_block_payload(ext_block_length,ext_block_level) { &lt;br/&gt;</td><td><b>C</b>&lt;br/&gt;</td><td><b>Descriptor</b>&lt;br/&gt;</td></tr><tr><td>ext_block_len_bits = 8 * ext_block_length &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>ext_block_use_bits = 0 &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>if( ext_block_level == xxx ) { &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>... &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>} &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>while( ext_block_use_bits++ &lt; ext_block_len_bits ) &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td><b>ext_dm_alignment_zero_bit</b>&lt;br/&gt;</td><td><b>0</b>&lt;br/&gt;</td><td><b>f(1)</b>&lt;br/&gt;</td></tr><tr><td>} &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td> &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>if( ext_block_level == xxx ) { &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>source_experience_ID &lt;br/&gt;</td><td><b>0</b>&lt;br/&gt;</td><td><b>u(8)</b>&lt;br/&gt;</td></tr><tr><td>sink_experience_ID &lt;br/&gt;</td><td><b>0</b>&lt;br/&gt;</td><td><b>u(8)</b>&lt;br/&gt;</td></tr><tr><td>operation_value &lt;br/&gt;</td><td><b>0</b>&lt;br/&gt;</td><td><b>u(32)</b>&lt;br/&gt;</td></tr><tr><td>ext_block_use_bits += 32 &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr><tr><td>} &lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td><td>&lt;b /&gt;&lt;br/&gt;</td></tr></tbody></table></table></tables>Table 3: Instance payload format

The source_experience_ID and sink_experience_ID shown in Table 3 can be defined and mapped at a high level. The operation_value metadata in Table 3 is a reserved location for one of various uses such as the following:

Time synchronization: When editing a video in a video application, a corresponding still photo can be updated in a photo application. In this example, the operation_value metadata can be a frame index or a timestamp.

Spatial resolution: Different multimedia experiences can have different scaling on the same attributes. For example, still photos often have a greater resolution than video. The operation_value metadata defines the resolution mapping between different experiences.

Propagation Options/Strength: Synchronization metadata 120 may indicate whether editing or translation of data between different multimedia experiences is allowed, and the "strength" of correspondence within or between experiences. For example, the operation_value metadata may be a simple binary flag indicating whether the workflow of converting standard dynamic range (SDR) photos to HDR videos is allowed. If the workflow of converting SDR photos to HDR videos is allowed, the operation_value metadata can define which type of SDR to HDR conversion algorithm should be used and its corresponding parameters. For example, it is possible to specify which TPB enumeration is selected so that a different color capacity can be specified for reverse reconstruction from SDR to HDR. In addition, for HDR video to SDR photo conversion, the intensity can indicate which TPB enumeration is selected and used to perform forward reshaping to generate SDR photos.

Metadata Transformation: Synchronized metadata 120 may indicate how to transform metadata from one multimedia experience to another, such as from a photo application to a video application or vice versa.

Auxiliary information: Synchronized metadata 120 may indicate metadata required to perform operations related to a multimedia experience.

When providing multiple multimedia experiences in a single bit stream, multiple extension blocks with the same hierarchy can be utilized. For example:

Multiple sinks: Converting a first experience A to a second experience B can affect a third experience C. In this scenario, two extension blocks with the same level may indicate these operations. For example, a one-bit stream contains SDR photos, HDR videos, and immersive HDR/SDR photos. Updating HDR video affects both SDR photos and immersive HDR/SDR photos.

Multiple sources: One-bit streams can contain SDR photos, HDR videos, and immersive HDR photos. One application can be converted into another application. For example, SDR photos can be converted from HDR videos, and SDR photos can be converted from immersive HDR photos.

Mid-level metadata 104 describes what media data is provided as input to a processing module, and describes how the operational metadata is applied to render the media data. In other words, the mid-level metadata 104 consists of a stream of media data bits and the corresponding low-level metadata 102 required for the rendering used. The media data can be packaged together with the experience metadata within the bitstream, or stored in a different location in the same bitstream. In some embodiments, format backward compatibility can be achieved by storing media data in a different location.

Each multimedia experience may contain zero, one, two or more low-level metadata 102 and different low-level metadata 102 may be encapsulated into an rpu_type format. In some implementations, there are 64 rpu_types available for assignment. In the illustrated example, mid-level metadata 104 contains an rpu_type_2 130 , an rpu_type_4 132 , an rpu_type_5 134 , and an rpu_type_63 136 . rpu_type_2 130 contains media data as a base-level bit stream, and contains two lower-level metadata: prediction metadata 110 and display management metadata 112 . rpu_type_4 132 contains photo metadata 114. rpu_type_5 134 contains immersive metadata 116. rpu_type_63_136 may be reserved for TPB payload.

The high-level metadata 106 includes several applications that utilize the mid-level metadata 104 and the synchronized metadata 120, such as a first application 150, a second application 152, and a third application 154, each application being a different Multimedia experience (such as a photo experience, a video experience, an SDR photo experience, an HDR photo experience, an SDR video experience, an HDR video experience, an immersive SDR/HDR video experience or the like). Multiple experiences of the same type can exist in the same bitstream. Each experience may also include side information that indicates whether media data and metadata are packaged together or positioned in separate locations.

The playback system determines what content to play based on playback computing resources, device capabilities, and end-user preferences. Playback computing resources indicate which experience levels the system can support. Device capabilities include whether the terminal device (eg, output device) can display HDR images and videos, the sensors available on the terminal device, and whether the terminal device has a touch screen. End user preferences include which experience a user has requested or preferred to be presented with.

In some embodiments, high-level metadata 106 includes a default presentation application 156 (eg, a native application). Alternatively, the user can select the presentation mode (or application). In some embodiments, the bitstream may be packaged in an existing container with a format that is preset for the presentation application 156 . For example, a photo experience may include a JPEG thumbnail as a default presentation without being associated with any low-level metadata 102 or mid-level metadata 104 . However, high-level metadata 106 may use synchronization metadata 120 to describe mechanisms between different applications with default profiles. In other embodiments, the bitstream is packaged in a container without a default profile. In these embodiments, the required application is launched through user or environmental input. &lt;br/&gt;<b>media playback</b>&lt;b /&gt;

A device implementing data structure 100 can play different experiences depending on the device's available computing resources, display capabilities, sensor availability, and user preferences. For example, FIG. 2 provides a play operation 200 for a multimedia experience (such as second application 152). The second application 152 may be, for example, a photo or video experience. When the second application 152 is stored in a legacy container (such as a JPEG container) (eg, as defined by the JPEG standard, EXIF and JFIF file formats, and the like) and the legacy container file is delivered to an end device, only Provides JPEG SDR still images.

However, if the second application 152 is provided to an end device that implements the data structure 100, the media data may be viewed as SDR or HDR still images using the photo player 204 that implements rpu_type_4 132, or may be viewed using the video player included in The prediction metadata 110 and display management metadata 112 in the rpu_type_2 130 implemented in 202 are viewed as an HDR live photo (eg video). Whether the media material is viewed using video playback 202 or photo playback 204 is based on user preference or user selection. &lt;br/&gt;<b>media editor</b>&lt;b /&gt;

A media profile can be edited to modify the experience defined by its rpu_type (e.g., its color grading, its resolution, its frame rate, or the like). Figure 3A illustrates an example in-experience editing operation 300 that allows editing to be performed on each experience (eg, each rpu_type) without affecting another experience (eg, another rpu_type). Figure 3B illustrates an example inter-experience editing operation 350 in which edits made in one experience propagate to other experiences. Whether intra-experience editing operations 300 or inter-experience editing operations 350 are performed and how they are propagated between experiences are defined by synchronization metadata 120 .

The in-experience editing operation 300 of Figure 3A allows editing within an experience (eg, second application 152) using an editing tool 310. Although the results of edits may be propagated to other experiences, media data and their corresponding metadata are edited within a single experience. The in-experience editing operation 300 of Figure 3A can perform several different operations based on whether edits made in one experience are automatically reflected in other experiences. These actions may include making individual edits in one experience without updating the other experience, editing one experience and automatically updating another experience with those edits, and editing one experience and updating other experiences with those edits as appropriate. Editing and updating are controlled by the synchronization metadata 120.

In the example of Figure 3A, the second application 152 includes an rpu_type_ <i>m</i>315 and rpu_type_ <i>n</i>320 and synchronization metadata 120. 1 edit rpu_type_ <i>m</i>Module 335 pair rpu_type_ <i>m</i>315 perform any edits. 1 edit rpu_type_ <i>n</i>Module 330 pair rpu_type_ <i>n</i>320 perform any edits. Once the data module 325 is updated and synchronized, the data 120 is updated and synchronized.

As an example of in-experience editing operations 300, synchronization metadata 120 controls the propagation of edits between an SDR still image and an HDR still image. In some embodiments, the SDR still images are edited independently of the HDR still images without updating the HDR still images. In some embodiments, editing HDR stills automatically updates SDR stills. In some embodiments, SDR stills are edited and HDR stills are conditionally updated using an inverse reversible TPB to upconvert SDR to HDR. When editing SDR still images contains new content (such as graphics and text), object-based (eg, mask) side information can be used. The added information can be upconverted differently and then merged in the HDR domain.

Inter-experience editing operations 350 allow joint editing of several experiences using an editing tool 360. For example, in some embodiments, one experience is edited with reference to other experiences, and the editing results automatically update the other experiences. In other embodiments, one experience is edited with reference to other experiences, and the editing results selectively update other experiences. Editing and updating are controlled by the synchronization metadata 120.

In the example of Figure 3B, the second application 152 includes an rpu_type_ <i>m</i>315 and rpu_type_ <i>n</i>320 and synchronization metadata 120. 1 edit rpu_type_ <i>m</i>Module 375 pair rpu_type_ <i>m</i>315 perform any edits. Once the data is updated and synchronized, the data module 365 is updated and the data is set 120 after synchronization. These changes are then propagated to update rpu_type_ <i>n</i>320 one update rpu_type_ <i>n</i>Mod 370.

As an example of inter-experience editing operations 350, synchronization metadata 120 controls the propagation of edits between an SDR still image and an HDR live photo (eg, video). In some embodiments, HDR live photos are edited and SDR stills are automatically updated. In this embodiment, the metadata indicates which frame in the HDR video is to be edited. In other embodiments, SDR stills are edited and HDR live photos are conditionally updated using inverse reversible TPB. In this embodiment, the inverse metadata indicates how to propagate editing results, such as cropping, resizing, and tone mapping. &lt;br/&gt;<b>Experience translation</b>&lt;b /&gt;

Experience translation transforms one experience into another and involves changing content between profiles and tiers even within the same experience. Figure 4A illustrates an example in-experience translation operation 400. Figure 4B illustrates an example inter-experience translation operation 450.

Figure 4A includes an application x 405, an application y 410, and a translation tool 415. Translation tool 415 translates application x 405 into application y 410 . Application x 405 and application y 410 may both be different profiles (or versions) of the same or a similar experience, such as switching from one photo experience to another. In-experience translation operations 400 may include media data translation, in which media data, such as a High Efficiency Video Coding (HEVC) encoded bitstream, is decoded to a common workspace and encoded to another workspace for conversion. Convert different settings such as resolution, color space, frame rate, SEI and VUI. This function is enabled by rpu_type_ <i>m</i>Intra-type translation module 430 is executed, the rpu_type_ <i>m</i>The intra-type translation module converts a rpu_type_ <i>m</i>420 is converted from applying x 405 to applying y 410. In-experience translation operations 400 may also include metadata translation in which an update sync metadata tool 435 updates the metadata (such as sync metadata 120) based on the transformation. For example, display management metadata 112 may be converted to new content.

Figure 4B includes an application x 455, an application y 460, and a translation tool 465. Translation tool 465 translates application x 455 into application y 460. Application x 455 may be, for example, a video experience, and application y 460 may be, for example, a photo experience. Inter-experience translation operations 450 may include media data translation, where media data is converted from one type of media to another type of video. For example, media data can be converted from photos to videos, or vice versa. Media data translation can be done via rpu_type_ <i>m</i>Transfer rpu_type_ <i>n</i>Inter-type translation module 480 is executed, which comes from rpu_type_ <i>m</i>Transfer rpu_type_ <i>n</i>The inter-type translation module converts rpu_type_ <i>m</i>470 translated to rpu_type_ <i>n</i>485. Inter-experience translation operations 450 may also include metadata translation, in which metadata of one type is converted to metadata of another type. For example, photo metadata 114 may be converted to video metadata (included in display management metadata 112) or vice versa. In some embodiments, prediction metadata 110 is transformed. Update synchronization metadata block 490 may update synchronization metadata 120 . &lt;br/&gt;<b>Container implementation</b>&lt;b /&gt;

Data structure 100 can be integrated into existing containers such as HEIF or JPEG. As an example, HEIF uses a video codec to encode an image using in-frame encoding. The syntax of HEIF is based on ISO BMFF (Basic Media File Format). ISO BMFF uses "blocks" to structure different types of data. Each block is guided by a four-character type (4CC). Blocks can be nested or hierarchical (for example, one block within another block). Each block has a size that is an integer number of the number of bytes in the specified block. Additionally, each block has a block type, such as compact.

Two different definitions are used to distinguish still images from video. First, still images are stored as projects. All image items are encoded independently and do not depend on any other items in their decoding. Any number of image items can be included in the same file. Image sequences (such as videos) are stored as tracks. An image sequence playback track is used when there are encoding dependencies between images or when the playback of images is timed. In contrast to the video track, the timing of the image sequence track is advisory.

The HEIF bitstream contains two main components: media data with 4CC as "meta" suffix data, and media data with 4CC as media data block "mdat" or item data block "idat". The metadata portion represents side information (for example, the structure of a multimedia application), and the media data stores multimedia data.

The components of data structure 100 may be implemented as blocks in a HEIF bitstream. For example, low-level metadata 102 can be defined as the block "dbom": <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>aligned(8) class ImagingOperationMetadata &lt;br/&gt;xtends FullBox('dbom', version, flags) &lt;br/&gt; &lt;br/&gt;nsigned int(32) metadata_type; &lt;br/&gt;it(8) data[]; &lt;br/&gt; &lt;br/&gt;</td></tr></tbody></table></table></tables>

For example, the above metadata_type can be "prdm" for prediction metadata 110, "dipm" for display management metadata 112, "phom" for photo metadata 114 and "synm" for sync metadata 120 ". Additional components of low-level metadata 102 may also be defined.

The mid-level metadata 104 can be defined as the block "dbex" containing the low-level metadata 102: <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>aligned(8) class ImagingExperienceMetadata &lt;br/&gt;xtends FullBox('dbex', version, flags) &lt;br/&gt; &lt;br/&gt;nsigned int(32) experience_type; &lt;br/&gt;nsigned int(32) operation_metadata_count; &lt;br/&gt;nsigned int(32) media_data_count; // consistency checking &lt;br/&gt;or(i = 0; i &lt; operation_metadata_count; i++) { &lt;br/&gt;magingOperationMetadata operation[]; &lt;br/&gt; &lt;br/&gt;f(experience_type == 2 ){ // Video Experience &lt;br/&gt; &lt;br/&gt;lse if(experience_type == 4 ){ // Photo Experience &lt;br/&gt; &lt;br/&gt;lse if(experience_type == 5 ){ // Immersive Experience &lt;br/&gt; &lt;br/&gt;lse{ &lt;br/&gt;/ not supported &lt;br/&gt; &lt;br/&gt; &lt;br/&gt;</td></tr></tbody></table></table></tables>

The association of media data with each experience metadata and operation metadata can be linked through an item property block ("iprp") included in the HEIF bitstream. For high-level metadata 106, an "iref" box defines the required metadata and media data for each application. In some implementations, blocks for synchronizing metadata 120 are stored at the highest level.

As another example, JPEG-XT is backward compatible with JPEG. JPEG-XT uses an APP tag (APP11) to store additional extended information. There are 16 APP tags present in JPEG. To take advantage of data structure 100, the format can be extended to APP11 without restriction. Since JPEG tags (16-bit 0xff followed by a 16-bit ID) have 16 bits of information indicating the number of bytes in the current tag, each tag only carries 2 ¹⁶= 64 kilobytes of information. If the metadata is larger than 64 KB, multiple APP tags are required.

In addition, in order to avoid false detection of mark byte "0xff", for a non-mark byte 0xff, each 0xff should be followed by a byte filled with 0x00. Using the block definitions and block structures used in ISO BMFF, headers and media data can be placed in multiple APP tags in JPEG. Therefore, while providing a raw experience as a JPEG still image, other devices can provide a similar experience. &lt;br/&gt;<b>make still images</b><b>HDR</b><b>ability</b>

The blocks described can be further extended to provide an HDR experience for still images. Table 4 provides example codec options for an imaging experience. <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>ID &lt;br/&gt;</td><td>Codec selection &lt;br/&gt;</td></tr><tr><td>1 &lt;br/&gt;</td><td>PQ HDR in HEIF &lt;br/&gt; PQ, R2020, 10 Knits &lt;br/&gt; YCbCr/IPT &lt;br/&gt; HEVC 10b &lt;br/&gt;</td></tr><tr><td>2 &lt;br/&gt;</td><td>HLG HDR in HEIF &lt;br/&gt; HLG, R2020, 1 Knits &lt;br/&gt;YCbCr &lt;br/&gt; HEVC 10b &lt;br/&gt;</td></tr><tr><td>3 &lt;br/&gt;</td><td>JPG + HDR (single stock in JPEG) &lt;br/&gt; γ, R709, 100nits; YCbCr; JPG 8bit &lt;br/&gt; PQ/HLG, R2020, 10knits; YCbCr/IPT; HEVC 10b &lt;br/&gt;</td></tr><tr><td>4 &lt;br/&gt;</td><td>SDR + HDR (two HEVCs, as a single stock in HEIF) &lt;br/&gt; γ, R709, 100nits; YCbCr; HEVC 10b &lt;br/&gt; PQ,/HLG R2020, 10knits; YCbCr/IPT; HEVC 10b &lt;br/&gt;</td></tr></tbody></table></table></tables>Table 4: Example of video experience codec &lt;b /&gt;

As an example, for a photo application, the photo metadata 114 may be used to store an HDR HEVC encoded still image. Alternatively, a hybrid log-gamma (HLG) transfer function can be used in HEVC-encoded still images instead of a perceptual quantization (PQ) transfer function, as shown in the following pseudocode: <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>FileTypeBox 'ftyp': major-brand='heic', compatible-brands='heic, mp41' &lt;br/&gt;etaBox: (container) &lt;br/&gt;andlerBox 'hdlr': 'pict' &lt;br/&gt;rimaryItemBox 'pitm': item_ID=1; &lt;br/&gt;temInfoBox 'iinf': entry_count=2 &lt;br/&gt;) 'infe': item_type='hvc1', item_ID=1; &lt;br/&gt;) 'infe': item_type='Exif', item_ID=2; &lt;br/&gt;temLocationBox'iloc': item_count=2 &lt;br/&gt;tem_ID=1, extent_count=1, extent_offset=X, extent_length=Y; &lt;br/&gt;tem_ID=2, extent_count=1, extent_offset=P, extent_length=Q; &lt;br/&gt;temReferenceBox 'iref': &lt;br/&gt;eferenceType ='cdsc', from_item_ID=2, ref_count=1, to_item_ID=1; &lt;br/&gt;temPropertiesBox 'iprp': &lt;br/&gt;temPropertyContainerBox 'ipco': &lt;br/&gt;hvcC' &lt;br/&gt;ispe' &lt;br/&gt;temPropertyAssociation 'ipma', entry_count=1: &lt;br/&gt;tem_ID=1, association_count=2, &lt;br/&gt;ssential=1, property_index=1; &lt;br/&gt;ssential=0, property_index=2; &lt;br/&gt;ediaDataBox 'mdat': &lt;br/&gt;EVC for HDR Image (at file offset X, with length Y) &lt;br/&gt;xif data block (at file offset P, with length Q) &lt;br/&gt;</td></tr></tbody></table></table></tables>

In another example, for a photo application, the EXIF metadata can be used to store an SDR JPEG encoded still image and the photo metadata can be used to store an HDR HEVC encoded still image, as described in the pseudocode below. exhibit: <tables><table><table border="1" borderColor="#000000" width="85%"><tbody><tr><td>FileTypeBox 'ftyp': major-brand='heic', compatible-brands='heic' &lt;br/&gt;etaBox 'meta': (container) &lt;br/&gt;andlerBox 'hdlr': 'pict' &lt;br/&gt;rimaryItemBox 'pitm': item_ID=1; &lt;br/&gt;temInfoBox 'iinf': entry_count=3 &lt;br/&gt;) 'infe': item_ID=1, item_type='jpeg'; &lt;br/&gt;) 'infe': item_ID=2, item_type='Exif'; &lt;br/&gt;) 'infe': item_ID=3, item_type='hvc1'; &lt;br/&gt;) 'infe': item_ID=3, item_type='dbex'; &lt;br/&gt;temLocationBox 'iloc': item_count=4 &lt;br/&gt;tem_ID=1, extent_count=1, extent_offset=X1, extent_length=Y1; &lt;br/&gt;tem_ID=2, extent_count=1, extent_offset=X2, extent_length=Y2; &lt;br/&gt;tem_ID=3, extent_count=1, extent_offset=X3, extent_length=Y3; &lt;br/&gt;tem_ID=4, extent_count=1, extent_offset=X4, extent_length=Y4; &lt;br/&gt;temReferenceBox 'iref': &lt;br/&gt;eferenceType='cdsc',from_item_ID=2,ref_count=1,to_item_ID=1; &lt;br/&gt;eferenceType='cdsc',from_item_ID=4,ref_count=1,to_item_ID=3; &lt;br/&gt;temPropertiesBox 'iprp': &lt;br/&gt;temPropertyContainerBox 'ipco': &lt;br/&gt;hvcC' &lt;br/&gt;ispe' &lt;br/&gt;temPropertyAssociation 'ipma': entry_count=2 &lt;br/&gt;) item_ID=1, association_count=2 &lt;br/&gt;ssential=1, property_index=1; &lt;br/&gt;ssential=0, property_index=2; &lt;br/&gt;) item_ID=1, association_count=2 &lt;br/&gt;ssential=1, property_index=3; &lt;br/&gt;ssential=0, property_index=4; &lt;br/&gt;ediaDataBox 'mdat' or 'idat': &lt;br/&gt;PEG SDR Image (at file offset X1, with length Y1) &lt;br/&gt;xif data block (at file offset X2, with length Y2) &lt;br/&gt;EVC HDR Image (at file offset X3, with length Y3) &lt;br/&gt;bex data block (at file offset X4, with length Y4) &lt;br/&gt;</td></tr></tbody></table></table></tables>

Figure 5 illustrates an example of a Basic Media File Format (BMFF) according to an embodiment. This BMFF is for a hypothetical new HDR photo format called, but not limited to, "Dolby Imaging" or "DI". It is based on the JPEG file format using the APP11 tag, although any other APP tag can be used. As depicted in Figure 5, BMFF contains: &lt;br/&gt; An APP tag (505) (for example, APP11) (2 bytes) Payload length (510) (2 bytes) An identification string (515) (for example, "DI") One empty byte (520) Dolby Imaging payload data (525), which contains HDR (eg, HEVC) image data (529) and rpu metadata (527).

The systems and methods described above can provide data structures for multimedia experiences. Systems, methods and devices according to the present disclosure may adopt any one or more of the following configurations.

(1) A data structure for implementing a plurality of multimedia applications, the data structure including: a first metadata level that includes low-level data for performing operations associated with media data in a one-bit stream. hierarchical metadata; a second metadata level that includes mid-level metadata for applying the low-level metadata to render the media data; and a tertiary metadata level that includes mid-level metadata for using The low-level metadata and the mid-level metadata are used to deliver high-level metadata of the plurality of multimedia applications, wherein the first metadata level further includes synchronized metadata for converting the media data, The low-level metadata, the mid-level metadata and the high-level metadata are converted from a first multimedia application in the plurality of multimedia applications to a second multimedia application in the plurality of multimedia applications. application.

(2) The data structure is as in (1), in which the low-level metadata includes: prediction metadata, which contains a polynomial prediction, a multivariable complex regression prediction and a volume product B-spline prediction; display Management metadata, which is used to display video using the media data; photo metadata, which is used to display an image using the media data; exchangeable image file format metadata, which contains metadata related to camera images; Empty metadata, which is used to implement the base media data; and immersive metadata, which contains an object map, a depth map and response curves for the media data.

(3) A data structure as in any one of (1) to (2), wherein the synchronization metadata includes a payload header indicating a number of extension blocks, each extension block including a payload header indicating its respective valid A length, level and header of the payload.

(4) The data structure of (3), wherein the payload header includes source_expierience_ID metadata, sink_expierience_ID metadata and operation_value metadata.

(5) The data structure of (4), wherein the operation_value metadata is a metadata selected from the group consisting of: a frame index of a still photo, a timestamp of a video frame , a resolution mapping function and a binary flag indicating whether to allow a workflow from a standard dynamic range (SDR) photo to a high dynamic range (HDR) video.

(6) The data structure of any one of (1) to (5), wherein the plurality of multimedia applications include a still photo application and a video application.

(7) The data structure of (6), wherein when the media data is manipulated in the still photo application, the synchronization metadata updates the media data, the low-level metadata, and the mid-level metadata for the video application. Metadata and the high-level metadata.

(8) The data structure of any one of (1) to (7), wherein the low-level metadata and the mid-level metadata are implemented as blocks within a High Efficiency Image Format (HEIF) container.

(9) The data structure of (8), wherein the blocks are implemented as application segment tags (APP tags) in a Joint Photographic Experts Group (JPEG) container.

(10) A controller for implementing the data structure of any one of (1) to (9), wherein the controller is configured to: edit a first application included in the plurality of multimedia applications a video in; and propagating the edit to a corresponding photo included in a second application of the plurality of multimedia applications, wherein the synchronization metadata includes a frame index for the corresponding photo.

(11) The data structure of any one of (1) to (9), wherein the synchronization metadata defines a resolution mapping between the first multimedia application and the second multimedia application.

(12) The data structure of any one of (1) to (9) or (11), wherein the synchronization metadata includes an indication of whether to allow conversion from a standard dynamic range (SDR) photo application to a high dynamic range (HDR) ) A binary flag for the workflow of a video application.

(13) The data structure of (12), wherein the synchronization metadata includes the SDR to HDR conversion algorithm and its corresponding parameters.

(14) A controller for implementing the data structure of any one of (1) to (9) or (11) to (13), wherein the controller is configured to: receive instructions for the plurality of multimedia A user input from one of the applications; and providing the media data as an SDR still photo application or an HDR video application based on the user input.

(15) A controller for implementing the data structure of any one of (1) to (9) or (11) to (13), wherein the controller is configured to perform a multimedia in-application editing operation , wherein the media data in a first multimedia application is edited without affecting a second multimedia application.

(16) A controller for implementing a data structure as in any one of (1) to (9) or (11) to (13), wherein the controller is configured to perform a multimedia inter-application editing operation , wherein the media data in a first multimedia application will be edited and spread to a second multimedia application.

(17) A controller for implementing the data structure of any one of (1) to (9) or (11) to (13), wherein the controller is configured to perform a multimedia in-application translation operation , wherein the media data is translated from a first multimedia application of a first type to a second multimedia application of a first type.

(18) A controller for implementing a data structure as in any one of (1) to (9) or (11) to (13), wherein the controller is configured to perform a multimedia inter-application translation operation , wherein the media data is translated from a first multimedia application of a first type to a second multimedia application of a second type.

(19) A controller for implementing a data structure as in any one of (1) to (9) or (11) to (13), wherein the controller is configured to: based on playback computing resources, devices At least one of capabilities and end-user preferences provide the media data as an SDR still photo application or an HDR video application.

(20) A controller for implementing the data structure of any one of (1) to (9) or (11) to (13), wherein for providing an imaging included in the plurality of multimedia applications For applications, the controller is configured to include at least one of: (a) a first codec including a perceptual quantization transfer function for HDR in a HEIF container; (b) a second a codec that includes a hybrid log-gamma transfer function for HDR in the HEIF container; (c) a third codec that includes JPG and HDR as a single library in a JPEG container; and (d) A fourth codec that includes both SDR and HDR as a single library in HEVC and the HEIF container.

With regard to the procedures, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such procedures, etc., have been described as occurring according to a specific ordered sequence, they may occur in an order different from that described herein. These procedures are practiced by performing the described steps in one order. It should be further understood that certain steps may be performed concurrently, additional steps may be added, or certain steps described herein may be omitted. In other words, the description of the procedures herein is provided for the purpose of illustrating particular embodiments and should in no way be construed as limiting the scope of the claims.

Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications in addition to the examples provided will be immediately apparent upon reading the above description. The scope should not be determined by reference to the above description, but instead should be determined by reference to the accompanying claims and the full scope of equivalents to which such claims are granted. It is anticipated and intended that future developments in the technology discussed herein will occur, and that the disclosed systems and methods will be incorporated into such future embodiments. In summary, it should be understood that this application is capable of modifications and variations.

All terms used in the claims are intended to be given the broadest reasonable interpretation and ordinary meaning of such terms as understood by one skilled in the art described herein, unless an express indication to the contrary is made herein. Specifically, the use of singular articles such as "a", "the", "said", etc. shall be read as stating one or more of the indicated elements, unless a technical solution statement indicates the contrary. One of the clear restrictions.

The Summary of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. The Abstract of the Disclosure is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, it can also be seen that in the foregoing embodiments, various features are grouped together in various embodiments for the purpose of simplifying the present disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly stated in each claim. Rather, as the following patent scope reflects, inventive subject matter lies in less than all features of a single disclosed embodiment. Therefore, the following patent claims are hereby incorporated into the embodiments, with each technical solution independently serving as a separately claimed subject matter.

<p>100: Three-level data structure/data structure &lt;br/&gt;02: Low-level metadata &lt;br/&gt;04: Mid-level metadata &lt;br/&gt;06: High-level metadata &lt;br/&gt;10: Prediction metadata &lt;br/&gt;12: Display management metadata &lt;br/&gt;14:Photo metadata &lt;br/&gt;16: Immersive metadata &lt;br/&gt;18: Exchangeable Image File Format/International Color Alliance Metadata &lt;br/&gt;20: Synchronize metadata &lt;br/&gt;22: Empty metadata &lt;br/&gt;30:rpu_type_2 &lt;br/&gt;32:rpu_type_4 &lt;br/&gt;34:rpu_type_5 &lt;br/&gt;36:rpu_type_63 &lt;br/&gt;50:First application &lt;br/&gt;52: Second application &lt;br/&gt;54:Third application &lt;br/&gt;56: Default presentation application &lt;br/&gt;00: Playback operation &lt;br/&gt;02: Video playback &lt;br/&gt;04:Photo playback &lt;br/&gt;00: Editing operation within the instance experience/Editing operation within the experience &lt;br/&gt;10: Editing tools &lt;br/&gt;15:rpu_type_ <i>m</i>&lt;br/&gt;20:rpu_type_ <i>n</i>&lt;br/&gt;25: Update the post-synchronization data module &lt;br/&gt;30: Edit rpu_type_ <i>n</i>module &lt;br/&gt;35: Edit rpu_type_ <i>m</i>module &lt;br/&gt;50: Editing operation between instance experiences/Editing operation between experiences &lt;br/&gt;60:Editing tools &lt;br/&gt;65: Update the post-synchronization data module &lt;br/&gt;70: Update rpu_type_ <i>n</i>module &lt;br/&gt;75: Edit rpu_type_ <i>m</i>module &lt;br/&gt;00: Instance translation operation/in-experience translation operation &lt;br/&gt;05:Application x &lt;br/&gt;10: Application y &lt;br/&gt;15: Translation tool &lt;br/&gt;20:rpu_type_ <i>m</i>&lt;br/&gt;30:rpu_type_ <i>m</i>Intra-type translation module &lt;br/&gt;35: Update synchronization metadata tool &lt;br/&gt;50: Translation operation between instances/experiences &lt;br/&gt;55: Application x &lt;br/&gt;60: Apply y &lt;br/&gt;65: Translation tool &lt;br/&gt;70:rpu_type_ <i>m</i>&lt;br/&gt;80: self rpu_type_ <i>m</i>Transfer rpu_type_ <i>n</i>Inter-type translation module &lt;br/&gt;85:rpu_type_ <i>n</i>&lt;br/&gt;90: Set data block after update synchronization &lt;br/&gt;05: Apply segmentation mark &lt;br/&gt;10: Payload &lt;br/&gt;15: Identify strings &lt;br/&gt;20: Empty tuple &lt;br/&gt;25: Dolby Imaging payload information &lt;br/&gt;27:rpu metadata &lt;br/&gt;29: High dynamic range image data </p>

These and other more detailed and specific features of various embodiments are more fully disclosed in the following description with reference to the accompanying drawings, in which:

Figure 1 illustrates an example data structure for implementing a multimedia experience.

Figure 2 illustrates an example playback operation for a multimedia experience.

Figure 3A illustrates an example in-experience editing operation for a multimedia experience.

Figure 3B illustrates an example inter-experience editing operation for a multimedia experience.

Figure 4A illustrates an example in-experience translation operation for a multimedia experience.

Figure 4B illustrates an example inter-experience translation operation for a multimedia experience.

Figure 5 illustrates an example base media file format according to an embodiment of the present invention.