System and method for management of scalability information in scalable video coding systems using control messages

[0001] The present invention relates to video communication systems. In particular, the present invention relates to techniques for transmitting the structure of a scalably encoded video bit stream, and its changes, in a video communication system, in a manner that reduces both bit rate and computational complexity.

BACKGROUND OF THE INVENTION [0002] New techniques for "scalable" coding of digital video and audio data that are aimed at the overall improvement of coding efficiency have a number of new structural features. In particular, scalability is an important new feature. In scalable coding, the original or source signal is represented using two or more hierarchically constructed bit streams. The hierarchical structure implies that decoding of a given bit stream is based on the availability of some or all of the other bitstreams that are lower in the hierarchy. Each bit stream, in conjunction with the bit streams on which it is based, provides a representation of the original signal with a specific time resolution, fidelity (eg in terms of signal-to-noise ratio (SNR)) or spatial (for video).

[0003] It is understood that the phrase "scalable" does not refer to numerical size or scale, but rather to the possibility of a given coding technique to provide a set of different bit streams corresponding to effective representations of the original or source signal on different "scales" of resolution or in generality of other features. The upcoming ITU-T H.264 specification, Appendix F, which will be referred to as Scalable Video Coding (SVC), is an example of a video encoding standard that provides the scalability of video encoding in all time, spatial and faithful dimensions. SVC is an extension of the H.264 standard (also known as Advanced Video Coding, AVC). An example of an earlier standard that also made all three types of scalability available is ISO MPEG-2 (also published as ITU-T H. 262). ITU G.729.1 (also known as G.729EV) is an example of a standard that provides scalable audio coding. [0004] Scalability has been introduced in video and audio coding as a solution to distribution problems in streaming and broadcasting, and with a perspective to allow a given communication system to work with different access networks (e.g., clients connected in different bands), network conditions (e.g. band fluctuations) and client devices (e.g., a personal computer equipped with a large display vs. a mobile device with a much smaller display).

[0005] Scalable video coding techniques that are specifically designed for use in interactive video communication such as video conferencing are described in the international patent publication of this authorized WO2008 / 060262. Furthermore, the international patent publication of this authorized WO2008 / 060262 describes a project of a new type of server, called the Scalable Video Communication Server (SVCS). The SVCS may advantageously use scalable video for video communication with high quality and low latency, and has a complexity that is significantly reduced compared to the classic switching or transcoding of Multipoint Control Units (MCUs). Similarly, the international patent publication of this authorized WO2007 / 076486 describes a Composite Scalable Video Coding Server (CSVCS) which has the same advantages as the SVCS, but generates a single encoded output bit stream. Scalable video coding project and architecture

PZ / 4046 / JDN 2 ΕΡ 2 005 607 BI

SVCS / CSVCS can be used in more advantageous ways, which are described, for example, in international patent publications of this authorized WO2007 / 075196, WO2008 / 051181, WO2007 / 067990, WO2007 / 095640, WO2007 / 103889. The above applications describe the use of scalable coding techniques and SVCS / CSVCS architecture for, respectively, trunking between servers, reduction of buffer delay jitter, fault tolerance and free access, thinning of scalable bitstream video streams to increase coding efficiency with reduced packet loss, and bandwidth control.

[0006] Particular attention is now paid to improved video and audio communication systems that use scalable video or audio coding. In particular, with the prospect of developing such systems, the attention is directed to the management of scalability information provided from the source of the video or audio stream to the recipient - directly or through one or more servers. The source may be a transmitting point that encodes and transmits video "live" via a communication network, a streaming server that transmits pre-encoded video, or a program module that provides access to a file stored in mass storage or other access device. Similarly, the recipient can be a receiving point,

SUMMARY OF THE INVENTION [0007] The object of the present invention is to provide systems and methods for improved information management on scalability in a video communication system using scalable coding. This object can be achieved by the features defined in the independent claims. Further extensions are characterized in the dependent claims.

[0008] Numerous examples of systems and methods communicate the structure of a scaledly coded bitstream, as well as changes that may take place over time, to system components. Communications techniques use differential encoding of relevant information and preferably achieve a reduced overhead bit rate and reduced computational complexity.

[0009] In a preferred embodiment for a video communication system that uses a coding format compliant with the SVD Standard H.264, the standard SEI messages with scalability information are modified and supplemented to include the ability to signal changes in the scalability information. The extended set of SEI message scalability information can inform recipients (eg servers, decoders / endpoints) about the structure of scalability layers of transmitted signals and their changes. Recipients can use information from the scalability layer to process or decode the received signals accordingly.

BRIEF DESCRIPTION OF THE DRAWINGS [0010] Further features, nature, and various advantages of the present invention will be more apparent from the following detailed description based on the preferred embodiment and the accompanying drawings in which:

FIG. 1 is a block diagram illustrating an exemplary architecture of an SVC transmission system according to an embodiment of the present invention;

FIG. 2 is a flow diagram of a scalable information state management process in accordance with an embodiment of the present invention;

PZ / 4046 / JDN 3 ΕΡ 2 005 607 BI

FIG. 3 is a diagram of a flow management process of the scalability information state for an alternative example solution.

[0011] In all figures, the same reference numerals and character numbers, if not otherwise indicated, are used to denote similar features, elements, components or parts of the illustrated embodiments. Furthermore, while the present invention will now be described using the example with reference to the figures, this is done in conjunction with the illustrative embodiments.

DETAILED DESCRIPTION [0012] Systems and methods are provided for improved management of scalability information in a video and audio communication system that uses scalable coding.

[0013] FIG. 1 illustrates an exemplary architecture of a communication system 100 that uses scalable coding. The communication system 100 includes a media server or encoder 110 (e.g., a streaming server or broadcasting point) that communicates video and / or audio signals to the client / receiver 120 via the network 130 via a media gateway 140.

[0014] For brevity, this description is limited to a video part of such a communication system. However, it will be understood that the systems and methods described herein for the video part can be used for scalable audio parts, understanding that the spatial scalability of the audio signal can not be provided, but multi-channel encoding can additionally be used in encoding the audio signal. Further, the systems and methods described herein may also be used for other multimedia data (e.g., graphics) that are encoded in a scalable manner. [0015] In the preferred embodiment of the communication system 100, the SVC H.264 encoding (SVC) format is used for video communication. (See, e.g., the SVC JD5 specification, T. Wiegand, G. Sullivan, J. Reichel, H. Schwarz, M. Wien (ed.) "Joint Draft 5: Scalable Video Coding," Joint Video Team, doc. JVTR201, Bangkok, Thailand, January 2005). SVC is a scalable extension of video encoding (Annex F) of the H.264 AVC video encoding standard.

[0016] SVC provides considerable flexibility when creating bit stream structures with scalability in several dimensions (e.g., spatial, temporal, fidelity or quality dimensions). A bitstream encoded with the help of SVC can be composed of many components or layers. The base layer provides a representation of the source signal with some basic fidelity. Additional layers (extension layers) provide information for improved signal representation in additional dimensions of scalability over the basic fidelity dimension. It should be noted that the encoded bit stream layers are typically formed into a pyramid structure in which the decoding of a given layer may require the presence of one or more lower layers. typically, the availability of the base layer is required to decode any extension layer. A pyramid layer structure does not always have a place. For example, when scalability is provided by multiple description coding or parallel simulation, independent decoding of some or all layers may be possible. The systems and methods described herein apply to all such scalability formats.

[0017] Scalability can help address system-level problems such as network and / or client heterogeneity, time-varying network performance, networks without flow guarantee or delivery, etc. However, to be able to be effective

PZ / 4046 / JDN 4 ΕΡ 2 005 607 BI to use these features, it is necessary to make them available to other system components, except coder and video decoder.

[0018] Various embodiments of the systems and methods of the present invention that effectively use all of the scalability features are understood with reference to FIG. 1. First, it should be noted that the use of media gateway 140 in system 100 is optional. The various embodiments of the systems and methods of the present invention are identical or similar when using the media server directly with the client, or when the media server is replaced by a file that is directly accessible to the client on storage or other access devices directly or indirectly (e.g. access to a file via a communication network). [0019] A simple operational scenario can be considered in which the media server / encoder 110 (e.g. a streaming server or encoder at the transmitting point) transmits the scalable media to the client / receiver 120 via the media gateway 140. This simple scenario requires that the connection be established between the media server and the client for transmitting a predetermined set of layers, which may, for example, be SVC NAL entities encapsulated in RTP. In addition, the media gateway 140 must be instructed (or inferred) to use the incoming packets in the best possible way (e.g., transmitted NAL SVC units encapsulated in RTP). In the case where the media gateway 140 has an SVCS / CSVCS architecture, this operational decision corresponds to the decision which packages should be rejected and which to forward. Further, for the decoder operation to work properly, the client / receiver 120 must know or be able to deduce which set of layers it should receive.

[0020] In order to enable these functionalities, the system 100 must represent and transmit the scalability structure of the transmitted bitstream to various system components. As an illustrative example, consider a video signal with two time resolutions, 15 and 30 fps (frames per second) and two spatial resolutions, QCIF and CIF. This corresponds to a four-layered scalability structure: the L0 layer containing the QCIF signal at 15 fps; L1 layer containing the QCIF signal extension to 30 fps; layer S0 containing the CIF signal extension to 15 fps; and layer S1 containing the CIF signal extension to 30 fps. The coding dependencies in a four-layer scalability structure may be such that L0 is a base layer, L1 relies on L0, S0 is based on L0 and S1 based on both L1 and S0.

[0021] The SVC provides a mechanism for describing the scalability structure of the encoded SVC of a video bitstream through its messages with information on scalability SEI (Scalability Information SEI message, SSEI)). Supplemental Enhancement Information (SEI) messages are data structures included in the SVC bitstream that provide sub-information about the encoded video signal, but are not necessary for the decoding process to work. The SSEI message, defined in section F.10.1.1 of SVC JD5 specification, contains descriptive information about each layer (e.g., frame rate, profile information) and, what is significant, information about dependencies in coding, i.e. on which other layers are given the layer is based for correct decoding. Each layer is identified, in the scope of a given bit stream, by the unique identifier of the layer "layer id". Information about dependencies in coding for a given layer is passed by encoding the number of directly dependent layers (num_directly_dependent_layers) and a sequence of differential values ​​(directly_dependent_layer_id_delta),

Which, when added to the identifier of a given layer, define the identifiers of the layers on which the decoding of a given layer is based.

[0022] SSEI messages as defined in section F.10.1.1 of the SVD JD5 specification have been designed to allow negotiation of capabilities (e.g., at the time of establishing the connection), stream adaptation (via a video server or intermediary media gateway) and low complexity processing ( e.g. without inference based on a detailed bit stream analysis). However, the SSEI JD5 project has numerous limitations that become evident in a closer examination of how information from SSEI messages must be used in a communication system.

[0023] In the operation of the communication system (e.g., system 100), there may be many situations in which the structure of the stream being transmitted is established or modified (either on the server / source or on the media gateway):

• Pre-definition of opportunities during negotiations. The media server provides a set of scalability layers, and the client accepts all or a subset of them, either directly or in the negotiation process (eg using the session description protocol, Session Description Protocol, SDP).

• Modification of the layer structure by a media server or media gateway to adapt to changing system conditions (eg increased packet loss between the media gateway and the client or change of the desired display resolution by the client).

• Modification of the structure of layers due to coder decisions. An example of such an encoder decision is related to Fine Granularity Scalability (FGS), a mechanism for providing fidelity scaling, where a FGS layer stream bit fragment can be used to produce a signal of reduced quality. In connection with how the FGS was designed in SVC JD5 it is possible that the interdependencies between layers change between consecutive images (or strictly speaking, between one slice and the next). In particular, a case may be considered in which the prediction is made for a given layer using, as a reference, a lower layer that uses FGS, and this prediction is based only on a fraction of the entire layer. In this case, it is possible that different images of the same layer depend on different lower layers, as some layers can be completely eliminated by bandwidth control. A possible solution that would not change the structure of dependencies between layers is the use of empty NAL units. However, this is not allowed in SVC JD5. This case can be perceived as similar to the previous one, with the difference that the time scale in which there are changes in dependencies between the layers is very high (in each patch).

[0024] Negotiation of capabilities is a feature of a communication system outside the scope of the SVC specification and scalable coding in general. Related areas of communication applications, such as IP telephony, have developed complex procedures for negotiating opportunities. For example, communication systems based on the Session Initiation Protocol (SIP) use the SDP offer / response. It should be noted that the use of SSEI itself for functionalities of negotiating opportunities is unnecessary. According to an embodiment of the present invention, SSEIs should not only indicate what the encoder is capable of producing, but preferably indicate what is (or what will be) actually contained in the transmitted bitstream. In this regard, the SSEI, according to the practice of the present invention, is modified,

PZ / 4046 / JDN 6 ΕΡ 2 005 607 B1 [0025] The modification of the layer structure (i.e., stream adaptation) may occur in the operation of the communication system on the media server / encoder or intermediary media gateway (more than one such media gateway may be present) . An example of a stream adaptation is the case in which the client / receiver 120 decides to switch or change the resolution (e.g. from CIF to QCIF). Switching or changing the resolution is accomplished by discarding corresponding packets on the media gateway 140 or on the transmitting media server 110. The resolution change call is achieved using classical signaling (shown as signaling paths 150 in FIG. 1). Particularly important for the proper switching operation is that the client / receiver 120 can be notified that the desired change has taken place. In the absence of such a notification, the client / receiver 120 will at some point receive only a subset of all packets or bits (those corresponding to the QCIF resolution). However, he can not know if the loss of high resolution packets or bits is deliberate (the result of his request) or because of packet losses or bits. The reasons for decoding in one loop may degrade the problem, as the sub-layer image may not be available to the client / receiver 120 to display in the absence of target layer data. [0026] As previously noted, such switching information may be provided by signaling messages from the media server or media gateway to the client / receiver 120 in a conventional manner (e.g., via signaling paths 150). However, various embodiments of the present invention allow to observe significant advantages (e.g. in terms of synchronization and response time under error conditions) if the layer structure information is carried in the band together with the coded data. Information about the structure of layers (and their changes) can be carried in the band in modified SSEI messages.

[0027] Another example of adapting a stream to changing system conditions relates to adaptation to packet losses caused by the network. In this case, the media gateway may decide to change or reduce, for example, the bandwidth of the stream being transmitted using the quality scaling functionality. Again, for the proper operation, the client / receiver 120 must be informed or instructed at the right time about the change in the bit stream that it receives. The changing information about the structure of the layers may be timely carried in the modified SSEI messages in the band together with the coded data.

[0028] Similar to the situation of adaptation of the stream are situations of changes in the structure of layers induced by the encoder. Unlike the stream adaptation situation discussed above, the changes in the layer structure caused by the encoder are not initiated by the client or media server in response to a change in system parameters, but are an "artefact" of the encoding process used by the video encoder. As in the case of signal adaptation, information about the variability of the layer structure can be timely conveyed by the modified SSEI messages in the band together with the encoded data for appropriate synchronization with the media gateway and the decoder.

[0029] Consideration of the operation of the system 100 in the above situations shows that there is a need to provide messages with scalability information at different times during a communication session and in time scales (or frequencies) that can be as high as at the patch level.

[0030] It should be noted that providing messages with scalability information (e.g., modified SSEI messages) can only be practical if the overhead resulting from the use of these messages is extremely small. In addition, due to the number in the communication system

PZ / 4046 / JDN Ί ΕΡ 2 005 607 BI clients served by a given server or gateway is usually maximized for economic reasons, it is of course desirable to minimize the computational burden related to analysis and the creation of relevant messages in the communication system.

[0031] It should be noted that the SVD JD5 specification sanctions the use of SSEI only for instantaneous decoding refresh (IDR) units (section F.10.2.1). IDR access units are equivalents of "intra" images that have a significant bit rate overhead and are typically used only once in interactive video communication systems (for coding the first image). This limitation also makes it impossible to use standard SSEI to signal changes in scalability information.

[0032] Because one of the basic uses of SSEI is to support servers and media gateways in properly servicing SVC streams, it is preferable that the processing complexity of such messages be minimized. This is typically achieved by using fixed length fields and as few options as possible. This simplicity contradicts the need for efficient compression in order to minimize the bit rate overhead associated with SSEI messages. The coding efficiency is particularly important if the granularity of SSEI message transmission is very high (and as noted above it can even be at the level of patches).

[0033] More effective representation of SSEI information from the compression point of view has been proposed. (See, e.g., I. Amonou, N. Cammas, S. Kervadec and S. Pateux, "Improved Signaling of Scalability Information," Joint Video Team, Doc. JVT-R068, Nice, Bangkok, January 2006, which is incorporated herein in whole on a reference basis.) However, the proposed representations have significant computational complexity. For example, the above-mentioned doc. JVT-R068 offers a representation that shows a reduction of the overhead by about 50% when a large number of scalability layers are used. The proposed representation is based on the approach to encoding different layers based on trees, where each node level corresponds to one of the three dimensions of scalability (spatial, temporal and qualitative, in this order). Presentation points correspond to the leaves of the tree. While, SVC JD5 syntax is based on sequential analysis of presentation points, encoded in any order. This is similar to the analysis of only the leaves of the tree structure mentioned in doc. JVT-R068, with the disadvantage that some information is unnecessarily duplicated from one layer to the next. In a tree-based approach, it is not necessary for lists with a common ancestor to duplicate information represented by their common parent (parents). However, this unnecessary duplication can be avoided by an alternative coding approach that uses the sensible use of option flags when creating SSEIs. For example, when only temporal_level changes between one layer and the next, it is practical to directly conclude

In the case of both approaches mentioned above (i.e., JVT-R068 and SVC JD5 syntax with an alternative interpretation of the option flag coding), it should be noted that the compromise "compilation-compression" is not ideal. In contrast to these approaches, the various embodiments of the present invention employ techniques based on differential encoding of inter-layer information, such that the layers can be freely added, replaced or removed. In contrast to the approaches mentioned above, various embodiments of the techniques of the present invention simultaneously reduce the bit rate overhead and design complexity to the minimum levels.

PZ / 4046 / JDN 8 ΕΡ 2 005 607 B1 [0035] The technique of the present invention for transmitting the necessary SSEI information may be understood with reference to its application to an example in which the initial SSEI is assumed and maintained as status information. on the server, gateway or client in the communication system. The needed bits to delete a layer or subset of layers from the SSEI state can be very small, as only enough "difference" information is needed to identify the layers on which the SSEI information structure will be truncated. In other words, the indicated layer, as well as all layers dependent on it, can be eliminated from the state information after the layer has been identified. Similarly, to add a new layer or a new set of layers to the SSEI state is enough, to send only differential information about the layer (or layers) that will be added. This approach intentionally eliminates the entire overhead associated with multiple encoding of layer structure information between one SSEI instance and another.

[0036] The reduction in the overall bit rate associated with encoding SSEI information due to such differential encoding allows the use of simple encoding structures for required messages, thus their media unconscious gateway analysis (at least at the video coding level (VLC) in H.264 terminology) ) is simplified. It should be noted that the techniques of the present invention make certain assumptions about how SSEI messages will be transported or, more importantly, whether and how they are transported reliably. All you need to do is ensure consistency of status if the same message is received more than once.

[0037] SSEI messages may be supplemented with information about the structure of the layers in any convenient way. In the embodiment, A SSEI is supplemented by three additional messages about control of the SEI scalability information corresponding to the operations of adding, replacing or deleting one or more layers. These operations are applied incrementally; in other words, the assumed input state before applying the indicated operation is the one that is in effect immediately before the operation. In an alternative example solution (exemplary solution B), the SSEI information is supplemented by two additional messages about the control of the SEI scalability information indicating respectively which layers are not present and which layers have changed the information about the decoding dependencies. On the contrary, in contrast to acting in an increasing manner in embodiment A, the operations in exemplary solution B are not made in a cumulative manner, but instead assume that the input state for applying the indicated operation is determined by the latest SSEI message. Exemplary solution B may be preferred in environments in which SEI messages may be lost. In this case, the initial SSEI may be forwarded to the receiver or media gateway during session establishment using reliable transport mechanisms. Loss of any emerging message about the control of scalability information in example solution B will not disrupt information on the state of the receiver or media gateway of the example solution B, as if it caused it in embodiment A. the operations in example solution B are not made in an incremental manner, but instead assume that the input state for applying the indicated operation is determined by the latest SSEI message. Exemplary solution B may be preferred in environments in which SEI messages may be lost. In this case, the initial SSEI may be forwarded to the receiver or media gateway during session establishment using reliable transport mechanisms. Loss of any emerging message about the control of scalability information in example solution B will not disrupt information on the state of the receiver or media gateway of the example solution B, as if it caused it in embodiment A. the operations in example solution B are not made in an incremental manner, but instead assume that the input state for applying the indicated operation is determined by the latest SSEI message. Exemplary solution B may be preferred in environments in which SEI messages may be lost. In this case, the initial SSEI may be forwarded to the receiver or media gateway during session establishment using reliable transport mechanisms. Loss of any emerging message about the control of scalability information in example solution B will not disrupt information on the state of the receiver or media gateway of the example solution B, as if it caused it in embodiment A. that the input state for applying the indicated operation is determined by the latest SSEI message. Exemplary solution B may be preferred in environments in which SEI messages may be lost. In this case, the initial SSEI may be forwarded to the receiver or media gateway during session establishment using reliable transport mechanisms. Loss of any emerging message about the control of scalability information in example solution B will not disrupt information on the state of the receiver or media gateway of the example solution B, as if it caused it in embodiment A. that the input state for applying the indicated operation is determined by the latest SSEI message. Exemplary solution B may be preferred in environments in which SEI messages may be lost. In this case, the initial SSEI may be forwarded to the receiver or media gateway during session establishment using reliable transport mechanisms. Loss of any emerging message about the control of scalability information in example solution B will not disrupt information on the state of the receiver or media gateway of the example solution B, as if it caused it in embodiment A. In this case, the initial SSEI may be forwarded to the receiver or media gateway during session establishment using reliable transport mechanisms. Loss of any emerging message about the control of scalability information in example solution B will not disrupt information on the state of the receiver or media gateway of the example solution B, as if it caused it in embodiment A. In this case, the initial SSEI may be forwarded to the receiver or media gateway during session establishment using reliable transport mechanisms. Loss of any emerging message about the control of scalability information in example solution B will not disrupt information on the state of the receiver or media gateway of the example solution B, as if it caused it in embodiment A.

[0038] Embodiment A according to the present invention introduces three new types of SEI payload in Annex D (section D.1) SVC JD5:

scalability_info_add (payload type 26) that adds one or more layers to the current SSEI information structure;

scalability_info_replace (payload type 27) that converts one or more layers from the current SSEI information structure; and

PZ / 4046 / JDN 9 ΕΡ 2 005 607 BI scalability_info_delete (payload type 28) that removes one or more layers from the current SSEI information structure.

[0039] The syntax for the "add" and "replace" versions is identical to the scalability_info (SSEI) structure. Only the meaning of the packaged data changes and only in terms of how the management of the SSEI state is performed (in other words, the interpretation of the meaning of individual fields remains the same). The syntax for the "delete" version can be simple, because only the identification of the layers to be deleted is necessary. It should be noted that the "replace" version is most useful for handling changes in layer dependency information. Due to the fact that there is no need to encode any additional information, the size of such a message may be extremely small (e.g., 30-40 bits, depending on the number of layers).

[0040] Herein, the use of the phrase "SSEI information" refers to any of these four types of scalability information, unless explicitly indicated otherwise.

[0041] Because the current SVC JD5 specification sanctions the use of SSEI only for IDR access units (section F.10.2.1), SSEI information in various embodiments of the present invention may occur at the beginning of any access entity.

[0042] When the parameter SPS (Sequence Parameter Set in SVC) is set to 0, the values ​​dependency_id, temporal_level and quality_level (hereinafter referred to as DTQ values) for all layers are specified in the SPS (section F.7.3.2). Assigning a given NAL to these values ​​is done using simple_priority_id if the NAL extension_flag syntax element is 0. If extension_flag is 1 then the simple_priority_id field is ignored according to the current specification and the NAL header values ​​for DTQ parameters are used. Note that if profile_idc is different than 83 then all NAL units type 20 and 21 must be set to extension_flag (section F.7.4.2).

[0043] If we assume that at the SPS level the nal_extension_flag flag is set to 0, then all NAL SVC headers will only use simple_priority_id, not the full DTQ set. In order to allow the NAL stream server or gateway, both messages must be analyzed. SSEI provides detailed information about layers, while SPS assigns a specific layer to the priority id identification values. There seems to be no convincing reason to include this information in the SPS, and there may be many reasons not to do so. When analyzing the remaining information contained in the SPS, it should be noted that none of them relates to transport issues. Thus, according to the present invention, a corresponding SPS portion can be removed and its coding transferred to the SSEI.

[0044] The use of nal_extension_flag should preferably take place in the entire stream, in other words simultaneous occurrence with NALs, with or without an extended NAL heading, should not be allowed. Although, theoretically, the inclusion of such functionality could create the appearance of providing more flexibility, in practice this functionality complicates the operation of servers and gateways without real benefits. Accordingly, the nal_extension_flag encoding according to various embodiments of the present invention is not carried out separately for each layer, but for SSEI. This is in accordance with SVC JD5, where coding is carried out for each SPS.

[0045] Along with the simple_prority_id coding performed at the SSEI level, there is no more need to provide optional DTQ value encoding and therefore the corresponding SSEI (decoding_ dependency_info_present_flag) flags can be removed.

[0046] As previously noted, SVC is an extension by scalable video coding (attachment F) of the video coding standard H.264 AVC. The AVC compatible layer implies the use of NAL unit headers without simple_priority_id or DTQ field extensions. The SVC JD5 specification assumes values ​​equal to 0. In various embodiments of the present invention, simple_priority_id values ​​or DTQ field extensions are signaled at the SSEI level.

[0047] TABLE I illustrates exemplary syntax changes that are made in section F.7.3.2, Sequence Parameter Set (SPS) SVC extension syntax, JD5 SVC specification according to the implementation principles of the present invention. The changes include removing the syntax elements associated with priority id and DTQ.

TABLE I.

<td colspan="3">SPS SVC EXTENSIONS MODIFICATIONS</td>

<td>seq_parameter_set_svc_extension () {</td><td>C</td><td>descriptor</td>

<td>extended_spatial_scalability</td><td>0</td><td>u (2)</td>

<td>If (extended_spatial_scalability&gt; 0) {</td><td></td><td></td>

<td>if (chroma_format_idc&gt; 0) {</td><td></td><td></td>

<td>chroma_phase_x_plus1</td><td>0</td><td>u (2)</td>

<td>chroma_phase_y_plus1</td><td>0</td><td>u (2)</td>

<td>}</td><td></td><td></td>

<td>if (extended_spatial_scalability = = 1) {</td><td></td><td></td>

<td>scaled_base_left_offset</td><td>0</td><td>se (v)</td>

<td>scaled_base_top_offset</td><td>0</td><td>se (v)</td>

<td>scaled_base_right_offset</td><td>0</td><td>se (v)</td>

<td>scaled_base_bottom_offset</td><td>0</td><td>se (v)</td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

[0048] TABLE II illustrates exemplary syntax changes (additions) which are made in Annex D, D.1, SEI load type syntax, according to the implementation principles of the present invention. Three new load types (26-28) are added that correspond to add, replace and delete operations.

TABLE II

<td colspan="3">TYPES OF SEI CARGO</td>

<td>sei_payload (payloadType, payloadSize) {</td><td>C</td><td>descriptor</td>

<td>if (payloadType = = 0)</td><td></td><td></td>

<td>Buffering_period (payloadSize)</td><td>5</td><td></td>

PZ / 4046 / JDN 11 EP 2 005 607 B1

<td colspan="3">TYPES OF SEI CARGO</td>

<td>else if (payloadType = = 1)</td><td></td><td></td>

<td>pic_timing (payloadSize)</td><td></td><td></td>

<td>e.t.c.</td><td></td><td></td>

<td>else if (payloadType = 22)</td><td></td><td></td>

<td>scalabilty_info (payloadSize) / * specified in Annex F * /</td><td>5</td><td></td>

<td>e.t.c.</td><td></td><td></td>

<td>else if (payloadType = 26)</td><td></td><td></td>

<td>scalability_info_add (payloadSize) / * specified in Annex F * /</td><td>5</td><td></td>

<td>else if (payloadType = 27)</td><td></td><td></td>

<td>scalability_info_replace (payloadSize) / * specified in Annex F * /</td><td>5</td><td></td>

<td>else if (payloadType = 28)</td><td></td><td></td>

<td>scalability_info_delete (payloadSize) / * specified in Annex F * /</td><td>5</td><td></td>

<td>else</td><td></td><td></td>

<td>reserved_sei_message (payloadSize)</td><td>5</td><td></td>

<td>if (! byte_aligned ()) {</td><td></td><td></td>

<td>bit_equal_to_one / * equals 1 * /</td><td>5</td><td>f (1)</td>

<td>while (! byte_aligned ())</td><td></td><td></td>

<td>bit_equal_to_zero / * equal to 0 * /</td><td>5</td><td>f (1)</td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

[0049] TABLE III shows exemplary syntax changes that take place in section F.10.1.1, message syntax with information about SCI scalability, according to the principles of an embodiment of the present invention. We notice the introduction of the nal_unit_extension_flag tag (moved from SPS) and the introduction of avc_compatible_flag. In addition, note that DTQ encoding is always present for non-AVC layers and that assignment of priority_id to DTQ is established if nal_unit_extension_flag is not set (i.e. when the NAL units do not carry explicit DTQ information).

TABLE III

<td colspan="3">SYNTHESIS OF SEI MESSAGES WITH INFORMATION ON SCALABILITY</td>

<td>scalability_info (payloadSize) {</td><td>C</td><td>descriptor</td>

<td>num_layers_minus1</td><td>5</td><td>ue (v)</td>

<td>nal_unit_extension_flag</td><td>5</td><td>u (1)</td>

VP / 4046 / JDN

EP 2 005 607 B1

<td colspan="3">SYNTHESIS OF SEI MESSAGES WITH INFORMATION ON SCALABILITY</td>

<td>for (i = 0; i &lt;= num_layers_minus1; i ++) {</td><td></td><td></td>

<td>layer_id [i]</td><td>5</td><td>u (8)</td>

<td>avc_compatible_flag [i]</td><td>5</td><td>u (1)</td>

<td>fgs _ layer _flag [i]</td><td>5</td><td>u (1)</td>

<td>sub_pic_layer_flag [i]</td><td>5</td><td>u (1)</td>

<td>sub_region_layer_flag [i]</td><td>5</td><td>u (1)</td>

<td>profile_level_info_present_flag [i]</td><td>5</td><td>u (1)</td>

<td>bitrate_info_present_flag [i]</td><td>5</td><td>u (1)</td>

<td>frm _ rate_ and nfo _ p rese nt _flag [i]</td><td>5</td><td>u (1)</td>

<td>frm_size_info_present_flag [i]</td><td>5</td><td>u (1)</td>

<td>if (! avc_compatible_flag [i]) {</td><td></td><td></td>

<td>laye r_ dependency_ and nfo_ present_flag [i]</td><td>5</td><td>u (1)</td>

<td>}</td><td></td><td></td>

<td>init_ pa ramete r_sets_i nfo_p rese nt_flag [i]</td><td>5</td><td>u (1)</td>

<td>if (profile_level_info_present_flag [i]) {</td><td></td><td></td>

<td>layer_ profile _ idc [i]</td><td>5</td><td>u (8)</td>

<td>layer_constraint_set0_flag [i]</td><td>5</td><td>u (1)</td>

<td>layer_constraint_set1_flag [i]</td><td>5</td><td>u (1)</td>

<td>layer_constraint_set2_flag [i]</td><td>5</td><td>u (1)</td>

<td>layer_constraint_set3_flag [i]</td><td>5</td><td>u (1)</td>

<td>reserved_zero_4bits / * equal to 0 * /</td><td>5</td><td>u (4)</td>

<td>layer_level_ idc [i]</td><td>5</td><td>u (8)</td>

<td>}</td><td></td><td></td>

<td>if (! avc_compatible_flag [i]) {</td><td></td><td></td>

<td>if (! nal_unit_extension_flag) {</td><td></td><td></td>

<td>priority_id [i]</td><td>5</td><td>in (6)</td>

<td>}</td><td></td><td></td>

<td>temporal_level [i]</td><td>5</td><td>u (3)</td>

<td>dependency_id [i]</td><td>5</td><td>u (3)</td>

<td>quality_level [i]</td><td>5</td><td>u (2)</td>

VP / 4046 / JDN

EP 2 005 607 B1

<td colspan="3">SYNTHESIS OF SEI MESSAGES WITH INFORMATION ON SCALABILITY</td>

<td>}</td><td></td><td></td>

<td>if (bitrate_info_present_flag [i]) {</td><td></td><td></td>

<td>avg _ bitrate [i]</td><td>5</td><td>u (16)</td>

<td>max_ bitrate [i]</td><td>5</td><td>u (16)</td>

<td>}</td><td></td><td></td>

<td>if (frm_rate_info_present_flag [i]) {</td><td></td><td></td>

<td>constant_frm_rate_idc [i]</td><td>5</td><td>u (2)</td>

<td>avg _frm _ rate [i]</td><td>5</td><td>u (16)</td>

<td>}</td><td></td><td></td>

<td>if (frm_size_info_present_flag [i]) {</td><td></td><td></td>

<td>frm_width_in_mbs_minus1 [i]</td><td>5</td><td>ue (v)</td>

<td>frm_height_in_mbs_minus1 [i]</td><td>5</td><td>ue (v)</td>

<td>}</td><td></td><td></td>

<td>if (sub_region_layer_f'lag [i]) {</td><td></td><td></td>

<td>base _ region layer_id [i]</td><td>5</td><td>u (8)</td>

<td>dynamic_rect_flag [i]</td><td>5</td><td>u (1)</td>

<td>if (dynamic_rect_f'lag [i]) {</td><td></td><td></td>

<td>horizontal_offset [i]</td><td>5</td><td>u (16)</td>

<td>vertical_offset [i]</td><td>5</td><td>u (16)</td>

<td>region _width [i]</td><td>5</td><td>u (16)</td>

<td>region _ height [i]</td><td>5</td><td>u (16)</td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

<td>if (sub_pic_layer_flag [i])</td><td></td><td></td>

<td>roi_id [i]</td><td>5</td><td>u (3)</td>

<td>if (layer_dependency_info_present_flag [i]) {</td><td></td><td></td>

<td>num _ directly _ dependent_layers [i]</td><td>5</td><td>ue (v)</td>

<td>for (j = 0; j &lt;num_directly_dependent_layers [i]; j ++)</td><td></td><td></td>

<td>directly_dependent_layer_id_delta [i] [j]</td><td>5</td><td>ue (v)</td>

<td>}</td><td></td><td></td>

VP / 4046 / JDN

EP 2 005 607 B1

<td colspan="3">SYNTHESIS OF SEI MESSAGES WITH INFORMATION ON SCALABILITY</td>

<td>if (init_parameter_sets_info_present_flag [i]) {</td><td></td><td></td>

<td>num_i n it_se q_parameter r_set_m inus1 [i]</td><td>5</td><td>ue (v)</td>

<td>for (j = 0; j &lt;= num_seq_parameter_set_minus1 [i]; j ++)</td><td></td><td></td>

<td>init_seq_parameter_set_id_delta [i] [j]</td><td>5</td><td>ue (v)</td>

<td>num _init_pic _ parameter _ set _ minus1 [i]</td><td>5</td><td>ue (v)</td>

<td>for (j = 0; j &lt;= num_pic_parameter_set_minus1 [i]; j ++)</td><td></td><td></td>

<td>init_pic_parameter_ set_id_delta [i] [j]</td><td>5</td><td>ue (v)</td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

[0050] Corresponding changes are also made to the semantic interpretation of SSEI message syntax elements (section F.10.2.1 of the SVC JD5 specification, SEI message semantics with information on scalability). In particular, the restriction on the position of IDRs is lifted. Furthermore, when present, the SSEI message will affect all NAL units that follow it and overwrite any and all previous scalability information. The importance of the message is current to the next message of the type scalability_FILE, scalability_info_add, scalability_info_replace or scalability_info_del. The meaning of nal_unit_extension_flag [i] is that a value of 0 specifies the relation dependency_id [i], temporal_level [i] and quality_id [i] with simple_priority_id [i] for the layer, whose layer_id [i] is contained in this SEI message with scalability information. nal_unit_extension_flag equal to 1 defines that the simple_priority_id [i] parameter is not present. When nal_unit_extension_flag is not present, by default it is equal to 1. The syntax element of NAL unit, extension_flag, all NAL units with nal_unit_type equal to 20 and 21, which are compatible with this SEI message with information about scalability will be equal to nal_unit_extension_flag. When profile_idc is different than 83 (identifying SVC), the extension_flag element of all NALs with nal_unit_type equal to 20 and 21, which refer to the current sequence parameter set (SPS), will be 1. Also, the value of nal_unit_extension_flag can not be different in any next messages scalability_info_add or scalability_info_replace, unless that it will be modified by the message scalability_info before. The meaning of avc_compatible_flag [i] is such that the value equal to 1 determines that the layer with layer_id [i] is compliant with the AVC specification. When avc_compatible_flag [i] equals 0, the current layer complies with the SVC specification. When avc_compatible_flag [i] equals 1, the NAL header does not carry extended header information. When avc_compatible_flag is equal to 1, all dependency_id [i], temporal_level [i] and quality_level [i] values ​​for the current i layer are assumed to be equal to 0. The significance of priority_id [i] is such that it specifies the value of simple_priority_id to be used in while setting values ​​for the syntax dependency_id, temporal_level and quality_level (as specified in section F.7.4.1 of the SVC JD5 specification). For all priority_id [i] values,

PZ / 4046 / JDN 15 ΕΡ 2 005 607 Bl temporal_level [i] and quality_level_list [i] are not present, dependency_id [i], temporal_level [i] and quality_level [i] will be 0. Finally, temporal_level [i], dependency_id [i] and quality_level [i] are equal respectively: temporal_level, dependency_id and quality_level, NAL units in a scalable layer with a layer identifier equal to i. If temporal_level [i], dependency_id [i] and quality_level [i] are not present, it is assumed they are equal to 0.

[0051] The syntax of the new SEI message "scalability_info_add" is identical to the syntax of the "scalability_info" SEI message. In terms of its meaning, when present, this SEI message will be effective for all subsequent NAL units. It adds the number of layers to the currently valid scalability information. If the layer_id of the layer described in this message is already present in the scalability information maintained by the receiver, then all information for this layer is deleted and the information contained in this message is used instead. The status of the scalability information applicable after the use of such message remains valid until the next scalable_Info SEIA message, scalability_info_add, scalability_info_replace or scalability_info_del. The meaning of all fields is the same as in scalability_info.

[0052] The syntax of the new SEI message "scalability_info_replace" is identical to the syntax of the "scalability_info" SEI message. In terms of its meaning, when present, this SEI message will be effective for all subsequent NAL units. It replaces information for the number of layers specified by their layer_id in the currently valid scalability information. Only the information contained in the message is replaced; any previous settings that are not modified by the message remain unchanged. The status of the scalability information applicable after applying this message remains valid until the next SEI message, scalability_info, scalability_info_add, scalability_info_replace or scalability_info_del. The meaning of all fields is the same as in scalability_info.

[0053] TABLE IV illustrates an exemplary syntax for a new SEI message "scalability_info_del" according to the principles of an embodiment of the present invention. As shown in the table, the message consists of a list of layer identifiers (layer_id) to be considered removed from the bit stream. When present, this SEI message will be effective for all subsequent NAL units. It deletes information for layers specified by their layer_id from the currently valid scalability information. The status of the scalability information applicable after applying this message remains valid until the next SEI message, scalability_info, scalability_info_add, scalability_info_replace or scalability_info_del. The meaning of all fields is identical to the corresponding fields in scalability_info.

TABLE IV

<td colspan="3">SYNTHES SEI MESSAGE REMOVING INFORMATION ABOUT SCALABILITY</td>

<td>scalability_info_del (payloadSize) {</td><td>C</td><td>descriptor</td>

<td>num_layers_minus1</td><td>5</td><td>ue (v)</td>

<td>for (i = 0; i &lt;= num_layers_minus1; i ++) {</td><td></td><td></td>

<td>layer_id [i]</td><td>5</td><td>u (8)</td>

<td>}</td><td></td><td></td>

PZ / 4046 / JDN 16 EP 2 005 607 B1

<td colspan="3">SYNTHES SEI MESSAGE REMOVING INFORMATION ABOUT SCALABILITY</td>

<td>}</td><td></td><td></td>

[0054] FIG. 2 is a flow diagram of an exemplary state processing algorithm 200 in embodiment A. The initial state of the layer structure (state) of the receiver is empty. In step 22, the receiver obtains SEI messages with information on scalability, bit stream or alternative methods such as signaling. The obtained SEI message scalability_info can act as an initializing SSEI message and is used to initialize the State. A similarly received SEI message, "add", "delete" and "replace" may, respectively, add to, delete from or modify the current state for one or more layers. It should be noted that each of these messages may concern multiple layers. However, for the sake of clarity, the diagram in FIG. 2 shows the effect for a single layer (layer (i)).

[0055] In some communication systems or communication scenarios, forwarding messages with SSEI information may never take place in the band together with video data, but always be transmitted during session establishment or other reliable means of communication. Due to the fact that video data can be transported through channels with non-zero packet losses, it may be desirable to ensure that such critical information is not at risk of loss.

[0056] For such communication systems or communication scenarios, in alternative embodiment B of the present invention, scalability information may be determined by the scalability_ILfo SEI message made available during the SDP negotiation or present in the IDR access unit (e.g., in a file format). The information in such SEI message scalabilityjnfo establishes information about scalability to the end of the sequence or to receive another SEI message, scalability or renegotiation in the case of SDP. Information on scalability can not change from this point, except for situations resulting from internal dependencies between layers. Furthermore, the difference in exemplary solution B relative to embodiment A relates to a method for handling state management. In embodiment A, state changes are introduced in a cumulative manner, which has the disadvantage that the loss of a single message can damage the status information. In alternative embodiment B, state changes are always entered relative to the latest SEI message scalability and the loss effects do not increase.

[0057] In exemplary embodiment B, the SSEI "add" message is not used, while the SSEI "delete" messages are called "layers_notpresent". The SSEI "layers_notpresent" message indicates that the given layer and all dependent layers will not be present in the bit stream starting with the access unit in which the SEI messages are contained. An indication of which layers are not present is always carried out in relation to the currently active SEI message scalability_info. It should be noted that a convenient mechanism to indicate that a given layer was added after it was removed consists in sending the "layers_notpresent" message, which removes all currently missing layers, except for the layer that is added. As an illustrative example, let's assume that the bit stream has 3 layers, numbers 0, 1 and 2. In addition, let's assume that after the initial transmission of all 3 layers, the media server sends the message "layers_notpresent" indicating that layers 1 and 2 are not present. In this case, layer 1 can be added later by sending a "layers_notpresent" message indicating that only layer 2 is not present. This, in the result,

PZ / 4046 / JDN indicates that layer 1 has been added. The media server (or media gateway) can start transmitting layer 1 data immediately after sending this SEI message "layers_notpresent".

[0058] In exemplary embodiment B, the SSEI message "replace" of the first embodiment is changed to the "dependency_change" message. The SEI message about the change in the dependency of scalability information indicates that the dependencies between layers for the given layer_id have changed and provide new information about the dependencies between the layers. The syntax for SSEI messages with dependency change does not allow any other change of information related to the scalability information.

[0059] TABLE V shows an exemplary syntax for the new alternative payload types SEI, scalability_info_layers_notpresent (No. 26) and scalability_info_dependency_change (No. 27), replacing the syntax from section D.1 of the SVC JD5 specification. Unlike embodiment A, in exemplary embodiment B, SEI messages with scalability information may be present only in IDR images. The meaning of the message, with the exception of the information on the dependencies between the layers, remains valid until the next SEI message of the same type.

TABLE V

<td colspan="3">ALTERNATIVE TYPES OF CARGO</td>

<td>sei_payload (payloadType, payloadSize) {</td><td>C</td><td>descriptor</td>

<td>if (payloadType = = 0)</td><td></td><td></td>

<td>Buffering_period (payloadSize)</td><td>5</td><td></td>

<td>else if (payloadType = = 1)</td><td></td><td></td>

<td>e.t.c.</td><td>5</td><td></td>

<td>else if (payloadType = 22)</td><td></td><td></td>

<td>scalabilty_info (payloadSize) / * specified in Annex F * /</td><td>5</td><td></td>

<td>e.t.c.</td><td></td><td></td>

<td>else if (payloadType = 26)</td><td></td><td></td>

<td>scalability_info_layers_notpresent (payloadSize) / * specified in Annex F * /</td><td>5</td><td></td>

<td>else if (payloadType = 27)</td><td></td><td></td>

<td>scalability_info_dependency_change (payloadSize) / * specified in Annex F * /</td><td>5</td><td></td>

<td>else</td><td></td><td></td>

<td>reserved_sei_message (payloadSize)</td><td>5</td><td></td>

<td>if (! byte_aligned ()) {</td><td></td><td></td>

<td>bit_equal_to_one / * equals 1 * /</td><td>5</td><td>f (1)</td>

<td>while (! byte_aligned ())</td><td></td><td></td>

<td>bit_equal_to_zero / * equal to 0 * /</td><td>5</td><td>f (1)</td>

<td>}</td><td></td><td></td>

VP / 4046 / JDN

EP 2 005 607 B1

<td colspan="3">ALTERNATIVE TYPES OF CARGO</td>

<td>}</td><td></td><td></td>

[0060] Specific layers may be marked as inactive (absent in the bitstream) by the SEI message scalability_info_layers_notpresent. An example of the syntax for this message is shown in TABLE VI.

TABLE VI

<td colspan="3">SEI MESSAGE WITH THE LAYERS-NOT-PRESENT FOR INFORMATION ON SCALABILITY</td>

<td>scalability_info_layers_notpresent (payloadSize) {</td><td>C</td><td>descriptor</td>

<td>num_layers</td><td>5</td><td>ue (v)</td>

<td>for (i = 0; i &lt;num_layers; i ++) {</td><td></td><td></td>

<td>layer_id [i]</td><td>5</td><td>u (8)</td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

[0061] When present, this SEI message will have effect for all subsequent NAL units. The message indicates that the layer data specified by their layer_id will not be present in the bitstream. The status of the scalability information applicable after applying this message remains valid until the next scalable_info SEI or scalability_info_layers_notpresent message. The num_layers value indicates the number of scalable layers or presentation points that will not be present in the bit stream after decoding this SEI message. The num_layers value is between 0 and 255, inclusive. When num_layers is 0, this indicates that all layers are present. The value of layer_id [i] indicates the identifier of the scalable layer, which is marked as absent in the bit stream. The message simply contains a list of layers, which are marked as absent. [0062] TABLE VII provides an example syntax for a SEI message, scalability_info_dependency_change. When present, this SEI message will be effective for all subsequent NAL units. Information about dependencies between layers for a given layer consists of information resulting from decoding elements synt_ num_directly_dependent_layer, directly_dependent_layer_id_delta_minus1 or alternatively a new element of the layer_dependency_info_src_layer_id_delta_minus1 syntax. This new element allows the system to identify a layer identifier for another layer that has the same information about inter-layer dependencies as the current layer (by encoding the difference of the layer identifiers minus 1). TABLE VII [0062] TABLE VII provides an example syntax for a SEI message, scalability_info_dependency_change. When present, this SEI message will be effective for all subsequent NAL units. Information about dependencies between layers for a given layer consists of information resulting from decoding elements synt_ num_directly_dependent_layer, directly_dependent_layer_id_delta_minus1 or alternatively a new element of the layer_dependency_info_src_layer_id_delta_minus1 syntax. This new element allows the system to identify a layer identifier for another layer that has the same information about inter-layer dependencies as the current layer (by encoding the difference of the layer identifiers minus 1). TABLE VII [0062] TABLE VII provides an example syntax for a SEI message, scalability_info_dependency_change. When present, this SEI message will be effective for all subsequent NAL units. Information about dependencies between layers for a given layer consists of information resulting from decoding elements synt_ num_directly_dependent_layer, directly_dependent_layer_id_delta_minus1 or alternatively a new element of the layer_dependency_info_src_layer_id_delta_minus1 syntax. This new element allows the system to identify a layer identifier for another layer that has the same information about inter-layer dependencies as the current layer (by encoding the difference of the layer identifiers minus 1). TABLE VII this SEI message will be effective for all subsequent NAL units. Information about dependencies between layers for a given layer consists of information resulting from decoding elements synt_ num_directly_dependent_layer, directly_dependent_layer_id_delta_minus1 or alternatively a new element of the layer_dependency_info_src_layer_id_delta_minus1 syntax. This new element allows the system to identify a layer identifier for another layer that has the same information about inter-layer dependencies as the current layer (by encoding the difference of the layer identifiers minus 1). TABLE VII this SEI message will be effective for all subsequent NAL units. Information about dependencies between layers for a given layer consists of information resulting from decoding elements synt_ num_directly_dependent_layer, directly_dependent_layer_id_delta_minus1 or alternatively a new element of the layer_dependency_info_src_layer_id_delta_minus1 syntax. This new element allows the system to identify a layer identifier for another layer that has the same information about inter-layer dependencies as the current layer (by encoding the difference of the layer identifiers minus 1). TABLE VII directly_dependent_layer_id_delta_minus1 or alternatively the new element layer_dependency_info_src_layer_id_delta_minus1. This new element allows the system to identify a layer identifier for another layer that has the same information about inter-layer dependencies as the current layer (by encoding the difference of the layer identifiers minus 1). TABLE VII directly_dependent_layer_id_delta_minus1 or alternatively the new element layer_dependency_info_src_layer_id_delta_minus1. This new element allows the system to identify a layer identifier for another layer that has the same information about inter-layer dependencies as the current layer (by encoding the difference of the layer identifiers minus 1). TABLE VII

<td colspan="3">SEI MESSAGES ABOUT CHANGING RELATIONSHIP WITH SCALABILITY INFORMATION</td>

<td>scalability_info_dependency_change (payloadSize) {</td><td>C</td><td>descriptor</td>

<td>num_layers_minus1</td><td>5</td><td>ue (v)</td>

<td>for (i = 0; i &lt;= num_layers_minus1; i ++) {</td><td></td><td></td>

PZ / 4046 / JDN 19 EP 2 005 607 B1

<td colspan="3">SEI MESSAGES ABOUT CHANGING RELATIONSHIP WITH SCALABILITY INFORMATION</td>

<td>layer_id [i]</td><td>5</td><td>u (8)</td>

<td>layer_dependency_info_present_flag [i]</td><td>5</td><td>u (1)</td>

<td>if (layer_depdendency_info_present_flag [i]) {</td><td></td><td></td>

<td>num _ directly _ dependent_layers [i]</td><td>5</td><td>ue (v)</td>

<td>for (j = 0; j &lt;num_directly_dependent_layei ^ s [i]: j ++)</td><td></td><td></td>

<td>directly _ dependent_ layer_ id _ delta_ minus1 [i] [j]</td><td>5</td><td>ue (v)</td>

<td>} else {</td><td></td><td></td>

<td>layer_ dependency_ info _ src _ layer_ id _ delta_ minus1 [i]</td><td>5</td><td>ue (v)</td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

<td>}</td><td></td><td></td>

[0063] The SEI message, scalability_info_dependency_change, indicates that the inter-layer dependency information for a particular layer identified by its layer_id will change according to the content of the message. The status of the scalability information applicable after applying this message remains valid until the next scalable_info SEI or scalability_info_dependency_change message.

[0064] num_layers_minus1 plus 1 indicates the number of scalable layers or presentation points for which information about the relationship between layers will change after decoding this SEI message. The value num_layers_minus1 is in the range 0 to 255, inclusive.

[0065] layer_id [i] indicates the identifier of the scalable layer for which the information on the relationship between the layers will change.

[0066] layer_dependency_info_present_flag [i] equal to 1 indicates the presence of interlayer dependency information for a scalable layer with layer identifier equal to layer_id [i] in the SEI message. The value 0 indicates that there is no presence of dependency information between the layers for a scalable layer with the layer identifier equal to and in the SEI message, and is identical to the layer indicated by layer_dependency_info_src_layer_id_delta_minus1 [i].

[0067] num_directly_dependent_layers [i] indicates the number of scalable layers on which the scalable layer with the layer identifier equal to layer_id [i] depends directly. For example, layer M depends directly on layer Q if there is at least one encoded image in layer M that uses prediction between layers from substrate Q. The value num_directly_dependent_layers is in the range from 0 to 255, inclusive.

[0068] dircctly_epeeneent_lyeer_id_eelta_miuus1 [i] [j] returns the difference between layer-ddn] and the identifier of the j-th scalable layer from which the scalable layer with the layer identifier equal to layer_id [i] depends directly and i. The identifier of the scalable direct dependency layer is equal ( layer_id [i] - directly_dependent_layer_id_delta_minus1 - 1).

PZ / 4046 / JDN 20 ΕΡ 2 005 607 Bl [0069] layer_dependency_info_src_layer_id_delta_minus1 [i] defines that layer with layer identifier equal to layer_id [i] has the same information about interdependencies between layers as layer with layer identifier equal (layer_id [i] - layer_dependency_info_src_layer_id_delta_minus1 [i] - 1). [0070] FIG. 3 shows a flow diagram of an exemplary state processing algorithm 300 in an exemplary embodiment B. The initial state of the layer structure (state) of the receiver is empty. In step 33, the receiver obtains SEI messages with information on scalability, bit stream or alternative methods, such as signaling. The SEI message scalability_info acts as an initializing SSEI message and uses it to initialize the state. "Layers_notpresent" and "dependency_change" messages, respectively, means one or more layers as absent or modify the information about dependencies of their decoding. It should be noted that in this alternative example solution B, the layers are neither added nor removed by non-initializing messages. This ensures consistency of the state (with the possible exception of decoding dependencies), as long as initializing messages are reliably received. It should be noted that each of these messages may concern multiple layers. However, for the sake of clarity, the diagram in FIG. 3, like the diagram in FIG. 2, shows the effect for a single layer (layer (i)). This ensures consistency of the state (with the possible exception of decoding dependencies), as long as initializing messages are reliably received. It should be noted that each of these messages may concern multiple layers. However, for the sake of clarity, the diagram in FIG. 3, like the diagram in FIG. 2, shows the effect for a single layer (layer (i)). This ensures consistency of the state (with the possible exception of decoding dependencies), as long as initializing messages are reliably received. It should be noted that each of these messages may concern multiple layers. However, for the sake of clarity, the diagram in FIG. 3, like the diagram in FIG. 2, shows the effect for a single layer (layer (i)).

[0071] Although a preferred embodiment is described herein by the H.264 SVC standard, it will be apparent to those skilled in the art that the techniques described herein can be directly applied or extended to any coding structure that allows for many spatial / qualitative and temporal levels, for video or audio or for any other multimedia data.

[0072] It will also be understood that the systems and methods of the present invention may be implemented using any convenient combination of hardware and software. The software (ie instructions) for the implementation and operation of the above-mentioned systems and methods can be made available on a computer-readable medium that can be contained in, without limitation to, firmware, memory, data storage devices, microcontrollers, microprocessors, integrated circuits, ASICS , on-line download media and other available media.

[0073] According to a first example, a digital media decoder that is configured to process encoded media data includes:

scalably encoded media data;

data initializing scalability information; and changes that occur over time to scalarization initialization information, wherein data with scalability information is used to inform the decoder about the presence and structure of the dependencies between the layers contained in the encoded media data, and in which changes with time to scalable initialization information they are encoded in a differential form.

[0074] According to a second example, in the decoder of the first example, the scalable media data is coded according to H.264 SVC, the scalability information is initialized by the SEI SVC messages scalability_info and the changes in the scalability information are applied by the decoder cumulatively. [0075] According to a third example, in the decoder of the second example, changes to scalability information further include a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that point forward. [0076] According to a fourth example, in the decoder of the second example, changes to the scalability information further include an SEI message scalability_info_replace indicating that

The scalability information for at least one layer present in the bitstream is replaced by the scalability information included in said SEI scalability_info_replace from here to future, with each scalability information assumed to be for layers that are not explicitly encoded in the SEI message scalability_info_replace, it retains its previous value.

[0077] According to a fifth example, in the decoder of the second example, changes to the scalability information further include an SEI message scalability_info_del indicating that at least one layer is being removed from the bit stream from that moment into the future.

[0078] According to a sixth example, in the decoder of the second example, changes to the scalability information may appear in any random access unit of the encoded video signal.

[0079] According to a seventh example, in the decoder of the first example, the scalable media data is coded according to the SVC H.264, the scalability information is initialized by the SEI SVC messages scalability_info and the changes in the scalability information are applied by the decoder individually using the latest message SEI scalability_info as a reference.

[0080] According to an eighth example, in a set-top decoder of the seventh example, changes to scalability information further include a scalable_info_layers_notpresent SEI message indicating that one or more scalable layers indicated in the scalar initialization information are not present in the bit stream from then on to the future .

[0081] According to a ninth example, in a set-top decoder of the seventh example, changes to scalability information further include a message scalability_info_dependency_change SEI indicating that the inter-layer dependency information for at least one layer indicated in the latest scalable initiation information is replaced by the information about the scalability. the relationship between the layers contained in said SEI message scalability_info_dependency_change from that moment into the future.

[0082] According to a tenth example, in the decoder of the seventh example, changes in scalability information may appear in any random access unit of the encoded video signal.

[0083] The eleventh example includes a method for decoding digital media by decoders, the digital media comprising scalable data media, data initializing scalability information and changing data initializing scalability information with time encoded in a differential form, the method comprising:

the use of scalable information data to inform the decoder about the presence and structure of dependencies between the layers contained in the encoded media data.

[0084] According to a twelfth example, the method of the eleventh example in which scalable media data is coded according to SV.2 H.264, scalability information is initialized by SEI SVC messages scalability_info, and the method further comprises applying changes to scalability information by a decoder cumulatively .

[0085] According to a thirteenth example, a method of the twelfth example, wherein changes to scalability information further include a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that point forward. [0086] According to a fourteenth example, the method of the twelfth example, wherein changes to scalability information further include an SEI message scalability_info_replace indicating that the scalability information for at least one layer present in the bit stream is

PZ / 4046 / JDN is replaced by the scalability information included in said scalability_info_replace SEI from here to future, the method further comprising:

the assumption that any scalability information for a layer that is not explicitly encoded in the SEI message, scalability_info_replace, retains its previous value.

[0087] According to a fifteenth example, the method of the twelfth example, wherein changes to scalability information further include an SEI message scalability_info_del indicating that at least one layer is being removed from the bit stream from that moment into the future.

[0088] According to a sixteenth example, the method of the twelfth example, wherein changes to scalability information may appear in any access unit of the encoded video signal.

[0089] According to a seventeenth example, the method of the eleventh example wherein the scalable media data is coded according to H.264 SVC and the scalarization initiation information is SEI SVC message scalability_info messages, the method further comprising: applying the scalability information changes by the decoder individually and apply the latest SEI message scalability_info as a reference.

[0090] According to an eighteenth example, the method of the seventeenth example wherein the changes to scalability information further include a scalable_info_layers_notpresent SEI message indicating that one or more scalability layers indicated in the scalarization initiation information are not present in the bit stream from that point in time. future.

[0091] According to a nineteenth example, the method of the seventeenth example wherein the changes to scalability information further include a message scalability_info_dependency_change SEI indicating that information about the relationship between layers for at least one layer indicated in the latest scalable initiation information is replaced by information about the dependencies between the layers contained in said SEI message scalability_info_dependency_change from that moment into the future.

[0092] According to a twentieth example, the method of the seventeenth example wherein changes to the scalability information may appear in any access unit of the encoded video signal.

[0093] According to the twenty-first example, computer-readable items including a set of instructions on how to perform the steps listed in at least one of the previous examples.

VP / 4046 / JDN

EP 2 005 607 B1