System for broadcasting multimedia content

System for broadcasting multimedia content

Technical field

The present invention relates to a telecommunication system for broadcasting multimedia content to client devices. The invention also relates to a server used in this type of system. The invention further relates to a client device for requesting said multimedia content from such a server. The invention finally relates to a method used in the system.

The present invention is used, for example, in an application for broadcasting live multimedia content to a client via the Internet or a mobile network.

Background technique

The streaming of live audio, video, and various multimedia content on the Internet is becoming common. As shown in Figure 1, the streaming session involves: a real-time encoder for real-time encoding of live multimedia content and providing the encoded data stream; a broadcast connection between the encoder and the streaming server; and the server and several Several point-to-point connections between clients. The standard protocol used for this type of real-time transmission on IP networks is the real-time protocol (RTP). Therefore, the encoded data stream is converted into RTP packets, which are sent to the server and further sent to the client.

The problem caused by the RTP protocol is that RTP packets are often blocked by firewalls and network address translators (NAT). Therefore, the client cannot receive the requested multimedia content.

The solution to circumvent this problem is known from the publication "An RTPto HTTP video gateway" by Mathias Johanson (2001 ACM, 1-58113-348-0/01/0005). The solution is to convert the RTP packet into a file, which is included in the web page of the server, and send it to the client using the Hypertext Transfer Protocol (HTTP) instead of the RTP protocol. The advantage of the HTTP protocol is that it can be accepted by all firewalls and works well with NAT.

In this prior art, video content including image sequences is encoded into MJPEG (Mobile Joint Photographic Experts Group) streams. MJPEG is a standard format for encoding video, which consists in independently encoding each image of a sequence using the JPEG format developed for still images. The MJPEG stream is further converted into the RFC as in the certification request The RTP stream described in 2435. The RTP stream is sent to the web server via the RTP multicast connection. The web server includes conversion means for converting the RTP stream into a multi-purpose Internet mail extended multi-part (MIME multi-part) file. MIME multipart is a standard for specifying and describing the format of Internet message bodies, which makes it possible to display JPEG image sequences in HTML web pages. In Johanson's solution, the JPEG images of the MJPEG stream are stored in a part of a MIME multipart file. The MIME file can be accessed through its URL (Uniform Resource Locator, see authentication request number 1738) address on a web page available on the web server. When the client browses the web page and clicks on the URL address, a specific Java applet is downloaded and loaded on the client side for sorting and synchronizing the download of continuous JPEG images of MJPEG files. Once received by the client, the JPEG image is decoded by the JPEG decoder, and the next JPEG image is downloaded at the same time. Therefore, the MJPEG video sequence is played in real time.

The main disadvantage of this solution is that it is very special. The MIME multipart format only accepts still image encoding formats like JPEG or GIF. The MJPEG format encodes a video sequence into a collection of independent JPEG images, and therefore does not take advantage of the temporal redundancy of the video sequence. The MJPEG format does not achieve sufficient compression ratio to allow via a low bit rate network connection like the Internet or mobile network. Video streaming. The MJPEG format is of great interest for music studio composition, but it is not universal at all for video streaming on the Internet. In order to use another video encoding format like MPEG-4, conversion is required, which may cause severe quality degradation.

Another disadvantage of this method is that it does not work on other types of multimedia content, such as audio or text, except for video. It also does not give a solution for synchronizing several multimedia sources. For example, this type of method does not provide any solution for streaming movies via the Internet.

Summary of the invention

The purpose of the present invention is to propose a more effective solution for broadcasting video and even more generally various multimedia contents on the Internet via a server.

This is using the telecommunication system as defined in claims 1 to 4, the server as defined in claims 5 to 7, the client device as defined in claim 8, the method as defined in claims 9 and 10, as The computer program defined in claim 11 and the signal defined in claim 12 are implemented.

According to the present invention, multimedia content is encoded into an encoded data stream. The encoded data stream is distributed to the server in real time via broadcast transmission. On the IP network, the broadcast transmission between the encoder and the server is generally a multicast connection in accordance with the RTP (Real Time Protocol) protocol. Then, the server can convert the received encoded data stream into a "progressive" file, which has a format compatible with sequential downloads. The server can also make the sequential file available to the client, for example, on a web page. The equipment is available.

The sequence file is sent from the server to the client via a peer-to-peer network connection. On an IP network, the point-to-point network connection generally conforms to the HTTP protocol (hypertext transfer protocol, see RFC 2616). The HTTP protocol, which is the foundation of the World Wide Web, has great advantages that are accepted by all firewalls and NATs.

The sequential download of the file consists in starting to decode the file before it is completely downloaded. This makes it possible for the file format to have a structure in which media data and metadata are interleaved. The media data includes audio, video, image or text tracks of the encoded multimedia content. Metadata describes how the media data is encoded. As long as the fragments include media data and metadata related to the media data, decoding can be completed on the segments of the file by using the file format.

According to the present invention, the client may be expected to be connected to the server at any time during the broadcasting of the multimedia content, and request to receive the multimedia content on the fly. For this purpose, the server can customize the sequence file into a client sequence file suitable for the client's request. The client sequence file includes metadata for allowing the client to grasp the current multimedia broadcast, such as initialization metadata, which is usually sent before starting the broadcast, for example, to configure the player.

In the preferred embodiment of the present invention, the file format used is ISO file format version 2, which can be read by a large number of multimedia data encoding standards, such as MPEG-4 or H.263 for video or audio AMR (Advanced Multi-Rate).

The first advantage of the present invention is that these standards are commonly used for multimedia data compression. For example, MPEG-4 or equivalent standards are widely used by content providers on the Internet. Therefore, there is no need for a transcoding device on the server side, which is often required in Johanson's solution in order to transcode MPEG-4 streams into M-JPEG streams.

The second advantage is that the video coding standard achieves a much better compression ratio than MJPEG at any bit rate from extremely low to extremely high bit rates. When the client is a mobile phone or a personal computer with a modem Internet connection, this quality gain is particularly relevant and limited by the low bit rate network connection.

The ISO file format version 2 can also interleave multimedia data from different sources such as audio, video, image or text, and therefore can provide encoded data to the clients player, where synchronized audio, video, and The text is also available. In conjunction with multimedia standards such as MPEG-4 that have been specifically designed to handle multimedia sources, the ISO file format version 2 allows multimedia data to be transmitted to the client via a download server. Therefore, another advantage of the present invention is to propose a solution for broadcasting any kind of multimedia content (ie synchronized audio, video, text and image) instead of just video, which is more suitable for the current Internet application.

The system according to the present invention is also advantageous because it enables the client to save a copy of the received client's sequential file. The server can also use DRM (Data Resource Management) to limit the number of authorized copies. Because there is no prior right to write data into the Java applet of the client's file system, it is not easy to do this using Johanson's solution.

These and other aspects of the present invention will become apparent from the embodiments described below, and are clarified with reference to the embodiments described below.

Description of the drawings

The present invention will be further explained with reference to the accompanying drawings:-Figure 1 is a diagram illustrating a telecommunication system for streaming multimedia data via a real-time network connection,-Figure 2 is a diagram illustrating a system for streaming multimedia data via a first network connection according to the present invention, a server and The second network is connected to a block diagram of a telecommunications system for broadcasting multimedia content,-Figure 3 describes the structure of a file according to ISO file format version 2,-Figure 4 shows in a functional manner how customized devices can be set up suitable for client requests according to the present invention The client sequence file,-Figure 5 describes the structure of the client sequence file according to the present invention,-Figure 6 is a schematic representation of an embodiment of the present invention, wherein the server includes a method for repairing the coded data stream received Repair device for the included media data.

detailed description

The telecommunications system according to the invention is depicted in FIG. 2. This type of telecommunication system includes an encoder 20, a first network connection 30 between the encoder 20 and a server 40, and a second network connection 50 between the server and the client device 60. The encoder encodes the multimedia content 10 from the content provider into an encoded data stream EDS.

The encoded data stream may include any number of media tracks, such as video tracks, audio tracks, and possibly text tracks or image tracks. It is sent in real time via said first network connection 30. In the preferred embodiment of the present invention, the RTP (Real Time Protocol) protocol is often used in the case of streaming applications, but this is not restrictive. The transport layer of the MPEG-2 standard called MPEG-2 TS can also be used. The encoded data stream EDS is thus encapsulated into RTP packets. The RTP packet includes some coded data also called media data and metadata as control data used to describe the media data.

It should be noted that the multimedia content MM may be live content or more generally any recorded multimedia program, but the multimedia content is broadcast and not made available on a "video on demand" server. The first network connection is therefore a multicast broadcast session, which is "heard" by a large number of clients and the server 40 among them.

The stream of the RTP packet is received by the receiving device 41 of the server 40, and converted into a sequential file PF by the stream-to-file conversion device 42. The sequential file PF has a file format including interleaved media data and metadata. In a preferred embodiment of the present invention, the sequence file follows the ISO file format version 2. It should be noted that in order to comply with ISO file format version 2, the file only needs to include meta and media data, and the data syntax is defined by the standard rather than their organization. Referring to FIG. 3, the live file according to the present invention is divided into data boxes (databoxes) connected in series, and one data box includes meta or media data. ISO file format version 2 defines three types of data boxes:-"MDAT" data boxes, which include interleaved data blocks of media data such as audio A, video V or text T sources. The data block does not have any structure or mark, a "MOOV" and a large number of "MOOF" data boxes, which include metadata for describing and accessing the media data. The ISO file format starts with a single "MOOV" data box. It is followed by alternating "MDAT" and "MOOF" boxes.

-The "MOOV" data box includes initialization media data, such as elements of the decoder configuration and some index tables for accessing the media data stored in the first MDAT. The "MOOF" data box includes an index table used to access media data stored in the usual subsequent MDAT.

It should be noted that another file format can also be used, such as e.g. MJPEG or e.g. Apple. Proprietary files such as the moov file format. The advantage of ISO file format version 2 is that it is compatible with a large number of standards for encoding multimedia data such as MPEG-4 for video tracks and AMR for audio tracks, which means that files using the format can be passed Follow the standard decoder to play. This is neither the case for MJPEG files that require an MJPEG decoder, nor is it specifically designed for Apple QuickTime players. The case of moov files.

The stream-to-file conversion device 42 is responsible for filling the meta and media data contained in the received RTP packet into the live file structure.

In the following, it will be assumed that the encoded data stream is an MPEG-4 encoded data stream. This is non-limiting as described above, and any other format compatible with ISO file format version 2 can be used.

The MPEG-4 encoded data stream is divided into multiple access units. An access unit is a data set that can be directly accessed. The RTP packet includes one or several access units from the MPEG-4 encoded data stream and some metadata about the access units, and the metadata forms an RTP header. In particular, the RTP header includes an access unit timestamp, which indicates when the access unit must be decoded.

The stream-to-file conversion device 42 mainly uses ISO file format version 2 to create a sequential file PF. This is done by:-copying the access unit related to the timestamp to one or several MDAT boxes, each MDAT box has An index,-the MOOV and MOOF tables indexed by associating the timestamp to the index of the MDAT box,-extracting the specified decoding from the SDP (Session Description Protocol, a protocol dedicated to initiating multimedia sessions) files The metadata of the device configuration, the files are generally sent to the server in parallel with the RTP stream, and they are copied to the MOOV table of the sequential file.

The obtained sequential file is suitable for sequential downloading because it is composed of independent data segments, including MOOF data boxes and MDAT data boxes, which can be independently decoded from any other data except MOOV data boxes . Therefore, as long as the client receives the MOOV data box, it can start decoding, which corresponds to a short delay.

The server 40 also includes a sending device 44 for sending the client sequence file to the client device 60. For example, the client device 60 includes a web browser for browsing web pages, where the client sequence file CPF is available as a downloadable file, for example. In response to the client's request RQ, the client sequence file CPF is sent to the client device 60 via the second network connection 50. In a preferred embodiment of the present invention, the second network connection 50 uses HTTP (Hypertext Transfer Protocol) protocol. The protocol, which is the basis of the World Wide Web, is responsible for transmitting HTML documents and managing services on the Internet. However, this is not restrictive, and FTP (File Transfer Protocol) can also be used.

The sequence file is given a basic URL address, such as http:server:port/american/live/madonna.mpg4. The sending device 44 includes a redirection sub-device. The redirection sub-device is to create a redirection file for containing a basic URL address. The redirection file is given a redirected URL address, such as http:server:port/redirection/madonna.m4r pointed to by a hypertext link on the webpage. Clicking on the redirection URL address causes the web browser of the client device 60 to download the redirection file. Once the file is downloaded, the redirected file is read by the web browser. The web browser can recognize the MPEG-4 file in the basic URL address, and directly call the player 61 suitable for processing this type of file format. Then, the player reads the URL address contained in the redirection file, and directly requests a download device 62 to download. The downloading device 62 is used to inform the player that the entire file has been downloaded, although only a part of the file is actually available, so that the player immediately opens the sequential file. Once opened, the part of the sequence file that is already available can be read according to the structure of the ISO file format version 2.

The advantage of the redirection device and the download device 62 according to the present invention is that sequential downloading is possible. Without any redirection, the sequence file PF is already downloaded by the web browser before being sent to the player. Before opening the sequence file PF, if there is no downloading device 62, the player waits until the downloading ends.

The server 40 finally includes a customizing device 43 for customizing the sequence file PF into a client sequence file CPF suitable for the client's request. The possible structure of the client sequence file is shown in FIG. 5 and will be described below.

It is assumed that the multimedia content has been received by the server 40 as an RTP stream since time t0, and the RTP stream is available on the server side as a hypertext link to the sequential file PF on the web page. A large number of clients may be playing the multimedia content at the same time. Referring to FIG. 4, it can also be assumed that a new client who is browsing the web page of the server requests to download the sequential file PF sequentially at time t. The purpose of the customization device 43 is to allow the new client to grab multimedia content as quickly as possible. For this purpose, the customization device 43 includes a primer sub-device 45 for providing initialization metadata to the client at time t. The initialization metadata mainly includes the decoder configuration, but more generally includes all the data needed by the client to start receiving real-time encoded data.

The important point is that the encoded data can only be accessed at a predetermined time stamp. The time stamp is related to the aforementioned access unit. Therefore, the access unit includes a timestamp indicating when to play the media data it contains. Some access units are random access points, that is, they can be accessed directly. For example, within a video track, a random access point corresponds to an "internal" image, that is, to an image that is coded independently of the previous image and can therefore be decoded independently.

The server includes a buffer BUF for temporarily storing a part of the sequential file corresponding to the last received RTP packet. The buffer can store segments of the sequential file, which can be decoded independently of RTP packets that have not yet been received. Such segments therefore include MOOF boxes and MDAT boxes. The MDAT box includes a large number of access units from tracks of different encoded data, for example, access units from audio and video tracks. The MOOF box includes an index table for accessing coded data contained in the MDAT box. The segment therefore includes more than one access unit timestamp. The "accessible time stamp" TS will hereinafter be referred to as the first access unit time stamp of the MDAT box.

Once the buffer is full, its content is sent to all connected clients simultaneously as burst data, and then the buffer stores new sequential file segments.

The buffer can store several seconds of encoded data. This means that the client will delay several seconds before receiving the live multimedia content. On the one hand, this delay should not be too high, especially for live events such as football matches, but on the other hand, the smaller the buffer, the higher the data overhead. In fact, it is not cost-free to restructure data into MOOF and MDAT, and a reasonable box size must be used so as not to affect the compression ratio.

The initial sub-device 45 can therefore:-respond to the client request by sending the decoder configuration INI to the client, the decoder configuration I NI as part of the MOOV box corresponding to the sequence file of the initial SDP file,-look for the time The next accessible time stamp TS occurring after t. If the time t is shorter than the next time stamp TS, the data contained in the sequential file is not accessible before the next time stamp TS. Between the two, the initial sub-device 45 can transmit additional padding data PAD to the new client, and the additional padding data PAD is used to make the client wait until the time stamp TS. These padding data PAD can simply provide a black screen or logo or even some commercial advertising programs.

The sequential file PF actually looks like a virtual file because it never exists as a whole on the server side. Only the segments of the live file are available in the buffer BUF at time t.

The customization device 43 also includes a starting sub-device 46. The starter device 45 is intended to start the transmission of the content of the buffer BUF to a new client from the time stamp TS. The activation sub-device 46 is, for example, to add the address of the client to the list of registered clients. Figure 4 shows the data received by the new client from the server from time t. Starting from the time stamp TS, the new client just receives the same data as other clients.

This is an additional important advantage of the present invention, that is, each client is sent the same data at the same time, because it allows advanced server performance with minimal hardware resources. In fact, in traditional video-on-demand, server performance decreases with the number of different parallel streams that the server must process. For example, depending on the size of the server's dynamic storage, a server that can serve 1,000 different streams in parallel may be able to serve 2,000 or more similar streams, depending on the availability of data in the dynamic storage instead of the hard disk. One of these storage media The difference in access speeds is great. Specifically, in the current situation, because the maximum memory size required to serve all clients at the same time is the above-mentioned buffer size, the performance of the video server is the most ideal, and the buffer size is much smaller than the typical server dynamics. Memory.

The decoder configuration INI, padding data PAD, and media data starting from the time stamp TS form a customized version of the sequence file PF, that is, the client sequence file CPF is particularly suitable for the requesting client. The client sequential file CPF is also a virtual file.

Unlike the first network connection 30, the second network connection 50 is a point-to-point connection between the server 40 and the client device 60, wherein the server and the client device are perceptible to each other. As shown in FIG. 4, the client device 60 includes request means 63 for requesting the sequence file PF available on the server 40, and for downloading the client sequence file CPF provided by the customization device 43 via the second network connection 50. The download device 62 of the client computer, and a player 61 for playing the received encoded data RED contained in the client sequence file in real time.

It should be noted that traditional players can only open local files, and cannot download remote files, that is, files located on remote servers. The downloading device 62, which is well known to those skilled in the art, enables the player 61 to process the received encoded data contained in the sequential file as if they were stored in a local file. For example, by using the HTTP command GET instead of the web browsing device 63, the download device can sort the download of the sequential files. As long as the coded data from the sequential file is received, the player 61 can recognize the ISO file format version 2 and start to decode the received coded data RED before the end of the download. The decoded multimedia content DMC is output and displayed.

The received encoded data RED forms a received client sequence file, which can be stored and replayed. It should be noted that the server can be designed to limit the number of authorized client copies. Such restrictions can be established, for example, by using DRM (Data Resource Management) technology, such as Open Mobile Alliance (OMA) download version 1.

It should be noted that the complete file size can exceed the memory size on the client. In this case, sequential downloading provides an additional advantage that the data corresponding to the beginning of the file can be erased during the playback process to give more new files. Data makes room; this method can effectively make endless programs available.

Another advantage of the client device according to the present invention is that it has no special features other than the ability to implement sequential downloading, which is known to those skilled in the art and is gradually becoming common. This means that the present invention will work for any client that includes a player and download device that can handle version 2 of the ISO file format.

In another embodiment of the present invention, the download server 40 further includes a repair device 49 for making holes in the sequence file PF complete, as shown in FIG. 6. The hole may be caused by possible data loss when passing through the first network connection 30 in the real-time data stream. For example, if the RTP protocol is used, some RTP packets may simply be lost during transmission or be recognized as errors by the RTP protocol on the server side. They may be rejected in the second case because there may not be time to request packet retransmission in real-time transmission. The loss or rejection of the RTP packet by the server 40 causes a "hole" in the sequence file created by the conversion device 42. The hole should not cause the player to crash on the client side, because by detecting, for example, the loss of the access unit timestamp, the compliant decoder is expected to be able to cope with the missing data in the encoded data stream. However, the hole will cause the quality of the displayed decoded multimedia content to decrease.

The system according to the present invention is capable of intercepting the encoded data during the transmission of the encoded data from the encoder 20 to the client 60. The advantage of the system is to benefit from this interception to repair the encoded data in the air. For this purpose, the repair device 48 can complete the hole by extrapolating the neighboring data using error resilience techniques. The error recovery techniques well known to those skilled in the art can handle compressed or decompressed data. The repaired sequence file RPF is output, and the client's repaired sequence file CRPF is sent to the client 60.

Additional advantageous settings of processing can be performed during the interception of data by the server 40. For example, by replacing an audio track with another audio track that is usually characterized by a track in a different language, it may consist in customizing the media data contained in the sequence file (PF) as the configuration file data assigned to the client device 60 The function. In fact, it is expected that very large-scale (that is, all over the country and even the world) programs will be distributed using a large number of servers. Each server is specific to a given country or region or town or region. In this case, some Replacing sequences with other sequences may make sense to users or have economic value for service providers, such as replacing general advertisements with advertisements that are more targeted to viewers of a given server. In addition, instead of having different processing as described above on a per server basis, the same server may also perform specific processing based on other criteria such as user preference or user profile. Examples of this type of processing include language selection and targeting ads.

The above drawings and their descriptions illustrate rather than limit the present invention. It is obvious that there are a large number of alternatives that fall within the scope of the appended claims. In this regard, the following closing remarks are made: there are a large number of methods to perform these functions by means of hardware or software items or a combination thereof. In this regard, the drawings are very diagrammatic, each of which represents only one possible embodiment of the invention. Therefore, although the drawings show different functions as different blocks, this does not exclude a single item of hardware or software to perform several functions, nor does it exclude a combination of hardware or software or both of them to perform a single function. For example, unlike what is illustrated in FIGS. 2, 4 and 6, the player 61 may also be a remote device independent of the client device 60. Any reference signs in the claims should not be construed as limiting the claims. The use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in the claims. The article "a" or "an" used before an element or step does not exclude the presence of a plurality of such elements or steps.