Medium streaming distribution system

TECHNICAL FIELD

0001The present invention relates to media streaming, and more particularly relates to streaming in a case where delay on a stream transmission path becomes larger mid-way through the transmission path, such as the case where a media stream is transmitted to a client via a wireless link. Note that, though a wireless link is taken as an example of the transmission path of a large delay in the following explanations, the present invention is not limited thereto.

BACKGROUND ART

0002In recent years, due to striking advances in communication technologies, it has become possible to perform multimedia delivery by means of wireless communication. In order for a client that receives wireless data delivery to playback multimedia in real time, QoS control over a communication path (transmission rate control and packet loss resilience control) become important.

0003Non-Patent Document 1, shown below, discloses that a wireless base station comprises an RTP monitoring agent after a shaping point (a flow control point). An RTP (Real-time Transfer Protocol) monitoring agent monitors a multimedia stream that arrives at the wireless base station and is to be transmitted via a wireless link, and feeds back to a media server the information indicating the congestion status in a wired network from the media server to the wireless base station. The client that receives data packets wirelessly from the wireless base station transmits reception acknowledge information to the media server according to RTCP protocol.

0004The media server determines the congestion status in the wired network and the link error condition over the wireless link based on the information fed back from the RTP monitoring agent and the reception report from the client. When a packet loss has occurred due to the congestion in a wired network, the media server reduces the media stream transmission rate, and when packet loss has occurred due to an error in the wireless link, it increases the packet loss resiliency of the encoding of the media. The packet loss resiliency can be increased by, for example, increasing the rate of intra-frame coding in video encoding.

0005In this type of conventional technique, once a packet loss occurs, considerable time is required before packet loss resiliency is increased. In a case where packets containing an I-picture in a video stream is discarded due to flow control in response to congestion in the wired network and packets of multiple subsequent P-pictures are then transmitted via a wireless link, the client is unable to reconstruct these P-pictures since the I-picture needed for reference to reconstruct the P-pictures is not available. Furthermore, even when a P-picture is discarded, no subsequent P-pictures dependent on that P-picture can be reconstructed until the next I-picture is received. In general TCP communications, a packet that is lost is retransmitted based on feedback information. However, in streaming where the media is reconstructed and played back in real time, retransmission may be too late in such a case where the buffer size of the client is small. When packets are discarded due to the congestion in a wired network, reconstruction errors occur not only for the discarded packet but also for the P-pictures contained in subsequent packets successfully received.

0006Therefore, there is a need to reduce the error propagation phenomenon where a packet loss occurring in the network upstream from the base station of a low-speed communication such as a wireless communication has an effect on reconstruction of the media contained in the subsequent packets.

DISCLOSURE OF THE INVENTION

0007A media streaming delivery system according to an aspect of the present invention comprises: a media delivery apparatus for transmitting a media stream in packets to a network according to a real time transfer protocol; and a wireless base station connected to the network for transmitting the media stream by a low-speed communication such as a wireless communication. The system further includes a packet analysis device for monitoring the packet arriving at the wireless base station, and transmitting feedback information relating to a packet loss to the media delivery device.

0008According to another aspect of the present invention, there is provided a media delivery apparatus included in the above-mentioned system.

0009According to an embodiment of the present invention, a packet analysis apparatus transmits a sequence number included in a packet header to the media delivery apparatus as feedback information.

0010Furthermore, in an embodiment of the present invention, the media delivery apparatus determines packet loss based on the feedback information and, depending on the determined packet loss, modifies the media stream so as to reduce the influence of the packet loss.

0011According to an embodiment of the present invention, the media stream is a movie including a sequence of I-pictures and P-pictures. The media delivery device includes a storage for storing a plurality of media streams for one movie including at least a first media stream including I-pictures in a first arrangement and a second media stream including I-pictures in a second arrangement which is different from the first arrangement.

0012The media delivery apparatus further includes switching means for, in response to the determination of the packet loss, for a destinated location where the loss has occurred, selecting a media stream in which the first I-picture after the picture in the lost packet appears earliest among the plurality of the media streams and switching the media stream to be sent to the selected media stream. This can reduce the error propagation phenomenon where a packet loss has an effect on play back of the media contained in the subsequent packet.

0013In an embodiment of the present invention, there are provided media streams with different frequencies of intra-frame coded I-pictures, and switching is made to the media stream with a higher frequency of I-pictures in response to packet loss.

0014Furthermore, according to another aspect of the present invention, the media delivery apparatus further includes an encoding device for generating the media stream. The encoding device is arranged to generate a media stream starting with an I-picture in response to the determination of packet loss.

0015According to an embodiment of the present invention, the encoding device increases the frequency of I-pictures at least for the media stream transmitted to the destination where the loss has occurred in response to detection of the packet loss based on the feedback information from the packet analysis apparatus.

0016Furthermore, according to an embodiment of the present invention, the media delivery apparatus receives destination feedback information regarding packet reception from a wireless terminal that is the destination of the media stream, and determines whether the packet loss has occurred in the wired transmission in the network or in the wireless transmission from the wireless base station based on the destination feedback information and the feedback information from the packet analysis apparatus.

0017Furthermore, in an embodiment of the present invention, the media delivery apparatus performs packet retransmission based on the feedback information on the packet loss.

0018Furthermore, according to an embodiment of the present invention, a transmission rate from the media delivery apparatus to the relay apparatus and a transmission rate from there lay apparatus to the terminal are obtained based on the feedback from both the media stream relay device and the terminal, and a surplus band is used for retransmission or forward error correction on the side of greater transmission rate, thereby improving communication quality.

BRIEF DESCRIPTION OF THE DRAWINGS

0019<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram showing concepts of an embodiment of the present inventions.

0020<figref idref="DRAWINGS">FIG. 2</figref> is a block diagram showing a construction of an embodiment of the present invention.

0021<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram showing a construction of an alternative embodiment of the present invention.

0022<figref idref="DRAWINGS">FIG. 4</figref> is a diagram showing examples of video compression coding formats.

0023<figref idref="DRAWINGS">FIG. 5</figref> is a diagram showing examples of MPEG I-picture and P-picture sequences.

0024<figref idref="DRAWINGS">FIG. 6</figref> is a sequence chart showing a communication sequence in accordance with a first embodiment and a second embodiment of the present invention.

0025<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram showing concepts of yet another embodiment of the present invention.

0026<figref idref="DRAWINGS">FIG. 8</figref> is a block diagram showing detailed constructions of the embodiment in <figref idref="DRAWINGS">FIG. 7</figref>.

0027<figref idref="DRAWINGS">FIG. 9</figref> is a sequence chart showing communication sequences in accordance with a third embodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

0028Hereinafter, an embodiment of the present invention will be explained with reference to the drawings. <figref idref="DRAWINGS">FIG. 1</figref> is a diagram showing an overall construction of an embodiment of the present invention. A media delivery device <b>11</b> performs real time delivery of multimedia. The multimedia includes images, audio, text, graphics, and the like, and streams of each are packet-transmitted, and are reconstructed and played back in real time by a receiving device. In the following explanations, these media are described with respect to a video stream. The media delivery device <b>11</b> can deliver the media stream in the form of on-demand video delivery, or in the form of a broadcast.

0029When a media delivery request comes from a wireless terminal <b>19</b> such as a mobile phone, a portable terminal (PDA), a computer provided with a wireless communication device, the media delivery device <b>11</b> packetizes the media stream, attaches an RTP (Real-time Transport Protocol) header and an IP header thereto, and transmits this via a wired network <b>21</b>. The packets are routed according to an Internet Protocol (IP), and arrive at a wireless base station <b>10</b>. The IP is a connectionless communication protocol, and when congestion occurs on the network, overflowing packets may be discarded from the network.

0030The wireless base station <b>10</b> transmits the packets received from the network <b>21</b> to the wireless terminal <b>19</b>. According to the present invention, a packet analysis device <b>15</b> is provided to the wireless base station <b>10</b> and monitors the media stream packets that arrive at the wireless base station <b>10</b>. The sequence of the media stream packets sent to a particular wireless terminal <b>19</b> can be identified by means of a pair of source and destination addresses included in the IP header and a pair of source and destination port numbers included in the UDP header. As will be explained in detail below, the RTP header is included in the packet, and the RTP header includes a sequence number that is unique to that packet in the media stream.

0031When detecting the flow of a sequence of packets having the same source addresses and destination addresses and also the same source port numbers and the destination port numbers, the packet analysis device <b>15</b> gives feedback (ACK) of the packet numbers contained in the RTP header, to the media delivery device. The media delivery device <b>11</b> can detect packet loss from this feedback information. When the media delivery device <b>11</b> detects that an I-picture packet was lost from the packet sequence numbers, the media format is switched to a format with a greater frequency of I-pictures.

0032With this operation, the wireless terminal <b>19</b> can receive the I-pictures early, and therefore a decline in the quality of reconstruction of the video can be prevented.

0033Referring next to <figref idref="DRAWINGS">FIG. 2</figref>, more detailed explanation is given regarding an embodiment of the present invention. The media delivery device <b>11</b> stores video contents A, B, C, and D into a multimedia content database <b>27</b> in two different formats. That is, the video A is stored as video contents A<b>1</b> in a first format, and video contents A<b>2</b> in a second format. Similarly, the video B is stored as contents B<b>1</b> in the first format and contents B<b>2</b> in the second format, and the video C is stored as contents C<b>1</b> in the first format and contents C<b>2</b> in the second format.

0034<figref idref="DRAWINGS">FIG. 5</figref> shows examples of the two formats. <figref idref="DRAWINGS">FIG. 5</figref>(A) shows an example of the first format, and <figref idref="DRAWINGS">FIG. 5</figref>(B) shows and example of the second format. The second format is set with a higher I-picture frequency than the first format. In the diagram, I indicates an intra-frame coded I-picture, and P indicates the P-picture that is prediction-coded based on the I-picture. When there is another preceding P-picture between the I-picture and the P-picture, the P-picture is prediction-encoded based on the I-picture and based on the preceding P-picture. In this embodiment, the movies are encoded in accordance with MPEG 4.

0035According to the MPEG 4 standard, the frames (pictures) of an inputted image is classified into I-pictures, each of which is encoded using information solely on that frame independent of images in the other frames, and P-pictures, each of which is predicted based on an I-picture, and residual prediction error is encoded for transmission.

0036The I-picture and the P-picture are each converted by DCT (Discrete Cosine Transform) into DCT coefficients. The DCT coefficients and motion information are entropy coded. At this time, one picture is divided into 8×8-pixel blocks, and DCT is performed on each block. Four adjacent blocks are combined into one macroblock (MB), and multiple macroblocks are combined as one group of blocks (GOB). As shown in <figref idref="DRAWINGS">FIG. 4</figref>(B), multiple macroblocks are placed in one packet and transmitted.

0037In this embodiment, the media delivery device <b>11</b> performs the on-demand multimedia delivery. In the diagram, the video content is only shown, but audio content which will be synchronized with the video and reconstructed is also stored in the database similarly, and is sent in parallel with video content.

0038A format switching section <b>29</b>, in response to an instruction from a feedback control section <b>35</b>, switches the content extracted from the database <b>27</b> between the first format and the second format.

0039A multimedia stream detecting section <b>37</b>, which is included in the packet analysis device <b>15</b>, detects the packet containing the RTP header from among the IP packets arriving at the wireless base station <b>10</b>, and detects the multimedia stream based on the pair of the source address and the destination address, and the pair of the source port number and the destination port number, of the packet. An RTP header monitor section <b>39</b> monitors the RTP header contained in the multimedia stream IP packets that are detected by the multimedia stream detecting section <b>37</b>, and inspects the packet sequence number contained in the RTP header. When loss of a sequence number is detected from inspecting the sequence numbers across multiple packets, the RTP header monitor section <b>39</b> transmits information for identifying the lost IP packet to a feedback generator <b>41</b>. In an embodiment, the information for identifying the IP packet includes the source address, the destination address, and the lost RTP packet's sequence number. Furthermore, in another embodiment, the RTP monitor section <b>39</b> transmits not the lost RTP packet's sequence number, but rather the sequence number of an RTP packet each time the RTP packet is detected, to the feedback generator <b>41</b>.

0040The feedback generator <b>41</b>, upon receiving each RTP packet, returns the sequence numbers thereof to the source address as an ACK. In an alternative embodiment, the feedback generator <b>41</b>, in response to the information transmitted from the RTP header monitor section <b>39</b>, determines whether the RTP packet is lost within a predetermined period based on the sequence numbers of the sequence of RTP packets, and transmits the sequence number of the lost RTP packet to the source address. In the feedback in this case, the RTCP can be extended and sent as well. These are sent to the wired network <b>21</b>.

0041The feedback control section <b>35</b> of the media delivery device <b>11</b> receives the feedback information from the packet analysis device <b>15</b>, and in response to reception of the lost packet's sequence number, transmits a media format switching instruction to the format switching section <b>29</b>. In another embodiment, the feedback control section <b>35</b>, based on the ACK information that is sent from the packet analysis device <b>15</b>, monitors lost packet sequence numbers, and when a lost one is detected, transmits the media format switching instruction to the format switching section <b>29</b>.

0042In all the embodiments, the feedback control section <b>35</b>, when it is detected that the packet is lost, transmits the format switching instruction to the format switching section <b>29</b>. When the lost packet is for a P-picture, the influence is less than in the case of the I-picture. Therefore, if the effect of the switching will be relatively small (e.g., when there are only a few P-pictures before the next I-picture, or the like) the format switching may be omitted.

0043Now, assuming that the media A<b>1</b> in the first format is provided to the wireless terminal <b>19</b>, the format switching section <b>29</b> switches the stream to the media A<b>2</b> which is in the second format. Referring back to <figref idref="DRAWINGS">FIG. 5</figref>(A), in the media A<b>1</b> steam, it is assumed that what is lost is either one or a plurality of multiple packets containing a macroblock of I-pictures <b>51</b>, or a packet containing a macroblock of P-pictures <b>51</b><i>a. </i>The format switching section <b>29</b>, in response to the instruction from the feedback control section <b>35</b>, switches the media to A<b>2</b> in the second format, and, after the I-picture <b>51</b> or P-picture <b>51</b><i>a </i>in the media A<b>1</b> is lost, starts the transmission from the I-picture <b>57</b> that appears first in the media A<b>2</b>.

0044At the wireless terminal <b>19</b>, since one or more packets of the I-picture <b>51</b> or packets of the P-picture <b>51</b><i>a </i>are lost, subsequent P-pictures <b>51</b><i>b, </i><b>51</b><i>c, </i><b>51</b><i>d, </i>and <b>51</b><i>e </i>which are encoded depending upon the I-picture <b>51</b> or the P-picture <b>51</b><i>a </i>cannot be reconstructed properly. However, the media delivery device <b>11</b> switches the media format to the second format where the I-picture frequency is higher, and starts transmission from the I-picture <b>57</b> in the second format shown in <figref idref="DRAWINGS">FIG. 5</figref>(B). Therefore, the wireless terminal <b>19</b> can properly reconstruct the media from the I-picture <b>57</b>.

0045<figref idref="DRAWINGS">FIG. 6</figref> shows an example of communication and processing sequences in the media delivery system in accordance with the present invention.

0046The content that is extracted from the database <b>27</b> is packetized at a packetizing section <b>31</b>, and is then sent out from a stream transmission section <b>33</b> to the wired network <b>21</b>. Referring to <figref idref="DRAWINGS">FIG. 6</figref>, the stream transmission section <b>33</b> executes an RTP processing routine <b>111</b> and an RTCP processing routine.

0047The RTP processing routine adds a compression code header for identifying the format of the compression coding to compressed data corresponding to one packet, and also adds the RTP header, the UDP header and the IP header. <figref idref="DRAWINGS">FIG. 4</figref>(A) shows an example of such a packet format. The RTP header contains the packet sequence number. An initial value of this sequence number is determined with a random number, and serial numbers starting from this initial value are added to each packet. The RTP header contains an SSRC identifier (Synchronization Source identifier). For this identifier the same value is added to multiple streams that should be handled in combination, for example, a voice stream and an image stream of the same user.

0048The processing routine <b>111</b> adds the RTP packet with the RTP header, the UDP header, and the IP header, for transmission according to a UDP/IP protocol. The IP header includes the source address indicating the transmission source, and the destination address indicating the destination. Furthermore, the UDP header includes the source port number indicating the port number of the transmission source, and the destination port number indicating the port number of the destination.

0049<figref idref="DRAWINGS">FIG. 4</figref>(B) shows how a picture in one frame is divided into multiple packets and sent in accordance with the MPEG standard. For example, the I-picture, because of its large information volume, is divided into multiple packets and then sent. When much congestion occurs on the network, a portion of the packets of the I-picture maybe discarded, and there is a possibility of the complete I-picture not arriving at the wireless terminal <b>19</b>. In <figref idref="DRAWINGS">FIG. 4</figref>(B), GOB indicates the group of blocks, and MB indicates the macroblock.

0050The RTCP processing routine <b>113</b> exchanges the control information with the network transmitting the RTP packets. The RTCP (Real-Time Transport Control Protocol, RTP Control Protocol) is a protocol that supplements the RTP, where the transmission device and the receiving device notify the control information to each other, whereby flow control, clock synchronization, and synchronization between media such as between audio data and video data are achieved.

0051In advance of transmission of the RTP-packet, the RTCP processing routine <b>113</b> turns the SDES (Source Description) which is a description about the information source into an RTCP packet <b>115</b> and transmit it onto the network using the UDP/IP protocol with the IP header and the UDP header added. After this, the RTP processing routine transmits RTP packets <b>117</b>, <b>119</b>, and <b>121</b> in sequence to the network.

0052In <figref idref="DRAWINGS">FIG. 2</figref>, a shaping point <b>13</b> placed before the wireless base station <b>10</b> represents one or a plurality of relay devices which govern flow controls in the wired network on the path from the media delivery device <b>11</b> to the wireless base station. The wireless base station <b>10</b> is provided with, for example, a layer <b>3</b> switch as a counterpart component of the shaping point for buffering the packets received from the wired network <b>21</b> at high speed and providing them to the wireless section at a speed adapted for the processing speed of the wireless section of the wireless base station <b>10</b>.

0053Referring back to <figref idref="DRAWINGS">FIG. 6</figref>, based on the RTCP packet <b>115</b> and the subsequent RTP packets <b>117</b>, <b>119</b>, and <b>121</b>, the packet analysis device <b>15</b> of the wireless base station <b>10</b> detects that they are parts of the same media stream, and calculates a packet loss rate, jitter and the like (<b>131</b>) from the packets received over a predetermined period. It then puts these pieces of information to an RTCP packet directed to the media delivery device <b>11</b>, and transmits this packet onto the wired network.

0054Each time the packet analysis device <b>15</b> detects an RTP packet from the same media stream from the media delivery device <b>11</b>, it transmits a reception acknowledge signal (ACK) containing the sequence number of that RTP packet to the media delivery device <b>11</b>. In an alternative embodiment, the packet analysis device <b>15</b> does not return the ACK each time the RTP packet is received, but monitors if the sequence numbers of the RTP packets belonging to the same media stream are contiguous. When the packet analysis device <b>15</b> detects that a sequence number is skipped, it transmits a reception failure signal (NACK) containing the sequence number of the lost RTP packet to the media delivery device <b>11</b>.

0055In the RTCP processing routine <b>113</b> of the media delivery device <b>11</b>, when an RTCP packet <b>133</b> which is a response to the RTCP packet <b>115</b> sent out is received, depending on the packet discard rate and other control information contained therein, the transmission rate of the media stream to be sent out is modified. Furthermore, when the packet loss rate detected in the RTCP processing routine <b>113</b> is high, the feedback control routine <b>35</b> transmits, to the format switching section <b>29</b> (<figref idref="DRAWINGS">FIG. 2</figref>), a control signal to switch the subsequent media stream format to the format with the higher I-picture frequency.

0056In the RTP processing routine <b>111</b> of the media delivery device <b>11</b>, each time ACK packets <b>105</b>, <b>107</b>, and <b>109</b> from the wireless base station <b>10</b> are received, the sequence numbers, which are contained in the ACK packets, of the RTP packets received from the wireless base station <b>10</b>, are transferred to the feedback control section <b>35</b>. The feedback control section <b>35</b> inspects whether or not any of the sent packets did not arrive at the wireless base station <b>10</b>, which is to say whether or not any packets are lost (<b>141</b>). When the feedback control section <b>35</b> detects that a packet is lost, the feedback control section <b>35</b> then judges whether or not the data contained in that RTP packet is an I-picture (<b>143</b>). When the lost packet is an I-picture, the above-mentioned format switching procedure (<b>145</b>) starts immediately.

0057At step <b>143</b>, when it is determined that the lost packet is in a P-picture, it is then determined whether or not the packet is lost up to a reference for retransmission set when a P-picture is lost (<b>147</b>). When the packet of the P-picture is lost, the influence is less than in the case of the I-picture. Therefore, when a predetermined reference value is reached the format is switched (<b>149</b>), and until this reference is reached the format is not switched. For example, the reference may be set so that the format is not switched in the case where there are only a few P-pictures before the next I-picture.

0058The wireless section of the wireless base station transmits the received IP packet by wireless. The wireless terminal <b>19</b> receives the wireless communication and reconstructs the media stream. Transmitting and receiving IP packets by wireless is a publicly known technique, so that detailed explanation thereof is omitted.

0059Depending on the construction of the wireless terminal <b>19</b>, the wireless terminal <b>19</b> can transmit the ACK signal to the media delivery device <b>11</b>. Specifically, a wireless terminal that conforms to the 3GPP standard cannot transmit ACK signals as the packet analysis device <b>15</b> does. When there is no need to conform with this standard, the wireless terminal can be constructed to transmit ACK signals. The feedback control section <b>35</b> compares the feedback information from the packet analysis device <b>15</b> with the information from the wireless terminal <b>19</b>, whereby the quality of the wireless communication from the wireless base station <b>10</b> to the wireless terminal <b>19</b> can be known. When it is judged that the quality of the wireless communication is low, the feedback control section issues to the format switching section <b>29</b> an instruction to switch the media form at to the format with the higher frequency of I-pictures. The formats for the media are not only two types. Many types of formats are available, and the format may be switched among many formats depending on the communication status. For example, when transmissions are being performed in a format with a high frequency of I-pictures, if sufficiently good communication status is detected, then the format can be switched to the format with the lower frequency of I-pictures. In this way, the media delivery device <b>11</b> can adjust the load of the stream transmission.

0060Referring next to <figref idref="DRAWINGS">FIG. 3</figref>, explanation is given regarding another embodiment of the present invention. In <figref idref="DRAWINGS">FIG. 3</figref>, elements that are similar to the embodiment in <figref idref="DRAWINGS">FIG. 2</figref> are indicated with the same reference numerals. At the media delivery device <b>11</b>, inputted images <b>30</b> are compressed/encoded in real time by an encoding section <b>28</b>, and then turned into packets by the packetizing section <b>31</b>, and then sent to the wired network <b>21</b> as the media stream. The inputted images <b>30</b> may be images that have been recorded in advance and stored in a storage device, or may be images that are captured in real time using a video camera as in a relay broadcast.

0061The encoding section <b>28</b> encodes the inputted images according to the MPEG4 standard which has already been explained. The encoding section <b>28</b> has a transcoding function, which in response to an instruction from the feedback control section <b>35</b>, modifies the frequency of the I-pictures in the encoding. That is, when, based on the feedback information from the packet analysis device <b>15</b>, it is detected that packets including a macroblock of I-pictures are lost in the media stream that arrives at the wireless base station <b>10</b>, the encoding section <b>28</b> switches the encoding so as to increase the frequency of the I-pictures.

0062Next, explanation is given regarding another embodiment for performing retransmission of the lost packet, based on the feedback from the wireless base station.

0063In the wireless link, delay time is normally long. The time for going back and forth between the media delivery device and the wireless base station, is much shorter than the time for going back and forth between the wireless base station and the wireless terminal client. Therefore, based on the size of the receiving buffer provided to the wireless terminal (i.e., the extent to which packet arrival can be delayed due to retransmission of the packets), when any packet is lost on the wired network from the media delivery device to the wireless base station and in the wireless link from the wireless base station to the wireless terminal, as a form of restoring the lost packet, it is determined whether to retransmit the lost packet on the wired network or wireless link, or to perform forward error correction (FEC), so that the throughput of each communication path can be optimized. FEC is specifically realized by using Reed-Solomon encoding or other error correction encoding commonly known to those skilled in the art. Furthermore, in order to further improve resistance to burst errors, interleaving can be performed, which is also commonly known to those skilled in the art. A commonly known example of using interleaving to improve the ability to correct burst errors is CIRC (Cross Interleaved Reed-Solomon Code) used, for example, for CD-DA.

0064In order to perform packet retransmission and FEC at their proper times, three modes may be provided as a form of restoring the lost packet on the two communication paths, based on the size of the wireless terminal's receiving buffer. These modes are explained in detail below.

0065<figref idref="DRAWINGS">FIG. 7</figref> shows an overall construction of a media stream delivery system provided with such a function. Constituent elements similar to <figref idref="DRAWINGS">FIGS. 1, 2</figref> and <b>3</b> are indicated by the same reference numerals. The wireless base station <b>101</b> is provided with a first feedback section <b>71</b>, an FEC decoder <b>72</b>, a shaping point <b>13</b>′ that performs flow control at the wireless base station, an FEC generator <b>18</b>, a second feedback section <b>73</b>, and a wireless transmission device <b>14</b>. The FEC decoder <b>72</b> functions, in the mode for using FEC to restore the lost packet on the network <b>21</b> (Mode <b>3</b> explained below), to restore the lost packet using error correction coding information that is sent over on the network <b>21</b>, and remove error correction code and transmit the main body of the data to the next stage. The shaping point <b>13</b>′, when packets reach in excess of the buffer capacity of the wireless base station <b>10</b>, discards a portion of the packets. That is, the shaping point <b>13</b>′ is the means for adjusting the rate of packets into the wireless link. The FEC generator <b>18</b> functions, in the mode for using the FEC to restore the lost packet on the wireless link between the wireless device <b>14</b> and the wireless terminal <b>19</b> (Mode <b>2</b> and Mode <b>3</b> explained below), to generate the error correction code to be attached to the data stream that will be sent over the wireless link.

0066<figref idref="DRAWINGS">FIG. 8</figref> is a diagram showing an embodiment in a case where, in the system shown in <figref idref="DRAWINGS">FIG. 7</figref>, the media delivery device employs a format switching system. It corresponds to the embodiment in <figref idref="DRAWINGS">FIG. 2</figref>. A multimedia stream detecting section <b>37</b>′ detects the packet containing the RTP header, from among the IP packets arriving at the wireless base station <b>10</b>, and detects multimedia stream based on the pair of the source address and destination address, and the pair of the source port number and the destination port number, of the packet.

0067An RTP header monitor section <b>39</b>′ monitors the RTP headers contained in the IP packets of the multimedia stream detected by the multimedia stream detecting section <b>37</b>′, and inspects the sequence numbers of the packets contained in the RTP headers. Based on the inspection of the sequence numbers across multiple packets, when it is discovered that any sequence number is lost, the RTP header monitor section <b>39</b>′ transmits information for identifying the lost IP packet, to the feedback generator <b>41</b>′. In one embodiment, the information for identifying the IP packet includes the source address, the destination address, and the lost RTP packet's sequence number. Furthermore, in another embodiment, the RTP monitor section <b>39</b>′ does not transmit the sequence number of the lost RTP packet, but transmits the sequence number of each RTP packet that is detected to the feedback generator <b>41</b>′.

0068The feedback generator <b>41</b>′ returns the sequence number of the packet to the source address as an ACK upon receiving each RTP packet. In an alternative embodiment, the feedback generator <b>41</b>′, in response to the transfer of the information from the RTP header monitor section <b>39</b>′, judges whether or not there is any lost RTP packet during a predetermined time based on the sequence of RTP packet's sequence numbers, and then transmits the sequence number of the lost RTP packet toward the source address. In yet another alternative embodiment, instead of making a response to the source address each time the individual RTP packets are received or each time the lost RTP packet is found, the response is given with respect to multiple RTP packets at a time. To explain it in more detail, the received/lost data indicating whether each packet was received normally or was lost, is returned for every N number of packets. The number N of packets that are grouped together is determined such that the response delay time falls within such a range that enables retransmission of the packet from the source address in time.

0069<figref idref="DRAWINGS">FIG. 9</figref> is a sequence chart showing a communication sequence in the system in <figref idref="DRAWINGS">FIG. 8</figref>. As was explained with reference to <figref idref="DRAWINGS">FIG. 6</figref>, the media delivery device <b>11</b> has the RTP processing section <b>111</b> and the RTCP processing section <b>113</b>. The RTCP processing section <b>113</b>, before transmitting a PTCP packet, sends out the RTCP packet <b>115</b> containing sender information (SDES) to the wired network <b>21</b> toward the wireless base station <b>10</b>. The first feedback section <b>71</b> of the wireless base station <b>10</b>, in response to the RTCP packet <b>115</b> and the subsequent RTP packets <b>117</b>, <b>119</b>, and <b>121</b>, calculates the packet loss rate and the network transmission delay time of the wired network <b>21</b>, and then mounts this information onto the RTCP packet <b>133</b> and transmits this to the media delivery device.

0070The second feedback section <b>73</b> of the wireless base station <b>10</b>, upon receiving each RTP packet of the media stream, sends the reception acknowledge signal (ACK) with the sequence number of the RTP packet back to the media delivery device. In <figref idref="DRAWINGS">FIG. 9</figref>, the ACK <b>105</b> is returned in response to the reception of the RTP packet <b>117</b>, the ACK <b>107</b> is returned in response to the reception of the RTP packet <b>119</b>, and the ACK <b>109</b> is returned in response to the RTP packet <b>121</b>. Of course, instead of transmitting the sequence numbers of the packets that were received normally back to the media delivery device as explained with respect to the feedback generator <b>41</b>′ in <figref idref="DRAWINGS">FIG. 8</figref>, the numbers of the lost packets may be sent back, or the arrived/yet unarrived responses may be returned for multiple packets grouped together.

0071Referring to <figref idref="DRAWINGS">FIG. 8</figref>, the second feedback section <b>73</b> is provided with the multimedia stream detecting section <b>37</b>′, the RTP header monitor section <b>39</b>′ and the feedback generator <b>41</b>′. Functions of each of these sections are the same as those of the multimedia stream detecting section <b>37</b>, the RTP header monitor section <b>39</b> and the feedback generator <b>41</b>, which are included in the packet analysis device <b>15</b> explained with reference to <figref idref="DRAWINGS">FIGS. 2 and 3</figref>.

0072The RTCP processing section <b>113</b> of the media delivery device <b>11</b> transfers the packet loss rate and the delay time, which are contained in the RTCP packet <b>133</b>, to the feedback control section <b>35</b>. The feedback control section <b>35</b> selects the transmission mode for the media stream based on this information (<b>157</b>).

0073The above-mentioned three modes are explained below, but explanation is first given regarding a method of obtaining parameters that are needed for the media delivery device <b>11</b> to determine the transmission rate and the transmission mode for the media stream using two types of feedback, Net-Feeds and SP-Feeds, from the wireless base station <b>10</b>, and feedback (client feedback) from the wireless terminal <b>19</b> represented by a line <b>25</b> in <figref idref="DRAWINGS">FIG. 8</figref>.

0074First, symbols which will be used are defined as follows: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0075"> l₁Packet loss rate on the wired network <b>21</b></li><li id="ul0002-0002" num="0076"> R*₁Maximum transmission rate permitted on the wired network <b>21</b></li><li id="ul0002-0003" num="0077"> R₁Transmission rate from the media delivery device <b>11</b></li><li id="ul0002-0004" num="0078"> R′₁Reception rate at the wireless base station <b>10</b></li><li id="ul0002-0005" num="0079"> R₁⁽²⁾Internal rate in wireless base station, after the FEC decoder <b>72</b> eliminates error correction information for FEC, and before packet discard processing is performed at the shaping point <b>13</b>′</li><li id="ul0002-0006" num="0080"> R₁⁽³⁾Internal rate in wireless base station after leaving the shaping point <b>13</b>′</li><li id="ul0002-0007" num="0081"> l₂Packet loss rate on the wireless link (i.e., between wireless device <b>14</b> and the wireless terminal <b>19</b>) </li><li id="ul0002-0008" num="0082"> R*₂Maximum transmission rate permitted on the wireless link </li><li id="ul0002-0009" num="0083"> R₂Transmission rate from the wireless base station <b>10</b></li><li id="ul0002-0010" num="0084"> R′₂Reception rate at the wireless terminal <b>19</b></li><li id="ul0002-0011" num="0085"> r Media stream coding rate </li></ul></li></ul>

0086Here, the maximum transmission rate R*₂permitted on the wireless link, R*<b>2</b>, can be determined at the wireless base station <b>10</b> by using one of the following two methods.

0087The first method uses ideal maximum transmission rate information in the process of setting the wireless link. That is, when setting the wireless link, resources assigned to this wireless link, specifically a bandwidth, is determined, so that based on this bandwidth the ideal or maximum possible transmission rate is determined. This value is utilized as R*₂. In this method, the rate used in the session where this wireless link is used, is fixed except in the case where the wireless base station to be used is replaced, such as when the location of the wireless terminal <b>19</b> is changed.

0088The second method is a method of deriving the maximum transmission rate R*₂based on the filling rate of the transmission buffer in the wireless base station <b>10</b> that has the packets that stand ready to be transmitted from the wireless base station <b>10</b> to the wireless link. That is, if the transmission buffer's filling rate drops or the buffer keeps empty, then it is understood that the current transmission rate is below the maximum rate and vice versa, if the filling rate is increasing. The rate R*₂obtained in this way fluctuates depending on the time in many cases.

0089The maximum transmission rate R*₂can be determined by, for example, the wireless device <b>14</b> in the base station <b>10</b>, or by a device or the like for controlling the wireless device <b>14</b>.

0090The transmission rates of the media delivery device <b>11</b> and the wireless base station <b>10</b>, R₁and R₂(i.e., the rates at the transmission ends), are the rates for all packets relating to the media stream, that is, all packets including not only the main bodies of the data of the media stream but also the packets for FEC which are added thereto, and retransmitted packets. Because of packet loss that can occur on the wired network or on the wireless link, rates at the respective reception ends, R′₁and R′₂, drop below the rates at the transmission ends. In other words, the following formula is established. <br />R′₁≦R₁, R′₂≦R₂

0091When the packets arrive at the wireless base station <b>10</b> from the wired network, first they are given to the FEC decoder <b>72</b>. If FEC has been applied, then the packets are decoded. As a result, the rate R₁⁽²⁾after the packets leave the FEC decoder drops below the reception rate R′₁of the wireless base station.

0092Here, in the FEC used in this embodiment, the output resulting from applying FEC consists of the original data packet and parity packet. Furthermore, the party packet is sent in a separate stream from the original data packet. Regarding this type of FEC, refer to, for example, RFC2733 (An RTP Payload Format or Generic Forward Error Correction). The FEC decoders (the FEC decoder <b>72</b> provided to the wireless base station <b>10</b> and the FEC decoder (not shown) provided to the wireless terminal <b>19</b>) use the parity packet sent in the other stream, to reconstruct the lost original data packet.

0093Before the packet leaves the wireless base station, the FEC generator <b>18</b> compares the rate R₁⁽³⁾after leaving the shaping point <b>13</b>′, with the maximum rate R*₂on the wireless link. If R₁⁽³⁾&lt;R*₂, then the FEC generator <b>18</b> uses the surplus bandwidth, R*₂−R₁⁽³⁾, for the FEC to restore the packets lost on the wireless link. Eventually, the transmission rate R₂from the wireless base station <b>10</b> approaches the maximum transmission rate R*₂permitted on the wireless link.

0094The shaping point <b>13</b>′ is provided between the FEC decoder <b>72</b> and the FEC generator <b>18</b>. When the rate R₁⁽²⁾after eliminating the FEC on the wired network <b>21</b>, is greater than the wireless link's bandwidth, that is when it is greater than the maximum rate R*₂, the packet is discarded. Furthermore, duplicate packets having the same RTP sequence numbers can be detected and eliminated here as well.

0095The two types of feedback, Net-Feeds and SP-Feeds, from the wireless base station <b>10</b> to the media delivery device <b>11</b>, are sent immediately before the FEC decoder <b>72</b> and the FEC generator <b>18</b>, respectively, via the first feedback section <b>71</b> and the second feedback section <b>73</b>.

0096The first feedback section <b>71</b>, which performs feedback before the shaping point <b>13</b>′, sends the feedback Net-Feeds to the media delivery device <b>11</b>, to notify the current status on the wired network. The feedback Net-Feeds is statistical feedback, and includes information collected at a packet window, such as a packet loss rate in amid-range time frame (e.g., several seconds) and the average and variance of round-trip time. This type of statistical feedback is commonly known. If necessary, refer to, for example, RFC1889 (Rtp: A transport protocol for real-time application).

0097The second feedback section <b>73</b>, which performs feedback after the shaping point <b>13</b>′, sends the feedback SP-Feeds in the form of a packet reception acknowledge packet (ACK) to the media delivery device <b>11</b>, so that the media delivery device <b>11</b> can determine which packet is lost before the wireless transmission. The feedback SP-Feeds is sent out in a short period (e.g., within 1 second).

0098In addition to the feedback Net-Feeds and SP-Feeds, in this embodiment, the wireless terminal <b>19</b> also sends out feedback at relatively short intervals with little delay, to the media delivery device <b>11</b>.

0099These types of feedback enable the media delivery device <b>11</b> to obtain information as listed below.

0100First, using the feedback Net-Feeds, the media delivery device <b>11</b> can know the maximum transmission rate R*₁permitted on the wired network. Furthermore, the packet loss rate l₁on the wired network is explicitly shown in the feedback Net-Feeds.

0101The feedback SP-Feeds offers feedback in high resolution at the ends of the wired network, which is to say feedback on the packet basis. Therefore, the media delivery device <b>11</b> can accurately grasp which packets have properly arrived at the wireless base station <b>10</b>.

0102Furthermore, using the feedback SP-Feeds and the client feedback from the wireless terminal <b>19</b>, the media delivery device <b>11</b> can the obtain packet loss rate l₂on the wireless link, and the maximum transmission rate R*₂permitted on the wireless link, in the following manner. That is, the feedback SP-Feeds is compared against the client feedback for only the original data packet stream where the parity packet stream for FEC has been eliminated. Thus, the media delivery device <b>11</b> can obtain the packet loss rate after the FEC is applied. Furthermore, from the client feedback for the parity packet stream, the media delivery device <b>11</b> can grasp how much FEC has been applied on the wireless link, which is to say the bandwidth used for the parity packet stream. If the packet loss rate after the FEC is applied, and the bandwidth for parity packet stream (i.e., the parity packet rate) for the FEC, are both known, then it becomes possible to obtain the “original” loss rate l₂on the wireless link before the FEC is applied.

0103In this embodiment, the wireless base station <b>10</b> automatically selects one of the three modes, based on the delay characteristic required of the media stream, which is to say the receiving buffer capacity of the wireless terminal <b>19</b>, and then operates. Hereinafter, explanation is given regarding these modes, Mode <b>1</b> through Mode <b>3</b>.

0104Mode <b>1</b>

0105This mode is applied in a case where the wireless terminal <b>19</b> (client) is provided with a large receiving buffer, which can allow significant jitter in the packet arrival timing, which is caused by retransmitting lost packets the requisite number of times on the wireless link. Note that, in third-generation mobile phone communication scheme, the wireless base station <b>10</b> may set up the wireless terminal <b>19</b> so as to retransmit on the link layer as is commonly known to those skilled in the art. Its specific construction or operation itself is not directly relevant to the present invention. If necessary, refer to H. Holma and A. Toskala, Eds., WCDMA for UMTS: Wireless Access for Third Generation Mobile Communications, Wiley, 2001. Therefore, in order to realize Mode <b>1</b>, a function for retransmitting packets by using third-generation mobile phone link layer can be used. That is, if the wireless base station <b>10</b> sets an error control mode for the wireless terminal <b>19</b> to the retransmitting mode, then, in a manner that is transparent from the application layer, the link layer will perform operations as follows. That is, at the link layer, when a lost packet is detected at the wireless terminal <b>19</b>, the wireless terminal <b>19</b> requests the wireless base station <b>10</b> to retransmit the given packet. When the wireless base station <b>10</b> receives the retransmission request, it compares the number of times the given packet has been retransmitted to date, with the maximum number of retransmission times which was set when that mode was set, and then notifies retransmission failure to the wireless terminal <b>19</b> in a case where the maximum retransmission times has already been reached. In a case where maximum retransmission times has not been reached yet, the wireless base station <b>10</b> retransmits to the wireless terminal <b>19</b> one of the already-transmitted packets that is temporarily stored and corresponds to the lost packet. The maximum retransmission times of the lost packet is determined based on the packet loss rate l₂on the wireless link, and a target value for improving the packet loss rate by performing retransmission, but an appropriate or standard value maybe known in advance, or the value maybe determined by making measurements as described above. The wireless terminal <b>19</b> that has received the retransmitted packet, uses the retransmitted packet to restore the media stream. Therefore, in terms of the application layer, the original error control mode was simply set to the retransmitting mode described above, and thus it looks as though the media stream has been received from the wireless base station <b>10</b> without any errors. Accordingly, while there is essentially no packet loss in the wireless link from the wireless base station <b>10</b> to the wireless terminal <b>19</b>, a great deal of variation does occur in the packet transmission delay.

0106Furthermore, using the feedback Net-Feeds from the wireless base station <b>10</b>, the media delivery device <b>11</b> performs congestion control on the wired network.

0107Furthermore, using the feedback SP-Feeds from the wireless base station <b>10</b>, the retransmission of the lost packet on the wired network is performed on the application layer. In Mode <b>1</b>, there are virtually no lost packets on the wireless link, so that the feedback from the wireless base station <b>10</b> is substantially the same as the feedback from the wireless terminal, but the feedback SP-Feeds arrives at the media delivery device <b>11</b> considerably faster than the feedback from the wireless terminal.

0108As compared to FEC, where the error correction code is always added, it is more efficient to perform packet retransmission only when a packet is actually lost. In Mode <b>1</b>, the delay and jitter which accompany packet retransmission, do not become a problem, so that retransmissions, and not FEC, are employed on both the wired network and the wireless link.

0109Mode <b>2</b>

0110This mode is used in a case where the size of the receiving buffer in the wireless terminal <b>19</b> is sufficient for the lost packet to be retransmitted the requisite number of times on the wired network, but is not large enough to allow such retransmission on the wireless link.

0111In Mode <b>2</b>, when the packet has been lost on the wired network, in a similar fashion to Mode <b>1</b>, the feedback SP-Feeds from the wireless base station <b>10</b> is used by the media delivery device <b>11</b> to retransmit the lost packet, and when the packet is lost on the wireless link between the wireless base station <b>10</b> and the wireless terminal <b>19</b>, this is solved by applying FEC.

0112Normally, the transmission rate on the wireless link is smaller than the wired network <b>21</b>. Therefore, the typical case is assumed here, that is, R*₁>R*₂. As described above, the media delivery device <b>11</b> can obtain R*₁, R*₁, and l₂from the feedback information. Accordingly, the media delivery device <b>11</b> can appropriately select a media coding rate (i.e., the transmission rate for the media stream) r. The bandwidth R*₂−r which is not used for the media stream on the wireless link, resists the loss rate l₂on the wireless link as error correction is performed. Thus, an error correction parity packet stream which is sufficient for restoring virtually all the lost packets can be transmitted.

0113On the wired network, the surplus bandwidth R*₂−r is used, to perform retransmission on the application layer. Furthermore, since duplicate packets are discarded at the wireless base station <b>10</b>, such packets do not occupy bandwidth on the wireless link.

0114Mode <b>3</b>

0115This mode is applied in a case where, because the buffer capacity of the wireless base station <b>10</b> is extremely small, any delay cannot be allowed, which is caused by not only the retransmissions on the wireless link, but also by retransmitting the lost packet the requisite number of times on the wired network.

0116In Mode <b>3</b>, the media delivery device <b>11</b> applies FEC adapted appropriately for the error rate and other such error characteristics on the wired network, and then sends out the media stream toward the wireless base station <b>10</b>. The parity stream for this FEC is eliminated by the FEC decoder <b>72</b> at the wireless base station <b>10</b>. The FEC generator <b>18</b> then applies another FEC adapted appropriately for the error characteristics of the wireless link, and then sends out to the wireless terminal <b>19</b>.

0117Referring back to <figref idref="DRAWINGS">FIG. 9</figref>, the feedback control section <b>35</b>, based on the information contained in the reception report from the wireless base station <b>10</b>, which is transferred over from the RTCP processing section, determines which of the three transmission modes described above is appropriate, and then determines the transmission mode (<b>157</b>).

0118The feedback control section <b>35</b>, based on the RTP packet reception acknowledge signal (ACK) that is sent over from the second feedback section <b>73</b>, determines whether packets in the media stream are lost (<b>141</b>). When it is detected that a packet is lost, the feedback control section <b>35</b> then determines whether or not the communications are being conducted in Mode <b>1</b> or Mode <b>2</b> (<b>151</b>). When the communications are being conducted in Mode <b>1</b> or Mode <b>2</b>, the packet retransmission procedures explained with respect to the blocks <b>143</b>, <b>145</b>, <b>147</b>, and <b>149</b> in <figref idref="DRAWINGS">FIG. 6</figref> are then executed (<b>153</b>). When the communications are being conducted in Mode <b>3</b>, if possible and desirable, the error correction code to be used is modified to a more appropriate one (<b>155</b>).

0119Explanations have been given so far regarding the specific embodiments of the present invention, but the present invention is not limited to the above-mentioned embodiments.