Adaptive forward error correction

This application claims the benefit of US provisional application number 60/632489 (agent reference number PU040331), filed December 2, 2004, entitled "ADAPTIVE FORWARD ERROR CORRECTION (FEC) FOR VIDEO STREAMING OVERWIRELESS NETWORKS". The entire US Provisional Application No. 60/632489 is incorporated herein by reference in its entirety.

The present invention generally relates to forward error correction (FEC).

Often, wireless network error rates are not sufficient for video applications due to the loss / drop of many large packets that can occur. Dropped or lost packets are often not recovered. However, in methods where packet recovery is performed, packets are recovered using retransmission, forward error correction (FEC), or a combination of both. FEC is widely used to correct errors without requiring retransmission. The FEC recovers the data contained in a corrupted, dropped, or lost packet by transmitting redundant information that can be used by the receiver to reconstruct the missing data. FEC provides faster data recovery with retransmissions and does not require feedback channels. Static FEC technology, which has been used by many researchers, adds FEC overhead to channels and networks.<u style="single">Status</u>Cannot be adapted to.

The use of FEC in wireless networks is an active area of research for wireless video applications. One advantage of the FEC is that it works well with multicast. Also, FEC does not require interaction with a video encoder and is therefore applicable to any video coding technique and to both storage and live video. However, static FEC algorithms can degrade performance due to the imbalance between network resources and the amount of redundancy that the algorithm adds that consumes the limited bandwidth of the wireless network.

Adaptive FEC technology has been shown to benefit network performance. All adaptive technologies require feedback to estimate the available network bandwidth, and then adapt the redundancy amount based on that feedback. Feedback may be explicit, such as in Real-time Control Protocol (RTCP) reports, or implicit, such as the use of packet retransmissions in the lower layers of wireless transmitters. Generally, the adaptive mechanism is the network<u style="single">Status</u>If is poor, increase the number of redundant packets to send. network<u style="single">Status</u>If is satisfactory, then no redundant packets need to be transmitted, or only a small number of redundant packets need to be transmitted. Many conventional methods target bit-level FEC.

These and other drawbacks and disadvantages of the prior art are solved by the present invention for adaptive forward error correction (FEC). One application of the adaptive FEC of the present invention is video streaming over a wireless network.

<p> Adaptive FEC equipment is provided.</p>

<p> According to another aspect of the invention, an adaptive FEC device is provided. The device includes a FEC coder and an adaptive FEC device. The FEC encoder encodes the source data of k packets into n packets. Where n&gt; k, n packets have redundant packets. The adaptive FEC device adapts and determines the number of redundant packets to send with k encoded packets based on at least one feedback message. At least one feedback message may indicate the condition or state of the network to which the FEC-encoded packet should be sent.</p><p> These and other aspects, features and advantages of the present invention will become apparent from the following detailed description of the examples which are examples to be read in connection with the accompanying drawings.</p><p> The present invention can be better understood according to the figure, which is an example below.</p>

The present invention is directed to adaptive forward error correction (FEC) in a preferred embodiment of video streaming over a wireless network. The present invention is a network<u style="single">Status</u>Provided are adaptive FEC methods and devices that improve video streaming performance by dynamically adjusting the FEC intensity based on the above. The number of redundant packets sent over the wireless network is the network<u style="single">Status</u>Optimized based on. The feedback signal is networked to the adaptive FEC device on the transmitting side.<u style="single">State</u>Used to inform. The feedback signal can originate from, but is not limited to, the receiver, intermediate node or transmitter link layer. network<u style="single">Status</u>Monitoring can be done on the transmitting side by accessing information from the transmitter's wireless hardware, such as a retransmission attempt, or by using the RTCP report if real-time control protocol (RTCP) is used. .. The use of sender information is network because RTCP reports are not sent frequently to save network resources.<u style="single">Status</u>Provides a faster response to changes in. However, RTCP reports have more information and can be used to more accurately estimate available network bandwidth. A combination of feedback messages may also be used. network<u style="single">Status</u>Based on, the adaptive FEC device determines how many redundant packets to send. network<u style="single">Status</u>If is inferior, then the network<u style="single">Status</u>But<u style="single">Good</u>More redundant packets are sent than in the case. It should be understood that the present invention describes packet level loss FEC, but other types of packet level FEC may also be utilized in accordance with the present invention while maintaining the spirit and scope of the present invention. is there.

The following description describes the principles of the embodiments of the present invention. Thus, one of ordinary skill in the art will practice the principles of the present invention and devise various configurations that are not explicitly stated or shown in the examples described herein, but are within the spirit and scope of the present invention. Can be.

All examples and conditional statements described herein are for the purpose of teaching to assist the reader in understanding the concepts devised by the inventor to develop the principles and prior art of the present invention. It should be considered not limited to the examples and conditions described.

Moreover, all text of the specification that describes the principles, aspects and embodiments of the invention includes both structural and functional equivalents, as in certain examples of the invention. Furthermore, such equivalents also include currently known equivalents, as well as future developed equivalents, that is, any developed element that performs the same function regardless of structure.

Thus, for example, those skilled in the art will appreciate that the block diagram presented herein represents a conceptual diagram of a circuit for illustration that implements the principles of the present invention. Similarly, any flowchart, flow diagram, state transition diagram, pseudocode, etc. may be executed by a computer or processor regardless of whether it is present on a substantially computer-readable medium and the computer or processor is explicitly indicated. It is understood to represent the various processes that can be performed.

The functionality of the various elements shown in the figure may be provided through the use of dedicated hardware, hardware that can execute the software associated with the appropriate software. When provided by a processor, functionality may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors that may be partially shared. Moreover, the explicit use of the term "processor" or "control" should not be considered to exclude hardware that can run the software. Also, the use of these terms includes digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), and non-volatile memory. Not limited to these.

Other conventional and / or customary hardware is also included. Similarly, any switch shown in the figure is conceptual. These functions may be performed through the program logic, through the dedicated logic, through the interaction of the program control and the dedicated logic, or manually. The individual techniques can be selected by the practitioner for further understanding from the context.

The claims of the present specification shall include any element expressed as a means of performing a particular function, and any method of performing the function, including (a) and (b) below. For example, (a) a combination of circuit elements that perform the function, or (b) any form of software combined with an appropriate circuit that executes the software to perform the function, thus firmware, microcode, etc. Include. The present invention as defined by such claims is based on the fact that the functions provided by various described means are combined and performed in conjunction with the method claimed by the claims. Applicants therefore consider any means capable of providing these functions as an equivalent of those set forth herein.

Packet level loss FEC is used to improve the reliability of wireless connections. Disappearance is easier to handle because the exact location of the error is known. The (n, k) vanishing FEC scheme encodes k source packets into n (n&gt; k) packets. The coding is sufficient to reconstruct the source data in any subset with k packets.

Referring to FIG. 1, an example coding / decoding FEC process is generally indicated by reference number 100. The source packet (k original packets) 110 is input to the encoder 120. Then, n packets 130 are output by the encoder 120. The n packets 130 have redundant packets 132 (ie, the shaded block of FIG. 1) along with the coded data corresponding to the k packets 131. Inserting a large number of FEC packets may be used to improve error correction performance over a large number of FEC blocks. The decoder 140 decodes n packets (having the original k packets 110 and redundant packets) to obtain the reconstructed data 150 (identical to the original k packets 110).

Wireless network packet loss rate is time, network<u style="single">Status</u>, Network load, etc. Therefore, the adaptive forward error correction control method is very useful for adjusting the amount of redundancy transmitted over the channel. Feedback-based adaptive forward error correction methods and devices are provided. network<u style="single">Status</u>But<u style="single">Good</u>In this case, it is not necessary to send a large number of redundant packets. In contrast, the higher the packet drop rate (above a predetermined threshold), the more redundant packets will be sent.

With reference to FIG. 2, the standard wireless video system architecture is generally indicated by reference number 200. The sender (having elements (205, 210, 215, 220, 225 and 230)) is a FEC-protected video source that may be stored or live content. The wireless video system architecture 200 has a video store having an output connected by signal communication with the input of the FEC encoder 210. Optionally, a video capture unit 215 having an input of the video encoder 220 and an output connected by signal communication may be used in place of the video store 205. The video encoder 220 also has an output connected by signal communication to the input of the FEC encoder 210. The output of the FEC encoder 210 is connected to the first input of the buffer 225 by signal communication. The output of buffer 225 is connected to the input of wireless network interface 230 by signal communication. The first output of the wireless network interface 230 is connected by signal communication to a wireless connection (also referred to herein as "wireless network") 235. The first and second inputs of the adaptive FEC device 240 are signaled to the second output of the wireless network interface 230 and the wireless connection 235, respectively. The output of FEC device 240 is connected by signal communication with the second input of buffer 225. The receiver 245 is connected to the wireless connection 235 by signal communication.

The transmitting FEC encoder 210 encodes the packet. Each packet has an integer number of macroblocks and a fixed number of bytes. The sender then packs the video data in real-time protocol / user datagram protocol (RTP / UDP) before it is passed over the wireless network interface 230 to the wireless network 235.

The sender transmits all the original packets without rest. The adaptive FEC block determines the number of redundant packets to send based on one or more feedback messages (hereinafter referred to as "feedback messages"). The feedback message can come from the receiver. The feedback message can then be sent to the transmitter via RTCP or obtained from the transmitter's wireless retransmission information. Radio retransmission information measures the number of times a packet has been retransmitted, and the network<u style="single">State</u>It can be used as an index. An effective adaptive approach uses both information with RTCP for a longer-term network.<u style="single">Status</u>I will provide a. At the same time, the sender information is on the network<u style="single">Status</u>Quick adaptation to<u style="single">To</u>To<u style="single">Do</u>Can be used for

The adaptive FEC device 240 determines the number of packets to be sent based on the feedback message. Many alternative techniques can be used to determine the quality of packets sent out. For example, one embodiment of the present invention utilizes an increase / decrease function. According to the principle of this embodiment, a "satisfactory" feedback message reduces the number of redundant packets sent by 1 to 0. In other cases, the number of redundant packets is increased by 1 to (nk) packets. Larger values of nk should be used because they have good adaptive rates.

In another alternative embodiment, adjustments may be made using the RTCP receiver report. In particular, the partial loss and cumulative number of lost packets RTCP receiver report fields can be used for tuning. These fields represent the number of lost packets since the last receiver report and the total number of lost packets since the start of transmission, respectively.

In many real 802.11 wireless local area network (WLAN) connections, the actual throughput is stable at some point and in some cases, and in fact the sending device attempts to transfer more data through the connection. So it goes down. This is illustrated in FIG. In FIG. 3, the curve of the receive bit rate vs. the provided transmit rate of the WLAN connection is generally indicated by reference number 300.

If the video transmitter (included in wireless network interface 230) is sending data at a rate that increases the reception rate as the transmission rate provided increases, the FEC will improve the actual amount of information received. However, when the video transmitter is operating on a flat, or worse, diminishing curve, FEC is useless and reduces the actual received video information when trying to send more data. obtain. For this reason, it is important for video transmitters to have knowledge of where in the curve of FIG. 3 the video transmitter actually operates. If the video transmitter is operating in the reduced part of the curve, the video transmitter should not increase the amount of FEC data, and in some other way (eg lower the bit rate) to improve performance, Therefore, an attempt should be made to reach the increasing part of the curve. If the video transmitter is already operating on the increasing part of the curve, the video transmitter may add additional FEC information to the good results.

One way to know where the video transmitter is on the curve is to use feedback from receiver 245 (eg RTCP report) or wireless transmitter retransmission information. If FEC is added and the video transmitter 245 receives a lower packet rate, the transmitter is operating in the reduced part of the curve. If FEC is added and the video transmitter 245 receives a higher packet rate, the transmitter is operating in the increasing part of the curve. This information is already available to the video transmitter if the video transmitter has already received information about the dropped packet through a feedback path. Alternatively, medium access control (MAC) level retransmission information from the radio transmitter can be used to estimate the number of dropped packets, as already described.

With reference to FIG. 4, adaptive FEC methods for video streaming over wireless networks are commonly indicated by reference number 400.

Functional block 405 encodes a video stream, generates fixed size video packets (eg MPEG2 transport packets), and passes control to functional block 410. In functional block 405, the number of redundant packets sent out (hereinafter also referred to as the variable "X") is equal to zero (0).

Functional block 410 encodes each k source packet using the loss code FEC, generates n packets with (nk) redundant packets, and then passes control to functional block 415. Functional block 415 sends out the first k source packets and passes control to decision block 420. Decision block 420 is the network<u style="single">Status</u>To measure and determine. network<u style="single">Status</u>If is good (ie, above a predetermined threshold), control is passed to functional block 425. Other networks<u style="single">Status</u>If is inferior (ie, below a predetermined threshold), control passes to decision block 430.

Functional block 425 sends X = max {X-1,0} redundant packets and returns control to functional block 410. That is, the functional block 425 sends out one less redundant packet than the number previously sent, with zero redundant packets as the lower limit. The determination block 430 determines the operating area of the video transmitter with respect to the received bit rate vs. the provided transmit rate (curve shown in FIG. 3). In particular, the determination block 430 determines whether the transmitter is operating in the increasing region of the curve in FIG. 3 or in the non-increasing region of the curve. If the transmitter is operating in the non-increasing region of the curve, control passes to decision block 440 and sends out X redundant packets. Control then returns to functional block 410. In addition, when the transmitter is operating in the increasing region of the curve, the control passes to the functional block 450 and sends out X = min {X + 1,nk} redundant packets. Control then returns to functional block 410. That is, the functional block 450 sends out one more redundant packet than the number previously sent, up to (nk) redundant packets. Returning to functional block 410 by functional blocks 425, 440 and 450 is a network<u style="single">Status</u>Makes the number of redundant packets sent based on adaptable.

These and other features and advantages of the present invention can be immediately elucidated by those skilled in the art based on the teachings herein. It is understood that the teachings of the present invention can be practiced in various forms of hardware, software, firmware, special purpose processors, or combinations thereof.

Most preferably, the teachings of the present invention are carried out as a combination of hardware and software. Further, the software is preferably implemented as an application program explicitly embedded in the program storage device. The application program may be uploaded and executed on a machine with any suitable architecture. Desirably, the machine is a computer platform with hardware such as one or more central processing units (CPU), random access memory (RAM), and input / output (I / O) interfaces. Will be implemented in. The computer platform may also have an operating system and microinstructions. The various processes and functions described herein may be part of a microinstruction code or part of an application program, or any combination thereof that can be executed by a CPU. In addition, various other peripherals such as additional data storage devices and printing devices may be connected to the computer platform.

Some of the components that make up the system and the methods shown in the drawings are preferably implemented in software, and the actual connection between the system components or between processing functional blocks depends on the method in which the present invention is programmed. It is further understood that they can be different. Those skilled in the art will be able to implement these and similar embodiments of the present invention according to the teachings of the present specification.

Although exemplary examples have been described herein with reference to the drawings, the invention is not limited to these detailed examples, and various changes and modifications are the scope or spirit of the invention. It is understood that it can be done by one of ordinary skill in the art without departing from. All changes and changes are included in the scope of the invention as defined in the claims.

<figref num="1">The figure of the coding / decoding forward error correction (FEC) processing which is an example to which this invention is applied is shown.</figref><figref num="2">The figure of the standard wireless video system architecture to which this invention is applied is shown.</figref><figref num="3">The curve of the receive bit rate vs. the offer transmit rate of the WLAN connection is shown.</figref><figref num="4">A flow diagram of an adaptive FEC method for video streaming over a wireless network according to the principles of the present invention is shown.</figref>