Determining congestion measures

<u>Technical Field</u>

0001The present invention relates to methods and apparatus for determining congestion measures.

0002More specifically, aspects of the present invention relate to methods and apparatus for determining congestion measures in respect of one or more of a plurality of users such as senders or receivers of data using a shared resource such as a data network or a forwarding node therein.

<u>Background to the Invention and Prior Art</u>

0003We start by presenting, as background information, some basic concepts to facilitate the understanding of the numerous monitoring and policing techniques that are presented afterwards.

Packets

0004A data sender usually splits data to be sent into small units known as packets. Each packet consists of a header and a payload carrying the data to be delivered. The header contains fields defined by the relevant communication protocol. The great majority of packets carried by commercial networks nowadays are so-called IP packets. IP is the Internet Protocol. This ensures that a network of routers can forward any packet from the source to its destination. IP is a connectionless protocol ― that means that the header information in each data packet is sufficiently self-contained for routers to deliver it independently of other packets; each packet could even take a different route to reach the destination.

Distributed bandwidth sharing and congestion

0005Data traversing the Internet follows a path between a series of routers, controlled by various routing protocols. Each router seeks to move the packet closer to its final destination. If too much traffic traverses the same router in the network, the router can become congested and packets start to experience excessive delays whilst using that network path. Between routers, data also traverses switches and other networking equipment that may also become congested. Throughout the following description the term router congestion will be used to imply congestion of any network equipment, without loss of generality. If sources persist in sending traffic through that router it could become seriously overloaded (congested) and even drop traffic (when its buffers overflow). If sources still persist in sending traffic through this bottleneck it could force more routers to become congested, and if the phenomenon keeps spreading, that can lead to a congestion collapse for the whole Internet ― which occurred regularly in the mid-1980s.

0006The solution to that problem has been to ensure that sources take responsibility for the rate at which they send data over the Internet by implementing congestion control mechanisms. Sources monitor feedback from the receiver of the metric that characterises path congestion in order to detect when the path their data is following is getting congested, in which case they react by reducing their throughput ― while they may slowly increase their rate when there is no sign of the path becoming congested.

0007The typical path characterisation metrics that sources monitor are the average roundtrip time (RTT) for the data path, the variance of the roundtrip time (jitter) and the level of congestion on the path. Congestion is one of the parameters controlling rate adaptation of a source sending data over a congested path.

0008The congestion level can be signalled either implicitly (through congested routers dropping packets) or explicitly (through mechanisms such as explicit congestion notification ― see next subsection). Currently the most common option is implicit signalling.

0009Sources using TCP are able to detect losses, because a packet loss causes a gap in the sequence; whenever a TCP source detects a loss, it is meant to halve its data transmission rate, but no more than once per round trip time, which alleviates the congestion on the router at the bottleneck.

Random Early Detection (RED)

0010Historically, routers would drop packets when they got completely saturated (which happens when a traffic burst cannot be accommodated in the buffer of the router) ― this policy is called drop-tail. Random early detection (RED) (discussed in reference "[RED]", bibliographic details of which are given later) is an improvement whereby routers monitor the average queue length in their buffer and when the average queue is higher than a given threshold, the router starts to drop packets with a probability which increases with the excess length of the queue over the threshold. RED is widely used in today's Internet because it avoids all flows receiving congestion signals at the same time (termed synchronisation) which would otherwise cause oscillations. RED also allows sources to react more promptly to incipient congestion and it keeps queues from growing unnecessarily long.

Explicit Congestion Notification

0011Explicit Congestion Notification (ECN) (see reference "[RFC3168]") further improves on RED by using a two-bit ECN field in the IP header to signal congestion. It runs the same algorithm as RED, but instead of dropping a packet, it sets its ECN field to the Congestion Experienced (CE) codepoint. The ECN standard requires the receiver to echo any congestion mark signalled in the data; for instance, a TCP receiver sets the Echo Congestion Experienced (ECE) flag in the TCP header, which the TCP source interprets as if the packet has been dropped for the purpose of its rate control. In turn the source then reacts to the congestion by halving its transmission rate.

0012ECN was originally defined for DECnet, the proprietary networking protocol developed by the Digital Equipment Corporation [DECbit]. As well as the idea being adopted in IP, it was also adopted in Frame Relay and ATM, but in these latter two protocols the network arranges feedback of the congestion signals internally, and the network enforces traffic limits to prevent congestion build-up (see [ITU-T Rec.l.371]).

0013The IEEE has standardised an explicit congestion approach where Ethernet switches not the end systems arrange to feedback the congestion signals, although the Ethernet device on the sending system is expected to co-operate by reducing its rate in response to the signals. The approach is tailored exclusively for homogeneous environments, such as data centres.

0014In the previously described approaches, each frame (or packet) carried just a binary flag and the strength of the congestion signal depended on the proportion of marked frames ― effectively a unary encoding of the congestion signal in a stream of zeroes and ones. However, the IEEE scheme signals a multibit level of congestion in each feedback frame, hence its common name: quantised congestion notification or QCN (see [IEEE802.1 Qau]).

Re-ECN

0015Re-ECN (see [re-ECN]) utilises a technique called re-feedback (discussed in [re-feedback] and in International application <patcit id="pcit0001" dnum="WO2005096566A"><text>WO2005/096566</text></patcit>) whereby packets indicate the congestion they are likely to experience on the rest of their path not just the congestion already experienced, that ECN indicates. It is similar to ECN but uses an extra unused bit in the packet header. This bit is combined with the two-bit ECN field to create four extra codepoints, as discussed in International application <patcit id="pcit0002" dnum="WO2006079845A"><text>WO2006/079845</text></patcit>.

0016The simplest way to understand the protocol is to think of each packet as having a different "colour" flag (where different "colours" correspond to different codepoints). At the start of a flow, a "green" flag ("FNE", meaning "Feedback Not Established") is used to indicate that the sender does not have sufficient knowledge of the path. Green flags are also used when the sender is unsure about the current state of the path.

0017By default packets are marked with "grey" flags. If they encounter congestion during their progress through the network the ECN marking applied by the congested router will be termed a "red" flag. The destination will feed back a count of the number of red flags it has seen. For every red flag it is informed of, the sender should mark an equivalent number of bytes it sends in a subsequent packet or packets with a "black" flag. The black flag re-echoes or reinserts the congestion feedback back into the forward-travelling stream of packets, hence the name "re-ECN". These black flags may not be modified once they have been sent by the sender. There is a small possibility that a black packet will in turn be marked red by a congested router, but the codepoints are chosen so that it is still possible to tell the packet was originally marked as black ― such packets are described as coloured "black-red".

0018At any intermediate node the upstream congestion is given by the proportion of red flagged bytes to total bytes. Thus the continually varying congestion level is effectively encoded in a stream of packets by interpreting the stream of red or non-red markings as a unary encoding of ones or zeroes respectively. Similarly, the congestion level of the whole path is encoded as a stream of black or non-black markings. The expected downstream congestion from any intermediate node can then be estimated from the difference between the proportions of black flags and of red flags, as described in International application <patcit id="pcit0003" dnum="WO2006079845A"><text>WO2006/079845</text></patcit>.

0019The IETF is in the process of defining an experimental change to the Internet protocol (IP) based on re-ECN, called Congestion Exposure (ConEx) (see [ConEx-abstr-mech]).

Discussion of Prior Techniques

0020The distributed congestion control responses to congestion of every data source do not necessarily share bandwidth equitably or efficiently. Firstly this approach relies on sources voluntarily responding in the prescribed way to the presence of congestion. Secondly, even if all sources respond as prescribed, the source of every data flow would not be taking account of how active or inactive it had been over time relative to others. Thirdly, equity should be judged between entities with real-world identities (e.g. users or customers of a network) not abstract data flows. Otherwise some real world entities can simply create many more data flows than others.

0021Due to this, network operators generally limit usage of a shared network. This is generically termed "policing".

0022The physical capacity of a communications link provides a natural physical limit on the bit-rate that the users of that link can achieve. The link provided to attach a customer site (e.g. home or business) to a network physically limits (or physically polices) the customer's traffic. However, often a logical rather than a physical limit is placed on the bit-rate to or from a customer site. This is because, as the peak bandwidth of access links has increased with advances in technology, average access link utilisation has decreased (currently 1% average utilisation during the peak period is typical). Therefore, when traffic from a large number of customers with low average utilisation is aggregated deeper into the network, it is uneconomic to provision shared capacity for the eventuality that every user might briefly use 100% of their access capacity.

0023Typically the average traffic from a large aggregate of customers is fairly predictable. It is possible for a network operator to provide enough capacity for this average, plus enough headroom to allow for daily variation. However, at peak times, everyone's experience is then determined by the heaviest users ― how many there are and how heavy they are.

Policing

0024A number of means have been devised to logically police usage of share capacity. Some are used in production networks, others are merely research proposals: <ul id="ul0001" list-style="none" compact="compact"><li><u>Token-bucket Policing:</u> With reference to <figref idref="f0001">Figure 1a</figref>, and as further discussed in [Turner86], the network operator allocates each customer i a contracted rate <b><i>u_i</i></b> and a contracted burst size <b><i>b_i.</i></b> A token bucket policer is associated with each customer, which is essentially an account that stores a single number d_i that characterises the customer's recent activity. Conceptually, d_i is the time-varying depth of fill of the customer's token bucket, which is filled with tokens at constant rate <b><i>u_i</i></b> and can store up to <b><i>b_i</i></b> tokens. A meter measures the customer's traffic and removes tokens from the bucket for every byte transferred. Therefore, a customer sending at time-varying bit-rate <b><i>x_i</i></b> will remove tokens from the bucket at rate <b><i>x_i.</i></b></li></ul>

0025A policer regulates the rate <b><i>y_i</i></b> at which the customer can send traffic dependent on the fill depth <i><b>d</b>_i</i> of the bucket. As long as the bucket is not empty (<i><b>d</b>_i</i>>0), the policer does not impede the customer's data flow <b><i>x_i.</i></b> But whenever there are insufficient tokens in the bucket (<b><i>d_i</i></b>=0), arriving data is discarded. If the customer is under-utilising the contract, the bucket will be full and further tokens filling the bucket will be discarded.

0026Variants are possible: e.g.: <ul id="ul0002" list-style="dash"><li>an overdraft at the bottom of the bucket where the probability of discard increases with the depth of the overdraft;</li><li>delay rather than discard (termed shaping rather than policing)</li><li>marking as out of contract, rather than discard (see RIO below).</li></ul>

0027<u>Paired token buckets</u>: A customer may be offered an assurance that they will always be able to use a certain bit-rate (their committed information rate or CIR), but they will also be allowed to use up to a peak information rate (PIR) if shared capacity is available. The two rates are also associated with allowed burst sizes above the rate: respectively the committed burst size (CBS) and the peak burst size (PBS). A CIR/PIR contract is generally policed by paired token buckets, filled respectively at the two rates and with depths of the two burst sizes respectively. This arrangement is typically called a three colour marker (TCM), because they often mark (or 'colour') outgoing traffic with one of three different classes of service depending on whether both, one or neither bucket is empty [RFC2697, RFC2698].

0028The CIR/PIR approach was common in Frame Relay and ATM, and it is common today on a shared link where the access capacity technology includes a mechanism for sharing out the capacity (e.g. time-division multiplexing in cable networks or passive optical networks and code-division multiplexing in cellular networks). Link capacity is provisioned so that it can support the sum of all the committed information rates. The approach is also used for whole networks, not just links, for example differentiated services (DiffServ) networks [RFC2698]. For a network, the committed rate may not be guaranteed - shared capacity may be provisioned so that it has a high probability of satisfying all the committed rates.

0029Another variant assures just one committed rate not two, with one associated committed burst size, but also a peak burst size is also allowed if available capacity permits [RFC2697].

0030<u>RED with in/out (RIO)</u>: RIO (further discussed in [Clark98]) separates the decision on which traffic is out of contract from the decision on whether to sanction out of contract traffic. It comes in two variants, each conceptually the dual of the other: <ul id="ul0003" list-style="dash" compact="compact"><li>Sender-based policing: At the ingress to a network any of the above policer designs can be used to determine which traffic is in-contract and which out. But out-of-contract traffic is merely tagged as such, rather than discarded. In fact, the customer can tag their own traffic to indicate which out-of-contract packets are least important to them; then the network operator merely has to check that the traffic tagged as in-contract, does actually fit within the contracted traffic profile.</li></ul>

0031If there is congestion at any forwarding node deeper into the network, packets tagged as out-of-contract can be discarded preferentially before in-contract packets are discarded. The RIO scheme proposed that nodes deeper into the network could simply run two instances of the RED algorithm, one with aggressive thresholds for out-of-contract traffic, and the other with a regular threshold configuration. <ul id="ul0004" list-style="dash" compact="compact"><li>Receiver-based policing: In this variant, the traffic is probabilistically marked with standard explicit congestion notification (ECN) if it passes through a congested queue. Then just before arriving at the receiver, the traffic is compared against the contracted profile using one of the policing techniques described above. But instead of discarding packets or tagging them out-of-contract, ECN markings are removed for all traffic within the profile.</li></ul>

0032<u>Weighted RED (WRED)</u>: WRED (further discussed in [WRED_Cisco]) is a variant of the sender-based version of RIO that has been widely implemented. Like RIO, on entry to the network, traffic is policed to a contract agreed with the customer. And like RIO, the policer tags rather than discards traffic that is out of profile. But rather than tag traffic as either in or out-of-contract, a WRED policer demotes out-of-contract traffic using potentially eight traffic class identifiers. For IP differentiated services, three classes are typically used, as standardised for the assured forwarding class of DiffServ [RFC2597]).

0033On interior routers, up to eight different sets of RED thresholds are configured for each class and one algorithm (rather than the two of RIO) determines the average queue length. Then each packet is compared against the thresholds relevant to its class, so that packets demoted to lower precedence classes will be more likely to be dropped.

0034<u>In-band-congestion-token-bucket policing</u>: Referring to <figref idref="f0002">Figure 1b</figref>, this is similar in operation to token bucket-policing, but it takes account of traffic only if it contributed to congestion. A prerequisite is that the proportion of the traffic's contribution to congestion elsewhere must have been tagged onto the traffic itself, as in-band congestion signalling. This is discussed further in [Jacquet08] and International application <patcit id="pcit0004" dnum="WO2006082443A"><text>W02006/082443</text></patcit>,

0035Typically each packet can either be marked or not, with a probability proportional to the congestion it has contributed to. This might be achieved with explicit congestion notification (ECN [RFC3168]) or congestion exposure (ConEx [ConEx-abstr-mech]). The meter measures only congestion marked packets and ignores the rest. It removes tokens from the congestion-token-bucket only for the bytes of marked packets. The network operator allocates each customer i a contracted congestion-bit-rate of <b><i>z_i</i></b> and a contracted congestion burst size <b><i>c_i.</i></b> Conceptually these are represented by a fill-rate and depth as with the traditional token bucket. Again, when a customer's congestion-token-bucket is empty, the policer limits their bit-rate.

0036Variants are possible: <ul id="ul0005" list-style="dash" compact="compact"><li>A dual token bucket might be used in which, as well as a defined token-fill-rate, the token-drain-rate is limited to a maximum. International application <patcit id="pcit0005" dnum="WO2010109201A"><text>W02010/109201</text></patcit> discusses this.</li><li>Instead of a binary congestion marking, each packet might be tagged with a real number between 0 &amp; 1 signifying the level of congestion it has experienced. For example the feedback frames in quantized congestion notification (QCN) [IEEE802.1Qau] are tagged in this way. Then, the meter would count the congestion-bytes to be removed from the bucket as the number of bytes in a data frame multiplied by the numeric congestion level associated with the frame.</li></ul>

0037<u>Weighted Fair Queuing (WFQ):</u> Referring to <figref idref="f0002">Figure 1c</figref>, WFQ partitions capacity between the entities actively using a link, without wasting capacity on inactive entities. Entities might be defined as whole customers or individual data flows. Each entity is associated with a weight, so that deliberately unequal shares can be provided. Traffic from each active entity is partitioned into separate queues. Access to the shared line is arbitrated by a scheduler, which serves each queue for a certain proportion of time, <b><i>w_i</i>/<i>Σw,</i></b> where <b><i>w_i</i></b> is the weight associated with entity <b><i>i</i></b> and <b><i>Σw</i></b> is the sum of the weights of all active entities. This gives each customer an assured minimum proportion of the link capacity Y, equal to <b><i>w_i Y</i>/ <i>Σw.</i></b> If a customer sends more than this, their queue just builds up. If they send less, their queue drains and whenever their queue empties, even in the brief periods between packets, the scheduler will give the other customers a higher proportion of the link, because <b><i>Σw</i></b> will not include the inactive user's weight while they have no packet waiting in the queue. This is discussed further in [WFQ89] [WFQ_Cisco].

0038<u>Deep packet inspection (DPI):</u> DPI machines use network processors to reconstruct application layer packet streams and identify which packets belong to which applications. It is then possible for the network operator to configure policies that discriminate against certain applications, which it infers are likely to occupy large amounts of capacity, but may not be particularly highly valued by most customers.

0039DPI machines are also configured to be able to recognise traffic from each individual customer and count total volume, or volume of a particular application(s) against each customer's account. A common approach is to combine the capabilities of the DPI machine to only limit the peer-to-peer file-sharing traffic of those users that have contributed a large proportion of the total traffic volume during the peak period of the day.

0040There is no standard DPI machine, the approach being entirely proprietary. But generally, the packet classification stage can be thought of as similar to the stage of all the schemes so far described that checks whether arriving traffic fits a profile, allowing traffic to be classified as in or out-of-contract. Alternatively, as with WRED, a spectrum between in and out can be defined.

0041Having classified how well traffic complies with a traffic contract, DPI boxes then use the full range of techniques already described to degrade out of contract traffic, ranging from discard to tagging for potential treatment elsewhere in the network if necessary.

0042In addition, DPI boxes may route certain classifications of traffic differently to improve or degrade its service.

0043<u>Comcast's Protocol-Agnostic Congestion Management System</u>: Comcast's system (see [Fairshare]), developed in conjunction with Sandvine, takes the following steps: <ul id="ul0006" list-style="dash"><li>It measures the volume of (upstream) traffic from each customer over a period of a few minutes and records the most recent per-customer metric.</li><li>The network monitors whether a particular segment is becoming congested.</li><li>If it is, the system identifies those users of that segment who have contributed most traffic in the recent past and assigns all their traffic to a lower priority class for a brief period.</li><li>Whenever the segment becomes congested, those users' traffic will then receive lower priority service than everyone else, and therefore may be delayed or dropped.</li><li>Once those customers reduce their contribution below a threshold, they are no longer assigned lower priority.</li></ul>

References

0044[RED]: <nplcit id="ncit0001" npl-type="s"><text>S. Floyd &amp; V. Jacobson: "Random Early Detection gateways for Congestion Avoidance", IEEE/ACM Transactions on Networking, Vol 1-4 (397-413) Aug 1993</text></nplcit>.

0045[RFC3168]: <nplcit id="ncit0002" npl-type="s"><text>Ramakrishnan, K.K., Floyd, S. &amp; Black, D.: "The Addition of Explicit Congestion Notification (ECN) to IP", Internet Engineering Task Force Request for comments 3168 (September 2001</text></nplcit>).

0046[DECbit] <nplcit id="ncit0003" npl-type="s"><text>Jain, R., Ramakrishnan, K. &amp; Chiu, D., "Congestion Avoidance in Computer Networks With a Connectionless Network Layer," Digital Equipment Corporation Technical report DEC-TR-506 (1987</text></nplcit>)

0047[ITU-T Rec. 1.371]: "<nplcit id="ncit0004" npl-type="s"><text>Traffic Control and Congestion Control in B-ISDN," ITU-T Recommendation I.371 (03/04) (March 2004</text></nplcit>) <http://www.itu.int/rec/recommendation.asp?type=folders〈=e&parent=T-REC-I.371>

0048[re-feedback]: <nplcit id="ncit0005" npl-type="b"><text>Briscoe, B., Jacquet, A., Cairano-Gilfedder, C.D., Salvatori, A., Soppera, A. &amp; Koyabe, M.: "Policing Congestion Response in an Internetwork Using Re-Feedback", Proc. ACM StGCOMM'05, Computer Communication Review 35(4): 277-288 ACM Press (Aug 2005</text></nplcit>).

0049[re-ECN]: <nplcit id="ncit0006" npl-type="s"><text>B. Briscoe, A. Jacquet, T. Moncaster, A. Smith: "Re-ECN: Adding Accountability for Causing Congestion to TCP/IP", IETF Internet Draft draft-briscoe-tsvwg-re-ecn-tcp-09 (Oct 2010</text></nplcit>).

0050[ConEx-abstr-mech]: <nplcit id="ncit0007" npl-type="s"><text>Mathis, M. &amp; Briscoe, B., "Congestion Exposure (ConEx) Concepts and Abstract Mechanism," Internet Engineering Task Force Internet Draft draft-mathis-conex-abstract-mech-00 (October 2010) (Work in progress</text></nplcit>)

0051[IEEE802.1Qau]: "<nplcit id="ncit0008" npl-type="s"><text>IEEE Standard for Local and Metropolitan Area Networks---Virtual Bridged Local Area Networks - Amendment: 10: Congestion Notification," IEEE Draft standard 802.1Qau Draft 2.4 (2006) Abstract URL: http://www.ieee802.org/1/paaes/802.1au.html</text></nplcit> [Turner86]: <nplcit id="ncit0009" npl-type="s"><text>Turner, J., "New Directions in Communications," IEEE Communications Magazine 24(10):8-15 (October 1986</text></nplcit>)

0052[RFC2697]: <nplcit id="ncit0010" npl-type="s"><text>Heinanen, J. &amp; Guerin, R., "A Single Rate Three Color Marker," Internet Engineering Task Force Request for comments 2697 (September 1999</text></nplcit>)

0053[RFC2698]: <nplcit id="ncit0011" npl-type="s"><text>Heinanen, J. &amp; Guerin, R., "A Two Rate Three Color Marker," Internet Engineering Task Force Request for comments 2698 (September 1999</text></nplcit>)

0054[Jacquet08]: <nplcit id="ncit0012" npl-type="s"><text>Jacquet, A., Briscoe, B. & Moncaster, T., "Policing Freedom to Use the Internet Resource Pool," In: Proc Workshop on Re-Architecting the internet (ReArch'08) ACM (December 2008) <http://www.bobbriscoe.net/pubs.html#polfree></text></nplcit>

0055[WFQ89]: <nplcit id="ncit0013" npl-type="s"><text>Demers, A., Keshav, S. &amp; Shenker, S., "Analysis and Simulation of a Fair-Queueing Algorithms," Computer Communication Review (SIGCOMM'89) 19(4):1-12 (September 1989</text></nplcit>)

0056[WFQ_Cisco]: "<nplcit id="ncit0014" npl-type="s"><text>Distributed Weighted Fair Queuing," Cisco Feature Guide Web page http://www.cisco.com/en/US/docs/ios/11_1/feature/guide/DWFQ.html (1992</text></nplcit>)

0057[Clark98]: <nplcit id="ncit0015" npl-type="b"><text>Clark, D.D. &amp; Fang, W., "Explicit allocation of best-effort packet delivery service," IEEE/ACM Transactions on Networking 6(4):362--373 IEEE Press (August 1998</text></nplcit>): DOI: http://dx.doi.org/10.1109/90.720870

0058[WRED_Cisco]: "<nplcit id="ncit0016" npl-type="s"><text>Weighted Random Early Detection on the Cisco 12000 Series Router," Cisco Systems Feature Guide IOS Software Release 11.2(1)GS (April 2005</text></nplcit>)

0059[RFC2597]: <nplcit id="ncit0017" npl-type="s"><text>Heinanen, J., Baker, F., Weiss, W. &amp; Wroclawski, J., "Assured Forwarding PHB Group," Internet Engineering Task Force Request for comments 2597 (June 1999</text></nplcit>)

0060[Fairshare]: <nplcit id="ncit0018" npl-type="s"><text>Bastian, C., Klieber, T., Livingood, J., Mills, J. &amp; Woundy, R., "Comcast's Protocol-Agnostic Congestion Management System," Internet Engineering Task Force Request for Comments 6057 (December 2010</text></nplcit>)

0061[uTP10]: Norberg, A., "uTorrent transport protocol," BitTorrent.org BitTorrent Enhancement Proposals (BEPs) 0029 (January 2010) (Draft) http://www.bittorrent.org/beps/bep_0029.html

Relevance of Prior Disclosures to the Present Invention:

0062<ul id="ul0007" list-style="none" compact="compact"><li>Of these prior disclosures, those believed to be of most relevance to the present invention are: <ul id="ul0008" list-style="dash"><li>In-band-congestion-token-bucket (IBCTB) policing (<figref idref="f0002">Figure 1b</figref>)</li><li>Deep packet inspection</li><li>Comcast's Protocol-Agnostic Congestion Management System</li></ul></li></ul>

0063These are all able to focus limits on those customers that contribute more volume during periods of peak usage. In-band-congestion-token-bucket (IBCTB) policing is believed to be of most relevance to the present invention, because it precisely identifies only the volume that directly contributes to congestion in queues.

0064The other two (DPI and Comcast's solution) only broadly identify those contributing most volume during a time when congestion is generally high. But the sources of high volume may not actually coincide with the periods of congestion in the specific queues that are congested.

0065This is problematic, because new 'background' congestion control protocols [uTP10] have recently been deployed that will rapidly reduce in rate at the first signs of congestion, yielding to other 'foreground' traffic. The foreground traffic can then briefly transfer at high rate, thus finishing earlier so that the background transfer can resume and still finish hardly any later. Unfortunately, approaches like DPI and Comcast's solution cannot detect such a fast response to congestion, so they falsely accuse such traffic of contributing high volume during congestion, when in fact the precise opposite is true. Given that about 20% of Internet traffic was using uTP when it was introduced in early 2010, this is a critical omission.

0066In contrast, ECN marks the volume of traffic that is present in the queue during the very brief microseconds when congestion actually occurs. Therefore, in-band-congestion-token-bucket (IBCTB) policing correctly allows through traffic that is highly responsive to congestion, even if very high volumes of it are transferred. While it correctly limits traffic that contributes large amounts of volume during congestion without responding so rapidly.

0067In-band-congestion-token-bucket (IBCTB) policing may be thought of in some ways as an ideal solution, because it precisely identifies the volume of traffic that a customer contributes to congestion, therefore measures precisely how much that customer constrains the experience of other customers or equivalently, how much that customer requires the capacity of the network to be increased. In economic terms, it measures the precise marginal cost of traffic.

0068However, in-band-congestion-token-bucket policing inherently requires the packets to be tagged with congestion information in-band. Therefore, at least congestion exposure (ConEx) has to have been deployed and ideally explicit congestion notification (ECN) too. Both these deployments may not happen for some years and may not be widespread. They require coordinated steps to deployment involving both end systems and networks.

0069Preferred embodiments of the present invention aim to provide a mechanism that allows the profile of congestion contributed by a customer to be compared with the allowed profile for that customer, where the congestion contributed is communicated from the site(s) of congestion to the policer "out-of-band" of the traffic.

<u>Summary of the Invention</u>

0070According to a first aspect of the present invention, there is provided a method according to claim 1

0071According to a second aspect of the present invention, there is provided an apparatus for performing a method according to the first aspect.

0072The various options and preferred embodiments referred to in the following description and dependent claims are applicable in relation to the first and second aspects

0073Preferred embodiments of the present invention may be thought of as providing a compromise that may not give the full benefits of congestion-policing based on in-band congestion signals such as ConEx and/or ECN, but they can be deployed unilaterally, at a single box in the network.

0074They have some similarities to the IBCTB policing described above and illustrated in <figref idref="f0002">Figure 1b</figref>, but according to preferred embodiments, should in general be deployed at locations in networks that are most likely to be congested, where they meter congestion locally, by associating each drop from a local queue with the customer that transferred the packet, rather than counting in-band explicit congestion marks on packets and associating them with customers. Then, as with the in-band-congestion-token-bucket policer, they drain each customer's congestion-token-bucket dependent on this locally counted congestion.

0075<figref idref="f0003">Figure 2</figref> illustrates a preferred embodiment that uses a technique which will be termed "out-of-band-congestion-token-bucket" (OOBCTB) policing. It can be seen that one bulk first-in first-out (FIFO) buffer is associated with the token-bucket policers of all the customers. The buffer is shown holding a queue of packets of varying sizes. This is the data-forwarding device that may become congested, which we have said would in general need to be located locally with the policer. Traffic from all the customers is served by this one simple buffer.

0076If this FIFO buffer becomes congested, each time a packet arrives that must be discarded, the size of the packet is counted against the customer that sent it. This is done by removing the same number of tokens from the customer's bucket as the number of their bytes discarded at the FIFO buffer. Therefore the bucket of customer <b><i>i</i></b> will drain at rate <b><i>v_i</i></b>=<i><b>p_i</b> y_i,</i> where <b><i>y_i</i></b> is the bit-rate of customer <i><b>i</b> that</i> is allowed through the policer, and <b><i>p_i</i></b> is the loss probability at the FIFO buffer. For random discards, <b><i>p_i</i></b> will be the same for all customers, therefore we can say <b><i>p_i</i></b> = <b><i>p</i></b> and <b><i>v_i</i>=<i>py_i.</i></b> This ensures that the customer's bucket drains at a rate proportionate to the amount of volume they contribute during congestion of the local queue.

0077In other respects, the token-bucket policer generally corresponds to the IBCTB policer already described (<figref idref="f0002">Figure 1b</figref>). The bucket of each customer <b><i>i</i></b> is filled with congestion tokens at rate <b><i>w_i.</i></b> And if the level <b><i>d_i</i></b> of a bucket drops to zero, newly arriving traffic from that customer is discarded until their bucket is replenished sufficiently by new tokens.

0078To achieve full effectiveness with this approach, the policer should be located at the point in the network where congestion occurs. ConEx and ECN allow packets to carry congestion information to the policer, which can be located in a control node at the 'customer-edge' of the network. In the absence of protocols such as ConEx, the policer generally has to be located at the congested device itself. If there were multiple points of congestion in the network, it would be harder to locate policers at all these points, because one fill rate associated with one customer must act as a shared feed to all the congestion-token-buckets for that customer - one at each location.

0079According to alternative embodiments, it would be possible to achieve distributed token filling by combining embodiments of the present invention with distributed congestion-policer techniques such as those outlined in International application <patcit id="pcit0006" dnum="WO2011045553A"><text>WO2011/045553</text></patcit>). Although this would make it possible to deploy congestion policing without requiring deployment of any in-band congestion signalling, it would be necessary to have a mechanism to distribute tokens to all the policers from a central source.

<u>Brief Description of the Drawings</u>

0080A preferred embodiment of the present invention will now be described with reference to the appended drawings, in which: <ul id="ul0009" list-style="none"><li><figref idref="f0001"><b>Figure 1a</b></figref> illustrates the technique known as "Token-Bucket Policing";</li><li><figref idref="f0002"><b>Figure 1b</b></figref> illustrates the technique which will be referred to as "In-Band-Congestion-Token-Bucket Policing";</li><li><figref idref="f0002"><b>Figure 1c</b></figref> illustrates the technique known as "Weighted Fair Queuing"; and</li><li><figref idref="f0003"><b>Figure 2</b></figref> illustrates a technique that will be referred to as "Out-of-Band-Congestion-Token-Bucket Policing", which is used according to preferred embodiments of the invention.</li></ul>

<u>Description of Preferred Embodiments of the Invention</u>

0081With reference in particular to <figref idref="f0003"><b>Figure 2</b></figref>, a method according to a preferred embodiment will now be described.

0082The example embodiment shown in <figref idref="f0003">Figure 2</figref> will serve to explain how the Out-of-Band-Congestion-Token-Bucket (OOBCTB) policer works. <ol id="ol0001"><li><u>1.</u><u>Classification:</u> When a packet arrives at the policer, a classifier (a sub-function within the policer that is not shown) determines which customer account i and therefore which token bucket <b><i>i</i></b> the packet is associated with. It may do this by inspecting the link layer source address or a virtual local area network (VLAN) identifier tag to determine which (possibly virtual) line the packet arrived from, or it may arrive on a physically distinct line from those of other customers, so that it can be directly associated solely with one customer.</li><li><u>2.</u><u>Token filling</u>: The time <b><i>t_i</i></b> that bucket <b><i>i</i></b> was last filled will have been stored in a structure associated with the bucket. The elapsed time from then to the time now <b><i>t_now</i></b> is determined, <b><i>Δt_i</i></b> = <i><b>t_now -</b> t_i.</i> Then <i>t_now</i> is stored as the new <b><i>t_i.</i></b> A number of tokens are added to the bucket equal to <i><b>Δt</b>_i<b>w</b>_i</i> where <b><i>w_¡</i></b> is the contracted fill-rate for that bucket (also stored with the bucket). However, the bucket cannot be filled more than its maximum depth <b><i>c_i.</i></b> Thus the new bucket level is <i><b>d</b>_i<b>← min( c</b>_i, <b>d</b>_i</i> + <b><i>Δt_i w_i )</i></b></li><li><u>3.</u><u>Policing</u>: The level <b><i>d_i</i></b> of the relevant congestion-token-bucket is compared with the size <b><i>s_i</i></b> of the packet in bytes. If the level is large enough, that is <b><i>d_i</i> ≥ <i>s_i,</i></b> the packet is allowed to pass. Otherwise, if the level is smaller, <b><i>d_i</i></b> &lt; <b><i>s_i,</i></b> the packet is discarded.</li><li><u>4.</u><u>Enqueuing</u>: The length of the queue in the FIFO buffer is read to determine whether the packet can be enqueued. An active queue management algorithm such as RED is used to determine whether the packet should be discarded probabilistically dependent on the smoothed queue length. If the algorithm decides to enqueue the packet in the buffer, no further action is required and the process returns to step 1 for the next packet.</li><li><u>5.</u><u>Metering and token draining:</u> If the AQM algorithm decides to discard the packet, the size <b><i>s_i</i></b> of the discarded packet in bytes is metered and a number of tokens equal to this metric are drained (subtracted) from the associated token bucket, with the proviso that the bucket depth cannot go negative. That is the new bucket level <b><i>d_i ← max(</i></b> 0, <b><i>d_i - s_i).</i></b> The process returns to step 1 for the next packet.</li></ol>

0083Note that in this preferred embodiment discards at the FIFO buffer lead to draining of the token bucket, while discards at the policer do not. This is preferred because otherwise discards at the policer would both be caused by an empty token bucket and also cause the token bucket to be empty, creating a vicious circle that would make it hard for a user to resume normal service once their bucket had become empty.

0084Also note that in this preferred embodiment the policing step precedes the enqueuing and metering steps. Otherwise, if policing followed enqueuing, the queue management algorithm would remove just enough traffic to keep the queue sufficiently short, then policing would unnecessarily remove more traffic. This would often leave the line idle, which is termed non-work-conserving, or simply underutilisation.

0085Instead discards at the policer should prevent out-of-contract traffic from even being considered for queuing, then out-of-contract traffic will not alter the token bucket's measure of how much congestion is caused by in-contract traffic that has been allowed to pass into the buffer.

0086Nonetheless, there is some scope for alternative embodiments to follow the above five steps in a different order to the preferred order enumerated above. For instance, the filling step may be done at any stage after classification, not necessarily straight after.

0087A numerical example scenario also referring to <figref idref="f0003">Figure 2</figref> will serve to explain the effect of the OOBCTB policer on different traffic streams. In this example, the variable <b><i>x_i</i></b> denotes the arriving bit-rate of all the traffic of customer <b><i>i</i></b> and <i><b>y</b>_i</i> denotes the bit-rate of the same user's traffic departing from the policer.

0088Consider a line of rate Y =1 Gb/s shared by numerous customers, of which we will focus on two. Imagine a brief period <b><i>Δt₁</i></b> = 50ms when the FIFO buffer is having to discard 1% of packets (<b><i>p</i></b>=1%), and a first customer is transferring 20% of the bit-rate at that time (<b><i>y₁</i> = <i>x₁</i> =</b> 20% * 1Gb/s = 200Mb/s). Then 20% of 1% of the bytes will be attributed as congestion caused by customer 1. While if a second customer had quickly reduced their rate at the first sign of the 1% congestion level, say to 0.5% of the capacity (<b><i>y₂</i> = <i>x₂</i></b> = 0.5% * 1Gb/s = 5Mb/s), then over this 50ms period when the congestion level <b><i>p</i></b> for all customers is 1%, only 0.5% of 1% of the congested bytes will be attributed to this second customer. Thus, the tokens drained from each customer's bucket over this period #1: <maths id="math0001"><math display="block"><mtable><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mrow><mi>i</mi><mo>,</mo><mn mathvariant="italic">1</mn></mrow></msub></mtd><mtd columnalign="left"><mo mathvariant="normal">=</mo><msub><mi mathvariant="bold-italic">Δt</mi><mn mathvariant="normal">1</mn></msub><mspace width="1em" /><msub><mi mathvariant="bold-italic">v</mi><mi mathvariant="normal">i</mi></msub></mtd></mtr><mtr><mtd><mspace width="1em" /></mtd><mtd columnalign="left"><mo mathvariant="normal">=</mo><msub><mi mathvariant="bold-italic">Δt</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msub><mi mathvariant="bold-italic">p y</mi><mi mathvariant="normal">i</mi></msub></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="italic">1</mn><mo mathvariant="italic">,</mo><mn mathvariant="italic">1</mn></mrow></msub></mtd><mtd columnalign="left"><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.05</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">%</mo><mo mathvariant="normal">*</mo><mn mathvariant="normal">1</mn><mo>⁢</mo><mi>Gb</mi><mo mathvariant="normal">/</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">20</mn><mo mathvariant="normal">%</mo><mspace width="2em" /><mo mathvariant="normal">=</mo><mn mathvariant="normal">100</mn><mo>⁢</mo><mi>kb</mi></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="italic">2</mn><mo mathvariant="italic">,</mo><mn mathvariant="italic">1</mn></mrow></msub></mtd><mtd columnalign="left"><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.05</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">%</mo><mo mathvariant="normal">*</mo><mn mathvariant="normal">1</mn><mo>⁢</mo><mi>Gb</mi><mo mathvariant="normal">/</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.5</mn><mo mathvariant="normal">%</mo><mspace width="2em" /><mo mathvariant="normal">=</mo><mn mathvariant="normal">2.5</mn><mo>⁢</mo><mi>kb</mi></mtd></mtr></mtable></math><img file="EP2575303A1_D0001.tif" /></maths>

0089Now imagine that, for a longer period <b><i>Δt₂</i></b> = 4 seconds, the loss probability <b><i>p</i></b> falls to 0.001% and the second customer quickly increases its rate up to 50% of capacity considerably exceeding the volume transferred by the first customer. Then: <maths id="math0002"><math display="block"><mtable><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mrow><mi>i</mi><mo>,</mo><mn mathvariant="bold-italic">2</mn></mrow></msub></mtd><mtd columnalign="left"><mo>=</mo><mi mathvariant="bold-italic">Δ</mi><mo>⁢</mo><msub><mi mathvariant="bold-italic">t</mi><mn>2</mn></msub><mspace width="1em" /><msub><mi mathvariant="bold-italic">p y</mi><mi>i</mi></msub></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="bold-italic">1</mn><mo mathvariant="bold-italic">,</mo><mn mathvariant="bold-italic">2</mn></mrow></msub></mtd><mtd columnalign="left"><mo>=</mo><mn mathvariant="normal">4</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>0.001</mn><mo>%</mo><mo>*</mo><mn>1</mn><mo>⁢</mo><mi>Gb</mi><mo>/</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>20</mn><mo>%</mo><mspace width="2em" /><mo>=</mo><mn>8</mn><mo>⁢</mo><mi>kb</mi></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="bold-italic">1</mn><mo mathvariant="bold-italic">,</mo><mn mathvariant="bold-italic">2</mn></mrow></msub></mtd><mtd columnalign="left"><mo>=</mo><mn>4</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>0.001</mn><mo>%</mo><mo>*</mo><mn>1</mn><mo>⁢</mo><mi>Gb</mi><mo>/</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>80</mn><mo>%</mo><mspace width="2em" /><mo>=</mo><mn>32</mn><mo>⁢</mo><mi>kb</mi></mtd></mtr></mtable></math><img file="EP2575303A1_D0002.tif" /></maths>

0090Therefore, over the whole 4.05s period: <maths id="math0003"><math display="block"><mtable><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mn>1</mn></msub><mo>=</mo><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="bold-italic">1</mn><mo mathvariant="bold-italic">,</mo><mn mathvariant="bold-italic">1</mn></mrow></msub><mo>+</mo><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="bold-italic">1</mn><mo mathvariant="bold-italic">,</mo><mn mathvariant="bold-italic">2</mn></mrow></msub><mo>=</mo><mn>100</mn><mo>⁢</mo><mi>kb</mi><mo>+</mo><mn>8</mn><mo>⁢</mo><mi>kb</mi></mtd><mtd><mo>=</mo><munder><mrow><mn>108</mn><mo>⁢</mo><mi>kb</mi><mspace width="1em" /></mrow><mo>̲</mo></munder><mspace width="1em" /><mi>drained from customer</mi><mspace width="1em" /><mn>1</mn><mo>⁢</mo><mi>ʹs bucket</mi></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">V</mi><mn>2</mn></msub><mo>=</mo><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="bold-italic">2</mn><mo mathvariant="bold-italic">,</mo><mn mathvariant="bold-italic">1</mn></mrow></msub><mo>+</mo><msub><mi mathvariant="bold-italic">V</mi><mrow><mn mathvariant="bold-italic">2</mn><mo mathvariant="bold-italic">,</mo><mn mathvariant="bold-italic">2</mn></mrow></msub><mo>=</mo><mn>2.5</mn><mo>⁢</mo><mi>kb</mi><mo>+</mo><mn>32</mn><mo>⁢</mo><mi>kb</mi></mtd><mtd><mo>=</mo><munder><mrow><mn>108</mn><mo>⁢</mo><mi>kb</mi><mspace width="1em" /></mrow><mo>̲</mo></munder><mspace width="1em" /><mi>drained from customer</mi><mspace width="1em" /><mn>2</mn><mo>⁢</mo><mi>ʹs bucket</mi><mn>.</mn></mtd></mtr></mtable></math><img file="EP2575303A1_D0003.tif" /></maths>

0091The volumes transferred by each customer over these two periods combined are: <maths id="math0004"><math display="block"><mtable><mtr><mtd><msub><mi mathvariant="bold-italic">X</mi><mrow><mn mathvariant="italic">1</mn><mo mathvariant="italic">,</mo><mn mathvariant="italic">1</mn></mrow></msub><mo>=</mo><mn mathvariant="normal">0.05</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">1</mn><mo>⁢</mo><mi>Gb</mi><mo mathvariant="normal">/</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">20</mn><mo mathvariant="normal">%</mo></mtd><mtd columnalign="left"><mo>=</mo><mn>10</mn><mo>⁢</mo><mi>Mb</mi></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">X</mi><mrow><mn mathvariant="italic">2</mn><mo mathvariant="italic">,</mo><mn mathvariant="italic">1</mn></mrow></msub><mo>=</mo><mn mathvariant="normal">0.05</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">1</mn><mo>⁢</mo><mi>Gb</mi><mo mathvariant="normal">/</mo><mi mathvariant="normal">s</mi><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.5</mn><mo mathvariant="normal">%</mo></mtd><mtd columnalign="left"><mo>=</mo><mn>0.25</mn><mo>⁢</mo><mi>Mb</mi></mtd></mtr></mtable></math><img file="EP2575303A1_D0004.tif" /></maths><maths id="math0005"><math display="block"><mtable><mtr><mtd><msub><mi mathvariant="bold-italic">X</mi><mrow><mn mathvariant="italic">1</mn><mo mathvariant="italic">,</mo><mn mathvariant="italic">2</mn></mrow></msub><mo>=</mo><mn>4</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>1</mn><mo>⁢</mo><mi>Gb</mi><mo>/</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>20</mn><mo>%</mo></mtd><mtd columnalign="left"><mo>=</mo><mn>800</mn><mo>⁢</mo><mi>Mb</mi></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">X</mi><mrow><mn mathvariant="italic">2</mn><mo mathvariant="italic">,</mo><mn mathvariant="italic">2</mn></mrow></msub><mo>=</mo><mn>4</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>1</mn><mo>⁢</mo><mi>Gb</mi><mo>/</mo><mi mathvariant="normal">s</mi><mo>*</mo><mn>20</mn><mo>%</mo></mtd><mtd columnalign="left"><mo>=</mo><mn>2</mn><mo>,</mo><mn>000</mn><mo>⁢</mo><mi>Mb</mi></mtd></mtr></mtable></math><img file="EP2575303A1_D0005.tif" /></maths><maths id="math0006"><math display="block"><mtable><mtr><mtd><msub><mi mathvariant="bold-italic">X</mi><mn>1</mn></msub><mo>=</mo><msub><mi mathvariant="bold-italic">X</mi><mrow><mn>1</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>+</mo><msub><mi mathvariant="bold-italic">X</mi><mrow><mn>1</mn><mo>,</mo><mn>2</mn></mrow></msub></mtd><mtd columnalign="left"><mo>=</mo><munder><mrow><mn>810</mn><mo>⁢</mo><mi>Mb</mi></mrow><mo>̲</mo></munder></mtd></mtr><mtr><mtd><msub><mi mathvariant="bold-italic">X</mi><mn>2</mn></msub><mo>=</mo><msub><mi mathvariant="bold-italic">X</mi><mrow><mn>2</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>+</mo><msub><mi mathvariant="bold-italic">X</mi><mrow><mn>2</mn><mo>,</mo><mn>2</mn></mrow></msub></mtd><mtd columnalign="left"><mo>=</mo><munder><mrow><mn>2</mn><mo>,</mo><mn>000.25</mn><mo>⁢</mo><mi>Mb</mi></mrow><mo>̲</mo></munder></mtd></mtr></mtable></math><img file="EP2575303A1_D0006.tif" /></maths>

0092Although the second customer transferred nearly 3 times the volume of the first customer, it achieved this while contributing under ¹/₃ of the congestion, therefore its congestion-token-bucket drained at slower than ¹/₃ of the rate of the first customer's. This is because it rapidly responded by reducing its rate drastically in response to congestion, and increasing again rapidly as soon as the congestion subsided.

0093Now imagine that both customers have a contract with the network operator that allows them to cause 20kb/s of congestion, that is <b><i>w₁</i></b> = <b><i>w₂</i></b> = 20kb/s.

0094If the above behaviour continued over subsequent periods, the customers' token buckets would continue to drain at respectively: <maths id="math0007"><math display="block"><msub><mi mathvariant="bold-italic">v</mi><mn>1</mn></msub><mo>=</mo><mn>108</mn><mo>⁢</mo><mi>kb</mi><mo>/</mo><mn>4.05</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo>,</mo><mo>≈</mo><mn>26.7</mn><mo>⁢</mo><mi>kb</mi><mo>/</mo><mi mathvariant="normal">s</mi><mo>,</mo></math><img file="EP2575303A1_D0007.tif" /></maths><maths id="math0008"><math display="block"><msub><mi mathvariant="bold-italic">v</mi><mn>1</mn></msub><mo>=</mo><mn>34.5</mn><mo>⁢</mo><mi>kb</mi><mo>/</mo><mn>4.05</mn><mo>⁢</mo><mi mathvariant="normal">s</mi><mo>,</mo><mo>≈</mo><mn>8.5</mn><mo>⁢</mo><mi>kb</mi><mo>/</mo><mi mathvariant="normal">s</mi><mo>,</mo></math><img file="EP2575303A1_D0008.tif" /></maths>

0095Thus the first customer's token bucket drains faster than its fill rate of 20kb/s. Therefore this bucket will eventually empty and the associated policer will start restricting the amount of traffic <b><i>y₁</i></b> that the first customer can contribute down to much less than it sends, <b><i>y₁ &lt;&lt; x₁.</i></b>

0096Meanwhile, even though the second customer is transferring much more volume, its bucket will remain full, because it is draining congestion tokens more slowly than the contracted fill-rate. This is because the second customer is highly responsive to brief periods of congestion, whereas the first is not. Therefore, the second customer will not be limited at all by the policer, <b><i>y₂</i> = x₂.</b>

0097As soon as the policer reduces <i><b>y</b>₁</i>, the proportion of congestion in the buffer attributed to customer 1 will also reduce (and congestion <b><i>p</i></b> is also likely to reduce, although that depends on everyone else's behaviour). This will reduce the drain rate <b>v<i>₁</i></b> = <b><i>p y₁</i></b> from customer 1's token bucket, which in turn will allow the bucket to fill faster than it drains. This, in turn again, will reduce or eliminate policer drops for customer 1. In the long run, customer 1's throughput will be limited to such a rate that it cannot cause more than its contracted rate of congestion.

Variants

0098<u>ECN as well as drop</u>: The buffer could be ECN-enabled. During congestion it would mark rather than drop any traffic that indicated it came from an ECN-capable transport (ECT). The token draining step would subtract as many tokens from the bucket as the size of the packet in bytes, whether the packet was dropped or ECN-marked to indicate congestion. but if the packet was neither marked nor dropped, no tokens would be drained.

0099Note an important distinction between the out-of-band and in-band metering methods. The in-band meter uses a pre-existing marking on the arriving packet to determine whether to drain tokens from the bucket. Whereas the out-of-band meter drains the bucket dependent on the size of the local queue, due to only local congestion. Then it applies an ECN marking as an output signal. In the in-band case, ECN markings provide the input to the process. Whereas, in the out-of-band case, ECN markings are merely an output, and indeed they are in fact merely an optional output.

0100This is why the metering technique is termed out of band; because the metered quantity is not determined from signalling within the packets. Rather the metered quantity is signalled from the buffer to the token bucket without using the data packets to carry the signals. Typically these signals will be carried between the buffer and the policer over the machine's internal bus, which is why they are termed out-of-band.

0101<u>Quantised congestion level:</u> Rather than draining the congestion-token-bucket dependent on a binary congestion notification (drop or mark), tokens could be drained dependent on the size of each packet and the congestion level of the buffer held as a real number. For instance, the packet size and congestion level could be multiplied together to determine how many tokens to drain.

0102<u>Various sanctions:</u> The first embodiment is framed within a policing scenario, but the main point is to combine a congestion-token-bucket with a different congestion metering method. The congestion policer itself is not the main aspect; and various sanctions may be used, whether the same as the policing action described, whether the same as other previously disclosed policers or different. Possible variations on the sanction applied include, but are not limited to: <ul id="ul0010" list-style="dash"><li>re-routing out-of contract traffic</li><li>delaying out of contract traffic</li><li>damaging out-of-contract traffic, for example truncating the payload but forwarding the header, perhaps with an out-of-contract indication that would distinguish it from other causes of packet damage or discard</li><li>downgrading the class of service of out-of-contract traffic, rather than discarding it In this last case, because the policer does not remove out of contract traffic, more flexibility is allowable in the order of the five steps of the preferred embodiment. For instance the policing step can follow the enqueuing step and/or the metering and draining step.</li></ul>

0103<u>Various Profile Comparator designs:</u> A token bucket is a convenient way to compare a service profile with a contracted profile, but it is not the only way to implement such a comparison function. Possible variations include, but are not limited to: <ul id="ul0011" list-style="dash"><li>a leaky bucket that fills with tokens as packets arrive and drains at a constant rate. A leaky bucket is merely the converse of a token bucket. Its external behaviour can be arranged to be identical to an equivalently sized token bucket by triggering the same action when it is full that the token bucket would trigger when it is empty.</li><li>using various combinations of buckets, such as a dual token bucket that limits both the fill and drain rates, a single-rate three-colour marker, a dual-rate three-colour marker, and so forth</li><li>considering traffic out of contract whenever the bucket depth is below a configured threshold rather than the size of the packet in question</li><li>increasing the probability that the meter deems traffic out-of-contract the lower the bucket level is below a threshold, rather than a binary in-out decision</li><li>using models with a similar outcome as a token bucket, but not implemented internally as a token bucket</li><li>determining a measure of how much the traffic is out of contract dependent on the difference between a user's rate of congestion and the contracted congestion rate for that user.</li></ul>

0104<u>No Policing, only metering:</u> The first embodiment is framed within a policing scenario, but the policing functionality of such embodiments is not necessarily different to that of previously disclosed congestion policers. An important distinction lies, however, in the use of a congestion-token-bucket in combination with a different congestion metering technique, and this may be incorporated within other scenarios. For instance the congestion metering technique could be used to detect whether traffic is outside a congestion contract then trigger a potential range of management actions other than (direct) policing, including (but not limited to): <ul id="ul0012" list-style="dash"><li>triggering an alarm for the operator</li><li>triggering a warning message to the customer</li><li>increasing the usage price of a network service</li><li>counting the volume of out-of-contract traffic, perhaps with a view to:</li><li>charging a higher usage charge for the volume counted</li><li>draining additional token buckets, which in turn might be designed to trigger policing or management actions</li><li>triggering application of a penalty clause in a contract</li><li>altering the per-user weights of a scheduler</li><li>altering the profile or classification of a user</li></ul>

0105<u>Buffer management variants:</u> The first embodiment is framed within a scenario where the buffer uses the RED active queue management algorithm, but any other algorithm could be used, including (but not limited to): <ul id="ul0013" list-style="dash"><li>no queue averaging</li><li>a simple threshold</li><li>a different shape function to calculate the probability of discard or marking dependent on queue length</li><li>no AQM algorithm at all (drop-tail)</li></ul>

0106Also, the first embodiment is framed within a scenario that uses a FIFO buffer, but other queuing disciplines may be appropriate, including (but not limited to): <ul id="ul0014" list-style="dash"><li>a buffer that is part of a more complicated queuing system, but is FIFO within itself</li><li>a weighted scheduler, such as weighted round-robin or weighted fair-queuing</li><li>a set of buffers with different scheduling priorities, but where the drop or marking probability of one buffer is dependent on the queue size in other buffers (e.g. [Gibbens02])</li></ul>

0107<u>Classifier variants:</u> The classifier may associate each packet with the customer account of the sender, the receiver, or some other entity held accountable for the traffic.

0108In relation to embodiments used within policing scenarios, policing is more likely to be applied close to the sender before traffic has traversed the network. Discarding out-of-contract traffic near a receiver serves little purpose as traffic will already have traversed the network. The source may reduce its rate in response to policer drops, but given the policer is designed to identify traffic sources that cause excessive congestion, it would seem inappropriate to assume the source will always respond sensitively to policer drops.

0109If the customer account to be used is that of the receiver, the classifier would typically use the destination address to associate each packet with a customer account.

0110The classifier may associate each packet with the account of a network provider, which in turn provides service to end-users, rather than associating each packet directly with each end-user. The network provider may itself offer its service commercially, or it may be a private concern, such as the network operator of a University campus or enterprise. In these cases, a service VLAN (S-VLAN) tag may typically be used to identify the appropriate user account.

0111Alternatively, identifiers unrelated to network addressing may be used to associate each packet with the correct customer account. For instance, the session identifier in the point-to-point protocol (PPP) may be used, or a cryptographically-generated identifier or sequence of identifiers.

0112<u>Enqueuing</u>: In the preferred embodiment, the drop decision is made on the packet in the process of being enqueued. This is not intended to preclude alternative queuing arrangements where the drop decision is made as packets are de-queued, or if packets are discarded from the middle of the queue.

0113Also, the drop decision may be made based on the size of a virtual queue that counts the size of packets arriving at a real queue but conceptually drains more slowly than the real queue.

0114In an alternative embodiment, the queue management algorithm could internally re-route packets towards an internal discard function, rather than immediately discard them. Then the token bucket associated with the appropriate user account could meter each packet en-route to this internal discard function, in order to drain an amount appropriate to the size of the packet. This alternative embodiment has the advantage of decoupling changes to the queue management function from changes to the metering function. In certain existing implementations such decoupling may allow a pre-existing metering function to be used without alteration, which may be advantageous if the function is embodied in hardware that is hard to change.

0115<u>Normalised weights:</u> The first embodiment has been described as if the fill-rates of each congestion-token-bucket are configured to known, fixed amounts.

0116Instead, fill-rates may be variable, depending on another aspect of the system. For instance, the fill-rate may be a part of a larger allowance allocated for multiple different purposes, e.g. processing and storage as well as bandwidth allocation. The amount of the allowance available for bandwidth allocation would then vary dependent on the amount used for the other purposes, and dependent on the algorithm used to divide up the larger allocation between its constituent parts.

0117Alternatively, a system could be devised where only the relative sizes of the weights would need to be configured, not their absolute values. In addition a parameter would need to be configured to set the proportion of customers it should aim to deem as out of contract - effectively how strongly policing would focus onto a few customers.

0118Then, internally, the system could adjust all the weights proportionately in order to keep the fill-level d_i of most queues close to the middle of their respective buckets, while allowing policing to focus on the configured few customers who were contributing the greatest proportion of congestion relative to their contracted shares.

0119<u>Not just packets:</u> Embodiments of the invention could be applied equally to traffic arriving in datagrams, frames or other discrete units of data.

0120<u>Distributed fill-rate:</u> Alternative embodiments may achieve or allow distributed token filling by combining appropriate embodiments of the invention with distributed congestion-policer techniques such as those outlined in International application <patcit id="pcit0007" dnum="WO2011045553A"><text>WO2011/045553</text></patcit>). Although this would make it possible to deploy congestion policing without requiring deployment of any in-band congestion signalling, it would generally be necessary to have a mechanism to distribute tokens to all the policers from a central source.

0121<u>Distributed buffers and/or meters:</u> The first embodiment has been described in relation to a scenario where the buffer and meters are within the same machine as the token buckets. An alternative embodiment would be to distribute these elements in cases where it was more convenient for the actual buffers to be remote from the token buckets and policers. For instance, distributed arrangements include (but are not limited to) the following: <ul id="ul0015" list-style="dash" compact="compact"><li>In a DOCSIS (data over cable system) cable system, the cable capacity is divided into channels separated using time-division multiplexing. A node central to a whole hybrid-fibre-coax cable called the cable-modem terminal server (CMTS) arbitrates access to each data channel, using a system of out-of-band messages. In the upstream direction, there are numerous buffers - at least one in each cable modem - on entry to the shared cable network. Thus, instead of the single FIFO buffer shown in <figref idref="f0003">Figure 2</figref>, this system involves numerous buffers, each holding a logically separate queue of traffic, but all waiting to enter a single physical resource that is logically divided into time-slots.</li></ul>

0122Although not as straightforward as the simple single buffer of <figref idref="f0003">Figure 2</figref>, it is possible for the CMTS to determine each customer's contribution to congestion by the number of unfulfilled or deferred requests for a slot to transmit data. This metered volume of congestion can then be used to drain a token bucket. The rest of the system may correspond to that already described. <ul id="ul0016" list-style="dash"><li>A passive optical network (PON) uses time-division multiplexing in a similar way to a DOCSIS cable system. Although of course PONs differ from DOCSIS cable networks in many details, the approach described above for DOCSIS translates directly into a PON scenario.</li><li>In a cellular system, the wireless spectrum is a shared link, conceptually similar to the shared cable in a DOCSIS system. In UMTS and LTE, the spectrum is separated into channels using code-division multiple access (CDMA). Similarly to cable, each mobile terminal (handset) has a buffer to queue data waiting to be allowed to transfer upstream over the radio spectrum. The radio network controller (RNC) serves a similar function to the CMTS in a cable network, arbitrating requests for transmission channels as the centre of the radio resource control system.</li></ul>

0123Again, determining each customer's contribution to congestion is not as straightforward as with the simple single buffer of <figref idref="f0003">Figure 2</figref>), but it is conceptually similar to the cable case and, at high level, conceptually similar to the single buffer case.

0124In the downstream direction, all these systems involve a centralised buffer, and can be modelled on the simpler scenario depicted in <figref idref="f0003">Figure 2</figref>.

0125<u>Not just bandwidth:</u> The preferred embodiment has been described in a scenario where the resource being shared is the bandwidth of a communications link and the items queuing to use the resource are packets. The following non-exhaustive list of alternative embodiments would be possible, where the invention is applied to other shared resources and other queued items: <ul id="ul0017" list-style="dash"><li>Requests to use the call-processing resource of a call server</li><li>Requests to use the processing resource of a shared computer</li><li>Requests to use the storage resource of a shared pool of memory (e.g. disks)</li><li>Requests to use a processing resource in a workflow system, where the processing resource may represent a mechanical or human operated function not only a computing function (e.g. shared machines, field engineers, call-centre staff)</li><li>Vehicles using a road transport system</li><li>Units of usage of an energy supply network, supplying for example electricity or gas.</li></ul>

References

0126[Gibbens02]: <nplcit id="ncit0019" npl-type="s"><text>Gibbens, R.J. &amp; Kelly, F.P., "On Packet Marking at Priority Queues, "IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002</text></nplcit>)