Methods and apparatus for flow-controllable multi-staged queues

RELATED APPLICATION

0001This application is a continuation of U.S. application Ser. No. 13/544,604, filed Jul. 9, 2012 (now U.S. Pat. No. 8,964,556), which claims priority to U.S. application Ser. No. 12/242,230, filed Sep. 30,2008 (now U.S. Pat. No. 8,218,442), which claims priority to the commonly owned U.S. Provisional Patent Application No. 61/096,209, entitled “Methods and Apparatus Related to Flow Control within a Data Center,” filed on Sep. 11, 2008, each of which are incorporated herein by reference in their entirety.

0002This application is also related to a co-pending U.S. patent application Ser. No. 12/242,224, filed Sep. 30, 2008 (now U.S. Pat. No. 8,154,996), entitled “Methods and Apparatus for Flow Control Associated with Multi-Staged Queues,” which is incorporated herein by reference in its entirety.

BACKGROUND

0003Embodiments described herein relate generally to flow control, and, in particular, to flow control associated with multi-staged queues.

0004Transmission of data from a transmitter to a receiver via a physical link (e.g., an Ethernet link) can be, for example, disrupted because of congestion at a queue configured to receive the data. In some instances, the disruption can cause head-of-line (HOL) blocking and/or result in the loss of at least portions of the data due to buffer overflow. Known flow control protocols such as Ethernet pause (Institute of Electrical and Electronics Engineers (IEEE) 802.3x) and priority pause (IEEE 802.1qbb) can be used to substantially prevent buffer overflow in some applications, and quantized congestion notification (QCN) (IEEE 802.1qau) can be used for management of data congestion within a multi-hop network that has relatively steady data flows. These known flow control protocols, however, may not adequately resolve congestion issues related to multi-stage queues and/or may not adequately handle the onset of congestion within a hop-by-hop network link caused by, for example, bursts of data.

0005Thus, a need exists for methods and apparatus for data flow control between multi-stage queues associated with a hop-by-hop network link.

SUMMARY OF THE INVENTION

0006In one embodiment, a method includes sending a first flow control signal to a first stage of transmit queues when a receive queue is in a congestion state. The method also includes sending a second flow control signal to a second stage of transmit queues different from the first stage of transmit queues when the receive queue is in the congestion state.

BRIEF DESCRIPTION OF THE DRAWINGS

0007<figref idref="DRAWINGS">FIG. 1</figref> is a schematic block diagram that illustrates multiple stages of flow-controllable queues, according to an embodiment.

0008<figref idref="DRAWINGS">FIG. 2</figref> is a schematic block diagram that illustrates multiple stages of flow-controllable queues, according to another embodiment.

0009<figref idref="DRAWINGS">FIG. 3</figref> is a schematic block diagram that illustrates a destination control module configured to define a flow control signal associated with multiple receive queues, according to an embodiment.

0010<figref idref="DRAWINGS">FIG. 4</figref> is a schematic diagram that illustrates a flow control packet, according to an embodiment.

0011<figref idref="DRAWINGS">FIG. 5</figref> is a flowchart that illustrates a method for defining a flow control packet, according to an embodiment.

0012<figref idref="DRAWINGS">FIG. 6</figref> is a schematic block diagram that illustrates a data center, according to an embodiment.

0013<figref idref="DRAWINGS">FIG. 7</figref> is a flowchart that illustrates a method for sending flow control signals to two different stages of queues, according to an embodiment.

DETAILED DESCRIPTION

0014Flow of data (e.g., a data packet such as an internet protocol (IP) packet) via a link between stages of queues at a source entity and stages of queues a destination entity can be controlled based on flow control signaling associated with multiple flow control loops. For example, if the destination entity (or a portion thereof) is unable to handle a flow of data from the source entity because the destination entity has limited buffering capability relative to the amount of data being sent, the destination entity can send a flow control signal associated with at least one of the flow control loops to the source entity. The flow control signal can be configured to trigger the source entity to suspend transmission of the flow of data to the destination entity. In some embodiments, the link between the source entity and the destination entity can be a physical link and can be associated with a single network hop (e.g., a network step that cannot be defined topographically, or a network step associated with a link between one media access control (MAC) device and another MAC device). In some embodiments, the source entity can be referred to as a transmitter and a destination entity can be referred to as a receiver.

0015Flow control loops associated with a link can be defined so that flow control signaling associated with the flow control loops can affect different portions of stages of queues spanning the link. For example, a flow control signal associated with a first flow control loop can trigger suspension of data transmission from a first stage queue and all second stage queues that fan into the first stage queue. A flow control signal associated with a different flow control loop can trigger suspension of data transmission from less than all of the second stage queues that fan into the first stage queue. In some embodiments, data received at one or more of the second stage queues that fan into the first stage queue can be from different (e.g., independent) network devices. In sum, the flow control signal associated with the second flow control loop can trigger different (e.g., more granular) data flow control than the flow control signal associated with the first flow control loop.

0016In some embodiments, a flow control module associated with one or more stages of queues can be configured to define a flow control signal associated with a flow control loop and/or can be configured to execute a flow-related action (e.g., suspend transmission of data) based on one or more parameter values associated with a flow control signal. In some embodiments, one or more parameter values associated with a flow control signal can be stored at the destination entity and/or a source entity. The stored parameter value(s) can be used to define another flow control signal and/or can be used to modify and/or schedule transmission of data from the source entity to the destination entity. In some embodiments, a flow control packet that includes parameter values related to several queues within a stage of queues can be communicated from a destination entity to a source entity via a flow control signal. In some embodiments, a flow control signal can be configured to indicate that at least a portion of a destination entity is available to receive data.

0017<figref idref="DRAWINGS">FIG. 1</figref> is a schematic block diagram that illustrates multiple stages of flow-controllable queues, according to an embodiment. As shown in <figref idref="DRAWINGS">FIG. 1</figref>, a transmit side of a first stage of queues <b>110</b> and a transmit side of a second stage of queues <b>120</b> are included in a source entity <b>130</b> on a transmit side of a physical link <b>100</b>. A receive side of the first stage of queues <b>110</b> and a receive side of the second stage of queues <b>120</b> are included in a destination entity <b>140</b> on a receive side of the physical link <b>100</b>. The source entity <b>130</b> and/or the destination entity <b>140</b> can be any type of computing device (e.g., a server, a personal computer, a router, a multi-stage switch) that can be configured to receive and/or transmit data via the physical link <b>100</b>. In some embodiments, the source entity <b>130</b> and/or the destination entity <b>140</b> can be associated with a data center. More details related to flow-controllable queues within a data center are discussed in connection with <figref idref="DRAWINGS">FIG. 6</figref>.

0018As shown in <figref idref="DRAWINGS">FIG. 1</figref>, the first stage of queues <b>110</b> includes transmit queues A₁through A₄on the transmit side of the physical link <b>100</b> (referred to as first-stage transmit queues <b>134</b>) and receive queues D₁through D₄on the receive side of the physical link <b>100</b> (referred to as first-stage receive queues <b>144</b>). The second stage of queues <b>120</b> includes transmit queues B₁and B₂on the transmit side of the physical link <b>100</b> (referred to as second-stage transmit queues <b>132</b>) and receive queues C₁and C₂on the receive side of the physical link <b>100</b> (referred to as second-stage receive queues <b>142</b>).

0019Flow of data via the physical link <b>100</b> can be controlled (e.g., modified, suspended) based on flow control signaling associated with flow control loops between the source entity <b>130</b> and the destination entity <b>140</b>. For example, data transmitted from the source entity <b>130</b> on the transmit side of the physical link <b>100</b> can be received at the destination entity <b>140</b> on the receive side of the physical link <b>100</b>. A flow control signal can be defined at and/or sent from the destination entity <b>140</b> to the source entity <b>130</b> when the destination entity <b>140</b> is unavailable to receive data from source entity <b>130</b> via the physical link <b>100</b>. The flow control signal can be configured to trigger the source entity <b>130</b> to modify the flow of the data from the source entity <b>130</b> to the destination entity <b>140</b>.

0020For example, if receive queue D₂is unavailable to handle data transmitted from transmit queue A₁, the destination entity <b>140</b> can be configured to send a flow control signal associated with a flow control loop to the source entity <b>130</b>; the flow control signal can be configured to trigger suspension of transmission of data from the transmit queue A₁to the receive queue D₂via a transmission path that includes at least a portion of the second stage of queues <b>120</b> and the physical link <b>100</b>. In some embodiments, the receive queue D₂can be unavailable, for example, when the receive queue D₂is too full to receive data. In some embodiments, the receive queue D<b>2</b> can change from an available state to an unavailable state (e.g., a congestion state) in response to data previously received from the transmit queue A₁. In some embodiments, transmit queue A₁can be referred to as a target of the flow control signal. The transmit queue A₁can be identified within the flow control signal based on a queue identifier associated with the transmit queue A₁. In some embodiments, the flow control signal can be referred to as a feedback signal.

0021In this embodiment, a flow control loop is associated with the physical link <b>100</b> (referred to as a physical link control loop), a flow control loop is associated with first the stages of queues <b>110</b> (referred to as a first stage control loop), and a flow control loop is associated with the second stage of queues <b>120</b> (referred to as a second stage control loop). Specifically, the physical link control loop is associated with a transmission path that includes the physical link <b>100</b>, and excludes the first stage of queues <b>110</b> as well as the second stage of queues <b>120</b>. Flow of data via the physical link <b>100</b> can be turned on and turned off based on flow control signaling associated with the physical link control loop.

0022The first stage control loop can be based on transmission of data from at least one of the transmit queues <b>134</b> within the second stage of queues <b>110</b> and a flow control signal defined based on an availability of (e.g., an indicator of an availability of) at least one of the receive queues <b>144</b> within the first stage of queues <b>110</b>. Thus, the first stage control loop can be referred to as being associated with the first stage of queues <b>110</b>. The first stage control loop can be associated with a transmission path that includes the physical link <b>100</b>, at least a portion of the second stage of queues <b>120</b>, and at least a portion of the first stage of queues <b>110</b>. Flow control signaling associated with the first stage control loop can trigger control of data flow from transmit queues <b>134</b> associated with the first stage of queues <b>110</b>.

0023The second stage control loop can be associated with a transmission path that includes the physical link <b>100</b> and includes at least a portion of the second stage of queues <b>120</b>, but excludes the first stage of queues <b>110</b>. The second stage control loop can be based on transmission of data from at least one of the transmit queues <b>132</b> within the second stage of queues <b>120</b> and a flow control signal defined based on an availability of (e.g., an indicator of an availability of) at least one of the receive queues <b>142</b> within the second stage of queues <b>120</b>. Thus, the second stage control loop can be referred to as being associated with the second stage of queues <b>120</b>. Flow control signaling associated with the second stage control loop can trigger control of data flow from transmit queues <b>132</b> associated with the second stage of queues <b>120</b>.

0024In this embodiment, the flow control loop associated with the second stage of queues <b>120</b> is a priority-based flow control loop. Specifically, each transmit queue from the second-stage transmit queues <b>132</b> is paired with a receive queue from the second-stage receive queues <b>142</b>; and each queue pair is associated with a level of service (also can be referred to as a class of service or quality of service). In this embodiment, second-stage transmit queue B₁and second-stage transmit queue C₁define a queue pair and are associated with level of service X. The second-stage transmit queue B₂and second-stage transmit queue C₂define a queue pair and are associated with service level Y. In some embodiments, different types of network traffic can be associated with a different level of service (and, thus a different priority). For example, storage traffic (e.g., read and write traffic), inter-processor communication, media signaling, session layer signaling, and so forth each can be associated with at least one level of service. In some embodiments, the second stage control loop can be based on, for example, the Institute of Electrical and Electronics Engineers (IEEE) 802.1qbb protocol, which defines a priority-based flow control strategy.

0025Flow of data via a transmission path <b>14</b>, shown in <figref idref="DRAWINGS">FIG. 1</figref>, can be controlled using at least one of the control loops. Transmission path <b>14</b> includes first-stage transmit queue A₂, second-stage transmit queue B₁, the physical link <b>100</b>, second-stage receive queue C₁, and first-stage receive queue D₃. Changes in data flow via a queue in one stage of the transmission path <b>14</b> based on a flow control loop associated with that stage, however, can impact data flow through another stage of the transmission path <b>14</b>. Flow control at one stage can affect data flow at another stage because the queues (e.g., transmit queues <b>132</b>, transmit queues <b>134</b>) within the source entity <b>130</b> and the queues (e.g., receive queues <b>142</b>, receive queues <b>144</b>) within the destination entity <b>140</b> are staged. In other words, flow control based on one flow control loop can have an impact on flow of data via elements associated with a different flow control loop.

0026For example, flow of data from first-stage transmit queue A₁via transmission path <b>14</b> to first-stage receive queue D₃can be modified based on one or more of the control loops—the first stage control loop, the second stage control loop, and/or the physical link control loop. Suspension of data flow to the first-stage receive queue D₃may be triggered because the first-stage receive queue D₃may have changed from an available state to an unavailable state (e.g., a congestion state).

0027If the data flowing to first-stage receive queue D₃is associated with level of service X, the flow of data via second-stage transmit queue B₁and second-stage receive queue C₁(which define the queue pair associated with level of service X) can be suspended based on flow control signaling associated with the second stage control loop (which is a priority-based control loop). But suspending transmission of data via the queue pair associated with level of service X can result in suspension of data transmissions from transmit queues that fan into the second-stage transmit queue B₁. Specifically, suspending transmission of data via the queue pair associated with level of service X can result in suspension of data transmissions from not only first-stage transmit queue A₂, but also of data transmissions from first-stage transmit queue A₁. In other words, flow of data from the first-stage transmit queue A₁is indirectly or collaterally affected. In some embodiments, data received at transmit queue A₁and data received at transmit queue A₂can be associated with the same level of service X, but the data received at transmit queue A₁and the data received at transmit queue A₂may be from, for example, from different (e.g., independent) network devices (not shown) that can be associated with a different level of service.

0028The data flowing to first-stage receive queue D₃can also be suspended by specifically suspending transmission of data from the first-stage transmit queue A₂based on flow control signaling associated with the first stage control loop. By directly suspending transmission of data from the first-stage transmit queue A₂, data transmissions from first-stage transmit queue A₁may not be disrupted. In other words, flow control of the first-stage transmit queue A₂can be directly controlled based on a flow control signal associated with the first stage control loop without suspending data transmission from other first-stage transmit queues such as the first-stage transmit queue A₁.

0029Flow of data to first-stage receive queue D₃can also be controlled by suspending transmission of data via the physical link <b>100</b> based on flow control signaling associated with the physical link control loop. But suspending transmission of data via the physical link <b>100</b> can result in suspension of all data transmissions via the physical link <b>100</b>.

0030The queues on the transmit side of the physical link <b>100</b> can be referred to as transmit queues <b>136</b> and the queues on the receive side of the physical link can be referred to as receive queues <b>146</b>. In some embodiments, the transmit queues <b>136</b> can also be referred to as source queues, and the receive queues <b>146</b> can be referred to as destination queues. Although not shown, in some embodiments, one or more of the transmit queues <b>136</b> can be included in one or more interface cards associated with the source entity <b>130</b>, and one or more of the receive queues <b>146</b> can be included in one or more interface cards associated with the destination entity <b>140</b>.

0031When source entity <b>130</b> transmits data via the physical link <b>100</b>, source entity <b>130</b> can be referred to as a transmitter disposed on a transmit side of the physical link <b>100</b>. Destination entity <b>140</b> can be configured to receive the data and can be referred to as a receiver disposed on a receive side of the physical link <b>100</b>. Although not shown, in some embodiments, the source entity <b>130</b> (and associated elements (e.g., transmit queues <b>136</b>)) can be configured to function as a destination entity (e.g., a receiver) and the destination entity <b>140</b> (and associated elements (e.g., receive queues <b>146</b>)) can be configured to function as a source entity (e.g., a transmitter). Moreover, the physical link <b>100</b> can function as a bidirectional link.

0032In some embodiments, the physical link <b>100</b> can be a tangible link such as an optical link (e.g., a fiber optic cable, a plastic fiber cable), a cable link (e.g., a copper-based wire), a twisted pair link (e.g., a category-5 cable), and so forth. In some embodiments, the physical link <b>100</b> can be a wireless link. Data transmissions via the physical link <b>100</b> can be defined based on a protocol such as an Ethernet protocol, a wireless protocol, and so forth.

0033In some embodiments, the second stage control loop can be referred to as being nested within the first stage control loop because the second stage of queues <b>120</b>, which is associated with the second stage control loop, is disposed inside of the first stage of queues <b>110</b>, which is associated with the first stage control loop. Similarly, the physical link control loop can be referred to as being nested within the second stage control loop. In some embodiments, the second stage control loop can be referred to as an inner control loop and the first stage control loop can be referred to as an outer control loop.

0034<figref idref="DRAWINGS">FIG. 2</figref> is a schematic block diagram that illustrates multiple stages of flow-controllable queues, according to an embodiment. As shown in <figref idref="DRAWINGS">FIG. 2</figref>, a transmit side of a first stage of queues <b>210</b> and a transmit side of a second stage of queues <b>220</b> are included in a source entity <b>230</b> disposed on a transmit side of a physical link <b>200</b>. A receive side of the first stage of queues <b>210</b> and a receive side of the second stage of queues <b>220</b> are included in a destination entity <b>240</b> disposed on a receive side of the physical link <b>200</b>. The queues on the transmit side of the physical link <b>200</b> can collectively be referred to as transmit queues <b>236</b> and the queues on the receive side of the physical link can collectively be referred to as receive queues <b>246</b>. Although not shown, in some embodiments, the source entity <b>230</b> can be configured to function as a destination entity, and the destination entity <b>240</b> can be configured to function as a source entity (e.g., a transmitter). Moreover, the physical link <b>200</b> can function as a bidirectional link.

0035As shown in <figref idref="DRAWINGS">FIG. 2</figref>, source entity <b>230</b> is in communication with destination entity <b>240</b> via the physical link <b>200</b>. Source entity <b>230</b> has a queue QP1 configured to buffer data (if necessary) before the data is transmitted via the physical link <b>200</b>, and destination entity <b>240</b> has a queue QP2 configured to buffer data (if necessary) received via the physical link <b>200</b> before the data is distributed at the destination entity <b>240</b>. In some embodiments, flow of data via the physical link <b>200</b> can be handled without the buffers queue QP1 and queue QP2.

0036Transmit queues QA₁through QA_N, which are included the first stage of queues <b>210</b>, can each be referred to as a first-stage transmit queue and can collectively be referred to as transmit queues <b>234</b> (or as queues <b>234</b>). Transmit queues QB₁through QB_M, which are included in the second stage of queues <b>220</b>, can each be referred to as a second-stage transmit queue and can collectively be referred to as transmit queues <b>232</b> (or as queues <b>232</b>). Receive queues QD₁through QD_R, which are included in the first stage of queues <b>210</b>, can each be referred to as a first-stage receive queue and can collectively be referred to as receive queues <b>244</b> (or as queues <b>244</b>). Receive queues QC₁through QC_M, which are in the second stage of queues <b>220</b>, can each be referred to as a second-stage receive queue and can collectively be referred to as receive queues <b>242</b> (or as queues <b>242</b>).

0037As shown in <figref idref="DRAWINGS">FIG. 2</figref>, each queue from the second stage of queues <b>220</b> is disposed within a transmission path between the physical link <b>200</b> and at least one queue from the first stage of queues <b>210</b>. For example, a portion of a transmission path can be defined by first-stage receive queue QD₄, second-stage receive queue QC₁, and the physical link <b>200</b>. Second-stage receive queue QC₁is disposed within the transmission path between first-stage receive queue QD₄and the physical link <b>200</b>.

0038In this embodiment, a physical link control loop is associated with the physical link <b>200</b>, a first stage control loop is associated with first the stages of queues <b>210</b>, and a second stage control loop is associated with the second stage of queues <b>220</b>. In some embodiments, the second stage control loop can be priority-based control loop. In some embodiments, the physical link control loop can include the physical link <b>200</b>, queue QP<b>1</b>, and queue QP<b>2</b>.

0039Flow control signals can be defined at and/or transmitted between a source control module <b>270</b> at the source entity <b>230</b> and a destination control module <b>280</b> at the destination entity <b>240</b>. In some embodiments, the source control module <b>270</b> can be referred to as a source flow control module, and the destination control module <b>280</b> can be referred to as a destination flow control module. For example, destination control module <b>280</b> can be configured to send a flow control signal to source control module <b>270</b> when one or more of the receive queues <b>246</b> (e.g., receive queue QD₂) at the destination entity <b>240</b> is unavailable to receive data. The flow control signal can be configured to trigger source control module <b>270</b> to, for example, suspend the flow of data from one or more of the receive queues <b>236</b> to the one or more receive queues <b>246</b>.

0040A queue identifier can be associated with data queued at a transmit queue from the transmit queues <b>236</b> by the source control module <b>270</b> before the data is transmitted. The queue identifier can represent and/or can be used to identify the transmit queue where the data is being queued. For example, when a data packet is queued at first-stage transmit queue QA₄, a queue identifier uniquely identifying first-stage transmit queue QA₄can be appended to the data packet or included in a field (e.g., a header portion, a trailer portion, a payload portion) within the data packet. In some embodiments, the queue identifier can be associated with data at the source control module <b>270</b>, or triggered by the source control module <b>270</b>. In some embodiments, the queue identifier can be associated with data just before the data is transmitted, or after the data has been transmitted from one of the transmit queues <b>236</b>.

0041The queue identifier can be associated with data transmitted from the transmit side of the physical link <b>200</b> to the receive side of the physical link <b>200</b> so that the source of the data (e.g., the source queue) can be identified. Accordingly, a flow control signal can be defined to suspend transmission of one or more of the transmit queues <b>236</b> based on the queue identifier. For example, a queue identifier associated with first-stage transmit queue QA_Ncan be included in a data packet transmitted from first-stage transmit queue QA_Nto first-stage receive queue QD₃. If after receiving the data packet, first-stage receive queue QD₃is unable to receive another data packet from first-stage transmit queue QA_N, a flow control signal requesting that first-stage transmit queue QA_Nsuspend transmission of additional data packets to first-stage receive queue QD₃can be defined based on the queue identifier associated with first-stage transmit queue QA_N. The queue identifier can be parsed from the data packet by the destination control module <b>280</b> and used by the destination control module <b>280</b> to define the flow control signal.

0042In some embodiments, data transmissions to first-stage receive queue QD_Rfrom several of the transmit queues <b>236</b> (e.g., first-stage transmit queues <b>234</b>) can be suspended in response to the first-stage receive queue QD_Rchanging from an available state to an unavailable state. Each of the several transmit queues <b>236</b> can be identified within a flow control signal based on their respective queue identifiers.

0043In some embodiments, one or more of the transmit queues <b>236</b> and/or one or more of the receive queues <b>246</b> can be a virtual queue (e.g., a logically defined group of queues). Accordingly, a queue identifier can be associated with (e.g., can represent) the virtual queue. In some embodiments, a queue identifier can be associated with a queue from a set of queues that define a virtual queue. In some embodiments, each queue identifier from a set of queue identifiers associated with the physical link <b>200</b> can be unique. For example, each transmit queues <b>236</b>, which are associated with the physical link <b>200</b> (e.g., associated with a hop), can be associated with a unique queue identifier.

0044In some embodiments, the source control module <b>270</b> can be configured to associate a queue identifier with only a specified subset of the transmit queues <b>236</b> and/or only a subset of data queued at one of the transmit queues <b>236</b>. For example, if data is transmitted from first-stage transmit queue QA₂to first-stage receive queue QD₁without a queue identifier, a flow control signal configured to request that transmission of data from first-stage transmit queue QA₂be suspended may not be defined because the source of the data may not be known. Accordingly, a transmit queue from the transmit queues <b>236</b> can be exempted from flow control by not associating (e.g., omitting) a queue identifier with data when the data is transmitted from the transmit queue.

0045In some embodiments, the unavailability of one or more of the receive queues <b>246</b> at the destination entity <b>240</b> can be defined based on a condition being satisfied. The condition can be related to a storage limit of a queue, a queue access rate, a flow rate of data into the queue, and so forth. For example, a flow control signal can be defined at the destination control module <b>280</b> in response to a status of one or more of the receive queues <b>246</b> such as second-stage receive queue QC₂changing from an available state to an unavailable state (e.g., a congestion state) based on a threshold storage limit being exceeded. The second-stage receive queue QC₂can be unavailable to receive data when in the unavailable state because, for example, the second-stage receive queue QC₂is considered too full (as indicated by the threshold storage limit being exceeded). In some embodiments, one or more of the receive queue <b>246</b> can be in an unavailable state when disabled. In some embodiments, the flow control signal can be defined based on a request to suspend transmission of data to a receive queue from the receive queues <b>246</b> when the receive queue is unavailable to receive data. In some embodiments, the status of one or more of the receive queues <b>246</b> can be changed from an available state to a congestion state (by destination control module <b>280</b>) in response to a specified subset of receive queues <b>246</b> (e.g., receive queues within a specified stage) being in a congestion state.

0046In some embodiments, a flow control signal can be defined at the destination control module <b>280</b> to indicate that one of the receive queues <b>246</b> has changed from an unavailable state to an available state. For example, initially, the destination control module <b>280</b> can be configured to define and send a first flow control signal to the source control module <b>270</b> in response to first-stage receive queue QD₃changing from an available state to an unavailable state. The first-stage receive queue QD₃can change from the available state to the unavailable state in response to data sent from first-stage transmit queue QA₂. Accordingly, the target of the first flow control signal can be first-stage transmit queue QA₂(as indicated based on a queue identifier). When the first-stage receive queue QD₃changes from the unavailable state back to the available state, the destination control module <b>280</b> can be configured to define and send a second flow control signal to the source control module <b>270</b> indicating the change from the unavailable state back to the available state. In some embodiments, the source control module <b>270</b> can be configured to trigger transmission of data from one or more of the transmit queues <b>236</b> to the first-stage receive queue QD₃in response to the second flow control signal.

0047In some embodiments, a flow control signal can have one or more parameter values that can be used by the source control module <b>270</b> to modify transmission from one of the transmit queues <b>236</b> (identified within the flow control signal by a queue identifier). For example, a flow control signal can include a parameter value that can trigger the source control module <b>270</b> to suspend transmission from one of the transmit queues <b>236</b> for a specified period of time (e.g., 10 milliseconds (ms)). In other words, the flow control signal can include a suspension-time-period parameter value. In some embodiments, the suspension time period can be indefinite. In some embodiments, the flow control signal can define a request to transmit data from one or more of the transmit queues <b>236</b> at a specified rate (e.g., specified number of frames per second, specified number of bytes per second).

0048In some embodiments, a flow control signal (e.g., the suspension time period within the flow control signal) can be defined based on a flow control algorithm. The suspension time period can be defined based on a time period during which a receive queue from the receive queues <b>246</b> (e.g., first-stage receive queue QD₄) will be unavailable. In some embodiments, the suspension time period can be defined based on more than one of the first stage receive queues <b>244</b> being unavailable. For example, in some embodiments, the suspension time period can be increased when more or less than a specified number of the first stage receive queues <b>244</b> is in a congestion state. In some embodiments, this type of determination can be made at the destination control module <b>280</b>. The time period during which the receive queue will be unavailable can be a projected (e.g., predicted) time period calculated by the destination control module <b>280</b> based on, for example, a flow rate (e.g., a historic flow rate, a prior flow rate) of data from the receive queue.

0049In some embodiments, the source control module <b>270</b> can deny or alter a request to modify the flow of data from one or more of the transmit queues <b>236</b>. For example, in some embodiments, the source control module <b>270</b> can be configured to decrease or increase a suspension time period. In some embodiments, rather than suspend transmission of data in response to a flow control signal, the source control module <b>270</b> can be configured to modify a transmission path associated with one of the transmission queues <b>236</b>. For example, if first-stage transmit queue QA₂has received a request to suspend transmission based on a change in status of first-stage receive queue QD₂, the source control module <b>270</b> can be configured to trigger transmission of data from first-stage transmit queue QA<b>2</b> to, for example, first-stage receive queue QD₃rather than comply with the request to suspend transmission.

0050As shown in <figref idref="DRAWINGS">FIG. 2</figref>, queues within the second stage of queues <b>220</b> fan into or fan out of the physical link <b>200</b>. For example, transmit queues <b>232</b> (i.e., queues QB₁through QB_M) on the transmit side of the physical link <b>200</b> fan into queue QP<b>1</b> on the transmit side of physical link <b>200</b>. Accordingly, data queued at any of the transmit queues <b>232</b> can be transmitted to queue QP<b>1</b> of the physical link <b>200</b>. On the receive side of the physical link <b>200</b>, data transmitted from the physical link <b>200</b> via queue QP<b>2</b> can be broadcast to receive queues <b>242</b> (i.e., queues QC₁through QC_M).

0051Also, as shown in <figref idref="DRAWINGS">FIG. 2</figref>, transmit queues <b>234</b> within a first stage of queues <b>210</b> fan into the transmit queues <b>232</b> within the second stage of queues <b>220</b>. For example, data queued at any of the first-stage transmit queues QA₁, QA₄, and QA_N-2can be transmitted to second-stage transmit queue QB₂. On the receive side of the physical link <b>200</b>, data transmitted from, for example, second-stage receive queue QC_Mcan be broadcast to first-stage receive queues QD_R-1and QD_R.

0052Because many of the flow control loops (e.g., first control loop) are associated with different fan-in and fan-out architectures, the flow control loops can have various affects on the flow of data via the physical link <b>200</b>. For example, when transmission of data from the second-stage transmit queue QB₁is suspended based on the second stage control loop, transmission of data from first-stage transmit queues QA₁, QA₂, QA₃, and QA_N-1via the second-stage transmit queue QB₁to one or more of the receive queues <b>246</b> is also suspended. In this case, transmission of data from one or more upstream queues (e.g., first-stage transmit queue QA₁) can be suspended when transmission from a downstream queue (e.g., second-stage transmit queue QB₁) is suspended. In contrast, if transmission of data from first-stage transmit queue QA₁along a transmission path that includes at least downstream second-stage transmit queue QB₁is suspended based on the first stage control loop, a flow rate of data from the second-stage transmit queue QB₁may be decreased without entirely suspending transmission of data from second-stage transmit queue QB₁; first-stage transmit queue QA₁, for example, may still be able to transmit data via second-stage transmit queue QB₁.

0053In some embodiments, the fan-in and fan-out architecture can be different than that shown in <figref idref="DRAWINGS">FIG. 2</figref>. For example, in some embodiments, some of the queues within the first stage of queues <b>210</b> can be configured to fan into the physical link <b>200</b>, bypassing the second stage of queues <b>220</b>.

0054Flow control signaling associated with the transmit queues <b>236</b> is handled by the source control module <b>270</b> and flow control signaling associated with the receive queues <b>246</b> is handled by the destination control module <b>280</b>. Although not shown, in some embodiments, flow control signaling can be handled by one or more control modules (or control sub-modules) that can be separate and/or integrated into a single control module. For example, flow control signaling associated with the first-stage receive queues <b>244</b> can be handled by a control module separate from a control module configured to handle flow control signaling associated with the second-stage receive queues <b>242</b>. Likewise, flow control signaling associated with the first-stage transmit queues <b>234</b> can be handled by a control module separate from a control module configured to handle flow control signaling associated with the second-stage transmit queues <b>232</b>. In some embodiments, one or more portions of the source control module <b>270</b> and/or the destination control module <b>280</b> can be a hardware-based module (e.g., a digital signal processor (DSP), a field programmable gate array (FPGA)) and/or a software-based module (e.g., a module of computer code, a set of processor-readable instructions that can be executed at a processor).

0055<figref idref="DRAWINGS">FIG. 3</figref> is a schematic block diagram that illustrates a destination control module <b>350</b> configured to define a flow control signal <b>28</b> associated with multiple receive queues, according to an embodiment. The stages of queues include a first stage of queues <b>310</b> and a second stage of queues <b>320</b>. As shown in <figref idref="DRAWINGS">FIG. 3</figref>, a source control module <b>360</b> is associated with a transmit side of the first stage of queues <b>310</b> and a destination control module <b>350</b> is associated with a receive side of the first stage of queues <b>310</b>. The queues on the transmit side of a physical link <b>300</b> can collectively be referred to as transmit queues <b>370</b>. The queues on the receive side of the physical link <b>300</b> can collectively be referred to as receive queues <b>380</b>.

0056The destination control module <b>350</b> is configured to send the flow control signal <b>28</b> to the source control module <b>360</b> in response to one or more receive queues within the first stage of queues <b>310</b> being unavailable to receive data from a single source queue at the first stage of queues <b>310</b>. The source control module <b>360</b> can be configured to suspend transmission of data from the source queue at the first stage of queues <b>310</b> to the multiple receive queues at the first stage of queues <b>310</b> based on the flow control signal <b>28</b>.

0057The flow control signal <b>28</b> can be defined by the destination control module <b>280</b> based on information associated with each unavailable receive queue within the first stage of queues <b>310</b>. The destination control module <b>350</b> can be configured to collect the information associated with the unavailable receive queues and can be configured to define the flow control signal <b>28</b> so that potentially conflicting flow control signals (not shown) will not be sent to the single source queue at the first stage of queues <b>310</b>. In some embodiments, the flow control signal <b>28</b> defined based on the collection of information can be referred to as an aggregated flow control signal.

0058Specifically, in this example, the destination control module <b>350</b> is configured to define the flow control signal <b>28</b> in response to two receive queues—receive queue <b>342</b> and receive queue <b>346</b>—at the receive side of the first stage of queues <b>310</b> being unavailable to receive data from a transmit queue <b>312</b> on the transmit side of the first stage of queues <b>310</b>. In this embodiment, receive queue <b>342</b> and receive queue <b>346</b> are changed from an available state to an unavailable state in response to data packets sent from transmit queue <b>312</b> via transmission path <b>22</b> and transmission path <b>24</b>, respectively. As shown in <figref idref="DRAWINGS">FIG. 3</figref>, transmission path <b>22</b> includes transmit queue <b>312</b>, transmit queue <b>322</b> within a second stage of queues <b>320</b>, the physical link <b>300</b>, receive queue <b>332</b> within the second stage of queues <b>320</b>, and receive queue <b>342</b>. Transmission path <b>24</b> includes transmit queue <b>312</b>, transmit queue <b>322</b>, the physical link <b>300</b>, receive queue <b>332</b>, and receive queue <b>346</b>.

0059In some embodiments, a flow control algorithm can be used to define the flow control signal <b>28</b> based on information related to the unavailability of receive queue <b>342</b> and/or information related to the unavailability of receive queue <b>346</b>. For example, if destination control module <b>350</b> determines that receive queue <b>342</b> and that receive queue <b>346</b> will be unavailable for different time periods, the destination control module <b>350</b> can be configured to define the flow control signal <b>28</b> based on the different time periods. For example, the destination control module <b>350</b> can request, via the flow control signal <b>28</b>, that transmission of data from transmit queue <b>312</b> be suspended for a time period calculated based on the different time periods (e.g., a time period equal to an average of the different time periods, a time period equal to the greater of the time different periods). In some embodiments, the flow control signal <b>28</b> can be defined based on individual suspension requests from the receive side of the first stage of queues <b>310</b> (e.g., a suspension request associated with receive queue <b>342</b> and a suspension request associated with receive queue <b>346</b>).

0060In some embodiments, the flow control signal <b>28</b> can be defined based on a maximum or a minimum allowable time period. In some embodiments, the flow control signal <b>28</b> can be calculated based on an aggregate flow rate of data from, for example, transmit queue <b>312</b>. For example, the suspension time period can be scaled based on the aggregate flow rate of data from transmit queue <b>312</b>. In some embodiments, for example, the suspension time period can be increased if the flow rate of data from transmit queue <b>312</b> is larger than a threshold value, and the suspension time period can be decreased if the flow rate of data from transmit queue <b>312</b> is lower than a threshold value.

0061In some embodiments, the flow control algorithm can be configured to wait for a specified period of time before defining and/or sending the flow control signal <b>28</b>. The wait time period can be defined so that multiple suspension requests related to transmit queue <b>312</b> and, which can be received at different times within the wait time period, can be used to define the flow control signal <b>28</b>. In some embodiments, the wait period can be triggered in response to at least one suspension request related to transmit queue <b>312</b> being received.

0062In some embodiments, the flow control signal <b>28</b> can be defined by a flow control algorithm based on a priority value associated with each receive queue within the first stage of queues <b>310</b>. For example, if receive queue <b>342</b> has a priority value that is higher than a priority value associated with receive queue <b>346</b>, the destination control module <b>350</b> can be configured to define the flow control signal <b>28</b> based on information associated with receive queue <b>342</b> rather than receive queue <b>346</b>. For example, the flow control signal <b>28</b> can be defined based on a suspension time period associated with receive queue <b>342</b> rather than a suspension time period associated with receive queue <b>346</b> because receive queue <b>342</b> can have a higher priority value than a priority value associated with receive queue <b>346</b>.

0063In some embodiments, the flow control signal <b>28</b> can be defined by a flow control algorithm based on an attribute associated with each receive queue within the first stage of queues <b>310</b>. For example, the flow control signal <b>28</b> can be defined based on receive queue <b>342</b> and/or receive queue <b>346</b> being a specified type of queue (e.g., a last-in-first-out (LIFO) queue, a first-in-first-out (FIFO) queue). In some embodiments, the flow control signal <b>28</b> can be defined based on receive queue <b>342</b> and/or receive queue <b>346</b> being configured to receive a specified type of data (e.g., a control data/signal queue, a media data/signal queue).

0064Although not shown, one or more control modules associated with a stage of queues (e.g., the first stage of queues <b>310</b>) can be configured to send information to a different control module where the information can be used to define a flow control signal. The different control module can be associated with a different stage of queues. For example, a suspension request associated with receive queue <b>342</b> and a suspension request associated with receive queue <b>346</b> can be defined at destination control module <b>350</b>. The suspension requests can be sent to a destination control module (not shown) associated with a receive side of the second stage of queues <b>320</b>. A flow control signal (not shown) can be defined at the destination control module associated with the receive side of the second stage of queues <b>320</b> based on the suspension requests and based on a flow control algorithm.

0065The flow control signal <b>28</b> can be defined based on a flow control loop associated with the first stage of queues <b>310</b> (e.g., a first stage control loop). One or more flow control signals (not shown) can also be defined based on a flow control loop associated with the second stage of queues <b>320</b> and/or a flow control loop associated with the physical link <b>300</b>.

0066Transmission of data associated with transmit queues within the first stage of queues <b>310</b> (other than transmit queue <b>312</b>) is substantially unrestricted by flow control signal <b>28</b> because flow of data to the receive queues <b>342</b> and <b>346</b> is controlled based on the first stage flow control loop. For example, transmit queue <b>314</b> can continue to transmit data via transmit queue <b>322</b> even though transmission of data from transmit queue <b>312</b> is suspended. For example, transmit queue <b>314</b> can be configured to transmit data via transmission path <b>26</b>, which includes transmit queue <b>322</b>, to receive queue <b>348</b> even though transmission of data from transmit queue <b>312</b> via transmit queue <b>322</b> has been suspended. In some embodiments, transmit queue <b>322</b> can be configured to continue to transmit data to receive queue <b>342</b> from, for example, transmit queue <b>316</b> even though transmission of data from queue <b>312</b> via transmission path <b>22</b> has been suspended based on flow control signal <b>28</b>.

0067If transmission of data to the receive queues <b>342</b> and <b>346</b> were instead suspended by controlling flow of data via transmit queue <b>322</b> based on a flow control signal (not shown) associated with the second stage control loop, transmission of data from transmit queue <b>314</b> and transmit queue <b>316</b> via transmit queue <b>322</b> would also be restricted (in addition to transmission of data from transmit queue <b>312</b>). Transmission of data from transmit queue <b>322</b> could be suspended because it is associated with a specified level of service, and the data that caused, for example, congestion at receive queues <b>342</b> and <b>346</b> may be associated with that specified level of service.

0068One or more parameter values defined within the flow control signal <b>28</b> can be stored at a memory <b>352</b> of the destination control module <b>350</b>. In some embodiments, the parameter value(s) can be stored at the memory <b>352</b> of the destination control module <b>350</b> after they have been defined and/or when the flow control signal <b>28</b> is sent to the source control module <b>360</b>. A parameter value defined within the flow control signal <b>28</b> can be used to track a state of, for example, transmit queue <b>312</b>. For example, an entry within the memory <b>352</b> can indicate that the transmit queue <b>312</b> is in a suspended state (e.g., a non-transmit state). The entry can be defined based on a suspension-time-period parameter value defined within the flow control signal <b>28</b>. When the suspension time period has expired, the entry can be updated to indicate that the state of the transmit queue <b>312</b> has changed to, for example, an active state (e.g., a transmit state). Although not shown, in some embodiments, the parameter value(s) can be stored at a memory (e.g., a remote memory) outside of the destination control module <b>350</b>.

0069In some embodiments, the parameter value(s) (e.g., state information defined based on the parameter value(s)) stored at the memory <b>352</b> of the destination control module <b>350</b> can be used by the destination control module <b>350</b> to determine whether or not an additional flow control signal (not shown) should be defined. In some embodiments, the parameter value(s) can be used by the destination control module <b>350</b> to define one or more additional flow control signals.

0070For example, if receive queue <b>342</b> is changed from an available state to an unavailable (e.g., a congestion state) in response to a first data packet received from transmit queue <b>312</b>, a request to suspend transmission of data from transmit queue <b>312</b> can be communicated via the flow control signal <b>28</b>. The flow control signal <b>28</b> can indicate, based on a queue identifier, that transmit queue <b>312</b> is a target of the request and can specify a suspension time period. The suspension time period and the queue identifier associated with transmit queue <b>312</b> can be stored in the memory <b>352</b> of the destination control module <b>350</b> when the flow control signal <b>28</b> is sent to the source control module <b>360</b>. After the flow control signal <b>28</b> is sent, receive queue <b>344</b> can be changed from an available state to a congestion state in response to a second data packet received from transmit queue <b>312</b> (transmission path is not shown in <figref idref="DRAWINGS">FIG. 3</figref>). The second data packet can be sent from the transmit queue <b>312</b> before transmission of data from the transmit queue <b>312</b> is suspended based on flow control signal <b>28</b>. The destination control module <b>350</b> can access the information stored in the memory <b>352</b> and can determine that an additional flow control signal targeted to transmit queue <b>312</b> should not be defined and sent to the source control module <b>360</b> in response to the change in state associated with receive queue <b>344</b> because flow control signal <b>28</b> has already been sent.

0071In some embodiments, the source control module <b>360</b> can be configured to suspend transmission from transmit queue <b>312</b> based on the most recent flow control signal parameter values. For example, after the flow control signal <b>28</b>, which is targeted to transmit queue <b>312</b>, has been sent to the source control module <b>360</b>, a later flow control signal (not shown) targeted to transmit queue <b>312</b> can be received at the source control module <b>360</b>. The source control module <b>360</b> can be configured to implement one or more parameter values associated with the later flow control signal rather than parameter values associated with flow control signal <b>28</b>. In some embodiments, the later flow control signal can trigger the transmit queue <b>312</b> to remain in a suspended state for a longer or shorter period of time than indicated in the flow control signal <b>28</b>.

0072In some embodiments, the source control module <b>360</b> can optionally implement one or more of the parameter values associated with the later flow control signal when a priority value associated with the parameter value(s) is higher (or lower) than a priority value associated with one or more of the parameter values associated with flow control signal <b>28</b>. In some embodiments, each priority value can be defined at the destination control module <b>350</b> and each priority value can be defined based on a priority value associated with one or more of the receive queues <b>380</b>.

0073In some embodiments, the flow control signal <b>28</b> and the later flow control signal (which are both targeted to transmit queue <b>312</b>) can both be defined in response to the same receive queue from the receive queues <b>380</b> being unavailable. For example, the later flow control signal can include updated parameter values defined by the destination control module <b>350</b> based on receive queue <b>342</b> remaining in an unavailable state for a longer period of time than previously calculated. In some embodiments, the flow control signal <b>28</b> targeted to transmit queue <b>312</b> can be defined in response to one of the receive queues <b>380</b> changing state (e.g., changing from an available state to an unavailable state), and the later flow control signal targeted to transmit queue <b>312</b> can be defined in response to another of the receive queues <b>380</b> changing state (e.g., changing from an available state to an unavailable state).

0074In some embodiments, multiple flow control signals can be defined at the destination control module <b>350</b> to suspend transmissions from multiple transmit queues from the first stage of queues <b>310</b>. In some embodiments, the multiple transmit queues can be transmitting data to a single receive queue such as receive queue <b>344</b>. In some embodiments, a history of the flow control signals to the multiple transmit queues from the first stage of queues <b>310</b> can be stored in the memory <b>352</b> of the destination control module <b>350</b>. In some embodiments, a later flow control signal associated with the single receive queue can be calculated based on the history of the flow control signals.

0075In some embodiments, suspension time periods that are associated with multiple transmit queues can be grouped and included in a flow control packet. For example, a suspension time period associated with transmit queue <b>312</b> and a suspension time period associated with transmit queue <b>314</b> can be included in a flow control packet (also can be referred to as a flow control packet). More details related to a flow control packet are described in connection with <figref idref="DRAWINGS">FIG. 4</figref>.

0076<figref idref="DRAWINGS">FIG. 4</figref> is a schematic diagram that illustrates a flow control packet, according to an embodiment. The flow control packet includes a header <b>410</b>, a trailer <b>420</b>, and a payload <b>430</b> that includes suspension-time-period parameter values (shown in column <b>412</b>) for several transmit queues represented by queue identifiers (IDs) (shown in column <b>414</b>). As shown in <figref idref="DRAWINGS">FIG. 4</figref>, transmit queues represented by queue IDs 1 through V (i.e., Queue ID₁through Queue ID_V) are each associated with a suspension-time-period parameter value 1 through V (i.e., Suspension Time Period₁through Suspension Time Period_V). The suspension-time-period parameter values <b>414</b> indicate time periods during which transmit queues represented by the queue IDs <b>412</b> should be suspended (e.g., prohibited) from transmitting data.

0077In some embodiments, the flow control packet can be defined at, for example, a destination control module such as destination control module <b>350</b> shown in <figref idref="DRAWINGS">FIG. 3</figref>. In some embodiments, the destination control module can be configured to define a flow control packet at regular time intervals. For example, the destination control module can be configured to define a flow control packet every 10 ms. In some embodiments, the destination control module can be configured to define a flow control packet at random times, when a suspension-time-period parameter value has been calculated, and/or when a specified number of suspension-time-period parameter values have been calculated. In some embodiments, the destination control module can determine that at least a portion of the flow control packet should not be defined and/or sent, for example, based on one or more parameter values and/or state information accessed by the destination control module.

0078Although not shown, in some embodiments, multiple queue IDs can be associated with a single suspension-time-period parameter value. In some embodiments, at least one queue ID can be associated with a parameter value other than a suspension-time-period parameter value. For example, a queue ID can be associated with a flow rate parameter value. The flow rate parameter value can indicate a flow rate (e.g., a maximum flow rate) at which transmit queues (represented by the queue IDs) should transmit data. In some embodiments, the flow control packet can have one or more fields configured to indicate whether or not a particular receive queue is available to receive data.

0079The flow control packet can be communicated from the destination control module to a source control module (such as source control module <b>360</b> shown in <figref idref="DRAWINGS">FIG. 3</figref>) via a flow control signal (such as flow control signal <b>28</b> shown in <figref idref="DRAWINGS">FIG. 3</figref>). In some embodiments, the flow control packet can be defined based on a layer-<b>2</b> (e.g., layer-<b>2</b> of the opens systems interconnection (OSI) model) protocol. In other words, the flow control packet can be defined at and used within layer-<b>2</b> of a network system. In some embodiments, the flow control packet can be transmitted between devices associated with layer-<b>2</b> (e.g., a MAC device).

0080Referring back to <figref idref="DRAWINGS">FIG. 3</figref>, one or more parameter values (e.g., state information defined based on the parameter value(s)) associated with the flow control signal <b>28</b> can be stored in a memory <b>362</b> of the source control module <b>360</b>. In some embodiments, the parameter value(s) can be stored at the memory <b>362</b> of the source control module <b>360</b> when the flow control signal <b>28</b> is received at the source control module <b>360</b>. A parameter value defined within the flow control signal <b>28</b> can be used to track a state of one or more of the receive queues <b>380</b> (e.g., receive <b>342</b>). For example, an entry within the memory <b>362</b> can indicate that receive queue <b>342</b> is unavailable to receive data. The entry can be defined based on a suspension-time-period parameter value defined within the flow control signal <b>28</b> and associated with an identifier (e.g., a queue identifier) of the receive queue <b>342</b>. When the suspension time period has expired, the entry can be updated to indicate that the state of the receive queue <b>342</b> has changed to, for example, an active state. Although not shown, in some embodiments, the parameter value(s) can be stored at a memory (e.g., a remote memory) outside of the source control module <b>360</b>.

0081In some embodiments, the parameter value(s) (and/or state information) stored at the memory <b>362</b> of the source control module <b>360</b> can be used by the source control module <b>360</b> to determine whether or not data should be transmitted to one or more of the receive queues <b>380</b>. For example, the source control module <b>360</b> can be configured to transmit data from transmit queue <b>316</b> to receive queue <b>344</b> rather than receive queue <b>342</b> based on state information related to receive queue <b>344</b> and receive queue <b>342</b>.

0082In some embodiments, the source control module <b>360</b> can analyze data transmission patterns to determine whether or not data should be transmitted from one or more of the source queues <b>370</b> to one or more of the receive queues <b>380</b>. For example, the source control module <b>360</b> can determine based on parameter values stored at the memory <b>362</b> of the source control module <b>360</b> that transmit queue <b>314</b> is sending a relatively high volume of data to receive queue <b>346</b>. Based on this determination the source control module <b>360</b> can trigger queue <b>316</b> to transmit data to receive queue <b>348</b> rather than receive queue <b>346</b> because receive queue <b>346</b> is receiving the high volume of data from transmit queue <b>314</b>. By analyzing transmission patterns associated with the transmit queues <b>370</b> the onset of congestion at one or more of the receive queues <b>380</b> can be substantially avoided.

0083In some embodiments, the source control module <b>360</b> can analyze parameter values (and/or state information) stored at the memory <b>362</b> of the source control module <b>360</b> to determine whether or not data should be transmitted to one or more of the receive queues <b>380</b>. By analyzing stored parameter values (and/or state information), the onset of congestion at one or more of the transmit queues <b>380</b> can be substantially avoided. For example, the source control module <b>360</b> can trigger data to be transmitted to receive queue <b>340</b> rather than receive queue <b>342</b> based on the historical availability of receive queue <b>340</b> compared with (e.g., being better than, being worse than) the historical availability of receive queue <b>342</b>. In some embodiments, for example, the source control module <b>360</b> can transmit data to receive queue <b>342</b> rather than receive queue <b>344</b> based on the historical performance of receive queue <b>342</b> compared with the historical performance of receive queue <b>344</b> with respect to data bursts patterns. In some embodiments, the analysis of parameter values related to one or more of the receive queues <b>380</b> can be based on a particular time window, a particular type of network transaction (e.g., inter-processor communication), a particular level of service, and so forth.

0084In some embodiments, the destination control module <b>350</b> can send status information (e.g., current status information) about the receive queues <b>380</b> that can be used by the source control module <b>360</b> to determine whether or not data should be transmitted from one or more of the source queues <b>370</b>. For example, the source control module <b>360</b> can trigger queue <b>314</b> to transmit data to queue <b>344</b> rather than queue <b>346</b> because queue <b>346</b> has more available capacity than queue <b>344</b> as indicated by the destination control module <b>350</b>. In some embodiments, any combination of current status information, transmission pattern analysis, and historical data analysis can be used to substantially prevent, or reduce the likelihood of the onset of congestion of one or more of the receive queues <b>380</b>.

0085In some embodiments, the flow control signal <b>28</b> can be sent from the destination control module <b>350</b> to the source control module <b>360</b> via an out-of-band transmission path. For example, the flow control signal <b>28</b> can be sent via a link dedicated to communications related to flow control signaling. In some embodiments, the flow control signal <b>28</b> can be transmitted via queues associated with the second stage of queues <b>320</b>, queues associated with the first stage of queues <b>310</b>, and/or the physical link <b>300</b>.

0086<figref idref="DRAWINGS">FIG. 5</figref> is a flowchart that illustrates a method for defining a flow control packet, according to an embodiment. As shown in <figref idref="DRAWINGS">FIG. 5</figref>, data queued at a transmit queue within a first stage of queues is associated with a queue identifier representing the transmit queue, at <b>500</b>. In some embodiments, the queue identifier can be appended to the data or included in a portion of a field associated with the data.

0087The data queued at the transmit queue is sent to a receive queue within the first stage of queues through a second stage of queues and a physical link, at <b>510</b>. In other words, the data is sent from the transmit queue to the receive queue via a transmission path that includes not only the transmit queue and the receive queue, but also at least one queue from the second stage of queues and the physical link. The second stage of queues can be disposed between the first stage of queues and the physical link. A transmit side of the first stage of queues can be associated with a switch core of a data center and the receive side of the first stage of queues can be associated with a network entity at the edge of the data center, and vice versa. The first stage of queues can be associated with a first stage control loop and the second stage of queues can be associated with a second stage control loop. In some embodiments, the physical link can be associated with a physical link control loop.

0088A status of the receive queue is modified in response to the data being received at the receive queue, at <b>520</b>. In some embodiments, the status of the receive queue can be changed from an available state to a congestion state in response to the data being received at the receive queue. In some embodiments, the status of the receive queue can be changed from an available state to a congestion state (by a destination control module) in response to more than one receive queue (or a specified subset of receive queues) within a stage being in a congestion state.

0089A request to suspend transmission of data from the transmit queue for a suspension time period is defined based on the status of the receive queue and based on the queue identifier, at <b>530</b>. The request to suspend transmission can include the queue identifier so that the transmit queue can be identified as the target of the request to suspend. In some embodiments, the request to suspend transmission can be defined based on the status of the receive queue changing to an inactive state or a disabled state. In some embodiments, the suspension time period can be defined based on a flow rate of data from the receive queue.

0090The suspension time period associated with the request is modified at a destination control module associated with a receive side of the second stage of queues, at <b>540</b>. The suspension time period for the transmit queue can be modified based on multiple requests to suspend transmission of data from the transmit queue. In some embodiments, the suspension time period can be modified at a portion of the destination control module configured to define a suspension-time-period parameter value for the transmit queue based on multiple requests to suspend transmission of data from the transmit queue. In some embodiments, the destination control module can be associated with a different portion of the receive side that may not include the second stage of queues. In some embodiments, the request to suspend can be denied.

0091A flow control packet that includes a parameter value representing the suspension time period is defined based on the request, at <b>550</b>. In some embodiments, the flow control packet can include suspension-time-period parameter values associated with multiple queue identifiers.

0092The flow control packet is sent to a source control module associated with the transmit queue, at <b>560</b>. In some embodiments, the flow control packet can be sent at a scheduled time or at a time specified at the source control module associated with the receive side of the second stage of queues. In some embodiments, the flow control packet can be sent via a flow control signal and/or can be sent via an out-of-band transmission path.

0093Transmission of data from the transmit queue within the first stage of queues is suspended based on the suspension-time-period parameter value included in the flow control packet, at <b>560</b>. In some embodiments, the suspension time period can be decreased by a flow control signal (e.g., a different flow control packet) received at the source control module after the flow control packet is received at the source control module.

0094<figref idref="DRAWINGS">FIG. 6</figref> is a schematic block diagram that illustrates a data center <b>600</b>, according to an embodiment. The data center <b>600</b> includes network entity <b>610</b>, network entity <b>620</b>, and network entity <b>630</b> (collectively referred to as network entities <b>680</b>), which are each in communication with a switch core <b>602</b> via at least one of the physical links <b>690</b>. For example, network entity <b>630</b> can be configured to transmit data to and/or receive data from the switch core <b>602</b> via physical link <b>632</b>. When network entity <b>630</b> transmits data via the physical link <b>632</b>, network entity <b>630</b> can be referred to as a transmitter disposed on a transmit side of the physical link <b>632</b>. Switch core <b>602</b> can be configured to receive the data and can be referred to as a receiver disposed on a receive side of the physical link <b>632</b>. Conversely, switch core <b>602</b> can also be configured to transmit data as a transmitter, and network entity <b>630</b> can be configured to receive the data as a receiver.

0095Flow of data via each of the physical links <b>690</b> (physical link <b>612</b>, physical link <b>622</b>, and/or physical link <b>632</b>) can be controlled based on more than one flow control loop such as those described in connection with <figref idref="DRAWINGS">FIGS. 1 through 5</figref>. The flow control loops can be associated with the physical links <b>690</b> and/or multiple stages of queues within portions of the data center <b>600</b>. For example, a flow control loop can be associated with physical link <b>622</b> and at least one flow control loop can be associated with each of several stages of queues spanning the physical link <b>622</b>.

0096In some embodiments, each stage of queues can have a first portion of queues on one side (e.g., a transmit side) of the physical link <b>622</b> and a second portion of queues on another side (e.g., a receive side) of the physical link <b>622</b>. For example, although not shown, a first queue within a stage of queues can be included in network entity <b>610</b> on one side of the physical link <b>612</b>, and a second queue within the stage of queues can be included in the switch core <b>602</b> on another side of the physical link <b>612</b>. A flow control loop can be associated with the first queue within the stage of queues included in the network entity <b>610</b> and the second queue within the stage of queues included in the switch core <b>602</b>.

0097For example, if the first queue (the receive queue) is unable to handle data transmitted from the second queue (the transmit queue), a flow control module associated with the first queue can send a flow control signal configured to trigger suspension of transmission of data from the second queue to the first queue. The second queue can be identified within the flow control signal based on a queue identifier associated with the second queue.

0098As shown in <figref idref="DRAWINGS">FIG. 6</figref>, network entity <b>610</b>, network entity <b>620</b>, and network entity <b>630</b> are at the edges of the data center <b>600</b> and function as gateways to network <b>614</b>, network <b>624</b>, and network <b>634</b>, respectively. Networks <b>614</b>, network <b>624</b>, and/or network <b>634</b> can be, for example, a virtual network, a local area network (LAN) and/or a wide area network (WAN), and can include one or more wired and/or wireless segments.

0099The switch core <b>602</b> of the data center <b>600</b> can be a switch fabric that has multiple stages (e.g., an ingress stage, an egress stage, a middle stage) through which data can be routed. In some embodiments, the switch core <b>602</b> can be defined based on a Clos network architecture (e.g., a non-blocking Clos network, a strict sense non-blocking Clos network, a Benes network). In some embodiments, a network architecture such as, for example, a Clos network and/or a Benes network can be reconfigurable (e.g., rearrangeable). In some embodiments, the switch core <b>602</b> can be defined by one or more multi-stage switches (not shown) that each include one or more switch fabrics.

0100<figref idref="DRAWINGS">FIG. 7</figref> is a flowchart that illustrates a method for sending flow control signals to two different stages of queues, according to an embodiment. As shown in <figref idref="DRAWINGS">FIG. 7</figref>, a first flow control signal is sent to a first stage of transmit queues when a receive queue is in a congestion state, at <b>700</b>. In some embodiments, the first flow control signal can be sent in response to an indicator that the receive queue is in the congestion state. In some embodiments, the indicator can be referred to as a congestion indicator. In some embodiments, a status of the receive queue can be changed from an available state to the congestion state in response to a portion of data sent from the first stage of transmit queues. In some embodiments, the first flow control signal can be associated with a first flow control loop.

0101A second flow control signal is sent to a second stage of transmit queues when the receive queue is in the congestion state, at <b>710</b>. In some embodiments, the second flow control signal can be sent in response to the indicator that the receive queue is in the congestion state. In other words, the indicator that triggers sending of the second flow control signal can be the same as the indicator that triggers sending of the first flow control signal. In some embodiments, the sending of the first flow control signal and the sending of the second flow control signal can be triggered by different indicators. In some embodiments, a status of the receive queue can be changed from an available state to the congestion state in response to a portion of data sent from the second stage of transmit queues. In some embodiments, the second flow control signal can be associated with a second flow control loop different than a first flow control loop associated with the first flow control signal.

0102Some embodiments described herein relate to a computer storage product with a computer-readable medium (also can be referred to as a processor-readable medium) having instructions or computer code thereon for performing various computer-implemented operations. The media and computer code (also can be referred to as code) may be those designed and constructed for the specific purpose or purposes. Examples of computer-readable media include, but are not limited to: magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as Compact Disc/Digital Video Discs (CD/DVDs), Compact Disc-Read Only Memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules; and hardware devices that are specially configured to store and execute program code, such as Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), and Read-Only Memory (ROM) and Random-Access Memory (RAM) devices.

0103Examples of computer code include, but are not limited to, micro-code or micro-instructions, machine instructions, such as produced by a compiler, code used to produce a web service, and files containing higher-level instructions that are executed by a computer using an interpreter. For example, embodiments may be implemented using Java, C++, or other programming languages (e.g., object-oriented programming languages) and development tools. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code.

0104While various embodiments have been described above, it should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The embodiments described herein can include various combinations and/or sub-combinations of the functions, components and/or features of the different embodiments described. For example, a source entity and/or a destination entity can have more than two flow-controllable stages of queues that span a physical link.