Network adapter with time-aware packet-processing pipeline

FIELD OF THE INVENTION

0001The present invention relates generally to packet networks, and particularly to time-aware packet processing in network adapters.

BACKGROUND OF THE INVENTION

0002Various techniques for processing packets in network adapters are known in the art. For example, U.S. Pat. No. 9,397,960 describes methods for steering packets, including receiving a packet and determining parameters to be used in steering the packet to a specific destination, in one or more initial steering stages, based on one or more packet specific attributes. The methods further include determining an identity of the specific destination of the packet in one or more subsequent steering stages, governed by the parameters determined in the one or more initial stages and one or more packet specific attributes, and forwarding the packet to the determined specific destination.

0003U.S. Pat. No. 10,027,601 describes techniques for flow-based packet modification. In some embodiments a communication apparatus includes a host interface, which is configured to be connected to a peripheral component bus, and a network interface, which is configured to be connected to a network. Packet processing circuitry is coupled between the host interface and the network interface and is configured to receive from a first interface, selected from among the host interface and the network interface, a data packet comprising a header containing multiple fields having respective values, to identify, responsively to a value of at least one of the fields, a corresponding entry in a header modification table, and to modify the header in accordance with the identified entry. The data packet with the modified header is transmitted through a second interface selected from among the host interface and the network interface.

SUMMARY OF THE INVENTION

0004An embodiment of the present invention that is described herein provides a network adapter including a host interface configured to communicate with a host, a network interface configured to communicate with a communication network, and packet processing circuitry. The packet processing circuitry is configured to receive a packet from the host via the host interface, or from the communication network via the network interface, to receive an indication of a network time used for synchronizing network elements in the communication network, to match the packet to a rule, the rule including a condition and an action, and to perform the action in response to the packet meeting the condition, wherein one or more of (i) the condition in the rule and (ii) the action in the rule, depend on the network time.

0005In some embodiments, the network adapter further includes a hardware clock that is synchronized to the network time and is configured to provide the indication of the network time to the packet processing circuitry. In some embodiments, the network adapter further includes a timestamping circuit that is configured to insert the indication of the network time into the packet upon arrival of the packet to the network adapter, or upon departure of the packet from the network adapter.

0006Typically, the packet processing circuitry is configured to match the packet to the rule by (i) determining first and second time values, at least one of which being derived from the network time, (ii) extending the first time value to a range of time values, and (iii) comparing the range of time values to the second time value. In various embodiments, the first and second time values are selected from a list including: a time indicated in the packet; an arrival time of the packet to the network adapter; a departure time of the packet from the network adapter; a time value specified in the rule; and a current value of the network time.

0007In an embodiment, upon a successful match to the rule, the packet processing circuitry is configured to control a departure time of the packet from the network adapter depending on the network time. In another embodiment, upon a successful match to the rule, the packet processing circuitry is configured to modify a time field in the packet depending on the network time. In yet another embodiment, upon a successful match to the rule, the packet processing circuitry is configured to insert into the packet a time field depending on the network time. In another embodiment, upon a successful match to the rule, the packet processing circuitry is configured to update statistical information depending on the network time.

0008In some embodiments, by dependence on the network time, the packet processing circuitry is configured to match the packet to the rule synchronously with application of another rule is another network device in the communication network. In a disclosed embodiment, the condition in the rule depends on the network time and on a value of a counter in the network adapter. In an example embodiment, the packet processing circuitry is configured to match the packet to the rule by comparing a time value, which depends on the network time, to a periodic time range.

0009There is additionally provided, in accordance with an embodiment of the present invention, a method for packet processing including, in a network adapter, receiving a packet from a host via a host interface, or from a communication network via a network interface. An indication of a network time, used for synchronizing network elements in the communication network, is received in the network adapter. The packet is matched to a rule using a match engine in the network adapter, the rule including a condition and an action, and the action is performed in response to the packet meeting the condition. One or more of (i) the condition in the rule and (ii) the action in the rule, depend on the network time.

0010The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

0011<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram that schematically illustrates a computing and communication system, in accordance with an embodiment of the present invention;

0012<figref idref="DRAWINGS">FIG. 2</figref> is a diagram that schematically illustrates examples of time-aware rule matching in a network adapter, in accordance with an embodiment of the present invention; and

0013<figref idref="DRAWINGS">FIG. 3</figref> is a flow chart that schematically illustrates a method for time-aware packet processing in a network adapter, in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Overview

0014Embodiments of the present invention that are described herein provide improved methods and systems for processing packets in network adapters. In the disclosed embodiments, a network adapter comprises a host interface for connecting to a host, a network interface for connecting to a communication network, and packet processing circuitry configured to process packets received from the host and/or from the network.

0015The packet processing circuitry comprises a match engine, typically implemented as a hardware pipeline, which is configured to match packets to a set of rules. A typical rule comprises a condition, and an action that is applied if the packet meets the condition.

0016In the embodiments disclosed herein, the match engine is time-aware, in the sense that one or more of the conditions specified in the rules, and/or one or more of the actions specified in the rules, depend on the network time. In the present context, the term “network time” refers to a common time-base that is used for synchronizing elements of the network to one another. The Precision-Time-Protocol (PIP), specified in the IEEE 1588 family of standards, is one example of a scheme for synchronizing network elements to a common network time.

0017In various embodiments, the match engine can be made aware of the network time in various ways. In some embodiments the network adapter comprises a hardware clock that is synchronized to the network time, e.g., a PTP Hardware Clock (PHC). The match engine may receive an indication of the network time directly from the PHC, and/or the PHC may be used for inserting timestamps into packets upon arrival or departure. In other embodiments, the network adapter comprises a free-running clock. In these embodiments, software running on the host may query the free-running clock (e.g., periodically) and correlate the value of the free-running clock with the network time. Having established this correlation, the host software can translate any desired network time to a corresponding value of the free-running clock.

0018Network-time dependent conditions may comprise, for example, declaring a match if the packet arrival time is before a specified time, during a specified time window, or after a specified time, or declaring a match if the packet carries a timestamp that differs from the packet's arrival time by more than a specified difference. Network-time dependent actions may comprise, for example, forcing a packet to depart before a specified time, at a specified time or after a specified time, modifying a timestamp in the packet in a specified way, inserting a timestamp into the packet, or updating a histogram that tracks the distribution of packet times of arrival. As can be appreciated, the above conditions and actions are only a few demonstrative, non-limiting examples.

0019In practice, many network-time dependent conditions can be formulated as a comparison between a time value (e.g., the current network time) and a time range extended from another time value (e.g., a time range around the packet arrival or departure time). In some embodiments the match engine evaluates the condition in a rule using this format, e.g., determines a first time value, extends it to form a time range, and compares a second time value to this time range.

0020The use of a network-time aware match engine enables network adapters to perform powerful packet processing operations that are useful in a wide variety of use cases. Several characteristic examples are described herein.

System Description

0021<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram that schematically illustrates a computing and communication system <b>20</b>, accordance with an embodiment of the present invention. System <b>20</b> comprises a host computer <b>24</b>, e.g., a server, a workstation, or any other suitable computing platform. Host computer <b>24</b> (also referred to herein simply as “host”) uses a network adapter <b>28</b> for communicating over a communication network <b>32</b>. Network <b>32</b> may comprise, for example, an Internet Protocol (IP) network or an Infiniband™ (IB) network. Depending on the applicable network protocol, network adapter <b>28</b> may be referred to, for example, as a Network Interface Controller (NIC) or a Host Channel Adapter (HCA), for example.

0022Host <b>24</b> comprises a Central Processing Unit (CPU) <b>36</b> and a host memory <b>40</b>, typically a Random Access Memory (RAM). Network adapter <b>28</b> comprises a host interface <b>44</b> for communicating with host <b>24</b>. In the present example, host interface <b>44</b> communicates with CPU <b>36</b> and with memory via a peripheral interconnect bus <b>42</b>, e.g., a Peripheral Component Interconnect Express (PCIe) bus. Network adapter further comprises network interface <b>48</b> for communicating with network <b>32</b>.

0023In some embodiments, network adapter <b>28</b> comprises a packet processor <b>52</b> that processes packets received from host <b>24</b> via interface <b>44</b>, and/or from network <b>32</b> via interface <b>48</b>. Typically, packet processor <b>52</b> transmits and receives packets associated with flows. Each flow may correspond to a Queue Pair (QP), to an Ethernet ring or to an IP tuple, for example.

0024Packet processor <b>52</b> comprises a match engine <b>60</b> that is configured to match packets to a set of rules. Typically, each rule specifies a condition and an action to be performed if a packet matches the condition. Conditions in rules may, for example, specify matching of one or more packet header fields, such as Ethertype, port number, protocol, source or destination IP or MAC address, VLAN tag or priority, upper layer (e.g., layer-4) source/destination address or port, Type of Service (TOS), or any other suitable field. Actions in rules may comprise, for example, dropping a packet, allowing a packet to proceed, forwarding a packet to a specified queue or port, modifying or adding a field in the packet, encapsulating or decapsulating a packet. Other actions may comprise, for example, updating some statistics being collected, e.g., update a count of packets that match a rule, or update the aggregate data size, e.g., byte count, of packets that match a rule.

0025Match engine <b>60</b> is typically implemented as a hardware pipeline, and is therefore also referred to as a packet-processing pipeline. In an example embodiment, the set of rules is stored in a rules database <b>56</b> in host memory <b>40</b>. A subset of the rules is cached in packet processor <b>52</b> and queried by match engine <b>60</b>.

0026In some disclosed embodiments, packet processor <b>52</b> of network adapter <b>28</b> (and specifically match engine <b>60</b>) is aware of, and synchronized to, the network time that governs system <b>20</b>. For example, various network elements (e.g., switches or routers in network <b>32</b>) may be synchronized to a common clock master (not shown) using PTP, and network adapter <b>28</b> is aware of this network time as well.

0027In an example embodiment, network adapter <b>28</b> comprises an internal PTP Hardware Clock (PHC) <b>64</b>, typically implemented using an oscillator and a counter, the PHC clock synchronized using PTP to the clock master of system <b>20</b>. Certain aspects of a hardware clock in a network adapter are addressed in U.S. patent application Ser. No. 16/779,611, entitled “Multihost Clock Synchronization,” filed Feb. 2, 2020, which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference.

0028In an embodiment, network adapter <b>28</b> comprises a time-stamping circuit <b>68</b> that is connected to PHC <b>64</b> and inserts timestamps into packets. Time-stamping circuit <b>68</b> may, for example, mark a packet with an indication of the network time at which the packet arrives in network adapter <b>28</b>, and/or with an indication of the network time at which the packet departs from network adapter <b>28</b>. Additionally or alternatively, PHC <b>64</b> may provide match engine <b>60</b> a direct indication of the current network time. Thus, match engine <b>60</b> may be made aware of the network time by receiving packets marked with the network time, and/or by receiving the network time directly.

0029In alternative embodiments, instead of PHC <b>64</b>, network adapter <b>28</b> comprises a free-running clock (not seen in the figures). In these embodiments, software running on host <b>24</b> may query the free-running clock (e.g., periodically or in response to some suitable trigger) and correlate the value of the free-running clock with the network time. Having established this correlation, the host software can translate any desired network time to a corresponding value of the free-running clock. The host may notify packet processor <b>52</b> of the translation. Additionally or alternatively, time-stamping circuit <b>68</b> may mark packets with indications of network times derived from the translation. In this manner, match engine <b>60</b> may be made time-aware without a need for an on-board PHC.

0030The configurations of host <b>24</b> and network adapter <b>28</b>, shown in <figref idref="DRAWINGS">FIG. 1</figref>, are example configurations that are depicted purely for the sake of conceptual clarity. Any other suitable configurations can be used in alternative embodiments.

0031In various embodiments, the various elements of host <b>24</b> and network adapter <b>28</b> may be implemented using suitable hardware, such as one or more Application-Specific integrated Circuits (ASIC) or Field-Programmable Gate Arrays (FPGA), using software, or using a combination of hardware and software elements.

0032In some embodiments, certain elements of host <b>24</b> and/or network adapter <b>28</b>, e.g., certain functions of packet processor <b>52</b>, may be implemented using one or more general-purpose processors, which are programmed in software to carry out the functions described herein. The software may be downloaded to the processors in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

0033Time-Aware Packet Processing in Network Adapter

0034As noted above, upon receiving a packet, match engine <b>60</b> matches the packet to a set of rules, e.g., to the rules in database <b>56</b> or parts thereof. Each rule comprises a condition, and an action to be performed if the packet matches the condition.

0035In some embodiments, for at least one of the rules, the condition and/or the action depends on the network time, as provided, for example, by PHC <b>64</b>. Thus, for some rules, match engine <b>60</b> may find that a given packet matches the rule at a certain network time, but does not match the same rule at another network time. In some embodiments, a time-dependent rule may be implemented by a cascade of first and second rules. The first rule is typically not time-dependent, but rather identifies a specific flow, a specific source, or other suitable packet attribute or attributes. Packets that match the first rule are matched against the second rule, which is time-aware.

0036In various embodiments, a condition in a rule may depend on one or more of the following, for example: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0037">A time of arrival of the packet in network adapter <b>28</b>, as marked in the packet by time-stamping circuit <b>68</b>.</li><li id="ul0002-0002" num="0038">A time of departure of the packet from network adapter <b>28</b>, as marked in the packet by time-stamping circuit <b>68</b>.</li><li id="ul0002-0003" num="0039">The current time, as provided by PHC <b>64</b>.</li></ul></li></ul>

0040A condition that depends on any of these time indications is regarded as depending on the network time. Additionally or alternatively, a condition may depend on the network time in any other suitable way.

0041A few non-limiting examples of conditions that depend on the network time comprise: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0042">Declare a match if the packet arrival time is before a specified time.</li><li id="ul0004-0002" num="0043">Declare a match if the packet arrival time falls during a specified time window.</li><li id="ul0004-0003" num="0044">Declare a match if the packet arrival time is after a specified time.</li><li id="ul0004-0004" num="0045">Declare a match if the packet carries a timestamp that differs from the packet's arrival time by more than a specified difference.</li></ul></li></ul>

0046Note that “packet arrival time” and “packet departure time” can be defined in various ways, because packet arrival and packet departure are multi-stage processes. In practice, it is not uncommon for a packet to travel via packet processor <b>52</b> for a time duration between several nanoseconds to several hundred nanoseconds. In some embodiments, time-stamping circuit <b>68</b> may time-stamp a packet with a “packet arrival time” in any of the packet arrival stages, possibly in multiple stages. Example types of packet arrival time may comprise (i) the time the packet processor completes Medium Access Control (MAC) processing of the packet, (ii) the time the packet processor completes decryption of the packet, (iii) the time the packet is scattered into the host, or any other suitable time. Any such time can be used for implementing time-aware rules. Different rules in the same rules database may use different types of packet arrival time, and/or different types of packet departure time. The distinction between different types of arrival time enables packet processor <b>52</b> to time-stamp a packet “close to the wire” (e.g., as close as possible to the point the packet is first assembled and recognized as a packet). The same holds for the packet departure time. This capability increases the time-stamping accuracy.

0047Further alternatively, any other suitable network-time dependent conditions can be used by match engine <b>60</b>. As seen, in some cases the rule depends on a range of times, e.g., a range of arrival times. Processing of rules of this sort is elaborated below.

0048In some embodiments, a condition may depend on the network time in combination with dependence on one or more other features, e.g., a matching of packet header field values. For example, a condition in a rule may require that the destination IP address of a packet match a certain value, and that the packet arrival time fall in some specified range.

0049In some embodiments, a condition may depend on the network time in combination with a counter value in the network adapter. For example, a condition in a rule may require that the packet arrival/departure time is within a specified time difference from the time a counter value crosses a threshold.

0050The above network-time dependent match conditions are demonstrative, non-limiting examples of match conditions that may be specified in rules. Additionally or alternatively, time-aware match engine <b>60</b> may process any other suitable rules whose conditions depend on the network time.

0051Additional or alternatively, for at least one rule, the action specified in the rule may depend on the network time. As noted above, examples of dependence on network time may comprise, for example, dependence on packet arrival time, on packet departure time, on the current network time, and the like. Thus, for some rules, in case of a match, match engine <b>60</b> may apply a certain action for a certain network time, and a different action at another network time.

0052In one example embodiment, a rule may control the departure time of a packet from the network adapter depending on the network time. For example, a rule may specify that, upon a successful match, the departure of the matching packet be delayed until a predefined network time. As another example, a rule may specify that a matching packet be forced to depart the network adapter before a specified network time, and otherwise dropped and a notification sent to the host software.

0053In another embodiment, a rule may specify insertion of a time field into a matching packet depending on the network time. The inserted time value may comprise, for example, the arrival time of the packet, the departure time of the packet, a time specified in the rule, or the current network time.

0054In yet another embodiment, a rule may specify modification of a time field in a packet depending on the network time. For example, a rule may specify that, upon a successful match, a timestamp field in the matching packet be corrected, e.g., by adding a constant value thereto. This sort of action is useful, for example, for implementing the PTP “transparent clock” mechanism, in which a network element increments a timestamp in a traversing packet by the internal propagation time of the packet in the network element. Note that the disclosed modification is performed within the action pipeline of the packet processor. Typically, modifying a timestamp field warrants an update of a Cyclic Redundancy Check (CRC) field in the packet. In an embodiment, given that the timestamp is modified by addition of a known value, this known value can be used for updating the CRC. An example process of updating a CRC in response to such an addition is described by Lu and Wong, in “A Fast CRC Update Implementation,” Proceedings of the 14^thAnnual Workshop on Circuits Systems and Signal Processing, November, 2003, pages 113-120.

0055Another rule may specify that a timestamp field in the matching packet be overwritten with a different value. Further additionally or alternatively, an action in a rule may update statistical information being collected, depending on the network time. For example, an action in a rule may specify that, upon a match, a certain counter be incremented. The counter may count, for example, the number of packets that match a respective rule, or the aggregate size of packets that match a respective rule. The counter may count bytes, packets or any other suitable data unit. This sort of mechanism is useful, for example, for constructing histograms and other statistical data structures, e.g., a histogram of packet arrival times for a predefined set of arrival-time ranges.

0056The above network-time dependent actions are demonstrative, non-limiting examples of actions that may be specified in rules. Additionally or alternatively, time-aware match engine <b>60</b> may process any other suitable rules whose actions depend on the network time.

0057In some embodiments, the disclosed techniques can be used for applying rules to packet in an atomic manner by two or more network devices (e.g., network adapters, network switches and the like). Since the various network devices are synchronized to the same network time, match engine <b>60</b> may apply a rule to a packet synchronously with another rule that is applied in another network device.

0058<figref idref="DRAWINGS">FIG. 2</figref> is a diagram that schematically illustrates examples of time-aware rule matching performed by match engine <b>60</b> in network adapter <b>28</b>, in accordance with an embodiment of the present invention. As noted above, some rules can be formulated as a comparison between a time value and a time range extended from another time value.

0059Thus, in some embodiments match engine <b>60</b> matches a packet to a rule by (i) evaluating a first time value and a second time value, (ii) extending the first time value to form a time range, and (iii) comparing the second time value to the time range.

0060The first time value may comprise, for example, a time indicated by a timestamp in the packet, the arrival time of the packet or the departure time of the packet. The time range may have any suitable size. In the figure the time range is denoted ±ΔT, and is symmetric on both sides of the first time value. Alternatively, the time range may be one-sided, i.e., only below or only above the first time value. In some embodiments, the time range is configurable.

0061The second time value may comprise, for example, a time indicated by a timestamp in the packet, the arrival time of the packet, the departure time of the packet, a time specified in the rule, or the current network time.

0062The comparison between the second time value and the range may be performed using any suitable comparison operator, e.g., <, >, ∈ or ∉. In other words, match engine <b>60</b> may evaluate the condition by checking whether the second time value falls below the time range, above the time range, inside the time range or outside the time range, for example.

0063In some embodiments, match engine <b>60</b> compares a time value to a periodic time range, e.g., by including a masking operation in the comparison. Consider, for example, a condition that declares a match if the time value is within the first 300 microseconds of a round second. Match engine <b>60</b> may evaluate such a condition, for example, by (i) masking the bits of the time value that correspond to values above 1 second, and (ii) checking whether the masked time value is smaller than 300 microseconds. Comparison to a periodic time range is useful, for example, for correcting drifts in a periodic streaming application. In video streaming, for example, a video frame should be streamed every 16 milliseconds. By comparing the arrival/departure time to a periodic time range of ±1 microseconds every 16 milliseconds, the start times of the video frames can be aligned to correct drift. Alternatively, comparison to a periodic time range, by masking or otherwise, can be used in any other suitable use-case.

0064<figref idref="DRAWINGS">FIG. 3</figref> is a flow chart that schematically illustrates a method for time-aware packet processing in network adapter <b>28</b>, in accordance with an embodiment of the present invention. For clarity, the flow chart refers to a single packet and a single rule. In practice, network adapter <b>28</b> typically processes a large number of packets and a large number of rules, possibly in parallel.

0065The method of <figref idref="DRAWINGS">FIG. 3</figref> begins with network adapter <b>28</b> receiving a packet, at a reception step <b>70</b>. The packet may arrive from host <b>24</b> via host interface <b>44</b>, or from network <b>32</b> via network interface <b>48</b>.

0066At a matching step <b>74</b>, match engine <b>60</b> attempts to match the packet to the condition in a rule. In some embodiments the condition depends on the network time as explained above.

0067At a checking step <b>78</b>, match engine <b>60</b> checks whether the match is successful. If not, the method loops back to step <b>70</b> above. In response to a successful match, match engine <b>60</b> applies the action specified in the rule, at an action step <b>82</b>. In some embodiments the action depends on the network time as explained above.

0068In some embodiments, although not necessarily, match engine <b>60</b> then updates statistics being collected, at a statistics updating step <b>86</b>. In some embodiments the statistics depends on the network time as explained above. The method loops back to step <b>70</b> above.

0069Although the embodiments described herein mainly address Ethernet and Infiniband network adapters, the methods and systems described herein can also be used in other applications, such as in enhanced Common Public Radio interface (eCPRI) equipment, Open vSwitch (OVS) offloading applications, Dynamic Adaptive Streaming over HTTP (DASH)—Also referred to as MPEG-DASH, telemetry applications that send events periodically, as well as time-sensitive applications such as Time-Sensitive Networking (TSN) communication and Society of Motion Picture and Television Engineers (SMPTE) 2110 video transmission.

0070It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.