Method to integrate co-processors with a protocol processing pipeline

TECHNICAL FIELD

0001The present invention relates to the integration of co-processors and a protocol processing pipeline that may operate at Layer 1 to Layer 7 of the OSI reference model.

BACKGROUND

0002A network adaptor (e.g., a Network Interface Controller (NIC)) may be, for example, network interface circuitry, such as on a PCI card connected to a host computer or other local device, an ASIC or an FPGA, via a host bus (such as a PCI bus or ARM ACE bus) or an IP block within an SoC (System On a Chip) that is connected to the system bus via a host bus. The network adaptor is typically used to couple the host CPU to a packet network through at least one interface, called a port. NIC circuitry has been an area of rapid development as advanced packet processing functionality and protocol offload have become requirements for so called “smart NICs”. In a previous U.S. Pat. No. 7,945,705 we have described how a co-processor is inserted into a protocol offload NIC processing pipeline using a dedicated SPI4.2 interconnect bus. We refer to this type of co-processor insertion as a data-path TAP point. Advantages of using such TAP points include being able to leverage the software that is available with the NIC while increasing flexibility and including value-added features.

0003Commercially, a NIC provider typically also provides a complete suite of software that has been included into the software release on different operating systems such as Linux, FreeBSD, or Microsoft Windows. Value added and extensibility features such as virtualization, deep packet inspection, access control and intrusion detection and prevention (ACL/IDS/IPS), encapsulation and de-capsulation, management interfaces, de-duplication, compression/de-compression, encryption/decryption can be inserted into the NIC processing pipeline while preserving the software investment and can be inserted without affecting the released software.

0004TAP points also allow separating the control path from the data path, and operating on one or the other independently, e.g. processing the control plane in the co-processor while transferring payload through the system bus.

SUMMARY

0005A network adaptor (or NIC) is equipped with multi-level protocol processing capability and is implemented with a protocol processing pipeline that has multiple tap points to enable the integration of co-processors to operate with the NIC. The capability leverages the protocol processing pipeline and all the existing NIC software while at the same time enabling the integration of value added co-processors to customize and enhance the NIC capabilities.

BRIEF DESCRIPTION OF FIGURES

0006<figref idref="DRAWINGS">FIG. 1</figref> is an example SoC with the protocol offload NIC IP with TAP points.

0007<figref idref="DRAWINGS">FIG. 2</figref> illustrates two examples of connecting the SoC NIC IP to the SoC system bus.

0008<figref idref="DRAWINGS">FIGS. 3 and 3-1</figref> are a block diagram illustrating the egress pipeline.

0009<figref idref="DRAWINGS">FIGS. 4 and 4-1</figref> are block diagrams illustrating the ingress pipeline.

0010<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram illustrating TAP points at each pipeline stage.

0011<figref idref="DRAWINGS">FIG. 6</figref> illustrates an example switch block for exit/entry into the protocol processing pipeline.

0012<figref idref="DRAWINGS">FIG. 7</figref> illustrates an example switch block configuration.

DETAILED DESCRIPTION

0013With the advent of SoC IP configurations of a protocol offload NIC there is an opportunity to generalize the TAP points as described in U.S. Pat. No. 7,945,705 and to enable different TAP points at different layers of the network protocol stack, e.g. optionally add L5-L7 processing in addition to the above application enabled by U.S. Pat. No. 7,945,705.

0014In the following detailed description of the preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

0015In U.S. Pat. No. 7,945,705, it is described how protocol encapsulation (therein, denoted as CPL for “Chelsio Protocol Language”) is used to direct messages to the co-processor via the SPI4.2 bus and how the CPL encapsulation is used to steer the messages injected from the co-processor. In accordance with the present invention, the mechanism is generalized so that data and control messages can exit and enter the protocol processing pipeline at various generalized TAP points. The TAP points can be at any of the layers L2-L7 of the ISO reference protocol stack, and the data-flow allowed includes, for example, ingress-to-ingress loopback, ingress-to-egress pass-through, egress-to-egress loopback, and egress-to-ingress pass-through. Unicast of the CPL messages, multi-cast, and broadcast may be supported. This means that a CPL message arriving at a TAP point may be switched to one, to many, or to all possible destinations of the particular TAP point.

0016The CPL messages include control plane messages and data plane messages, where the control plane messages configure the hardware, indicate progress, and indicate completions of tasks. The CPL data plane messages are used to carry payload. A TAP point can be configured such that all CPL data plane messages that arrive at the TAP point are switched to the co-processor before being injected by the co-processor back into the processing pipeline, while all the control plane messages are switched to the next stage in the processing pipeline, thereby bypassing the co-processor. This separation has the advantage of simplifying the co-processor design and implementation, as the co-processor may only need to know how to process the subset of CPL messages that carry payload and the co-processor only needs to handle the traffic load generated by payload carrying messages. In a similar manner a TAP point can be configured such that all the control plane messages are switched to the co-processor while all the data plane messages are switched to the next stage in the processing pipeline and do not go through the co-processor. This has the advantage of being able to move the control processing from the host computer to a co-processor, thereby potentially simplifying the host computer or being able to replace the host computer with a different type of devices, for example an ASIC, an FPGA, or other type of programmable logic device that is well suited to data plane processing while it might not be as well suited for control plane processing. The co-processor in this case might employ a conventional host processor.

0017Refer now to <figref idref="DRAWINGS">FIG. 1</figref> that shows an example SoC with the NIC IP referred to as the CIM+Transport Processor <b>100</b> with the TAP points connecting to value added features referred to as custom features <b>110</b>. The example SoC includes two quad core Cortex A15 ARM processors <b>130</b> and <b>131</b> connected together over the ARM CoreLink™ CCN network <b>140</b> and the processors are also connected via the CoreLink™ to DRAM <b>181</b> and to SATA/SAS controllers <b>150</b><b>170</b>. The DRAMs <b>181</b> are connected to the CoreLink™ via DMC-<b>520</b> controllers <b>180</b>. The SATA/SAS controllers <b>151</b> and <b>171</b> are typically connected to the CoreLink™ via an ACE Lite controller <b>150</b> and <b>170</b>. Finally, the NIC is connected to CoreLink™ either via a PCIe endpoint/root, or via an ACE or ACE Lite controller <b>180</b>. The NIC has custom features integrated via TAP Points <b>111</b>, and finally the NIC is connected via Ethernet MAC to the interface of the SoC where it is connected to Ethernet <b>120</b>.

0018Refer now to <figref idref="DRAWINGS">FIG. 2</figref> that shows two options for coupling the NIC IP to the CoreLink™ system network. The first example is via a PCI Endpoint <b>201</b> and PCI root <b>200</b> and the second example is via an ACE bus or ACE Lite bus interface module <b>202</b>. The Chelsio Transport Processor communicates with the system network using DMA, PIO, and memory access MEM <b>210</b>. The Control uP <b>230</b> implements the control plane processing required by the IP and the Edge Switch <b>240</b> is a switch that is integrated with the IP. The P<b>0</b><b>250</b> and P<b>1</b><b>260</b> blocks are two Ethernet MAC interfaces. Finally the TAP Points <b>270</b> are connected to the IP at different stages of the processing pipeline to the Edge Switch and between the Edge Switch and MAC.

0019Refer now to <figref idref="DRAWINGS">FIGS. 3 and 3-1</figref> that show the egress processing pipeline with TAP points inserted at different layers of the protocol stack. The TAP points are individually optional and one or more can be included depending on the desired custom features in the SoC. When a particular co-processor is not required, it is not included in the pipeline, and when a particular TAP point is not required it is replaced with a point to point connection. For example when the L5-L7 TAP point <b>105</b> is not required the CPL message entry point <b>104</b><i>a </i>is directly connected to the ulp <b>106</b>. When CPL messages (i.e., a payload encapsulated by a CPL header) arrive from the SoC system via the PCIe bus or ACE interface module <b>104</b><i>a</i>, the CPL messages are optionally switched to the L5-L7 co-processor <b>130</b> before proceeding into the processing pipeline <b>102</b>. For offloaded TCP connections (i.e., the NIC is handling TCP protocol processing rather than a TCP protocol stack that is part of the host CPU operating system), the CPL messages carry TCP payload and no L2, L3 or L4 headers, since the L2, L3 and L4 headers are formed on the NIC as part of the offloaded protocol processing. This is therefore an opportune point in the pipeline to operate on the application payload, e.g. perform ACL/IDS/IPS, de-duplications, compression/de-compression, encryption/decryption, etc.

0020The packet next proceeds into PDU encapsulation block ulp <b>106</b>—e.g., TLS and/or iSCSI and/or iWARP and the L4+ TAP point <b>132</b> is an opportune point to operate on the Protocol Data Unit (PDU). For example, the PDU header can be audited for ACL/IPS/IPS purposes, the PDU can be encrypted, and the header can be transformed, for example, translating from one format to a different format.

0021The packet is processed and an Ethernet encapsulated TCP packet is formed <b>120</b> and the L4 TAP <b>111</b> is an opportune time to process layer L4 packet in co-processor <b>132</b>. An example of Layer 4 processing that may be performed by co-processor <b>132</b> is TLS encryption where the L4 payload is encrypted.

0022The packet then proceeds to the L3 TAP <b>113</b> and examples of Layer 3 processing in co-processor <b>134</b> include NAT and IPsec encryption.

0023The packet then proceeds through the L2 switch and to the L2 TAP <b>116</b>. Examples of Layer 2 processing in co-processor <b>133</b> include encapsulations such as VXLAN, GENEVE, and NVGRE.

0024Refer now to <figref idref="DRAWINGS">FIGS. 4 and 4-1</figref> that show the steps performed by a NIC processing pipeline for an ingress packet with TAP points inserted at different layers of the protocol stack. When Ethernet packets arrive <b>140</b>, they optionally go to the L2 TAP point <b>150</b> before going to the L2 switch <b>160</b>. The co-processor connected to this TAP point <b>150</b> can for example remove VXLAN, GENEVE, NVGRE encapsulation.

0025The packets then proceed to the Ethernet TCP/UDP/IP packet parser <b>170</b> before optionally proceeding to the L3 TAP <b>180</b>. The co-processor <b>190</b> connected to this TAP point <b>180</b> can, for example, perform IPsec decryption and/or NAT processing.

0026The packets then are injected into the processing pipeline <b>104</b><i>c </i>and proceed to the L4 TAP <b>115</b>. The co-processor <b>121</b> connected to this TAP point <b>115</b> can for example perform TLS decryption.

0027When L4 carries a PDU in the payload, the PDU's are recovered next in the processing pipeline and the L4+ TAP point <b>116</b> is where it is opportune to inject PDU header auditing co-processing, performed by co-processor <b>122</b> for ACL/IPS/IPS purposes, PDU decryption and the header transformation for example translating from one format to a different format.

0028The packets then are injected into the L5-L7 TAP <b>117</b> and are then injected into system memory via the PCIe bus or ACE interface module. The L5-L7 TAP co-processor <b>123</b> is for co-processors that operate on application payload, e.g. perform ACL/IDS/IPS, de-duplications, compression/de-compression, encryption/decryption.

0029Refer now to <figref idref="DRAWINGS">FIG. 5</figref> that summarizes the relationship of the pipeline steps and the TAP points. The co-processors CP(0), CP(1), . . . , CP(9) are optional as are the TAP points that are connected to them. A particular co-processor is instantiated in an SoC only when it is required, and the TAP point is only instantiated when it is required, and when neither is required the TAP point is replaced with a point-to-point connection between processing stages. The egress TAP points and co-coprocessors from <figref idref="DRAWINGS">FIG. 3</figref> are shown again as CP(0)-CP(4) <b>500</b>, <b>501</b>, <b>502</b>, <b>503</b> and <b>504</b>. The ingress TAP points and co-processors from <figref idref="DRAWINGS">FIG. 4</figref> are shown again as CP(5)-CP(9) <b>510</b>, <b>511</b>, <b>512</b>, <b>513</b>, and <b>514</b>.

0030Refer now to <figref idref="DRAWINGS">FIG. 6</figref> that shows the port number convention used to switch messages into and out of TAP points. A TAP point has 4 ports numbered 1-4, and CPL packets can arrive on each of the ports and packets can leave on each of the ports; the ports are bi-directional. The ports are numbered in clockwise fashion starting at the top port. A CPL message going in the egress direction and going to the next stage in the pipeline will arrive on port E=4 and proceed out of port 2 to the next stage in the egress pipeline. A CPL message that goes to CP(0) will instead be switched to port 1 and enter CP(0). A CPL message coming from CP(0) and arriving on port 1 can be switched to port 2 or 3. A CPL message coming from CP(0) or E=4 and looping around will be switched to port 3 in TAP point marked A in the Figure and will be switched to port 4 in the TAP point marked B in the Figure.

0031Refer now to <figref idref="DRAWINGS">FIGS. 7A to 7C</figref> that show an example switch box configuration for the TAP point switch box shown in <figref idref="DRAWINGS">FIG. 6</figref>. The message type is encoded as 8-bits in this example and the switching action is determined by looking up, in a per-port table, the port destination for the arriving message type. The port destination is a bit vector and a copy of a message is sent to each of the ports with a value 1 in the bit vector. Three switching table fragments are shown in the Figure. In <figref idref="DRAWINGS">FIG. 7A</figref>, the CPL_TX_DATA message with type 0x0C is switched to port 2, because bit 2 is set in the bit-vector, when it arrives on port 4 in TAP A. The CPL_TX2RX_DATA message with type 0x31 is switched to port 1 in TAP A which is CP(0). We assume that when this message type comes back from CP(0) on port 1 it is still encoded as CPL_TX2RX_DATA and based on the <figref idref="DRAWINGS">FIG. 7B</figref>, it is therefore switched to port 3 in TAP A., Finally when it arrives on port 1 in TAP B, based on <figref idref="DRAWINGS">FIG. 7C</figref>, it is switched to port 4, and thereby it completing the egress-to-egress loopback action with CP(0) getting an opportunity to process the payload while the CPL message is looped back.

0032Each of tap points in <figref idref="DRAWINGS">FIG. 5</figref> that are used can be configured in the manner shown in <figref idref="DRAWINGS">FIGS. 7A to 7C</figref> and can be used as a flexible framework to extend the operation of the basic protocol offload egress and ingress pipelines while retaining software NIC compatibility. The configuration can, for example, proceed in similar fashion to the one described in the U.S. Pat. No. 7,945,705 that uses PIO accesses or it can use CPL messages that have the effect of writing per-port switching tables in in the different TAP points being used.

0033We have thus described a generalization of TAP points for an interface device that offloads protocol processing, including advantageously enabling of different TAP points at different layers of the network protocol stack.