Integrated circuit switch with embedded processor

INTEGRATED CIRCUIT SWITCH WITH EMBEDDED PROCESSOR

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Patent Application 60/152,849, filed September 8, 1999, and of U.S. Provisional Patent Application 60/175,339, filed January 10, 2000, which are assigned to the assignee of the present patent application and which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to systems and methods for high-speed data transfer, and specifically to intelligent switch devices for use in switching fabrics and networks.

BACKGROUND OF THE INVENTION

In current-generation computers, the central processing unit (CPU) is connected to the system memory and to peripheral devices by a parallel bus, such as the ubiquitous Peripheral Component Interface (PCI) bus. As data path-widths grow, and clock speeds become faster,' however, the parallel bus is becoming too costly and complex to keep up with system demands. In response, the computer industry is moving toward fast, packetized, serial input/output (I/O) bus architectures, in which computing hosts and peripheral are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, including Next Generation I/O״ (NGIO) and Future I/O״ (FIO), culminating in the InfiniBand״ architecture, which has been advanced by a consortium led by a group of industry leaders (including Intel, Sun, Hewlett Packard, IBM, Compaq, Dell and Microsoft). Storage Area Networks (SAN) provide a similar, packetized, serial approach to

WO 01/18977 high-speed storage access, which can also be implemented using an InfiniBand fabric.

Components within the InfiniBand architecture communicate by transferring packets of data between themselves. A channel adapter is a component within a processor node or an input/output unit which typically interfaces with a bus therein, and converts data between packet and parallel forms. The channel adapter generates packets for transmission, as well as consuming received packets. A switch is a component which, except for management packets, does not generate or consume packets. The switch passes the packet along, based on a destination address in a header of the packet. Each channel adapter and switch are typically controlled by one or more dedicated CPUs external to the components.

WO 01/18977

SUMMARY OF THE INVENTION

It is an object of some aspects of the present invention to provide a switching network interface operating between a plurality of ports at an improved rate.

It is a further object of some aspects of the present invention to provide a switching network interface whose configuration can be easily upgraded.

In preferred embodiments of the present invention, a switching network interface comprises a packet switch internally coupled within the interface to a dedicated Embedded RISC (Reduced Instruction Set Computer) Processor. Most preferably, the interface also comprises a channel adapter which is coupled to the switch, and a random access memory (RAM) which is coupled to and which is available for the use of the Embedded RISC Processor (ERP). The interface is implemented as a single device, most preferably as a VLSI (very large scale integrated circuit) , and is coupled by a bus to one or more peripheral devices external to the interface. The packet switch comprises a plurality of external ports coupled to a switching fabric, between which ports and the bus packets generated within the fabric are routed by the packet switch and the ERP. Packets are routed between the channel adapter and the packet switch via one. or more ports internal to the interface.

On initialization of the network interface, microcode is loaded into the RAM from a boot ROM (read only memory) comprised in the switching fabric. Once the RAM has been loaded, the ERP utilizes the microcode in order to control and set the routing of the packet switch. The close proximity of the ERP to the packet switch, and the implementation of the two elements as one physical interface, enables the switch to be configured and to be managed substantially faster than switching systems known in the art. The one physical interface also

WO 01/18977 - PCT/IL00/00544 achieves significant savings in cost, compared to using separate components. Furthermore, the presence.of the RAM in the network interface means that operating instructions for the ERP can be easily rewritten, so that operation of the packet switch within the interface can be easily reconfigured.

There is therefore provided, according to a preferred embodiment of the present invention a switching interface for a packet-switching network, including:

a packet switch, adapted to convey data packets generated in the packet-switching network; and a central processing unit (CPU), adapted to control the packet switch, such that the packet switch and CPU are implemented as a single integrated circuit device.

Preferably, the interface includes a channel adapter coupled to the packet switch, wherein the channel adapter is adapted to reconfigure the data packets transferred between the packet switch and the adapter and. is at least partly controlled by the CPU and is implemented within the single device.

Preferably, the data packets include a management packet containing control instructions for the channel adapter, and the CPU is operative to assist the channel adapter in processing the management packet.

Preferably, the interface includes a CPU external to the single device, which external CPU is adapted to at least partly control the channel adapter.

Further preferably, tasks performed by the CPU and the external CPU are partitioned so as to optimize performance of the interface.

Preferably, the data packets include a management packet containing control instructions for the packet switch, and the CPU is operative to assist the packet switch in processing the management packet.

WO 01/18977 ׳ PCT/IL00/00544

Preferably, the interface includes a random access memory (RAM) implemented within the single device, wherein is stored code received from a source external to the device which is adapted to operate the CPU.

Preferably, the CPU includes a Reduced Instruction Set Computer (RISC) Processor.

Preferably, the packet-switching network includes an Infiniband switching fabric.

There is further provided, according to a preferred embodiment of the present invention, a method for switching data within a packet-switching network, including:

conveying data packets generated within the packetswitching network via a packet switch;

controlling the packet switch with a central . processing unit (CPU), which is implemented together with the packet switch as a single integrated circuit device.

Preferably, the method includes:

coupling a channel adapter to the packet switch, which adapter is adapted to reconfigure the'data packets transferred between the packet switch and the adapter;

controlling the channel adapter at least partly by the CPU; and implementing the channel adapter in the single device.

Preferably, the data packets include a management packet containing control instructions for the channel adapter, and controlling the channel adapter includes the CPU assisting the channel adapter to process the management packet.

Preferably, controlling the channel adapter includes at least partly controlling the channel adapter by another CPU external to the single device.

Further preferably, controlling the channel adapter includes partitioning tasks performed by the CPU and the

WO 01/18977 external CPU so as to optimize performance of the single device.

Preferably, the method includes implementing a random access memory (RAM) within the' single device, wherein is stored code received from a source external to the device which is adapted to operate the CPU.

Preferably, the CPU is a Reduced Instruction Set Computer (RISC) Processor.

Preferably, the packet-switching network includes an Infiniband switching fabric.

Preferably, the data packets include a management packet containing control instructions for the packet switch, and controlling the packet switch includes the CPU assisting the packet switch to process the management packet.

The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings in which:

WO 01/18977

PCT/ILOO/OOS44

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a schematic block diagram of a switching network interface, according to a preferred embodiment of the present invention; and

Fig. 2 is a flowchart that schematically illustrates a method by which the interface of Fig. 1 operates, according to a preferred embodiment of the present invention.

WO 01/18977 PCT/ILOO/O0544

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Fig. 1 is a schematic block diagram of a switching network interface 20, according to a preferred embodiment of the present invention. Interface 20 comprises a switching fabric packet switch 24 which is coupled by a plurality of external ports 1,2,3,4 to a switching fabric

22. Interface 20 is coupled to a standard I/O bus 34, such as a PCI bus, for the purpose of communicating with, one or more peripheral devices 3 6 on the bus, such as an I/O adapter and/or a central processing unit (CPU) . The switching fabric preferably comprises an InfiniBand fabric, as described in the Background of the Invention, and some of the terms used hereinbelow are specific to the InfiniBand architecture. It will be understood, however, that the system architecture and methods of communication. described herein are in no way limited to InfiniBand, and that other switching fabrics, as are known in the art, may be configured to handle and convey data packets in a similar manner.

Switch 24 is coupled by a first internal port 0 to an embedded RISC processor (ERP) 30 which functions as a central processing unit (CPU) controlling the operation of the switch. Inter alia, ERP 30 operates as a subnet manager configuring switch 24 and loading a forwarding table which is used by the switch to route individual packets through the switch. ERP 30 also supports transfer of general service queue pair (QP) packets, and handles packets which are classed as exceptions. The operation of ERP 30 is described in more detail below with reference to Fig. 2. In addition to being coupled to switch 24, the ERP is coupled to and is able to at least partly control the operation of a channel adapter 26.

Switch 24 is also coupled by a second internal port 5 to channel adapter 26. If interface 20 is installed in a host unit of the switching fabric, channel adapter 26 most preferably comprises a host channel adapter.

WO 01/18977

Alternatively, if interface 20 is installed in a peripheral unit of the switching fabric, channel adapter 26 most preferably comprises a target channel adapter. Most preferably, adapter 26, in addition to being controlled by ERP 30, is at least partly controlled by a CPU 28, external to interface 20, comprised in the respective unit within which interface 20 is installed. Further most preferably, data packets are transferred between adapter 26 and the unit within which interface 20 is installed via bus 34. A random access memory (RAM) 32, comprised in interface 20, is coupled to ERP 30 and is utilized to store operating instructions for the ERP. Switching interface 20 is implemented as a single device, most preferably as a VLSI (very large scale integrated circuit), comprising the elements adapter 26, switch 24, ERP 30 and RAM 32, and appropriate connections between the elements.

Fig. 2 is a flowchart that schematically illustrates a method by which interface 20 operates, according to a preferred embodiment of the present invention. In an initialization step, switching fabric 22 is booted and code for interface 20 is read from a non-volatile memory, comprised in the fabric, such as a read only memory (ROM) or a disk operating the fabric. In a load step the operating code for interface 20 is read, most preferably as microcode, into RAM 32. Once the operating code has been read and acknowledged in RAM 32, packet switch 24 is configured by ERP 30, which loads one or more forwarding tables into the switch, and which controls the switch. It will be appreciated that the operation of interface 20 can be easily upgraded by loading new operating code into RAM 32.

In a packet reception step, switch 24 determines a type of packet received by the switch. The type of packet is most preferably one of a group consisting of a management packet destined for switch 24 or adapter 26, a

WO 01/18977 general queue pair (QP) packet/ and a packet not included in either of the above groups. Switch 24 determines the type of packet according to a destination address, specifying a local identifier (LID) ׳of the packet destination, comprised in the packet, and according to how the switch has been configured by the ERP in the load step.

Management packets contain control instructions for switch 24 or adapter 26. They are placed in a dedicated register of the switch or adapter, as appropriate, which then attempts to decode the instructions and carry them out. Typically, the processing capabilities of the switch and adapter are very limited, and they are assisted by ERP 30 in dealing with at least some of these management packets. For example, if the packet is a management packet comprising an update to a forwarding table of switch 24, ERP 30 accepts the packet and configures the switch to perform the update required by the packet. In this case the packet is consumed by switch

24. Under instruction of ERP 30, switch 24 may also generate a management packet and transmit the packet to its destination. For example, ERP 30 can be utilized to collect and process statistics on traffic through interface 20. The statistics and/or results are then forwarded as one or more management packets to other components in fabric 22, where they can be utilized to improve the operation of the fabric, for example, by reconfiguring routing tables in the entire fabric based on dynamic load information.

If the packet is a QP packet destined for the unit within which interface 20 is installed, switch 24 transfers the packet to channel adapter 26, which in turn reconfigures the packet and transfers it to the unit via bus 34.

Packets which are neither management packets nor QP packets are typically one of three types - a packet which

WO 01/18977

PCT/ILO0/OO544 is the result of an external interrupt, a packet which is unrecognized after arriving at peripheral devices 36, and a packet which occurs after an unrecognized transaction has occurred on bus 34. When such a packet is received by switch 24, the packet is transferred to ERP 30. The ERP generates an exception packet and sends the exception packet to a destination appropriate to the packet causing the exception to be generated, for example the originating address. If the packet generating the exception packet is a data packet, the exception packet is preferably sent on the same path as the data packet.

In preferred embodiments of the present invention, tasks performed by ERP 30 and by another CPU, such as CPU 28 or a CPU present in peripheral devices 36, may be partitioned between the two processors so as to optimize system performance. An example of two possible methods of partitioning tasks comprised in transferring an interrupt packet is described in U.S. patent application 09/559352, which application is assigned to the assignee of the present invention, and whose disclosure is incorporated herein by reference. The presence of ERP 30 as an integral component of interface 20 significantly enhances the performance of the interface compared with systems where at least some of the processor, switch, and channel adapter are separate elements.

It will be appreciated that the preferred embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.