Bridge between parallel buses over a packet-switched network

ηιυρο ranan ηνΐηιρα *ni>n ranya pi "ivy>

BRIDGE BETWEEN PARALLEL BUSES OVER A PACKET-SWITCHED NETWORK MELLANOX TECHNOLOGIES LTD. C: 45062 am niaibiw vpubo PCT/IL00/00538 148263/2 WO 01/18988

RRinr.F. BETWEEN PARALLEL BUSES OVER A PACKET-SWITCHED NETWORK

FIELD OF THE INVENTION

The present invention relates generally to computing systems, and specifically to systems that use packet-switching fabrics to connect a computer host to peripheral devices.

BACKGROUND OF THE INVENTION

It should be noted that PCT document WO 01/18989 Al is incorporated here within in its entirety.

In current-generation computers,' the central processing unit (CPU) is connected to the system memory and to peripheral devices by a parallel bus, such as the ubiquitous Peripheral 15 Component Interface (PCI) bus. As data path-widths grow, and dock speeds become faster, however, the parallel bus is becoming too costly and complex to keep up with system demands.

In response, the computer industry is moving toward fast, packetized, serial Input/output (I/O) bus architectures, in which computing hosts and peripheral are linked by a switching network, commonly referred to as a switching fabric, A number of architectures of this type have been 20 proposed, including "Next Generation I/O" (NGIO) and "Future I/O" (FIO), culminating in the "InfiniBand" architecture, which has been advanced by a consortium led by a group of industry leaders (including Intel, Sun, Hewlett Packard, IBM, Compaq, Dell and Microsoft). Storage Area Networks (SAN) provide a similar, packetized, serial approach to high-speed storage

-sA access, which can also be implemented using an InfiniBand fabric. 25 PCI local buses are limited in tbe number of different devices that can be attached to them and in the physical distance that is allowed between any two devices on the bus. Tq . overcome these limitations, PCI-to-PCI bridges (referred to herein as "P2P" bridges) are used to link .two or more PCI local buses together. The bridge must implement logic that enables a device on one of the buses to write to and read from devices on the other bus, without violating 30 bus protocols and transaction ordering rules. Failure to observe the bus rules can lead to WO 01/18988 PCT/IL00/00S38 deadlocks and errors. PCI bus protocols and rules are described in detail in the PCI Local Bus

Specification, Revision 2.2 (1998), published by the PCI Special Interest Group (Hillsboro,

Oregon), which is incorporated herein by reference. See particularly section 3.2.5 (pages 40-44), "Transaction Ordering and Posting," and Appendix E (pages 267-277), "System t 5 Transaction Ordering."

According to the specification, there are nine types of PCI bus transactions, which can be divided into two general groups - posted and non-posted - based on how they are handled by an intermediate agent, such as a bridge. Posted transactions complete at the originating (master) device before they reach their ultimate destination. The master can thus proceed with other 10 work, including other bus transactions, before the posted transaction reaches the destination.

The bridge accepts the data on behalf of the actual destination device and assumes responsibility for assuring that the access completes properly at the destination. The PCI bus specification requires memory writes (including Memory Write and Memory Write and Invalidate commands) to be posted. 15 Non-posted transactions on the PCI bus cannot complete at the originating device until they have reached their ultimate destination. The master cannot proceed with any other dependent work until the transaction has completed and returned from the destination. In the case of a read transaction, this means that the requested data has been returned from the destination. For a write transaction, a write acknowledge must be returned. According to the 20 PCI specification, memory read transactions (Memory Read, Memory Read Line and Memory Read Multiple), as well as configuration transactions (Configuration Read and Configuration Write) and input/output (I/O) transactions (I/O Read and I/O Write) are non-posted.

Table I below summarizes the recognized PCI bus transactions, which are based on the division of the overall PCI address space into Memory, I/O and Configuration address spaces: /) 2 WO 01/18988 PCT/IL00/00538

TABLE I - PCI BUS TRANSACTIONS

Transaction name Meaning Posted? I/O Read Read data from agent mapped to I/O Address Space No I/O Write Write data to agent mapped to I/O Address Space No Configura- tion Read Read data from agent mapped to Configuration Address Space No Configura- tion Write Write data to agent mapped to Configuration Address Space No Memory Write Write data to agent mapped to Memory Address Space Yes Memory Write and Invalidate Write data to agent mapped to Memory Address Space, with minimum transfer of one complete cache line Yes Memory Read Read data from agent mapped to Memory Address Space No Memory Read Line Read a complete cache line of data from agent mapped to Memory Address Space No Memory Read Multiple Read multiple cache lines of data from agent mapped to Memory Address Space No

Other PCI bus cycles include Interrupt Acknowledge, Special Cycle and Dual Address Cycle, as 5 described in the above-mentioned PCI bus specification. The Dual Address Cycle is used by 32-bit devices to access a 64-bit address space in two phases, and must be supported by P2P bridges. Support for Interrupt Acknowledge and Special Cycle is optional.

Conventionally, a P2P bridge comprises a bridge device on each of the two PCI buses that it connects, with a logical and physical connection between the bridge devices to link the 10 buses. When a master device on a first bus wishes to initiate a transaction with a destination device on a second bus, it sends the appropriate bus command to an address of the destination 3 WO 01/18988 PCT/ILO0/OO538 device in the address space of the first bus, which was assigned to the destination device at boot-up. This address is within a range of addresses belonging to the bridge device on the first bus. The bridge device is configured to "claim" all cycles that are addressed to destination devices that reside on the second bus. After claiming the cycle, the bridge device on the first bus 5 forwards it to the bridge device on the second bus, which passes it on to the final destination.

When the command is a Posted Memory Write (PMW), the bridge device acknowledges the command immediately, and passes on the data to be written to the destination device. Other, non-posted Write commands, as well as Read commands, are treated as Delayed Write Requests (DWRs) and Delayed Read Requests (DRRs), respectively. When the bridge device receives 10 such a delayed request, it returns a Retry response to the master and passes the appropriate command over the bridge to the second bus. The bridge device then waits for an appropriate Delayed Write Completion (DWC), reporting whether or not the write was successfully performed, or Delayed Read Completion (DRC), returning the requested data, to come back over the bridge from the destination device. The transaction is completed only when the bridge 15 device has returned the DWC or DWR to the originating master.

To avoid errors and possible deadlocks, the bridge must service all transactions that it handles in an order that is consistent with PCI bus protocols. The above-mentioned Appendix E ofthe PCI bus specification sets forth the following ordering rules, which the bridge must obey: 1. A subsequent PMW submitted to the bridge target device cannot pass a 20 previously-accepted PMW, i.e., PMW transactions must complete in the order in which they are received. 2. A read transaction (DRR) must push posted write data ahead in order to maintain ordering, i.e., the read cannot pass the PMW. 3. A non-posted write transaction (DWR) must push posted write data ahead in order 25 to maintain ordering, i.e., the DWR cannot pass the PMW. 4. Before completion of a read transaction (DRC), any write data must first be pushed back to the originating bus of the read transaction, i.e., the DRC cannot pass the PMW (assuming the write command was accepted on the second bus before the read transaction was completed there). 30 5. A PMW must be allowed to pass a DRR or DWR to avoid deadlocks. 4 WO 01/18988 PCT/IL00/00538 6. A delayed completion (DWC or DRC) must be allowed to pass a delayed request (DWR or DRR) to avoid deadlocks. 7. A PMW must be allowed to pass a DWC or DRC to avoid deadlocks.

Other pairs of transactions do not have ordering constraints and can be handled by the bridge in any convenient order.

Because the PCI bus itself is a parallel bus, most P2P bridges known in the art uses similar parallel architectures. Recently, however, the Texas Instruments Semiconductor Group, of Santa Clara, California, announced a PCI-to-PCI bridge/transceiver chipset, which is said to deliver serialized PCI over twisted-pair cable at the theoretical maximum PCI bus speed of 132 MB/s. The chipset is designed to allow PCI buses to be expanded and extended over distances beyond normal PCI parallel bus limits by allowing two separate PCI parallel buses to be connected by a twisted-pair cable.

SUMMARY OF THE INVENTION

It is an object of some aspects of the present invention to provide an improved serial bridge between two parallel buses, and particularly between two PCI buses.

It is a further object of some aspects of the present invention to provide methods and devices for bridging between parallel buses over a packet-switched network.

It is yet a further object of some aspects of the present invention to provide methods and devices that map PCI bus commands to serial data packets and vice versa, while observing PCI transaction ordering rules.

In preferred embodiments of the present invention, a bridge links a first parallel bus to a second, remote parallel bus over a packet-switched network. Preferably, the first and second parallel buses comprises PCI buses, and the packet-switched network comprises an InfiniBand (IB) network, although the principles of the present invention are similarly applicable to buses and networks of other types. The bridge receives PCI bus read and write commands issued by a master device on the first bus, translates them into IB packets for transmission over the network, and then translates them back into the appropriate PCI bus commands for delivery to a destination device on the second bus. The response of the destination device is similarly translated and conveyed back over the bridge to the master device. (Commands issued by masters on the second bus with destinations on the first bus are preferably treated in like 5 WO 01/18988 PCT/IL00/00538 manner.) In transmitting and processing the IB packets, the bridge observes all PCI transaction ordering rules.

The bridge comprises a respective bridge device on each of the first and second PCI buses. Each bridge device preferably comprises a target channel adapter (TCA), connecting to 5 the respective bus, and a network interface, connecting to the IB network. Preferably, the network interface comprises a switch, but substantially any interface obeying IB rules may be used. The TCA converts each PCI read or write command that it receives into an IB read or write packet for transmission via the interface over the IB fabric. Preferably, the TCA assigns a respective IB network channel to each of the PCI address spaces to which the read and write 10 commands may relate, so that PCI memory, configuration and I/O transactions each get their own fixed channel through the network. The channel, in this context, refers to a particular path through the network and a service (or priority) level, wherein all packets on a given channel are transmitted and received in a fixed order. Different types of memory read transactions provided on the PCI bus are preferably distinguished by assigning an appropriate value to a data length 15 field in the corresponding IB read packets.

The respective channel for each address space is assigned in advance and is known to the TCA of the bridge device on the destination bus. When this destination TCA receives a read or write packet, it is thus able to determine from the packet's channel whether to issue a memory, configuration or I/O command on the destination bus. This use of network channels, in which 20 packet ordering is maintained, means that delayed read and write requests are conveyed and serviced in the order required by PCI rules.

In order to assure that posted write commands are allowed to be executed ahead of delayed read and write commands, as required by the rules, each TCA preferably comprises a request service buffer, most preferably a first-in-first-out (FIFO) buffer. When the TCA on the 25 destination bus receives a delayed read or write command over the bridge, it places the command in the buffer to await execution. Whenever the TCA receives a posted write command (identified as a write packet coming over the channel of the network assigned to memory transactions), however, it places the command on the destination bus immediately, passing the delayed commands in the buffer. 30 In some preferred embodiments of the present invention, three or more PCI buses are bridged together over a common IB network. In such embodiments, each of the buses has a 6 WO 01/18988 PCT/IL00/00538 bridge device that performs the functions described herein, and network channels are assigned for transmission of read and write packets between each pair of the bridge devices. Bridges

known in the art, whether parallel or serial, are capable only of bridging between a pair of PCI buses (or other parallel buses), so that multiple bridges are required when multiple buses are to 5 be connected. Because preferred embodiments of the present invention use the IB network (or other packet network) for bridging, they are not subject to this limitation.

There is therefore provided, in accordance with a preferred embodiment of the present invention, a method for bridging between first and second parallel buses having a common set of bus transactions and ordering rules governing execution of the transactions, the method 10 including: receiving at a target address on the first bus a bus command, associated with one of the transactions in the set, which command is issued by an originating device on the first bus for execution by a destination device on the second bus; sending a data packet embodying the received bus command from the target address on 15 the first bus over a packet-switched network to a master address on the second bus; and responsive to receiving the data packet at the master address, delivering the bus command over the second bus to the destination device in accordance with the ordering rules.

Preferably, the first and second parallel buses include Peripheral Component Interface (PCI) buses, and the packet-switched network includes an InfiniBand network. 20 Preferably, receiving the bus command at the target address includes assigning a range of addresses in an address space on the first bus to a bridge device for communication with the second bus, the range of address including the target address. Most preferably, assigning the range of addresses includes assigning multiple ranges of addresses to the bridge device, and mapping each of the ranges to a respective channel in the network, and sending the packet 25 includes sending the packet over the channel to which the range including the target address is mapped. In a preferred embodiment, assigning the multiple ranges includes assigning to the bridge device a respective range of addresses in each of a plurality of address spaces on the first bus, wherein the plurality of address spaces includes memory, configuration and input/output (I/O) address spaces. Additionally or alternatively, delivering the bus command over the second 30 bus includes mapping the channel over which the packet is received to one of the plurality of address spaces on the second bus. 7

<img img-format="tif" img-content="drawing" file="IL148263AD00021.tif" id="idf0001" />

5 WO 01/18988 PCT/IL00/00538 10 15 20 25 30

Further additionally or alternatively, receiving the command includes receiving multiple commands at one or more addresses in one of the assigned ranges, and mapping the target address range includes mapping all of the one or more addresses in the range to the same channel such that the packets are sent over the channel through the network in the order in which they were received at the bridge device.

Preferably, receiving the bus command includes receiving a read command for execution by the destination device, and wherein sending the data packet includes sending a network read request packet. Further preferably, receiving the read command includes receiving a command to read a given quantity of data from a memory associated with the destination device, and sending the network read request packet includes setting a data length field in the read request packet responsive to the given quantity. Additionally or alternatively, receiving the command to read the given quantity includes receiving a command to read one or more cache lines from the memory.

Further additionally or alternatively, the method includes receiving data over the second bus at the master address from the destination device responsive to the read command, and sending a response packet containing the received data from the master address over the network to the target address. Preferably, receiving the read command includes receiving first and second read commands, and wherein delivering the bus command includes delaying delivery of the second read command until the response packet containing the data received responsive to the first read command is sent.

Additionally or alternatively, receiving the bus command includes receiving a write command for execution by the destination device, and sending the data packet includes sending a network write request packet. Preferably, the method includes receiving over the second bus at the master address an acknowledgment from the destination device of execution of the write command, and sending a response packet responsive to the acknowledgment from the master address over the network to the target address.

Preferably, receiving the bus command includes receiving a request to execute a delayed transaction on the second bus, followed before completion of the delayed transaction by a request to execute a posted transaction on the second bus, and delivering the bus command includes servicing the posted transaction before completing the delayed transaction. Further preferably, delivering the bus command includes placing the delayed transaction request in a 8 WO 01/18988 PCT/ILOO/OOS38

<img img-format="tif" img-content="drawing" file="IL148263AD00022.tif" id="idf0002" />

5 10 15 20 25 30 buffer at the master address, and allowing the posted transaction request to bypass the buffer. Additionally or alternatively, the method includes receiving over the second bus at the master address a response to the delayed transaction request and sending a response packet embodying the response from the master address over the network to the target address before receiving the posted transaction request, wherein servicing the posted transaction includes placing the response in a buffer at the target address and conveying the response to the originating device only after servicing the posted transaction.

Additionally or alternatively, receiving the bus command includes receiving a request to execute a posted transaction on the second bus, followed by a request to execute a further transaction on the second bus before completion of the posted transaction, and delivering the bus command includes ensuring that the posted transaction is completed before servicing the further transaction.

There is also provided, in accordance with a preferred embodiment of the present invention, a method for bridging between an originating parallel bus and first and second destination parallel buses, all of the buses having a common set of bus transactions and ordering rules governing execution of the transactions, the method including: assigning first and second ranges of addresses in an address space on the originating bus to a bridge device for communication with the first and second destination buses respectively; receiving bus commands at first and second target addresses in the first and second ranges, respectively, which bus commands are associated with one or more of the transactions in the set and are issued by at least one originating device on the originating bus for execution by first and second destination devices on the first and second destination buses, respectively; sending first and second data packets embodying the received bus commands from the bridge device over respective serial links to first and second master addresses on the first and second destination buses, respectively; and responsive to receiving the data packets at the master addresses, delivering the bus commands over the first and second destination buses to the respective destination devices in accordance with the ordering rules.

Preferably, the serial links include paths through a packet-switched network, wherein assigning the first and second ranges of addresses includes mapping each of the first and second ranges to respective first and second channels in the network, and wherein sending the packets 9 WO 01/18988 PCT/IL00/00S38 includes sending the first and second data packets over the first and second channels, respectively.

There is additionally provided, in accordance with a preferred embodiment of the present invention, a method for bridging between first and second Peripheral Component Interface (PCI) 5 buses, including: receiving at a target address on the first PCI bus a PCI bus command issued by an originating device on the first PCI bus, the command instructing a destination device on the second PCI bus to carry out a PCI bus transaction; sending a data packet embodying the received bus command from the target address on 10 the first PCI bus over a serial link to a master address on the second PCI bus; and responsive to receiving the data packet at the master address, delivering the bus command over the second PCI bus to the destination device in accordance with PCI ordering rules.

Preferably, receiving the PCI bus command includes receiving multiple types of PCI read 15 commands selected from a group of commands consisting of Configuration Read, Input/Out (I/O) Read, Memory Read, Memory Read Line and Memory Read Multiple commands, and wherein sending the data packet includes mapping the multiple types to a single type of read request packets for transmission over the serial link. Most preferably, mapping the multiple types includes assigning the Configuration Read, I/O Read and Memory Read commands to 20 different channels on the serial link over which the read request packets corresponding to the commands are to be transmitted. Additionally or alternatively, mapping the multiple types includes assigning values to a data length field in the read request packets responsive to whether the command includes a Memory Read, Memory Read Line or Memory Read Multiple command. 25 Further additionally or alternatively, receiving the PCI bus command includes receiving multiple types of PCI write commands selected from a group of commands consisting of Configuration Write, Input/Out (I/O) Write, Memory Write, and Memory Write and Invalidate commands, and wherein sending the data packet includes mapping the multiple types to a single type of write request packets for transmission over the serial link. 30 Still further additionally or alternatively, receiving the PCI bus command includes receiving multiple, different types of PCI bus commands selected from a group of commands 10 WO 01/18988 PCT/ILOO/00538 consisting of Configuration commands, Input/Output (I/O) commands and Memory commands, and wherein sending the data packet includes mapping each of the different types of commands to different, respective channels on the link, such that data packets embodying the different types of commands are transmitted over the respective channels. Preferably, assigning the commands 5 includes associating the channels with respective ranges of addresses in the Configuration, I/O and Memory address spaces of the first PCI bus. Additionally or alternatively, receiving the multiple, different types of bus commands includes receiving the bus commands in a sequential order, and sending the data packet includes preserving the order of the bus commands among the data packets transmitted over each of the channels. Further additionally or alternatively, 10 delivering the bus command includes determining the type of bus command to deliver on the second PCI bus responsive to the channel over which the packet was received.

Preferably, the method includes receiving a response over the second PCI bus at the master address from the destination device responsive to the bus command, and sending a response packet embodying the response over the link to the target address on the first PCI bus. 15 Further preferably, receiving the PCI bus command includes receiving a delayed transaction request, followed by a Posted Memory Write (PMW) before completion of the delayed transaction, and wherein delivering the bus command includes servicing the PMW before completing the delayed transaction. Most preferably, delivering the bus command includes placing the delayed transaction request in a buffer at the second PCI bus, and allowing 20 the PMW to bypass the buffer.

There is further provided, in accordance with a preferred embodiment of the present invention, a device for bridging between first and second parallel buses having a common set of bus transactions and ordering rules governing execution of the transactions, the device including: a target channel adapter (TCA), which is configured to receive at a target address on the 25 first bus a bus command, associated with one of the transactions in the set, which command is issued by an originating device on the first bus for execution by a destination device on the second bus and to generate a data packet embodying the received bus command; and a network interface adapter, configured to transmit the data packet over a packet-switched network to a master address on the second bus, such that responsive to 30 receiving the data packet at the master address, the bus command is delivered over the second bus to the destination device in accordance with the ordering rules. 11 WO 01/18988 PCT/ILOO/00538

Preferably, a range of addresses including the target address in an address space on the first bus is assigned to the TCA. Further preferably, the range of addresses includes multiple ranges of addresses, each of which is mapped by the device to a respective channel in the network, such that the network interface adapter sends the packet over the channel to which the 5 range including the target address is mapped. Most preferably, the multiple ranges include a respective range of addresses in each of a plurality of address spaces on the first bus.

In a preferred embodiment, the bus command includes an outgoing bus command, and the data packet includes an outgoing data packet, and the TCA includes: a target unit, which is adapted to receive the outgoing bus command on the first bus and 10 to map the address of the command to the channel over which the outgoing packet is to be sent; and a master unit, which is adapted to receive an incoming packet sent over a further channel of the network from the second bus and responsive to the incoming packet, to map the further channel to a given address on the first bus and to convey an incoming command to the given 15 address responsive to the incoming packet.

Additionally or alternatively, the bus command includes a read command for execution by the destination device, responsive to which read command the TCA is adapted to generate a network read request packet to be transmitted over the network. In a preferred embodiment, the read command includes an outgoing read command, the read request packet includes an 20 outgoing read request packet, and the TCA includes: a target unit, which is adapted to receive the outgoing read command on the first bus and to generate the outgoing read request packet; and a master unit, which is adapted to receive an incoming read packet sent from the second bus and responsive to the incoming read packet, to deliver an incoming read command on the 25 first bus.

Preferably, the master unit is further adapted to receive incoming read data responsive to the incoming read command and to generate a response packet containing the received data to be sent to the second bus.

Further additionally or alternatively, the bus command includes a write command for 30 execution by the destination device, responsive to which write command the TCA is adapted to generate a network write request packet to be transmitted over the network. In a preferred 12 WO 01/18988 PCT/ILO0/00538

<img img-format="tif" img-content="drawing" file="IL148263AD00023.tif" id="idf0003" />

5 10 15 20 25 30 embodiment, the write command includes an outgoing write command, the write request packet includes an outgoing write request packet, and the TCA includes: a target unit, which is adapted to receive the outgoing write command on the first bus and to generate the outgoing write request packet; and a master unit, which is adapted to receive an incoming write packet sent from the second bus and responsive to the incoming write packet, to deliver an incoming write command on the first bus.

There is moreover provided, in accordance with a preferred embodiment of the present invention, a device for bridging between an originating parallel bus and first and second destination parallel buses, all of the buses having a common set of bus transactions and ordering rules governing execution of the transactions, the device including: a target channel adapter (TCA), to which first and second ranges of addresses in an address space on the originating bus are assigned for communication with the first and second destination buses respectively, and which is adapted to receive bus commands at first and second target addresses in the first and second ranges, respectively, which bus commands are associated with one or more of the transactions in the set and are issued by at least one originating device on the originating bus for execution by first and second destination devices on the first and second destination buses, respectively, and to generate first and second data packets embodying the received bus commands; and a network adapter, configured to transmit the first and second data packets over respective serial links to first and second master addresses on the first and second destination buses, respectively, such that responsive to receiving the data packets at the master addresses, the bus commands are delivered over the first and second destination buses to the respective destination devices in accordance with the ordering rules.

There is furthermore provided, in accordance with a preferred embodiment of the present invention, a device for bridging between first and second Peripheral Component Interface (PCI) buses, including: a target channel adapter (TCA), adapted to receive at a target address on the first PCI bus a PCI bus command issued by an originating device on the first PCI bus, the command instructing a destination device on the second PCI bus to carry out a PCI bus transaction, and to generate a data packet embodying the received bus command; and 13 WO 01/18988 PCT/ILOO/00538

<img img-format="tif" img-content="drawing" file="IL148263AD00024.tif" id="idf0004" />

5 10 15 20 25 a network adapter, configured to transmit the data packet over a serial link to a master address on the second PCI bus, such that responsive to receiving the data packet at the master address, the bus command is delivered over the second PCI bus to the destination device in accordance with PCI ordering rules.

In a preferred embodiment, the PCI bus command includes an outgoing command, the data packet includes an outgoing data packet, and the TCA includes: a target unit, which is adapted to receive the outgoing PCI bus command on the first PCI bus and to generate the outgoing data packet; and a master unit, which is adapted to receive an incoming data packet sent from the second bus and responsive to the incoming data packet, to deliver an incoming PCI bus command on the first bus.

The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof^ taken together with the drawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a block diagram that schematically illustrates a system for bridging between multiple parallel buses over a packet-switched network, in accordance with a preferred embodiment of the present invention;

Fig. 2 is a block diagram that schematically illustrates details of a bridging device in the system of Fig. 1, in accordance with a preferred embodiment of the present invention;

Fig. 3 is a flow chart that schematically illustrates a method for carrying out a read transaction over a bridge, in accordance with a preferred embodiment of the present invention;

Fig. 4 is a flow chart that schematically illustrates details of a method for transmitting a bus command through a packet-switched network, in accordance with a preferred embodiment of the present invention;

Fig. 5 is a flow chart that schematically illustrates details of a method for servicing a bus read request received through a packet-switched network, in accordance with a preferred embodiment of the present invention; and

Fig. 6 is a block diagram that schematically illustrates a further detail of the bridging device of Fig. 2, in accordance with a preferred embodiment of the present invention. 14 WO 01/18988 PCT/ILOO/00538

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Fig. 1 is a block diagram that schematically illustrates a system 20 for bridging between multiple PCI buses 22, 34 and 40, in accordance with a preferred embodiment of the present invention. Each of the PCI buses is linked by a respective bridge device 28, 32 or 38 to an InfiniBand (IB) packet-switched network 30. Each pair of bridge devices, together with network 30, forms a PCI-to-PCI (P2P) bridge between the respective buses. While preferred embodiments are described hereinbelow with reference to PCI buses and the IB network, it will be understood that the principles of the present invention may be applied, mutatis mutandis, to bridge between parallel buses of other types, and over packet-switched networks obeying different standards.

In the example shown in Fig. 1, a master device on bus 22, such as a central processing unit (CPU) 24, wishes to read from or write to a destination peripheral device 36 on bus 34. The master device issues an appropriate PCI read or write command to an address in the address space of PCI bus 22 that is assigned to bridge device 28. The remainder of the PCI bus address space is typically assigned to peripheral and I/O devices 26 on bus 22. Bridge device 28 generates an IB command based on the CPU request and sends it over network 30 to bridge device 32. The latter bridge device decodes the IB command and generates the appropriate PCI bus command over bus 34 to peripheral device 36. The peripheral device returns the appropriate response (typically data, in the case of a read command, or an acknowledgment of a write command) back to bridge device 32. This bridge device generates a corresponding IB response packet and returns the response over network 30 to bridge device 28, which passes the response over bus 22 to the CPU.

In accordance with the PCI standard, each device on PCI bus 22 can have multiple address "windows," or segments, assigned to it in each of the memory, configuration and I/O address spaces on the bus 22. Preferably, sixteen of these windows in each of the three address spaces are assigned to bridge device 28, although alternatively, a larger or smaller number of windows may be assigned. Further aspects of the allocation and definition of these segments are described in the above-mentioned PCT patent application entitled "Parallel Bus Communications over a Packet-Switching Fabric." Upon start-up or reconfiguration of system 20, CPU 24 is informed of an address for peripheral device 36, which is within one of the address windows assigned to bridge device 28. The CPU thus addresses all memory read and write commands 15 WO 01/18988 PCT/IL00/00538

<img img-format="tif" img-content="drawing" file="IL148263AD00025.tif" id="idf0005" />

10 15 20 25 destined for devices on bus 34 to the appropriate memory window of bridge device 28 in the PCI memory address space; all configuration read and write commands to the appropriate configuration window of bridge device 28 in the PCI configuration address space; and all I/O read and write commands to the appropriate I/O window of bridge device 28 in the PCI I/O address space. Optionally, among the windows assigned to bridge device 28, there are also windows associated with channels over network 30 to bridge device 38, for addressing a peripheral device 42 on bus 40, as well as to additional PCI buses not shown in the figure.

Each of the address windows assigned to bridge device 28 is associated with a different channel, or Work Queue Pair (WQP) in IB network 30. As noted above, each channel in the IB network corresponds to a fixed path and level of service through the network, and packet ordering is maintained among all packets carried by a given channel. Similar methods of maintaining packet order are known in other types of networks known in the art. As a consequence of this channel structure, all read and write requests to a given part of the address space of PCI bus 34 (memory, configuration or I/O) are received by bridge device 32 in the exact order in which they were sent by bridge device 28. Responses of peripheral 36 are similarly sent to back to bridge device 32 and are returned to CPU 24 in the proper order.

Fig. 2 is a block diagram that schematically illustrates details of bridging device 28, in accordance with a preferred embodiment of the present invention. The structure of devices 32 and 38 is preferably also as shown in this figure. Device 28 comprises a target channel adapter (TCA) 50, which is coupled to PCI bus 22, and a switch 52, coupled to network 30. Preferred switches for this purpose are described in the above-mentioned provisional patent applications 60/152,849 and 60/175,339, for example. The design of switch 52 is beyond the scope of the present patent application, however, and substantially any packet switch meeting InfiniBand specifications, or which at least has the feature of maintaining packet order -within a channel, will be suitable for carrying out the function of switch 52. TCA 50 is responsible for converting PCI commands (or cycles) into IB packets, and vice versa. The TCA comprises a PCI target unit (PCIT) 54 and a PCI master unit (PCIM) 56. PCIT 54 responds to transactions originating on PCI bus 22, which are initiated by a bus master such as CPU 24. In particular, the PCIT receives read and write commands from the PCI bus and generates the appropriate IB read and write request packets for transmission via switch 52. When response packets are received from the destination bus, PCIT 54 receives these packets 30 16 WO 01/18988 PCT/ILOO/00538 and generates the appropriate response cycles on PCI bus 22 (returning data in response to a read request or acknowledgment of a write request). PCIM 56, on the other hand, is responsible for receiving IB read and write request packets from network 30, sent by a master device on a remote originating PCI bus. It converts these requests into the appropriate read and write 5 commands on bus 22, which in this case is the destination bus. The PCIM then packetizes the resulting response to send back to the originating bus.

Fig. 3 is a flow chart that schematically illustrates a method for carrying out a PCI read transaction over IB network 30, in accordance with a preferred embodiment of the present invention. At a read request step 70, CPU 24 issues a read command, which is directed to 10 peripheral 36. As noted above, the CPU sends the request to an address within an appropriate window in the address space of bus 22 that is assigned to TCA 50. As a result, the request is received and processed by PCIT 54. PCIT 54 responds to the read request with a PCI "retry" response to CPU 24 on bus 22, at an acknowledgment step 72, informing the CPU that the requested data are not yet available, 15 so that the CPU should retry the request later. As noted above, when the CPU issues a posted memory write (PMW) command, the PCIT acknowledges the command immediately. The CPU must continue to retry non-posted read and write requests, however, until the delayed response is received from destination bus 34. At the same time as it issues the retry response, the PCIT records the read request in a buffer 58 and allocates space to receive the read response data that 20 are expected to arrive from device 36. The read request is recorded in the buffer with sufficient identifying information so that the PCIT will be able to associate the read response packet that it later receives from device 36 with the particular read request that led to it.

As shown in Table I above, there are five different types of PCI read requests that CPU 24 can send. Depending on the type and address of the read request, PCIT 54 constructs an 25 appropriate IB Remote Direct Memory Access (RDMA) read request packet, and submits the packet to switch 52, at a packet transmission step 74. The PCIT issues the IB RDMA packets in exactly the order in which the corresponding read and write commands were received from bus 22. The PCIT maps the PCI address in the PCI read command to a predetermined IB channel and virtual address (i.e., logical address) that will direct the corresponding IB read request 30 packet to bridge device 32 on destination bus 34 and from there to peripheral 36. Further aspects of the mapping of PCI commands to IB requests and network channels are described in the 17 WO 01/18988 PCT/ILOO/00538

<img img-format="tif" img-content="drawing" file="IL148263AD00026.tif" id="idf0006" />

10 15 20 25 above-mentioned PCT patent application entitled "Parallel Bus Communications over a Packet-Switching Fabric."

The IB RDMA read request packet sent by bridge device 28 is received by bridge device 32 at a packet reception step 76. For the purposes of the present embodiment, device 32 is considered to be substantially identical in construction to device 28, although other designs may also be used. (In fact, one advantage of the mapping between PCI address windows and IB channels provided by preferred embodiments of the present invention is that it allows different types of bridging devices on different buses to communicate with one another in a straightforward manner. There is preferably no information embedded in the IB packets that is specific or proprietary to the bridge devices themselves. Furthermore, device 28 is preferably capable of sending packets to substantially any IB-compliant end-point, such as an I/O controller on network 30, and not only to the other bridge devices.) Upon receiving the packet, switch 52 of device 32 submits it to PCIM 56 of this device, which places the request in a buffer 60 until the bus is available to service the request. PCIM 56 of device 32 maps the channel and virtual address of the IB packet to the physical address of PCI peripheral 36 on bus 34, at an address mapping step 78. Depending on the channel and data length of the RDMA packet, the PCIM chooses the appropriate type of PCI read command to send to the peripheral, and receives the requested data in accordance with PCI bus procedures. The PCIM and generates an IB read response packet for transmission back to bridge device 28. The response packet is submitted to switch 52 and is transmitted by the switch over network 30, at a response transmission step 80. The PCIM sends the read response packet, to complete the delayed read transaction (DRC), before servicing the next read or write request in buffer 60. Delayed write completion (DWC) acknowledgments are likewise given priority over new pending requests.

When the IB read response packet returns to bridge device 28, switch 52 routes it to PCIT 54, which buffers the data, at a response reception step 82. The PCIT compares the response information to the pending requests in buffer 58 and retires from the buffer the delayed read request (DRR) that has now been fulfilled. On the next retry of this read request by CPU 24, PCIT 54 conveys the received data via bus 22 to the CPU, at a data delivery step 84. If any posted memory write (PMW) commands are issued on PCI bus 22 while the response data are 30 18 WO 01/18988 PCT/IL00/00538 waiting in the buffer, however, PCIT 54 will not attempt to place the data on the bus until the PMW has been completed, so as to ensure that PCI transaction ordering rules are observed.

Fig. 4 is a flow chart that schematically illustrates a method for converting a PCI bus command to an IB packet, in accordance with a preferred embodiment of the present invention. 5 This conversion is used, inter alia, at step 74 in the method of Fig. 3. It enables the nine different types of PCI read and write commands (as listed in Table I) to be mapped to appropriate read and write request packets on the IB network.

Various types of packets may be sent through the IB network switching fabric, invoking different sorts of responses at the packet destination. The packet type is specified by an 10 Extended Transport Header (ΕΊΉ) associated with the transport layer of the IB protocol stack. Read and write operations are invoked by a Remote Direct Memory Access (RDMA) ETH. The first message of a RDMA read or write request contains a RDMA ETH specifying a virtual address for the operation, an access key (R_key) and a total length of the data buffer to read or write. The RDMA write request includes a payload containing the data to be written (which 15 may be continued in subsequent RDMA write packets). An entity that receives the write request responds with an acknowledge packet, having an Acknowledge ΕΤΗ (AETH). The AETH is also present in Read Response packets returned by an entity that receives the read request. Such Read Response packets also have a payload containing the requested data. Those skilled in the art will recognize that packets types and operations similar to these are also encountered in 20 packet-switched networks of other sorts known in the art.

At a cycle reception step 90, PCIT 54 of bridge device 28 receives a read or write command on the PCI bus. At an address space determination step 92, the PCIT examines to which ofthe three PCI address spaces the command is addressed: memory, configuration or I/O. The PCIT determines the IB channel over which a corresponding packet is to be sent depending 25 on the address space. Thus, if the address is in the I/O space, a RDMA read or write packet is transmitted on channel "A," for example, at an I/O packet construction step 94. In this case, the length of the data buffer will be no more than 32 bits, in accordance with PCI convention. (This channel number and the other channel numbers cited below are chosen arbitrarily, for the sake of illustration.) On the other hand, if the address is in the configuration space, the 30 corresponding RDMA read or write packet is transmitted on channel "B," at a configuration packet construction step 96. Here the length of the data buffer will similarly be no more than 4 19 WO 01/18988 PCT/ILOO/00538 bytes, as indicated by the Byte Enable (BE) bits written by the master device (CPU 24), in accordance with PCI convention.

If the address for the PCI bus command is in the memory space of the PCI bus, the corresponding RDMA read or write packet is designated to be sent on channel "C," at a

5 memory packet selection step 98. Construction of the IB packet depends on whether it is to carry a read or write request. As shown in Table I, there are two types of memory write transactions: Write, and Write and Invalidate. At an invalidation step 100, it is determined whether the current cycle is a Write and Invalidate cycle. If so, a dedicated Memory Write and Invalidate (MWI) bit is set in a configuration portion of the RDMA write packet, at a MWI 10 setting step 102. The packet is then prepared, at a write packet sending step 104, with a data length given by the length of the actual data specified by the PCI write command. For memory and I/O write commands, an IB send packet is preferably used, which does not explicitly specify an address on bus 34. Bridge device 32 then maps the resulting PCI command to a command buffer of destination device 36. Configuration write commands, on the other hand, preferably 15 use an IB RDMA write packet and specify the address on bus 34 to which the data are to be written. Alternatively, however, all PCI write commands may be mapped to either IB write or IB send packets, depending on how master device 24 is to interact with destination device 36. The mapping in each case is preferably associated with the assigned IB channel.

If a PCI memory read command is encountered, a RDMA read packet is sent on channel 20 C, at a read packet sending step 106. In this case, the data length in the RDMA packet is determined by the type and prefetch length of the PCI memory read that has been invoked. For Memory Read Line, the data length is determined by the memory length to be read up to the next cache line of master device 24. For Memory Read Multiple, the data length is given by the multiple cache lines to be read, up to the next page boundary in the memory.

25 Fig. 5 is a flow chart that schematically illustrates a method for converting an IB RDMA read request packet to a PCI bus read command, in accordance with a preferred embodiment of the present invention. This conversion is used at step 78 in the method of Fig. 3. It is further described in the above-mentioned PCT patent application entitled "Parallel Bus Communications over a Packet-Switching Fabric." The read request packet is received by PCIM 56, at a packet 30 reception step 110. The PCIM examines the IB channel over which the packet was received, at a channel determination step 112. If the packet was received on channel A, the PCIM issues an 20

<img img-format="tif" img-content="drawing" file="IL148263AD00027.tif" id="idf0007" />

WO 01/18988 PCT/ILOO/00538 I/O read command to the appropriate device on the PCI bus, at an I/O reading step 114.

Similarly, if the packet was received on channel B, the PCIM issues a configuration read command, at a configuration reading step 116. I/O and configuration write commands are generated in a similar fashion.

If the IB RDMA read request was received on channel C, it corresponds to a memory read command. At a read length examination step 118, PCIM 56 determines what type of read command to issue on the PCI bus depending on the data length of the IB packet. If the data length is less than four bytes, a simple PCI Memory Read command is issued, at a read step 120. If the data length is more than four bytes, but does not cross any cache line boundary, the PCIM issues a Memory Read Line command, at a read line step 122. Otherwise, if the data length is more than four bytes and reading the data results in crossing a cache line boundary, the PCIM must issue a Memory Read Multiple command, at a multiple read step 124.

For IB write requests on channel C (not shown in this figure), PCIM 56 issues the appropriate Memory Write command on the PCI bus. If the MWI bit in the IB packet is set, and the packet data include an integral number of cache lines with an address that is cache line-aligned, the PCIM will issue a Memory Write and Invalidate command. Otherwise, an ordinary Memory Write will be issued. Thus, all of the different types of PCI read and write operations are mapped to IB RDMA read and write request packets, and are then converted back to the appropriate types of PCI operations upon arrival at the destination bus.

Fig. 6 is a block diagram that schematically illustrates a detail of PCI master unit (PCIM) 56 and its associated buffer 60, in accordance with a preferred embodiment of the present invention. When PCIM 56 receives any delayed request (DRR or DWR) over IB network 30, it places the request in buffer 60, preferably in a FIFO queue 130. If the queue overflows, the PCIM preferably notifies the sending PCIT that it has discarded the excess RDMA request packet, and the PCIT resends the packet subsequently.

On the other hand, when PCIM 56 receives a posted write request 132, it does not enter it in buffer 60, but rather passes it directly through for delivery to the PCI bus. An arbiter 134 gives priority to request 132 over all of the pending delayed requests waiting in the buffer. Thus, when a PMW reaches PCIM 56, it will always be allowed to pass pending DRRs and DWRs waiting in the buffer. Similarly, DRCs and DRWs are not allowed to pass to TCA 50 if 21 WO 01/18988 PCT/ILO0/00538 there are any non-completed PMWs waiting in PCIM 56. The delayed completion packets are allowed, however, to pass previously-arrived delayed request packets.

In summary, reference is now made to the seven PCI ordering rules listed in the

Background of the Invention. It will be observed that the operation of PCIM 56, and 5 particularly its management of the queues in buffer 60, ensures that the present bridge observes rules 5 and 6, i.e., that PMWs, DRCs and DWCs are allowed to pass DRRs and DWRs. Furthermore, because PCIT 54 and IB network 30 preserve the order of the read and write commands received from the PCI bus, there is no possibility that a PMW will be passed by any other type of transaction. Thus, rules I, 2 and 3 are observed. Finally, because PCIT 54 buffers 10 DRCs and DWCs that it receives from the destination bridge device, and does not place the corresponding data on the PCI bus until any pending PMWs on the bus have been completed, rules 4 and 7 are observed, as well. System 20 thus meets all of the requirements of a P2P bridge, with the added flexibility afforded by bridging through IB network 30. It will be apparent to those skilled in the art that elements of system 20 and of the methods described 15 hereinabove may be altered in a straightforward manner if necessary to adapt to buses, networks, transactions and ordering rules of other types.

It will thus be appreciated that the preferred embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both 20 combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. 22 , crnxan rwzn ατα inia^n ριοιιτ ρηιζη ιγπ πτ naoa ,ρηη ηχΒ- laoana mavia n^maa πρ^ποώ .zrtwan ™·Μ rnirnn τπυ1? oxnm □ιηπη τν evaCtOffi ^·*'*^·*· * ***9^"·** {yyj — 1$ JH2CIB OS S3:45 40200

<img img-format="tif" img-content="drawing" file="IL148263AD00028.tif" id="idf0008" />

.(mcna navin) crus&amp;amp;'an nwa