Parallel bus communications over a packet-switching fabric

PARALLEL BUS COMMUNICATIONS OVER A PACKET-SWITCHING FABRIC

FIELD OF THE INVENTION

The present invention relates generally to computing systems, and specifically to devices and methods for bridging between a parallel bus and a switching fabric or network.

BACKGROUND OF THE INVENTION

In current-generation computers, the central processing unit (CPU) is connected to the system memory and to peripheral devices by a parallel bus, such as the ubiquitous Peripheral 15 Component Interface (PCI) bus. PCI bus-protocols and rules are described in detail in the PCI Local Bus Specification, Revision 2.2 (1998), published by the PCI Special Interest Group (Hillsboro, Oregon), which is incorporated herein by reference. Briefly, the PCI specification defines a set of bus commands, or cycles, each identified by a 4-bit code on the C/BE (Bus Command and Byte Enable) lines of the bus. A bus master, such as a CPU, sends commands to 20 a target device on the bus by specifying a 32־ or 64-bit bus address of the target device. Table I below lists the PCI bus command codes for reference. The different types of read and write commands in the table are based on the division of the overall PCI address space into Memory, I/O and Configuration address spaces.

TABLE I - PCI BUS COMMANDS

<td> C/BE</td><td> Command Type</td>

<td> 0000</td><td> Interrupt acknowledge</td>

<td> 0001</td><td> Special cycle</td>

<td> 0010</td><td> I/O read</td>

WO 01/18989

<td> 0011</td><td> I/O write</td>

<td> 0100</td><td> Reserved</td>

<td> 0101</td><td> Reserved</td>

<td> 0110</td><td> Memory read</td>

<td> 0111</td><td> Memory write</td>

<td> 1000</td><td> Reserved</td>

<td> 1001</td><td> Reserved</td>

<td> 1010</td><td> Configuration read</td>

<td> 1011</td><td> Configuration write</td>

<td> 1100</td><td> Memory read multiple (multiple cache lines)</td>

<td> 1101</td><td> Dual address cycle</td>

<td> 1110</td><td> Memory read line (single cache line)</td>

<td> 1111</td><td> Memory write and invalidate</td>

As data path-widths grow, and clock speeds become faster, the parallel bus is becoming too costly and complex to keep up with system demands. In response, the computer industry is moving toward fast, packetized, serial input/output (I/O) bus architectures, in which computing 5 hosts and peripheral are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, including Next Generation I/O (NGIO) and Future I/O (FIO), culminating in the InfiniBand״ architecture, which has been advanced by a consortium led by a group of industry leaders (including Intel, Sun, Hewlett Packard, IBM, Compaq, Dell and Microsoft). Storage Area Networks (SAN) provide a 10 similar, packetized, serial approach to high-speed storage access, which can also be implemented using an InfiniBand fabric.

SUMMARY OF THE INVENTION

It is an object of some aspects of the present invention to provide devices and methods for bridging between a parallel bus and a packet-switched network or fabric.

WO 01/18989

It is a further object of some aspects of the present invention to provide methods and devices that map PCI bus commands to serial data packets and vice versa.

In preferred embodiments of the present invention, a bridge device links a parallel bus to a packet-switched network. Preferably, the parallel bus comprises a PCI bus, and the packet-switched network comprises an InfiniBand (IB) network, but the principles of the present invention are similarly applicable to buses and networks of other types. The bridge device receives bus commands that are issued by a master device to a specified address on the bus, and translates them into packets for transmission over the network. The packets are sent to a network address, typically a virtual address on the network, that is assigned to conespond to the specified bus address. Similarly, the bridge device receives packets from the network that are destined for devices on the bus, and translates them into appropriate bus responses or commands, to be conveyed to the proper address on the bus. The bridge device thus enables master devices on the bus to communicate transparently with destination devices on the network, as though the destination devices were actually present on the bus. It similarly enables devices on the network to communicate transparently with target devices on the bus.

In some preferred embodiments of the present invention, in order to allow master devices on the bus to communicate over the network, one or more segments are allocated to the bridge device in the address space of the bus. For the PCI bus, the segments are preferably allocated in each of the Memory, I/O and Configuration address spaces. Each segment is divided into one or more address ranges, and each range is associated with a unique channel on the network. Preferably, a channel lookup table held by the bridge device lists attributes of each of the channels. When the master device on the PCI bus sends a bus command to an address in one of the segments that are allocated to the bridge device, the bridge device looks up the address in the table in order to determine the channel to use and the attributes of the channel. These attributes also including a mapping between bus addresses and network addresses to which the packet is to be sent. The bridge device uses the attributes in the table to construct and transmit a packet of the appropriate type to the proper network address over the assigned channel. This flexible allocation of address segments and ranges, and the use of the lookup table to associate channel properties with address ranges on the PCI bus, make the bridge device particularly convenient to configure and use.

WO 01/18989

When the bridge device receives packets over the network destined for devices on the

PCI bus, a similar arrangement is preferably used to map the channels and network addresses to the corresponding bus addresses of the destination devices.

There is therefore provided, in accordance with a preferred embodiment of the present invention, a method for bridging between a parallel bus and a packet-switched network, including:

allocating at least one segment in an address space of the parallel bus for communications between devices on the bus and the packet-switched network;

defining a correspondence between bus addresses within the at least one segment and respective network addresses on the network;

receiving a bus command directed to a specified one of the bus addresses within the at least one segment; and transmitting a packet over the network to one of the network addresses responsive to the correspondence.

Preferably, allocating the at least one segment includes allocating a plurality of segments, wherein the bus includes a Peripheral Component Interface (PCI) bus, and the address space includes Memory, Input/Output (I/O) and Configuration address spaces, and wherein allocating the plurality of segments includes allocating the segments in at least two of the addresses spaces. Most preferably, the packet-switched network includes an InfiniBand switching fabric.

Preferably, defining the correspondence includes assigning a channel in the network to correspond to a given range of the bus addresses within the at least one segment, the given range including the specified bus address, and transmitting the packet includes sending the packet over the assigned channel. Optionally, the given range includes substantially all of the at least one segment. Alternatively, assigning the channel includes dividing the segment into a plurality of non-overlapping ranges, one of which is the given range.

Further preferably, assigning the channel includes recording attributes of the channel in a lookup table, and transmitting the packet includes looking up the attributes in the table responsive to the bus command so as to transmit the packet in accordance with the attributes. Preferably, looking up the attributes includes determining an index to the lookup table based on the bus address specified by the bus command. Alternatively or additionally, the recorded

WO 01/18989 attributes of the channel include a status of the channel, and transmitting the packet includes deciding, responsive to the status, whether to transmit the packet or to return to a device on the bus that issued the bus command an indication that the packet could not be sent.

Preferably, transmitting the packet includes transmitting a request packet to a destination device on the network, and the method includes receiving a response packet over the network from the destination device, and generating a response on the parallel bus based on the response packet. Preferably, receiving the bus command includes receiving the command from a master device on the bus, and generating the response includes returning the response to the master device, such that the command and response are substantially indistinguishable to the master device from commands sent to and responses received from other devices on the bus with which the master device is in communication.

In a preferred embodiment, receiving the bus command includes receiving a read command for execution by a destination device at the one of the network addresses, and transmitting the packet includes sending a network read request packet to the destination device. Preferably, receiving the read command includes receiving a command to read a given quantity of data from the destination device, and sending the network read request packet includes setting a data length field in the read request packet responsive to the given quantity. Optionally, receiving the command to read the given quantity includes receiving a command to read one or more cache lines.

In another preferred embodiment, receiving the bus command includes receiving a write command for execution by a destination device at the one of the network addresses, and transmitting the packet includes sending a network write request packet to the destination device.

Preferably, defining the correspondence includes defining a relation between the one or more bus addresses and one or more virtual addresses on the network.

There is also provided, in accordance with a preferred embodiment of the present invention, a method for bridging between a packet-switched network and a parallel bus, including:

allocating at least one channel on the packet-switched network for communications with devices on the parallel bus;

WO 01/18989 PCT/IL00/00543 defining a correspondence, for use in the communications over the at least one channel, between one or more network addresses on the network that are accessible via the at least one channel and one or more respective bus addresses in an address space of the parallel bus;

receiving a packet at one of the network addresses over the at least one channel; and conveying a bus command over the bus to one of the bus addresses responsive to the correspondence.

Preferably, the at least one channel includes a plurality of channels. In a preferred embodiment, the bus includes a Peripheral Component Interface (PCI) bus, and the network includes an InfiniBand switch fabric, wherein allocating the at least one channel includes associating a plurality of channels with respective InfiniBand Work Queue Pairs (WQPs).

Preferably, allocating the at least one channel includes recording attributes associated with the channel in a lookup table, and conveying the bus command includes looking up the attributes in the table responsive to a characteristic of the packet so as to convey the bus command in accordance with the recorded attributes. Further preferably, the characteristic of the packet includes a channel number assigned to the channel in the network over which the packet is received, and looking up the attributes includes determining an index to the lookup table based on the identification number and finding the attributes in the table indicated by the index.

In a preferred embodiment, determining the index includes using as the index a first portion of the identification number, and looking up the attributes includes matching a second portion of the identification number to a tag stored in the table in order to verify that the attributes indicated by the index pertain to the channel over which the packet is received. Preferably, the method includes refreshing the attributes recorded in the table with respect to the channel over which the packet is received when the second portion of the identification number does not match the tag.

Preferably, conveying the bus command includes determining the bus address to which to convey the command by transforming the network address at which the packet is received, based on the attributes recorded in the table. Most preferably, defining the correspondence includes defining a correspondence between one or more virtual addresses on the network and the one or more respective bus addresses.

WO 01/18989

Further preferably, conveying the bus command includes conveying the command to a destination device on the bus at the one of the bus addresses, and the method includes receiving a response over the bus from the destination device, and transmitting a response packet over the at least one channel of the network responsive to the response. In a preferred embodiment, receiving the packet includes receiving a read request packet for execution by the destination device, and conveying the bus command includes generating a read cycle on the bus addressed to the destination device. Preferably, receiving the read request packet includes receiving a packet specifying a length of data to read from the destination device, and generating the read cycle includes selecting one of a plurality of types of read cycles to generate, responsive to the specified data length. In another preferred embodiment, receiving the packet includes receiving a write rpquest packet for execution by the destination device, and conveying the bus command includes generating a write cycle on the bus addressed to the destination device.

There is additionally provided, in accordance with a preferred embodiment of the present invention, a bridge device for bridging between a parallel bus and a packet-switched network, including:

a target channel adapter (TCA), adapted to receive a bus command over the parallel bus directed to a specified one of a plurality of bus addresses within a segment in an address space of the parallel bus, which segment is allocated to the TCA for communications between devices on the bus and the packet-switched network, the TCA being further adapted to generate, responsive to the bus command, a data packet for transmission over the packet-switched network to a network address corresponding to the specified bus address in accordance with a predefined address correspondence; and a network interface adapter, configured to transmit the data packet over the packet-switched network to the network address.

In a preferred embodiment, the network interface adapter includes a switch.

There is further provided, in accordance with a preferred embodiment of the present invention, a device for bridging between a packet-switched network and a parallel bus, including:

a network interface adapter, configured to receive a data packet sent to a specified network address over a channel on the packet-switched network, which channel is allocated on the network for communications with devices on the parallel bus; and

WO 01/18989 a target channel adapter (TCA), which is adapted to select a bus address in an address space of the parallel bus corresponding to the specified network address in accordance with an address correspondence, defined for use in the communications over the at least one channel, between network addresses that are accessible via the channel and respective bus addresses in the address space of the parallel bus, and to convey a bus command to the selected bus address responsive to the received data packet.

The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a block diagram that schematically illustrates a system for bridging between a parallel bus and a packet-switched network, in accordance with a preferred embodiment of the present invention;

Fig. 2 is a block diagram that schematically illustrates details of a bridging device in the system of Fig. 1, in accordance with a prefened embodiment of the present invention;

Fig. 3 is a flow chart that schematically illustrates a method for converting a bus command to a packet for transmission over a packet-switched network, in accordance with a prefened embodiment of the present invention;

Fig. 4 is a block diagram that schematically illustrates construction of a channel index for use in the method of Fig. 3, in accordance with a preferred embodiment of the present invention;

Figs. 5 and 6 are block diagrams that schematically illustrate schemes for segmenting memory space on a parallel bus, in accordance with a preferred embodiment of the present invention; and

Fig. 7 is a flow chart that schematically illustrates a method for converting a packet received over a packet-switched network to a bus command, in accordance with a preferred embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Fig. 1 is a block diagram that schematically illustrates a system 20 for bridging between a PCI bus 22 and an InfiniBand (IB) packet-switched network 30, in accordance with a preferred embodiment of the present invention. In this example, a master device on bus 22, such as a central processing unit (CPU) 24, wishes to read from or write to a destination

WO 01/18989 device, such as an I/O device 38, over network 30. Alternatively, a second bridge device 32 may be used to communicate with a peripheral device 36 on bus 34. The use of bridge devices 28 and 32 in cooperation constitutes a PCI-to-PCI bridge and is described in greater detail in the above-mentioned patent application entitled Bridge between Parallel Buses over a Packet-Switched Network.״ While preferred embodiments are described hereinbelow with reference to the PCI bus and the IB network, it will be understood that the principles of the present invention may be applied, mutatis mutandis, to bridge between parallel buses and networks of other types and obeying different standards.

In a typical scenario, CPU 24 issues an appropriate PCI read or write command to an address in the address space of PCI bus 22 that is assigned to bridge device 28. The remainder of the PCI bus address space is typically assigned to peripheral and I/O devices 26 on bus 22. Bridge device 28 generates an IB packet based on the CPU request and sends it over network 30 to device 38. The I/O device returns the appropriate response (typically data, in the case of a read command, or an acknowledgment of a write command) back to bridge device 28. This bridge device generates a corresponding PCI bus cycle, in which it passes the response over bus 22 to the CPU.

In accordance with the PCI standard, each device on PCI bus 22 can have multiple address segments, assigned to it in each of the Memory, Configuration and I/O address spaces on the bus 22. Preferably, bridge device 28 is designed so that thirty-two such segments can be assigned to the device, although alternatively, a larger or smaller number of segments may be available. Further preferably, the segments are divided into ranges, most preferably up to 128 ranges in all for the Memory and I/O segments, and another eight ranges for the Configuration segments, (The configuration ranges are preferably set aside for communication with the Configuration address space of other, remote PCI buses, such as bus 34.)

Each range is associated with a channel over network 30, as described in detail hereinbelow. Upon start-up or reconfiguration of system 20, CPU 24 is informed of a PCI address for I/O device 38 that is within one of the address segments assigned to bridge device 28. The CPU addresses all read and write commands destined for device 38 to this address. Preferably, among the segments assigned to bridge device 28, there are also ranges associated with channels over network 30 to bridge device 32, for addressing peripheral device 36 on bus 34, as well as to additional network devices and PCI buses not shown in the figure.

WO 01/18989

Each of the address ranges assigned to bridge device 28 is associated with a different channel, or Work Queue Pair (WQP) in IB network 30. Each channel in the IB network corresponds to a fixed path and level of service through the network, and packet ordering is preferably maintained among all packets carried by a given channel. Similar methods of maintaining packet order are known in other types of networks known in the art. As a consequence of this channel structure, all read and write requests directed to device 38 are received by the device in the exact order in which they were sent by bridge device 28. Responses of device 38 are similarly sent to back to bridge device 28 and are returned to CPU 24 in the proper order. This ordering is used in maintaining PCI bus conventions, as described in the above-mentioned patent application entitled Bridge between Parallel Buses over a Packet-Switched Network.

Fig. 2 is a block diagram that schematically illustrates details of bridging device 28, in accordance with a prefened embodiment of the present invention. Device 28 comprises a target channel adapter (TCA) 50, which is coupled to PCI bus 22, and a network interface, such as a switch 52, coupled to network 30. Preferred switches for this purpose are described in the above-mentioned provisional patent applications nos. 60/152,849 and 60/175,339, for example. The design of switch 52 is beyond the scope of the present patent application, however, and substantially any packet switch capable of interfacing with TCA 50 will be suitable for carrying out the function of switch 52.

TCA 50 is responsible for converting PCI commands (or cycles) into IB packets, and vice versa. The TCA comprises a PCI target unit (PCIT) 54 and a PCI master unit (PCIM) 56. PCIT 54 responds to transactions originating on PCI bus 22, which are initiated by a bus master such as CPU 24. In particular, the PCIT receives read and write commands from the PCI bus and generates the appropriate IB read and write request packets for transmission via switch 52. When response packets are received from the destination device or bus, PCIT 54 receives these packets and generates the appropriate response cycles on PCI bus 22 (typically returning data in response to a read request or an acknowledgment in response to a write request). PCIM 56, on the other hand, is responsible for receiving IB read and write request packets from network 30, sent by IB device 38 or bridge device 32, for example. It converts these requests into the appropriate read and write commands on bus 22, which in this case is

WO 01/18989 the destination bus. The PCIM then packetizes the resulting response to send back to the originating bus.

Fig. 3 is a flow chart that schematically illustrates a method by which PCIT 54 processes a PCI bus cycle and generates a corresponding IB packet, in accordance with a preferred embodiment of the present invention. As noted above, the PCIT is preferably capable of capturing PCI bus cycles with addresses in any of thirty-two different segments. Each segment is delimited by a lower bar address and an upper limit address, which are preferably programmable for each segment individually. The bar and limit for each segment are listed in a segment header, along with an identification of the PCI address space (Memory, I/O or Configuration) to which the segment belongs. The bar and limit are most preferably set on 1 MB boundaries in the PCI address space, so that the 20 least significant bits (LSB) of the bar are set to 00000, and the 20 LSB of the limit are set to FFFFF.

At a cycle reception step 70, PCIT 54 receives a cycle on the PCI bus and examines its C/BE code and destination address. At an address checking step 72, the address is compared to the bars and limits in the PCIT segment headers in order to determine whether it falls within one of the configured segments. Setting the bar and limit on 1 MB boundaries is useful in this regard, as it allows rapid comparison of the address to the bars and limits without using a great deal of logic or computing power. Cycles having addresses outside all of the PCIT segments are ignored, at a cycle discard step 74.

When the cycle address is found to be within one of the segments, PCIT 54 proceeds to assemble the information needed to prepare a packet for transmission, beginning with a segment parameter reading step 76. The PCIT must first determine the IB channel that has been assigned to carry packets associated with the current cycle address, and it must then obtain relevant channel parameters for preparing and sending the packet. As noted above, each such channel corresponds to a range of the addresses within the segment, which may comprise the entire segment. The channel parameters are held in a target channel lookup table (CLT) 58 (Fig. 2). These parameters preferably include, but are not limited to, the following attributes:

• IB Work Queue Pair (WQP) context, providing connection information for the channel such as the WQP number and the service type and level, destination ID.

WO 01/18989

PCI7IL00/00543 • IB channel properties, such as the channel status - whether the channel is enabled or disabled, packet serial number (PSN), and the number of delayed read or write request packets that the channel can handle.

• IB packet properties, for example -

- Whether PCI Write cycles are to be conveyed on this channel as IB Remote Direct Memory Access (RDMA) packets or as Send packets. (RDMA packets specify an address in the memory of the destination device to which data are to be written, while Send packets do not.)

- What data length to specify in RDMA Read packets, depending on the cache line size of the master device on the PCI bus and the type of PCI Read cycle.

- Latency parameters, i.e., how much data to accumulate at IB network device 38 before sending a read response packet back to bridge device 28.

- Address parameters, for mapping the PCI bus address to a corresponding IB virtual address.

Storing these parameters in CLT 58 allows them to be accessed quickly and easily when a PCI bus cycle is addressed to one of the configured segments of the PCIT, and also allows simple reprogramming of channel attributes as needed.

The segment headers held by the PCIT preferably include, for each segment, a channel base index (CBI) and offset and mask parameters, which are read at step 76. The CBI and the offset and mask parameters are used in calculating a channel index (CI) at an index computation step 78, based on the PCI address range in which the bus command was received at step 70. The CI identifies the channel over which the packet is to be sent in response to the PCI bus command. It also serves as the index to CLT 58, at a lookup step 80, in order to determine the necessary attributes of the channel. Further details regarding the assignment of channels to PCI address ranges and of the computation of the CI are described hereinbelow.

The attributes retrieved from CLT 58 are used to build and send the corresponding packet, at a packet sending step 82. If the channel is unavailable, because the channel properties indicate that it is disabled or that its queue of delayed requests is full, PCIT 54 will not send the packet, but will rather return a retry or failure indication on PCI bus 22. Assuming the channel to be enabled, however, various types of packets may be sent through the IB network switching fabric, invoking different sorts of responses at the packet destination.

WO 01/18989

PCT/ILO0/OO543

According to IB convention, the packet type is specified by an Extended Transport Header (ETH) associated with the transport layer of the IB protocol stack. As noted above, read and write requests to a specific virtual address on the IB network are invoked by a Remote Direct Memory Access (RDMA) ETH, while write requests without a specific virtual address are 5 carried by Send packets. The virtual address is preferably calculated by PCIT 54 by applying mask and address parameters, stored in CLT 58, to the PCI address of the corresponding read or write bus command. The first message of a RDMA read or write request contains a RDMA ETH specifying a virtual address for the operation, an access key (R_key) and a total length of the data buffer to read or write. The RDMA write request includes a payload containing the 10 data to be written (which may be continued in subsequent RDMA write packets).

Table Π below lists all of the PCI bus commands that may be received by PCIT 54 on bus 22, along with the coresponding IB packet types that the PCIT will send over network 30 at step 82, in accordance with a preferred embodiment of the present invention:

TABLE Π - CONVERSION OF PCI CYCLES TO IB PACKETS

<td> C/BE</td><td> Command</td><td> IB packet</td>

<td> 0000</td><td> INTA</td><td> None</td>

<td> 0001</td><td> Special</td><td> None</td>

<td> 0010</td><td> I/O read</td><td> RDMA read (length &lt;32 bits)</td>

<td> 0011</td><td> I/O write</td><td> RDMA write or send (length &lt;32 bits)</td>

<td> 0110</td><td> Memory read</td><td> RDMA read (length according to prefetch length in CLT 58, up to page boundary)</td>

<td> 0111</td><td> Memory write</td><td> RDMA write or send (length depending on data length in PCI cycle)</td>

<td> 1010</td><td> Configuration read</td><td> RDMA read (length &lt; 32 bits depending on PCI BE)</td>

<td> 1011</td><td> Configuration write</td><td> RDMA write (length &lt; 32 bits depending on PCI BE)</td>

<td> 1100</td><td> Memory read multiple</td><td> RDMA read (length according to multiple prefetch length in CLT 58, up to page boundary)</td>

WO 01/18989

PC17ILO0/0O543

<td> 1101</td><td> Dual address</td><td> None</td>

<td> 1110</td><td> Memory read line</td><td> RDMA read (length up to next aligned cache line, specified in CLT 58, or to page boundary)</td>

<td> 1111</td><td> Memory write/ invalidate</td><td> RDMA write or send</td>

When an entity such as IB device 38 receives the write request, it responds with an acknowledge packet, having an Acknowledge ΕΤΗ (ΑΕΊΉ). The AETH is also present in read response packets returned by device 38 when it receives the read request. Such read response packets also have a payload containing the requested data. Those skilled in the art will recognize that packets types and operations similar to these are also encountered in packet-switched networks of other sorts known in the art. PCIT 54 receives the response packet from device 38 and generates a corresponding PCI bus cycle on bus 22. Further details of the process by which PCIT 54 services read and write transactions initiated on bus 22 are described in the above-mentioned patent application entitled Bridge between Parallel Buses over a Packet-Switched Network.

Fig. 4 is a block diagram that schematically illustrates details of index computation step 78, in accordance with a preferred embodiment of the present invention. A channel index (CI) 94 is constructed by concatenating a specified number of mask bits 96 taken from a PCI address 90 at which the command was received, with a number of the most significant bits (MSB) 98 of channel base index (CBI) 92. A configuration bit 100 is preferably added to CI 94 to indicate whether address 90 belongs to one of the 128 ranges assigned to bridge device 28 in the memory and I/O address spaces of the PCI bus to the configuration address space, or to one of the eight special ranges assigned to the configuration space. The mask parameter in the segment header specifies the number of mask bits 96 to select from the PCI address (two bits in the example of Fig. 4), at an offset from the least significant bit (LSB) of the PCI address that is given by the offset parameter. Bits 98 are taken from CBI 92 to give a total CI length of 8 bits. The CBI bits are concatenated between mask bits 96 and configuration bit 100 to obtain the 8-bit CI 94.

WO 01/18989

PCT/ILO0/OO543

Figs. 5 and 6 are block diagrams that schematically illustrate the assignment of ranges in PCI Memory spaces 110 and 120 to IB channels, using the method of Fig. 4. In Fig. 5, Memory space 110 includes segments 3 and 4. Segment 3 has a bar and limit on 4 MB boundaries and is divided into four ranges, each 1 MB wide. Given an offset parameter of 20, 5 the two mask bits 96 cycle in order through the segment from 0 to 3 (binary 00 to 11). With CBI 92 set to 8 (binary 100), CBI bits 98 in CI 94 will be 0001. Thus, the four ranges of segment 3 receive channel indices of 8, 9, A and B, respectively, Similarly, segment 4 has its bar aligned on a 16 MB boundary, with an offset of 22. The single mask bit 96 in this case sequentially takes the values 0 and 1, giving two 4MB ranges. With CBI 92 set in this case to 10 C, the two ranges have channel indices of C and D.

In Fig. 6, Memory space 120 includes segments 0, 1 and 2, which in this case are not aligned on 4 MB boundaries. Thus, although segment 2 is divided into four ranges, having the same mask and offset values as segment 3 (Fig. 5), mask bits 96 of segment 2 sequence as 2, 3, 0, 1. With CBI 92 set to 4, the 1 MB ranges in segment 2 receive channel indices 6, 7, 4 and 5, 15 in that order. Segment 0 is an example of non-uniform division of a segment into ranges. The two mask bits 96 for this segment are equal to 00 in the lowest 1 MB range, 01 over the next 2 MB, and 10 in the upper 1 MB, giving channel indices of 0, 1 and 2, respectively. Finally, by setting the offset and mask parameters to zero in segment 1, the entire 5 M־r segment receives the same channel index, given by its CBI, which is set to 3.

²⁰ F’g- ⁷ is a flow chart that schematically illustrates a method by which PCIM 56 (Fig. 2) processes an IB packet and generates a corresponding PCI bus cycle, in accordance with a preferred embodiment of the present invention. In servicing IB packets, the PCIM refers to a master channel lookup table (CLT) 60, which, like target CLT 58, provides the channel parameters pertinent to generating the PCI cycle. The parameters include, but are not limited 25 to:

• IB Work Queue Pair (WQP) context, providing connection information for the channel as in target CLT 58.

• IB channel properties, also as in target CLT 58, and including the message serial number (MSN) to be used in preparing the response packets to IB request packets that

PCIM 56 receives.

• PCI cycle properties, for example -

WO 01/18989

- Cycle type - whether this channel corresponds to PCI Memory, I/O or Configuration cycles.

- Addressing attributes, including address map and mask, for converting IB virtual addresses to PCI bus addresses.

PCIM 56 receives an IB packet, at a packet reception step 130, and attempts to look up the corresponding channel parameters in CLT 60. For this purpose, the PCIM preferably extracts the seven LSB of the WQP number for use as an index to the CLT, at an indexing step 132. It uses the index to look up the indicated CLT entry, at a lookup step 134. Preferably, master CLT 60 includes 128 entries, for up to 128 active channels, and the master CLT serves as a sort of data cache for parameters regarding these channels. Parameters of additional channels over which PCIM 56 can receive packets, beyond the active 128, are preferably stored in another memory. In order to determine whether the record in CLT 60 indicated by the channel index at step 134 is indeed the correct one for this channel, the PCIM checks the upper seventeen bits of the WQP number against a Master WQP tag stored in the CLT for this channel entry, at a tag matching step 136. If the tag matches, it means that the cache holds the correct data for this channel. Otherwise, the data in this entry of CLT 60 are preferably refreshed from memory, at a cache refresh step 138. Alternatively, it may be that this channel has not been configured in PCIM 56, in which case the packet is discarded, and an error is reported.

Having found the appropriate entry in the CLT, PCIM 56 reads the PCI attributes that it will need in order to construct a corresponding PCI cycle, at an attribute reading step 140. Based on the attributes in the CLT, the PCIM validates the packet, checking, for example, that the packet has arrived in the proper order and carries appropriate protection parameters (P_Key and R_Key in IB network convention). In other words, the PCIM implements transport layer functions of network 30. The PCIM then calculates the address on PCI bus 22 to which the cycle is to be addressed, at an addressing step 142. For RDMA read and write packets, the address map and mask parameters in the CLT are used to determine the PCI address based on the virtual address of the IB packet that was received. For IB send packets, PCIM 56 will generate write cycles to a PCI bus address specified in the CLT. The address may be incremented from one packet to the next.

WO 01/18989

Finally, PCIM 56 generates the appropriate PCI cycle for the packet that it has received, at a cycle generation step 144. Table ΠΙ below lists the types of PCI cycles that are generated in response to incoming IB packets, in accordance with a preferred embodiment of the present invention. Further details of the process by which PCIM 56 services read and write 5 requests received over IB network 30 are described in the above-mentioned patent application entitled Bridge between Parallel Buses over a Packet-Switched Network.

TABLE ΠΙ - CONVERSION OF IB PACKETS TO PCI CYCLES

<td> Master channel type</td><td> IB packet attributes</td><td> PCI cycle (C/BE)</td>

<td rowspan="2"> I/O</td><td> RDMA read</td><td> 0010 - I/O read</td>

<td> RDMA write/send</td><td> 0011 - I/O write</td>

<td rowspan="2"> Config- uration</td><td> RDMA read</td><td> 1010 - Configuration Read</td>

<td> RDMA write/send</td><td> 1011 - Configuration Write</td>

<td rowspan="5"> Memory</td><td> RDMA read, length &lt; 4 bytes</td><td> 0110 - Memory Read</td>

<td> RDMA read, length &gt; 4 bytes and up to cache line</td><td> 1110 - Memory Read Line</td>

<td> RDMA read, length &gt; 4 bytes and crosses cache line boundary</td><td> 1100 - Memory Read Multiple</td>

<td> RDMA write/send with Memory Write and Invalidate bit enabled - aligned to cache lines</td><td> 1111 - Memory Write and Invalidate</td>

<td> Other RDMA write/send</td><td> 0111 - Memory Write</td>

It will thus be observed that system 20 (Fig. 1) meets all of the requirements of both

PCI bus 22 and IB network 30. Bridge device 28 enables devices on the PCI bus, such as CPU

WO 01/18989

24, to cany out transactions transparently with IB device 38 and with PCI peripheral device 36 on PCI bus 34 as though these remote devices were actually located on bus 22. The bridge device similarly enables IB device 38 to interact transparently with PCI bus devices. It will be apparent to those skilled in the art that the principles of system 20 and of the methods described 5 hereinabove may be applied in a straightforward manner to buses, networks and transactions of other types. It will therefore be appreciated that the preferred embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, 10 as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.