Fault tolerant modular computing system

0001The present invention relates to computing systems formed of one or more arrays of interconnected hardware modules. More particularly, the present invention relates to a digital computer system in which its capacity and fault tolerance are determined by the number and arrangement of modular hardware elements installed in and thereby connected in one or more receptacle arrays. Robotic installation and removal of modules in the environment of the arrays is an aspect of the present invention.

0002The assignee of the present invention, has pioneered the concept of fault tolerance in connection with computing systems. Basically, that concept holds that failure of one element or subsystem of a computing system should never be permitted to cause the whole system to fail. In practice, the use of parallel processors, parallel data paths to peripheral equipment and parallel power supplies has resulted in a fault tolerant computing system which has significantly advanced the art. This advance is document, for example, in U.S. Patent No. 4,228,496, the disclosure of which is incorporated herein by reference.

0003In practice, discrete hardware elements of a computer, such as processors, peripheral device controllers, central memory, data mass storage devices, and power supplies are formed as separate hardware elements, and these elements are typically interconnected by cabling, and sometimes by direct plug in connection to back planes or mother boards.

0004One commonly encountered hardware element is the rotating data storage device, sometimes called "disk drive". Disk drives, whether based on magnetic or optical storage technologies, are frequently used in computing systems to provide rapid, random access to stored data. Over the years, disk drive technology has progressed to a point where a massive amount of data may now be stored in a disk drive device using small diameter non-removable disks (e.g. eight inch) and having a very small physical package.

0005Heretofore, the servicing of existing digital computer systems, including systems practicing fault tolerance, has involved the disconnection of cables and harnesses interconnecting the various subsystems such as disk drives. Such servicing has typically included the removal and reinstallation of cabinetry and mounting hardware. Because of the way in which prior computering systems have been designed, constructed and assembled, removal of a circuit board or a cable during servicing has either required that all primary power to the subsystem be removed or has led to failure of backup or mirror subsystems, causing a single point failure within the computing system.

0006Also, the manner in which computing systems have been assembled has not been conducive of automatic maintenance procedures. While industrial robots are known and widely employed in industrial manufacturing processes, little if any effort has been given to the adaptation of computing systems and programmed robotic equipment to facilitate automatic maintenance procedures.

0007Thus, a hitherto unsolved need has arisen for a fault tolerant digital computer incorporating subsystems and hardware elements thereof which may be serviced, either manually or automatically by robotic maintenance equipment, without disturbance of the cabinet and cabling housing and connecting the disk drive to be serviced.

0008A system is described in JP-A-58-222354 in which multiple packages are contained in a frame. Software operably disconnects a failed package, replacing it with a spare system. The mounting location of the package is determined, and a lateral position guide, riding on a lateral guide rail, is moved to a lateral position of the failed package while a detachable guide moves along a longitudinal guide rail to the longitudinal position, beneath the failed package, thereby identifying the package location, permitting it to be pulled out by a removable handle.

0009A general object of the present invention is to provide a fault tolerant computing system formed of multiple subsystems of uniform hardware modules in one or more interconnecting arrays which overcomes the limitations and drawbacks of the prior art.

0010In accordance with the present invention, there is provided a fault tolerant computing facility comprising at least one housing array containing plural pairs of modules, each containing a hardware element of the computing system, and a plurality of power supply modules, there being one power supply module for each hardware element, each of the pair of modules comprising a primary module and a back-up mirror module, characterized by:    at least a pair of power distribution modules, one power distribution module operating to distribute primary power from a first primary source connection to the primary modules and associated power supply modules, the other power distribution module operating to distribute secondary power from a second primary source connection to the back-up modules and associated power supplies; and    interconnect means for automatic electrical disconnection of removed modules and reconnection of replacement modules    A feature of the system is to enable the provision of an array for multiple hardware elements such as disk drives which facilitates configurations of a single string of disk drives or two separate strings of disk drives without installation of any additional internal cabling within the array.

0011A further feature of a system in accordance with the present invention is to enable the provision of improved packaging for hardware elements of a fault tolerant computing system such as a data storage subsystem formed of one or more strings of multiple disk drives which achieves a significant improvement in serviceability and overall system reliability, and which makes the subsystem less sensitive to facility power faults.

0012Yet another feature of a system in accordance with the present invention is to enable the provision of a housing for mating with hardware attachment carrier modules of a computing system in a manner which enables the module to be accepted into the housing and thereupon automatically connected with power, control and data signals, control panel functions and provided with cooling without the use of tools and without need for access to the cabling installed within the cabinet.

0013A further feature of a system in accordance with the present invention is to enable the provision of a housing for receiving multiple uniformly sized hardware element modules and a mating attachment carrier module for each hardware element, such as a disk drive, in the form of a drawer which is mechanically and electrically blind-mateable with the housing so as not to require special adjustment of alignment in over for the hardware element it is carrying to be placed in service.

0014One more feature of a system in accordance with the present invention is to enable the provision of a housing for multiple hardware element modules which receives a power supply for each module in close proximity to the modules in order to minimize power supply cabling lengths and which enables the supply to be accepted into the housing and thereupon automatically connected to the power distribution cabling of the housing without theuse of tools and without need for access to the cabling installed within the housing.

0015Yet another feature of a system in accordance with the present invention is to enable the provision of a housing, a first primary power distribution module (PDM) powered from one facility primary power source, and a second back-up or mirror image primary PDM powered from another primary power source within the facility, with the first PDM supplying power to first hardware elements such as disk drives, first primary power supplies and one half of the cooling apparatus, and with the second PDM supplying power to second hardware elements such as back-up, mirror image disk drives, back-up power supplies and the other half of the cooling apparatus, so that loss of one facility power buss or circuit breaker will not cause the entire subsystem to fail.

0016A still further feature of a system in accordance with the present invention is to enable the provision of a fault tolerant computing system facility comprising modularized elements blind-mateable with a housing providing all inter- connections so as to enable removal and replacement of each element by one or more programmed robots.

0017Thus, one embodiment of a fault tolerant digital computer subsystem in accordance with the present invention includes a housing which includes provisions for receiving a plurality of modularized hardware elements, such as disk drives for example, including at least one primary module and at least one back up mirror module. A plurality of modularized power supplies are receivable within the housing, there being a separate power supply module electrically associated with each separate hardware module. At least two separate power distribution modules (PDM's) are provided in the housing. One PDM distributes primary power from a first facility primary source connection to the primary modules and power supply modules associated with primary modules, and one half of cooling apparatus in the housing, and a second PDM distributes primary power from a second facility primary source connection to back up mirror modules and power supply modules associated with back up mirror modules and the other half of the cooling apparatus within the housing. Cabling within the housing enables the hardware modules and the power supply modules to be mechanically and electrically blind-mateable by providing automatic disconnect and reconnect when the modules are removed and inserted into the cabinet, without the use of tools, either by hand or by robotic maintenance equipment.

0018An electrical configuration arrangement in the housing which is cooperatively associated with the cabling enables said subsystem to be alternatively configured as a single string of hardware modules and as multiple strings of hardware modules without physical modification of the internal cabling.

0019A control panel is provided for configuring logically each module and for indicating condition of the hardware modules and power supply modules installed in the cabinet.

0020The invention is described further hereinafter, by way of example only, with reference to the accompanying drawings, in which:- <ul id="ul0001" list-style="none"><li>Fig. 1 is an orthogonal view of the front side of one presently preferred embodiment of the invention including a cabinet for receiving multiple disk drive and power supply modules. A front door has been omitted to facilitate illustration of the cabinet structure.</li><li>Fig. 2 is an orthogonal view of the back side of the cabinet depicted in Fig. 1. A rear door has been omitted to facilitate illustration.</li><li>Fig. 3 is an enlarged exploded orthogonal view of a drive and drive carrier module drawer illustrated in Fig. 1.</li><li>Fig. 4 is a diagram of the logical organization of the disk drives and power supply distribution connections within the cabinet depicted in Fig. 1.</li><li>Fig. 5 is a diagram of the external data cable orientation and connections of the cabinet depicted in Fig. 1.</li><li>Fig. 6 is a schematic diagram of external data cable, jumper and termination connection facilitating a single string configuration for the drives contained in the cabinet depicted in Fig. 1.</li><li>Fig. 7 is a schematic diagram of external data cable and termination connections facilitating a two string configuration for the drives contained in the cabinet depicted in Fig. 1, and further illustrating daisy chaining of a plurality of subsystems of the type depicted in Fig. 1.</li><li>Fig. 8 is an orthogonal view of a portion of the front side of one bay of a modularized, fault tolerant computing facility adapted for automatic robotic maintenance apparatus and procedures comprising a second presently preferred embodiment of the present invention.</li><li>Fig. 9 is a diagrammatic top plan view of the overall computing facility which is partially depicted in the orthogonal view of Fig. 8.</li><li>Fig. 10 is a diagrammatic front side view in elevation and partial section of the computing facility depicted in Figs. 8 and 9.</li></ul>

Disk Drive Data Storage Subsystem 10

0021A first preferred embodiment of the present invention is illustrated as a fault tolerant multiple disk drive data storage subsystem 10, depicted in Figs. 1 to 7. As seen in Fig. 1, the subsystem 10 is illustrated as being contained in a small cabinet 12 which requires, for example, about six square feet of floor space, and stands about three feet in height. The cabinet 12 includes a top 14, a control panel 16 and sidewalls 18. The beveled, transparent control panel 16 is secured to the top 14 at the front of the cabinet 12. The control panel provides for access to configuration controls and acts as a window for light emitting signalling devices 17 which indicate the status of the disk drives installed within the cabinet 12. The logical identities of the disk drives installed within the cabinet 12 are established by programmable identification plugs which may be inserted into and removed from the control panel 16.

0022Inside the cabinet, shelves 20 and vertical dividers 22 are provided to define spaces for eight disk drive carrier module drawers 24. Each drawer 24 contains a disk drive 26, such as a fixed rotating multiple eight-inch disk magnetic media data storage device having a data storage capacity of 168 megabytes, unformatted, for example. While magnetic media storage disks are preferred, it is to be understood that the cabinet and drawer arrangement may be used effectively with a wide variety of hardware elements of a fault tolerant computing system. With eight drives 26 installed in the cabinet 12, the subsystem provides an overall data storage capability of 1.344 gigabytes of unformatted data. A shelf 28 at the bottom of the cabinet 12, and vertical dividers 30, define spaces for eight plug-in disk drive power supply modules 32. Thus, there is a separate power supply module 32 provided for each disk drive module and each power supply module 32 is located in close proximity with its associated disk drive module drawer 24.

0023A plurality of fans, not shown, are provided on a front door of the cabinet 12, also not shown, to provide a portion of the cooling required to maintain the interior of the cabinet within proper ambient operating range.

0024Turning to Fig. 2, each disk drive shelf 20 is provided with an electrical connection backplane 34. The backplane structure 34 includes a connector plug 36 for mating with a complementary connector jack 38 located on the back of the disk drive carrier drawer, as seen in Fig. 3. All power, control and data connections for the disk drive 26 pass through the plug 36 and jack 38, as a single location of connection. The backplane structure 34 includes a horizontal metal support plate 35 and a printed circuit connection board 37 having connection traces aligned along a rearward edge thereof. Data cabling, not shown, is provided to connect each connection board 37 with an associated drive jack on a data cable connection plane 44 located at the rear of the cabinet 12 and below the shelves 20, as shown in Fig. 2. Once the data cabling is connected to each connection board, electrical/logical configuration of the disk drives 26 in the cabinet 12 may be made by connections at the connection plane 44.

0025A primary facility power distribution module 40 is connected to a first source of facility primary power. The module 40 distributes primary power to one half of the power supply modules: for example, modules 32A, 32C, 32E and 32G, which are associated with primary drives 26A, 26C, 26E, and 26G, as shown (sic: in) Fig. 4. A mirror facility power distribution module 42 is connected to a second source of facility primary power. The module 42 distributes primary power to the other half of the power supply modules: modules 32B, 32D, 32F, and 32H, which are associated with mirror drives 26B, 26D, 26F, and 26H.

0026It is to be understood that the subsystem 10 achieves a first level of fault tolerance by providing a backup or mirror drive for each primary drive 26. Thus, the drive 26B "mirrors" and backs up the drive 26A by containing the same useful data therein as is stored in the primary drive. In the event of a failure of one source of facility primary power, rendering one of the drives 26A or 26B inoperative, the other drive, containing a mirror image of the data held in the first drive remains on line and available for use without interruption in computing service.

0027Cables 46 facilitate connection of the subsystem 10 to two parallel processors of the type described in the referenced U.S. Patent No. 4,228,496. Termination plugs 48 contain suitable termination resistances so that the impedances of the cables 46 are properly matched and so that standing waves do not interfere with high speed data and control signal transmissions from the supervisory parallel processors and the disk drives 26.

0028Details of the disk drive carrier drawer module 24 are illustrated in Fig. 3. Therein the drawer 24 is seen to include two opposite and parallel side rails 50 having lower longitudinal reinforcing lips 51. A front panel 52 is joined to the front ends of the opposed side rails 50. The panel defines a rectangular opening 53 which facilitates passage of cooling air currents about the disk drive 26. The panel 52 also includes provision for a handle 54 and for two knurled looking screws 56 which mate with corresponding threaded portions at the front of the cabinet 12 to lock the drawer 24 in position within the cabinet 12.

0029Four inset L-shaped brackets 58 are integrally formed with the side rails 50. These brackets 58 define mounting holes which are aligned with threaded mounting members of four sidemounted shock absorbers 60 of the disk drive 26. The spacing of the side walls 50 and of the brackets 88 is such as to accommodate the mounting dimensions of the disk drive 26. In practice, a disk drive model No. M-2322K available from Fujitsu America, Inc., 3058 Orchard Drive, San Jose, CA 95134-2017, is Presently preferred as the implementing drive 26. Screws 59 are used to mount the disk drive 26 to the drawer 24 at the brackets 58, as diagrammed in Fig. 3.

0030A transverse fan bracket 62 is mounted by screws to the side rails 50 at a location just behind the space provided for the drive 26. The bracket 62 supports a cooling fan 64 which forces cooling air currents to pass over the disk drive 26 for carrying away heat generated from the spindle motor, actuator motor and electronics of the disk drive. The fan 64 is connected to the facility distribution module 40 or 42 which supplies power to the power supply 32 associated with the particular drive. The fan bracket includes an opening 66 for a data and control cable which connects the drive 26 and the jack 38, and which also provides a second handle to facilitate lifting of the drawer during installation and removal. Side openings 68 are also defined by the fan bracket 62, and these openings facilitate other connections to the disk drive 62 and also the circulation of cooling air currents.

0031A rear end plate 70 is formed integrally with the side rails 50, and the plate includes an upper transverse reinforcing lip 72. A machined, precise alignment opening 74 is defined through the end plate 70 on the right side, as shown in Fig. 3, and a second, oval alignment opening 76 is formed through the left side of the end plate 70. Tapered alignment pins extending toward the front of the cabinet are provided on the vertical portion of the backplane structure 34. These pins enter the openings 74 and 76 and automatically register and align the plug 36 and jack 38 prior to engagement thereof as the drawer 24 is slid along the shelf, until the plug and jack mate as the drawer 24 reaches its installation position within the cabinet 12. Then, the screws 56 are rotated manually to secure the drawer 24 in place.

0032It will be understood from Figs. 4-7 that the logical and electrical identification of each disk drive 26 is established by the physical location of the disk drive within the cabinet 12. Thus, as seen from the front, the upper right drive 26A, the first primary drive, is mirrored by the drive 26B just below it. The next primary drive 26C is mirrored by the drive 26D at the lower right. The upper left third primary drive 26E is mirrored by its back up drive 26F, and the fourth primary drive 26G is mirrored by its back up drive 26H at the lower left.

0033The control panel 16 is arranged as shown in Fig. 4 so that an indicator display 17 is associated with each drive 26. Thus, the indicator display 17A displays the status of the drive 26A, display 17B displays the status of the drive 26B, display 17C displays the status of the drive 26C, display 17D displays the status of the drive 26D, display 17E displays the status of the drive 26E, display 17F displays the status of the drive 26F, display 17G displays the status of the drive 26G, and display 17H displays the status of the drive 26H.

0034Each disk drive 26 is provided with two data connectors on the data cable connector plane 44 as shown schematically in Fig. 5. This dual connection arrangement facilitates connection of each drive 26 to two disk drive controllers of the fault tolerant parallel processing system to which the subsystem 10 is connected. If one controller or its cabling fails in service, the other controller remains on line, thereby providing fault tolerance of the overall system.

0035Jumper cables 47 and terminations 48 enable the subsystem 10 to be configured as a single daisy-chained array of storage devices as shown in Fig. 6. The term "daisy-chain" as used herein is synonymous in meaning with party line, broadcast mode or unibus and means that a disk drive controller broadcasts each control message to all of the disk drives of the chain even though the message is intended only for one of them. The particular disk drive 26 for which the message is intended recognizes the message and responds to the controller, also typically via a common communication path. Other connection arrangements, such as private lines between each disk drive 26 and the controller are within the purview of the present invention.

0036Alternative connections, shown in Fig. 7 enable the subsystem 10 to be multiple daisy-chained internally, and in connection to two other like subsystems 10ʹ and 10ʺ.

Modular Fault Tolerant Computing Facility 100

0037A fault tolerant computing facility 100 employing the concept of standardized electrically and mechanically blind-mateable carrier module drawers, such as the drawer 24 depicted and previously discussed in conjunction with the subsystem 10, is illustrated in Figs. 8-10. Therein, the facility is arranged into a group of four spaced apart, generally parallel equipment arrays or bays 102, a portion of one being shown in Fig. 8. The facility 100 provides for a computing system which may be expanded in virtually unlimited fashion, and also provides for and facilitates use of automatic robotic maintenance procedures and equipment. While the drawers 24 described in connection with the subsystem 10 are depicted in Fig. 8, it is to be understood that a hardware element other than a disk drive may be mounted in a particular drawer. For example, one drawer may include a central processor, another may include a peripheral processor, a third may include a device controller, a fourth may include central memory, and so forth.

0038As with the subsystem 10, each bay of the facility 100 provides an array of shelves 120 and vertical dividers 122 which define uniform spaces for the drawers 24. However, there is no requirement that the spaces be uniform, so long as they are accessible from a major side of each array 102. And, there is no constraint on the length of a particular bay 102, and a bay could be extended to any practical length in any dimension, remembering that the practical constraints are direct access to modules and cables and cable lengths between computing elements which limit the speeds at which control signals and data may be passed back and forth.

0039Shelves 128 and vertical dividers 130 are provided for e.g. four power supply modules 32 directly under each column of four spaces for the module drawers 24. Each power supply location is associated with a particular space for a drawer, so as to keep power supply cable lengths as short as practicable.

0040One arrangement of the facility is depicted in Fig. 9 wherein e.g. four parallel, spaced apart bays 102 are closely spaced from a back plane array 104 arranged normal to each bay. With this arrangement, cables 106 from each bay 102 may be plugged into and interconnected within the back plane array 104. The back plane array 104 may include monitoring apparatus and controls which are accessible to an operator at a control panel 116 along an outer wall 108.

0041Each array or bay 102 is mounted on a suitable pedestal 110 so that the spaces for modules and power supplies are elevated a suitable distance from the floor 112. The bays 102 and the back plane array 104 may be enclosed by a top wall 114, side walls 116, and a back wall 118 in order to define a forced air plenum for cooling the facility 100.

0042Cooling fans such as e.g. a pusher fan 140 and a puller fan 142 may be provided to move cooling air currents through the plenum and the bays 102 in order to draw off heat generated by the hardware elements within the module drawers 24 and the power supplies 32. While only two fans are illustrated, it is apparent that any number of cooling elements, whether air or liquid, and whether associated with the entire facility or with each one or a few adjacent drawers of a bay may be employed, it being remembered that one important concept of fault tolerance in computing systems is the avoidance of any single point of failure of the computing facility 100.

0043In accordance with one embodiment of the present invention, an industrial robot 150 is provided for maintenance, installation and/or reconfiguration procedures within the facility 100. The robot 150 may follow a variety of forms and be of any suitable structure. Its purpose is to be able to remove and replace any drawer 24, power supply 32 or cable 106 within the facility, automatically and without human operator intervention. One suitable form of robot 180 grossly resembles a remote controlled (by low powered local radio signals) fork lift and includes a platform 152 carried by a battery powered wheel carriage having wheels 154 which engage tracks 156 extending throughout the facility 100 at locations which give the robot 150 precise access to each space for every drawer 24 and power supply module 32. A vertical rack 158 enables a horizontal table 160 to move vertically between the spaces of a column of the array 102. The table 160 may be provided with a general or specific gripping mechanism 162 to enable the robot 180 to grasp each drawer 24 or power supply 32. Once the mechanism 162 has engaged e.g. the front panel 52 of a drawer 24, then the table 160 is translated horizontally in order to draw the drawer 24 from its space in the array.

0044The robot 150 then transports the drawer 24 and its hardware elements contents to a suitable service area 164 where a replacement drawer 24 is available. The robot 150 deposits the drawer it just removed into a pick up area for a service technician, picks up the replacement drawer 24, returns to the particular space of the bay 102 from which the original drawer was removed and inserts and moves the new drawer 24 into the space until it becomes mechanically engaged and electrically connected in the manner previously described in connection with the subsystem 10.

0045The robot 180 is also provided with cable gripping equipment for removing and replacing each of the cables 106 under program control.

0046Doors or other suitable closures or strips 166 may be provided in the back wall 118 in the vicinity of the tracks 156 so that a forced air plenum arrangement is maintained while still providing access for the robot 180 to each of the arrays 102.

0047A locating system is provided for the robot 180 within the facility 100. One suitable indexing arrangements may be a scale 168 along the base 110 of each array 102. The scale 168 may be two dimensional and read optically by suitable optical sensors in the robot 150, or may be three dimensional and be read by electromechanical cam following equipment. Each drawer 24 is preferably provided with a unique bar code, such as a Universal Product Code, in a standard arrangement and location. An optical scanning device aboard the robot 150 may then scan the bar code to verify that the correct module is being removed for service.

0048The robot 150 operates automatically under program control. A special computer may be provided for the robot 150, or the computing facility 100 may include special service routines as part of its resources management in order to make effective use of the robot. In any event, the robot maintenance service program maintains a facilities resource map which records the precise location of each robotically accessible hardware element within the facility 100 and correlates an identifying number of each module, power supply and cable (preferably recorded on the bar code affixed to the particular element) with its present location. When e.g. a module 24 is replaced, the identifying number of the replacement module is substituted in the facilities map automatically during execution of the service program.

0049While tracks 156 are shown, painted lines may also be provided and followed by a steerable robot having suitable optional sensors for following the lines.