Restoring access to a failed data storage device in a redundant memory system

BACKGROUND

1. Field of the Invention

The present invention relates generally to controlling access to computer memory systems and, more particularly, to restoring access to a failed data storage device in a redundant memory system.

2. Related Art

A computer memory module commonly includes a plurality of integrated circuits (ICs), each of which stores millions of binary digits (bits) of data. Most memory ICs store data bits in columns, rows and planes of memory cells, with each cell comprising a relatively small capacitor. When data is written to a memory cell, its capacitor is either charged to a predetermined voltage to represent a “1” bit, or the capacitor is discharged to represent a “0” bit. If the capacitor's charge changes significantly between the time data is written to the memory cell and the time the memory cell is read, data read from the memory cell will not correctly represent the data previously written to that cell. Such an occurrence is commonly referred to as a memory error.

Memory errors can be classified as hard or soft, depending on whether the errors occur repeatably or randomly. For example, a failed capacitor usually causes its memory cell to be read as a “0” regardless of whether a “1” or a “<b>0</b>” was written to the memory cell. Thus, a failed capacitor usually causes repeatable, or hard, memory errors. In contrast, random or soft memory errors are usually caused by sporadic events, most commonly cosmic rays. A sufficiently high-energy cosmic ray passing through a memory cell capacitor can change the capacitor's charge, altering data stored in the memory cell. Because of their relatively narrow beams, cosmic rays typically affect only one or a small number of memory cells of a memory module.

Progressively smaller capacitors have been used in successive generations of memory ICs, yielding higher densities of memory cells and, therefore, higher memory capacities. Unfortunately, such higher-density memory modules are more susceptible to cosmic ray-induced memory errors than their lower-density counterparts. Smaller capacitors require lower voltages to represent a “1” bit, enabling weaker cosmic rays to alter the contents of the memory cells. In addition, because such memory cells are more densely packed in the ICs, a single cosmic ray can pass through, and therefore affect, a greater number of capacitors than in lower-density memory ICs. Thus, higher-density memory ICs are more likely to incur soft memory errors and are more likely to incur multi-bit, as opposed to single-bit, soft errors than lower-density memory ICs.

Various protocols have been developed to manage memory errors. For example, some memory systems include capabilities similar to those used in redundant arrays of independent disk (RAID) storage systems. In the context of memory systems, the term “RAID” traditionally refers to redundant arrays of industry-standard DIMMs (dual in-line memory modules), although the term “RAIM” (redundant array of independent memory) is also commonly used to refer to such systems, and will be used herein. If one of the redundant storage devices (disk drives or memory modules) fails, the redundancy enables the memory system to use data from the surviving storage devices to reconstruct data stored on the failed device. This process of reconstructing lost data is commonly referred to as error correction.

A RAIM memory system uses a quantity of memory modules (typically four) to store data, and an additional (e.g., a fifth) memory module to store parity information. Data to be stored is divided into four blocks. Each block is stored in one of the data memory modules in a process commonly known as striping. Parity information calculated from the four blocks is stored in the parity memory module. When retrieving data from the memory modules, the ECC logic typically included in the RAIM system attempts to automatically correct detected data errors. If the error cannot be corrected (i.e., it is “uncorrectable”), the data fetched from the failed memory module is reconstructed using the data in the remaining three data memory modules and the parity information in the parity memory module. In addition, the RAIM memory system ceases reading (i.e., takes off-line) the memory module that incurred the uncorrectable error.

After a failed memory module is taken off-line, the remaining memory modules do not thereafter provide the redundancy necessary to be able to recover from an uncorrectable error. That is, if one of the three remaining memory modules, or the parity memory module, subsequently incurs an uncorrectable error, the RAIM memory system will be unable to reconstruct the data. Instead, it will signal an unrecoverable memory error, typically causing the host computer system to crash.

SUMMARY OF THE INVENTION

In one aspect of the present invention, a method of configuring access to a failed memory module is disclosed. The method comprises determining a type of the error; and configuring access to the memory module based on the error type.

In another aspect of the present invention, a computer system is disclosed. The computer system comprises a memory system that comprises a plurality of memory modules; and a memory controller that accesses the plurality of memory modules to service memory requests. The computer system also comprises an error-type memory controller that configures the noted access such that the memory controller can continue to access a failed one of the plurality of memory modules that incurred a soft error.

In a further aspect of the present invention, a memory system for restoring access to a memory module that incurred an error is disclosed. The memory system comprises means for determining a type of the error; and means for restoring access to the memory module if said type of error is determined to be a soft error.

BRIEF DESCRIPTION OF THE DRAWINGS

<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of an exemplary computer in which embodiments of the present invention can be practiced.

<figref idref="DRAWINGS">FIG. 2</figref> is a simplified block diagram illustrating functional components of a memory control apparatus, such as one that can be employed in the computer of <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 3</figref> is a high-level flowchart of operations performed by one embodiment of the memory control apparatus of the present invention.

<figref idref="DRAWINGS">FIG. 4</figref> is a simplified block diagram of the memory controller shown in <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 5</figref> is a simplified schematic diagram of a one-bit slice of one of the switches shown in <figref idref="DRAWINGS">FIG. 4</figref>, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 6</figref> is a simplified diagram of control and status registers (CSR) shown in <figref idref="DRAWINGS">FIG. 4</figref>, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 7</figref> is a data structure diagram of an error log, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 8</figref> is a data structure diagram of error information stored in the error log shown in <figref idref="DRAWINGS">FIG. 7</figref>, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 9</figref> is a block diagram illustrating interactions between hardware and software of a memory system shown in <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 10</figref> is a flowchart illustrating operation of memory control logic shown in <figref idref="DRAWINGS">FIG. 4</figref>, according to one embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 11</figref> is a flowchart illustrating operation of the software shown in <figref idref="DRAWINGS">FIG. 9</figref>, according to one embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention are directed toward providing continued access to a memory module that has incurred an uncorrectable soft error. The inventors have recognized that after a redundant memory system detects an uncorrectable memory error, the memory module that incurred the error (the “failed” memory module) can continue to be accessed with no loss of reliability when the uncorrectable error is soft. Accordingly, embodiments of the present invention configure access to a failed memory module based on the type of uncorrectable memory error; that is, whether the memory error is soft or hard.

To determine the type of error, embodiments of the present invention preferably continue to write data to the failed memory module as though no error occurred, and the failed memory module is monitored (i.e., put on “probation”). While a memory module is on probation, the memory system reconstructs data from the remaining memory modules rather than reading potentially unreliable data from the failed memory module. If, upon analysis, certain embodiments of the present invention determines that the uncorrectable error was hard, the failed memory module is no longer used (i.e., it is “condemned” and taken off-line). On the other hand, if the uncorrectable error was soft, the memory module is brought back on-line (i.e., “exonerated”) for reading, thereby restoring full data redundancy in the memory system. In those embodiments in which data is continually written to a failed memory module during probation, exonerated memory modules will contain current data when brought back on-line.

As noted, as memory densities have increased, there has been a concomitant increase in the likelihood of soft errors. Detection of an uncorrectable memory error, regardless of type, results in the memory module being taken off-line in conventional redundant memory systems. Because soft errors occur frequently, particularly in high-capacity memory devices, conventional redundant memory systems frequently take memory modules off-line and are unable to recover from subsequent uncorrectable errors in the remaining memory modules. In contrast, embodiments of the present invention continue to use memory modules that have experienced uncorrectable soft errors. Because most memory errors are soft, such embodiments of the present invention significantly reduce the likelihood that an unrecoverable error will occur.

Embodiments of the present invention can be implemented in connection with any type of redundant memory system. Embodiments will be described below in the context of one class of redundant memory system, known as a redundant array of independent memory (RAIM) memory system. It should be appreciated, however, that embodiments of the present invention can be implemented in connection with all applicable redundant memory systems unless otherwise noted.

<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of an exemplary computer <b>100</b> implementing a RAIM memory system <b>102</b>. Memory system <b>102</b> comprises a plurality of memory modules <b>104</b>A–E connected to and controlled by a RAIM memory controller <b>108</b>. Computer <b>100</b> further comprises a central processor <b>110</b>, I/O devices <b>112</b> and <b>114</b> and other devices and components (not shown), all directly or indirectly interconnected by a memory bus <b>116</b>.

Data is stored in memory modules <b>0</b>–<b>3</b> (<b>104</b>A–D) while parity information is stored in memory module P (<b>104</b>E). Memory modules <b>104</b> are preferably Rambus® memory modules, or RIMMs® (RAMBUS and RIMM are registered trademarks of Rambus, Inc.) It should be appreciated, however, that dual in-line memory modules (DIMMs) as well as other types of memory modules can also be used. Memory modules <b>104</b> provide redundant storage for data stored by RAIM memory controller <b>108</b>. Although five memory modules are shown in <figref idref="DRAWINGS">FIG. 1</figref>, embodiments of the present invention can be implemented in connection with redundant memory systems having any quantity of memory modules.

<figref idref="DRAWINGS">FIG. 2</figref> is a simplified block diagram illustrating functional components of an error-type memory controller <b>200</b>, according to one embodiment of the present invention. Error-type memory controller <b>200</b> is preferably implemented as a combination of hardware and software components, as described in detail below. The hardware components can, for example, be included in RAIM memory controller <b>108</b>, and the software components can, for example, be executed by central processor <b>110</b>. Alternatively, memory controller <b>200</b> can be implemented solely in hardware or solely in software.

Error-type memory controller <b>200</b> comprises a memory error type identifier <b>202</b> that determines whether an uncorrectable error incurred by a memory module <b>104</b> is hard or soft. As noted, hard memory errors are errors that repeatably occur while soft memory errors are errors that randomly occur. In the exemplary embodiment illustrated in <figref idref="DRAWINGS">FIG. 2</figref>, identifier <b>202</b> receives an indication <b>201</b> that an uncorrectable error has occurred in a memory module <b>104</b>. It should be appreciated that such information can be received, calculated, derived, etc., by another component of memory system <b>102</b>, or by memory error type identifier <b>202</b> based on information provided by other component(s) of memory system <b>102</b>.

Error-type memory controller <b>200</b> also comprises a memory module access configurator <b>204</b> that configures read and write access to memory modules <b>104</b> based on the type of uncorrectable error that has been incurred. Memory access configurator <b>204</b> receives from identifier <b>202</b> indications <b>203</b>, <b>205</b> identifying the type of uncorrectable error that has been incurred, and the particular memory module <b>104</b> in which it occurred. In addition, memory access configurator <b>204</b> also receives error indication <b>201</b>. It should be appreciated that such information need not be strictly received as depicted, but can be derived from other information provided by identifier <b>202</b>, or received or derived from information provided by other component(s) of memory system <b>102</b>.

When an uncorrectable error is detected, memory module access configurator <b>204</b> prevents read access to the failed memory module, thereby placing the memory module on probation until memory error type identifier <b>202</b> determines the type of error that has occurred. As will be described in detail below, memory access configurator <b>204</b> preferably permits continued write access to the failed memory module <b>104</b> during probation. If the error is determined to be a hard error, memory module access configurator <b>204</b> continues to prevent read access to the memory module <b>104</b>. However, if the uncorrectable error is determined to be a soft error, memory module access configurator <b>204</b> re-enables read access to the failed memory module <b>104</b>. Memory access configuration <b>204</b> controls read and write access to memory modules <b>104</b> through the generation of one or more read and write access control signals <b>207</b>. The quantity, format and content of signals <b>207</b> are appropriate for controlling selected components of memory system <b>102</b>.

<figref idref="DRAWINGS">FIG. 3</figref> is a high-level flowchart of operations performed by certain embodiments of the present invention. Process <b>300</b> is invoked in response to the occurrence of an uncorrectable error in a memory module <b>104</b>. As noted, an uncorrectable memory error is one which could not be corrected by the implemented ECC logic. At <b>302</b>, read access to the memory module <b>104</b> that incurred the uncorrectable error is discontinued, thereby placing the memory module on probation. Later, depending on whether the uncorrectable error is determined to be soft or hard, read access to the failed memory module <b>104</b> might be resumed.

At <b>304</b>, write access to the failed memory module <b>104</b> is continued to ensure the memory module contains current data if and when it is exonerated. In certain applications, memory controller <b>108</b> automatically discontinue write access to a memory module <b>104</b> that has incurred an uncorrectable error. When implemented in such systems, embodiments of the present invention preferably inhibit or reverse such operations to ensure continued write access to the failed memory module <b>104</b>.

At <b>306</b>, the uncorrectable data in error is reconstructed using whatever technique is implemented in the redundant memory system. In the exemplary RAIM memory system <b>102</b>, for example, the requested data is reconstructed using the data from the remaining three memory modules <b>104</b> and the parity information stored in memory module <b>104</b>E. It should be appreciated, however, that other data reconstruction techniques can be implemented. The reconstructed data is supplied to central processor <b>110</b>, I/O device <b>112</b> or <b>114</b> or other component which requested the data. In addition, at <b>308</b>, reconstructed data is stored in the failed memory module <b>104</b>; that is, the data is scrubbed so that a next uncorrectable error, if any, that is detected in the same memory module <b>104</b> can be interpreted as a second uncorrectable error rather than a legacy of the first uncorrectable error. Scrubbing the failed memory module <b>104</b> will also ensure that the failed memory module will contain valid data if and when it is exonerated.

The error is logged at <b>310</b>. This current and possibly prior error(s) incurred by the memory module <b>104</b> are analyzed at <b>312</b> to determine whether this error was soft or hard. Such a determination can be based on any number of considerations, examples of which are provided below. Access to the failed memory module <b>104</b> is then configured based on the error type. For example, failed memory modules <b>104</b> that have incurred a soft error are exonerated, as depicted at <b>316</b>. Here, read access is re-enabled. Because write access was not disabled at <b>304</b>, the failed memory module <b>104</b> is thereafter fully functional and system redundancy is restored.

On the other hand, at <b>314</b>, a failed memory module <b>104</b> that has incurred a hard error is condemned. The above embodiments of the present invention continue to prevent read access but permits continued write access. In an alternative embodiment, write access is also disabled. Such an alternative embodiment may be implemented, for example, in memory systems that support ‘hot swap’ or ‘active replacement’ of memory modules <b>104</b> during system operations.

As noted, embodiments of the present invention are preferably implemented as a combination of hardware and software components. A description of one such embodiment is now provided, beginning with a description of its hardware components, followed by a description of its software components and an explanation of how these components interact. In the exemplary embodiment described below, memory error type identifier <b>202</b> is implemented in software executing on central processor <b>110</b> (<figref idref="DRAWINGS">FIG. 1</figref>) and memory module access configurator <b>204</b> is implemented as hardware in RAIM memory controller <b>108</b> (<figref idref="DRAWINGS">FIG. 1</figref>). It should be understood that such an allocation of functionality is exemplary only; the present invention can be implemented in any combination of hardware and software, as noted above.

<figref idref="DRAWINGS">FIG. 4</figref> is a simplified block diagram of memory controller <b>108</b>, in accordance with one embodiment of the present invention. In this embodiment, memory controller <b>108</b> is a RAIM memory controller. Memory control logic <b>404</b> implements memory module access configurator <b>204</b>, as described below. RAIM memory controller <b>108</b> comprises RAIM logic <b>402</b> coupled to, and controlled by, memory control logic <b>404</b> via connection <b>406</b>. In this exemplary embodiment, memory control logic <b>404</b> implements memory module access configurator <b>204</b> as described below.

RAIM logic <b>402</b> is also coupled to memory bus <b>116</b> via a data path <b>408</b>. In addition, RAIM logic <b>402</b> is coupled to memory modules <b>104</b>A–E by data lines <b>410</b>A–D and <b>412</b>, respectively. Separately-controllable, in-line switches <b>414</b>A–D are located in corresponding data lines <b>410</b>A–D. Switches <b>414</b> are controlled by memory control logic <b>404</b> via control lines <b>416</b>A–D, respectively, to configure read and write access to memory modules <b>104</b>. Switches <b>414</b> are described in detail below with reference to <figref idref="DRAWINGS">FIG. 5</figref>.

As noted with respect to <figref idref="DRAWINGS">FIG. 1</figref>, RAIM memory controller <b>108</b> communicates with central processor <b>110</b> and other components of computer <b>100</b> via memory bus <b>116</b>. In the embodiment illustrated in <figref idref="DRAWINGS">FIG. 4</figref>, RAIM memory controller <b>108</b> communicates with external components via an interrupt request line <b>418</b> and a set of registers collectively referred to as control and status registers (CSR) <b>420</b>. In this embodiment, RAIM logic <b>402</b> and memory control logic <b>404</b> read and write from/to CSR <b>420</b> via connections <b>422</b> and <b>424</b>, respectively. Thus, via CSR <b>420</b>, external components, such as the software implementation of memory error type identifier <b>202</b> being executed by central processor <b>110</b>, can obtain status information from, and control operations of, memory control logic <b>404</b> and RAIM logic <b>402</b>.

RAIM logic <b>402</b> distributes data to be stored by memory system <b>102</b> (<figref idref="DRAWINGS">FIG. 1</figref>) across memory modules <b>104</b>A–D in a process commonly known as striping. RAIM logic <b>402</b> receives data to be stored in memory system <b>102</b> over data path <b>408</b>. In this illustrative embodiment, RAIM logic <b>402</b> divides the received data into four blocks and calculates parity information from the four blocks. RAIM logic <b>402</b> then sends the four data blocks and parity information to memory modules <b>104</b> via data paths <b>410</b>A–D and <b>412</b>, respectively. Conversely, when data is fetched from memory system <b>102</b>, RAIM logic <b>402</b> reads four data blocks and parity information from memory modules <b>104</b> over data paths <b>410</b> and <b>412</b>. In addition to managing the storage of data in memory modules <b>104</b>, RAIM logic <b>402</b> includes ECC logic that detects and possibly corrects certain data errors. RAIM logic <b>402</b> sends the (possibly corrected) data to memory bus <b>116</b> via data path <b>408</b>.

In the embodiment illustrated in <figref idref="DRAWINGS">FIG. 4</figref>, RAIM logic <b>402</b> comprises primarily conventional components that perform the above-noted functions. In accordance with aspects of the present invention, once an uncorrectable memory error is detected, RAIM logic <b>402</b> notifies memory control logic <b>404</b> of the error via connection <b>406</b>. In response, memory control logic <b>404</b> (re)configures access to the failed memory module <b>104</b> by controlling certain operations of RAIM logic <b>402</b> via connection <b>406</b>, and by controlling switches <b>414</b>A–D via their corresponding control lines <b>416</b>A–D. Once the type of the error is determined, the software embodiment of memory error identifier <b>202</b> executing on central processor <b>110</b> commands memory control logic <b>404</b> and RAIM logic <b>402</b> via CSR <b>420</b> to further (re)configure access to the failed memory module <b>104</b>. Memory control logic <b>404</b> is described in detail below.

As noted, access to memory modules <b>104</b>A–D is partly or wholly controlled by the states of switches <b>414</b>A–D, respectively. Each switch <b>414</b>A–D controls the transfer of data over a corresponding data bus <b>410</b>A–D to control read and write access by RAIM logic <b>402</b> to a corresponding memory module <b>104</b>A–D. Each data path <b>410</b> is preferably a multi-bit bus, and each switch <b>414</b> preferably controls all the data bits of its corresponding bus <b>410</b>. Each switch <b>414</b> controls whether data from RAIM logic <b>402</b> can be sent (written) to the switch's corresponding memory module <b>104</b> for storage therein. Independent of its write setting, each switch <b>414</b>A–D controls whether data can be fetched (read) by RAIM logic <b>402</b> from the corresponding memory module <b>104</b>A–D.

<figref idref="DRAWINGS">FIG. 5</figref> is a simplified schematic diagram of an exemplary one-bit slice <b>500</b> of one of the data path switches <b>414</b> introduced above with reference to <figref idref="DRAWINGS">FIG. 4</figref>. One-bit bi-directional line <b>502</b> connects bit slice <b>500</b> to its corresponding memory module <b>104</b>. One-bit read data line <b>504</b> and one-bit write data line <b>506</b> connect bit slice <b>500</b> to RAIM logic <b>402</b>. Read enable line <b>508</b> is part of one of the control lines <b>416</b>, by which memory control logic <b>404</b> controls bit slice <b>500</b>. When read enable line <b>508</b> is true, data from the corresponding memory module <b>104</b> arriving on line <b>502</b> is passed by AND gate <b>510</b>, via read data line <b>504</b>, to RAIM logic <b>402</b>. Thus, when read enable line <b>508</b> is true, RAIM logic <b>402</b> can read the corresponding data bit from the corresponding memory module <b>104</b>, and when read enable line <b>508</b> is false, RAIM logic <b>402</b> cannot read data from the corresponding memory module <b>104</b>.

Write enable line <b>512</b> is also part of the control line <b>416</b> that carries signals from memory control logic <b>404</b> to the switch <b>414</b> that includes data slice <b>500</b>. When write enable line <b>512</b> is true, data from RAIM logic <b>402</b> arriving on write data line <b>506</b> is placed by tri-state driver <b>514</b> on bi-directional line <b>502</b> and, thereby, sent to the corresponding memory module <b>104</b>. Thus, when write enable line <b>512</b> is true, RAIM logic <b>402</b> can write data to the corresponding memory module <b>104</b>, and when write enable line <b>512</b> is false, RAIM logic <b>402</b> cannot write data to the memory module.

As noted, in the embodiment illustrated in <figref idref="DRAWINGS">FIG. 4</figref>, control and status registers (CSRs) <b>420</b> are utilized to facilitate communications between external devices, such as central processor <b>110</b>, and memory control logic <b>404</b> and RAIM logic <b>402</b>. <figref idref="DRAWINGS">FIG. 6</figref> is a simplified diagram of one embodiment of CSRs <b>420</b>. CSRs <b>420</b> comprise several flags and fields. Specifically, CSR <b>420</b> includes a RAIM enabled flag <b>602</b>, a RAIM memory error flag <b>604</b>, an update RAIM map flag <b>606</b>, a RAIM write map mask field <b>608</b> and a RAIM read map mask field <b>610</b>.

RAIM enabled field <b>602</b> is a one-bit, read/write flag that controls and indicates the RAIM capability of RAIM logic <b>402</b>. For example, central processor <b>110</b> (<figref idref="DRAWINGS">FIG. 1</figref>) can set RAIM enabled flag <b>602</b> to a “1” to turn on the RAIM capability, or central processor <b>110</b> can clear this flag to a “0” to turn off the RAIM capability. In addition, central processor <b>110</b> can read RAIM enabled flag <b>602</b> to ascertain if the RAIM capability is currently enabled.

RAIM memory error flag <b>604</b> is a one-bit field that indicates whether RAIM logic <b>402</b> (<figref idref="DRAWINGS">FIG. 4</figref>) has detected an error in data from one or more memory modules <b>104</b>. When such an error is detected, read/write RAIM memory error flag <b>604</b> is set to “1,” and memory control logic <b>404</b> generates an interrupt request to central processor <b>110</b> via interrupt request line <b>418</b>. In addition, a value is placed in read/write RAIM read map mask <b>610</b> to indicate which memory module <b>104</b> incurred the error.

RAIM read map mask field <b>610</b> is, in this embodiment, a 5-bit field, with each bit corresponding to one memory module <b>104</b>A–E. RAIM logic <b>402</b> sets individual bits in RAIM read map mask fields <b>610</b> to indicate which memory module <b>104</b> has experienced a memory error. For example, if an error is detected in data from memory module <b>104</b>B (memory module number <b>1</b>), RAIM read map mask <b>610</b> is set to 00010 (binary), with the “0” bits indicating that memory modules <b>104</b>A and <b>104</b>C–E have not experienced an error, and the “1” bit indicating the error that occurred in memory module <b>104</b>B.

In conjunction with write-only update RAIM map flag <b>606</b>, RAIM map mask fields <b>608</b> and <b>610</b> can be used to selectively enable or disable any combination of memory modules <b>104</b> for writing and/or reading. If update RAIM map flag <b>606</b> is set to “1,” memory control logic <b>404</b> (<figref idref="DRAWINGS">FIG. 4</figref>) reads RAIM map masks <b>608</b> and <b>610</b> and enables or disables memory modules <b>104</b> for reading, according to the values of RAIM map masks <b>608</b> and <b>610</b>. For example, to enable all memory modules <b>104</b> for writing and reading, except memory module <b>104</b>B (memory module number <b>1</b>) for reading, central processor <b>110</b> can set RAIM read map mask <b>610</b> to 11101 (binary) and set update RAIM map flag <b>606</b> to “1.” The “0” bit in 11101 (binary) disables memory module <b>104</b>B for reading, while the “1” bits enable the other memory modules <b>104</b>A, C, D and E for reading. In response, memory control logic <b>404</b> controls switches <b>414</b> to enable and disable (as appropriate) individual memory modules <b>104</b> for reading. Bits in RAIM write map mask <b>608</b> similarly enable and disable write access to memory modules <b>104</b>, as noted above.

As one of ordinary skill in the art would appreciate, to prevent memory system <b>102</b> from returning inconsistent data, changes to settings of switches <b>414</b> to (re)enable read access to memory modules <b>104</b> preferably occur only during a dynamic random-access memory (DRAM) refresh cycle or other time when no read or write access to memory modules <b>104</b> can occur. If read access to a memory module <b>104</b> were to be enabled, for example, while RAIM logic <b>402</b> is reading remaining memory modules <b>104</b> to reconstruct data for a failed memory module, inconsistent data could be retrieved by RAIM logic <b>402</b>, causing an error in the reconstructed data. Because access to memory modules <b>104</b> is prohibited during a refresh cycle, memory control logic <b>404</b> delays the settings of switches <b>414</b> until the next refresh cycle.

As noted, embodiments of the present invention configure read and write access to memory modules <b>104</b> based on the types of errors (soft or hard) incurred by the memory modules. To determine the type of memory error, embodiments of the present invention preferably analyze an error log of, for example, historic error information. <figref idref="DRAWINGS">FIG. 7</figref> is a data structure diagram of one exemplary embodiment of an error log <b>700</b>. Error log <b>700</b> contains error information <b>702</b>A–E corresponding to memory modules <b>104</b>A–E, respectively. <figref idref="DRAWINGS">FIG. 8</figref> is a data structure diagram of an exemplary embodiment of the error information <b>702</b> stored in error log <b>700</b>. In this exemplary embodiment, error information <b>702</b> comprises the quantity of errors that occurred during one or more predetermined time periods. In the embodiment shown in <figref idref="DRAWINGS">FIG. 8</figref>, for example, error information <b>702</b> comprises error counts <b>802</b>A–C for each of several days. Other embodiments of error log <b>700</b> and error information <b>702</b> are, of course, possible, as would be appreciated by those of ordinary skill in the art. For example, in one alternative embodiment, error counts <b>802</b> are accumulated for other time periods, such as per hour. In further embodiments, more complex error log information is maintained and/or calculated. In alternative embodiments, for example, a moving average number of errors or error rate can be maintained or calculated for each memory module <b>104</b>. It should be further appreciated that in all the above and other embodiments, this error information can be more finely resolved. For example, error information can be maintained for each page of each memory module <b>104</b>.

Having described the hardware implementation of memory module access configurator <b>204</b> of this embodiment of the present invention, embodiments of a corresponding software implementation of memory error identifier <b>202</b> will now be described with reference to <figref idref="DRAWINGS">FIG. 9</figref>. As noted, when memory system <b>102</b> (<figref idref="DRAWINGS">FIG. 1</figref>) detects an uncorrectable error in data from a memory module <b>104</b>, it generates an interrupt request to central processor <b>110</b> via interrupt request line <b>418</b> (<figref idref="DRAWINGS">FIG. 4</figref>). <figref idref="DRAWINGS">FIG. 9</figref> is a block diagram of memory error type identifier software <b>900</b> being executed by central processor <b>110</b> to handle an interrupt <b>902</b>. An interrupt service routine <b>904</b> is configured to receive control when memory control logic <b>404</b> (<figref idref="DRAWINGS">FIG. 4</figref>) generates interrupt <b>902</b>. An operating system <b>906</b> is preferably used to dispatch interrupt <b>902</b> to interrupt service routine <b>904</b>. Interrupt service routine <b>904</b> is also configured to control memory system <b>102</b> by reading and writing CSR <b>420</b>, as shown by arrow <b>908</b>. Error log <b>700</b> is preferably used by interrupt service routine <b>904</b> to keep track of information regarding errors incurred by memory system <b>102</b>. Alternatively, error log <b>700</b> can be maintained in another memory such as in RAIM memory controller <b>108</b>. Still other implementations could maintain such error information within the RAIM subsystem.

The operations performed by embodiments of the hardware and software components are described below with reference to <figref idref="DRAWINGS">FIGS. 10 and 11</figref>. As noted above with reference to <figref idref="DRAWINGS">FIGS. 1 and 4</figref>, memory control logic <b>404</b> comprises memory modules access configurator <b>204</b>. <figref idref="DRAWINGS">FIG. 10</figref> is a flowchart illustrating operations performed by hardware of one embodiment of memory control logic <b>404</b> when an uncorrectable data error is detected in a memory module <b>104</b>. At <b>1002</b>, memory control logic <b>404</b> sets a value in RAIM read map mask <b>610</b> (<figref idref="DRAWINGS">FIG. 6</figref>) to indicate which memory module <b>104</b> incurred an error. Memory control logic <b>404</b> also sets RAIM memory error flag <b>604</b> to “1.”

At <b>1004</b>, memory control logic <b>404</b> generates an interrupt request to central processor <b>110</b> via interrupt request line <b>418</b>. Operations associated with the processing of such an interrupt by memory error identifier <b>202</b> are, as noted, described below with reference to <figref idref="DRAWINGS">FIG. 11</figref>.

As noted, in certain applications, RAIM logic <b>402</b> ceases writing to a memory module <b>104</b> upon detection of an uncorrectable error. For example, RAIM logic <b>402</b> and other portions of memory controller <b>108</b> may be implemented with off-the-shelf integrated circuits (ICs). Some such ICs automatically disable write access to a failed memory module <b>104</b> after detecting an uncorrectable data error from that module. In such applications, memory control logic <b>104</b> re-enables write access to the failed memory module at block <b>1006</b>. Preferably, such write access is re-enabled prior to the completion of the current memory cycle. This avoids the circumstance in which central processor <b>110</b> issues a subsequent write request to memory system <b>102</b> prior to the re-enablement of write access to the failed memory module <b>104</b>.

At <b>1008</b>, RAIM logic <b>402</b> writes the reconstructed data to the failed memory module <b>104</b>. As previously described, when a memory module <b>104</b> incurs an error, RAIM logic <b>402</b> reconstructs the data stored in the failed memory module <b>104</b>, and forwards the requested data over memory bus <b>116</b> to service the memory request. At <b>1008</b>, the reconstructed data is also written to the failed memory module <b>104</b> in an operation referred to above as scrubbing. If the error in the failed memory module <b>104</b> is soft, scrubbing the data in the failed memory module <b>104</b> enables the memory module to subsequently fetch data from the same memory cells without generating additional errors. On the other hand, if the error is hard, the next time the data is fetched, another uncorrectable error will occur.

As noted with respect to <figref idref="DRAWINGS">FIGS. 9 and 10</figref>, when memory control logic <b>404</b> (hardware) detects an error in data from one or more of the memory modules <b>104</b>, the memory control logic generates an interrupt <b>902</b> at <b>1002</b>. <figref idref="DRAWINGS">FIG. 11</figref> is a flowchart illustrating operations of an embodiment of interrupt service routine <b>904</b> which handles interrupt <b>902</b>. At <b>1102</b>, CSRs <b>420</b> (<figref idref="DRAWINGS">FIG. 6</figref>) are read to ascertain which memory module(s) <b>104</b> incurred an error. If necessary, RAIM memory error flag <b>604</b> in CSRs <b>420</b> is cleared to “0” to indicate to RAIM memory controller <b>108</b> that the interrupt is being handled. At <b>1104</b>, information pertaining to the error is logged, such as in error log <b>700</b> (<figref idref="DRAWINGS">FIG. 7</figref>). For the embodiment of error log <b>700</b> illustrated in <figref idref="DRAWINGS">FIGS. 7 and 8</figref>, the error count <b>802</b> corresponding to the current day is incremented. In the embodiment illustrated in <figref idref="DRAWINGS">FIG. 8</figref>, a periodic process (not shown) shifts the error counts <b>802</b> down one bucket <b>802</b> at the end of each day.

Returning to <figref idref="DRAWINGS">FIG. 11</figref>, at <b>1106</b>, memory error identifier software <b>900</b> determines whether the error threshold has been exceeded. As noted, the error threshold can be determined based on one or more of a myriad of factors and calculations. If the error threshold has not been exceeded, control passes to <b>1108</b> at which memory error identifier software <b>900</b> writes to CSRs <b>420</b> (<figref idref="DRAWINGS">FIG. 6</figref>) to re-enable read access to the failed memory module <b>104</b>.

Otherwise, control passes to <b>1110</b> at which an error message is displayed identifying the failed memory module <b>104</b>. Referring back to <figref idref="DRAWINGS">FIG. 9</figref>, interrupt service routine <b>904</b> can invoke operating system <b>906</b> to display this message. Other well-known techniques, such as illuminating LEDs on the failed memory module <b>104</b>, can be employed. In either case, at <b>1112</b> the interrupt is dismissed.

In an alternative embodiment, portions of a failed memory module are taken out of service, without taking the entire memory module off-line. In this embodiment, operating system <b>906</b> (<figref idref="DRAWINGS">FIG. 9</figref>) allocates memory modules <b>104</b> in fixed- or variable-size portions called pages, in a well-known manner. Operating system <b>906</b> maintains a bad page list <b>912</b> that catalogs faulty or questionable memory pages. Once committed to bad page list <b>912</b>, a memory page is thereafter not allocated for use. Returning to <figref idref="DRAWINGS">FIG. 11</figref>, in this alternative embodiment, if the error threshold is exceeded at <b>1106</b>, control passes from <b>1110</b> to <b>1114</b>, instead of to <b>1112</b>. At <b>1114</b>, the page(s) that incurred the error is/are added to bad page list <b>910</b>. Read access to the failed memory module <b>104</b> is then re-enabled at <b>1108</b>, and the interrupt is dismissed at <b>1112</b>. In this alternative embodiment, checking the error threshold at <b>1106</b> is optional; control can pass from <b>1104</b> directly to <b>1110</b> or <b>1114</b>.

While various embodiments of the present invention have been described herein, it should be understood that they have been presented by way of example only, and not limitation. For example, embodiments of the present invention can be implemented in connection with other redundant memory systems having the same or different storage devices that incur hard and soft errors due to conditions or circumstances other than those noted above. For example, embodiments of the present invention can be implemented in mirrored memory systems. It can also be used with alternatives to RAID <b>4</b> redundancy, such as RAID <b>5</b> or RAID <b>6</b> redundancy or combinations of redundancy technologies. In addition, embodiments of the present invention can be used in systems that permit “hot swapping” of failed memory modules. Furthermore, the invention is not limited to computer applications.

For example, embodiments of the present invention can be used in any system that requires fault-tolerant memory, such as navigation systems and portable defibrillators. As another example, aspects of the present invention that are implemented in software or firmware can be stored in a memory, and control operation of a computer such as a personal computer, workstation, mainframe, control processor, microprocessor or a control processor embedded in another system. The memory can, but need not, be part of an integrated circuit that includes the control processor or microprocessor. The software or firmware can be stored on a removable or fixed computer-readable medium, examples of which include, but are not limited to, CD-ROM, CD-RW, DVD-ROM, DVD-RW, ZIP disk, hard disk or floppy disk. In addition, the software or firmware can be transmitted over a wireless or wired communication link, such as a public or private local or wide area computer network, including the Internet, or a telephone network. Alternatively, aspects of the present invention can be implemented in hardware. For example, the above functionality can be implemented in a single integrated circuit or in a combination of integrated and/or discrete circuits. All or portions of the functionality of the interrupt service routine or memory control logic can be implemented as combinatorial logic, an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).