Failure-specific data collection and recovery for enterprise storage controllers

BACKGROUND

1. Field of the Invention

This invention relates to enterprise storage controllers, and more particularly to apparatus and methods for collecting data and recovering from failures in enterprise storage controllers.

2. Background of the Invention

In enterprise storage controllers, such as the IBM DS8000™ storage controllers, a “warmstart” may be used as both a debug mechanism and a recovery method. The warmstart may initiate a state-save operation to collect state-save data. This state-save data may contain data structures that are deemed necessary to analyze the problem or condition that necessitated the warmstart. In addition, the state-save data may contain component-specific microcode path traces and general traces. Component-specific traces may be saved in separate buffers. General traces, which record all the activity of each defined processor, may also be saved in separate buffers.

The warmstart itself may allow the storage controller to reinitialize certain data structures, rebuild linked lists, and restore the storage controller to a more known state. During the warmstart, there are periods of time where I/O may be blocked and therefore must be redriven. Accordingly, the duration of the warmstart should be as short as possible to minimize the impact on the storage system. This is particularly important in applications such as time-sensitive banking and reservation systems.

A warmstart may be automated for both debug and recovery when the storage controller detects certain conditions, such as microcode detected errors (MDEs) and microcode logic errors (MLEs). In certain cases, a warmstart may be initiated by host software or applications. For example, a host may send a state-save command to the storage controller when the host detects certain conditions that would warrant collecting data at the storage controller. Typically, after an application fails, the host will send a state-save command to the volume that was used by the application. In addition, a user can manually force a warmstart on the storage controller command line at any time.

When a warmstart occurs, the state-save data that is collected at a customer site may be transmitted to a customer support center. The state-save data may be very large, possibly hundreds of megabytes. As a result, it can take several hours for the state-save data to be successfully transmitted to the customer support center. Unfortunately, the time spent in transit may delay support and problem analysis. For a customer-critical problem, the problem may reoccur at the storage controller while the state-save data is in transit and before the customer support center has a chance to understand the problem or provide an action plan. For a customer, this situation can be very aggravating. The impact of the problem can also increase as the problem reoccurs.

Once the state-save data arrives at a customer support center, the data may be analyzed to determine the cause of the problem. Many times, only a small amount of the state-save data is needed to determine what caused the problem. Thus, much of the data may be unneeded, and much of the delay associated with the unneeded data may be unnecessary. Furthermore, much of the resources, time, and effort required to collect and transmit the state-save data may be wasted.

Many of the problems that occur in the storage controller are typically related to a host/device relationship. This is because a storage controller may be device-centric, meaning that a host (e.g., a open system or mainframe server) may access a specific device (e.g., a disk drive, a tape drive, etc.) to perform I/O operations (i.e., reads and writes) to that device. Consequently, only data directly related to the problem (e.g., data related to a host/device relationship, data related to a specific piece of hardware with a problem, data related to events surrounding an MDE or MLE, or the like) may be needed to analyze the problem. For example, data may be needed with respect to the state of device data structures, the data path used for the host-to-device communications, or the like. The data needed may include data structures for several components, where most or all of the data structures are related to a specific device/host relationship.

Other issues may also arise when a warmstart collects state-save data. For example, when a state save occurs, even though the problem is related to a specific host and/or device, all devices connected to the storage controller may become unavailable to all hosts attached to the storage controller. This means I/O activity is suspended for all hosts and all devices during the warmstart. This is a heavy penalty to pay for an isolated problem between a specific host and device.

It has also been observed that a warmstart recovery may fail for reasons unrelated to the original problem. For instance, since all I/O activity may be suspended during the warmstart, all data queued in cache may need to be destaged to the storage devices (e.g., disk drives). If the queue of data in the cache is huge, this may cause the warmstart to take longer than normal. In the worse case scenario, this delay may cause another warmstart. In such cases, the storage controller may be unable to recover on its own.

In view of the foregoing, what are needed are apparatus and methods to reduce the number and/or frequency of warmstarts that occur in enterprise storage controllers. Further needed are apparatus and methods to more closely tailor data collection and recovery processes on specific failures and host/device relationships, thereby allowing I/O to continue between hosts and storage devices unrelated to the problem. Yet further needed are apparatus and methods to reduce the amount of data collected in association with a failure condition, thereby reducing the time and resources needed to analyze the data or transmit the data to a customer support center.

SUMMARY

The invention has been developed in response to the present state of the art and, in particular, in response to the problems and needs in the art that have not yet been fully solved by currently available apparatus and methods. Accordingly, the invention has been developed to provide improved apparatus and methods for handling failure conditions in storage controllers. The features and advantages of the invention will become more fully apparent from the following description and appended claims, or may be learned by practice of the invention as set forth hereinafter.

Consistent with the foregoing, a method for handling a failure condition in a storage controller is disclosed herein. In certain embodiments, such a method may include initially detecting a failure condition in a storage controller. The failure condition may be associated with a specific host and specific storage device connected to the storage controller. The method may further include determining a failure ID associated with the failure condition. Using the failure ID, an entry may be located in a data collection and recovery table. This entry may indicate one or more data collection and/or recovery processes to execute in response to the failure condition. The method may further include executing the data collection and/or recovery processes indicated in the entry. While executing the data collection and/or recovery processes, I/O activity between hosts and storage devices not associated with the failure condition may be maintained.

In selected embodiments, the method may further include monitoring the number of times the failure condition occurs. In the event the number of times exceeds a threshold value, the method may execute a more drastic recovery process such as a warmstart or failover. In certain embodiments, the method may be designed to execute a pre-data-collection process prior to and/or while executing the data collection process. For example, the pre-data-collection process may cause a storage device associated with the failure condition to return a busy indicator in response to I/O from any host.

A corresponding apparatus and computer program product are also disclosed and claimed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through use of the accompanying drawings, in which:

<figref idrefs="DRAWINGS">FIG. 1</figref> is a high-level block diagram showing one embodiment of a system comprising a storage controller, multiple hosts, and multiple storage devices;

<figref idrefs="DRAWINGS">FIG. 2</figref> is a high-level block diagram showing the internal components of one embodiment of a server in the storage controller;

<figref idrefs="DRAWINGS">FIG. 3</figref> is a high-level block diagram showing various software modules that may be stored in the server's memory;

<figref idrefs="DRAWINGS">FIG. 4</figref> is a high-level block diagram showing one example of an entry in a data collection and recovery table;

<figref idrefs="DRAWINGS">FIG. 5</figref> is a flow chart showing one example of a method for handling a failure condition in the storage controller; and

<figref idrefs="DRAWINGS">FIG. 6</figref> is a high-level block diagram showing another example of an entry in the data collection and recovery table.

DETAILED DESCRIPTION

It will be readily understood that the components of the present invention, as generally described and illustrated in the Figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the invention, as represented in the Figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of certain examples of presently contemplated embodiments in accordance with the invention. The presently described embodiments will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout.

As will be appreciated by one skilled in the art, the present invention may be embodied as an apparatus, system, method, or computer program product. Furthermore, the present invention may take the form of a hardware embodiment, a software embodiment (including firmware, resident software, micro-code, etc.) configured to operate hardware, or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “module” or “system.” Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code stored in the medium.

Any combination of one or more computer-usable or computer-readable medium(s) may be utilized. The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium may include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CDROM), an optical storage device, or a magnetic storage device. In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions or code. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

Referring to <figref idrefs="DRAWINGS">FIG. 1</figref>, one embodiment of a system <b>100</b> comprising a storage controller <b>102</b>, one or more host devices <b>104</b><i>a</i>-<i>d</i>, and one or more storage devices <b>106</b><i>a</i>-<i>d </i>is illustrated. In general, the storage controller <b>102</b> may enable one or more hosts <b>104</b><i>a</i>-<i>d </i>(e.g., open system and/or mainframe servers) to access (i.e., read or write) data in one or more storage devices <b>106</b><i>a</i>-<i>d </i>(e.g., disk drives, tape drives, etc.). In selected embodiments, the storage controller <b>102</b> may include one or more servers <b>108</b><i>a</i>, <b>108</b><i>b</i>. The storage controller may also include host adapters <b>110</b> and device adapters <b>112</b> to enable the hosts <b>104</b><i>a</i>-<i>d </i>and storage devices <b>106</b><i>a</i>-<i>d </i>to communicate with the storage controller <b>102</b>. The servers <b>108</b><i>a</i>, <b>108</b><i>b </i>may provide redundancy to ensure that data is always available to the hosts <b>104</b><i>a</i>-<i>d</i>. Thus, when one server <b>108</b><i>a </i>fails, the other server <b>108</b><i>b </i>may remain functional to ensure that I/O is able to continue between the hosts <b>104</b><i>a</i>-<i>d </i>and the storage devices <b>106</b><i>a</i>-<i>d</i>. This process may be referred to as a “failover.”

One example of a storage controller <b>102</b> having an architecture similar to that illustrated in <figref idrefs="DRAWINGS">FIG. 1</figref> is the IBM DS8000™ enterprise storage controller. The DS8000™ is a high-performance, high-capacity storage controller providing disk storage that is designed to support continuous operations. DS8000™ series models use IBM's POWER5™ servers <b>108</b>, which are integrated with IBM'sirtualization engine technology. The virtualization engine provides a partitioning capability that allows the server's resources and storage resources to be divided up to do different tasks.

Nevertheless, embodiments of the invention are not limited to the IBM DS8000™ enterprise storage controller, but may include any comparable or analogous storage system, regardless of the manufacturer, product name, or components or component names associated with the system. Furthermore, any system that could benefit from one or more embodiments of the invention is deemed to fall within the scope of the invention. Moreover, apparatus and methods in accordance with the invention may be used with any storage system that exhibits the same issues or problems disclosed herein. Such systems are also deemed to fall within the scope of the present invention. Thus, the IBM DS8000™ is presented only by way of example and is not intended to be limiting.

Referring to <figref idrefs="DRAWINGS">FIG. 2</figref>, in selected embodiments, each server <b>108</b> may include one or more processors <b>200</b> (e.g., an n-way symmetric multiprocessor), memory <b>202</b>, and various ports <b>204</b> and/or interfaces <b>204</b>. The memory <b>202</b> may include volatile memory <b>208</b> (e.g., RAM) as well as non-volatile memory <b>210</b> (e.g., ROM, EPROM, EEPROM, hard disks, flash memory, etc.). In selected embodiments, the volatile memory <b>208</b> may include a cache <b>212</b>. Whenever a host <b>104</b> performs a read operation, the servers <b>108</b> may fetch data from the storages devices <b>106</b> and save it in the cache <b>212</b> in the event it is required again. If the data is requested again by a host <b>104</b>, the server <b>108</b> may fetch the data from the cache <b>212</b> instead of fetching it from the storage device <b>106</b>, saving both time and resources.

The volatile memory <b>208</b> and non-volatile memory <b>210</b> may store software modules that run on the processor(s) <b>200</b> and are needed to access data in the storage devices <b>106</b>. The servers <b>108</b> may host at least one instance of these software modules, which collectively may also be referred to as a “server,” albeit in software form. These software modules may manage all read and write requests to logical volumes in the storage devices <b>106</b>.

In selected embodiments, a server <b>108</b> may communicate with a management console <b>214</b>. The management console <b>214</b> may be used to monitor, manage, and configure the server <b>108</b>, as well as notify a user and/or a customer service center <b>216</b> when the storage controller <b>102</b> requires service. For example, the management console <b>214</b> may be used to configure and manage logical partitions in the storage devices <b>106</b>, as well as monitor and identify hardware/software problems in the server <b>108</b>. In selected embodiments, the management console <b>214</b> is a dedicated workstation or computer that provides a GUI interface to configure the storage controller <b>102</b> and storage devices <b>106</b> in a desired manner. The management console <b>214</b> may act as a focal point for a service representative to determine an appropriate service strategy for the storage controller <b>102</b>. In selected embodiments, the management console <b>214</b> may be connected to a network <b>218</b> to enable centralized management of the storage controller <b>102</b> from a remote location.

Referring to <figref idrefs="DRAWINGS">FIG. 3</figref>, in selected embodiments, one or more modules for performing various operations in accordance with the invention may be stored in the memory <b>202</b> of the servers <b>108</b>. In selected embodiments, these modules may include a data collection and recovery module <b>300</b> for performing failure-specific data collection and recovery processes in the event a failure occurs in a server <b>108</b>. A “failure” is defined broadly herein to include any failure, error, problem, or event that occurs in a host device <b>104</b>, the storage controller <b>102</b>, or a storage device <b>106</b> where it may be desirable to execute data collection and/or recovery processes in the storage controller <b>102</b>.

As previously mentioned, conventional data collection and recovery processes may involve performing a “warmstart” in a server <b>108</b> when a failure or error condition occurs. This warmstart may initiate a state-save operation that collects data in order to analyze the problem or condition that necessitated the warmstart. The warmstart may also execute various recovery processes after the state-save data is collected.

Unfortunately, when a warmstart occurs, all storage devices <b>106</b> connected to the storage controller <b>102</b> may become unavailable to all hosts <b>104</b> attached to the storage controller <b>102</b>. This is a heavy penalty to pay for an isolated problem between a specific host <b>104</b> and a specific device <b>106</b>. The warmstart may also collect more data than is actually needed to analyze and resolve the problem, wasting both time and resources.

The data collection and recovery module <b>300</b> may address these problems by focusing data collection and recovery processes on specific failures and host/device relationships, without disabling I/O activity between hosts <b>104</b> and storage devices <b>106</b> that are unrelated to the failure. This may be accomplished in many cases without performing a warmstart. By focusing on specific failures, rather than using a “kitchen sink” approach to data collection, the data collection and recovery module <b>300</b> may significantly reduce the amount of data that is collected to analyze and resolve a failure.

In selected embodiments, the data collection and recovery module <b>300</b> may include one or more of a data collection and recovery table <b>302</b>, a detection module <b>304</b>, a location module <b>306</b>, a pre-data-collection module <b>308</b>, a data collection module <b>310</b>, a recovery module <b>312</b>, a maintenance module <b>314</b>, a count module <b>316</b>, a threshold module <b>318</b>, and a reset module <b>320</b>. The term “table” is defined broadly herein to include any table, matrix, array, file, or data structure for arranging data. The illustrated modules are presented only by way of example. For example, some embodiments of the invention may include more or fewer modules than those illustrated in <figref idrefs="DRAWINGS">FIG. 3</figref>.

The data collection and recovery table <b>302</b> may store one or more entries <b>322</b>. Each entry <b>322</b> may correspond to a particular failure condition. In certain embodiments, each entry <b>322</b> may describe data structures and traces to be collected and recovery processes to be executed in response to a particular failure condition. Since failure conditions may differ, actions taken in response to the failure conditions may also differ. Furthermore, since a failure may be associated with a specific host <b>104</b> and a specific storage device <b>106</b>, action may be taken for the specific host <b>104</b> and storage device <b>106</b> while allowing I/O to continue between hosts <b>104</b> and storage devices <b>106</b> not associated with the failure. Several examples of entries <b>322</b> and information contained in the entries <b>322</b> are described in association with <figref idrefs="DRAWINGS">FIGS. 4 and 6</figref>.

In selected embodiments, a detection module <b>304</b> may detect and identify a failure condition in the storage controller <b>102</b>. For example, where the storage controller <b>102</b> is an IBM enterprise storage controller, the detection module <b>304</b> may detect and identify a failure condition such as a microcode detected error (MDE) or microcode logic error (MLE). The detection module <b>304</b> may also identify host(s) <b>104</b> and/or storage device(s) <b>106</b> that are associated with the failure condition. This may be accomplished, for example, by identifying host(s) <b>104</b> and/or storage device(s) <b>106</b> that were actively performing I/O when the failure condition occurred.

A location module <b>306</b> may locate an entry <b>322</b> in the data collection and recovery table <b>302</b> that corresponds to the detected failure condition. This may be accomplished, for example, by determining a failure ID (such as an MDE or MLE number) associated with the failure condition and then locating an entry <b>322</b> in the table <b>302</b> by locating the failure ID. Thus, in selected embodiments, a failure ID may provide an index to the data collection and recovery table <b>302</b>, although other indexing methods are also possible. The failure ID will be discussed in more detail hereafter.

A pre-data-collection module <b>308</b> may initiate processes to execute prior to and/or while the data collection process is executing. For example, the pre-data-collection module <b>308</b> may cause the storage controller <b>102</b> to return a “busy” signal in response to all host I/O to the storage device <b>106</b> associated with the failure condition while data is being collected. For example, in an IBM S/<b>390</b> host and storage device relationship, the storage controller <b>102</b> may present CU BUSY status (control unit busy status) in response to I/O from any host <b>104</b> until all necessary state-save data is collected.

A data collection module <b>310</b> may initiate processes for collecting state-save data <b>324</b> associated with the failure condition. For example, the data collection module <b>310</b> may collect data structures <b>326</b>, component-specific traces <b>328</b> (i.e., traces specific to a particular hardware or software component), global traces <b>330</b>, or other data that may be helpful to analyze, fix, or resolve the failure condition. Unlike the data collection that may occur during a warmstart, the data collection module <b>310</b> may collect state-save data that is specific to the particular failure condition. This will likely reduce the amount of data that is collected, saving both time and resources, and potentially accelerate any response to the problem since the data may be transmitted to the appropriate individual or service center and analyzed in a more expeditious manner.

A recovery module <b>312</b> may be used to invoke recovery processes (as identified in the corresponding failure condition entry <b>322</b>) associated with the failure condition. The recovery process may include all actions or operations needed to recover from the failure. In certain embodiments, this may include causing the storage controller <b>102</b> to return a “busy” signal in response to all host I/O to the storage device <b>106</b> associated with a failure condition while the recovery processes are executing.

A maintenance module <b>314</b> may be used to maintain communication between hosts <b>104</b> and storage devices <b>106</b> not related to the failure condition while the other modules <b>304</b>, <b>306</b>, <b>308</b>, <b>310</b>, <b>312</b>, <b>316</b>, <b>318</b>, <b>320</b> are operating. Thus, devices <b>104</b>, <b>106</b> not associated with the failure condition may continue to operate while the failure is being handled.

In selected embodiments, a count module <b>316</b> may be used to count and update a count value reflecting the number of times a particular error condition has occurred. This may be helpful to determine the severity or frequency of the problem and take a more appropriate response. For example, if a problem is recurring, an escalated recovery process such as a warmstart or failover may be needed. In selected embodiments, a threshold module <b>318</b> may be use to invoke an escalated recovery process when the count value has reached a threshold value. For example, a warmstart or failover may be performed if the count value reaches a particular threshold value. A reset module <b>320</b> may be used to reset the count value once the escalated recovery process is invoked.

As mentioned, escalated recovery processes may include warmstarts and failovers, with the failover typically being the more severe response. In selected embodiments, a warmstart module <b>332</b> may be used to perform a warmstart in the event the count value reaches the threshold value. The warmstart module <b>332</b> may include a data collection module <b>334</b> and a recovery module <b>336</b>. The data collection module <b>334</b> may collect state-save data, which will typically be larger in size and more comprehensive than the state-save data collected by the data collection module <b>310</b> in the data collection and recovery module <b>300</b>. Similarly, the recovery module <b>336</b> may invoke recovery processes associated with a warmstart. This may include temporarily suspending I/O between all hosts <b>104</b> and all storage devices <b>106</b>, regardless of whether they were associated with the failure condition or not.

Similarly, a failover module <b>338</b> may be used to perform a failover operation in the event a server <b>108</b> cannot recover from a failure condition using a focused recovery process or a warmstart. The failover may be more severe than any of the other responses and may include shutting down one of the servers <b>108</b><i>a </i>and operating exclusively with the other server <b>108</b><i>b</i>. This operation may continue until corrective action is taken.

Referring to <figref idrefs="DRAWINGS">FIG. 4</figref>, one embodiment of an entry <b>322</b> (or record <b>322</b>) in the data collection and recovery table <b>302</b> is illustrated. As mentioned, each entry <b>322</b> may correspond to a particular failure condition. Each entry <b>322</b> may contain a number of fields, although each field may be populated differently. These fields are presented only by way of example and are not intended to be limiting.

In the illustrated embodiment, each entry <b>322</b> includes a field <b>400</b> storing a failure ID. This failure ID may include a unique identifier (such as an MDE or MLE number) associated with a particular error or failure condition. The entry <b>322</b> may also include a field <b>402</b> associated with one or more pre-data-collection processes. This field <b>402</b>, for example, may store pointers to pre-data-collection processes to execute before or while collecting data associated with the failure condition. The entry <b>322</b> may also include a field <b>404</b> associated with one or more collection processes. This field <b>404</b> may indicate which data structures and/or traces to collect, or may store pointers to processes used to collect such data. In selected embodiments, the data collected may be a subset of data collected during a warmstart, thereby reducing the amount of data collected.

The entry <b>322</b> may also include a field <b>406</b> associated with one or more recovery processes. This field <b>406</b>, for example, may store pointers to processes used to recover from a particular failure. This field <b>406</b> may also optionally store pointers to processes to execute while the server <b>108</b> is recovering. For example, this field <b>406</b> may point to a process to return a “busy” signal in response to all host I/O to a storage device <b>106</b> associated with a failure condition.

In selected embodiments, the entry <b>322</b> may also store a count value <b>408</b>. As previously mentioned, this count value <b>408</b> may be incremented or modified each time a failure associated with the entry <b>322</b> occurs. When the count value <b>408</b> reaches a threshold value <b>410</b>, a more drastic measure such as an escalated recovery process (e.g., a warmstart or failover) may be executed. For example, one benefit of a warmstart is that it may reset certain data structures. In selected embodiments, a field <b>412</b> may be associated with one or more escalated recovery processes. Like the previous examples, this field <b>412</b> may, in certain embodiments, store pointers to these escalated recovery processes.

It should be recognized that some of the fields in the entry <b>322</b> may be empty in certain circumstances. For example, it may be desirable to collect data for certain failures without executing any recovery process. In other cases, it may be desirable to execute recovery processes without collecting any data. For example, if a command sent by a host <b>104</b> is rejected by a storage device <b>106</b>, this may cause the application that generated the command to terminate in the host <b>104</b>, causing an error or failure condition in the host <b>104</b>. The host <b>104</b> may not have any way to initiate a warmstart or other action in the storage controller <b>102</b> (which could be used to collect data indicating why the command was rejected).

In selected embodiments, an entry <b>322</b> may be included in the table <b>302</b> for such a rejected command. This entry <b>322</b> may be used to initiate data collection in the storage controller <b>102</b> in the event the command is rejected. In such cases, data may be collected in the storage controller <b>102</b> (in order to analyze the problem) without initiating any recovery processes in the storage controller <b>102</b> (since the problem is in the host <b>104</b>). In other embodiments, the data collection and recovery module <b>300</b> may be configured to initiate data collection and/or recovery processes in response to other commands or actions initiated by the host <b>104</b>.

Referring to <figref idrefs="DRAWINGS">FIG. 5</figref>, one embodiment of a method <b>500</b> for handling a failure condition in a storage controller <b>102</b> is illustrated. In this embodiment, the method <b>500</b> may continually check <b>502</b> for the occurrence of a failure condition. If a failure has occurred, the method <b>500</b> may determine <b>504</b> a failure ID (such as an MDE or MLE number) associated with the failure condition. The method <b>500</b> may also determine <b>506</b> which host(s) <b>104</b> and storage devices <b>106</b> are associated with the failure condition. The method <b>500</b> may then locate <b>508</b>, using the failure ID, an entry <b>322</b> in the data collection and recovery table <b>302</b> that corresponds to the failure condition.

At this point, the method <b>500</b> may compare <b>510</b> the count value <b>408</b> in the entry <b>322</b> to the threshold value <b>410</b>. If the count value is not equal to the threshold value, the method <b>500</b> may execute <b>512</b> any pre-data-collection process(es) and collect <b>514</b> any data that is indicated in the entry <b>322</b>. The method <b>500</b> may also execute <b>516</b> any recovery process(es) that are indicated in the entry <b>322</b>. The method <b>500</b> may then increment <b>518</b> the count value.

On the other hand if, at the decision step <b>510</b>, the count value <b>408</b> is equal to the threshold value <b>410</b>, the method <b>500</b> may execute <b>520</b> any escalated recovery processes that are indicated in the entry <b>322</b>. This may include, for example, initiating a warmstart or failover operation. The method <b>500</b> may then reset <b>522</b> the count value <b>408</b>.

Referring to <figref idrefs="DRAWINGS">FIG. 6</figref>, another embodiment of an entry <b>322</b> (or record <b>322</b>) in the data collection and recovery table <b>302</b> is illustrated. Like the previous example, the entry <b>322</b> may include fields <b>400</b>, <b>402</b>, <b>404</b>, <b>406</b> associated with a failure ID, pre-data-collection process(es), data collection process(es), and recovery process(es). However, unlike the entry <b>322</b> described in <figref idrefs="DRAWINGS">FIG. 4</figref>, the entry <b>322</b> may store multiple count values <b>408</b><i>a</i>, <b>408</b><i>b</i>, multiple threshold values <b>410</b><i>a</i>, <b>410</b><i>b</i>, and multiple fields <b>412</b><i>a</i>, <b>412</b><i>b </i>for different escalated recovery processes. The first count value <b>408</b><i>a </i>may be incremented or modified each time the failure associated with the entry <b>322</b> occurs. When the first count value <b>408</b><i>a </i>reaches a first threshold value <b>410</b><i>a</i>, a first escalated recovery process (as indicated in the field <b>412</b><i>a</i>), such as a warmstart, may be executed. Similarly, a second count value <b>408</b><i>b </i>may be incremented or modified each time the first escalated recovery process <b>412</b><i>a </i>is executed. When the second count value <b>408</b><i>b </i>reaches a second threshold value <b>410</b><i>b</i>, a second escalated recovery process (as indicated in the field <b>412</b><i>b</i>), such as a failover operation, may be executed. This embodiment allows more drastic measures to be taken if a failure condition continues to reoccur.

The flowcharts and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.