Automated integrated high availability of the in-memory database cache and the backend enterprise database

PRIORITY CLAIM

0001This application claims benefit of U.S. Provisional Application No. 61/243,874, filed Sep. 18, 2009, the entire contents of which is hereby incorporated by reference as if fully set forth herein, under 35 U.S.C. §119(e). This application claims benefit under 35 U.S.C. §120 as a Continuation-in-part of application Ser. No. 12/562,928, filed Sep. 18, 2009, entitled “Distributed Consistent Grid Of In-Memory Database Caches,” by Hoang, et al, the entire contents of which application is hereby incorporated by reference as if fully set forth herein. The applicant(s) hereby rescind any disclaimer of claim scope in the parent application(s) or the prosecution history thereof and advise the USPTO that the claims in this application may be broader than any claim in the parent application(s). This application is also related to U.S. patent application Ser. No. 12/030,113, filed Feb. 12, 2008, entitled “Database System with Dynamic Database Caching,” by Hoang et al.; and U.S. patent application Ser. No. 12/030,094, filed Feb. 12, 2008, entitled “DATABASE SYSTEM WITH ACTIVE AND STANDBY NODES,” by Aranha et al., the entire contents of each of which are hereby incorporated by reference for all purposes as if fully set forth herein.

FIELD OF THE INVENTION

0002Embodiments of the invention described herein relate generally to mid-tier databases, and, more specifically, to techniques for providing high availability for a mid-tier database.

BACKGROUND

0003The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

0004Service providers provide end users with services that involve data originating from persistent data sources. The data sources themselves are often unequipped to provide such services. Instead, end users connect via clients to applications that provide these services. Applications, in turn, interact with the data sources to provide these services.

Database and Database Servers

0005One source of data is a database. A database is a collection of logically related data and metadata. From a high-level perspective, that data and metadata is organized into logical structures, for example, according to relational and/or object-relational database constructs. Database metadata defines database objects, such as tables, object tables, views, or complex types.

0006One database implementation involves the maintenance of data representative of the database in a storage mechanism such as, for example, a persistent storage device. A database server or like process(es) provides access to the data and metadata by reading from and writing to the storage. In many cases, the data representative of the database is stored in storage-based structures that differ from the logical structure of the database—for example, in data blocks on a hard disk. Accordingly, the database server translates some of the data representative of the database into logical structures, such as tables, rows, and columns, in order to perform operations that make reference to those logical structures. When the database server has finished performing the operations, if data in any of the logical structures has changed, the database server translates the logical structures back into the storage-based structures and causes those structures to be stored in the storage.

0007Applications interact with database servers via database commands such as SQL statements. These database commands cause the database servers to perform the above mentioned database operations. These operations may include, for example, providing a subset of data to the application, modifying records in the database, searching for data that meets certain criteria, sorting data, performing an analysis of the data, and so on. In many cases, as a result of the operations, database servers return “result sets” to the applications in the form of logically structured data.

Tiers

0008The above described interactions are characterized as occurring across three levels or “tiers.” The first of these levels is known as the “client-side,” and involves the end user operating a client. For example, an end user may operate a web browser or other user interface to request online services such as booking a flight. Or, as another example, a user may operate a cell phone to request a telephone service.

0009The second level is a server-side level known as the “mid-tier,” and involves applications. Generally, a component may be described as being in the “mid-tier” if it is deployed between two components in a chain of interactions—more specifically, if it responds to requests from one component by sending a request to another component. As used herein, however, the term “mid-tier” may more specifically refer to any component that responds to requests from any other component by interacting with data originating from one or more backend data sources, such as a backend database server. For example, a website running at the mid-tier may provide a web interface for booking a flight to a user, and in response to requests from the user, request flight data from a database server. Or, as another example, cellular switching software at the mid-tier may rely on data from a database server to determine how to respond to a cell phone's request to make a call.

0010The final level is a server-side level known as the “backend,” and involves one or more data sources, such as a file system, web server, or a database server and database. For example, where the one or more backend data sources is a database system, the backend level comprises, essentially, one or more persistent databases stored at one or more persistent storage devices, any database servers that interact directly with the persistent storage device to provide access to logical structures stored in the one or more persistent databases, and any components that manage the persistent storage devices or the backend database servers. For example, the backend may comprise a clustered database system that is managed by a cluster manager.

0011Because backend database systems are a particularly common form of backend data, the remainder of this application shall refer extensively to the use of backend database systems. However, the techniques described herein are just as applicable to any other source of data stored persistently at the backend level.

0012In some embodiments, the computing devices implementing components at each level are physically distinct from the computing devices at each other level. For example, database servers and mid-tier applications are often implemented at different computing devices. However, in other embodiments, physical separation is not strictly maintained.

Mid-Tier Database

0013For some applications, it is advantageous to implement one or more database systems on computing devices that are in the mid-tier—in other words, computing devices that implement mid-tier applications as opposed to backend components. Mid-tier databases provide mid-tier applications with faster access to data by bringing the data closer to the applications. Moreover, in some embodiments a mid-tier database may be stored entirely within a memory that is faster than the storage mechanism used predominately for the backend databases, further increasing the speed with which mid-tier applications can access data. For example, data representative of a mid-tier database may be stored in a volatile, random access memory. However, benefits may be realized from using a mid-tier database even without the mid-tier database being stored in such a memory.

Mid-Tier Caching

0014In an embodiment, only a small subset of the data required by the mid-tier applications can be maintained in the mid-tier level. This may be true for a variety of reasons. For example, the amount of memory or storage available at the mid-tier level may be much smaller than the amount of data needed by the mid-tier application. For this reason, the mid-tier applications rely on a backend database. Unfortunately, access to a backend database incurs various “roundtrip” communication costs resulting from, for example, transmission overhead, limited transmission speeds, and/or limited transmission bandwidth. Thus, mid-tier applications sometimes cache certain objects from the backend database at the mid-tier level.

0015One technique for implementing a mid-tier cache is to utilize a mid-tier database as a cache of certain critical data in the backend database. The subset of data stored in the mid-tier database may be selected because it is most frequently accessed or most critical to the mid-tier applications. For other data, the mid-tier applications access the backend database. In this manner, a mid-tier database can be used to cache frequently accessed data from the backend database so as to avoid requiring the mid-tier applications to constantly incur round-trips to the backend database server.

0016In an embodiment, a cache agent is responsible for controlling which elements of the database are cached in the mid-tier database. For example, the cache agent implements a policy that calls for loading cached copies or “instances” of backend elements into the mid-tier database dynamically when their corresponding elements are requested by the mid-tier applications. The cache agent also manages the size of the mid-tier database by ensuring that older and/or less used cache instances are removed from the mid-tier database as needed. An example of such an embodiment is described in “Database System with Dynamic Database Caching.”

Directly-Linked Mid-Tier Applications

0017Like backend database systems, mid-tier database systems may feature a database server to which mid-tier applications may connect to access data in the mid-tier database. However, in some embodiments, mid-tier applications execute in the same memory space that stores the mid-tier database, and thus may read and write directly to the mid-tier database. Thus, these mid-tier applications forego the need to establish a communication link with a database server, further increasing the speed with which the mid-tier applications may access data in the mid-tier database. Such mid-tier applications are hereinafter referred to as “directly-linked” applications. A vendor may simplify development of directly-linked mid-tier applications for third parties with the provision of pre-compiled libraries of instructions that perform many of the same functions as a database server, as well as an application programming interface (API) for utilizing those instructions. For convenience, any functions described herein as being performed by a database server at the mid-tier level, may also be understood as being performed by an executing instance of such instructions, invoked by a directly-linked application.

0018Note that the internal functioning of a mid-tier database server is different than a backend database server, and thus the two types of database servers are not to be confused. For example, mid-tier database servers may be optimized to compile queries differently than backend database servers, owing to factors such as differing assumptions about the speed at which data may be retrieved from memory, as well as consideration of the caching relationship between the backend database and the mid-tier database.

Reliability of Mid-Tier Databases

0019One challenge in deploying mid-tier databases is dealing with failures of the mid-tier databases. One approach is to deploy multiple copies of the mid-tier database. Each copy of the mid-tier database is monitored by one or more threads or processes that propagate transactions to other copies of the mid-tier database. (Note that, for simplification, the terms thread and process shall subsequently be used interchangeably throughout this application). The mid-tier applications are made aware of these multiple copies, and when one of the copies fails, the mid-tier applications are configured to interact instead with another copy of the mid-tier database.

0020One such scheme is described in “DATABASE SYSTEM WITH ACTIVE AND STANDBY NODES.” An active copy and a standby copy of a mid-tier database are maintained. Both copies may be readable. However, applications may only perform write operations at the active copy. Transactions at the active copy are propagated to the standby copy. When the active copy fails, a database administrator sends a command to a process linked to the standby copy to cause the standby copy to become the active copy. The administrator also sends commands to restart any directly linked applications, so that they are linked to the new active copy. Applications and clients can then be configured to interact with the new active copy instead of the failed active copy.

0021A drawback to deploying multiple copies of a mid-tier database is that such deployments are administratively complex. Administrators must furthermore constantly monitor the mid-tier databases to identify failures and react accordingly. Administrators can develop script-based processes to automate the monitoring of a database and failover to a standby database, but development and maintenance of such scripts can be expensive and inconvenient. For example, the scripts must be changed any time the configuration of the database changes. Also, the administrator must configure processes to monitor any directly-linked applications, and, in the event of a directly linked application failing, restart the directly-linked application with an appropriate configuration. Moreover, each non-directly-linked application must be made aware of the multiple copies and configured with logic for detecting and reacting to a failed database. These and other obstacles impede and in many cases prevent the realization of many of the benefits of using mid-tier databases.

BRIEF DESCRIPTION OF THE DRAWINGS

0022The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

0023<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a system <b>100</b> in which the techniques described herein may be practiced;

0024<figref idref="DRAWINGS">FIG. 2</figref> is a block diagram <b>200</b> illustrating various components of a mid-tier database cluster <b>140</b> in greater detail, along with various interactions between the components of system <b>100</b> at large;

0025<figref idref="DRAWINGS">FIG. 3</figref> is a flow chart <b>300</b> illustrating an example method of configuring a mid-tier database cluster;

0026<figref idref="DRAWINGS">FIG. 4</figref> is a flow chart <b>400</b> illustrating an example method for recovering from a failure at an active node;

0027<figref idref="DRAWINGS">FIG. 5</figref> is a flow chart <b>500</b> illustrating an example method for recovering from a failure at a standby node;

0028<figref idref="DRAWINGS">FIG. 6</figref> is a flow chart <b>600</b> illustrating an example method for recovering from the failures of both nodes of an active-standby mid-tier database pair; and

0029<figref idref="DRAWINGS">FIG. 7</figref> is block diagram of a computer system upon which embodiments of the invention may be implemented.

DETAILED DESCRIPTION

0030In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

0031Embodiments are described herein according to the following outline:

00321.0. General Overview

00332.0. Structural Overview <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0034">2.1. Tiers</li><li id="ul0002-0002" num="0035">2.2. Mid-Tier Database Hosts</li></ul></li></ul>

00363.0. Functional Overview <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0037">3.1. Mid-Tier Cluster Configuration and Monitoring</li><li id="ul0004-0002" num="0038">3.2. Detecting Node Failure</li><li id="ul0004-0003" num="0039">3.3. Recovering from Failure of an Active Node</li><li id="ul0004-0004" num="0040">3.4. Recovering from Failure of a Standby Node</li><li id="ul0004-0005" num="0041">3.5. Recovering from Failure of both an Active and Standby Node</li></ul></li></ul>

00424.0. Additional Implementation Examples <ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0000"><ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0043">4.1. Recovery for Read-Only Subscribers</li><li id="ul0006-0002" num="0044">4.2. Configuration Files</li></ul></li></ul>

00455.0. Implementation Mechanism—Hardware Overview

00466.0. Extensions and Alternatives

1.0. General Overview

0047Approaches, techniques, and mechanisms are disclosed for maintaining a cluster of mid-tier databases. A cluster manager is configured to manage a plurality of copies of a mid-tier database as a mid-tier database cluster. The cluster manager manages the cluster by, among other actions, monitoring each copy of the mid-tier database for failures, as well as quickly recovering from such failures using a variety of recovery strategies.

0048According to an embodiment, the cluster manager may also concurrently manage a backend database cluster to which at least a subset of the data in the mid-tier database is kept synchronized. Because the same cluster management software may be used to manage both the mid-tier database and the backend database, deployment of the mid-tier database cluster is simplified.

0049According to an embodiment, a mid-tier database cluster comprises at least two mid-tier database nodes. At any given time, a first of these mid-tier database nodes is designated as an active node that accepts updates from applications, while a second of these mid-tier database nodes is designated as a hot standby node whose database is always up-to-date with the database of the active node. The standby node is capable of becoming the active node almost instantaneously in the event of failure at the active node. Multiple of such mid-tier database clusters may reside in the mid-tier, any or all of which may be connected to a single backend database cluster.

0050In an embodiment, each mid-tier database node (or an agent associated therewith) is initialized by accessing common configuration information defining the mid-tier database cluster. The configuration information may specify, for example, one or more hosts that are available to implement mid-tier database nodes, one or more synchronization and/or backup schemes for the clustered data, one or more applications to execute in association with various nodes of the mid-tier database cluster, virtual address information, and so on. Based on this information, at least one of the nodes in the database cluster is configured to replicate changes in a copy of an element in the mid-tier database to a copy of the element in a backend database. Further based on this information, each node (or its associated agent) configures a cluster manager to: 1) monitor status information for the node; and 2) in response to detecting a failure of the node's copy of the mid-tier database, trigger one or more recovery processes in the mid-tier database cluster. Among other recovery responses, the recovery processes may recover a copy of the mid-tier database at the failed node or a spare node. The recovery processes may also cause a new node to provide to a client services involving transactions in the mid-tier database, which services were, at the time of the failure, being provided by the failed node. According to an embodiment, each copy of the mid-tier database may be, for example, a memory resident database. Thus, a node reloads the entire database into memory to recover a copy of the database.

0051According to an embodiment, the cluster manager is configured to automatically respond to a failure at an active node by causing, nearly instantaneously, a standby node to assume the role of the active node. Furthermore, the cluster manager causes a copy of the database to be recovered. The recovery may occur at either the failed node, once it returns to service, or a spare node. The recovery node may recover the copy by, for example, loading a new copy of the database into memory based on recovery information communicated by the new active node and/or stored in a shared or local storage. The recovery node assumes the role of standby node and as such begins replicating transactions performed by the new active node.

0052According to an embodiment, the cluster manager is configured to automatically respond to a failure at a standby node by causing a copy of the database to be recovered at a recovery node. The recovery node may be either the standby node, once it returns to service, or an spare node. The recovery node assumes the role of standby node.

0053According to an embodiment, the cluster manager is configured to automatically respond to concurrent failures of the active node and the standby node by causing a copy of the database to be recovered at two recovery nodes. The recovery node may recover the copy by, for example, loading a new copy of the database into memory based on recovery information communicated by the backend database system and/or stored in a shared or local storage. The recovery nodes may include the failed nodes, once they return to service, and/or spare nodes. One recovery node is assigned as an active node, while another is assigned as a standby node.

0054According to an embodiment, one or more applications executing at the same host as a node of the mid-tier database cluster are designated as dependent upon the node, in that they rely upon data in the node's copy of the mid-tier database in order to properly provide clients with services. In response to a failure of the node's copy of the mid-tier database, the one or more applications are automatically relocated to the node that assumes the role of the failed node. For example, one or more applications may be directly linked to an active node. When the active node fails, the one or more applications are stopped at the host of the failed active node, and started at the host of the new active node. According to an embodiment, these dependencies are described in the cluster configuration information. Based on the described dependencies, the node or its associated agent configures the cluster manager to cause this application relocation in response to a database failure.

0055According to an embodiment, each node's host is assigned a virtual address. When a node fails, if the node's failed database is recovered at a spare node, the cluster manager causes the node's virtual address to be reassigned to the host of the spare node. Applications may be configured to route database commands through each node's virtual address as opposed to through the physical addresses of each node's respective host. In this manner, applications may be transparently redirected to replacement nodes, without having to implement special logic to detect the existence of the replacement nodes. Moreover, clients of directly linked applications may communicate with those applications via the virtual address. Since the cluster manager will also automatically start the applications at the host of the replacement node, the reassignment of the virtual address to the replacement node ensures that the clients will automatically be redirected to working versions of the applications in the event of database failure. In other aspects, the invention encompasses a computer apparatus and a computer-readable medium configured to carry out the foregoing steps.

2.0. Structural Overview

0056<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a system <b>100</b> in which the techniques described herein may be practiced, according to an embodiment of the invention. <figref idref="DRAWINGS">FIG. 1</figref> is but one example of a system capable of implementing the techniques described herein. Other systems may include more or fewer components in different arrangements. Moreover, the distribution of work between components may vary from system to system.

2.1. Tiers

0057System <b>100</b> comprises a backend <b>110</b>, mid-tier <b>150</b>, and client-side <b>190</b>. Backend <b>110</b> comprises a shared storage space <b>115</b> and a database system <b>120</b>. Mid-tier <b>130</b> comprises a mid-tier database cluster <b>140</b> and multiple applications <b>181</b>-<b>185</b>. Client-side comprises clients <b>191</b>-<b>195</b>. A cluster manager <b>129</b> is implemented by components in both mid-tier <b>130</b> and backend <b>110</b>.

Client-Side

0058Clients <b>191</b>-<b>195</b> may be, for example, web browsers, desktop applications, mobile applications, and so on. Clients <b>191</b>-<b>195</b> request services from mid-tier applications <b>181</b>-<b>185</b>.

0059Mid-Tier

0060Mid-tier applications <b>181</b>-<b>185</b> may be, for example, web applications, servers, service providers, and so forth. In an embodiment, mid-tier applications <b>181</b>-<b>185</b> are any applications that rely upon data originating from backend database system <b>120</b> but were not provided by the vendor of backend database system <b>120</b>. However, in other embodiments, some or all of mid-tier applications <b>181</b>-<b>185</b> may be vendor-provided. To provide the services requested by clients <b>191</b>-<b>195</b>, mid-tier applications <b>181</b>-<b>185</b> rely on data that is persistently maintained at database system <b>120</b>, some of which data may be cached for more efficient access at mid-tier database cluster <b>140</b>. Mid-tier applications <b>181</b>-<b>185</b> may further rely upon data that is found exclusively at mid-tier database cluster <b>140</b>.

0061Mid-tier database cluster <b>140</b> comprises nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b>. Nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> are each, in essence, self-contained mid-tier database systems, comprising both a data store with one or more databases as well as implementations of database access routines for accessing the data stored therein. Nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> are each capable of responding to database commands from applications <b>181</b>-<b>185</b> by retrieving and/or manipulating the data stored in their respective databases. Nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> may be implemented on separate hosts. Or, one or more of nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> may be implemented on a shared host. Furthermore, some or all of applications <b>181</b>-<b>185</b> may be implemented on the same hosts as nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b>.

0062Nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> are all considered part of the same mid-tier database cluster because they each store a copy of a particular data set. Nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> maintain the consistency of the particular data set through any of a variety of synchronization techniques, including replication and/or propagation. Such techniques are described in, for example, “DATABASE SYSTEM WITH ACTIVE AND STANDBY NODES.”

0063At least one of nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> maintains consistency between this particular data set and the database system <b>120</b>. For example, the data in mid-tier database cluster <b>140</b> may include a cache of a subset of data found in database system <b>120</b>, kept consistent with database system <b>120</b> by node <b>150</b>. Again, consistency may be maintained through any of a variety of techniques, including techniques such as described in “Distributed Consistent Grid Of In-Memory Database Caches.” However, in some embodiments the particular data set may or may not have originated from database system <b>120</b>. Furthermore, nodes <b>150</b>, <b>160</b>, <b>141</b>, <b>142</b>, and <b>143</b> may store data sets other than the particular data set.

0064With respect to the particular data set, node <b>150</b> is considered the “active node.” The mid-tier applications thus access the particular data set primarily through node <b>150</b>. In fact, write access to the particular data set is provided exclusively through active node <b>150</b>. Meanwhile, node <b>160</b> is designated as a “standby node.” Node <b>160</b> is kept consistent at least asynchronously with transactions at node <b>150</b>, so that if node <b>150</b> were to fail, node <b>160</b> may take the place of node <b>150</b> with minimal recovery efforts. For example, changes to node <b>150</b> may be constantly replicated at node <b>160</b> based on transaction logs communicated by node <b>150</b> to node <b>160</b>. In an embodiment, mid-tier database cluster <b>140</b> may feature additional standby nodes, or no standby node at all. Moreover, mid-tier database cluster <b>140</b> may feature multiple active nodes, kept consistent through any suitable means.

0065Nodes <b>141</b>-<b>143</b> are “subscriber nodes,” and may be used to provide read-only access to the particular data set (e.g. for application <b>184</b>). In an embodiment, changes at active node <b>150</b> are propagated or replicated to these nodes less frequently than to standby node <b>160</b>. For example, changes may be replicated from standby node <b>160</b> instead of active node <b>150</b>, thereby introducing additional delay in synchronizing subscribed nodes <b>141</b>-<b>143</b>. Or, transaction logs may be buffered and communicated less frequently to subscriber nodes <b>141</b>-<b>143</b>. Mid-tier database cluster <b>140</b> may feature any number of subscriber nodes, including none at all.

0066Mid-tier database cluster <b>140</b> further comprises spare node <b>170</b>. Spare node <b>170</b> does not provide the particular data set to applications <b>181</b>-<b>185</b>, and in fact may not be involved in any replication or propagation scheme involving the particular data set. Rather, spare node <b>170</b> may be utilized for recovering from failures at any of nodes <b>141</b>-<b>143</b>, <b>150</b>, or <b>160</b>. Specifically, spare node <b>170</b> may replace any of nodes <b>141</b>-<b>143</b>, <b>150</b>, or <b>160</b>, should any of nodes <b>141</b>-<b>143</b>, <b>150</b>, or <b>160</b> be unable to recover from a failure. In an embodiment, any number of spare nodes may be specified for mid-tier database cluster <b>140</b>, including none at all.

0067In an embodiment, mid-tier <b>130</b> may comprise multiple mid-tier database clusters <b>140</b>. Each mid-tier database cluster <b>140</b> is associated with a different data set. Some or all of the mid-tier nodes may store data for multiple sets of clustered data, and therefore belong to multiple mid-tier database clusters <b>140</b>. For example, node <b>150</b> may be an active node for one set of data, and a standby node for another set of data.

Backend

0068Database system <b>120</b> comprises a database server <b>122</b>, a database server <b>123</b>, and a database <b>124</b>, each being implemented by one or more shared or separate host devices. Database <b>124</b> is a collection of logically related data stored at, for example, a pool of one or more persistent storage devices such as hard disks or storage arrays. Database <b>124</b> may be, for example, a relational database. In an embodiment, database <b>124</b> is distinguished from databases in mid-tier cluster <b>140</b> in that it is stored in media that performs read and/or write operations more slowly than the media upon which databases in mid-tier cluster <b>140</b> are stored. In an embodiment, database <b>124</b> is distinguished from databases in mid-tier cluster <b>140</b> in that mid-tier cluster <b>140</b> stores only a subset of the data found in database <b>124</b>.

0069In an embodiment, database servers <b>122</b> and <b>123</b> directly provide access to database <b>124</b> for one or more applications. For example, as depicted, database server <b>122</b> directly provides access to database <b>124</b> for application <b>185</b>. Specifically, database server <b>122</b> responds to database commands from application <b>185</b> by performing operations for storing, retrieving, and manipulating data stored in database <b>124</b>. Database servers <b>122</b> and <b>123</b> further provide indirect access to database <b>124</b> for one or more applications by providing data to be cached in mid-tier database cluster <b>140</b>. For example, as depicted, database server <b>122</b> provides data to be cached at mid-tier database cluster <b>140</b>. The cached data is then accessed by applications <b>181</b>-<b>184</b>.

0070In an embodiment, the nodes of mid-tier database cluster <b>140</b> are distinguished from backend database system <b>120</b> because they are implemented on hosts at which mid-tier applications such as applications <b>181</b>-<b>185</b> execute, whereas no mid-tier applications execute on any host that implements database system <b>120</b>. However, this distinction need not be strictly maintained.

0071While the embodiment depicted in <figref idref="DRAWINGS">FIG. 1</figref> consists of only a single backend database system with two backend database servers and a backend database, other embodiments permit caching of data from any number of backend database systems with any number of database servers and databases.

Cluster Manager

0072Cluster manager <b>129</b> manages database system <b>120</b> as a backend database cluster. That is to say, cluster manager <b>129</b> monitors database <b>124</b> and database server <b>122</b> and <b>123</b> to ensure that they operate collectively in a consistent manner. Among other functions, cluster manager <b>129</b> may employ various techniques for recovering from failures at any of the components in database system <b>120</b>. Cluster manager <b>129</b> may be implemented by one or more processes executing on any of a number of host servers in the backend, including the hosts executing database servers <b>122</b> and <b>123</b>, as well as a dedicated host server. In an embodiment, each machine hosting a component of database system <b>120</b> executes an instance of cluster management logic, and these instances communicate with each other in a coordinated fashion to collectively implement cluster manager <b>129</b>. An example cluster manager is described in “Oracle Clusterware: Administration and Deployment Guide” by Oracle Corporation, October 2008, available at the time of writing at http://download.oracle.com/docs/cd/B28359_—01/rac.111/b28255.pdf, the entire contents of which are hereinafter incorporated by reference as if set forth in their entirety.

0073Cluster manager <b>129</b> relies upon shared storage <b>115</b> for storage of various information necessary to manage database system <b>120</b>, such as data describing cluster configuration settings, cluster resources, system status, and/or ownership mappings. In an embodiment, shared storage <b>115</b> is further utilized by mid-tier nodes <b>141</b>-<b>143</b>, <b>150</b>, and <b>160</b> to store recovery information, such as snapshots and transaction logs. Shared storage <b>115</b> may be implemented by one or more persistent storage devices. Although depicted as residing in backend <b>110</b>, some or all of shared storage <b>115</b> may be provided by devices in mid-tier <b>130</b>.

0074The functionality of cluster manager <b>129</b> is extended to automatically manage mid-tier database cluster <b>140</b>. Among other functions, cluster manager <b>129</b> starts, monitors, and/or reacts to failures at the various nodes of mid-tier cluster <b>140</b>, so as to ensure that the data in mid-tier cluster <b>140</b> remains both consistent and highly available. In an embodiment, cluster manager <b>129</b> also provides various availability services to some or all of applications <b>181</b>-<b>185</b>. Examples of interactions between cluster manager <b>129</b> and the various components of mid-tier <b>130</b> are described more fully throughout the remaining sections of this disclosure.

0075In an embodiment, one or more of the machines hosting a node <b>141</b>-<b>143</b>, <b>150</b>, <b>160</b>, or <b>170</b> executes an instance of the cluster management logic, thereby participating in the implementation of cluster manager <b>129</b>. In an embodiment, cluster manager <b>129</b> is implemented exclusively in the backend.

2.2. Mid-Tier Database Hosts

0076<figref idref="DRAWINGS">FIG. 2</figref> is a block diagram <b>200</b> illustrating various components of mid-tier database cluster <b>140</b> in greater detail, along with various interactions between the components of system <b>100</b> at large. Again, <figref idref="DRAWINGS">FIG. 2</figref> provides but one example of a system capable of implementing the techniques described herein. Other systems may include more or fewer components in different arrangements. Moreover, the distribution of work between components may vary from system to system.

0077As depicted in <figref idref="DRAWINGS">FIG. 2</figref>, mid-tier database nodes <b>150</b>, <b>160</b>, and <b>170</b> of <figref idref="DRAWINGS">FIG. 1</figref> are implemented by hosts <b>250</b>, <b>260</b>, and <b>270</b>, respectively. Hosts <b>250</b>, <b>260</b>, and <b>270</b> are deployed in mid-tier <b>130</b>, and are distinct from any hosts used to implement backend database system <b>120</b>. Note that, for simplification, the hosts used to implement backend database system <b>120</b> and various other components of <figref idref="DRAWINGS">FIG. 2</figref> are not depicted in <figref idref="DRAWINGS">FIG. 2</figref>. Hosts <b>250</b>, <b>260</b>, and <b>270</b> also comprise a number of other components depicted in <figref idref="DRAWINGS">FIG. 2</figref>, including application logic <b>282</b> and <b>283</b> for executing applications <b>182</b> and <b>183</b>, a virtual address <b>231</b>, cluster configuration data <b>258</b>, and cluster agents <b>259</b>, <b>269</b>, and <b>279</b>.

0078<figref idref="DRAWINGS">FIG. 2</figref> further depicts various components already described with respect to <figref idref="DRAWINGS">FIG. 1</figref>, including client <b>192</b>, application <b>181</b>, shared storage <b>115</b>, cluster manager <b>129</b>, and subscriber nodes <b>145</b>, each of which may be implemented at any one or more hosts, including hosts <b>250</b>, <b>260</b>, and <b>270</b> as well as other hosts not depicted.

0079Mid-tier database nodes <b>150</b> and <b>160</b> each respectively comprise the following components: a daemon <b>251</b> or <b>261</b>, a database server <b>252</b> or <b>262</b>, a mid-tier database <b>254</b> or <b>264</b>, and sync agent(s) <b>256</b> or <b>266</b>. Nodes <b>150</b> and <b>160</b> may further comprise any number of additional components as needed. Each component is implemented by one or more processes at each node's respective host <b>250</b> or <b>260</b>. Although spare node <b>170</b> has the potential to implement the above listed components in response to failures at nodes <b>150</b> or <b>160</b>, spare node <b>170</b> may or may not comprise such components while nodes <b>150</b> and <b>160</b> are operating as active and standby nodes, respectively. Because such components of node <b>170</b>, even if running while nodes <b>150</b> and <b>160</b> function as active and standby nodes, are nonetheless not actively participating in the replication of the particular data set involved in mid-tier database cluster <b>140</b>, the components are not depicted in <figref idref="DRAWINGS">FIG. 2</figref>.

0080For convenience, nodes <b>150</b>, <b>160</b>, and <b>170</b> are described herein as performing tasks that are actually performed by subcomponents such as daemons <b>251</b> and <b>261</b>, sync agents <b>256</b> and <b>266</b>, as well as associated components such as cluster agents <b>259</b> and <b>269</b>. Thus, any task described or implied as being performed by a node should be understood as being performed by any one or more processes executing at or in association with the node. Similarly, tasks described or implied as being performed by hosts <b>250</b>, <b>260</b>, or <b>270</b>, should be understood as being performed by any one or more processes executing at hosts <b>250</b>, <b>260</b>, or <b>270</b>.

Mid-Tier Databases

0081Mid-tier databases <b>254</b> and <b>264</b> are collections of data stored in mid-tier <b>130</b>. In an embodiment, mid-tier databases <b>254</b> and <b>264</b> each comprise a plurality of tables stored entirely within a volatile memory at their nodes' respective host, such as a Random Access Memory (RAM). Mid-tier databases <b>254</b> and <b>264</b> may further comprise additional components stored in a volatile memory, including indexes and temporary space. Other components of mid-tier databases <b>254</b> and <b>264</b> may include transaction logs and recovery data, such as snapshots, stored in a non-volatile memory such as a flash disk or hard disk. In an embodiment, additional or even all components of mid-tier databases <b>254</b> and <b>264</b> are stored in a non-volatile memory.

0082Mid-tier database nodes <b>150</b>, <b>160</b>, <b>170</b>, and <b>141</b>-<b>143</b> may each additionally comprise other mid-tier databases, some of which may be grouped into other mid-tier database clusters, and some of which may not be involved in any cluster.

0083For simplification, the techniques described herein are described primarily with respect to an embodiment where the set of clustered data encompasses an entire mid-tier database. However it should be understood that certain actions or steps involving a “mid-tier database” (or copies thereof) may in some embodiments involve only a particular subset of data, within a mid-tier database, that has been defined as belonging to a particular mid-tier cluster.

Mid-Tier Database Servers

0084Mid-tier database servers <b>252</b> and <b>262</b> are executing instances of logic for responding to database commands by performing operations for storing, retrieving, and manipulating data stored in mid-tier databases <b>154</b> and <b>164</b>, respectively. In an embodiment, a vendor-provided server process invokes database servers <b>154</b> and <b>164</b> to handle database commands received from application <b>181</b> over a communication channel (e.g. SQL statements transmitted via TCP/IP over an open socket to a database server port).

0085In an embodiment, one or more of database servers <b>152</b> and <b>162</b> may be temporary instances of database access routines, instantiated as needed for single database transactions. In such embodiments, database servers <b>152</b> and <b>162</b> are instantiated via calls to a database library API by a directly-linked application. Note that, because each database application may invoke its own database server thread, multiple database server threads may run concurrently. In an embodiment, database servers <b>152</b> and <b>162</b> may be invoked at the same time both by directly-linked applications, as well as by a background database server process responding to statements over a communication channel.

0086For example, mid-tier database <b>254</b> may be resident within a memory that is directly accessible to applications <b>182</b> and <b>183</b>. Applications <b>182</b> and <b>183</b> may therefore read data from mid-tier database <b>254</b> directly, without having to send a query over a communication channel. To simplify the process of applications <b>182</b> and <b>183</b> reading mid-tier database <b>254</b> directly, a vendor may supply a code library of database access routines. Applications <b>182</b> and <b>183</b> may thus read data from mid-tier database <b>254</b> by making calls via the database library API to execute the provided routines. These calls result in, essentially, applications <b>182</b> and <b>183</b> invoking database server threads, which for simplicity are hereinafter referred to as database servers.

Synchronization Agents

0087Sync agents <b>256</b> and <b>266</b> are each one or more agents responsible for synchronizing data between mid-tier database nodes and/or backend database system <b>120</b>. Synchronization may be accomplished by any suitable synchronous or asynchronous techniques. For simplification, this application shall refer to any process of synchronizing changes from one database or database copy to another as the replication or propagation of changes. However, for the purposes of this disclosure, “replication,” “propagation,” and like terms should all be understood to equally refer to any technique for ensuring that a change in one data store is also made in another data store.

0088Sync agents <b>256</b> and <b>266</b> further include any number of components to assist in synchronization. For example, sync agents <b>256</b> and <b>266</b> may include one or more components that function as replication agents that replicate transactions from active node <b>150</b> to standby node <b>160</b>. The replication agents may further replicate the transactions from standby node <b>160</b> to subscriber nodes <b>145</b>. As another example, sync agents <b>256</b> and <b>266</b> may further include one or more components that function as propagation agents, in that they propagate changes back and forth between mid-tier cluster <b>140</b> and database system <b>120</b>. As yet another example, sync agents <b>256</b> and <b>266</b> may include one or more components that function as mid-tier caching agents. Sync agents <b>256</b> and <b>266</b> may further be responsible for backing up recovery information to shared storage <b>115</b>. For example, sync agents <b>256</b> and <b>266</b> may periodically record transaction logs and/or snapshots to shared storage <b>115</b> and/or persistent storage at one or more of hosts <b>250</b>, <b>260</b>, and <b>270</b>.

Daemons

0089Daemons <b>251</b> and <b>261</b> are each one or more processes responsible for various resource management tasks at their respective nodes. Functions performed by daemons <b>251</b> and <b>261</b> may include, for example, maintaining and reporting statistics regarding data usage and connections at mid-tier database <b>254</b> or <b>264</b>, managing shared memory access and policies, and starting and/or recovering various components of their respective mid-tier nodes, including one or more of mid-tier databases <b>254</b> and <b>264</b>, database servers <b>252</b> and <b>262</b>, and sync agents <b>256</b> and <b>266</b>.

0090For example, daemons <b>251</b> and <b>261</b> may each be capable of launching processes for various components of mid-tier node <b>150</b> or <b>160</b>, respectively, and may do so in response to the daemon being started or in response to the daemon receiving various commands over an open socket or command-line interface.

0091In an embodiment, daemons <b>251</b> and <b>261</b> may also be responsible for monitoring some or all of these various components for failures. Daemons <b>251</b> and <b>261</b> may report these failures to other components, or may react to these failures by re-launching any appropriate processes and, if necessary, recovering their respective mid-tier databases. However, in other embodiments, some or all of this monitoring is performed by other components, such as cluster agent <b>259</b> or cluster manager <b>129</b>, which in turn instruct daemon <b>251</b> or <b>261</b> to perform any appropriate recovery tasks.

Applications

0092Mid-tier host <b>250</b> executes mid-tier applications <b>182</b> and <b>183</b>. Applications <b>182</b> and <b>183</b> are executing instances of application logic <b>282</b> and <b>283</b>, respectively. Application logic <b>282</b> and <b>283</b> resides at or is readily accessible to host <b>250</b> in any suitable form, including executable code and/or hardware. To facilitate various functionalities described herein, copies of application logic <b>282</b> and <b>283</b> also reside at or are readily accessible to hosts <b>260</b> and <b>270</b>. Mid-tier host <b>250</b> also comprises or communicates with application <b>181</b>. Mid-tier host <b>250</b> may further comprise or communicate with any number of additional applications.

0093Mid-tier applications <b>181</b>-<b>183</b> communicate with database server <b>252</b> via, for example, database commands, to access data stored in mid-tier database <b>254</b>. Applications <b>181</b>-<b>183</b> use this data, for example, to provide services requested by various of clients <b>191</b>-<b>195</b>.

0094In an embodiment, application <b>181</b> communicates with database server <b>252</b> via a communication mechanism that is slower than the communication mechanism by which applications <b>182</b> and <b>183</b> communicate with database server <b>252</b>. For example, application <b>181</b> may execute at a host other than host <b>250</b>, and may therefore need to communicate with database server <b>252</b> over a relatively slow network connection. Applications <b>182</b> and <b>183</b>, by contrast, may reside on host <b>250</b> and therefore be capable of foregoing communication via the slow network connection.

0095In an embodiment, application <b>181</b> must establish a communication link via an open socket with a persistently executing instance of database server <b>252</b>, whereas applications <b>182</b> and <b>183</b> are capable of communicating with database server <b>252</b> without having to open any sockets. For example, applications <b>182</b> and <b>183</b> may communicate with database <b>252</b> via in-memory API calls. In fact, applications <b>182</b> and <b>183</b> may instantiate their own objects <b>252</b> as needed to access mid-tier database <b>254</b>.

0096In an embodiment, application <b>181</b> is configured with data identifying the locations of various hosts of components of the mid-tier database cluster (e.g. the IP addresses of hosts at which the active and standby nodes may reside). Application <b>181</b> periodically monitors the nodes at these locations to determine which location corresponds to active node <b>150</b>. For example, application <b>181</b> may periodically address polling requests to a certain port at the locations of hosts <b>250</b> and <b>260</b>. Nodes <b>150</b> and <b>160</b> may receive these polling requests and respond to application <b>181</b> with status information indicating which of nodes <b>150</b> and <b>160</b> is the active node <b>150</b>. At any given time, application <b>181</b> will only address write requests to the location at which the active node resides. Client <b>192</b> may be similarly configured to poll pre-configured locations to determine which of hosts <b>150</b> or <b>160</b> currently host application <b>183</b>.

0097By contrast, by virtue of various techniques described in subsequent sections, applications <b>182</b> and <b>183</b> are guaranteed to always be executing at the same host as the active node, and therefore require no such configuration information or status polling.

0098As depicted, mid-tier hosts <b>260</b> and <b>270</b> do not execute applications <b>182</b> and <b>183</b>. However, mid-tier hosts <b>260</b> and <b>270</b> have easy access to application logic <b>282</b> and <b>283</b>, so that they may execute applications <b>182</b> and <b>183</b> in the event of a failure at host <b>250</b>. Moreover, mid-tier hosts <b>260</b> and <b>270</b> may execute any number of other applications. For example, host <b>260</b> may execute certain read-only applications designated as “standby” applications to be run in association with a standby node in mid-tier cluster <b>140</b>.

Virtual Address

0099Host <b>250</b> is assigned a virtual address <b>231</b> by which components external to host <b>250</b> may address communications to host <b>250</b> and its various components. For example, application <b>181</b> and client <b>192</b> communicate with components of host <b>250</b> by addressing communications to virtual address <b>231</b>.

0100Virtual addresses <b>231</b> is considered “virtual” in that it is automatically assigned to another host in mid-tier <b>130</b> upon the occurrence of certain events. For example, if node <b>150</b> were to fail, and its database were to be recovered either as a standby database or active database at spare host <b>270</b>, virtual address <b>231</b> would be reassigned to spare host <b>270</b>. In an embodiment, this reassignment may cause application <b>181</b> and client <b>192</b> to communicate with components of host <b>270</b> instead. The various components of host <b>270</b> may be configured to behave in a manner consistent with the various components of host <b>250</b>, so that application <b>181</b> and client <b>192</b> are unaware of the fact that they are communicating with host <b>270</b> instead of host <b>250</b>.

0101In an embodiment, virtual address <b>231</b> is a virtual Internet Protocol (IP) address. In an embodiment, virtual address <b>231</b> co-exists with one or more real IP addresses assigned to specific physical interfaces at host <b>250</b>. In contrast to virtual IP address <b>231</b>, the real IP addresses are held persistently by host <b>250</b>, in that they are not reassigned when virtual address <b>231</b> is reassigned. In fact, messages to and from virtual address <b>231</b> may be routed through these real IP addresses until certain failures occur, at which time virtual address <b>231</b> will be re-routed through the real IP addresses of another host.

0102In an embodiment, each node in mid-tier cluster <b>140</b> is associated with a different virtual address.

Cluster Agents

0103Each of hosts <b>250</b>, <b>260</b>, and <b>270</b> maintains or has access to a copy of the same cluster configuration data <b>258</b>. Cluster configuration data <b>258</b> may describe a number of aspects of mid-tier cluster <b>140</b>, including: hosts at which to implement mid-tier database nodes, the number and types of nodes to include in the cluster (e.g. active/standby nodes), the database(s) and application(s) to cluster, management access command for each clustered application, mid-tier and backend replication schemes, and so on. Configuration data <b>258</b> may be maintained locally or in a shared storage in any suitable form, including one or more files, databases, registry entries, and/or objects.

0104Based on configuration data <b>258</b>, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> and/or other cluster components take any of a number of steps to cause hosts <b>250</b>, <b>260</b>, and <b>270</b> to implement mid-tier cluster <b>140</b>. For example, cluster agents <b>259</b> and <b>269</b> may start daemons <b>251</b> and <b>261</b>, respectively, thereby initializing mid-tier nodes <b>150</b> and <b>160</b>. Cluster agents <b>259</b> and <b>269</b> may also pass configuration parameters to daemons <b>251</b> and <b>261</b> to assist daemons <b>251</b> and <b>261</b> in initializing mid-tier databases <b>254</b> and <b>264</b>. Cluster agents <b>259</b> and <b>269</b> may also start applications <b>182</b> and <b>183</b>. Cluster agents <b>259</b> and <b>269</b> may also start various sync agents <b>256</b> and <b>266</b>. Cluster agents <b>259</b> and <b>269</b> may then configure the sync agents <b>256</b> and <b>266</b> to employ one or more synchronization schemes, such as replication from active node <b>150</b> to standby node <b>160</b> and subscriber nodes <b>145</b>, as well as synchronization between active node <b>150</b> and backend database system <b>120</b>. Cluster agents <b>259</b> and <b>269</b> may further configure sync agents <b>256</b> and <b>266</b> to periodically store recovery information, such as transaction logs and/or snapshots to shared storage <b>115</b>.

0105In an embodiment, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> each communicate with cluster manager <b>129</b> to cause cluster manager <b>129</b> to manage nodes <b>150</b> and <b>160</b> and applications <b>182</b> and <b>183</b>. In an embodiment, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> accomplish this step by providing cluster manager <b>129</b> with information describing which components of mid-tier cluster <b>140</b> should be running, how to determine if said components are running correctly, and a list of one or more actions to take in response to a failure.

0106In an embodiment, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> further generate scripts that, when executed, cause hosts <b>250</b>, <b>260</b>, and/or <b>270</b> to perform various recovery steps, such as restarting components, altering replication schemes, reloading mid-tier databases into memory, acquiring a certain virtual address, and so on. Cluster agents <b>259</b>, <b>269</b>, and <b>279</b> may then instruct cluster manager <b>129</b> to cause an appropriate one or more of these scripts to be executed in response to detecting various failures.

0107In an embodiment, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> instruct cluster manager <b>129</b> to inform cluster agents <b>259</b>, <b>269</b>, and <b>279</b> of any detected failures. Based on configuration data <b>258</b>, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> then coordinate with each other to determine which recovery steps to perform and how to perform those steps.

0108In an embodiment, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> are configured to automatically launch when hosts <b>250</b>, <b>260</b>, and <b>270</b>, respectively, restart. In an embodiment, each cluster agent <b>259</b>, <b>269</b>, and <b>279</b> must be manually started. In an embodiment, only a first cluster agent need be started. The first cluster agent then causes cluster agents to be started at each of the hosts identified by configuration data <b>258</b> as belonging to mid-tier cluster <b>140</b>.

0109In an embodiment, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> are configured to communicate with each other via, for instance, a common port. Cluster agents <b>259</b>, <b>269</b>, and <b>279</b> may share state information indicating the status of each node in mid-tier cluster <b>140</b>. So, for example, a cluster agent at a previously failed node may utilize this state information to determine whether the failed node should return as an active node (in the case of no other active node existing), a standby node, or a subscriber node. However, in other embodiments, this determination is left to other components. Cluster agents <b>259</b>, <b>269</b>, and <b>279</b> may further communicate with each other to propagate configuration data <b>258</b> and share recovery scripts.

0110In an embodiment, some or all of these steps described above may be triggered instead by manual user commands, and/or scheduled jobs. In an embodiment, some or all of the steps described above may be taken by any other suitable component of hosts <b>250</b>, <b>260</b>, and <b>270</b>, including daemons <b>251</b> and <b>261</b>, based on configuration data <b>258</b>. Note that, although depicted as separate from mid-tier nodes <b>150</b>, <b>160</b>, and <b>170</b>, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> may alternatively be conceptualized as part of mid-tier nodes <b>150</b>, <b>160</b>, and <b>170</b>. In fact, cluster agents <b>259</b> and <b>269</b> may be part of daemons <b>251</b> and <b>261</b>.

0111Note that each of subscriber nodes <b>145</b> may likewise be associated with a similar cluster agent and have access to similar configuration data.

Cluster Manager

0112Cluster manager <b>129</b> manages both a backend database cluster in the form of database system <b>120</b>, as well as mid-tier database cluster <b>140</b>. For example, cluster manager <b>129</b> may be resident both at backend hosts and mid-tier hosts. As used herein, “management” of a cluster comprises at least: a) monitoring for failures in processes specified as belonging to the cluster; and b) performing prescribed recovery actions in the event of some or all of said failures.

0113Cluster manager <b>129</b> comprises one or more processes executing at any number of hosts in backend <b>110</b> or mid-tier <b>130</b>, including, potentially, hosts <b>250</b>, <b>260</b>, and <b>270</b>. For example, cluster manager <b>129</b> may actually be a collection of coordinated cluster manager components executing at each of hosts <b>250</b>, <b>260</b>, and <b>270</b>. However, cluster manager <b>129</b> need not necessarily execute at each or even any host used to implement mid-tier cluster <b>140</b>. For instance, 1custer manager <b>129</b> may be implemented at a separate server host. Cluster manager <b>129</b> may be implemented by any clustering software capable of: 1) receiving configuration information from cluster agents <b>259</b>, <b>269</b>, and <b>279</b>; 2) monitoring processes at hosts <b>250</b>, <b>260</b>, and <b>270</b> in accordance with that configuration information; and 3) performing actions described in that configuration information in response to observing failures at the monitored processes. For example, cluster manager <b>129</b> may be a collection of one or more executing instances of Oracle Clusterware.

0114Cluster manager <b>129</b> manages a variety of components in mid-tier cluster <b>140</b>. As mentioned previously, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> configure cluster manager <b>129</b> to manage these components. Cluster manager <b>129</b> maintains a registry of “resources” identifying each of the components it has been configured to manage, along with any information necessary to manage the components. The cluster-managed components may include, for example, some or all of: daemons <b>251</b> and <b>261</b>, applications <b>182</b> and <b>183</b>, database servers <b>252</b> and <b>262</b>, mid-tier databases <b>254</b> and <b>264</b>, sync agents <b>256</b> and <b>266</b>, cluster agents <b>259</b>, <b>269</b>, and <b>279</b>, and virtual address <b>231</b>. However, in an embodiment, daemons <b>251</b> and <b>261</b> or cluster agents <b>259</b> and <b>269</b> may be responsible for managing most of these components, and cluster manager <b>129</b> is configured to deal only with failures to daemons <b>251</b> and <b>261</b> or cluster agents <b>259</b> and <b>269</b>.

0115Cluster manager <b>129</b> monitors these managed components by periodically determining their statuses. Cluster manager <b>129</b> may determine the statuses of the managed components using any of a variety of means, including executing command line functions, pinging the components over an open socket, waiting for pings from the components, reading log files, analyzing system-collected statistics, and so on. In an embodiment, cluster manager <b>129</b> monitors some of the cluster-managed components only indirectly. For example, certain cluster-managed components may be directly monitored by daemons <b>251</b> and <b>261</b> or cluster agents <b>259</b> and <b>269</b>, which in turn may relay the components' statuses to cluster manager <b>129</b>.

0116In an embodiment, cluster manager <b>129</b> is configured to perform one or more prescribed actions in response to a component's failure to respond, or in response to status information that indicates that the component has failed. These actions may include, for example, causing any or all of hosts <b>250</b>, <b>260</b>, and <b>270</b> to execute scripts, terminate processes, start processes, or acquire a different virtual IP address. These actions may further include causing any or all of nodes <b>145</b>, <b>150</b>, <b>160</b>, and <b>170</b> to assume different roles and/or load a mid-tier database into memory. These actions may be accomplished by executing command line instructions directly on hosts <b>250</b>, <b>260</b>, or <b>270</b>, making one or more remote procedure calls, sending instructions to cache agents <b>259</b>, <b>269</b>, or <b>279</b>, or any other suitable means. Examples of actions that cluster manager <b>129</b> may be configured to take in order to recover from various failures are described in subsequent sections.

0117In an embodiment, each managed component may report its status as being in any of a variety of states, such as “active,” “restarting,” “suspended,” “failing,” and so on. Cluster manager <b>129</b> may be configured to take a different action in response to some or all of these states.

0118In an embodiment, cluster agents <b>259</b>, <b>269</b>, and <b>279</b> are each part of cluster manager <b>129</b>.

3.0. Functional Overview

3.1. Mid-Tier Cluster Configuration and Monitoring

0119<figref idref="DRAWINGS">FIG. 3</figref> is a flow chart <b>300</b> illustrating an example method of configuring a mid-tier database cluster, according to an embodiment of the invention.

0120At step <b>310</b>, a cluster administrator generates cluster configuration data, such as cluster configuration data <b>258</b>. For example, the administrator may manually create a cluster configuration file such as described in section 4.1. Or, as another example, the administrator may utilize a configuration tool to generate configuration data in a shared configuration database.

0121The configuration data may be stored in any suitable location. For example, the administrator may save a cluster configuration file in a shared location accessible to each host, at a host from which the administrator intends to initialize the cluster, or at each host itself.

0122At step <b>320</b>, a cluster administration utility at a first host, such as a standalone application or cluster agent <b>259</b>, accesses the cluster configuration data to determine the cluster configuration. Step <b>330</b> may be triggered in response to the cluster administration utility being started, the cluster administration utility receiving a command from an administrator, and/or the cluster administration utility detecting the presence of a configuration file in a specified location. Step <b>320</b> and other subsequent steps may also or instead be performed by other cluster components, including a mid-tier database daemon or a cluster manager itself.

0123At step <b>330</b>, the cluster administration utility communicates with cluster agents in the mid-tier database cluster to determine the state of the mid-tier database cluster, if any. For example, the cluster administration utility may need to determine whether active and standby nodes already exist. If an active node does not exist, the cluster administration utility may further need to initiate a selection process to decide which cluster agents will be responsible for initializing the active and standby nodes. The selection process may involve user input and/or may be based on a voting protocol. In an embodiment, if the cluster administration utility cannot locate a cluster agent at a host specified to be in the mid-tier database cluster, the cluster administration utility attempts to initialize a cluster agent at that host. In an embodiment, the cluster administration utility relies upon the configuration data to learn which hosts are defined to be in the cluster, as well as the manner in which the cluster administration utility should attempt to contact cluster agents at those hosts (e.g. a specified port number to which each cluster agent will listen).

0124At step <b>340</b>, the cluster administration utility causes one or more cluster agents to initialize one or more mid-tier database nodes. For example, a cluster agent at the first host may start some or all of the components of mid-tier node <b>150</b>. Information necessary to accomplish this task, such as the location(s) of scripts and executables required to start each component, are described in the configuration data read in step <b>330</b>. In an embodiment, as a result of this step, a mid-tier database is created in the first host's memory. However, in an embodiment, some or all of the various resources relied upon to implement each mid-tier database node are not actually started until later steps.

0125In an embodiment, the initialization of step <b>340</b> involves the creation of one or more replication schemes. For example, based on both the configuration data read in step <b>320</b> and the state information communicated in step <b>330</b>, the cluster administration utility may cause a first cluster agent or other cluster component at a first node to configure the first node to participate in one or more replication and/or backup schemes. For example, if there is no active node for the mid-tier database cluster, the first cluster agent may configure the first node as an active node. As another example, if the cluster administration utility receives cluster state information indicating that there is already an active node, the first cluster agent may configure the first node to be a standby node. As another example, the first cluster agent may configure the first node to periodically store snapshots, transaction logs, and other recovery information to a shared storage identified in the cluster configuration data. As another example, the first cluster agent may configure the first node to implement a mid-tier cache of a certain backend data specified in the cluster configuration data. As yet another example, if the state information for the cluster indicates that sufficient active and standby nodes already exist for the mid-tier cluster, the first cluster agent may configure the first node as a spare node. As yet another example, if the configuration data indicates that the first node is a read-only subscriber of data from an active node, the first node may be configured as a subscriber node.

0126At step <b>350</b>, the cluster administration utility optionally causes one or more hosts to acquire a virtual address. For example, the cluster configuration data may specify that each of the cluster's active and standby nodes should have a virtual address selected from the group of 10.0.0.2 and 10.0.0.3. If the first node is the active node, for example, the first cluster agent may thus cause the first host to acquire one of the specified virtual addresses. However, in some embodiments, no virtual address is required. In an embodiment, the cluster administration utility accomplishes step <b>350</b> by simply creating virtual address resources at each of the one or more hosts, which resources may subsequently be started and monitored by the cluster manager. A virtual address resource, when started, may cause its respective host to acquire a virtual address.

0127At step <b>360</b>, the first cluster administration utility configures a cluster manager, such as cluster manager <b>129</b>, to monitor for and react to failures at the various nodes of the mid-tier database cluster. The cluster agent may accomplish this step, for example, by communicating with the cluster manager to define one or more cluster resources for the cluster manager to manage. For example, the cluster administration utility may store cluster configuration information in a shared storage such as a cluster registry, which is managed and replicated by the cluster manager. The defined resources may include the first node in general, any subcomponents of the first node, and any applications associated with the first node, per step <b>360</b>. In an embodiment, the resources created for each host will vary, depending on the role assigned to the node at each host. For example, a resource may be defined for monitoring the replication of transactions from at an active node, but no such resource may defined at a spare node. In an embodiment, each resource definition describes one or more actions (e.g. an executable command) that the cluster manager may take to start, stop, and check the status of the resource. In an embodiment, some of the resource definitions may further describe dependency relationships with other resources. For instance, the definition may indicate that to start the resource, another resource must already be started.

0128Information necessary to create these resources, such as the location(s) of scripts and executables required to start each application, may be described in the configuration data read in step <b>320</b>. The cluster administration utility may further generate certain scripts necessary to implement certain resources based on the cluster configuration data. Moreover, the cluster administration utility may rely upon the state information and/or the results of the selection process in step <b>330</b> to determine which resources should be created. For example, the cluster configuration data may define a set of applications as “active” applications associated with the active node. The cluster administration utility would create resources corresponding to these applications, and associate them with the active node.

0129In an embodiment, each resource definition further describes one or more actions that the cluster manager should take in response to detecting that the resource has failed. These actions may be as simple as restarting the resource or executing a recovery script at the first host. For instance, the first cluster agent may generate one or more recovery scripts that the cluster manager should execute at the first host in response to the cluster manager detecting that the first node's sync agents have stopped responding to status checks. Or the actions taken by the cluster manager in response to a failure may be more complex, such as issuing commands to other nodes in the cluster to take the place of the first node, while at the same time attempting to restart the first node. Specific types of actions that may be taken in response to specific types of failures are discussed subsequently.

0130At step <b>370</b>, the cluster administration utility causes any unstarted applications and/or resources that are defined for the mid-tier database cluster to be started. The cluster administration utility may perform this step using any of a variety of methods, including instructing the cluster manager and/or the cluster agents to start the various applications and resources. For example, the cluster administration utility may instruct the cluster manager to start directly-linked applications <b>182</b> and <b>183</b> of <figref idref="DRAWINGS">FIG. 2</figref> by executing application logic <b>282</b> and <b>283</b>.

0131At step <b>380</b>, the cluster manager monitors for and reacts to failures at some or all of the nodes, as configured in step <b>360</b>. In an embodiment, only nodes with certain roles are monitored—for example, the active node, the standby node, and subscriber nodes. In an embodiment, all nodes are monitored.

0132In an embodiment, steps <b>310</b>-<b>380</b> are performed in other orders than that described above. Furthermore, some steps may be omitted—for instance, the cluster administration utility may not always communicate with cluster agents to determine the cluster state; rather, in some instances the cluster administration utility may make assumptions regarding the state of the cluster. Furthermore, the cluster administration utility may omit step <b>370</b> where there are no applications or unstarted resources are associated with a node, and may omit step <b>350</b> when there is no virtual address to assign to a host. In an embodiment, cluster administration utility performs some or all of the steps of <b>310</b>-<b>380</b> based on instructions from a user. In other embodiments, the cluster administration utility performs these steps without user input.

3.2. Detecting Node Failure

0133Subsequent sections of this disclosure describe recovery strategies for failures at specific types of mid-tier database nodes. The recovery strategies are applicable to any types of failures at a node, including failures of any subcomponents and/or linked applications. However, in an embodiment, the recovery strategies discussed below are employed only in response to failures from which recovery is not possible within a defined period of time. For example, one may recover from failures at many failed directly-linked applications by simply restarting the failed applications. As another example, it may be possible to recover from the failure of a sync agent <b>256</b> by simply restarting the sync agent <b>256</b> and performing a number of background recovery operations. Therefore, in an embodiment, “node failure” and like phrases refers more specifically to the failure of a component such as daemon <b>251</b>, cluster agent <b>259</b>, mid-tier database <b>254</b>, or host <b>250</b>.

0134In an embodiment, a cluster manager is configured to first attempt one or more simpler recovery strategies, such as issuing a restart command to the failed component, prior to employing any of the strategies discussed below. The strategies discussed below are performed only if the component does not restart after these one or more simpler recovery actions.

0135In an embodiment, a cluster manager detects node failure by periodically polling for status information from the node. The cluster manager determines that the node has failed when it receives no response to a poll, or where the node responds in a manner that is inconsistent with a properly functioning node (e.g. data indicating a status of “failing”).

0136In an embodiment, a mid-tier database daemon or cluster agent may be capable of detecting node failure more quickly than the cluster manager. For example, the mid-tier database daemon may detect that the mid-tier database has been corrupted. Rather than waiting for the cluster manager to learn of such a failure through polling a mid-tier database monitoring process, the daemon may “push” a message to the cluster manager indicating that a failure is occurring. This message may be pushed by the daemon directly to the cluster manager, or relayed through a cluster agent. The cluster manager may thus respond much more rapidly to node failure. In fact, depending on how well the daemon can predict a failure, the cluster manager may be able to switch active nodes before a node has fully failed.

00003.3. Recovering from Failure of an Active Node

0137<figref idref="DRAWINGS">FIG. 4</figref> is a flow chart <b>400</b> illustrating an example method for recovering from a failure at an active node, according to an embodiment of the invention. A cluster manager may be configured to perform this method by a cluster agent at an activate node, per step <b>380</b> of <figref idref="DRAWINGS">FIG. 3</figref>. While the cluster manager is described as performing the steps of flow chart <b>400</b>, the cluster manager may instead be configured to cause another component to perform any step of flow chart <b>400</b> by issuing one or more commands to that component.

0138At step <b>410</b>, the cluster manager detects that an active mid-tier database node at a first host has failed. Detection of such a failure is discussed in section 3.2. The remaining steps of flow chart <b>400</b> are performed in response to this detection.

0139At step <b>420</b>, if necessary and possible, the cluster manager stops the failed active node, as well as any applications executing at the first host that are directly linked to the failed active node.

0140At step <b>430</b>, the cluster manager configures a standby node at a second host to assume the role of active node. The standby node is accordingly hereinafter referred to as the former standby node or the new active node. Among the many consequences of this role switch, the new active node may take steps to recover any database transactions that had been performed at the failed active node but not replicated to the standby node. For example, the new active node may consult transaction logs and other recovery information stored in a shared storage or at the backend database system to identify unreplicated transactions. Further among the consequences of the role switch, the new active node may begin permitting write transactions to its copy of the mid-tier database. Yet further among the consequences of the role switch, if the failed active node had been configured to synchronize a particular set of data with a backend data source, the standby node may begin to synchronize the particular set of data in its copy of the mid-tier database with the backend data source.

0141At step <b>440</b>, the cluster manager starts, at the second host, any applications that were linked to the failed active node at the first host. These applications will become linked to new active node. In an embodiment, the cluster manager is able to accomplish this step because the application logic for these applications is accessible to both the first and second host, either as locally stored copies, or in a shared storage.

0142At step <b>450</b>, if possible, the cluster manager restarts the formerly failed node at the first host. However, in some embodiments, the cluster manager may simply wait for the failed node to restart itself and announce its availability to the cluster manager.

0143At step <b>460</b>, the cluster manager configures the formerly failed node or a spare node as a new standby node. In an embodiment, as a consequence of this step, a copy of the clustered mid-tier database will be created anew at the new standby node. For example, the cluster manager may cause the new standby node to access recovery information at the first host or shared storage to reconstruct the mid-tier database as it existed at the time of failure. Transactions performed at the new active node will then be replicated to the new standby node. In an embodiment, a previous version of the mid-tier database may still be available at the new standby node, and the standby mid-tier database may be brought up to date simply by replicating any transactions that have been performed at the new active node since the failure.

0144In an embodiment, as part of step <b>460</b>, the cluster manager may also start one or more “standby” applications at the first host. These standby applications would have been linked to the former standby node at the time the active node failed. The cluster manager will therefore also stop these applications at the former standby node.

0145At step <b>470</b>, in the event that the failed database is recovered to a spare node instead of the failed node, the cluster manager optionally reassigns to the spare host a virtual IP address that had been formerly assigned to the first host. Consequentially, any clients or applications that had been communicating with applications or database servers at the first host via the virtual IP address will be transparently redirected to the spare host.

0146In some embodiments, various steps <b>420</b>-<b>470</b> may occur in different orders or even not at all. In an embodiment, the cluster manager returns the failed node as a spare node. In an embodiment, when the failed node recovers, the cluster manager reconfigures the failed node to again be the active node.

00003.4. Recovering from Failure of a Standby Node

0147<figref idref="DRAWINGS">FIG. 5</figref> is a flow chart <b>500</b> illustrating an example method for recovering from a failure at a standby node, according to an embodiment of the invention. A cluster manager may be configured to perform this method by a cluster agent at a standby node, per step <b>380</b> of <figref idref="DRAWINGS">FIG. 3</figref>. While the cluster manager is described as performing the steps of flow chart <b>500</b>, the cluster manager may instead be configured to cause another component to perform any step of flow chart <b>500</b> by issuing one or more commands to that component.

0148At step <b>510</b>, the cluster manager detects that a standby mid-tier database node at a first host has failed. Detection of such a failure is discussed in section 3.2. The remaining steps of flow chart <b>500</b> are performed in response to this detection.

0149At step <b>520</b>, if necessary and possible, the cluster manager stops the failed standby node, as well as any applications executing at the first host that are directly linked to the failed standby node.

0150At step <b>530</b>, the cluster manager deploys a copy of the mid-tier database at a spare node on a second host. In an embodiment, the entire mid-tier database is recreated at the spare node. For example, the spare node may create its copy of the mid-tier database based on recovery information available in a shared storage. The spare node may also or instead communicate directly with an active node to obtain the data necessary to create its copy of the mid-tier database.

0151At step <b>540</b>, the cluster manager configures the spare node as a standby node. As a consequence, transactions performed at the active node will be replicated at the spare node. The spare node will hereinafter be referred to as the former spare node or the new standby node.

0152At step <b>550</b>, the cluster manager starts, at the second host, any applications that were linked to the failed standby node at the first host. These applications will become linked to new standby node. In an embodiment, the cluster manager is able to accomplish this step because the application logic for these applications is accessible to both the first and second host, either as locally stored copies, or in a shared storage.

0153At step <b>560</b>, the cluster manager optionally reassigns to the second host the virtual IP address that had been formerly assigned to the first host. Consequentially, any clients or applications that had been communicating with applications or database servers at the first host via the virtual IP address will be transparently re-routed to the new standby host. In an embodiment, the active node also addressed the former standby node using this virtual IP address, and is thus redirected automatically to the new standby node for transaction replication.

0154At step <b>570</b>, if possible, the cluster manager starts the formerly failed node at the first host, and configures it as a spare node. However, in some embodiments, the cluster manager may simply wait for the failed node to restart itself and announce its availability to the cluster manager.

0155In some embodiments, various steps <b>520</b>-<b>570</b> may occur in different orders or even not at all. In an embodiment, once the failed node returns, the cluster manager configures the failed node to again function as a standby node, and returns the new standby node back to its former role as a spare node.

00003.5. Recovering from Failure of Both an Active and Standby Node

0156<figref idref="DRAWINGS">FIG. 6</figref> is a flow chart <b>600</b> illustrating an example method for recovering from the failures of both nodes of an active-standby mid-tier database pair, according to an embodiment of the invention. A cluster manager may be configured to perform this method, for example, by a cluster agent at an active node, per step <b>380</b> of <figref idref="DRAWINGS">FIG. 3</figref>. In an embodiment, these recovery steps are associated with an active node resource—in other words, the cluster manager is configured to perform these recovery steps in response to the failure of the active node if the recovery steps of flow chart <b>400</b> fail to bring up a new active node. In an embodiment, a separate cluster resource may be configured to monitor the active and standby nodes as a pair, and the recovery steps of flow chart <b>600</b> may instead be associated with that cluster resource. While the cluster manager is described as performing the steps of flow chart <b>600</b>, the cluster manager may instead be configured to cause another component to perform any step of flow chart <b>600</b> by issuing one or more commands to that component.

0157At step <b>610</b>, the cluster manager detects that both nodes of an active-standby mid-tier database pair have failed concurrently. For example, while attempting to recover from the failure of an active node, the cluster manager may be unable to cause the performance of various recovery steps at the standby node. The remaining steps of flow chart <b>600</b> are performed in response to this detection.

0158At step <b>620</b>, the cluster manager identifies two recovery nodes. In an embodiment, the recovery nodes were both spare nodes at the time of failure. In an embodiment, one or both of these nodes may be the failed nodes themselves, should the failed nodes return to the cluster within a sufficient period of time.

0159At step <b>630</b>, the cluster manager causes copies of the mid-tier database to be deployed at each of the recovery nodes. For example, each recovery node may create a copy of the mid-tier database based on recovery information available in a shared storage. The recovery nodes may further be instructed to query the backend database system to recover some or all of their respective copies of the mid-tier database. If a recovery node is in fact one of the nodes that had failed, the recovery node may further utilize locally stored recovery information in creating its copy of the mid-tier database.

0160At step <b>640</b>, the cluster manager configures a first recovery node as an active node, and a second recovery node as a standby node. The cluster manager further configures the recovery nodes to implement other synchronization schemes as necessary.

0161At step <b>650</b>, the cluster manager starts any directly-linked clustered applications at the active node's host. The cluster manager also starts “standby” applications at the standby node's host, if any such applications have been defined.

0162At step <b>660</b>, if the recovery active node resides at a different host than did the failed active node, the cluster manager optionally reassigns an active virtual IP address to the host of the recovery active node. Likewise, if the recovery standby node resides at a different host than did the failed active node, the cluster manager optionally reassigns a standby virtual IP address to the host of the recovery standby node.

0163In some embodiments, various steps <b>620</b>-<b>660</b> may occur in different orders or even not at all.

4.0. Additional Implementation Examples

4.1. Recovery for Read-Only Subscribers

0164According to an embodiment, the cluster manager may further be configured to monitor for and react to failures at subscriber nodes. Each subscriber node works with its own cluster agent in much the same manner as described with respect to active nodes, standby nodes, and spare nodes. In an embodiment, the cluster manager is configured to react to a failure of the subscriber node in only one way: restarting the subscriber node. However, in other embodiments, more complex recovery strategies may be used. In an embodiment, a subscriber node may also be considered a spare node.

4.2. Configuration Files

0165According to an embodiment of the invention, cluster configuration data, such as cluster configuration data <b>258</b>, is stored at each host in the form of one or more files. For example, the cluster configuration data may be stored at each host in a file by the name of cluster.ini. Example contents of this file may be as follows:

0000<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="center" /><thead><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row><row><entry>[cluster1]</entry></row><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="offset" colwidth="28pt" align="left" /><colspec colname="1" colwidth="189pt" align="left" /><tbody valign="top"><row><entry /><entry>MasterHosts=host1,host2,host3,host4</entry></row><row><entry /><entry>SubscriberHosts=host5,host6</entry></row><row><entry /><entry>ScriptInstallDir=/mycluster/TTscripts</entry></row><row><entry /><entry>MasterVIP=192.168.1.1, 192.168.1.2</entry></row><row><entry /><entry>SubscriberVIP=192.168.1.3</entry></row><row><entry /><entry>VIPInterface=eth0</entry></row><row><entry /><entry>VIPNetMask=255.255.255.0</entry></row><row><entry /><entry>CacheConnect=y</entry></row><row><entry /><entry>AutoRecover=y</entry></row><row><entry /><entry>RepBackupDir=/shared_drive/dsbackup</entry></row><row><entry /><entry>RepBackupPeriod=3600</entry></row><row><entry /><entry>AppName=reader</entry></row><row><entry /><entry>AppType=Standby</entry></row><row><entry /><entry>AppStartCmd=/mycluster/reader/app_start.sh</entry></row><row><entry /><entry>AppStopCmd=/clusterware/reader/app_stop.sh</entry></row><row><entry /><entry>AppCheckCmd=/clusterware/reader/app_check.sh</entry></row><row><entry /><entry>AppName=update</entry></row><row><entry /><entry>AppType=Active</entry></row><row><entry /><entry>AppStartCmd=/mycluster/update/app2_start.sh</entry></row><row><entry /><entry>AppStopCmd=/clusterware/update/app2_stop.sh</entry></row><row><entry /><entry>AppCheckCmd=/clusterware/update/app2_check.sh</entry></row><row><entry /><entry namest="offset" nameend="1" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0166The line reading “[cluster1]” serves two purposes. First, it defines the name of the mid-tier database cluster. Second, it signifies that every setting after it, up until the next bracketed line (if any), belongs to the “cluster1” cluster. Note that, in an embodiment, the same configuration file may be used to specify multiple logical clusters of different data sets, and that consequentially a single cluster agent may be used for multiple logical clusters.

0167The line beginning “MasterHosts” identifies a set of hosts on which the active node, standby node, and spare nodes may be implemented. The line beginning “StandbyHosts” identifies a set of hosts on which subscriber nodes may be implemented. The line beginning “MasterVIP,” along with the three lines following it, define virtual addresses information for the cluster. The line beginning “CacheConnect” indicates that the clustered data is to be a mid-tier cache of backend data. The line beginning “AutoRecover,” along with the two lines following it, define a backup scheme for the clustered data on a shared storage.

0168The line reading “AppName=reader,” along with the four lines following it, identify an application that should be instantiated in association with the standby node. The line reading “AppName=update,” along with the four lines following it, identify an application that should be instantiated in association with the active node.

5.0. Implementation Mechanism—Hardware Overview

0169According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices, which are also referred to herein as “hosts” or “machines.” The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.

0170For example, <figref idref="DRAWINGS">FIG. 7</figref> is a block diagram that illustrates a computer system <b>700</b> upon which an embodiment of the invention may be implemented. Computer system <b>700</b> includes a bus <b>702</b> or other communication mechanism for communicating information, and a hardware processor <b>704</b> coupled with bus <b>702</b> for processing information. Hardware processor <b>704</b> may be, for example, a general purpose microprocessor.

0171Computer system <b>700</b> also includes a main memory <b>706</b>, such as a random access memory (RAM) or other dynamic storage device, coupled to bus <b>702</b> for storing information and instructions to be executed by processor <b>704</b>. Main memory <b>706</b> also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor <b>704</b>. Such instructions, when stored in storage media accessible to processor <b>704</b>, render computer system <b>700</b> into a special-purpose machine that is customized to perform the operations specified in the instructions.

0172Computer system <b>700</b> further includes a read only memory (ROM) <b>708</b> or other static storage device coupled to bus <b>702</b> for storing static information and instructions for processor <b>704</b>. A storage device <b>710</b>, such as a magnetic disk or optical disk, is provided and coupled to bus <b>702</b> for storing information and instructions.

0173Computer system <b>700</b> may be coupled via bus <b>702</b> to a display <b>712</b>, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device <b>714</b>, including alphanumeric and other keys, is coupled to bus <b>702</b> for communicating information and command selections to processor <b>704</b>. Another type of user input device is cursor control <b>716</b>, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor <b>704</b> and for controlling cursor movement on display <b>712</b>. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

0174Computer system <b>700</b> may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system <b>700</b> to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system <b>700</b> in response to processor <b>704</b> executing one or more sequences of one or more instructions contained in main memory <b>706</b>. Such instructions may be read into main memory <b>706</b> from another storage medium, such as storage device <b>710</b>. Execution of the sequences of instructions contained in main memory <b>706</b> causes processor <b>704</b> to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

0175The term “storage media” as used herein refers to any media that store data and/or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device <b>710</b>. Volatile media includes dynamic memory, such as main memory <b>706</b>. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

0176Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus <b>702</b>. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

0177Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor <b>704</b> for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system <b>700</b> can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus <b>702</b>. Bus <b>702</b> carries the data to main memory <b>706</b>, from which processor <b>704</b> retrieves and executes the instructions. The instructions received by main memory <b>706</b> may optionally be stored on storage device <b>710</b> either before or after execution by processor <b>704</b>.

0178Computer system <b>700</b> also includes a communication interface <b>718</b> coupled to bus <b>702</b>. Communication interface <b>718</b> provides a two-way data communication coupling to a network link <b>720</b> that is connected to a local network <b>722</b>. For example, communication interface <b>718</b> may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface <b>718</b> may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface <b>718</b> sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

0179Network link <b>720</b> typically provides data communication through one or more networks to other data devices. For example, network link <b>720</b> may provide a connection through local network <b>722</b> to a host computer <b>724</b> or to data equipment operated by an Internet Service Provider (ISP) <b>726</b>. ISP <b>726</b> in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” <b>728</b>. Local network <b>722</b> and Internet <b>728</b> both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link <b>720</b> and through communication interface <b>718</b>, which carry the digital data to and from computer system <b>700</b>, are example forms of transmission media.

0180Computer system <b>700</b> can send messages and receive data, including program code, through the network(s), network link <b>720</b> and communication interface <b>718</b>. In the Internet example, a server <b>730</b> might transmit a requested code for an application program through Internet <b>728</b>, ISP <b>726</b>, local network <b>722</b> and communication interface <b>718</b>.

0181The received code may be executed by processor <b>704</b> as it is received, and/or stored in storage device <b>710</b>, or other non-volatile storage for later execution.

6.0. Extensions and Alternatives

0182In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

0183According to an embodiment, each host of a mid-tier database node is assigned a virtual address. Each virtual address is mapped to a specific node role—e.g. active, or standby, subscriber, and so on. When the role of a node changes, it is assigned the virtual address mapped to that role. Applications may be configured to route database commands through the active node's virtual address, without actively monitoring the node at the virtual address to ensure that it is still active. When the active node fails, the cluster manager causes the active node's virtual address to be reassigned to the host of the new active node (e.g. the former standby node) Likewise, the virtual address of a failed standby node's host may be reassigned to the host of a new standby node. In this manner, applications may be transparently redirected to a new active node, without having to implement special logic to detect node failures and relocations. Moreover, clients of directly linked applications may communicate with those applications via the virtual address. Since the cluster manager will also automatically start the applications at the host of the new active node, the reassignment of the virtual address to the new active node ensures that the clients will automatically be redirected to working versions of the applications in the event of database failure.

0184In an embodiment, each cluster agent acts as its own cluster administration tool, in that each cluster agent is responsible for reading cluster configuration data when it comes online, automatically determining its role in the cluster, and then creating any appropriate cluster resources for the cluster manager to monitor.

0185Embodiments of the invention may also be applicable to any environment where databases are stored entirely within a volatile memory, regardless of where the databases are deployed or whether a backend database system is involved. Furthermore, embodiments of the invention may be applicable to any environment where a single machine hosts both applications and the data upon which those applications rely, again regardless of where the hosts are deployed or whether a backend database system is involved.