Techniques for performing resynchronization on a clustered system
Summary by NHIP
Clustered system resynchronization
The method identifies write requests targeting a first cluster and a second cluster, then detects failures causing unsynchronized dirty region information. It resynchronizes successfully written data from the second cluster to the first cluster using log information stored in a dirty region log.
Claim Score by NHIP
Abstract
Various embodiments are generally directed an apparatus and method for receiving information to write on a clustered system comprising at least a first cluster and a second cluster, determining that a failure event has occurred on the clustered system creating unsynchronized information, the unsynchronized information comprising at least one of inflight information and dirty region information, and performing a resynchronization operation to synchronize the unsynchronized information on the first cluster and the second cluster based on log information in at least one of an inflight tracker log for the inflight information and a dirty region log for the dirty region information.
Term
8.7 yearsleft in the term
Expires 21 June 2035, including 244 days of term adjustment.
- Priority and filed
- Granted
- Today
- Expires
20 claims: 3 independent, 17 dependent
- 1A method, comprising:identifying, by a processing component, information, of a write request, to write to a first cluster and a second cluster, wherein the write request targets the first cluster and is replicated as a replicated write request to the second cluster;determining that a failure event has occurred resulting in unsynchronized information between the first cluster and the second cluster, the unsynchronized information comprising dirty region information;andperforming a resynchronization operation to synchronize the unsynchronized information between the first cluster and the second cluster based on log information in a dirty region log for the dirty region information, wherein the dirty region log is used by the resynchronization operation to synchronize replicated data, successfully written to the second cluster by the replicated write request, from the second cluster to the first cluster where the write request was not successfully completed at the first cluster.
- 11Broadest claimClaim Score 60, broad(NHIP)A computer-readable storage medium comprising a plurality of instructions that, when executed by processing circuitry, enable the processing circuitry to:identify information, of a write request, to write to a first cluster and a second cluster;create an entry within an inflight tracker log for the write request, wherein the entry indicates that data of the write request has not completed at both the first cluster and the second cluster, and wherein the entry is cleared once the data is written to both the first cluster and the second cluster;determine that a failure event has occurred resulting in unsynchronized information between the first cluster and the second cluster, the unsynchronized information comprising inflight information;andperform a resynchronization operation to synchronize the unsynchronized information between the first cluster and the second cluster based on log information in the inflight tracker log for the inflight information.
- 16A computing device, comprising:a memory containing computer-readable storage medium having stored thereon instructions for performing a method;anda processor unit coupled with the memory, the processor unit configured to execute the instructions to cause the processor unit to: identify information, of a write request, to write to a first cluster and a second cluster, wherein the write request targets the first cluster and is replicated as a replicated write request to the second cluster;determine that a failure event has occurred resulting in unsynchronized information between the first cluster and the second cluster, the unsynchronized information comprising dirty region information received by a clustered system;andperform a resynchronization operation to synchronize the unsynchronized information between the first cluster and the second cluster based on log information in a dirty region log for the dirty region information, wherein the dirty region log is used by the resynchronization operation to synchronize replicated data, successfully written to the second cluster by the replicated write request, from the second cluster to the first cluster where the write request was not successfully completed at the first cluster.
Independent claims3
115 paragraphs in 4 sections, as filed
TECHNICAL FIELD
Embodiments described herein generally relate to resynchronizing storage clusters on a clustered system.
BACKGROUND
Clustered storage systems may store and provide information to one or more computing systems in a network, such as a storage area network (SAN). More specifically, a computing system may write information to a storage system and read information from the storage system over one or more network connections. These clustered storage systems may include storage devices, such as disks, in an array to store the information.
The information stored on the clustered storage systems may be critical and important to clients utilizing the clustered storage system. Thus, the clustered storage systems may provide backup services to the clients. More specifically, a backup or a copy of the information may be stored on the clustered storage system and, in some instances, on storage devices that are separate from where the original information is stored. However, in some instances, errors or failures may cause information stored on the clustered storage system to become unsynchronized.
BRIEF DESCRIPTION OF THE DRAWINGS
Embodiments described herein are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.
<figref idref="DRAWINGS">FIG. 1A</figref> illustrates an exemplary embodiment of a storage computing system.
<figref idref="DRAWINGS">FIG. 1B</figref> illustrates a second exemplary embodiment of a storage computing system.
<figref idref="DRAWINGS">FIG. 1C</figref> illustrates a third exemplary embodiment of storage computing system.
<figref idref="DRAWINGS">FIG. 1D</figref> illustrates a fourth exemplary embodiment of a storage computing system.
<figref idref="DRAWINGS">FIG. 2</figref> illustrates a logic flow to communicate replication transfer.
<figref idref="DRAWINGS">FIG. 3</figref> illustrates a logic flow to schedule replication transfers.
<figref idref="DRAWINGS">FIG. 4</figref> illustrates a logic flow diagram to handle new information received during a resynchronous operation.
<figref idref="DRAWINGS">FIG. 5A</figref> illustrates an exemplary embodiment of a region.
<figref idref="DRAWINGS">FIG. 5B</figref> illustrates a second exemplary embodiment of a region.
<figref idref="DRAWINGS">FIG. 5C</figref> illustrates a third exemplary embodiment of a region.
<figref idref="DRAWINGS">FIG. 5D</figref> illustrates a fourth exemplary embodiment of a region.
<figref idref="DRAWINGS">FIG. 6</figref> illustrates an exemplary embodiment of a logic flow.
<figref idref="DRAWINGS">FIG. 7</figref> illustrates an exemplary embodiment of a computing system.
<figref idref="DRAWINGS">FIG. 8</figref> illustrates an embodiment of a first computing architecture.
DETAILED DESCRIPTION
Various embodiments are directed systems, devices, apparatuses, methods and so forth to perform resynchronization operations on a clustered system. For example, a clustered system may include a primary cluster and a secondary cluster, where data is synchronized on both clusters to provide failover and backup capabilities. Generally, information may be received from one or more client devices by the primary cluster and copied to the secondary cluster in a parallel manner. When the system is operating “normally”, the information may be copied in a parallel manner by writing to storage devices associated with the primary cluster and storage devices associated with the secondary cluster in parallel.
However, at various points in time, for any number of reasons, the primary cluster and secondary cluster may fall out of sync or become unsynchronized. In other words, information stored on the storage device associated with the primary cluster may not match or mirror the information stored on storage device associated with a secondary. For example, a failure event, such as a communication failure, a hardware failure, a software failure, and so forth may occur on either of the clusters which may cause information to not be written to one or more of the storage devices. Thus, various embodiments are directed to performing resynchronization operations so that the primary cluster and secondary cluster are synchronized after the failure event or whatever has caused the clustered system to become unsynchronized is resolved.
In some embodiments, one or more synchronization logs may be used to track information for storage on the clustered system and are used to perform the resynchronization operation when information becomes out-of-sync. For example, the clustered system may include an inflight tracker log which may store information to track inflight information including data and other information that is in the process of being written to the storage devices associated with both the primary and secondary clusters, but not committed on both the primary and secondary clusters. Only after a confirmation that the inflight information has been committed to storage devices associated with the primary cluster and the secondary cluster will the inflight tracker log for that information be cleared.
When the clustered system becomes out of the sync one or more components of the clustered system may read the inflight tracker log, determine information that has not be fully committed or written to on the clustered system, and perform a resynchronous operation to retrieve the information and fully commit it on the clustered system. For example, the clustered system may determine regions of a data structure on storage devices associated with primary cluster at least allocated to have inflight information based on the inflight tracker log and copy the inflight information to a corresponding region of a data structure on storage devices associated with the secondary cluster.
In some instances, since the inflight information may not have been committed on either the primary cluster or the secondary cluster the information copied from the primary cluster to the secondary cluster may be “old” information or data. However, in some instances the inflight information may have been committed on the primary cluster, but not on the secondary cluster and thus “new” information may be copied from the primary cluster to the secondary cluster during the resynchronization operation. However, in some other instances, inflight information may have been committed on the secondary cluster, but not on the primary cluster and “old” information may be copied from the primary cluster to the secondary cluster during the resynchronization operation. Copying “old” information from the primary cluster to the secondary cluster may be desirable since the primary cluster is the source of information to the client devices. However, various embodiments are not limited in this manner. For example, when the secondary cluster has committed the inflight information, it may be copied from secondary cluster to the primary cluster.
Various embodiments may also include a dirty region log to track dirty region information or data and information that has been written to one of the clusters, either the primary cluster or the secondary cluster, but not both. The clustered system may also use this dirty region log to perform resynchronization operations to ensure that the dirty region information is synchronized across the clustered system, on both the primary cluster and the secondary cluster. For example, various components of the clustered may read the dirty region log and determine the dirty region information that is not synchronized on the clustered. Further, the information may be copied from storage devices associated with the cluster having the information to storage devices associated with the cluster not having the information. These and other details will become more apparent with the following description.
Various embodiments also relate to apparatus or systems for performing these operations. This apparatus may be specially constructed for the required purpose or it may include a general-purpose computer as selectively activated or reconfigured by a computer program stored in the computer. The procedures presented herein are not inherently related to a particular computer or other apparatus. Various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these machines will appear from the description given.
Reference is now made to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding thereof. It may be evident, however, that the novel embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate a description thereof. The intention is to cover all modifications, equivalents, and alternatives consistent with the claimed subject matter.
<figref idref="DRAWINGS">FIG. 1A</figref> illustrates a general overview of an exemplary clustered system <b>100</b> including a first cluster <b>106</b>-<b>1</b> having one or more nodes <b>108</b>-<b>1</b>-x configured to provide processing and storage services on a plurality of storage devices <b>114</b>-<b>1</b>-m of a storage array <b>112</b>-<b>1</b>, where x and m may be any positive integer. The clustered system <b>100</b> may also include a second cluster <b>106</b>-<b>2</b> having a plurality of nodes <b>108</b>-<b>2</b>-y also coupled with a plurality of storage devices <b>114</b>-<b>2</b>-n of a storage array <b>112</b>-<b>2</b>, where y and n may be any positive integer. In the exemplary embodiment, <figref idref="DRAWINGS">FIG. 1A</figref> illustrates a client <b>102</b> coupled with clusters <b>106</b>-<b>1</b> and <b>106</b>-<b>2</b> through one or more network connections <b>101</b>. In some embodiments, the one or more network connections <b>101</b> may include any type of wired or wireless connection or combination thereof. The client <b>102</b> may communicate any type of information to the clusters <b>106</b> for storage on one or more storage devices <b>114</b> in a storage array <b>112</b> over the one or more network connections <b>101</b>. Although <figref idref="DRAWINGS">FIG. 1</figref> illustrates only one client device, any number of client devices may communicate with the clustered system <b>100</b>.
The clustered system <b>106</b> may be configured to provide failover and backup capabilities for the client <b>102</b>. In some embodiments, the first cluster <b>106</b>-<b>1</b> may be considered a primary cluster and the second cluster <b>106</b>-<b>2</b> may be considered a secondary cluster. The client <b>102</b> may communicate information with first cluster <b>106</b>-<b>1</b> as a primary cluster and the information may be communicated between the first cluster <b>106</b>-<b>1</b> and the second clustered system <b>106</b>-<b>2</b> via the cluster switching fabric <b>116</b> to provide the failover and backup capabilities.
In an exemplary embodiment, if an error or a failover event occurs on the clustered system <b>100</b>, the client <b>102</b> may switch from communicating with the first cluster <b>106</b>-<b>1</b> to communicate with the second cluster <b>106</b>-<b>2</b>. Since the information is copied or backed up on the second cluster <b>106</b>-<b>2</b>, the switching between the clusters <b>106</b> occurs transparently and seamlessly to the client <b>106</b>. In addition, various embodiments are not limited to switching from the first cluster <b>106</b>-<b>1</b> to the second cluster <b>106</b>-<b>2</b> and switching may occur in either direction. In some embodiments, the client <b>102</b> may switch back and communicate information with the first cluster <b>106</b>-<b>1</b> once the error or failure event has been resolved. In the same or other embodiments, the second cluster <b>106</b>-<b>2</b> may be operating as the primary cluster and the client <b>102</b> may switch and communicate with the first cluster <b>106</b>-<b>1</b> during a failure event.
Client <b>102</b> may communicate any type of information to be stored on the clustered system and may be any type of computing device having a plurality of components. These components may include a central processing unit (CPU), main memory, I/O devices, and storage devices (for example, flash memory, hard drives and others). The main memory may be coupled to the CPU via a system bus or a local memory bus. The main memory may be used to provide the CPU access to data and/or program information that is stored in main memory at execution time. Typically, the main memory is composed of random access memory (RAM) circuits. A computer system with the CPU and main memory is often referred to as a host system.
Client <b>102</b> may be configured to interact with nodes <b>108</b> of the clusters <b>106</b> in accordance with a client/server model of information delivery. That is, a client <b>102</b> may request the services of a node <b>108</b>, and the node <b>108</b> may return the results of the services requested by the client <b>102</b> over a network including one or more network connections <b>101</b>. Client <b>102</b> may be configured to execute processor executable instructions, shown as application(s) <b>104</b> for reading and writing information at the cluster <b>106</b>-<b>1</b> and cluster <b>106</b>-<b>2</b>. Such application(s) <b>104</b> may include a database application, a financial management system, an electronic mail application or any other application type.
The client <b>102</b> may communicate over any type of one or more network connections <b>101</b> including an Ethernet network, a Fibre Channel (FC) network or any other network type. Moreover, the nodes <b>108</b> may be interconnected by a cluster switching fabric <b>116</b> which may be embodied as a Gigabit Ethernet switch or any other interconnect type.
Further, the client <b>102</b> may issue packets using application(s) <b>104</b> including file-based access protocols, such as the Common Internet File System (“CIFS”) protocol or the Network File System (“NFS”) protocol, over the Transmission Control Protocol/Internet Protocol (“TCP/IP”) when accessing information in the form of certain data containers. Alternatively, the client <b>102</b> may issue packets using application(s) <b>104</b> including block-based access protocols, such as the Small Computer Systems Interface (“SCSI”) protocol encapsulated over TCP (“iSCSI”) and SCSI encapsulated over Fibre Channel (“FCP”), when accessing information in the form of other data containers, such as blocks.
In various embodiments a node <b>108</b> may be any type of computing device and may include any number of functional components that cooperate to provide distributed storage system architecture for the client <b>106</b>. Each node <b>108</b> is generally organized as a network element and a storage device element. The network element may include functionality that enables nodes <b>108</b> to connect to the client <b>102</b> over one or more network connections <b>101</b>, while each storage device element connects to one or more storage devices <b>114</b> or a storage array <b>112</b> having a plurality of storage devices <b>114</b>. Further and as will be discussed in more detail below nodes <b>108</b> may include other functional components for providing the backup and failover capabilities and to ensure synchronization of information on the clustered system.
Each of the clusters <b>106</b>-<b>1</b> and <b>106</b>-<b>2</b> including the nodes <b>108</b> may be coupled with or include a storage array <b>112</b>-<b>1</b> and <b>112</b>-<b>2</b> having or more storage devices <b>114</b>. The storage arrays <b>112</b>-<b>1</b> and <b>112</b>-<b>2</b> may include a number of elements and components to provide storage services to the client <b>102</b>. More specifically, the storage arrays <b>112</b> may implement a high-level module, such as a file system, to logically organize the information as a hierarchical structure of directories or files and may have a data structure such as logical units identified by a logic unit number (LUN) on the storages devices <b>114</b>.
In various embodiments, the LUN may be a unique identifier used to designate individual or collections of hard disk devices for address by a protocol associated with a SCSI, iSCSI, Fibre Channel (FC), and so forth. Logical units are central to the management of block storage arrays shared over a storage area network (SAN). Each LUN identifies a specific logical unit, which may be a part of a hard disk drive, an entire hard disk or several hard disks in a storage device <b>114</b>, for example. As such, a LUN could reference an entire RAID set, a single disk or partition, or multiple hard disks or partitions. The logical unit is treated as if it is a single device and is identified by the LUN.
Each logical unit may be divided into a number of smaller regions which may be further divided into blocks. Each region of a data structure may have a particular region size, such as <b>1</b> Megabyte (MB) and each block may have a block size, such as <b>4</b> Kilobytes (KB), 8 KB, 16 KB, 32 KB, 64 KB, and so forth. Information received from a client <b>102</b> may be stored and written to the logical units on a block-by-block basis or in chunks of blocks. Further and as will be discussed in more detail, information targeted to be written to a logical unit may be tracked and logged in one or more logs on a region-by-region basis. This tracking and logging of information may be used to ensure that the primary cluster and secondary cluster are in sync and to enable the resynchronization of information across the clustered system when it falls out of sync, such as when a failure event occurs.
<figref idref="DRAWINGS">FIG. 1B</figref> illustrates an exemplary embodiment of a system <b>125</b> for providing syncing and resyncing capabilities in a clustered system environment. System <b>125</b> may include cluster <b>106</b>-<b>1</b> having node <b>108</b>-<b>1</b> and cluster <b>106</b>-<b>2</b> having node <b>108</b>-<b>2</b> and may be the similar to or the same as the liked named elements discussed above with respect to <figref idref="DRAWINGS">FIG. 1A</figref>. <figref idref="DRAWINGS">FIG. 1B</figref> only shows each cluster having one node for illustrative purposes only and as discussed above with respect to <figref idref="DRAWINGS">FIG. 1A</figref>, a cluster may have any number of nodes. Further and in some embodiments, cluster <b>106</b>-<b>1</b> may be a primary cluster and cluster <b>106</b>-<b>2</b> may be a secondary cluster. However, cluster <b>106</b>-<b>1</b> and cluster <b>106</b>-<b>2</b> are not limited in this manner and at certain times, such as when a failure event occurs, cluster <b>106</b>-<b>2</b> may be the primary cluster and cluster <b>106</b>-<b>1</b> may be the secondary cluster.
In some embodiments, a client may communicate with the primary cluster and send it information to store on storage devices associated with the primary cluster. The information may be sent to the secondary cluster to store as a backup and to provide failover capabilities in case of a failure. More specifically, a client may send information to cluster <b>106</b>-<b>1</b> when it is operating as a primary cluster and the information may be copied to cluster <b>106</b>-<b>2</b>. In another example, a client may send information to cluster <b>106</b>-<b>2</b> when it is operating as a primary cluster and the information may be copied to cluster <b>106</b>-<b>1</b>. Various embodiments are not limited in this manner.
<figref idref="DRAWINGS">FIG. 1B</figref> illustrates one exemplary configuration where node <b>108</b>-<b>1</b> includes components for receiving information from a client and processing the information as part of cluster <b>106</b>-<b>1</b> operating as a primary cluster. When node <b>108</b>-<b>1</b> is operating in a primary cluster, it may include a splitter <b>127</b>, a resync engine <b>129</b> and a sync transfer engine <b>131</b>-<b>1</b>. Node <b>108</b>-<b>2</b> is illustrated as having components for processing information as part of cluster <b>106</b>-<b>2</b> operating as a secondary cluster and includes a sync transfer engine <b>131</b>-<b>2</b> and a writer <b>133</b>. Any node operating in a primary cluster may include components illustrated in node <b>108</b>-<b>1</b> and any node operating in a secondary cluster may include components illustrated in node <b>108</b>-<b>2</b>. When the clusters <b>106</b>-<b>1</b> and <b>106</b>-<b>2</b> switch operating roles, the components illustrated operating on the cluster <b>106</b>-<b>1</b> may operate on cluster <b>106</b>-<b>2</b>, and vice versa. However, for simplification purposes <figref idref="DRAWINGS">FIG. 1B</figref> is discussed with cluster <b>106</b>-<b>1</b> as the primary cluster and cluster <b>106</b>-<b>2</b> as the secondary cluster.
In various embodiments, a splitter <b>127</b> on a primary cluster may be responsible for receiving information from one or more clients, such as client <b>102</b> and providing synchronous write capabilities. The information received from the client <b>102</b> may be received as one or more write requests to write information on storage devices, such as storage devices <b>114</b>-<b>1</b> and <b>114</b>-<b>2</b>. The splitter <b>127</b> may send the information to storage devices associated with both the primary cluster and the secondary cluster in parallel or in a synchronous manner. For example, the splitter <b>127</b> may send the original information to the storage devices <b>114</b>-<b>1</b> associated with the primary cluster and may send a copy of the information to storage devices <b>114</b>-<b>2</b> associated with the secondary cluster at the same time or nearly the same time. Various embodiments are not limited in this manner and the splitter <b>127</b> may first send the information to the storage devices <b>114</b>-<b>1</b> associated with the primary cluster and then to the storage devices <b>114</b>-<b>2</b> associated with the secondary cluster in a serial manner. However, by sending the information in parallel or at nearly the same time, latency times may be reduced when performing read and write requests.
In various embodiments, the splitter <b>127</b> may utilize the sync transfer engine <b>131</b>-<b>1</b> to send information, such as the copy of the information, to the secondary cluster. The sync transfer engine <b>131</b>-<b>2</b> operating on the secondary cluster may receive the copy of the information from the sync transfer engine <b>131</b>-<b>1</b> and may send it to a writer <b>133</b> for writing and committing the information to storage devices <b>114</b>-<b>2</b> in storage array <b>112</b>-<b>2</b>.
The splitter <b>127</b> may also communicate with a resync engine <b>129</b> and provide it with log information, such as inflight log information and dirty region log information to store in one or more logs. The inflight log information may track inflight information or data and information received by a clustered system but not yet written on one or more storage devices associated with a primary cluster and/or a secondary cluster on a region-by-region basis. The inflight log information may track the inflight information and indicate that the inflight information has not been successfully stored on storage devices associated with the primary cluster, storage devices associated with the secondary cluster, or both.
In various embodiments, the storage devices associated with each of the clusters, primary and secondary, may send a response back to the splitter <b>127</b> indicating that the inflight information has been successfully stored. Only when storage devices associated with both clusters have sent a response back to the splitter <b>127</b> indicating success, will the inflight log information be updated to reflect that the information has been stored.
The splitter <b>127</b> may first send the inflight tracker log information to the resync engine <b>129</b> prior to sending the inflight information to the storage devices of the primary cluster and a copy of the inflight information to the sync transfer engine <b>133</b>-<b>1</b> for storage on the secondary cluster. This may be done to ensure that the inflight information is tracked by a log if a failure event were to occur during the communication of the inflight information. As will be discussed in more detail below with respect to <figref idref="DRAWINGS">FIG. 1D</figref>, the inflight tracker log information may be stored in an inflight tracker log <b>177</b>.
The splitter <b>127</b> may also send dirty region log information to the resync engine <b>129</b> to track and log information stored on one the clusters, either the primary cluster or the secondary cluster, but not both during a failure event. In other words, the dirty region log information indicates information has been successfully written to at least one storage device associated with one of the clusters and a response has been received by the splitter <b>127</b>. In various embodiments, the dirty region log information may be sent to the resync engine <b>129</b> during a failure event to track regions written to during the failure event.
In one example, information may be stored on storage devices associated with a primary cluster, but not on storage devices associated with a secondary cluster during a failure event and each region written to on the primary cluster may be tracked by the dirty region tracker log <b>179</b>. In another example, information may be stored in a data structure on the storage devices associated with the secondary cluster, but not the primary cluster during the failure event and each region written to on the secondary cluster may be tracked by the dirty region log <b>179</b>. In a third example, some information may be stored in a data structure associated with the primary cluster and some other information may be stored in a data structure associated with the secondary cluster during a failure event and the each region written on both the primary cluster and secondary cluster may be tracked in the dirty region log <b>179</b>.
In various embodiments, a clustered system may include a resync engine <b>129</b> to ensure that the clusters are in sync. More specifically, the resync engine <b>129</b> may utilize a resynchronous operation to ensure that inflight information is stored on the clustered system and data structures for each cluster are in sync. More specifically, the resync engine <b>129</b> may perform the resynchronous operation after a failure event has been resolved by copying information from one cluster to another cluster based on the inflight tracker log information and dirty region log information. Thus, each of the logs, the inflight tracker log and the dirty region log, may provide information to the resync engine <b>129</b> to ensure that a clustered system is synchronized after a failure event or any other event that may cause the clusters to become unsynchronized.
With respect to the inflight information, the resync engine <b>129</b> may perform a resynchronous operation and use the inflight tracker log information in the inflight tracker log <b>177</b> to determine inflight information that may not have been written to storage devices due to the failure event or loss of data. In some instances, since the inflight information may not have been committed on either the primary cluster or the secondary cluster the information copied by the resync engine <b>129</b> from the primary cluster to the secondary cluster may be “old” information or data. However, in some instances the inflight information may have been committed on the primary cluster, but not on the secondary cluster and thus “new” information may be copied by the resync engine <b>129</b> from the primary cluster to the secondary cluster during the resynchronization operation. However, in some other instances, inflight information may have been committed on the secondary cluster, but not on the primary cluster and “old” information may be copied by the resync engine <b>129</b> from the primary cluster to the secondary cluster during the resynchronization operation. Copying “old” information from the primary cluster to the secondary cluster may be desirable since the primary cluster is the source of information to the client devices. However, various embodiments are not limited in this manner. For example, when the secondary cluster has committed the inflight information, the resync engine <b>129</b> may copy information from secondary cluster to the primary cluster. The resync engine <b>129</b> may repeat this operation for all of the inflight information indicated in the inflight tracker log <b>177</b> on a region-by-region basis.
The resync engine <b>129</b> may also ensure the storage devices of the primary cluster and secondary cluster are in sync based on the dirty region log information in the dirty region log <b>179</b>. As previously discussed, the dirty region log <b>179</b> may have dirty region log information indicating one or more regions that may have been written to while the clustered system was in a failed state. Since these regions were written to while the system was the failed state, the information may not be synced across the primary and secondary clusters. To resync the information, the resync engine <b>129</b> may perform the resynchronous operation using a sweeping hand method to copy information from one cluster having the information to another cluster.
The sweeping hand method may include reading information and writing information from the data structures or logical units in a block-by-block manner, or by chunks of blocks. The sweeping hand method may be performed on each region indicated in the dirty region log <b>179</b> as having dirty region information. As will be discussed in more detail below, the resync engine <b>129</b> may start the resynchronous operation at one end point of a region and finish at another end point of the region. In some embodiments, the starting end point may be a block having the lowest block number and the finishing end point may be a block having the highest block number. However various embodiments are not limited in this manner and starting end point may be the block having the highest block number and the finishing end point may be the block having the lowest block number.
During the resynchronous operation, the resync engine <b>129</b> may read information from a portion of a region, such as a block or chunk of blocks on a storage device and write the information to a corresponding portion of a region on a different storage device. In some embodiments, the information may be read from a storage device associated with a primary cluster and written to a storage device associated with a secondary cluster, or vice versa. To prevent corruption of the information being resynced, the resync engine <b>129</b> may enable a lock on the portion of the region being read from and written to. By enabling a lock on the portion, the resync engine <b>129</b> may prevent the corruption of information while it is being sync'd across the clusters.
Further, the clustered system may receive new information from a client for storage while the resynchronous operation is being conducted. The resync engine <b>129</b> may handle the writing of the new information to the storage devices of both the primary cluster and the secondary cluster during the resynchronous operation. More specifically, the resync engine <b>129</b> may receive the new information from the splitter <b>127</b> and may determine whether the new information is to be stored in a portion of a region that already has been resynced, in a portion of the region that is currently being resynced or in a portion of the region that will be resynced at a future point in time.
The resync engine <b>129</b> may enable the new information to be written to storage devices associated with both the primary cluster and the secondary cluster when the new information is to be stored in a portion of the region already resynced. When the new information is be stored in the portion of the region currently being resynced, the resync engine <b>129</b> may wait until the resynchronous operation of this portion is complete and then enable the new information to be written to storage devices associated with both the primary cluster and the secondary cluster. As previously discussed, the resync engine <b>129</b> may enable a lock on the portion of the region currently being resynced to prohibit reading and writing to this portion. The lock may be lifted or disabled once resyncing has completed. Finally, the resync engine <b>129</b> may enable writing new information to a portion of a region that will be resynced at some future point in time. In this case, the new information only needs to be written to the portion to be resynced because it will be copied to the other cluster during the resynchronous operation.
<figref idref="DRAWINGS">FIG. 1C</figref> illustrates an exemplary detailed embodiment of a resync engine <b>129</b> having components to perform the operations described above including resynchronous operations, processing new information, and ensuring clustered systems are in sync. In various embodiments, the resync engine <b>129</b> may include sync logs <b>151</b>, such as the inflight tracker log <b>177</b> and the dirty region log <b>179</b> as illustrated in <figref idref="DRAWINGS">FIG. 1D</figref>. The resync engine <b>129</b> may also include a resync control <b>153</b> and a resync scanner <b>155</b>. Various embodiments are not limited in this manner and the resync engine <b>129</b> may include more or less components to provide various resync functionality.
The resync control <b>153</b> may control various aspects of a resynchronous operation including initialization, resyncing information from a data structure on a primary cluster to a data structure on a secondary structure or vice versa, synchronizing lost inflight information, logging dirty region information, handling new information during a resynchronous operation, and so forth. For example, the resync control <b>153</b> may initialize or start a resynchronous operation based on input from a user or information received from other system components. More specifically, a user may manually start a resynchronous operation via one or more inputs using an input device, such as a keyboard, mouse, touchscreen interface, and so forth. In some embodiments, one or more components (not shown) may monitor and determine when a failure event has occurred and send information to the resync control <b>153</b> to start a resynchronous operation once the failure event has been resolved.
Further, the resync control <b>153</b> may also control the resynchronous operation by reading the sync logs <b>151</b> and determining information, such as inflight information and dirty region information that needs to be resynchronized across a clustered system. Once the resync control <b>153</b> determines the information to resynchronize, it may perform various operations including synchronizing the information across the clustered system including the primary cluster and the secondary cluster. In particular, the resync control <b>153</b> may send the inflight information directly to the storage devices associated with the primary cluster and may send a copy of the information to the secondary cluster via sync transfer engines operating on both sides of the clustered system. The information and the copy of the information may be written into one or more portions of one or more regions of a data structure based on write requests included with the inflight information.
In some embodiments, the resync control <b>153</b> may handle copying information between the primary cluster and secondary cluster based on inflight log information in the inflight log <b>177</b> and/or the dirty region log information stored in the dirty region log <b>179</b>. As previously discussed, the inflight log <b>177</b> and dirty region log <b>179</b> may store or indicate regions of data structures stored on the primary cluster, the secondary cluster, or both. The resync control <b>153</b> may perform forward resynchronization operations and utilize the resync scanner <b>155</b> to copy information from a data structure associated with the primary cluster to a corresponding data structure associated with the secondary cluster. Furthermore, the resync control <b>153</b> may perform reverse resynchronization operations using the resync scanner <b>155</b> and copy information from a data structure associated with the secondary cluster to a corresponding data structure associated with the primary cluster.
In various embodiments, the resync control <b>153</b> may perform the resynchronization operations utilizing a sweeping hand method. For example, the resync control <b>153</b> may copy information from one end point of a region to another end point of the region. In some embodiments, the resync control <b>153</b> may start the resynchronous operation at an end point having the lowest block number and copying the information on a block-by-block basis or in groups of blocks using the resync scanner <b>155</b> until an end point having a highest block number is reached, or vice versa. The resync control <b>153</b> may determine the region to copy information from based on the dirty region log information in the dirty region log <b>179</b>.
In some embodiments, the resync control <b>153</b> may also control the synchronization of new information received from one or more clients while the resynchronous operation is executing. For example, the resync control <b>153</b> may enable writing information to a region or a portion of a region that already has been resynced on both the primary cluster and secondary cluster. In another example, the resync control <b>153</b> may wait for the resynchronous operation to complete on a portion of a region when the new information is to being written to that portion of the region currently being resynced. In a third example, when the new information is to be located in a portion of a region that needs to be resynced, the resync control <b>153</b> may write the new information to a data structure on only one cluster of the clustered system, either on the primary cluster or the secondary cluster wherever information is being read from.
<figref idref="DRAWINGS">FIG. 1D</figref> illustrates an exemplary embodiment of sync logs <b>151</b> including an inflight tracker log <b>177</b> and a dirty region log <b>179</b> to store log information. The inflight tracker log <b>177</b> may store inflight tracker log information and the dirty region log <b>179</b> may store dirty region log information. The inflight tracker log information may record any incoming writes from one or more clients that have not been fully committed or written on a clustered system. Moreover, the inflight tracker log information may log the incoming writes when the write is first received by the clustered system. This is to ensure that if a relationship falls out of sync between clusters, such as a primary cluster and secondary cluster, the regions corresponding to the writes can be brought back into sync, i.e. resynced. Once information relating to the write instruction for information has been stored on both sides it is no longer considered inflight information and that region can be cleared in the inflight tracker log <b>179</b>.
The inflight tracker log <b>177</b> may be a file stored in cache in random access memory, or on a persistent storage device such as a hard drive and may indicate each region having inflight information to be written in. In some embodiments, the file may be in a bitmap file format where each bit corresponds to a region of a data structure such as a logical unit. When a bit corresponding to a region is set to one, inflight information is to be saved in that corresponding region. When a bit corresponding to a region is set to zero, the corresponding region is clear and does not having any inflight information to be written to it. This coarse grain approach, i.e. each bit corresponding to a region, allows the file or inflight tracker log <b>177</b> to be small and stored in cache.
Similarly, the dirty region log <b>179</b> may store dirty region log information indicating information that may be written to regions during a failure event on one of the clusters. The splitter <b>127</b> may send dirty region log information to the dirty region log <b>179</b> when the clustered system is in a failed state or during a failure event. The dirty region log <b>179</b> may also implement and track dirty region information in a bitmap file on a region-by-region basis. Each bit corresponds to a region of a data structure such as a logical unit. When a bit corresponding to a region is set to one, dirty region information is saved in that corresponding region on either the primary cluster or the secondary cluster, but not both. When a bit corresponding to a region is set to zero, the corresponding region is in sync on both the primary cluster and secondary cluster.
<figref idref="DRAWINGS">FIG. 2</figref> illustrates one exemplary embodiment of a logic flow <b>200</b> for processing information on a clustered system and tracking inflight information and dirty region information. Logic flow <b>200</b> may be representative of some or all of the operations executed by one or more embodiments described herein. Logic flow <b>200</b> is discussed with reference to the clustered system of <figref idref="DRAWINGS">FIGS. 1A-1D</figref> for illustrative purposes. However, various embodiments are not limited in the manner and the logic flow may be implemented on any computing system or device.
At block <b>202</b> a clustered system may receive information including one or more write instructions to store. For example, a primary cluster, such as cluster <b>106</b>-<b>1</b> may receive write instructions from a client <b>102</b> to store information and data on cluster array <b>112</b>-<b>1</b>. Cluster <b>106</b>-<b>1</b> may receive the information and send it to one or more components including a splitter <b>127</b>. The splitter <b>127</b> may prepare the information to be synchronously written across the clustered system on both the primary cluster and the secondary cluster. In particular, the splitter <b>127</b> may send inflight tracker log information to a resync engine <b>129</b> to store in an inflight tracker log <b>177</b>. The inflight tracker log information may indicate one or more regions in which the information is to written to on the clusters and the resync engine <b>129</b> may store the inflight tracker log information in the inflight tracker log <b>177</b> at block <b>204</b>.
In various embodiments, the splitter <b>127</b> may perform a synchronous write operation to write the information on both the primary cluster and secondary cluster at block <b>206</b>. More specifically, the splitter <b>127</b> may send the information to be written on the storage devices associated with the primary cluster and send a copy of the information to be written on storage devices associated with the secondary cluster. In some embodiments, the splitter <b>127</b> may utilize a sync transfer engines <b>131</b> to communicate the copy of the information to secondary cluster and a writer <b>133</b> may write the copy of the information to the secondary cluster's storage devices. At block <b>208</b>, the splitter <b>127</b> may determine whether the information has been successfully written to the primary cluster and secondary cluster. In some embodiments, the splitter <b>127</b> may receive information from the storage devices on each of the clusters indicate that the write operation successfully completed. If so, the splitter <b>127</b> may clear the inflight tracker log information from the inflight tracker log <b>177</b> for that information at block <b>210</b>. The clustered system may continue to receive information and perform write operations.
If the write operation was not successful at block <b>208</b>, a determination may be made as to whether a failure event has occurred at decision block <b>212</b>. A failure event may include a network failure, a hardware failure, a software failure, etc. and may occur between clusters, nodes on the clusters, the client and the clustered system, and so forth. If a failure event has not occurred at block <b>212</b>, a resynchronous operation may be performed on the clustered system at block <b>220</b> by a resync engine <b>129</b>, for example. In this case, only the inflight information not successfully written to the clusters may be resynced. The resync engine <b>129</b> may perform a resynchronous operation for the inflight information by reading the inflight tracker log <b>177</b> to determine which regions need to be resynced, and copying the data and information on the primary cluster to the secondary cluster once the inflight information is received. In some instances, the data and information copied may be “old” data and information previously received on the clustered system. In other instances, the data and information may be “new” data and information received when the clustered system made an entry in the inflight tracker log. The clustered system may continue to receive information to write after the resynchronous operation has been performed.
If a failure event is determined to have occurred at block decision block <b>212</b>, the clustered system may continue to receive information that may be successfully written to one of the clusters, either the primary cluster or secondary cluster, but not both. At block <b>214</b>, the splitter <b>127</b> may send dirty region log information to a dirty region log <b>179</b> for the dirty region information or information that has been successfully written on one of the clusters but not both. Moreover, the dirty region log information may indicate which of the regions of a data structure that have been written to on one of the clusters. At decision block <b>216</b>, a determination may be made as to whether the failure event has been resolved or not. For example, one or more components may send information to the splitter <b>127</b> indicating that the failure event has been resolved. In some embodiments, the splitter <b>127</b> may poll one or more components to determine if the failure event has been resolved. If the failure event has not been resolved and the splitter <b>127</b> continues to receive new information <b>218</b>, the splitter <b>127</b> may track this new information in the dirty region log <b>179</b>. The splitter <b>127</b> may track all of the information written to the clustered system in the dirty region log <b>179</b> during a failure event.
If the failure event is determined to be resolved at decision block <b>216</b>, a resynchronous operation may be performed at block <b>220</b>. In this case, the resync engine <b>129</b> may perform the resynchronous operation to resync the inflight information not successfully written on the clustered system and any dirty region information received during the failure event. The resync engine <b>129</b> may perform the resynchronous operation for the inflight information as previously discussed above copying information on the primary cluster and to the secondary cluster on a region-by-region basis. Further, the resync engine <b>129</b> may perform the resynchronous operation for the dirty region information by reading the dirty region log <b>179</b> to determine the regions that need to be resynced and copying the information from the regions having the information to corresponding regions on a cluster not having the information. In some embodiments, the resync engine <b>129</b> may utilize a sweeping hand method to copy information for regions and may perform the resynchronous operation for every region indicated in the dirty region log <b>179</b>. Although logic flow <b>200</b> illustrates certain operations occurring in a certain order, various embodiments are not limited in this manner. Various operations may occur before or after other operations and in some embodiments inflight information may be lost without a failure event occurring, the inflight information may be resynced on the clustered system. In some embodiments, a failure event may occur without any inflight information being lost and dirty region information may be resynced on the clustered system.
<figref idref="DRAWINGS">FIG. 3</figref> illustrates one exemplary embodiment of a logic flow <b>300</b> for performing a resynchronous operation based on dirty region log information and inflight tracker log information. Logic flow <b>300</b> may be representative of some or all of the operations executed by one or more embodiments described herein. Logic flow <b>300</b> is discussed with reference to the clustered system of <figref idref="DRAWINGS">FIGS. 1A-1D</figref> for illustrative purposes. However, various embodiments are not limited in the manner and the logic flow may be implemented on any computing system or device.
At decision block <b>302</b> a determination may be made as whether information, such as inflight tracker log information, is in the inflight tracker log <b>177</b> and dirty region information is in the dirty region log <b>179</b>. For example, the resync engine <b>129</b> may read the inflight tracker log <b>177</b> to determine whether inflight tracker information needs to be resynced across the clustered system. The inflight tracker log information indicates which regions of the primary and/or second clusters that may have information not fully committed on both the primary cluster and secondary cluster. Logic flow <b>300</b> may also include determining whether dirty region log information is in the dirty region log <b>179</b> at block <b>302</b>. The dirty region log <b>179</b> may include dirty region log information indicating regions that are out of sync on the clusters, i.e. where information is stored on the primary cluster or the secondary cluster, but not both.
At block <b>304</b>, the inflight tracker log <b>177</b> may be used to determine one or more regions allocated for storage of information received from a client device, i.e. inflight information. Further, the dirty region log <b>179</b> may be used to determine one or more regions having dirty region information. For example, the resync engine <b>129</b> may read the inflight tracker log and the dirty region log <b>177</b> and determine which regions need to be resynced on the clusters.
The resync engine <b>129</b> may perform a resynchronous operation for each region indicated in the inflight tracker log <b>177</b> and the dirty region log <b>179</b>. More specifically, the resync engine <b>129</b> may enable a lock on a portion of a region in which information will be read at block <b>306</b>. The lock may be enabled to prevent information from being written to the portion of the region while the resynchronization is being performed. The resync engine <b>129</b> may read information from the portion of the region at block <b>308</b> and may disable the lock on the portion of the region at block <b>310</b>. The resync engine <b>129</b> may write the information to a corresponding portion of a region on a corresponding cluster at block <b>312</b>. For example, the resync engine <b>129</b> may read information from a portion of a region on the primary cluster and write information to a corresponding portion of a region on the secondary cluster, or vice versa.
In various embodiments, a determination as to whether a region has completely been resynced is made at decision block <b>314</b>. If the region has been completely resynced, blocks <b>302</b> through <b>316</b> may be repeated for each region indicated in the inflight tracker log <b>177</b> and the dirty region log <b>179</b>. However, if the region currently being resynced isn't complete, the resync engine <b>129</b> may move to the next portion of the region to copy information at block <b>316</b>. The resync engine <b>129</b> may copy information from a region in a sweeping hand manner as previously discussed. Although <figref idref="DRAWINGS">FIG. 3</figref> illustrates a number of blocks in a certain order, various embodiments are not limited in this manner.
<figref idref="DRAWINGS">FIG. 4</figref> illustrates an exemplary embodiment of a logic flow diagram <b>400</b> for handling new information received during a resynchronous operation. Logic flow <b>400</b> may be representative of some or all of the operations executed by one or more embodiments described herein. Logic flow <b>400</b> is discussed with reference to the clustered system of <figref idref="DRAWINGS">FIGS. 1A-1D</figref> for illustrative purposes. However, various embodiments are not limited in the manner and the logic flow may be implemented on any computing system or device.
At block <b>402</b>, the clustered system may receive information to store on the clusters. If the clustered system is not performing a resynchronous operation, the information may be synchronously written to the primary cluster and secondary cluster by the splitter <b>127</b>, as previously discussed. However, in some embodiments the clustered system may receive new information while a resynchronous operation is being performed. The resync engine <b>129</b> may handle the new information received during a resynchronous operation.
More specifically and in various embodiments, the resync engine <b>129</b> may determine where to write the new information at block <b>404</b>. For example, the new information may include a write operation which may indicate a portion of a region to write the data to, such as one or more addresses, block numbers, or the like. The resync engine <b>129</b> may determine the status of that portion of the region including whether the portion has already been resynced, is currently being resynced or is to be resynced. For example, the resync engine <b>129</b> may determine if the portion of the region has already been resynced by the resynchronous operation at decision block <b>406</b>. If so, the resync engine <b>129</b> may write or instruct the splitter <b>127</b> to synchronously write the information to the primary cluster and secondary cluster at block <b>412</b>.
However, if the new information is to be written to a portion of a region that is currently being resynced as determined at decision block <b>408</b>. The resync engine <b>129</b> may wait for the resynchronous operation to complete on that portion at block <b>410</b>. In some embodiments, the resync engine <b>129</b> may enable a lock on the portion of the region currently being resynced to prevent corruption of data due to overwrite of data. Once the resynchronous operation has been completed on the portion to receive the new information, the resync engine <b>129</b> may synchronously write the new information to the clusters at block <b>412</b>. In some embodiments, the resync engine <b>129</b> may instruct or permit the splitter <b>127</b> to synchronously write the new information to the clusters.
In some embodiments, the new information may be intended to be written to a portion of a region that is to be resynced at some future point in time, as determined at block <b>414</b>. In this case, the resync engine <b>129</b> may write or instruct the splitter <b>127</b> to write the new information to the portion on the cluster that is to be read from during the resynchronous operation at block <b>416</b>. For example, if the resynchronous operation is to copy information from the primary cluster to the secondary cluster, then the new information is to be written to the portion on primary cluster. Similarly, if the resynchronous operation is to copy information from the secondary cluster to the primary cluster, then the new information is to be written to the portion on the secondary cluster. By only writing the new information on one cluster, processing cycles and network bandwidth may be conserved. The new information will be synchronized on the clustered system when the resynchronous operation is performed on the portion that has the new information.
<figref idref="DRAWINGS">FIGS. 5A-5D</figref> illustrate a region <b>500</b> during different stages of a resynchronous operation and handling new information received during a resynchronous operation. <figref idref="DRAWINGS">FIG. 5A</figref> illustrates a typical region during a resynchronous operation. As previously discussed, the clustered system may store information as logical units which may be divided into a number of smaller regions of a data structure, such as region <b>500</b>, which may be further divided into one or more portions or blocks <b>512</b>. Each region may have a particular region size, such as one Megabyte (MB) and each block may have a block size, such as four Kilobytes (KB), eight KB, 16 KB, 32 KB, 64 KB, and so forth. Thus, each region may have any number of blocks based on the region size and the block size. <figref idref="DRAWINGS">FIG. 5A</figref> illustrates region <b>500</b> having blocks <b>512</b>-<b>1</b> through <b>512</b>-t, where t may be any positive integer value. In some embodiments, block <b>512</b>-<b>1</b> may have the lowest block number and block <b>512</b>-t may have the highest block number.
During resynchronous operations, regions of a logical unit may be identified as out of sync in one or more logs, such as the inflight tracker log <b>177</b> and the dirty region log <b>179</b>. Each of the regions identified may be resynced during the resynchronous operation. In some embodiments, the resync engine <b>129</b> may perform a resynchronous operation on a region utilizing a sweeping hand method and copying information starting with the lowest block number and finishing with the highest block number as illustrated by arrow <b>508</b>. The resync engine <b>129</b> may read and write the information from the region <b>500</b> on a block-by-block basis or in groups of blocks.
<figref idref="DRAWINGS">FIG. 5B</figref> illustrates region <b>500</b> during a resynchronous operation having one or more synced blocks <b>502</b>, one or more locked blocks <b>504</b>, and one or more dirty blocks <b>506</b>. The one or more synced blocks <b>502</b> are blocks that have already been synced on the clusters during the resynchronous operation. Further, the one or more locked blocks <b>504</b> are blocks that are currently being resynced by the resynchronous operation and are locked from being written to by an outside operation. Finally, the one or more dirty blocks <b>506</b> are blocks that need to be resynced by the resynchronous operation.
<figref idref="DRAWINGS">FIG. 5B</figref> illustrates the clustered system receiving new information intended to be written to region <b>500</b> as illustrated by arrow <b>510</b>. In this exemplary embodiment, the new information is to be written to one or more blocks in the synced blocks <b>502</b> area of region <b>500</b>. In this example, the resync engine <b>129</b> may synchronously write the new information on the primary cluster and secondary cluster since these blocks are in sync across the clustered system. <figref idref="DRAWINGS">FIG. 5C</figref> illustrates new information intended to be written to region <b>500</b> in the locked blocks <b>504</b> area as illustrated by arrow <b>510</b>. In this exemplary embodiment, the resync engine <b>129</b> may wait until the resynchronous operation is complete on the blocks to be written. Then the resync engine <b>129</b> may synchronously write the new information on both the primary cluster and secondary cluster.
In another exemplary embodiment, <figref idref="DRAWINGS">FIG. 5D</figref> illustrates new information intended to be written to region <b>500</b> in the dirty blocks <b>506</b> area as illustrated by arrow <b>510</b>. In this example, the resync engine <b>129</b> may write the new information to the region <b>500</b> prior to the resynchronous operation being performed on these blocks. Since these blocks still need to be resynced on the clusters, the resync engine <b>129</b> may only write the new information to the cluster being resynced from. For example, if the resync engine <b>129</b> is copying information from the primary cluster to the secondary cluster during the resynchronous operation, the new information may be written to the primary cluster. In another example, if the resync engine <b>129</b> is copying information from the secondary cluster to the primary cluster during the resynchronous operation, the new information may be written to the secondary cluster. The new information will be synced across the clusters when the resynchronous operation is performed on the dirty blocks <b>506</b> area. Various embodiments are not limited in this manner and the resync engine <b>129</b> may synchronously write the new information to both the primary and secondary cluster.
<figref idref="DRAWINGS">FIG. 6</figref> illustrates an embodiment of logic flow <b>600</b>. The logic flow <b>600</b> may be representative of some or all of the operations executed by one or more embodiments described herein. For example, the logic flow <b>600</b> may illustrate operations performed by systems of <figref idref="DRAWINGS">FIGS. 1A-1D</figref>. However, various embodiments are not limited in this manner.
In the illustrated embodiment shown in <figref idref="DRAWINGS">FIG. 6</figref>, the logic flow <b>600</b> may include receiving information from a client device to write on a clustered system comprising at least a first cluster and a second cluster at block <b>605</b>. The clustered system may include the first cluster and second cluster to provide data integrity and system redundancy. For example, in some embodiments, the first cluster may be considered a primary cluster and the second cluster may be considered a secondary cluster. Information received by the clustered system may be synchronously written in a parallel manner to the both the primary cluster and secondary cluster to create data redundancy and a backup copy of the information.
In some embodiments, the information sent by the client device may include data and write operations to store data on the clustered system. The data may be any type of data including financial data, medical data, industry data, database data, product data, scientific data, experimental data, and so forth. In addition, the write operations may include one or more locations to store the data. For example, the write operations may include one or more block numbers of a logical unit indicating a location to store the data. The locations may be used to write the data on the clustered system, and in some instances, to track information in one or more logs.
At block <b>610</b>, the logic flow <b>600</b> may include determining that a failure event has occurred on the clustered system creating unsynchronized information, the unsynchronized information comprising at least one of inflight information and dirty region information. As previously discussed, a failure event may be any type of event that may cause the clustered system to operate in a non-optimal manner, become out of sync, and may include a hardware failure, a software failure, and/or a networking failure. During the failure event the clustered system may continue to receive information from the client device. This information may be unsynchronized and may be handled by writing information to an inflight tracker log or a dirty region log.
In some embodiments, the logic flow <b>600</b> at block <b>615</b> may include performing a resynchronization operation to synchronize the unsynchronized information on the first cluster and the second cluster based on log information in at least one of an inflight tracker log and a dirty region log. For example, the inflight tracker log may include log inflight information or information that has not been fully committed on the clustered due to the failure event or for some other reason. During the resynchronous operation, the inflight tracker log may be read, and the information stored on the primary cluster may be copied to the secondary cluster regardless as to whether the information is the newly received information from a client device, for example. In some embodiments, the dirty region log may log dirty region information or information that has been written to one of the clusters, the first or the second cluster, but not both of the clusters. Therefore, the dirty region information may need to be copied from one cluster to another cluster after the failure event has been resolved. A resynchronization operation may be performed to copy information from the primary cluster to the secondary cluster based on the inflight tracker log and the dirty region log.
<figref idref="DRAWINGS">FIG. 7</figref> illustrates an exemplary embodiment of hardware architecture of a cluster node <b>700</b>. In some embodiments, cluster node <b>700</b> may be the same or similar as one of the cluster nodes <b>108</b> included in the storage system <b>100</b>. Cluster node <b>700</b> may include processor <b>702</b>, memory <b>704</b>, storage operating system <b>706</b>, network adapter <b>708</b> and storage adapter <b>710</b>. In various embodiments, the components of cluster node <b>700</b> may communicate with each other via one or more interconnects, such as one or more traces, buses and/or control lines.
Processor <b>702</b> may be one or more of any type of computational element, such as but not limited to, a microprocessor, a processor, central processing unit, digital signal processing unit, dual core processor, mobile device processor, desktop processor, single core processor, a system-on-chip (SoC) device, complex instruction set computing (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or any other type of processor or processing circuit on a single chip or integrated circuit. In various embodiments, cluster node <b>700</b> may include more than one processor.
In one embodiment, cluster node <b>700</b> may include a memory unit <b>704</b> to couple to processor <b>702</b>. Memory unit <b>704</b> may be coupled to processor <b>702</b> via an interconnect, or by a dedicated communications bus between processor <b>702</b> and memory unit <b>704</b>, which may vary as desired for a given implementation. Memory unit <b>704</b> may be implemented using any machine-readable or computer-readable media capable of storing data, including both volatile and non-volatile memory. In some embodiments, the machine-readable or computer-readable medium may include a non-transitory computer-readable storage medium, for example. The embodiments are not limited in this context.
The memory unit <b>704</b> may store data momentarily, temporarily, or permanently. The memory unit <b>704</b> may store instructions and data for cluster node <b>700</b>. The memory unit <b>704</b> may also store temporary variables or other intermediate information while the processor <b>702</b> is executing instructions. The memory unit <b>704</b> is not limited to storing the above discussed data; the memory unit <b>704</b> may store any type of data. In various embodiments, memory <b>704</b> may store or include storage operating system <b>706</b>
In various embodiments, cluster node <b>700</b> may include storage operating system <b>706</b> to control storage operations on the cluster node <b>700</b>. In some embodiments, storage operating system <b>706</b> may be stored in memory <b>704</b> or any other type of storage device, unit, medium, and so forth. The storage operating system <b>706</b> may implement a write-anywhere file system that cooperates with virtualization modules to “virtualize” the storage space provided on the storage arrays and storage devices. The file system may logically organize the information as a hierarchical structure of named directories and files on the disks. Each “on-disk” file may be implemented as set of disk blocks configured to store information, such as data, whereas the directory may be implemented as a specially formatted file in which names and links to other files and directories are stored. The virtualization modules allow the file system to further logically organize information as a hierarchical structure of logical data blocks on the disks that are exported as logical unit numbers (LUNs).
The network adapter <b>708</b> may include the mechanical, electrical and signaling circuitry needed to connect the cluster node <b>700</b> to one or more hosts and other storage systems over a network, which may include a point-to-point connection or a shared medium, such as a local area network.
In various embodiments, the storage adapter <b>710</b> cooperates with the operating system <b>706</b> executing on the cluster node <b>700</b> to access information requested by a host device, guest device, another storage system and so forth. The information may be stored on any type of attached array of writable storage device media such as video tape, optical, DVD, magnetic tape, bubble memory, electronic random access memory, micro-electro mechanical and any other similar media adapted to store information, including data and parity information. Further, the storage adapter <b>710</b> includes input/output (I/O) interface circuitry that couples to the disks over an I/O interconnect arrangement, such as a conventional high-performance, FC serial link topology.
<figref idref="DRAWINGS">FIG. 8</figref> illustrates an embodiment of an exemplary computing architecture <b>800</b> suitable for implementing various embodiments as previously described. In one embodiment, the computing architecture <b>800</b> may include or be implemented as part of computing system, such as storage systems <b>100</b>, <b>125</b>, <b>150</b> and <b>175</b>.
As used in this application, the terms “system” and “component” are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution, examples of which are provided by the exemplary computing architecture <b>800</b>. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical and/or magnetic storage medium), an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers. Further, components may be communicatively coupled to each other by various types of communications media to coordinate operations. The coordination may involve the uni-directional or bi-directional exchange of information. For instance, the components may communicate information in the form of signals communicated over the communications media. The information can be implemented as signals allocated to various signal lines. In such allocations, each message is a signal. Further embodiments, however, may alternatively employ data messages. Such data messages may be sent across various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.
The computing architecture <b>800</b> includes various common computing elements, such as one or more processors, multi-core processors, co-processors, memory units, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input/output (I/O) components, power supplies, and so forth. The embodiments, however, are not limited to implementation by the computing architecture <b>800</b>.
As shown in <figref idref="DRAWINGS">FIG. 8</figref>, the computing architecture <b>800</b> includes a processing unit <b>804</b>, a system memory <b>806</b> and a system bus <b>808</b>. The processing unit <b>804</b> can be any of various commercially available processors.
The system bus <b>808</b> provides an interface for system components including, but not limited to, the system memory <b>806</b> to the processing unit <b>804</b>. The system bus <b>808</b> can be any of several types of bus structure that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. Interface adapters may connect to the system bus <b>808</b> via slot architecture. Example slot architectures may include without limitation Accelerated Graphics Port (AGP), Card Bus, (Extended) Industry Standard Architecture ((E)ISA), Micro Channel Architecture (MCA), NuBus, Peripheral Component Interconnect (Extended) (PCI(X)), PCI Express, Personal Computer Memory Card International Association (PCMCIA), and the like.
The computing architecture <b>800</b> may include or implement various articles of manufacture. An article of manufacture may include a computer-readable storage medium to store logic. Examples of a computer-readable storage medium may include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. Examples of logic may include executable computer program instructions implemented using any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like. Embodiments may also be at least partly implemented as instructions contained in or on a non-transitory computer-readable medium, which may be read and executed by one or more processors to enable performance of the operations described herein.
The system memory <b>806</b> may include various types of computer-readable storage media in the form of one or more higher speed memory units, such as read-only memory (ROM), random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM (DDRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, polymer memory such as ferroelectric polymer memory, ovonic memory, phase change or ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, magnetic or optical cards, an array of devices such as Redundant Array of Independent Disks (RAID) drives, solid state memory devices (e.g., USB memory, solid state drives (SSD) and any other type of storage media suitable for storing information. In the illustrated embodiment shown in <figref idref="DRAWINGS">FIG. 8</figref>, the system memory <b>806</b> can include non-volatile memory <b>810</b> and/or volatile memory <b>812</b>. A basic input/output system (BIOS) can be stored in the non-volatile memory <b>810</b>.
The computer <b>802</b> may include various types of computer-readable storage media in the form of one or more lower speed memory units, including an internal (or external) hard disk drive (HDD) <b>814</b>, a magnetic floppy disk drive (FDD) <b>816</b> to read from or write to a removable magnetic disk <b>818</b>, and an optical disk drive <b>820</b> to read from or write to a removable optical disk <b>822</b> (e.g., a CD-ROM or DVD). The HDD <b>814</b>, FDD <b>816</b> and optical disk drive <b>820</b> can be connected to the system bus <b>808</b> by a HDD interface <b>824</b>, an FDD interface <b>826</b> and an optical drive interface <b>828</b>, respectively. The HDD interface <b>824</b> for external drive implementations can include at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.
The drives and associated computer-readable media provide volatile and/or nonvolatile storage of data, data structures, computer-executable instructions, and so forth. For example, a number of program modules can be stored in the drives and memory units <b>810</b>, <b>812</b>, including an operating system <b>830</b>, one or more application programs <b>832</b>, other program modules <b>834</b>, and program data <b>836</b>. In one embodiment, the one or more application programs <b>832</b>, other program modules <b>834</b>, and program data <b>836</b> can include, for example, the various applications and/or components of the system <b>100</b>.
A user can enter commands and information into the computer <b>802</b> through one or more wire/wireless input devices, for example, a keyboard <b>838</b> and a pointing device, such as a mouse <b>840</b>. Other input devices may include microphones, infra-red (IR) remote controls, radio-frequency (RF) remote controls, game pads, stylus pens, card readers, dongles, finger print readers, gloves, graphics tablets, joysticks, keyboards, retina readers, touch screens (e.g., capacitive, resistive, etc.), trackballs, trackpads, sensors, styluses, and the like. These and other input devices are often connected to the processing unit <b>804</b> through an input device interface <b>842</b> that is coupled to the system bus <b>808</b>, but can be connected by other interfaces such as a parallel port, IEEE 1394 serial port, a game port, a USB port, an IR interface, and so forth
A monitor <b>844</b> or other type of display device is also connected to the system bus <b>808</b> via an interface, such as a video adaptor <b>846</b>. The monitor <b>844</b> may be internal or external to the computer <b>802</b>. In addition to the monitor <b>844</b>, a computer typically includes other peripheral output devices, such as speakers, printers, and so forth.
The computer <b>802</b> may operate in a networked environment using logical connections via wire and/or wireless communications to one or more remote computers, such as a remote computer <b>848</b>. The remote computer <b>848</b> can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer <b>802</b>, although, for purposes of brevity, only a memory/storage device <b>850</b> is illustrated. The logical connections depicted include wire/wireless connectivity to a local area network (LAN) <b>852</b> and/or larger networks, for example, a wide area network (WAN) <b>854</b>. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global communications network, for example, the Internet.
When used in a LAN networking environment, the computer <b>802</b> is connected to the LAN <b>852</b> through a wire and/or wireless communication network interface or adaptor <b>856</b>. The adaptor <b>856</b> can facilitate wire and/or wireless communications to the LAN <b>852</b>, which may also include a wireless access point disposed thereon for communicating with the wireless functionality of the adaptor <b>856</b>.
When used in a WAN networking environment, the computer <b>802</b> can include a modem <b>858</b>, or is connected to a communications server on the WAN <b>854</b>, or has other means for establishing communications over the WAN <b>854</b>, such as by way of the Internet. The modem <b>858</b>, which can be internal or external and a wire and/or wireless device, connects to the system bus <b>808</b> via the input device interface <b>842</b>. In a networked environment, program modules depicted relative to the computer <b>802</b>, or portions thereof, can be stored in the remote memory/storage device <b>850</b>. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used.
The computer <b>802</b> is operable to communicate with wire and wireless devices or entities using the IEEE 802 family of standards, such as wireless devices operatively disposed in wireless communication (e.g., IEEE 802.11 over-the-air modulation techniques). This includes at least Wi-Fi (or Wireless Fidelity), WiMax, and Bluetooth™ wireless technologies, among others. Thus, the communication can be a predefined structure as with a conventional network or simply an ad hoc communication between at least two devices. Wi-Fi networks use radio technologies called IEEE 802.11x (a, b, g, n, etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wire networks (which use IEEE 802.3-related media and functions).
The various elements of the storage system <b>100</b>, <b>125</b>, <b>150</b>, and <b>175</b> as previously described with reference to <figref idref="DRAWINGS">FIGS. 1-8</figref> may include various hardware elements, software elements, or a combination of both. Examples of hardware elements may include devices, logic devices, components, processors, microprocessors, circuits, processors, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), memory units, logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. Examples of software elements may include software components, programs, applications, computer programs, application programs, system programs, software development programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. However, determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints, as desired for a given implementation.
Some embodiments may be described using the expression “one embodiment” or “an embodiment” along with their derivatives. These terms mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. Further, some embodiments may be described using the expression “coupled” and “connected” along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, some embodiments may be described using the terms “connected” and/or “coupled” to indicate that two or more elements are in direct physical or electrical contact with each other. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other.
It is emphasized that the Abstract of the Disclosure is provided to allow a reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein,” respectively. Moreover, the terms “first,” “second,” “third,” and so forth, are used merely as labels, and are not intended to impose numerical requirements on their objects.
What has been described above includes examples of the disclosed architecture. It is, of course, not possible to describe every conceivable combination of components and/or methodologies, but one of ordinary skill in the art may recognize that many further combinations and permutations are possible. Accordingly, the novel architecture is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims.
Contents4
15 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US10331363B2 | Cited by | United States of America | Applicant |
| US10684994B2 | Cited by | United States of America | Search report |
| US2017091299A1 | Cited by | United States of America | Search report |
| US11928005B2 | Cited by | United States of America | Applicant |
| US11461280B2 | Cited by | United States of America | Applicant |
| US11341101B2 | Cited by | United States of America | Search report |
| US11468014B2 | Cited by | United States of America | Applicant |
| US2012166390A1 | Cites | United States of America | Search report |
| US2013110782A1 | Cites | United States of America | Search report |
| US6823349B1 | Cites | United States of America | Search report |
| US7039661B1 | Cites | United States of America | Search report |
| US7383264B2 | Cites | United States of America | Search report |
| US7409587B2 | Cites | United States of America | Search report |
| US7702906B1 | Cites | United States of America | Search report |
| US7925633B2 | Cites | United States of America | Search report |
| US8396830B2 | Cites | United States of America | Search report |
| US9317576B2 | Cites | United States of America | Search report |
| US20120166390A1 | Cites | United States of America | Search report |
| US20130110782A1 | Cites | United States of America | Search report |
7 members in 1 office
Priority claims2
| Document | Office | Kind | Date |
|---|---|---|---|
| 201414518422 | United States of America | A | |
| US201414518422 | – | – | – |
Members7
| Document | Office | Kind | |
|---|---|---|---|
| US2016110378A1 | United States of America | A1 | |
| US9720752B2This record | United States of America | B2 | |
| US2017322837A1 | United States of America | A1 | |
| US10379919B2 | United States of America | B2 | |
| US2019324829A1 | United States of America | A1 | |
| US11449373B2 | United States of America | B2 | |
| US2023016882A1 | United States of America | A1 |
52 transactions on the USPTO file
Allowed after 1 non-final rejection.
- Non-final rejections
- 1
- Final rejections
- 0
- RCEs
- 0
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Recordation of Patent Grant MailedPGM/ | PGM/ | |
| Patent Issue Date Used in PTA CalculationAllowedPTAC | PTAC | |
| Email NotificationEML_NTR | EML_NTR | |
| Issue Notification MailedAllowedWPIR | WPIR | |
| Email NotificationEML_NTR | EML_NTR | |
| Mail Response to 312 Amendment (PTO-271)MN271 | MN271 | |
| Dispatch to FDCD1935 | D1935 | |
| Response to Amendment under Rule 312N271 | N271 | |
| Pubs Case Remand to TCPUBTC | PUBTC | |
| Application Is Considered Ready for IssuePILS | PILS | |
| Amendment after Notice of Allowance (Rule 312)AllowedA.NA | A.NA | |
| Issue Fee Payment VerifiedN084 | N084 | |
| Issue Fee Payment ReceivedIFEE | IFEE | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Notice of AllowanceAllowedMN/=. | MN/=. | |
| Notice of Allowance Data Verification CompletedAllowedN/=. | N/=. | |
| Reasons for AllowanceEX.R | EX.R | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Non-Final ActionA... | A... | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Email NotificationEML_NTR | EML_NTR | |
| Application ready for PDX access by participating foreign officesCCRDY | CCRDY | |
| PG-Pub Issue NotificationPG-ISSUE | PG-ISSUE | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Email NotificationEML_NTR | EML_NTR | |
| Change in Power of Attorney (May Include Associate POA)PA.. | PA.. | |
| Correspondence Address ChangeC.AD | C.AD | |
| Email NotificationEML_NTR | EML_NTR | |
| Application Is Now CompleteCOMP | COMP | |
| Filing Receipt - UpdatedFLRCPT.U | FLRCPT.U | |
| Sent to Classification ContractorPGPC | PGPC | |
| FITF set to YES - revise initial settingFTFS | FTFS | |
| Patent Term Adjustment - Ready for ExaminationPTA.RFE | PTA.RFE | |
| Additional Application Filing FeesADDFLFEE | ADDFLFEE | |
| Applicant has submitted a new specification to correct Corrected Papers problemsCORRSPEC | CORRSPEC | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Email NotificationEML_NTR | EML_NTR | |
| Corrected PaperCPAP | CPAP | |
| Filing ReceiptFLRCPT.O | FLRCPT.O | |
| Cleared by OIPE CSRL194 | L194 | |
| Oath or Declaration Filed (Including Supplemental)C602 | C602 | |
| Applicants have given acceptable permission for participating foreignAPPERMS | APPERMS | |
| IFW Scan & PACR Auto Security ReviewSCAN | SCAN | |
| Entity status set to undiscounted (initial default setting or status change)BIG. | BIG. | |
| Initial Exam Team nnIEXX | IEXX |
5 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Maintenance fee paymentMAFP | MAFP | |
| Maintenance fee paymentMAFP | MAFP | |
| Information on status: patent grantGrantedSTCF | STCF | |
| Information on status: patent grantGrantedSTCF | STCF | |
| AssignmentAS | AS |
Numbers
- Publication
- 09720752
- Publication, DOCDB
- 9720752
- Publication, EPODOC
- US9720752
- Application
- 14518422
- Application, DOCDB
- 201414518422
- Application, EPODOC
- US201414518422
Titles
- English
- Techniques for performing resynchronization on a clustered system
Patent term adjustment
- A delay
- +263 daysthe office missed an examination deadline
- Applicant delay
- −19 days
- Net adjustment
- 244 days
Classification
- CPC, 6
- G06F11/006
- G06F11/2097
- G06F11/00
- G06F2201/80
- G06F11/2038
- G06F11/2048
- IPC, 2
- G06F17 30
- G06F11 00
- USPC, 1
- 001001000