Systems and methods for stored data verification

RELATED APPLICATIONS

p-0002This application is a continuation of U.S. patent application Ser. No. 10/990,286, filed Nov. 15, 2004, which claims the benefit of U.S. Provisional patent application No. 60/520,452 titled SYSTEM AND METHOD FOR STORED DATA ARCHIVE VERIFICATION, filed Nov. 13, 2003, each of which is hereby incorporated herein by reference in its entirety.

p-0003This application is also related to the following pending applications, each of which is hereby incorporated herein by reference in its entirety: <ul><li id="ul0001-0001" num="0003">U.S. Pat. No. 6,418,478, titled PIPELINED HIGH SPEED DATA TRANSFER MECHANISM, issued Jul. 9, 2002;</li><li id="ul0001-0002" num="0004">U.S. Provisional Patent Application Ser. No. 60/460,234, titled SYSTEM AND METHOD FOR PERFORMING STORAGE OPERATIONS IN A COMPUTER NETWORK, filed Apr. 3, 2003;</li><li id="ul0001-0003" num="0005">U.S. patent application Ser. No. 10/818,749, titled SYSTEM AND METHOD FOR DYNAMICALLY PERFORMING STORAGE OPERATIONS IN A COMPUTER NETWORK, filed Apr. 5, 2004, now U.S. Pat. No. 7,246,207, issued Jul. 17, 2007;</li><li id="ul0001-0004" num="0006">U.S. patent application Ser. No. 10/260,209, titled SYSTEM AND METHOD FOR ARCHIVING OBJECTS IN AN INFORMATION STORE, filed Sep. 30, 2002, now U.S. Pat. No. 7,107,298, issued Sep. 16, 2006; and</li><li id="ul0001-0005" num="0007">U.S. patent application Ser. No. 09/495,751, titled HIGH SPEED DATA TRANSFER MECHANISM, filed Feb. 1, 2000, now U.S. Pat. No. 7,209,972, issued Apr. 24, 2007.</li></ul>

COPYRIGHT NOTICE

p-0004A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosures, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.

BACKGROUND

p-00051. Field

p-0006The invention disclosed herein relates generally to data storage systems in computer networks and, more particularly, to improvements in storing and verifying accurate archiving of electronic data.

p-00072. Description of the Related Art

p-0008Storage architectures used by individual computers or data stores to store electronic data typically include volatile storage media such as Random Access Memory “RAM”, and one or more nonvolatile storage devices such as hard drives, tape drives, optical disks, and other storage devices that form a part of or are directly associated with an individual computer. A network of computers such as a Local Area Network “LAN” or a Wide Area Network “WAN”, typically store electronic data via servers or stand-alone storage devices accessible via the network. Stand-alone storage devices are generally connected to one individual computer or a network of computers. Network storage devices commonly known in the art typically include physical drives in which tapes or other storage media are stored and a robotic arm which is used to place the tapes or storage media into the drives. Examples of network storage devices include networkable tape drives, optical libraries, Redundant Arrays of Inexpensive Disks “RAID”, CD-ROM jukeboxes, and other devices.

p-0009Electronic data is typically archived according to a schedule, for example, data is designated to be copied and stored once a day. Generally, data is archived in the event that an original copy becomes unavailable, for example, the data is destroyed, lost or otherwise inaccessible. In general, the data is directed to a system component to be copied or backed up to storage media, such as an auxiliary copy, copy backup or quick recovery copy. Some systems check the copy or backup to ensure the copy or backup is accurate. Generally, the accuracy check includes steps such as analyzing each data item copied and comparing it to the original data, or other method. Such verification methods can be lengthy and time consuming for copies of large volumes of data, requiring significant use of system resources. Alternatively, some systems use cursory data checks, such as comparing filenames, which is less time consuming, however, it yields a less reliable data check.

SUMMARY

p-0010The present invention addresses the problems discussed above, and includes a method for verifying stored data. Methods and systems are described for verifying stored data. The system includes components such as a storage manager, media agent, storage device, jobs manager, archive check manager, index, copy manager, client computer, and data store. The storage system verifies data by receiving a first set of metadata associated with a first set of stored data, generating a second set of metadata associated with a second set of stored data, and comparing the first set of metadata and second set of metadata. Alternatively, the storage system generates a first set of metadata associated with a first set of stored data, generates a second set of stored data associated with the first set of stored data, generates a second set of metadata associated with the second set of stored data, and compares the first set of metadata and the second set of metadata.

p-0011The method involves receiving a first set of metadata associated with a first set of stored data. The first set of metadata is typically generated based on the first set of stored data, or otherwise associated with the first set of stored data. The first set of metadata is received, for example, from a local system component, or external source. The first set of metadata may also be received in response to a retrieval storage operation initiated by a system component. The method generates a second set of metadata associated with a second set of stored data. Preferably, the second set of metadata is generated according to the same procedure used to generate or create the first set of metadata. Additionally, it is preferred that the second set of stored data be data that is associated with the first set of stored data, such as a backup, auxiliary copy, or other copy. The method compares the first set of metadata and the second set of metadata. In general, a comparison of the first set of metadata and the second set of metadata can reveal that the metadata sets are the same, or different. If the comparison reveals that the first and second sets of metadata are the same, then generally, the first set of stored data and the second set of stored data are also the same, and there was an accurate copy made in the second set of stored data. If the comparison of the first and second sets of metadata reveals differences, then the first and second sets of stored data are not the same, and there may have been an error in creating the second set of stored data. In the event that inconsistencies are found between the first and second sets of metadata, the method may initiate a storage operation, such as a copy or backup the first set of stored data.

p-0012In another embodiment of the invention, the method for verifying stored data includes performing a storage operation, for example, a copy or backup operation, or other storage operation as further described herein. In performing the storage operation, a first set of metadata is generated and is associated with a first set of stored data. Preferably, the first set of metadata is generated based on the first set of stored data. The first set of stored data is preferably an original or primary copy, or other original set of data. Additionally, a second set of stored data is generated and is associated with the first set of stored data. Preferably, the second set of stored data is a backup, auxiliary copy, other copy, or otherwise based on the first set of stored data. The method generates a second set of metadata based on the second set of stored data. Preferably the second set of metadata is generated according to the same procedure as the first set of metadata. The first set of metadata and the second set of metadata are compared. The comparison of the first set of metadata and the second set of metadata provide an accuracy check for the first set of stored data and the second set of stored data.

BRIEF DESCRIPTION OF THE DRAWINGS

p-0013The invention is illustrated in the figures of the accompanying drawings which are meant to be exemplary and not limiting, in which like references are intended to refer to like or corresponding parts, and in which:

p-0014<figref idrefs="DRAWINGS">FIG. 1</figref> is block diagram showing a high-level view of the storage network architecture and components according to an embodiment of the invention;

p-0015<figref idrefs="DRAWINGS">FIG. 2</figref> is a flow diagram for verifying stored data according an embodiment of the invention;

p-0016<figref idrefs="DRAWINGS">FIG. 3</figref> is a flow diagram for verifying stored data according to an embodiment of the invention;

p-0017<figref idrefs="DRAWINGS">FIG. 4</figref> is a block diagram showing a chunk, according to an embodiment of the invention;

p-0018<figref idrefs="DRAWINGS">FIG. 5</figref> is a flow diagram for a storage operation according to an embodiment of the invention;

p-0019<figref idrefs="DRAWINGS">FIG. 6</figref> is a flow diagram for verifying stored data, according to an embodiment of the invention;

p-0020<figref idrefs="DRAWINGS">FIG. 7</figref> is a block diagram showing a stored data verification table according to an embodiment of the invention; and

p-0021<figref idrefs="DRAWINGS">FIG. 8</figref> is a detailed flow diagram of a step of the flow diagram of <figref idrefs="DRAWINGS">FIG. 6</figref> for verifying stored data, according to an embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

p-0022Preferred embodiments of the invention are now described with reference to the drawings. An embodiment of the system of the present invention is shown in <figref idrefs="DRAWINGS">FIG. 1</figref>. The system and components of the system as presented in <figref idrefs="DRAWINGS">FIG. 1</figref> are exemplary of a modular storage system such as the COMMVAULT GALAXY backup and retrieval system and QINETIX storage management system, available from CommVault Systems, Inc. of Oceanport, N.J., and further described in U.S. Provisional Patent Application No. 60/460,234, titled SYSTEM AND METHOD FOR PERFORMING STORAGE OPERATIONS IN A COMPUTER NETWORK, filed Apr. 3, 2003, which is incorporated herein by reference in its entirety.

p-0023As shown in <figref idrefs="DRAWINGS">FIG. 1</figref>, the system includes a data store <b>100</b>, a client computer <b>110</b>, a storage manager (or storage manager component) <b>120</b>, an SM index <b>130</b>, a jobs manager <b>140</b>, an archive check manager <b>150</b>, one or more media agents (or media management components) <b>160</b>, an MA index <b>170</b>, a copy manager <b>180</b>, and one or more storage devices <b>190</b>. The data store <b>100</b> contains data generated by a typical enterprise machine, such as client computer <b>110</b>. For example, a data store <b>100</b> may contain ORACLE data for a database, or mail server data. In the event that a data store <b>100</b> loses data, the data store <b>100</b> obtains the lost data from a backup or other data copy. The data store <b>100</b> is generally in communication with the client computer <b>110</b>

p-0024The client computer <b>110</b> may be a single client computer <b>110</b> or a plurality of client computers <b>110</b>. Client computer <b>110</b> generally refers to a computer with data, e.g. a primary data set, which may be backed up, such as a personal computer, a workstation, a server computer, a host computer, etc. The client computer <b>110</b> typically also includes software to facilitate data transfers from the data store <b>100</b> to the other system components. The client computer <b>110</b> is generally in communication with the storage manager <b>120</b>. In some embodiments of the invention, the client computer <b>110</b> is also in communication with the media agent <b>160</b>.

p-0025The storage manager <b>120</b> is generally a software module or modules capable of coordinating and controlling a storage system. In preferred embodiments, the storage manager <b>120</b> contains an SM index <b>130</b>, a jobs manager <b>140</b>, and an archive check manager <b>150</b>. In other embodiments of the invention, the SM index <b>130</b>, jobs manager <b>140</b>, or archive check manager <b>150</b> may exist independently within the storage system, or reside within different components of the storage system, such as the media agent <b>160</b>. The storage manager <b>120</b> communicates directly the client computer <b>110</b> and media agent <b>160</b> to initiate, perform and manage system storage operations. Storage operations include, but are not limited to, creation, storage, retrieval, migration, deletion, and tracking of primary or production volume data, secondary volume data, primary copies, secondary copies, auxiliary copies, snapshot copies, backup copies, incremental copies, differential copies, HSM copies, archive copies, Information Lifecycle Management (“ILM”) copies, and other types of copies and versions of electronic data. The storage manager <b>120</b> manages system storage operations, and together with the job manager <b>140</b> and archive check manager <b>150</b>, manages and directs data verification, including, for example, updating the data verification information to SM index <b>130</b>.

p-0026The storage manager <b>120</b> includes SM index <b>130</b>, which is generally any storage index that is maintained to store index data for storage operations, such as data verification operations. Data stored to the SM index <b>130</b> in data verification operations includes for example, data copied or backed up, the metadata generated in association with data sets, the results of verification checks, and other information generated by the system during storage operations. The SM index <b>130</b> can also be a storage index known in the art, or other storage index, such as, the index cache of CommVault's GALAXY as further described in U.S. patent application Ser. No. 10/260,209, now U.S. Pat. No. 7,107,298, which is incorporated herein by reference in its entirety. The SM index <b>130</b> stores data related to the storage system components, including information related to each data set, copies, backups, metadata, or other information. Index data is useful because, among other benefits, it provides the system with an efficient mechanism for locating information associated with performing storage operations.

p-0027The storage manager <b>120</b> also includes the jobs manager <b>140</b>, which is a module (or modules) that schedules and monitors storage operations that are to be performed, are being performed, or have been completed by the system. The jobs manager <b>140</b> can schedule storage operations automatically, for example, based on storage policies. Storage operations can also be scheduled manually, for example, a user may enter an input to the jobs manager <b>140</b>, using, for example, a user interface to schedule a storage operation. The jobs manager <b>140</b> directs the archive check manager <b>150</b> to initiate data verification operations, as further described herein. The jobs manager <b>140</b> is preferably included in the storage manager <b>120</b>, but may exist independently in the system, or within another system component, such as media agent <b>160</b>.

p-0028A storage policy is generally a data structure or other information which includes a set of preferences and other storage criteria for performing a storage operation. The preferences and storage criteria may include, but are not limited to: a storage location, relationships between system components, network pathway to utilize, retention policies, data characteristics, compression or encryption requirements, preferred system components to utilize in a storage operation, and other criteria relating to a storage operation. A storage policy may be stored to a storage manager index, to archive media as metadata for use in restore operations or other storage operations, or to other locations or components of the system.

p-0029The archive check manager <b>150</b> is also included in the storage manager <b>120</b>, and is a module (or modules) that manages data verification operations within the system, as further described herein. For example, the archive check manager <b>150</b> identifies data to be verified in the data verification operation and manages the data verification operation. The archive check manager <b>150</b> is preferably included in the storage manager <b>120</b>, but may exist independently in the system, or within another system component, such as media agent <b>160</b>.

p-0030The media agent <b>160</b> is generally a module or modules capable of managing storage operations, such as copying and backups, or other storage-related operation. The media agent <b>160</b> preferably includes MA index <b>170</b> and copy manager <b>180</b>. The media agent <b>160</b> is in communication with the storage manager <b>120</b> and storage device <b>190</b>, for example, via a local connection or a network. In some embodiments of the invention, the media agent <b>160</b> is also in communication with the client computer <b>110</b>. The media agent <b>160</b> is capable of communicating instructions from a storage manager <b>120</b> to a storage device <b>190</b> to undertake a storage operation.

p-0031Additionally, the media agent <b>160</b> is capable of directing the copy manager <b>180</b>, which is responsible for initiating and performing a copy operation. The media agent <b>160</b> stores data related to storage operations and media, such as auxiliary copies, to MA index <b>170</b>. Examples of data stored by the media agent <b>160</b> to the MA index <b>170</b> include metadata or other data associated with storage operations.

p-0032The MA index <b>170</b> may be an index, such as SM index <b>130</b> described herein, the media agent index described in U.S. Provisional Patent Application No. 60/460,234, titled SYSTEM AND METHOD FOR PERFORMING STORAGE OPERATIONS IN A COMPUTER NETWORK, filed Apr. 3, 2003, or other storage index known in the art. The copy manager <b>180</b> is a module or modules used to initiate, manage and perform copy operations or other copy operations. A copy operation is a storage operation that generally includes creating a duplicate copy of data for storage. Copies are typically made for disaster recovery purposes.

p-0033The copy manager <b>180</b> preferably resides within the media agent <b>160</b>, but can also exist as an independent system component, or within another system component.

p-0034The storage device <b>190</b> is generally one or more devices as described herein. The storage device <b>190</b> can be a CommVault GALAXY storage device, or other storage device capable of performing a storage operation. Storage device <b>190</b> typically includes components such as media, slots, media drives and a robotic arm or arms.

p-0035Data generated by the system or otherwise obtained by the system, is typically received by data store <b>100</b>, or client computer <b>110</b>. A copy of the data, for example, an auxiliary copy or other copy, is made according to storage policies, as further described herein. When data is copied, the copy is generally transferred to the storage device <b>190</b> via the storage manager <b>120</b> or media agent <b>160</b>. The data copy is checked to verify that the contents of the copy are identical to the original data.

p-0036Referring to <figref idrefs="DRAWINGS">FIG. 2</figref>, which depicts a flow chart of a method of an embodiment of the invention for verifying data, a first set of metadata associated with a first set of stored data is received, step <b>200</b>. Generally, when a copy of the original or first set of data is created, a set of metadata related to the first set of data is created. In preferred embodiments, the metadata is generated by the copy manager, or other system component, and is received by a media agent, copy manager or other system component. The first set of metadata is associated with a first set of stored data, for example, the first set of metadata is generated based on the first set of stored data. In preferred embodiments, the first set of stored data is an original set of data, primary copy, other original data copy or verified copy. The first set of metadata is preferably associated with the first set of stored data according to a procedure known to the storage system. For example, the first set of metadata can be generated according to a routine based on the first set of stored data, obtained using variables based on the first set of stored data, generated as a chunk header, as further described herein, or other procedure for obtaining metadata based on the first set of stored data, or other data. In preferred embodiments, the first set of metadata is stored to an index, such as the SM index or MA index.

p-0037A second set of metadata associated with a second set of stored data is generated, step <b>210</b>. In preferred embodiments, the second set of stored data is associated with the first set of stored data. For example, the second set of stored data is typically a backup, auxiliary copy or other copy of the first set of stored data. The second set of metadata is generated based on the second set of stored data. Preferably, the procedure for generating the second set of metadata is the same procedure used to generate the first set of metadata. In one embodiment of the invention, the second set of metadata is generated by a copy manager, a media agent, or other system component. In some embodiments, the second set of metadata is generated in a partial copy operation, wherein, a set of data is read, metadata based on the set of data is generated, but no copy of the data file is made. In another embodiment of the invention, the second set of metadata is generated by the archive check manager. In preferred embodiments, the second set of metadata is stored to an index, such as the SM index or MA index.

p-0038The first set of metadata and the second set of metadata are compared, step <b>220</b>. The first set of metadata and the second set of metadata are compared to determine whether the sets of metadata are similar, equivalent, or otherwise indicate that the underlying data which generated the metadata are substantially similar. If the comparison of the first set of metadata and the second set of metadata indicates that the sets of metadata are similar, the underlying data used to generate the first and second sets of metadata, in this case, the first and second sets of stored data, are substantially similar. Thus, the second set of stored data is determined to be an accurate or true copy of the first set of stored data. When the first set of metadata and the second set of metadata are determined to be different or inconsistent, or otherwise indicate that the underlying data used to generate the first and second sets of metadata are different, the first and second sets of stored data may not be equivalent, and therefore, the second set of stored data is likely an inaccurate or otherwise defective copy of the first set of stored data.

p-0039In preferred embodiments, a copy manager compares the first set of metadata and the second set of metadata. In another embodiment, an archive check manager compares the first set of metadata and the second set of metadata. The first set of metadata and the second set of metadata are compared to determine whether the sets of metadata are the same. In some embodiments, one or both of the sets of metadata are obtained from an index, such as the MA index, SM index, or are generated as needed, for example, by performing a partial storage operation, further described herein, or other method for obtaining metadata.

p-0040Referring to <figref idrefs="DRAWINGS">FIG. 3</figref>, which depicts a flowchart of another embodiment of the invention, a storage operation is initiated, step <b>250</b>. The storage operation is normally initiated by a storage manager, for example, by a jobs manager, which, as described herein, schedules storage operations, for example, based on storage policies. In other embodiments, other system components initiate system storage operations. The jobs manager, or other system component, schedules storage operations and directs an archive check manager, or other system component, to initiate a storage operation. The storage operation initiated at step <b>250</b> is preferably an operation similar to a copy operation. For example, a data file in a client data store is copied. A copy manager, or other system component, generates a first set of metadata associated with a first set of stored data, step <b>260</b>. For example, the copy manager initiates a copy operation, and obtains the first set of stored data. The copy manager reads the first set of stored data and generates a first set of metadata. As described herein, the first set of metadata is generated according to a procedure, known to the system, such as according to a routine based on the first set of stored data, or obtained using a variable based on the first set of stored data, generated as a chunk header, or other procedure for obtaining metadata. The first set of stored data is generally an original set of data, such as a primary copy, other original copy or verified copy.

p-0041The copy manager or other system component generates a second set of stored data associated with first set of stored data, step <b>270</b>. Preferably, the second set of stored data is a backup, auxiliary copy, or other copy of the first set of stored data. The second set of stored data is generally obtained by directing the copy manager to read the first set of stored data and write a copy, such as the second set of stored data, but may also be generated using other copy methods and system components. The copy manager, or other system component, generates a second set of metadata associated with the second set of stored data, step <b>280</b>. Specifically, the second set of metadata is generated based on the second set of stored data. The second set of metadata is preferably obtained using the same procedure used to generate the first set of metadata in step <b>260</b>, or other procedure for generating metadata, for example, generating metadata based on data from a primary copy. The copy manager or other system component, compares the first set of metadata and the second set of metadata, step <b>290</b>. The first set of metadata and the second set of metadata are compared to determine whether the first set of metadata and the second set of metadata are equivalent. As described herein, the comparison of the first and second sets of metadata may be used as an indicator of whether the second set of stored data is an accurate copy of the first set of stored data.

p-0042When data is copied, for example, when the media agent or copy manager copies a first set of stored data and creates a second set of stored data, the data is generally copied and transferred between system components using chunks, such as the system and methods described in U.S. Pat. No. 6,418,478, titled PIPELINED HIGH SPEED DATA TRANSFER MECHANISM, issued Jul. 9, 2002. Chunks are useful because they can be used to assemble data into packages that are transmitted across system networks, for example according to system resource availability.

p-0043Referring to <figref idrefs="DRAWINGS">FIG. 4</figref>, which depicts a chunk according to an embodiment of the invention, chunks are generally a fixed file size, such as 512 MB, and include a chunk header <b>300</b> and a chunk payload <b>310</b>. The chunk header <b>300</b> includes data or information such as an archive file ID <b>320</b>, a chunk sequence number <b>330</b>, a chunk volume <b>340</b>, an offset <b>350</b>, a chunk volume file mark number <b>360</b>, and other information <b>370</b>. A chunk payload <b>310</b> includes data files or a subset of data files that were copied, e.g. an initial or first set of data. Each of the data items or information included in the chunk header <b>300</b> is related to the file data <b>380</b>-<b>389</b> in the chunk payload <b>310</b>. The chunk header <b>300</b> data items include metadata generally derived from the stored data or files in the chunk payload <b>310</b>. For example, a copy manager or other system component which creates, packages, or otherwise handles chunks, reads the data contained in the chunk payload <b>310</b> to generate metadata or other information in the chunk header <b>300</b>.

p-0044The archive file ID <b>320</b> is an alphanumeric identifier for the file being archived, for example, in the payload <b>310</b>. The chunk sequence number <b>330</b> provides the sequence information for the chunk. The chunk volume <b>340</b> indicates the volume of the chunk utilized, for example, the volume of the data or files on the chunk. Offset <b>350</b> indicates the distance the data item is offset, for example, which is information related to, and used in reassembling the chunk files. Chunk volume file mark number <b>360</b> refers to information related to the volume of the file. The chunk header <b>300</b> also contains other information <b>370</b>, which may be data, variables, information or other metadata related to files <b>380</b>-<b>389</b> in the payload <b>310</b>. Files <b>380</b>-<b>389</b> of the payload represent that a subset of a data file, a set of data, a file or a block, or several sets of data, files or blocks may be stored in a chunk payload, for example, file 1 to file n. The information contained in the chunk header <b>300</b> is preferably stored to an index, such as an MA index or SM index. Storing the information in the chunk header <b>300</b> to the index is useful so that the information can be readily accessed by a media agent, storage manager, copy manager or other system component.

p-0045Referring to <figref idrefs="DRAWINGS">FIG. 5</figref>, which depicts a flow diagram of a storage operation, a storage operation is initiated, step <b>400</b>. Preferably the storage operation is initiated by a storage manager, or storage manager components, such as a jobs manager, an archive check manager, a media agent, or media agent components, such as a copy manager or other system component. The storage operations initiated at step <b>400</b> include, for example, system backups, media movement, imports, exports, migrations, restores, data verification and other storage-related operations. Files are identified for the storage operation, step <b>410</b>. The files are identified by the media agent, storage manager, archive check manager, copy manager, or other system component. In some embodiments, the files may be identified according to storage policies associated with a particular file. As further described herein, information related to files and storage policies are stored in, and may be obtained from an index, such as an SM index or MA index, or other locations within the system.

p-0046In general, the storage operation is assigned to a particular media agent to manage the operation. The media agent is typically associated with the file, storage device, or other system component utilized in the storage operation. Additionally, the media agent may be designated to manage the operation by storage policies. In some embodiments, the media agent may be dynamically reassigned during a storage operation according to storage policies, system resource availability or other variable. The copy manager used in storage operations is typically the copy manager associated with the media agent assigned to manage the operation. It is understood that the copy manager may be dynamically reassigned during the storage operation, for example, when its media agent is reassigned, or according to system resource availability or other variables.

p-0047A chunk header is generated for the files identified in step <b>410</b>, and the files are packaged into chunks, step <b>420</b>. The chunk header is generated according to a procedure known to the system, such as the procedure described herein in connection with generating metadata. For example, the chunk header is generated according to specific terms or variables based on the files, such as the data or information items included in the chunk header. Preferably, the copy manager creates the chunk header. The files are packaged into chunks, for example, into a payload. The chunk header information is updated to an index, such as an SM index or MA index.

p-0048The chunk is transferred, step <b>430</b>, for example, via the networks using, or as directed by, a media agent, or other system component. In general, the chunk is transferred through the networks according to system resource availability. The chunk transfer destination is preferably a storage device assigned, for example, according to storage policies, or as directed by media agent, storage manager, or other system component.

p-0049The chunk is received, step <b>440</b> at the storage device, mentioned herein. The storage device preferably communicates data indicating that the chunk has been received to the media agent, storage manager or other system component. The chunk receipt data is updated to an index, such as the SM index or MA index. Upon receipt of the chunk, the chunk is processed, step <b>450</b>. The chunk is generally processed by sequence number, or other information that is contained in the chunk header to reassemble the files. In preferred embodiments, the copy manager reads the chunk header and stores the data contained in the chunk header to an index, such as the SM index or MA index. In some embodiments, the data or files contained in the chunk payload are extracted from the chunk according to the chunk header information. The copy manager copies the chunk file information, for example, to storage media, such as disks, tapes, CD-ROMs or other media for storage. The data extracted from the chunk is stored, step <b>460</b>. The data is generally stored to a storage device.

p-0050Referring to <figref idrefs="DRAWINGS">FIG. 6</figref>, which depicts a flow chart for verifying stored data according to one possible embodiment of the invention, a data verification operation is initiated, step <b>500</b>. In preferred embodiments the data verification operation is scheduled by the jobs manager according to storage policies, and the job manager directs the archive check manager to initiate a data verification operation. In other embodiments, the data verification operation is initiated by the storage manager, media agent, or other system component. The data verification operation, as described herein, determines and verifies whether a copy of data is an accurate copy of an original data set.

p-0051Data to be verified in the data verification operation is identified, step <b>510</b>. In preferred embodiments, the data to be verified is identified, for example, by referencing a stored data verification table, such as the stored data verification table <b>600</b> depicted in <figref idrefs="DRAWINGS">FIG. 7</figref>. In other embodiments, the data to be verified is identified according to information contained in an index, such as an SM index, an MA index, storage policy, or other system component, or as directed by a user. Preferably, the archive check manager identifies the data to be verified by referring to the stored data verification table <b>600</b>, the index, such as the SM index or MA index, or other data source. In other embodiments of the invention, the storage manager, jobs manager, media agent, copy manager or other system component identifies the data to be verified. In some embodiments, the stored data verification table <b>600</b> data may be stored, for example to the SM index or MA index.

p-0052Referring to <figref idrefs="DRAWINGS">FIG. 7</figref>, the data verification table <b>600</b> includes data such as the data shown in a column titled chunk ID <b>610</b>, a column titled chunk verified <b>620</b>, a column titled result <b>630</b>, or other information. Chunk ID <b>610</b> is an alphanumeric identifier for the file, block, chunk or data set stored, such as the files, blocks, chunks, or data sets stored to chunk payload. Chunk verified <b>620</b> provides a yes or no flag indicating whether the chunk or data has been verified, for example, in a data verification operation, as described herein. Result <b>630</b> indicates the result of the data verification operation, such as good copy, unsuccessful, bad file, bad incremental block, or other result. In addition to verifying whether the data is accurate, the data verification operation identifies with particularity which part of the file or copied data is not a good copy, such as the result in column result <b>630</b> which indicates that file 5 is bad.

p-0053Referring again to <figref idrefs="DRAWINGS">FIG. 6</figref>, as mentioned herein, in general, the jobs manager schedules data verification operations and other storage operations, and directs the archive check manager to identify data to be verified. In some embodiments, the archive check manager refers to an index, such as an MA index to identify data to be verified. In other embodiments, the archive check manager identifies the data to be verified in step <b>510</b> by referring to the stored data verification table. In some embodiments, the archive check manager refers to the stored data verification table by chunk identifier to determine whether the chunk has been verified, for example, according to the data contained in a chunk verified column. In other embodiments, the archive check manager refers to the stored data verification table by data contained in the chunk verified column to determine which chunks have not been verified, and obtaining the chunk identifier for unverified chunks.

p-0054The archive check manager obtains the list of data for verifying, as described herein, and sends it to the copy manager, or other system component, step <b>520</b>. The list of data for verifying can be sent, for example as data stored to an index, such as an SM index or MA index, or in a communication to the copy manager. Generally, upon receipt of the list of data to be verified, the copy manager verifies the data, step <b>530</b>.

p-0055An embodiment of step <b>530</b> is depicted in the detailed flowchart of <figref idrefs="DRAWINGS">FIG. 8</figref>. The copy manager, or other system component, initiates a copy operation, step <b>700</b>. The copy manager receives a first set of stored data, or a primary copy, or other original copy of data, and a first set of metadata, step <b>710</b>. The first set of stored data is typically stored to a storage device and information associated with the first set of stored data, such as the first set of metadata, is stored to an index, or other system component. The index, or other system component, provides the source of the first set of stored data and the first set of metadata received by the copy manager. In another embodiment, the copy manager, or other system component, generates the first set of metadata, as described herein, for the first set of stored data. In preferred embodiments, the copy manager performs a copy read operation, wherein, the first set of stored data is read, and a set of metadata is generated. The first set of metadata is stored to an index, or other system component.

p-0056The copy manager, or other system component, generates a second set of metadata, step <b>720</b>. In preferred embodiments, the copy manager reads a copy of the first set of stored data, such as an auxiliary copy, or other copy and performs a partial storage operation. The partial storage operation is generally a copy operation, such as a copy operation used to create a copy of the first set of stored data, auxiliary copy or other copy. The partial storage operation includes reading the data file, creating metadata associated with the data file, but does not complete the copy operation. Thus, only metadata is generated in the partial storage operation. In preferred embodiments, the procedure for generating the metadata in the partial storage operation is the same as the procedure used to generate the first set of metadata, and the first and second sets of metadata are preferably chunk headers, such as chunk header <b>300</b>. A copy of the second set of metadata can be stored for later reference to an index, or used immediately in a comparison of the first set of metadata and second set of metadata, step <b>730</b>.

p-0057Referring back to <figref idrefs="DRAWINGS">FIG. 6</figref>, the result of the data comparison, for example, the comparison of the first set of metadata is updated to the stored data verification table, step <b>540</b>. If the first and second sets of metadata are the same, the copy manager updates the stored data verification table to reflect the result to indicate the second set of stored data is a good copy. Alternatively, if the first and second sets of metadata are different, the copy manager can determine which of the files, for example, which of the files in the chunk, are inconsistent, and update the results data accordingly, for example, the data included in the column results <b>630</b> of <figref idrefs="DRAWINGS">FIG. 7</figref>, such as file 5 is bad, bad incremental block, the verification was unsuccessful, or other result. In preferred embodiments, when the data verification result is something other than good copy, the copy manager initiates another copy or backup operation to copy or back up the first set of stored data.

p-0058Systems and methods disclosed herein are used generally to check or verify all data in a storage system. In some embodiment, the systems and methods can be applied selectively to portions of data. For example, in some embodiments, data within the system can be arbitrarily grouped into backup sets or in CommVault system backup sets, generally denoting arbitrary groups of data wherein data in one backup set can also be in another backup set, such that it is not exclusively in one backup set. At the sub-client level, data in one sub-client group is not in another sub-client group and is exclusive to a sub-client. As described herein, backup sets may have several sub-clients in a set, at the application level, or as specified in a storage policy for a group of data.

p-0059While the invention has been described and illustrated in connection with preferred embodiments, many variations and modifications as will be evident to those skilled in this art may be made without departing from the spirit and scope of the invention, and the invention is thus not to be limited to the precise details of methodology or construction set forth above as such variations and modification are intended to be included within the scope of the invention.