Methods and apparatus for deduplication in storage system

DESCRIPTION OF THE INVENTION

p-00021. Field of the Invention

p-0003This invention generally relates to storage technology and, more specifically, to deduplication of data stored in storage systems.

p-00042. Description of the Related Art

p-0005Recently, a deduplication technique has been developed for storage systems to in order to reduce the storage area cost and the management costs. With deduplication, when the storage system receives data or file from host computers, the storage system first compares the received data with data or files that have already been stored in the storage system. If the same data or file is detected, the storage system creates compact link information to the detected data or file instead of storing the entire received data or file. In other words, the storage system stores the received data or file in a virtual copy, as opposed to physically storing it on a storage media. By means of deduplication, the amount of used physical storage area in the storage system can be made smaller than the size of the corresponding logical area used by the computers. That is, deduplication enables one to use the physical storage area of a storage system in a more efficient manner.

p-0006A deduplication method as applied to a storage system is described in U.S. Pat. No. 7,065,619, incorporated herein in its entirety. U.S. patent publication No. US2006/0059207 to Hirsch et al. shows deduplication method as applied to a storage system such as a virtual tape library. U.S. Pat. No. 6,704,730 shows deduplication method as applied to a computer file system. In addition, Network Appliance provides file storage system (NAS) using deduplication, as described in http://www.netapp.com/products/storage-systems/near-line-storage/asis-dedup.html.

p-0007Current deduplication techniques generally have one or more of the following problems. Because deduplication presupposes that multiple virtual data objects are established by only one physical object, it is associated with loosing large amount of data or files when the number of duplication (virtualized data related to one physical data) is large. For example, if the physical data or file is lost due to a storage system failure, a large number of the corresponding virtual data or files may be also lost. In other words, due to deduplication, the damage and impact from unexpected storage failure incidents become larger. Moreover, as other example, if the physical data or file suffers a performance problem, the access to the corresponding virtual data or files also suffers a similar performance problem. Such performance problem can be caused by storage system failure, human error for system mis-configuration, concentration of access load, and the like. This can happen when multiple virtual data objects or files, which correspond to the same physical object, experience simultaneous accesses from multiple clients, thereby resulting in concentration of access load on the respective physical data object. These effects become larger when the degree of deduplication increases.

p-0008Thus, the existing technology is deficient in its ability to provide a deduplication technology for a storage system having good reliability characteristics.

SUMMARY OF THE INVENTION

p-0009The inventive methodology is directed to methods and systems that substantially obviate one or more of the above and other problems associated with conventional techniques for data deduplication.

p-0010In accordance with one aspect of the inventive concept, there is provided a computerized data storage system including: at least one host computer; a management terminal; and a storage system. The storage system in turn includes an interface configured to communicate with the at least one host computer; a storage device having multiple data objects; and a deduplication controller configured to perform a deduplication of data stored in the storage device. The deduplication controller maintains a threshold with respect to allowed degree of deduplication, counts a number of links for each data object and does not perform deduplication when the counted number of links for the data object exceeds the threshold even if duplication is detected.

p-0011In accordance with another aspect of the inventive concept, there is provided a computerized data storage system. The inventive storage system includes at least one host computer; a management terminal; and a storage system. The storage system in turn includes: an interface operable to communicate with the at least one host computer; a normal reliability storage area; a high reliability data storage area; a data migration controller configured to migrate data between the normal reliability storage area and the high reliability data storage area; and a deduplication controller configured to perform deduplication of data stored in the normal reliability data storage area or the high reliability data storage area. The deduplication controller maintains a threshold with respect to allowed degree of deduplication and counts a number of links for each object. The deduplication controller is further configured to cause the data migration controller to migrate a data object to the high reliability storage area when the counted number of links for the data object exceeds the threshold.

p-0012In accordance with yet another aspect of the inventive concept, there is provided a method performed by a storage system including an interface configured to communicate with at least one host computer and at least one storage device having multiple data objects. The inventive method involves: determining whether a first data is duplicated in the at least one duplicate data object; maintaining a threshold with respect to allowed degree of deduplication; and counting a number of links for the at least one duplicate data object. If the first data is duplicated in the at lest one duplicate data object and if the counted number of links does not exceed the threshold, deduplication of the data in the at least one duplicate data object is performed; and if the counted number of links exceeds the threshold, the deduplication of the data in the at least one duplicate data object is not performed.

p-0013In accordance with a further aspect of the inventive concept, there is provided a method performed by a storage system including an interface configured to communicate with at least one host computer and at least one storage device having multiple data objects. The inventive method involves: determining whether the a data is duplicated in the at least one duplicate data object of the multiple data objects; maintaining a threshold with respect to allowed degree of deduplication; and counting a number of links for the at least one duplicate data object. If the first data is duplicated in the at lest one duplicate data object, deduplication of the data in the at least one duplicate data object is performed; and if the counted number of links exceeds the threshold, the at least one duplicate data object is migrated to a high reliability storage area.

p-0014In accordance with yet further aspect of the inventive concept, there is provided a computer-readable medium storing a set of instruction, the set of instructions, when executed by a storage system including an interface configured to communicate with at least one host computer and at least one storage device having multiple data objects; causing the storage system to: determine whether a first data is duplicated in the at least one duplicate data object of the multiple data objects; maintain a threshold with respect to allowed degree of deduplication; and count a number of links for the at least one duplicate data object. If the first data is duplicated in the at lest one duplicate data object, deduplication of the data in the at least one duplicate data object is performed; and if the counted number of links exceeds the threshold, the at least one duplicate data object is migrated to a high reliability storage area.

p-0015In accordance with yet further aspect of the inventive concept, there is provided a computer-readable medium storing a set of instruction, the set of instructions, when executed by a storage system including an interface configured to communicate with at least one host computer and at least one storage device having data objects; causing the storage system to: determine whether a first data is duplicated in the at least one duplicate data object; maintain a threshold with respect to allowed degree of deduplication; and count a number of links for the at least one duplicate data object. If the first data is duplicated in the at lest one duplicate data object and if the counted number of links does not exceed the threshold, deduplication of the data in the at least one duplicate data object is performed; and if the counted number of links exceeds the threshold, the deduplication of the data in the at least one duplicate data object is not performed.

p-0016Additional aspects related to the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Aspects of the invention may be realized and attained by means of the elements and combinations of various elements and aspects particularly pointed out in the following detailed description and the appended claims.

p-0017It is to be understood that both the foregoing and the following descriptions are exemplary and explanatory only and are not intended to limit the claimed invention or application thereof in any manner whatsoever.

BRIEF DESCRIPTION OF THE DRAWINGS

p-0018The accompanying drawings, which are incorporated in and constitute a part of this specification exemplify the embodiments of the present invention and, together with the description, serve to explain and illustrate principles of the inventive technique. Specifically:

p-0019<figref idrefs="DRAWINGS">FIG. 1</figref> illustrates an exemplary system configuration of the first embodiment.

p-0020<figref idrefs="DRAWINGS">FIG. 2</figref> illustrates an exemplary embodiment of the Parity group information.

p-0021<figref idrefs="DRAWINGS">FIG. 3</figref> illustrates usage of volumes in an exemplary embodiment of the Storage system.

p-0022<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates an exemplary embodiment of the Chunk information.

p-0023<figref idrefs="DRAWINGS">FIG. 5</figref> illustrates an exemplary embodiment of the segment information.

p-0024<figref idrefs="DRAWINGS">FIG. 6</figref> illustrates the first part of an exemplary embodiment of the write process.

p-0025<figref idrefs="DRAWINGS">FIG. 7</figref> illustrates the second part of an exemplary embodiment of the write process.

p-0026<figref idrefs="DRAWINGS">FIG. 8</figref> illustrates the third part of an exemplary embodiment of the write process.

p-0027<figref idrefs="DRAWINGS">FIG. 9</figref> illustrates the fourth part of an exemplary embodiment of the write process.

p-0028<figref idrefs="DRAWINGS">FIG. 10</figref> illustrates an exemplary embodiment of a process for performing read request.

p-0029<figref idrefs="DRAWINGS">FIG. 11</figref> illustrates an exemplary deduplication process as a post process.

p-0030<figref idrefs="DRAWINGS">FIG. 12</figref> illustrates the first part of an exemplary embodiment of the write process.

p-0031<figref idrefs="DRAWINGS">FIG. 13</figref> illustrates the second part of an exemplary embodiment of the write process.

p-0032<figref idrefs="DRAWINGS">FIG. 14</figref> illustrates the third part of an exemplary embodiment of write process.

p-0033<figref idrefs="DRAWINGS">FIG. 15</figref> illustrates an exemplary migration process.

p-0034<figref idrefs="DRAWINGS">FIG. 16</figref> illustrates an exemplary embodiment of Migration information.

p-0035<figref idrefs="DRAWINGS">FIG. 17</figref> describes another example of deduplication process as a post process.

p-0036<figref idrefs="DRAWINGS">FIG. 18</figref> illustrates an exemplary system configuration of the second embodiment.

p-0037<figref idrefs="DRAWINGS">FIG. 19</figref> illustrates an exemplary embodiment of file system data structure.

p-0038<figref idrefs="DRAWINGS">FIG. 20</figref> illustrates usage of actual storage area in Storage system.

p-0039<figref idrefs="DRAWINGS">FIG. 21</figref> illustrates an exemplary embodiment of Substantial file information.

p-0040<figref idrefs="DRAWINGS">FIG. 22</figref> illustrates an exemplary embodiment of Virtual file information.

p-0041<figref idrefs="DRAWINGS">FIG. 23</figref> illustrates one example of the deduplication process as post process.

p-0042<figref idrefs="DRAWINGS">FIG. 24</figref> illustrates an exemplary system configuration of the third embodiment.

p-0043<figref idrefs="DRAWINGS">FIG. 25</figref> illustrates an exemplary system configuration of the fourth embodiment.

p-0044<figref idrefs="DRAWINGS">FIG. 26</figref> illustrates an exemplary embodiment of a computer platform upon which the inventive system may be implemented.

DETAILED DESCRIPTION

p-0045In the following detailed description, reference will be made to the accompanying drawing(s), in which identical functional elements are designated with like numerals. The aforementioned accompanying drawings show by way of illustration, and not by way of limitation, specific embodiments and implementations consistent with principles of the present invention. These implementations are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other implementations may be utilized and that structural changes and/or substitutions of various elements may be made without departing from the scope and spirit of present invention. The following detailed description is, therefore, not to be construed in a limited sense. Additionally, the various embodiments of the invention as described may be implemented in the form of a software running on a general purpose computer, in the form of a specialized hardware, or combination of software and hardware.

A. First Embodiment

A.1. System Configuration

p-0046<figref idrefs="DRAWINGS">FIG. 1</figref> illustrates an exemplary system configuration of the first embodiment of the inventive storage system. A storage system that is related to one embodiment of the invention incorporates one or more of the following components: a Storage system <b>100</b>, an Array controller <b>110</b>, a Main processor <b>111</b>, a Switch <b>112</b>, a Block I/F controller <b>113</b>, a Memory <b>200</b>, a Cache <b>300</b>, a Disk controller <b>400</b>, a disk (e.g. HDD) <b>610</b> and a Backend path (e.g. Fibre Channel, SATA, SAS, iSCSI(IP)) <b>601</b>.

p-0047The Main processor <b>111</b> performs various processes required for proper functioning of the Array controller <b>110</b>. The Main processor <b>111</b> and other components use the following information stored in Memory <b>200</b>: Parity group information <b>201</b>; Chunk information <b>202</b>, Segment information <b>203</b>, Consolidation threshold <b>204</b> and Migration information <b>205</b>.

p-0048The Main processor <b>111</b> performs the aforesaid processes by executing the following programs stored in Memory <b>200</b>: a Write process program <b>211</b>, a Read process program <b>212</b>, a Deduplication (post process) program <b>213</b> and a Migration program <b>214</b>. The detailed description of the above processes will be provided below.

p-0049The Host <b>500</b> and the Management terminal <b>520</b> are connected to the Block interface <b>113</b> via a SAN <b>901</b>. This SAN <b>901</b> may be implemented using Fibre Channel or iSCSI(IP) technology. The Host <b>500</b> and the Management terminal <b>520</b> are interconnected via LAN <b>903</b> (e.g. IP network). The Management terminal <b>520</b> is also connected to the Array controller <b>110</b> via out-of-band Network <b>902</b>, such as an IP-based network.

p-0050To facilitate the computing capability, the Host <b>500</b> and the Management terminal <b>520</b> incorporate various computing resources such as processor and memory (not shown in <figref idrefs="DRAWINGS">FIG. 1</figref>). The Host <b>500</b> also incorporates an application software <b>501</b>, an OS <b>502</b>, and a File system <b>503</b>. In addition, the Management terminal <b>520</b> incorporates the Management software <b>521</b>.

p-0051The storage volumes (Logical Units) provided by the Storage system <b>100</b> are composed from a collection of storage areas located in HDDs. The data in these volumes may be protected by storing a parity code, which is accomplished by using the RAID configuration and the associated technology. The collection of HDDs that provides a volume is called a Parity group <b>600</b>. In one embodiment, various parity group configurations (RAID configurations) and various numbers of disks for the parity group can be applied. As one example of a RAID configuration, a double parity configuration (i.e. RAID6) can be applied. With RAID6, the stored data can be recovered even if two HDD in a Parity Group <b>600</b> breaks down. Therefore, the RAID6 configuration provides very high reliability for storing data. The aforesaid RAID6 configuration is described in “Building Storage Networks” written by Marc Farley, published in 2000 by the McGraw-Hill Company and incorporated herein by reference.

p-0052The Host <b>500</b> can store data in the storage volume and access the data stored in the storage volume. In other words, the Host <b>500</b> writes data to the storage volume and reads data from the storage volume. The Array controller <b>110</b> manages the Parity group <b>600</b> using the Parity group information <b>201</b>. <figref idrefs="DRAWINGS">FIG. 2</figref> illustrates an exemplary embodiment of the Parity group information <b>201</b>. As shown in <figref idrefs="DRAWINGS">FIG. 2</figref>, the Parity group information <b>201</b> includes a parity group ID, a number of disks in each Parity group <b>600</b>, a type of disk configuration (RAID type), IDs of disks in each Parity group <b>600</b> and a capacity information for each disk.

p-0053<figref idrefs="DRAWINGS">FIG. 3</figref> illustrates exemplary usage of volumes in the Storage system <b>100</b>. The Storage system <b>100</b> provides Volumes <b>630</b> (i.e. data storage area) to the Host <b>500</b>. The Host <b>500</b> performs write and read access operations to store/access data in the Volumes <b>630</b> via the host paths (e.g. SAN <b>901</b>). When the new data are stored in the Volumes <b>630</b>, the Storage system <b>100</b> performs deduplication of the contents of the Volume <b>630</b>. The processes for the aforesaid deduplication operation will be described in detail below. As shown in <figref idrefs="DRAWINGS">FIG. 3</figref>, the storage area in the Volume <b>630</b> is logically divided into multiple segments having a fixed size. On the other hand, the physical storage area provided by a RAID parity group is divided into multiple chunks. The size of a chunk is the same as the size of a segment. The Array controller <b>110</b> assigns a chunk to a segment during the write process and stores the data corresponding to the segment in the respective chunk. Moreover, as described below, when the Array controller <b>110</b> detects the same content in other segments, the Array controller <b>110</b> physically stores in a chunk only one copy of the redundant content and creates a link relationship between multiple segments and one chunk. For example, in <figref idrefs="DRAWINGS">FIG. 3</figref>, segment <b>2</b> and segment <b>3</b> has same content stored in the chunk <b>1</b>.

p-0054In order to achieve the above deduplication, the Array controller <b>110</b> uses the Chunk information <b>202</b> and the Segment information <b>203</b>. <figref idrefs="DRAWINGS">FIG. 4</figref> illustrates an exemplary embodiment of the Chunk information <b>202</b>. The Chunk information <b>202</b> is used to manage physical area in Storage system <b>100</b>. In <figref idrefs="DRAWINGS">FIG. 4</figref>, the Chunk information <b>202</b> includes parity group ID, chunk ID, usage (status of use), hash value and the information on the number of respective deduplication links. A set of a parity group ID and a chunk ID uniquely identifies the chunk. The hash value is a value which is used to detect the same data. The hash value is semi-unique value calculated based on the data in each chunk. The size of the hash value is smaller than the size of the chunk itself. Therefore, the hash values are easy to compare and, therefore, the hash value are very suitable for use in detecting the same data. The hash value is generated by a hash function such as SHA-1 and SHA-256. The number of links indicates the number of established relations between a chunk (i.e. physical area) and a segment (i.e. logical or virtual area), as shown in <figref idrefs="DRAWINGS">FIG. 3</figref>. The usage information indicates the current usage of each chunk (i.e. physical area).

p-0055<figref idrefs="DRAWINGS">FIG. 5</figref> illustrates an exemplary embodiment of the Segment information <b>203</b>. The Segment information <b>203</b> maintains a relation (link) between the chunk (i.e. physical area) and the corresponding segment (i.e. logical or virtual area). In <figref idrefs="DRAWINGS">FIG. 5</figref>, the Segment information <b>203</b> includes a volume ID, a segment ID, a status of the assignment, a parity group ID and a chunk ID of the chunk linked to the segment. A set of volume ID and segment ID uniquely identifies the segment. The status of the assignment is ‘No’ if no chunk is assigned to the segment while ‘Yes’ indicates that some chunk has been assigned to the segment. Linked chunk (parity group ID and chunk ID) indicates the chunk linked to the segment. That is, data in the segment is stored in the chunk indicated as the linked chunk. In one embodiment, the above information is stored as list or directory of each element for quick search.

A.2. First Example of Write Process: Limitation of Deduplication

p-0056The Storage system <b>100</b> performs deduplication as a part of the process for handling a data write request. <figref idrefs="DRAWINGS">FIG. 6</figref> illustrates the first part of the exemplary write process. In this part, detection of duplication is performed.

p-0057At step <b>1001</b>, the Array controller <b>110</b> receives a write request and the associated data from the Host <b>500</b> via the SAN <b>901</b>. The Array controller <b>100</b> then takes the first segment of the target area specified in the write request as the current target segment to be processed.

p-0058At step <b>1002</b>, the Array controller <b>110</b> checks the Segment information <b>203</b> with respect to the target segment. If the target segment has already had a corresponding chunk assigned at step <b>1003</b>, the Array controller <b>110</b> obtains new data for the target segment from the write data (step <b>1004</b>). If not, the Array controller <b>110</b> obtains the new data from old data stored in the chunk and the write data (step <b>1005</b>).

p-0059At step <b>1006</b>, the Array controller <b>110</b> calculates the hash value of the new data by means of the hash function.

p-0060At step <b>1007</b>, the Array controller <b>110</b> searches the Chunk information <b>202</b> for the same hash value.

p-0061At step <b>1008</b>, if the Array controller <b>110</b> finds a chunk having the same hash value, the process proceeds to step <b>1009</b>. If not, the process proceeds to step <b>1201</b> in <figref idrefs="DRAWINGS">FIG. 8</figref>.

p-0062At step <b>1009</b>, the Array controller <b>110</b> compares the threshold value recorded in Consolidation threshold record <b>204</b> with the number of links to the chunk found in step <b>1008</b> by referring to the Consolidation threshold <b>204</b> and Chunk information <b>202</b>. The consolidation threshold represents a threshold value for the number of virtual data objects corresponding to one physical data object. In other words, this threshold represents the threshold with respect to the degree of deduplication. If the number of links is smaller than the threshold value, the process proceeds to step <b>1010</b>. If not, the process proceeds to step <b>1201</b> in <figref idrefs="DRAWINGS">FIG. 8</figref>.

p-0063At step <b>1010</b>, the Array controller <b>110</b> compares the new data with the data in the found chunk by binary-level comparison.

p-0064At step <b>1011</b>, as the result of the comparison, if both data are the same, the process proceeds to step <b>1101</b> in <figref idrefs="DRAWINGS">FIG. 7</figref>. If not, the process proceeds to step <b>1201</b> in <figref idrefs="DRAWINGS">FIG. 8</figref>.

p-0065<figref idrefs="DRAWINGS">FIG. 7</figref> illustrates the second part of the exemplary write process. In this part, a link (relation) is created or updated instead of physically storing the received data.

p-0066At step <b>1101</b>, the Array controller <b>110</b> checks the Segment information <b>203</b> in order to locate the target segment.

p-0067If the target segment has already had a corresponding chunk assigned at step <b>1102</b>, the process proceeds to step <b>1104</b>. If not, the Array controller <b>110</b> updates the Segment information <b>203</b> to create link between the target segment and the found chunk having same as the new data (step <b>1103</b>).

p-0068At step <b>1104</b>, the Array controller <b>110</b> updates the Segment information <b>203</b> by changing the chunk for the target segment to the found chunk having the same data.

p-0069At step <b>1105</b>, the Array controller <b>110</b> updates the Chunk information <b>202</b> by subtracting 1 from the number of links of the old chunk for the target segment.

p-0070If the number of links of the old chunk equals to 0 at step <b>1106</b>, the Array controller <b>110</b> updates the Chunk information <b>202</b> to label the old chunk as unused (step <b>1107</b>). If not, the process proceeds to step <b>1108</b>.

p-0071At step <b>1108</b>, the Array controller <b>110</b> updates the Chunk information <b>202</b> by adding 1 to the number of links of the found chunk having the same as the new data. After that, the process proceeds to step <b>1301</b> in <figref idrefs="DRAWINGS">FIG. 9</figref>.

p-0072<figref idrefs="DRAWINGS">FIG. 8</figref> illustrates the third part of the exemplary write process. In this part, the write data is physically stored.

p-0073At step <b>1201</b>, the Array controller <b>110</b> checks the Segment information <b>203</b> with respect to the target segment.

p-0074At step <b>1202</b>, if the target segment has already had a corresponding chunk assigned, the process proceeds to step <b>1203</b>. If not, the process proceeds to step <b>1205</b>.

p-0075At step <b>1203</b>, the Array controller <b>110</b> checks the Chunk information <b>202</b> with respect to the assigned chunk.

p-0076At step <b>1204</b>, if the number of links pointing to the assigned chunk is larger than 1, the process proceeds to step <b>1208</b>. If not, the process proceeds to step <b>1212</b>.

p-0077At step <b>1205</b>, the Array controller <b>110</b> searches the Chunk information <b>202</b> for an unused chunk and obtains the unused chunk.

p-0078At step <b>1206</b>, the Array controller <b>110</b> updates the Segment information <b>203</b> and the Chunk information <b>202</b> to assign the chunk to the target segment.

p-0079At step <b>1207</b>, the Array controller <b>110</b> stores the new data in the chunk. After that, the process proceeds to step <b>1301</b> in <figref idrefs="DRAWINGS">FIG. 9</figref>.

p-0080At step <b>1208</b>, the Array controller <b>110</b> searches the Chunk information <b>202</b> for an unused chunk and obtains the unused chunk.

p-0081At step <b>1209</b>, the Array controller <b>110</b> updates the Segment information <b>203</b> by changing the assignment of the chunk for the target segment to the newly obtained chunk.

p-0082At step <b>1210</b>, the Array controller <b>110</b> updates the Chunk information <b>202</b> by subtracting 1 from the number of links pointing to the old chunk for the target segment.

p-0083At step <b>1211</b>, the Array controller <b>110</b> stores the new data in the new chunk. After that, the process proceeds to step <b>1301</b> in <figref idrefs="DRAWINGS">FIG. 9</figref>.

p-0084At step <b>1212</b>, the Array controller <b>110</b> stores the new data in the assigned chunk. After that, the process proceeds to step <b>1301</b> in <figref idrefs="DRAWINGS">FIG. 9</figref>.

p-0085<figref idrefs="DRAWINGS">FIG. 9</figref> illustrates the fourth part of the exemplary write process. This figure describes the termination condition of the process.

p-0086If the whole of the target area of the write request has been processed at step <b>1301</b>, the Array controller <b>110</b> reports completion of the process of the write request to the host (step <b>1302</b>), and the process terminates. If not, the Array controller <b>110</b> designates the next segment as target segment (step <b>1303</b>), and then the process proceeds to step <b>1002</b> in <figref idrefs="DRAWINGS">FIG. 6</figref>.

p-0087In the above process, if the number of links of a chunk reaches the deduplication threshold, further consolidation of the same data (i.e. deduplication) is not performed. User or Management software <b>521</b> can specify the threshold (the Consolidation threshold <b>204</b>) through the Management terminal <b>520</b>. With the above described process, the risk to stored data and the problems such as reduction in reliability and performance across a large number of data objects can be effectively mitigated.

A.3. Read Process

p-0088<figref idrefs="DRAWINGS">FIG. 10</figref> illustrates an exemplary embodiment of a process for a read request.

p-0089At step <b>1401</b>, the Array controller <b>110</b> receives a read request from the Host <b>500</b> via the SAN <b>901</b>. The Array controller <b>100</b> designates the first segment of target area corresponding to the read request as the current target segment to be processed.

p-0090At step <b>1402</b>, the Array controller <b>110</b> refers to the Segment information <b>203</b> and obtains the chunk for the target segment.

p-0091At step <b>1403</b>, the Array controller <b>110</b> transfers the data in the chunk to the host. If the entire target area corresponding to the read request has been processed at step <b>1404</b>, the Array controller <b>110</b> reports the completion of the process of the read request to the host (step <b>1405</b>), and terminates the process. If not, the Array controller <b>110</b> designates the next segment as the target segment (step <b>1406</b>), and then the process proceeds to step <b>1402</b>.

A.4. First Example of Deduplication Process as a Post Process: Limitation of Deduplication

p-0092Deduplication of data stored in Storage system <b>100</b> can be performed independently of the write process. That is, deduplication can be performed as a post process after storing the data. This process can be also performed for mirror/snapshot data after decoupling of the mirror.

p-0093<figref idrefs="DRAWINGS">FIG. 11</figref> illustrates the deduplication process as post process.

p-0094At step <b>1501</b>, the Array controller <b>110</b> calculates hash value of data stored in a chunk corresponding to the current segment using the hash function.

p-0095At step <b>1502</b>, the Array controller <b>110</b> searches the Chunk information <b>202</b> for the same hash value.

p-0096At step <b>1503</b>, if the Array controller <b>110</b> finds a chunk having the same hash value, the process proceeds to step <b>1504</b>. If not, the process proceeds to step <b>1510</b>.

p-0097At step <b>1504</b>, the Array controller <b>110</b> compares the threshold value recorded in the Consolidation threshold record <b>204</b> with the number of links pointing to the chunk found in step <b>1503</b> by referring to the Consolidation threshold record <b>204</b> and the Chunk information <b>202</b>. If the number of links is smaller than the threshold, the process proceeds to step <b>1505</b>. If not, the process proceeds to step <b>1510</b>.

p-0098At step <b>1505</b>, the Array controller <b>110</b> compares the data in the current segment with the data in the found chunk using a binary-level comparison.

p-0099At step <b>1506</b>, if, based on the comparison, it is determined that both data are the same, the process proceeds to step <b>1507</b>. If not, the process proceeds to step <b>1510</b>.

p-0100At step <b>1507</b>, the Array controller <b>110</b> updates the Segment information <b>203</b> by changing the chunk for the current segment to the found chunk having the same data.

p-0101At step <b>1508</b>, the Array controller <b>110</b> updates the Chunk information <b>202</b> by adding the number of links of the old chunk for the current segment to the number of links of the found chunk having the same data.

p-0102At step <b>1509</b>, the Array controller <b>110</b> updates the Chunk information <b>202</b> by labeling the old chunk unused.

p-0103If the whole of the target area of the deduplication process has been processed at step <b>1510</b>, the process terminates. If not, the Array controller <b>110</b> designates the next segment as the current segment (step <b>1511</b>), and after that the process proceeds to step <b>1501</b>.

p-0104Also in the above deduplication process, if the number of links pointing to a chunk reaches the deduplication threshold, the consolidation of same data (i.e. deduplication) is not performed. Using the above inventive process, the risk to stored data and the problems such as reduction of reliability and performance across a large number of deduplicated data objects can be mitigated.

A.5. Second Example of Write Process: Data Migration Based on the Number of Links

p-0105<figref idrefs="DRAWINGS">FIG. 12</figref>, <figref idrefs="DRAWINGS">FIG. 13</figref> and <figref idrefs="DRAWINGS">FIG. 14</figref> illustrate another example of a write process with deduplication. With the shown process, data migration between areas of different reliability is performed based on the comparison between the threshold and the number of links pointing to a chunk after deduplication.

p-0106<figref idrefs="DRAWINGS">FIG. 12</figref> illustrates the first part of the exemplary write process. In this part, the detection of duplication is performed. The process is the same as the process described in <figref idrefs="DRAWINGS">FIG. 6</figref> except for that the step <b>1009</b> has been eliminated.

p-0107<figref idrefs="DRAWINGS">FIG. 13</figref> describes the second part of the write process. In this part, a link (relation) is created or updated instead of physically storing the data. This process is the same as the process described the <figref idrefs="DRAWINGS">FIG. 7</figref> except for the addition of the following procedure after the addition/subtraction step for the number of links: (i) comparison between the number of links and the threshold and (ii) data migration between areas of different reliability based on the result of the comparison. The related steps are the following.

p-0108If the number of links of the old chunk equals to the threshold at step <b>1707</b>, the Array controller <b>110</b> moves the data in the old chunk from high reliability area such as RAID6 to normal reliability area such as RAID1 or RAID5 (step <b>1708</b>). The threshold is recorded in the Consolidation threshold record <b>204</b> and indicates a threshold with respect to the number of virtual data objects for one physical data object. In other words, this threshold indicates a threshold with respect to the degree of deduplication.

p-0109If the number of links pointing to the chunk equals to the threshold plus 1 at step <b>1711</b>, the Array controller <b>110</b> moves the data in the chunk from the normal reliability area such as RAID1 and RAID5 to the high reliability area such as RAID6 (step <b>1712</b>).

p-0110<figref idrefs="DRAWINGS">FIG. 14</figref> illustrates the third part of the write process. In this part, the data is physically written to a storage device. This process is the same as the corresponding process described the <figref idrefs="DRAWINGS">FIG. 8</figref>, except for the addition of following procedure after the subtraction operation with respect to the number of links: (i) comparison between the number of links and the threshold and (ii) data migration between areas of different reliability based on the result of the comparison. The related steps are the following.

p-0111If the number of links pointing to the old chunk equals to the threshold at step <b>1812</b>, the Array controller <b>110</b> moves the data in the old chunk from the high reliability area such as RAID6 to the normal reliability area such as RAID1 or RAID5 (step <b>1813</b>).

p-0112In the above process, if the number of links pointing to a chunk exceeds the deduplication threshold, the data in the chunk is migrated to the high reliability area such as RAID6. Using the above inventive process, the risk of loss to the stored data and the problems such as reduction of reliability and performance across a large number of data objects can be mitigated.

p-0113<figref idrefs="DRAWINGS">FIG. 15</figref> illustrates the migration process mentioned above.

p-0114At step <b>1901</b>, the Array controller <b>110</b> seeks an unused chunk by referring to the Parity group information <b>201</b> and the Chunk information <b>202</b>. The unused chunk is obtained from high reliability area (e.g. RAID6) or normal reliability area (e.g. RAID1 or RAID5) according to the condition mentioned in the aforesaid processes.

p-0115At step <b>1902</b>, the Array controller <b>110</b> creates an entry in Migration information <b>205</b> for the chunk to be moved. <figref idrefs="DRAWINGS">FIG. 16</figref> shows an exemplary embodiment of the Migration information <b>205</b>. Migration information <b>205</b> has sets of parity group ID information and Chunk ID information for both of the source and the destination. It also has a copy pointer that denotes the progress of the copy operation.

p-0116At step <b>1903</b>, the Array controller <b>110</b> copies the data in the source chunk to the chunk selected as the destination. According to progress of the copying operation, the copy pointer in Migration information <b>206</b> is updated by moving it forward.

p-0117At step <b>1904</b>, after the completion of the copying operation, the Array controller <b>110</b> updates the Segment information <b>203</b> to change chunk for the related segment(s) from the source chunk to the destination chunk. In other words, the Array controller <b>110</b> changes mapping between the segment(s) and the chunk. This accomplishes the transparent migration of the data for Host <b>500</b>.

p-0118At step <b>1905</b>, the Array controller <b>110</b> updates the Chunk information <b>202</b> to label the source chunk as unused.

p-0119At step <b>1906</b>, the Array controller <b>110</b> deletes the entry for the chunk.

A.6. Second Example of Deduplication Process as a Post Process: Data Migration Based on the Number of Links

p-0120<figref idrefs="DRAWINGS">FIG. 17</figref> describes illustrates another example of the deduplication process as a post process. With this process, data migration between areas of different reliability is performed based on comparison between the threshold and the number of links pointing to a chunk after the deduplication.

p-0121The process shown in <figref idrefs="DRAWINGS">FIG. 17</figref> is the same as the corresponding process described the <figref idrefs="DRAWINGS">FIG. 11</figref>, except for deletion of the step <b>1504</b> and addition of following procedure, which is performed after the addition operation performed on the number of links: (i) comparison between the number of links and the threshold and (ii) data migration between areas of different reliability based on the result of the comparison. The related steps are the following.

p-0122If the number of links pointing to the chunk equals to the threshold plus 1 at step <b>2009</b>, the Array controller <b>110</b> moves the data in the chunk from the normal reliability area such as RAID1 and RAID5 to the high reliability area such as RAID6 (step <b>2010</b>).

p-0123Also in the above deduplication process, if the number of links pointing to a chunk exceeds the deduplication threshold, the data in the chunk is migrated to the high reliability area such as RAID6. Using the above inventive process, the risk of loss to the stored data and the problems such as reduction of reliability and performance across a large number of data objects can be mitigated.

B. Second Embodiment

B.1. System Configuration

p-0124<figref idrefs="DRAWINGS">FIG. 18</figref> illustrates an exemplary system configuration of the second embodiment. In the system of the second embodiment, the Array controller <b>110</b> incorporates a Network I/F controller <b>114</b> instead of the Block interface controller <b>113</b> of the first embodiment. The Memory <b>200</b> maintains a Substantial file information <b>222</b> and a Virtual file information <b>223</b> instead of the Chunk information <b>202</b> and the Segment information <b>203</b> of the first embodiment.

p-0125In addition to the programs mentioned in the first embodiment, the Memory <b>200</b> contains a File service program <b>215</b>. The File service program <b>215</b> exports files (i.e. it makes files accessible) via network file sharing protocols such as NFS (Network File System) and CIFS (Common Internet File System). Moreover, the File service program <b>215</b> interprets requests from the Host <b>500</b> and handles those requests by invoking the Write process program <b>211</b> and the Read process program <b>212</b>. The addition of the Network I/F controller <b>114</b> and the File service program <b>215</b> enables the Storage system <b>100</b> to function as a NAS (Network Attached Storage) system.

p-0126The Host <b>500</b> and the Management terminal <b>520</b> are connected to the Network interface <b>114</b> via an IP-SAN <b>904</b> (i.e. IP network). In addition to the configuration mentioned in the first embodiment, the Host <b>500</b> incorporates a Network file system client software <b>504</b>. The Network file system client <b>504</b> sends the Storage system <b>100</b> file I/O requests via IP-SAN <b>904</b> using network file sharing protocols such as NFS and CIFS in response to an instruction from the Application software <b>501</b> and the like. Other components of the system of the second embodiment are generally the same as the respective components described with reference to the first embodiment.

p-0127<figref idrefs="DRAWINGS">FIG. 19</figref> illustrates an exemplary embodiment of the file system data structure. Each mode can be used to indicate a file or a directory. If the mode indicates a file (if its File Type field is “file” such as <b>804</b> and <b>807</b>), the Data Blocks pointed from Block Pointer in the mode contain actual data of the file. If a file is stored in a plurality of Data Blocks (e.g. 10 blocks), the addresses of the 10 Data Blocks are recorded in Block Pointer. If the mode indicates a directory (if the File Type field is “directory” such as <b>801</b> and <b>803</b>), the Data Blocks pointed from Block Pointer in the mode stores the list of Inode Numbers and names of all files and directories that resides in the directory (the list is called Directory Entry). The aforesaid several types of information used to manage files are stored in volumes in the Storage system <b>100</b> according to a predetermined placement rule.

p-0128<figref idrefs="DRAWINGS">FIG. 20</figref> illustrates usage of physical storage area in the Storage system <b>100</b>. The Storage system <b>100</b> stores files used by the Hosts <b>500</b> and continues to present (export) these files to the Hosts <b>500</b>. In addition, the Storage system <b>100</b> performs deduplication for the files if the Storage system <b>100</b> detects files having the same content. As shown in <figref idrefs="DRAWINGS">FIG. 20</figref>, the Storage system <b>100</b> provides virtual files to the Host <b>500</b> and the virtual files are linked to the physical files that are physically stored in storage area of the Storage system <b>100</b>. This relation between the virtual files and the physical files is similar to the relation between the segments and chunks mentioned with respect to the first embodiment.

p-0129The above relationship (link) and the related information are expressed by Substantial file information <b>222</b> and Virtual file information <b>223</b> shown in <figref idrefs="DRAWINGS">FIG. 21</figref> and <figref idrefs="DRAWINGS">FIG. 22</figref>. The substantial file information <b>222</b> is information necessary to manage substantial (physical) file physically stored in the Storage system <b>100</b>. In <figref idrefs="DRAWINGS">FIG. 21</figref>, the Substantial file information <b>222</b> includes a substantial file ID identifying each substantial file, a hash value generated from the content of the substantial file and the number of links pointing to the substantial file. The number of links represents the number of relationships between the substantial file and virtual files shown in <figref idrefs="DRAWINGS">FIG. 20</figref>.

p-0130The virtual file information <b>223</b> is information necessary to manage virtual files provided to the Host <b>500</b>. In <figref idrefs="DRAWINGS">FIG. 22</figref>, the Virtual file information <b>223</b> includes a virtual file ID identifying each virtual file and a substantial file ID of the linked substantial file. In this embodiment, the virtual file ID and the substantial file ID are independent.

B.2. Deduplication Process

p-0131The deduplication processes such as the write deduplication process and deduplication process performed as a post process can be performed in the same manner as described with reference to the first embodiment. <figref idrefs="DRAWINGS">FIG. 23</figref> illustrates one example of the deduplication process performed as a post process in this embodiment.

p-0132At step <b>2101</b>, the Array controller <b>110</b> calculates the hash value of the data in current file (virtual file) using the hash function.

p-0133At step <b>2102</b>, the Array controller <b>110</b> searches the Substantial file information <b>222</b> for the same hash value.

p-0134At step <b>2103</b>, if the Array controller <b>110</b> can find a substantial file having the same hash value, the process proceeds to step <b>2104</b>. If not, the process proceeds to step <b>2110</b>.

p-0135At step <b>2104</b>, the Array controller <b>110</b> compares the threshold value recorded in Consolidation threshold record <b>204</b> with the number of links of the substantial file found in step <b>2103</b> by referring to the Consolidation threshold record <b>204</b> and the Substantial file information <b>222</b>. The consolidation threshold indicates a threshold with respect to the number of virtual data files for one substantial (physical) data file. In other words, this threshold represents a threshold with respect to the degree of deduplication. If the number of links is smaller than the threshold, the process proceeds to step <b>2105</b>. If not, the process proceeds to step <b>2110</b>.

p-0136At step <b>2105</b>, the Array controller <b>110</b> compares the data in the current virtual file with data in the found substantial file by binary-level comparison.

p-0137At step <b>2106</b>, as result of the comparison, if the both data are same, the process proceeds to step <b>2107</b>. If not, the process proceeds to step <b>2110</b>.

p-0138At step <b>2107</b>, the Array controller <b>110</b> updates the Virtual file information <b>223</b> by changing the substantial file for the current virtual file to the found substantial file having the same data.

p-0139At step <b>2108</b>, the Array controller <b>110</b> updates the Substantial file information <b>222</b> by adding the number of links pointing to the old substantial file associated with the current file to the number of links of the found substantial file having the same data.

p-0140At step <b>2109</b>, the Array controller <b>100</b> deletes the substantial file associated with the current file by updating the Substantial file information <b>222</b>. If all the target files of the deduplication process have been processed at step <b>2110</b>, the process terminates. If not, the Array controller <b>110</b> designates the next file as the current file (step <b>2111</b>), and the process proceeds to step <b>2101</b>.

p-0141In the above deduplication process, if the number of links pointing to the substantial file reaches the deduplication threshold, consolidation of the same data (i.e. deduplication) is not performed. Using the above inventive process, the risk of loss to the stored data and the problems such as reduction of reliability and performance across a large number of data objects can be mitigated.

p-0142Just like the above-described process, by considering chunk and segment in the first embodiment as substantial file and virtual file in this embodiment, each process mentioned with reference to the first embodiment can be applied to the file storage system (Storage system <b>100</b>) in the second embodiment. The Chunk information <b>201</b> and the Segment information <b>203</b> in the first embodiment correspond to each of the Substantial file information <b>222</b> and the Virtual file information <b>223</b> in this embodiment. Obtaining and releasing (making unused) chunks also corresponds to creation and delectation of substantial files. As another example of high reliability area, storage area in which data is protected by remote replication to other storage system can be applied.

p-0143By applying the processes such as the write process deduplication or the deduplication process performed as a post process, as described hereinabove, the risk of loss to the stored data and the problems such as reduction of reliability and performance across a large number of data objects in network file storage systems (NAS) having deduplication capability can be successfully mitigated.

C. Third Embodiment

C.1. System Configuration

p-0144<figref idrefs="DRAWINGS">FIG. 24</figref> illustrates an exemplary system configuration of the third embodiment. In addition to components mentioned with reference to the first embodiment, the system in this embodiment incorporates a Backup server <b>510</b> and Storage system <b>700</b>. The Backup server <b>510</b> incorporates the Backup software <b>511</b>. The Host <b>500</b> and the Storage system <b>700</b> are connected by a SAN <b>901</b>, which enables the Host <b>500</b> to store data the Storage system <b>700</b>. Array controller <b>110</b> incorporates a Tape (stream) interface controller <b>114</b> instead of the Block interface controller <b>113</b> of the first embodiment. In addition to the programs mentioned in the first embodiment, the Memory <b>200</b> contains a Virtual tape program <b>216</b>.

p-0145The Virtual tape program <b>216</b> emulates a tape device and a tape library. That is, it interprets requests for a tape device from the Host <b>500</b> and handles these requests by invoking the Write process program <b>211</b> and the Read process program <b>212</b>. The Backup server <b>510</b> and the Host <b>500</b> can use the Storage system <b>100</b> as a tape library with a tape media. This enables the Storage system <b>100</b> to function as a VTL (Virtual Tape Library). The other components of the system of the third embodiment are generally the same as the corresponding components described with reference with the first embodiment.

C.2. Deduplication Process in Backup Operation

p-0146In the backup operation, the Backup software <b>511</b> reads data used by the Host <b>500</b> from the Storage system <b>700</b> and writes the data to the Storage system <b>100</b>. In the restore operation, the Backup software <b>511</b> reads the data to be restored from the Storage system <b>100</b> and writes it to the Storage system <b>700</b>.

p-0147Because the processes in the Storage system <b>100</b> are same as the corresponding process mentioned with reference to the first embodiment, except for virtual tape function including interpreting and converting of requests, each deduplication-related process mentioned with reference to the first embodiment can be applied to the Storage system <b>100</b> in the third embodiment.

p-0148By applying the processes such as write deduplication process or deduplication process performed as a post process described above, the risk of loss to the stored data and the problems such as reduction of reliability and performance across a large number of data objects in VTL having deduplication capability can be successfully mitigated. In another exemplary storage system configuration, the controller having the deduplication capability can be separated, for example as an appliance, from the rest of the storage system having HDDs.

D. Fourth Embodiment

p-0149<figref idrefs="DRAWINGS">FIG. 25</figref> illustrates an exemplary system configuration of the fourth embodiment. In this embodiment, the Backup software <b>511</b> incorporates a Deduplication program <b>512</b> in order to provide the functions and capability with respect to deduplication mentioned in the above embodiments. During the backup operation, the Backup software <b>511</b> reads data used by the Host <b>500</b> from the Storage system <b>700</b> and writes the data to the Backup storage system <b>710</b>, such as a tape device. During the restore operation, the Backup software <b>511</b> reads the data to be restored from the Backup storage system <b>710</b> and writes the data to the Storage system <b>700</b>.

p-0150By applying the processes such as the write deduplication process or deduplication process performed as a post process described above to processes performed by the Backup software <b>511</b>, the risk of loss to the backup data and the problems such as reduction of reliability and performance across a large number of backup data objects can be successfully mitigated.

E. Exemplary Computer Platform

p-0151<figref idrefs="DRAWINGS">FIG. 26</figref> is a block diagram that illustrates an embodiment of a computer/server system <b>2600</b> upon which an embodiment of the inventive methodology may be implemented. The system <b>2600</b> includes a computer/server platform <b>2601</b>, peripheral devices <b>2602</b> and network resources <b>2603</b>.

p-0152The computer platform <b>2601</b> may include a data bus <b>2604</b> or other communication mechanism for communicating information across and among various parts of the computer platform <b>2601</b>, and a processor <b>2605</b> coupled with bus <b>2601</b> for processing information and performing other computational and control tasks. Computer platform <b>2601</b> also includes a volatile storage <b>2606</b>, such as a random access memory (RAM) or other dynamic storage device, coupled to bus <b>2604</b> for storing various information as well as instructions to be executed by processor <b>2605</b>. The volatile storage <b>2606</b> also may be used for storing temporary variables or other intermediate information during execution of instructions by processor <b>2605</b>. Computer platform <b>2601</b> may further include a read only memory (ROM or EPROM) <b>2607</b> or other static storage device coupled to bus <b>2604</b> for storing static information and instructions for processor <b>2605</b>, such as basic input-output system (BIOS), as well as various system configuration parameters. A persistent storage device <b>2608</b>, such as a magnetic disk, optical disk, or solid-state flash memory device is provided and coupled to bus <b>2601</b> for storing information and instructions.

p-0153Computer platform <b>2601</b> may be coupled via bus <b>2604</b> to a display <b>2609</b>, such as a cathode ray tube (CRT), plasma display, or a liquid crystal display (LCD), for displaying information to a system administrator or user of the computer platform <b>2601</b>. An input device <b>2610</b>, including alphanumeric and other keys, is coupled to bus <b>2601</b> for communicating information and command selections to processor <b>2605</b>. Another type of user input device is cursor control device <b>2611</b>, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor <b>2604</b> and for controlling cursor movement on display <b>2609</b>. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

p-0154An external storage device <b>2612</b> may be connected to the computer platform <b>2601</b> via bus <b>2604</b> to provide an extra or removable storage capacity for the computer platform <b>2601</b>. In an embodiment of the computer system <b>2600</b>, the external removable storage device <b>2612</b> may be used to facilitate exchange of data with other computer systems.

p-0155The invention is related to the use of computer system <b>2600</b> for implementing the techniques described herein. In an embodiment, the inventive system may reside on a machine such as computer platform <b>2601</b>. According to one embodiment of the invention, the techniques described herein are performed by computer system <b>2600</b> in response to processor <b>2605</b> executing one or more sequences of one or more instructions contained in the volatile memory <b>2606</b>. Such instructions may be read into volatile memory <b>2606</b> from another computer-readable medium, such as persistent storage device <b>2608</b>. Execution of the sequences of instructions contained in the volatile memory <b>2606</b> causes processor <b>2605</b> to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.

p-0156The term “computer-readable medium” as used herein refers to any medium that participates in providing instructions to processor <b>2605</b> for execution. The computer-readable medium is just one example of a machine-readable medium, which may carry instructions for implementing any of the methods and/or techniques described herein. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device <b>2608</b>. Volatile media includes dynamic memory, such as volatile storage <b>2606</b>. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise data bus <b>2604</b>. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

p-0157Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, a flash drive, a memory card, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.

p-0158Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to processor <b>2605</b> for execution. For example, the instructions may initially be carried on a magnetic disk from a remote computer. Alternatively, a remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system <b>2600</b> can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on the data bus <b>2604</b>. The bus <b>2604</b> carries the data to the volatile storage <b>2606</b>, from which processor <b>2605</b> retrieves and executes the instructions. The instructions received by the volatile memory <b>2606</b> may optionally be stored on persistent storage device <b>2608</b> either before or after execution by processor <b>2605</b>. The instructions may also be downloaded into the computer platform <b>2601</b> via Internet using a variety of network data communication protocols well known in the art.

p-0159The computer platform <b>2601</b> also includes a communication interface, such as network interface card <b>2613</b> coupled to the data bus <b>2604</b>. Communication interface <b>2613</b> provides a two-way data communication coupling to a network link <b>2614</b> that is connected to a local network <b>2615</b>. For example, communication interface <b>2613</b> may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface <b>2613</b> may be a local area network interface card (LAN NIC) to provide a data communication connection to a compatible LAN. Wireless links, such as well-known 802.11a, 802.11b, 802.11g and Bluetooth may also used for network implementation. In any such implementation, communication interface <b>2613</b> sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

p-0160Network link <b>2613</b> typically provides data communication through one or more networks to other network resources. For example, network link <b>2614</b> may provide a connection through local network <b>2615</b> to a host computer <b>2616</b>, or a network storage/server <b>2617</b>. Additionally or alternatively, the network link <b>2613</b> may connect through gateway/firewall <b>2617</b> to the wide-area or global network <b>2618</b>, such as an Internet. Thus, the computer platform <b>2601</b> can access network resources located anywhere on the Internet <b>2618</b>, such as a remote network storage/server <b>2619</b>. On the other hand, the computer platform <b>2601</b> may also be accessed by clients located anywhere on the local area network <b>2615</b> and/or the Internet <b>2618</b>. The network clients <b>2620</b> and <b>2621</b> may themselves be implemented based on the computer platform similar to the platform <b>2601</b>.

p-0161Local network <b>2615</b> and the Internet <b>2618</b> both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link <b>2614</b> and through communication interface <b>2613</b>, which carry the digital data to and from computer platform <b>2601</b>, are exemplary forms of carrier waves transporting the information.

p-0162Computer platform <b>2601</b> can send messages and receive data, including program code, through the variety of network(s) including Internet <b>2618</b> and LAN <b>2615</b>, network link <b>2614</b> and communication interface <b>2613</b>. In the Internet example, when the system <b>2601</b> acts as a network server, it might transmit a requested code or data for an application program running on client(s) <b>2620</b> and/or <b>2621</b> through Internet <b>2618</b>, gateway/firewall <b>2617</b>, local area network <b>2615</b> and communication interface <b>2613</b>. Similarly, it may receive code from other network resources.

p-0163The received code may be executed by processor <b>2605</b> as it is received, and/or stored in persistent or volatile storage devices <b>2608</b> and <b>2606</b>, respectively, or other non-volatile storage for later execution. In this manner, computer system <b>2601</b> may obtain application code in the form of a carrier wave.

p-0164It should be noted that the present invention is not limited to any specific firewall system. The inventive policy-based content processing system may be used in any of the three firewall operating modes and specifically NAT, routed and transparent.

p-0165Finally, it should be understood that processes and techniques described herein are not inherently related to any particular apparatus and may be implemented by any suitable combination of components. Further, various types of general purpose devices may be used in accordance with the teachings described herein. It may also prove advantageous to construct specialized apparatus to perform the method steps described herein. The present invention has been described in relation to particular examples, which are intended in all respects to be illustrative rather than restrictive. Those skilled in the art will appreciate that many different combinations of hardware, software, and firmware will be suitable for practicing the present invention. For example, the described software may be implemented in a wide variety of programming or scripting languages, such as Assembler, C/C++, perl, shell, PHP, Java, etc.

p-0166Moreover, other implementations of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. Various aspects and/or components of the described embodiments may be used singly or in any combination in the computerized storage system with data replication functionality. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.