System and method for data encryption
Claim Score by NHIP
Abstract
A method of processing data from a file includes obtaining a first portion of the file, encrypting the first portion of the file to create a first encrypted portion, obtaining a second portion of the file, encrypting the second portion of the file to create a second encrypted portion, and storing the first and second encrypted portions such that each of the first and the second encrypted portions can be individually accessed. A method of processing data from a file includes receiving a request to access a first portion of the file, wherein data in the first portion of the file is encrypted, and data in a second portion of the file is encrypted, and decrypting the data in the first portion, and not the data in the second portion.

Term
3.1 yearsto projected expiry
Projected expiry 16 November 2029, counted from filing; an application has no term until it is granted.
- Priority and filed
- Published
- Today
- Projected expiry
37 claims: 10 independent, 27 dependent
- 1A method of processing data from a file, comprising:obtaining a first portion of the file;encrypting the first portion of the file to create a first encrypted portion;obtaining a second portion of the file;encrypting the second portion of the file to create a second encrypted portion;and storing the first and second encrypted portions such that each of the first and the second encrypted portions can be individually accessed.
- 14A system for processing data from a file, comprising:means for obtaining a first portion and a second portion of the file;means for encrypting the first portion of the file to create a first encrypted portion, and the second portion of the file to create a second encrypted portion;and means for storing the first and second encrypted portions such that each of the first and the second encrypted portions can be individually accessed.
- 15A computer product having a computer-useable medium storing a set of instruction, wherein an execution of the instruction causes a process to be performed, the process comprising:obtaining a first portion of the file;encrypting the first portion of the file to create a first encrypted portion;obtaining a second portion of the file;encrypting the second portion of the file to create a second encrypted portion;and storing the first and second encrypted portions such that each of the first and the second encrypted portions can be individually accessed.
- 16A method of processing data from a file, comprising:receiving a request to access a first portion of the file, wherein data in the first portion of the file is encrypted, and data in a second portion of the file is encrypted;and decrypting the data in the first portion, and not the data in the second portion.
- 25Broadest claimClaim Score 98, very broad(NHIP)The method of claim. 23 , wherein the plurality of sub-units is associated with a compression unit.
- 33A system for processing data from a file, comprising:means for receiving a request to access a first portion of the file, wherein data in the first portion of the file is encrypted, and data in a second portion of the file is encrypted;and means for decrypting the data in the first portion, wherein the means for decrypting the data in the first portion is capable of decrypting the data in the first portion without decrypting the data in the second portion.
- 34A computer product having a computer-useable medium storing a set of instruction, wherein an execution of the instruction causes a process to be performed, the process comprising:receiving a request to access a first portion of the file, wherein data in the first portion of the file is encrypted, and data in a second portion of the file is encrypted;and decrypting the data in the first portion, and not the data in the second portion.
- 35A method of processing data from a file, comprising:decrypting at least a portion of the file using a first encryption key;encrypting the at least a portion of the file using a second encryption key;and updating a state indicator to indicate that the at least a portion of the file has been re-encrypted.
- 36A system for processing data from a file, comprising:means for decrypting at least a portion of the file using a first encryption key;means for encrypting the at least a portion of the file using a second encryption key;and means for updating a state indicator to indicate that the at least a portion of the file has been re-encrypted.
- 37A computer product having a computer-useable medium storing a set of instruction, wherein an execution of the instruction causes a process to be performed, the process comprising:decrypting at least a portion of the file using a first encryption key;encrypting the at least a portion of the file using a second encryption key;and updating a state indicator to indicate that the at least a portion of the file has been re-encrypted.
Independent claims10
134 paragraphs in 6 sections, as filed
RELATED APPLICATION DATA
0001This application is related to U.S. application Ser. No. ______, entitled “System and method for data de-duplication” having attorney docket No. OI7052242001, and U.S. application Ser. No. ______, entitled “System and method for data compression” having attorney docket No. OI7052222001.
FIELD
0002This application relates generally to systems and methods for storing and accessing data, and more specifically, to systems and methods for storing and accessing LOB data.
BACKGROUND
0003Files, such as LOB files, serve the goal of providing a content-rich store for data. Some applications give rise to the duplicated storage of LOB data, and thereby waste a significant amount of storage space. The ability to identify LOBs that have identical content and for those LOBs to share a single data repository is therefore desirable.
0004LOB data, residing in mainline or archived storage devices, can grow into very large sizes. This provides disk space and disk management challenges to administrators. Data compression is a commonly used mechanism to minimize disk space requirements. It is thus desirable to compress LOB data and provide random access to compressed data. In existing approaches, an algorithm is used to compress or decompress a source LOB to a destination LOB. The destination LOB is either a temporary LOB or an existing LOB, in which case it is overwritten. This technique requires creation of an explicit destination LOB and does not provide random access to LOB data.
0005Another problem with existing technique for storing LOB data is that the LOB data is stored unencrypted on-disk. However, in many cases, securing sensitive information is critical for meeting business and compliance requirements.
SUMMARY
0006In accordance with some embodiments, a method of processing data from a file includes obtaining a first portion of the file, encrypting the first portion of the file to create a first encrypted portion, obtaining a second portion of the file, encrypting the second portion of the file to create a second encrypted portion, and storing the first and second encrypted portions such that each of the first and the second encrypted portions can be individually accessed.
0007In accordance with other embodiments, a system for processing data from a file includes means for obtaining a first portion and a second portion of the file, means for encrypting the first portion of the file to create a first encrypted portion, and the second portion of the file to create a second encrypted portion, and means for storing the first and second encrypted portions such that each of the first and the second encrypted portions can be individually accessed.
0008In accordance with other embodiments, a computer product having a computer-useable medium storing a set of instruction, wherein an execution of the instruction causes a process to be performed, the process includes obtaining a first portion of the file, encrypting the first portion of the file to create a first encrypted portion, obtaining a second portion of the file, encrypting the second portion of the file to create a second encrypted portion, and storing the first and second encrypted portions such that each of the first and the second encrypted portions can be individually accessed.
0009In accordance with other embodiments, a method of processing data from a file includes receiving a request to access a first portion of the file, wherein data in the first portion of the file is encrypted, and data in a second portion of the file is encrypted, and decrypting the data in the first portion, and not the data in the second portion.
0010In accordance with other embodiments, a system for processing data from a file includes means for receiving a request to access a first portion of the file, wherein data in the first portion of the file is encrypted, and data in a second portion of the file is encrypted, and means for decrypting the data in the first portion, wherein the means for decrypting the data in the first portion is capable of decrypting the data in the first portion without decrypting the data in the second portion.
0011In accordance with other embodiments, a computer product having a computer-useable medium storing a set of instruction, wherein an execution of the instruction causes a process to be performed, the process includes receiving a request to access a first portion of the file, wherein data in the first portion of the file is encrypted, and data in a second portion of the file is encrypted, and decrypting the data in the first portion, and not the data in the second portion.
0012In accordance with other embodiments, a method of processing data from a file includes decrypting at least a portion of the file using a first encryption key, encrypting the at least a portion of the file using a second encryption key, and updating a state indicator to indicate that the at least a portion of the file has been re-encrypted.
0013In accordance with other embodiments, a system for processing data from a file includes means for decrypting at least a portion of the file using a first encryption key, means for encrypting the at least a portion of the file using a second encryption key, and means for updating a state indicator to indicate that the at least a portion of the file has been re-encrypted.
0014In accordance with other embodiments, a computer product having a computer-useable medium storing a set of instruction, wherein an execution of the instruction causes a process to be performed, the process includes decrypting at least a portion of the file using a first encryption key, encrypting the at least a portion of the file using a second encryption key, and updating a state indicator to indicate that the at least a portion of the file has been re-encrypted.
0015Other aspects and features will be evident from reading the following detailed description of the embodiments.
BRIEF DESCRIPTION OF THE DRAWINGS
0016The drawings illustrate the design and utility of embodiments, in which similar elements are referred to by common reference numerals. In order to better appreciate how advantages and objects of the embodiments are obtained, a more particular description of the embodiments will be illustrated in the accompanying drawings.
0017<figref idref="DRAWINGS">FIG. 1</figref> illustrates a system having a data receiving module and a data de-duplication module in accordance with some embodiments;
0018<figref idref="DRAWINGS">FIG. 2</figref> illustrates a process performed by the data receiving module of <figref idref="DRAWINGS">FIG. 1</figref> in accordance with some embodiments;
0019<figref idref="DRAWINGS">FIG. 3</figref> illustrates a process performed by the data de-duplication module of <figref idref="DRAWINGS">FIG. 1</figref> in accordance with some embodiments;
0020<figref idref="DRAWINGS">FIG. 4A</figref> illustrates metadata associated with a stored file in accordance with some embodiments, particularly showing file “XYZ” having been stored;
0021<figref idref="DRAWINGS">FIG. 4B</figref> illustrates metadata associated with a stored file in accordance with other embodiments;
0022<figref idref="DRAWINGS">FIG. 4C</figref> illustrates metadata associated with a stored file in accordance with other embodiments;
0023<figref idref="DRAWINGS">FIG. 5A</figref> illustrates metadata associated with a stored file in accordance with some embodiments, particularly showing two files, “XYZ” and “ABC” having the same data that are stored once;
0024<figref idref="DRAWINGS">FIG. 5B</figref> illustrates metadata associated with two stored files in accordance with some embodiments, particularly showing two files, “XYZ” and “ABC” having the same data that are stored twice;
0025<figref idref="DRAWINGS">FIG. 6</figref> illustrates metadata associated with a stored file in accordance with some embodiments, particular showing that the file “XYZ” has been removed;
0026<figref idref="DRAWINGS">FIG. 7</figref> illustrates metadata associated with a stored file in accordance with some embodiments, particularly showing that the file is separated into a plurality of blocks;
0027<figref idref="DRAWINGS">FIG. 8</figref> illustrates a system having a data compression module in accordance with some embodiments;
0028<figref idref="DRAWINGS">FIG. 9</figref> illustrates a process performed by the data compression module of <figref idref="DRAWINGS">FIG. 8</figref> in accordance with some embodiments;
0029<figref idref="DRAWINGS">FIG. 10</figref> illustrates compression units having sub-units in accordance with some embodiments;
0030<figref idref="DRAWINGS">FIG. 11A</figref> illustrates metadata of a stored file in accordance with some embodiments, particularly showing metadata having information regarding data compression;
0031<figref idref="DRAWINGS">FIG. 11B</figref> illustrates metadata of a stored file in accordance with other embodiments, particularly showing blocks of the file having their respective data compression information;
0032<figref idref="DRAWINGS">FIG. 12</figref> illustrates data compression maps associated with the compression units of <figref idref="DRAWINGS">FIG. 10</figref> in accordance with some embodiments;
0033<figref idref="DRAWINGS">FIG. 13</figref> illustrates compression units in accordance with other embodiments;
0034<figref idref="DRAWINGS">FIG. 14</figref> illustrates a system having a data encryption module in accordance with some embodiments;
0035<figref idref="DRAWINGS">FIG. 15</figref> illustrates a process performed by the data encryption module of <figref idref="DRAWINGS">FIG. 14</figref> in accordance with some embodiments;
0036<figref idref="DRAWINGS">FIG. 16A</figref> illustrates metadata of a stored file in accordance with some embodiments, particularly showing metadata having information regarding data encryption;
0037<figref idref="DRAWINGS">FIG. 16B</figref> illustrates metadata of a stored file in accordance with other embodiments, particularly blocks of the file having their respective data encryption information;
0038<figref idref="DRAWINGS">FIG. 17A-17B</figref> illustrate a technique of using a state indicator to keep track of a file that has been re-keyed; and
0039<figref idref="DRAWINGS">FIG. 18</figref> illustrates a block diagram of a computer system that can be used to perform various functions described herein in accordance with some embodiments.
DETAILED DESCRIPTION OF THE EMBODIMENTS
0040Various embodiments are described hereinafter with reference to the figures. It should be noted that the figures are not drawn to scale and elements of similar structures or functions are represented by like reference numerals throughout the figures. It should also be noted that the figures are only intended to facilitate the description of embodiments. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an aspect described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiments.
0041De-duplication
0042<figref idref="DRAWINGS">FIG. 1</figref> illustrates a system <b>10</b> configured to process data in accordance with some embodiments. In the illustrated embodiments, the system <b>10</b> is configured to receive a request from a client <b>12</b> to store data, receive the data, process the data, and store the data in a database <b>14</b> so that the data can be accessed at a later time. The client <b>12</b> may be a computer, or a handheld device, such as a phone, a PDA, a MP3 player, or other devices. In the illustrated embodiments, the clients <b>12</b><i>a, </i><b>12</b><i>b </i>are communicatively connected to the system <b>10</b> via the internet. Alternatively, the clients <b>12</b><i>a, </i><b>12</b><i>b </i>may be communicatively coupled to the system <b>10</b> using other techniques, such as, through a cable, or Bluetooth technology. Although only two clients <b>12</b><i>a, </i><b>12</b><i>b </i>are shown, it should be understood that the system <b>10</b> may communicate with more than two clients <b>12</b>. In the illustrated embodiments, the database <b>14</b> is communicatively coupled to the system <b>10</b>. In other embodiments, the database <b>14</b> may be a part of the system <b>10</b>. The system <b>10</b> may be implemented using a hardware, such as a computer, or a processor. In other embodiments, the system <b>10</b> may be implemented using software. In further embodiments, the system <b>10</b> may be implemented using a combination of hardware and software.
0043In the illustrated embodiments, the system is configured to receive LOB file from a client. The LOB file may have a large size, e.g., a size that is larger than 500 kb, and more particularly, a size that is larger than 1 Mb. Alternatively, the LOB file may have other sizes. In some embodiments, the LOB file may be an image file (e.g., a .gif file) or an audio file (e.g., a MP3 file). In other embodiments, the LOB file may be of other data types. In further embodiments, the system is configured to receive other type of files or objects from client(s) <b>12</b>.
0044As shown in <figref idref="DRAWINGS">FIG. 1</figref>, the system <b>10</b> includes a data receiving module <b>20</b> and a de-duplication module <b>22</b>. The data receiving module <b>20</b> is configured to receive and pass data to the de-duplication module <b>22</b> in a manner prescribed by the system <b>10</b>. The de-duplication module <b>20</b> is configured to analyze data received from a client <b>12</b>, and determine whether the received data is already stored at database <b>14</b>. In some cases, if the data is already stored at the database <b>14</b>, and duplication of the stored data is not desired, the de-duplication module <b>20</b> then updates the database <b>14</b> to reflect the fact that more than one client <b>12</b> has requested the same data be stored. In such cases, the duplication module <b>22</b> would not store an additional copy of the data. In other cases, if the data is already stored at the database <b>14</b>, and duplication of the stored data is desired, the de-duplication module <b>20</b> may then store the additional data even though a copy of which is already stored at the database <b>14</b>.
0045<figref idref="DRAWINGS">FIG. 2</figref> illustrates an example of a process <b>100</b> that is performed by the data receiving module <b>20</b> in accordance with some embodiments. In the illustrated example, the data requested by a client <b>12</b> (e.g., client <b>12</b><i>a</i>) to be stored by the system <b>10</b> is a LOB file <b>50</b>. However, it is understood that the system <b>10</b> can be configured to store other types of data. First, the data receiving module <b>20</b> receives a portion of the LOB file <b>50</b> (Step <b>102</b>). Next, the data receiving module <b>20</b> determines a size of the total received portion (Step <b>104</b>). The data receiving module <b>20</b> then compares the size with a prescribed data processing threshold (Step <b>106</b>). The data processing threshold may be, for examples, 260 kB, 1 MB, or other values set by a user. If the size of the cumulative collected data is equal or larger than the prescribed data processing threshold, the data receiving module <b>20</b> then passes the collected data (e.g., block <b>52</b><i>a</i>) downstream for further processing, and continues to collect the remaining portions of the LOB file <b>50</b> (Step <b>109</b>). On the other hand, if the size of the cumulative collected data is less than the prescribed data processing threshold, the data receiving module <b>20</b> then continues to collect additional portions of the LOB file <b>50</b> (Step <b>110</b>) until the size of the total collected portions reach the prescribed data processing threshold. As a result of the process <b>100</b>, the LOB file <b>50</b> is passed downstream (e.g., to the de-duplication module <b>22</b>) in the form of blocks <b>52</b> (e.g., blocks <b>52</b><i>a</i>-<b>52</b><i>e </i>in the example), each of which having a size that is equal or less then the prescribed processing threshold. Although five blocks <b>52</b> are shown in the example, in other examples, the file <b>50</b> may be separated into other numbers of portions/blocks <b>52</b>. Such technique is advantageous in that the de-duplication module <b>22</b> does not need to wait for an entire LOB file <b>50</b> to be collected before it starts analyzing the data of the LOB file. Also, in some cases, if the de-duplication module <b>22</b> (or another processing unit downstream) does not have enough memory to store the entire LOB file, the above data passing technique would allow the de-duplication module <b>22</b> (or another processing unit downstream) to process the data of the LOB file <b>50</b>.
0046In the illustrated embodiments, the system <b>10</b> includes a user interface that allows a user, such as an administrator, to input the prescribed data processing threshold. The user interface may include, for example, a screen, a keyboard, and a mouse. Also, in some embodiments, the user interface may allow a user to activate or deactivate the data receiving module <b>20</b>. If the data receiving module <b>20</b> is deactivated, data of the LOB file <b>50</b> received from the client <b>12</b> would be transmitted to the de-duplication module <b>22</b> without being processed by the data receiving module <b>20</b>.
0047<figref idref="DRAWINGS">FIG. 3</figref> illustrates a process <b>120</b> that is performed by the de-duplication module <b>22</b> in accordance with some embodiments. As shown in the figure, the de-duplication module <b>22</b> receives the LOB file <b>50</b> (Step <b>122</b>), and determines whether the LOB file <b>50</b> was already stored (Step <b>124</b>). Various techniques may be used to determine whether a LOB file has been stored previously. In some embodiments, the de-duplication module <b>22</b> is configured to calculate a hash value associated with the LOB file using an algorithm such as SHA1 or MD5. In such case, as the LOB file data is being received or written to a storage or temporary memory, a rolling hash value for the portion of the LOB file that has been received is calculated. A final hash value of the LOB file data is calculated at completion of the writing process, which uniquely identifies the LOB file data. In some embodiments, a B-tree may be used to maintain the calculated rolling hash value(s). Any of the techniques known in the art may be used to calculate the hash value.
0048For example, upon receiving the first block <b>52</b><i>a </i>of the LOB file <b>50</b>, the de-duplication module <b>22</b> then calculates a hash value for the block <b>52</b><i>a. </i>The de-duplication module <b>22</b> then checks to see if the calculated hash value can be found in the database <b>14</b>. For example, the de-duplication module <b>22</b> can look up a hash value table or a B-tree. If the calculated hash value cannot be found, then the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is not yet stored by the system <b>10</b>. Alternatively, if the calculated hash value can be found, the de-duplication module <b>22</b> then continues to receive the next block <b>52</b><i>b </i>of the LOB file <b>50</b>, and calculates a second hash value using the data from block <b>52</b><i>b. </i>The de-duplication module <b>22</b> then checks the database <b>14</b> again to see if the second calculated hash value can be found. If the second calculated hash value cannot be found, then the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is not yet stored by the system <b>10</b>. Alternatively, if the second calculated hash value can be found, the de-duplication module <b>22</b> then continues to receive the next block <b>52</b><i>c </i>of the LOB file, and calculates a third hash value using the data from block <b>52</b><i>b. </i>The above process is repeated until the last block <b>52</b> (e.g., block <b>52</b><i>e</i>) of the LOB file <b>50</b> is received and processed. If the last calculated hash value cannot be found, then the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is not yet stored by the system <b>10</b>. Alternatively, if the last calculated hash value can be found, then the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is already stored in database <b>14</b>.
0049In other embodiments, instead of performing a hash lookup for each of the blocks, the hash lookup may be performed after the LOB file <b>50</b> has been received. For example, the system <b>10</b> may be configured to detect an end of file (EOF), and upon detecting the EOF, the system <b>10</b> then performs data de-duplication, as discussed herein. Such technique may have the benefit of reducing consumption of CPU resource.
0050In the above embodiments, the data receiving module <b>20</b> passes portions of the LOB file <b>50</b> to the de-duplication module <b>22</b>, thereby allowing the de-duplication module <b>22</b> to process blocks <b>52</b> of the LOB file <b>50</b> have certain prescribed size. However, in other embodiments, the system <b>10</b> may not include the data receiving module <b>20</b>. In such cases, the de-duplication module <b>22</b> receives the entire LOB file <b>50</b> before it starts processing the LOB file <b>50</b>. Also, in further embodiments, the de-duplication module <b>22</b> may not need to calculate hash value(s). For example, in other embodiments, the calculation of hash value(s) for the LOB file <b>50</b> may be performed by another system (e.g., another computer or software that may communicate with the system <b>10</b> using the internet or other communication devices), or by the client <b>12</b>. In such cases, the de-duplication module <b>22</b> receives the hash value(s) and determines whether the LOB file <b>50</b> desired to be stored by the client <b>12</b> is already stored by the system <b>10</b> based on the received hash value(s).
0051Returning to <figref idref="DRAWINGS">FIG. 3</figref>, if the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is not already stored, the de-duplication module <b>22</b> then stores the LOB file <b>50</b> in the database <b>14</b> (Step <b>126</b>). In some embodiments, the system <b>10</b> is configured to receive the entire LOB file <b>50</b> before it is passed downstream to be stored. In other embodiments, the system <b>10</b> allows a data collection threshold to be inputted (e.g., via a user interface). In such cases, data of the LOB file <b>50</b> is passed downstream to be stored based on the prescribed data collection threshold. For example, the system <b>10</b> may be configured to monitor the size of the portion of the LOB file <b>50</b> that has been processed by the de-duplication module <b>22</b>. When the size of the portion reaches or exceeds the prescribed data collection threshold, the system <b>10</b> then stores the portion of the LOB file <b>50</b>.
0052In the illustrated embodiments, the system <b>10</b> also maintains metadata regarding the stored LOB file <b>50</b>. <figref idref="DRAWINGS">FIG. 4A</figref> illustrates a table <b>300</b> of metadata for the LOB file <b>50</b> that may be stored in the database <b>14</b>. As shown in the figure, the metadata includes the final calculated hash value <b>302</b> that uniquely identifies the LOB file <b>50</b>, an identifier <b>304</b> of the LOB file <b>50</b>, an address <b>306</b> of the physical storage location that stores the LOB file data, a counter <b>308</b>, and a de-duplication flag <b>310</b>. The identifier <b>304</b> of the LOB file <b>50</b> may be prescribed by the client (e.g., client <b>12</b><i>a</i>), and may be the name of the LOB file <b>50</b> that the client <b>12</b> wishes to use. In the illustrated example, the client <b>12</b><i>a </i>has requested that the name “XYZ” be used for the LOB file <b>50</b>. The address <b>306</b> may be implemented using a pointer that points to the physical storage location storing the LOB file data. In the illustrated example, the physical address <b>306</b> for the LOB file <b>50</b> “XYZ” is “A1.” The counter <b>308</b> represents the total number of client(s) <b>12</b> that have requested the same LOB file data to be stored by the system <b>10</b>. In the illustrated example, since the client <b>12</b><i>a </i>is the first one that requested the LOB file <b>50</b> be stored, the counter <b>308</b> is set to “1.” The de-duplication flag <b>310</b> is used to indicate whether duplication of already stored data is desired. In the illustrated example, the de-duplication flag <b>310</b> for the LOB file is set to “ON,” indicating that no duplication of already stored data is desired. Alternatively, setting the de-duplication flag <b>310</b> to “OFF” would indicate that duplication of already stored data is desired.
0053In the illustrated embodiments, the system <b>10</b> may include a user interface that allows a user, such as an administrator, to input the de-duplication flag <b>310</b> information. For example, the user may input certain file type(s) for which data de-duplication is desired. In other examples, the user may input a source address for which data de-duplication is desired. In such case, if the system <b>10</b> receives data from the prescribed source address, the system <b>10</b> then performs data de-duplication. In other embodiments, the whether to perform data de-duplication may be determined by the client <b>12</b> transmitting the LOB file <b>50</b>. For example, the client <b>12</b> may transmit the LOB file <b>50</b> to the system <b>10</b>, and requesting the system <b>10</b> to perform data de-duplication. In such cases, the system <b>10</b> determines that data de-duplication is desired if it receives a request from the client <b>12</b> to perform data de-duplication.
0054It should be noted that various techniques may be used to store the LOB file data and the metadata for the LOB file <b>50</b>, and that the system <b>10</b> should not be limited to using the example of the table <b>300</b> shown. For example, in other embodiments, the database <b>14</b> may include a hash value table <b>350</b> that identifies a unique index <b>352</b> for each hash value, and an index table <b>360</b> that contains metadata for each index value <b>352</b> (<figref idref="DRAWINGS">FIG. 4B</figref>). In such cases, after the de-duplication module <b>22</b> calculates the unique hash value. <b>302</b> for the LOB file <b>50</b>, the de-duplication module <b>22</b> then assigns and associates an index <b>352</b> with the hash value <b>302</b>. The index <b>352</b> may be used to retrieve metadata for the LOB file, and/or access the LOB file. For example, as shown in the figure, once the index <b>352</b> has been determined; the index table <b>360</b> may be used to obtain the address (which is “A1” in the example) of the physical location storing the LOB file data. In further embodiments, the metadata for the LOB file <b>50</b> may be stored using other techniques (e.g., using more than two tables), and may or may not be in table form.
0055In any of the embodiments described herein, the de-duplication flag <b>310</b> may be contained in the table <b>350</b>, instead of table <b>360</b> (<figref idref="DRAWINGS">FIG. 4C</figref>).
0056Referring again to <figref idref="DRAWINGS">FIG. 3</figref>, if the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is already stored, the de-duplication module <b>22</b> then updates the database <b>14</b> (Step <b>128</b>). Following the above example, assuming client <b>12</b><i>b </i>now transmits a request to the system <b>10</b> requesting that a LOB file having identifier “ABC” be stored, wherein the LOB file data is the same as that of file “XYZ” stored for client <b>12</b><i>a. </i>Because the LOB data for files “XYZ” and “ABC” are the same, the calculated hash value for the file “ABC” would be the same as the hash value for the “XYZ” file. The de-duplication module <b>22</b>, upon checking the table <b>300</b>, will determine that data being the same as that of the file “ABC” is already stored by the system <b>10</b> because the calculated hash value (“H3” in the example) can be found. The de-duplication module <b>22</b> then checks the table <b>300</b> to see if the already stored LOB file has a de-duplication flag <b>310</b> that is set “ON.” In the illustrated example, the LOB file associated with the hash value H<b>3</b> has a de-duplication flag <b>310</b> that is set to “ON,” indicating that there should not be any duplication of the LOB file data. As a result, the de-duplication module <b>22</b> does not store a duplicate copy of the LOB file data, but updates the database <b>14</b> by incrementing the counter value <b>308</b> by one (e.g., from “1” to “2”), indicating that there are now two clients <b>12</b> (clients <b>12</b><i>a </i>and <b>12</b><i>b </i>in the example) that have requested the same LOB file data to be stored by the system <b>10</b> (<figref idref="DRAWINGS">FIG. 5A</figref>). The de-duplication module <b>22</b> also updates the database <b>14</b> by inserting a LOB identifier “ABC” prescribed by the second client <b>12</b><i>b. </i>The above technique is advantageous in that the system <b>10</b> does not need to store multiple copies of the same LOB file for different clients <b>12</b>. Alternatively, if the de-duplication flag <b>310</b> is set to “OFF,” the de-duplication module <b>22</b> then stores the LOB file data transmitted by client <b>12</b><i>b </i>at another physical storage location (<figref idref="DRAWINGS">FIG. 5B</figref>), which in the example, is “B1.”
0057Assuming, in the example, that client <b>12</b><i>b </i>now wishes to access the LOB data for LOB file named “ABC.” The client <b>12</b><i>b </i>sends a request to the system <b>10</b> to access the data for LOB file “ABC.” Upon receiving the request, the system <b>10</b> checks the list of LOB identifiers (e.g., from table <b>300</b>) to see if the LOB identifier “ABC” can be found. As shown in <figref idref="DRAWINGS">FIG. 5A</figref>, the LOB file identifier “ABC” can be found in the table <b>300</b>, and has an address value of “A1,” which is the same address for the LOB file data stored for client <b>12</b><i>a. </i>The system <b>10</b> uses the address “A1” to retrieve the LOB file data requested by the client <b>12</b><i>b, </i>and transmits the LOB file data to the client <b>12</b><i>b. </i>
0058Following the above example, assuming that client <b>12</b><i>a </i>now sends a request to the system <b>10</b> requesting that the LOB file “XYZ” be deleted from the database <b>14</b>. Upon receiving the request, the system <b>10</b> checks the list of LOB identifiers (e.g., from table <b>300</b>) to see if the LOB identifier “XYZ” can be found. As shown in <figref idref="DRAWINGS">FIG. 5A</figref>, the LOB file identifier “XYZ” can be found in the table <b>300</b>, and has a counter value of “2.” The de-duplication module <b>22</b> then updates the counter from “2” to “1” and remove the LOB identifier “XYZ,” thereby “deleting” the LOB file “XYZ” without actually deleting the LOB file data (<figref idref="DRAWINGS">FIG. 6</figref>). As long as the counter value is larger than zero, indicating that the stored LOB file data needs to be preserved for at least one client (which in the example is client <b>12</b><i>b</i>), the actual LOB file data would not be deleted from the system <b>10</b>.
0059Following the above example, assuming that client <b>12</b><i>b </i>now sends a request to the system <b>10</b> requesting that the LOB file “ABC” be deleted from the database <b>14</b>. Upon receiving the request, the system <b>10</b> checks the list of LOB identifiers (e.g., from table <b>300</b>) to see if the LOB identifier “ABC” can be found. As shown in <figref idref="DRAWINGS">FIG. 6</figref>, the LOB file identifier “ABC” can be found in the table <b>300</b>, and has a counter value of “1.” The de-duplication module <b>22</b> then updates the counter from “1” to “0.” Upon detecting that the counter is now equal to “0” (representing the fact that there is no remaining client <b>12</b> that wishes the LOB file data be stored), the de-duplication module <b>22</b> deletes the LOB file data in the database <b>14</b>, thereby releasing the storage space for other purposes. In some embodiments, upon detecting that the counter <b>308</b> is “0,” the de-duplication module <b>22</b> does not delete the LOB file data immediately, but wait for a certain prescribed period (e.g., 3 months, 1 week, several days, etc.) before deleting the LOB file data from the database <b>14</b>. This allows the client <b>12</b> to undo the delete operation in the event that the client <b>12</b> changes its mind within the prescribed period.
0060In other embodiments, the counter <b>308</b> can be used for other purposes. For example, in some embodiments, instead of, or in addition to, using the counter <b>308</b> to determine when to release storage space, the counter <b>308</b> can be used to determine when to start a new index. For example, in the embodiments of <figref idref="DRAWINGS">FIG. 4B</figref>, the LOB files <b>50</b> associated with index “I6” are considered to be in a group. However, in some cases, if the number of LOB files <b>50</b> becomes too big in a group, it may take the system <b>10</b> to long to process data in the group. As such, it may be desirable to limit the number of LOB files <b>50</b> that in a group. For example, an upper counter limit, e.g., 100, may be prescribed by an administrator. In such cases, if the counter <b>308</b> is equal to or exceeds 100, a new index may be provided to start a new group. The new index would have a counter that starts from “1” and is associated with the same hash value (“H3” in the example).
0061In further embodiments, the counter <b>308</b> can be used to indicate that a client <b>12</b> has requested one or more copies of the file be made. For example, assuming that the system <b>10</b> already stores a LOB file <b>50</b> having a hash value “H3,” which corresponds to a request to store the file “XYZ” from client <b>12</b><i>a </i>and a request to store the file “ABC” from client <b>12</b><i>b </i>(i.e., the files “XYZ” and “ABC” have the same data and therefore, the same hash value). If the client <b>12</b><i>a </i>sends a request to make a copy of the LOB file “XYZ” in the database <b>14</b>, the system <b>10</b> can satisfy such request by updating the counter <b>308</b> for the LOB file “XYZ” by one (e.g., from “1” to “2”). As such, the system <b>10</b> satisfies the request without explicitly making and storing a copy of the stored LOB file <b>50</b>. In such cases, the metadata may include a plurality of counter values <b>308</b> for the LOB file <b>50</b>, wherein each counter value <b>308</b> is associated with a LOB file identifier (e.g., “XYZ,” “ABC,” etc.). In the above example, the counter value <b>308</b> for the LOB file <b>50</b> “XYZ” has been updated from “1” to “2,” and the counter value <b>308</b> for the LOB file <b>50</b> “ABC” remains as “1.” As such, the total counter value for the file associated with hash value “H3” is “3,” indicating that there have been three requests to store the same file data (two requests from client <b>12</b><i>a, </i>and one request from client <b>12</b><i>b</i>).
0062As another example, assuming that the following is stored in the system <b>10</b>:
0000<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="5"><colspec colname="1" colwidth="42pt" align="center" /><colspec colname="2" colwidth="49pt" align="left" /><colspec colname="3" colwidth="42pt" align="center" /><colspec colname="4" colwidth="35pt" align="center" /><colspec colname="5" colwidth="49pt" align="center" /><thead><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row><row><entry>Hash Val</entry><entry>LOB ID</entry><entry>Dedup Flag</entry><entry>Count</entry><entry>StorageAddr</entry></row><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>H3</entry><entry>XYZ, ABC</entry><entry>ON</entry><entry>2</entry><entry>A1</entry></row><row><entry>H4</entry><entry>XYZ1, ABC1</entry><entry>ON</entry><entry>2</entry><entry>A2</entry></row><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>
If the client wishes to copy the data from XYZ to XYZ
1
(that is, XYZ
1
is the destination of the copy, and is to have the same data as XYZ), then the diagram becomes
0000<tables id="TABLE-US-00002" num="00002"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="5"><colspec colname="1" colwidth="35pt" align="center" /><colspec colname="2" colwidth="63pt" align="left" /><colspec colname="3" colwidth="42pt" align="center" /><colspec colname="4" colwidth="35pt" align="center" /><colspec colname="5" colwidth="42pt" align="center" /><thead><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row><row><entry>Hash Val</entry><entry>LOB ID</entry><entry>Dedup Flag</entry><entry>Count</entry><entry>StorageAddr</entry></row><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>H3</entry><entry>XYZ, ABC, XYZ1</entry><entry>ON</entry><entry>3</entry><entry>A1</entry></row><row><entry>H4</entry><entry>ABC1</entry><entry>ON</entry><entry>1</entry><entry>A2</entry></row><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>
As shown in the example, the counter value
308
has been decremented for hash value H
4
, and incremented for hash value H
3
.
0064In the above embodiments, the LOB file data is associated with a single address (e.g., “A1” in the example). However, in any of the embodiments described herein, the LOB file data may be associated with more than one address. For example, in other embodiments, each block <b>52</b> of the LOB file <b>50</b> may have an associated address. <figref idref="DRAWINGS">FIG. 7</figref> illustrates an example in which the LOB file <b>50</b> is separated into five blocks (such as blocks <b>52</b><i>a</i>-<b>52</b><i>e </i>shown in <figref idref="DRAWINGS">FIG. 2</figref>). Blocks <b>52</b><i>a</i>-<b>52</b><i>e </i>have block identifiers “B1”-“B5,” respectively. The sizes of blocks <b>52</b><i>a</i>-<b>52</b><i>e </i>are 250 kb, 250 kb, 248 kb, 252 kb, and 30 kb, respectively. Also, in the illustrated example, blocks <b>52</b><i>a</i>-<b>52</b><i>e </i>are stored in addresses “A1”-“A5,” respectively. Assuming that the client <b>12</b><i>a </i>sends a request to the system <b>10</b> to access a portion of the saved LOB file <b>50</b> (e.g., data from 260 kb to 800 kb). The system <b>10</b> then looks up the table <b>300</b>, and based on the sizes of the blocks, determines that the requested data can be obtained from blocks B<b>2</b>-B<b>4</b>. In response to the client's <b>12</b><i>a </i>request, the system <b>10</b> then retrieves the data that correspond to blocks B<b>2</b>-B<b>4</b> using addresses A<b>2</b>-A<b>4</b>, respectively, and transmits the data to the client <b>12</b><i>a. </i>Such technique is advantageous in that the system <b>10</b> needs not process the entire LOB file <b>50</b> in order to allow the client <b>12</b> to access a portion of the LOB file <b>50</b>. In particular, since the system <b>10</b> can retrieve the requested data by accessing only the block(s) <b>52</b> that contains the requested data (individually accessing the block(s) <b>52</b>), the system <b>10</b> can provide the client <b>12</b> access to any one of the blocks <b>52</b> without having to access the entire LOB file <b>50</b> that contains all the file data.
0065In any of the embodiments described herein, the database <b>14</b> can be one or a combination of storage devices that can store data. For example, the database <b>14</b> can be a single storage device that is configured to store various information described herein. In some cases, the storage device maybe partitioned into a plurality of sub-storage devices, thereby allowing different information to be organized and maintained. In other examples, the database <b>14</b> can include two or more storage devices that are communicatively coupled (e.g., through internet or cable(s)) to each other. In such cases, the storage devices are configured to store different information described herein. In some embodiments, one or more of the storage devices may be partitioned to form a plurality of sub-storage devices.
0066Also, in other embodiments, the metadata described herein need not be stored in the database <b>14</b> according to the examples of the format shown previously. For example, in other embodiments, the metadata described herein may be stored in one or more tables. If a plurality of tables are used, one or more data from one table may be associated with one or more data from another table (e.g., using a common variable or a pointer). In further embodiments, the metadata described herein need not be stored in table format, and may be stored in the database <b>14</b> in other forms that are known in the art.
0067Data Compression
0068In some embodiments, the system <b>10</b> may further include a data compression module <b>24</b> (<figref idref="DRAWINGS">FIG. 8</figref>). The data compression module <b>24</b> is configured to compress data before the data is stored at the system <b>10</b>. The data compression module <b>24</b> may also be configured to decompress data in response to a client's request to retrieve/access the stored data.
0069<figref idref="DRAWINGS">FIG. 9</figref> illustrates a process <b>400</b> that includes data compression performed by the data compression module <b>24</b> in accordance with some embodiments. As shown in the figure, the de-duplication module <b>22</b> receives the LOB file <b>50</b> (Step <b>122</b>), and determines whether the LOB file <b>50</b> was already stored (Step <b>124</b>). If the LOB file <b>50</b> was already stored, then the system <b>10</b> updates the database <b>14</b> without storing a duplicate of the LOB file <b>50</b> (Step <b>128</b>). On the other hand, if the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is not yet stored, the de-duplication module <b>22</b> then passes the LOB file data to the data compression module <b>24</b>.
0070Upon detecting that there is a LOB file that is desired to be stored at the system <b>10</b>, the data compression module <b>24</b> first checks data compression criteria (Step <b>402</b>). The data compression criteria prescribes whether and/or how to perform data compression for the LOB file data based on certain rules set by a user. In the illustrated embodiments, the system <b>10</b> may include a user interface that allows a user, such as an administrator, to input the data compression criteria. For example, an administrator may prescribe one of four levels of compression, namely, “None,” “Low,” “Medium,” and “High” for a certain type or/and size of file. “None” compression is prescribed when no compression is desired to be performed for the file. “Low” compression is prescribed when some compression is desired. “High” compression is prescribed when significant compression is desired. In some cases, the user may, for example, prescribe a file size limit as a data compression criteria. In such cases, if the LOB file size is below the prescribed file size, the data compression module <b>24</b> may then perform a “Low” level of data compression, or may not perform any data compression at all. The user may, as another example, prescribe a file type and assign a certain level (e.g., “Medium”) of compression for such file type. For example, the user may prescribe all “.gif” file type be compressed at “Medium” level of compression. In other embodiments, the user interface may also allow the user to prescribe which data compression algorithm (which may or may not corresponds to a desired level of compression) to use for certain file based on the file type and/or file size. In other examples, the user may input a source address for which data compression is desired. In such case, if the system <b>10</b> receives data from the prescribed source address, the system <b>10</b> then performs data compression. The user may also prescribe which compression algorithm to use for data coming from certain prescribed source address. In other embodiments, the data compression criteria may be determined by the client <b>12</b> transmitting the LOB file <b>50</b>. For example, the client <b>12</b> may transmit the LOB file <b>50</b> to the system <b>10</b>, and requesting the system <b>10</b> to store the file <b>50</b> in a compressed form. In such cases, the system <b>10</b> determines that data compression is desired if it receives a request from the client <b>12</b> to compress the LOB file <b>50</b>.
0071In further embodiments, the data compression module <b>24</b> may be configured to determine compression efficiency on at least a portion of the LOB file, and automatically determines a level of data compression for the LOB file. For example, if the data compression module <b>24</b> determines that the compression efficiency is low (e.g., below a prescribed limit), the data compression module <b>24</b> may not perform any data compression for the LOB file. On the other hand, if the data compression module <b>24</b> determines that the compression efficiency is high (e.g., above a prescribed limit), the data compression module <b>24</b> may perform a “High” level of data compression for the LOB file.
0072Returning to <figref idref="DRAWINGS">FIG. 9</figref>, next, the data compression module <b>24</b> compresses the LOB file data based on the compression criteria obtained from Step <b>402</b> (Step <b>404</b>), and then stores the compressed LOB file data in the database <b>14</b> (Step <b>406</b>). In some cases, if it is determined by the data compression module <b>24</b>, based on the data compression criteria, that the received LOB file data is not to be compressed, the data compression module <b>24</b> then stores the LOB file data in uncompressed form at database <b>14</b> (Step <b>408</b>). The storing of the LOB file may be performed based on a data collection threshold, as similarly discussed herein.
0073In the illustrated embodiments, the data compression module <b>24</b> is configured to create one or more compression unit for the LOB file <b>50</b>. <figref idref="DRAWINGS">FIG. 10</figref> illustrates an example of a LOB file <b>50</b>, which is separated into blocks <b>52</b><i>a</i>-<b>52</b><i>e </i>by the data receiving module <b>20</b>. In the illustrated examples, blocks <b>52</b><i>a</i>-<b>52</b><i>e </i>have data sizes 250 kb, 250 kb, 248 kb, 252 kb, and 30 kb, respectively. The data compression module <b>24</b> then creates three compression units <b>500</b><i>a</i>-<b>500</b><i>c, </i>wherein each compression unit <b>500</b> corresponds to one or more blocks <b>52</b> having a total size that is less than (or equal to) a prescribed unit size. In the illustrated example, the prescribed unit size is 500 kb, which may be input by a user, such as an administrator. As a result, each compression unit <b>500</b> includes compressed data that correspond to uncompressed block(s) <b>52</b> having a total size that is 500 kb or less. In other embodiments, the uncompressed total size associated with a compression unit <b>500</b> can exceed the prescribed threshold. For example, in some cases, the system <b>10</b> may be configured to create a compression unit <b>500</b> when the total uncompressed data received exceeds the prescribed threshold (500 kb in the example).
0074As shown in the figure, compression unit <b>500</b><i>a </i>includes a data compression map <b>502</b><i>a, </i>a sub-unit <b>504</b><i>a </i>that corresponds to the block <b>52</b><i>a, </i>and a sub-unit <b>504</b><i>b </i>that corresponds to the block <b>52</b><i>b. </i>In particular, the sub-unit <b>504</b><i>a </i>is the compressed data of block <b>52</b><i>a, </i>and the sub-unit <b>504</b><i>b </i>is the compressed data of block <b>52</b><i>b. </i>The data compression map <b>502</b><i>a </i>is a variable-sized map for the compression unit <b>500</b><i>a, </i>which tracks the physical length and logical length of the sub-units <b>504</b><i>a, </i><b>504</b><i>b. </i>Similarly, compression unit <b>500</b><i>b </i>includes a data compression map <b>502</b><i>b, </i>a sub-unit <b>504</b><i>c </i>that corresponds to the block <b>52</b><i>c, </i>and a sub-unit <b>504</b><i>d </i>that corresponds to the block <b>52</b><i>d, </i>and compression unit <b>500</b><i>c </i>includes a compression map <b>502</b><i>c </i>and a sub-unit <b>504</b><i>e </i>that corresponds to the block <b>52</b><i>e. </i>As shown in the illustrated example, compression unit <b>500</b><i>a </i>corresponds to blocks <b>52</b><i>a, </i><b>52</b><i>b </i>having a total size of 500 kb (which is equal to the prescribed unit size of 500 kb), compression unit <b>500</b><i>b </i>corresponds to blocks <b>52</b><i>c, </i><b>52</b><i>d </i>having a total size of 500 kb (which is equal to the prescribed unit size of 500 kb), and compression unit <b>500</b><i>c </i>corresponds to block <b>52</b><i>e </i>having a total size of 30 kb (which is less than the prescribed unit size of 500 kb).
0075<figref idref="DRAWINGS">FIG. 11A</figref> illustrates a table <b>300</b> of metadata for the LOB file <b>50</b> that may be stored in the database <b>14</b> after the LOB file data have been processed by the data compression module <b>24</b>. As shown in the figure, the metadata includes the final calculated hash value <b>302</b> that uniquely identifies the LOB file <b>50</b>, an identifier <b>304</b> of the LOB file <b>50</b>, a counter <b>308</b>, and a de-duplication flag <b>310</b>. The hash value <b>302</b>, identifier <b>304</b>, counter <b>308</b>, and the de-duplication flag <b>310</b> are similar to those discussed with reference to <figref idref="DRAWINGS">FIG. 4A</figref>. In the illustrated example, the metadata also includes a data compression flag <b>602</b>, a data compression level <b>604</b>, a data compression algorithm identifier <b>606</b>. The data compression flag <b>602</b> is “ON,” indicating that the stored LOB file data has been compressed. The data compression level in the example is “High,” indicating that a high level of data compression is desired for the LOB file data. The data compression algorithm identifier <b>606</b> indicates that data compression algorithm “AL2” is used to perform data compression for the LOB file <b>50</b>. Alternatively, the data compression flag <b>602</b> may be “OFF,” indicating that no data compression for the LOB file <b>50</b> has been performed. In other embodiments, the metadata stored in the database <b>14</b> may not include one or some of the ones discussed. For example, in other embodiments, the metadata may not include the data compression algorithm identifier <b>606</b>.
0076In the illustrated example, the metadata also includes compressed unit identifiers <b>608</b><i>a</i>-<b>608</b><i>c </i>that are identifiers of compressed units <b>500</b><i>a</i>-<b>500</b><i>c, </i>respectively. The identifiers <b>608</b><i>a</i>-<b>608</b><i>c </i>have respective values “CU1,” “CU2,” and “CU3.” The metadata also includes the size <b>610</b> of the portion of the LOB file <b>50</b> that corresponds with each respective compressed unit <b>500</b>. In the illustrated example, the size <b>610</b><i>a </i>of the portion of the LOB file <b>50</b> associated with the compressed unit <b>500</b><i>a </i>is 500 kb, the size <b>610</b><i>b </i>of the portion of the LOB file <b>50</b> associated with the compressed unit <b>500</b><i>b </i>is 500 kb, and the size <b>610</b><i>c </i>of the portion of the LOB file <b>50</b> associated with the compressed unit <b>500</b><i>c </i>is 30 kb. In the illustrated example, the metadata further includes addresses <b>612</b><i>a</i>-<b>612</b><i>c </i>that represent the physical storage location associated with the respective compressed units <b>500</b><i>a</i>-<b>500</b><i>c. </i>In some embodiments, the addresses <b>612</b> may be implemented using pointers.
0077<figref idref="DRAWINGS">FIG. 12</figref> illustrates an example of the data compression maps <b>502</b><i>a</i>-<b>502</b><i>c. </i>The data compression map <b>502</b><i>a </i>includes block identifiers <b>650</b><i>a, </i><b>650</b><i>b, </i>the block sizes <b>652</b><i>a, </i><b>652</b><i>b </i>for blocks <b>52</b><i>a, </i><b>52</b><i>b, </i>respectively, and sub-unit sizes <b>654</b><i>a, </i><b>654</b><i>b, </i>for sub-units <b>504</b><i>a, </i><b>504</b><i>b, </i>respectively. The data compression map <b>502</b><i>a </i>also includes sub-unit addresses <b>656</b><i>a, </i><b>656</b><i>b </i>for the sub-units <b>504</b><i>a, </i><b>504</b><i>b, </i>respectively. The data compression map <b>502</b><i>b </i>includes block identifiers <b>650</b><i>c, </i><b>650</b><i>d, </i>the block sizes <b>652</b><i>c, </i><b>652</b><i>d, </i>for blocks <b>52</b><i>c, </i><b>52</b><i>d, </i>respectively, and sub-unit sizes <b>654</b><i>c, </i><b>654</b><i>d, </i>for sub-units <b>504</b><i>c, </i><b>504</b><i>d, </i>respectively. The data compression map <b>502</b><i>b </i>also includes sub-unit addresses <b>656</b><i>c, </i><b>656</b><i>d </i>for the sub-units <b>504</b><i>c, </i><b>504</b><i>d, </i>respectively. The data compression map <b>502</b><i>c </i>includes block identifier <b>650</b><i>e, </i>the block size <b>652</b><i>e </i>for block <b>52</b><i>e, </i>and sub-unit size <b>654</b><i>e </i>for the sub-unit <b>504</b><i>e. </i>The data compression map <b>502</b><i>c </i>also includes sub-unit address <b>656</b><i>e </i>for the sub-unit <b>504</b><i>e. </i>
0078Assuming that the client <b>12</b><i>a </i>wishes to access a portion of the LOB file <b>50</b> “XYZ,” wherein the portion is from the 260th kb to the 750th kb of the LOB file <b>50</b>. The client <b>12</b><i>a </i>sends a request to the system <b>10</b>, which looks up the metadata for the LOB file <b>50</b> “XYZ.” Based on the block sizes <b>610</b><i>a, </i><b>610</b><i>b, </i>from table <b>300</b>, which cover the range of data requested by the client <b>12</b><i>a, </i>the system <b>10</b> determines that the requested portion of the LOB file <b>50</b> can be obtained by accessing compression units <b>500</b><i>a </i>“CU1,” and compression unit <b>500</b><i>b </i>“CU2.” The system <b>10</b> then accesses the compression maps <b>502</b><i>a, </i><b>502</b><i>b </i>of the compression units <b>500</b><i>a, </i><b>500</b><i>b. </i>From the compression maps <b>502</b><i>a, </i>the system <b>10</b> determines that the first block <b>52</b><i>a </i>“B1” does not contain any requested data because it only covers data from 0 kb to 250 kb, and that the second block <b>52</b><i>b </i>“B2” contains at least a portion of the requested data because it covers data from 251 kb to 500 kb. Using the address <b>656</b><i>b </i>“SUA2” for the second sub-unit <b>504</b><i>b, </i>the system <b>10</b> then retrieves the compressed data for the second sub-unit <b>504</b><i>b. </i>The system <b>10</b> then decompresses the data (having a size of 20 kb) from the sub-unit <b>504</b><i>b </i>to obtain the uncompressed data (having a size of 250 kb), and transmits it to the client <b>12</b><i>a. </i>
0079In the example, since sub-unit <b>504</b><i>b </i>only provides data up to the 500th kb, in order to provide the remaining requested data (the 501st kb to the 750th kb) to the client <b>12</b><i>a, </i>the system <b>10</b> accesses the next data compression map <b>502</b><i>b. </i>According to the compression map <b>502</b><i>b, </i>the next sub-unit <b>504</b><i>c </i>can provide the 501st kb to the 748th kb. As such, the system <b>10</b> uses the sub-unit address <b>656</b><i>c </i>to retrieve the compressed data for the sub-unit <b>504</b><i>c. </i>The system <b>10</b> then decompresses the data (having a size of 5 kb) from the sub-unit <b>504</b><i>c </i>to obtain the uncompressed data (having a size of 248 kb), and transmits it to the client <b>12</b><i>a. </i>
0080In the example, since sub-units <b>504</b><i>a, </i><b>504</b><i>b </i>only provide data up to the 748th kb, in order to provide the remaining requested data (the 749th kb to the 750th kb) to the client <b>12</b><i>a, </i>the system <b>10</b> accesses the next sub-unit <b>504</b><i>d. </i>According to the compression map <b>502</b><i>b, </i>the next sub-unit <b>504</b><i>d </i>can provide the 749th kb to the 750th kb. As such, the system <b>10</b> uses the sub-unit address <b>656</b><i>d </i>to retrieve the compressed data for the sub-unit <b>504</b><i>d. </i>The system <b>10</b> then decompresses the data (having a size of 30 kb) from the sub-unit <b>504</b><i>d </i>to obtain the uncompressed data (having a size of 252 kb), and transmits it to the client <b>12</b><i>a. </i>
0081As shown in the above embodiments, the system <b>10</b> is configured to perform two mappings, i.e., mapping the sub-units <b>504</b> with their respective blocks <b>52</b> (logical mapping), and mapping the sub-units <b>504</b> with their respective addresses <b>656</b> (physical mapping). Such technique is advantageous in that it allows the client <b>12</b> to access a portion of the LOB file <b>50</b> without having to process (e.g., perform data decompression) on the entire LOB file <b>50</b>.
0082In the above embodiments, different blocks <b>52</b> of the same LOB file <b>50</b> have the same data compression criteria. In other embodiments, different blocks <b>52</b> of the same LOB file <b>50</b> may have different compression requirements (e.g., compression levels, compression algorithms, etc.). <figref idref="DRAWINGS">FIG. 11B</figref> illustrates a variation of the metadata stored in the database <b>14</b>, wherein different compression requirements are prescribed for different blocks <b>52</b> of the LOB file <b>50</b> “XYZ.” In the example, compression unit <b>500</b><i>a </i>“CU1” is obtained using compression algorithm “AL2” which provides “High” level of data compression for the corresponding block <b>52</b><i>a. </i>Compression unit <b>500</b><i>b </i>“CU2” is obtained using compression algorithm “AL3” which provides “Low” level of data compression for the corresponding block <b>52</b><i>b. </i>The compression flag for compression unit <b>500</b><i>c </i>“CU3” is set to “OFF,” indicating that the data in the compression unit <b>500</b><i>c </i>are not compressed. As shown in the figure, the compression level for the compression unit <b>500</b><i>c </i>is set to “None,” and no data compression algorithm is designated. In other embodiments, in addition to compression algorithm(s), the table <b>300</b> or the data compression map(s) <b>502</b> may also include information regarding which algorithm(s) to use for compressing and/or decompressing data from different sub-unit(s) <b>504</b>.
0083In the above embodiments, different sub-units <b>504</b> associated with same compression unit <b>500</b> may use different compression algorithms and decompression algorithms. In other embodiments, different sub-units <b>504</b> associated with the same compression unit <b>500</b> may use the same compression algorithm and decompression algorithm, but sub-units <b>504</b> from different compression units <b>500</b> may use different compression algorithms and de-decompression algorithms.
0084In the above embodiments, a compression unit <b>500</b> may correspond to a plurality of blocks <b>52</b> from the LOB file <b>50</b>. This is because the prescribed processing threshold (which is 260 kb in the above example) used by the data receiving module <b>20</b> to determine a size of a block <b>52</b>, is different from the prescribed unit size (which is 500 kb in the above example) used by the data compression module <b>24</b> to determine how many block(s) <b>52</b> is covered by a compression unit <b>500</b>. In other embodiments, each compression unit <b>500</b> may correspond to a single block <b>52</b>. <figref idref="DRAWINGS">FIG. 13</figref> illustrates an alternative embodiment in which each block <b>52</b> from the LOB file <b>50</b> is compressed to form a single compression unit <b>500</b>. In the illustrated example, the LOB file <b>50</b> is separated by the data receiving module <b>22</b>, based on the prescribed processing threshold, into five blocks <b>52</b><i>a</i>-<b>52</b><i>e. </i>The data compression module <b>24</b> then performs data compression to create compression units <b>500</b><i>a</i>-<b>500</b><i>e </i>for blocks <b>52</b><i>a</i>-<b>52</b><i>e, </i>respectively.
0085In the above embodiments, the data compression module <b>24</b> is configured to store the LOB file <b>50</b> in the form of a plurality of compression units <b>500</b>, such that the client <b>12</b> can perform random access of a portion of the LOB file <b>50</b>. However, in other embodiments, the system <b>10</b> may not provide such feature. In such cases, the data compression module <b>24</b> may receive the LOB file <b>50</b> and then perform data compression to create a single compressed file for the LOB file <b>50</b>.
0086As described in the above embodiments, the system <b>10</b> of <figref idref="DRAWINGS">FIG. 8</figref> includes both the de-duplication module <b>22</b> and the data compression module <b>24</b>. However, in other embodiments, the system <b>10</b> may not include the data de-duplication module <b>22</b>. In such cases, the data compression module <b>24</b> is not configured to receive LOB file data from the de-duplication module <b>22</b>, but from the data receiving module <b>20</b>. In some embodiments, if the system <b>10</b> does not include the de-duplication module <b>22</b>, each LOB file that is requested by a client <b>12</b> to be stored at the system <b>10</b> is compressed by the data compression module <b>24</b>, and is then stored at a physical storage unit, without accounting for the possibility that the LOB file may be a duplicate of an already stored file. In other embodiments, duplication of data may be detected by another system (e.g., by the client <b>12</b>, or by another processing unit coupled to the system <b>10</b>). In such cases, the data compression module <b>22</b> will compress LOB file data that is not already stored at the database <b>14</b>.
0087In any of the embodiments described herein, the system <b>10</b> may be configured to allow a data processing threshold to be inputted (e.g., via a user interface) for the data compression module <b>24</b>. The data processing threshold for the data compression module <b>24</b> is the prescribed size of a portion of the LOB file <b>50</b> to be processed by the data compression module <b>24</b>. For example, the data compression module <b>24</b> may be configured to keep track a size of the portion of the LOB file that has been processed by the data de-duplication module <b>22</b>. When the size reaches or exceeds the prescribed data processing threshold for the data compression module <b>24</b>, the data compression module <b>24</b> then performs the various functions associated with data compression described herein. The data processing threshold for the data compression module <b>24</b> may be the same as, or different from, that for the data de-duplication module <b>22</b>.
0088Data Encryption
0089In some embodiments, the system <b>10</b> may further include a data encryption module <b>26</b> (<figref idref="DRAWINGS">FIG. 14</figref>). The data compression module <b>26</b> is configured to encrypt data before the data is stored at the system <b>10</b>. The data encryption module <b>26</b> may also be configured to decrypt data in response to a client's request to retrieve/access the stored data.
0090<figref idref="DRAWINGS">FIG. 15</figref> illustrates a process <b>700</b> that includes data encryption performed by the data encryption module <b>26</b> in accordance with some embodiments. As shown in the figure, the de-duplication module <b>22</b> receives the LOB file <b>50</b> (Step <b>122</b>), and determines whether the LOB file <b>50</b> was already stored (Step <b>124</b>). If the LOB file <b>50</b> was already stored, then the system <b>10</b> updates the database <b>14</b> without storing a duplicate of the LOB file <b>50</b> (Step <b>128</b>). On the other hand, if the de-duplication module <b>22</b> determines that the LOB file <b>50</b> is not yet stored, the de-duplication module <b>22</b> then passes the LOB file data to the data compression module <b>24</b>, which performs data compression (Step <b>404</b>), as described previously.
0091After the LOB file data has been compressed, the LOB file data is then passed to the encryption module <b>26</b>. The data encryption module <b>26</b> first checks data encryption criteria (Step <b>702</b>). The data encryption criteria prescribes whether and/or how to perform data encryption for the LOB file data based on certain rules set by a user. In the illustrated embodiments, the system <b>10</b> may include a user interface that allows a user, such as an administrator, to input the data encryption criteria. For example, the user may input certain file type(s) for which data encryption is desired. The user may also prescribe which encryption algorithm to use for certain file type. In other examples, the user may input a source address for which data encryption is desired. In such case, if the system <b>10</b> receives data from the prescribed source address, the system <b>10</b> then performs data encryption. The user may also prescribe which encryption algorithm to use for data coming from certain prescribed source address. In other embodiments, the data encryption criteria may be determined by the client <b>12</b> transmitting the LOB file <b>50</b>. For example, the client <b>12</b> may transmit the LOB file <b>50</b> to the system <b>10</b>, and requesting the system <b>10</b> to store the file <b>50</b> in an encrypted form. In such cases, the system <b>10</b> determines that data encryption is desired if it receives a request from the client <b>12</b> to encrypt the LOB file <b>50</b>.
0092If the data encryption module <b>26</b> determines, based on the data encryption criteria, that the LOB file <b>50</b> is not to be encrypted, the data encryption module <b>26</b> then passes the LOB file <b>50</b> to be stored in the database <b>14</b> (Step <b>708</b>). The storing of the LOB file may be performed based on a data collection threshold, as similarly discussed herein.
0093On the other hand, if the data encryption criteria indicates that the LOB file <b>50</b> needs to be encrypted, the data encryption module <b>26</b> then performs data encryption for the LOB file <b>50</b> (Step <b>704</b>), and passes the encrypted LOB file <b>50</b> downstream to be stored at the database <b>14</b> (Step <b>706</b>). In some embodiments, the LOB file <b>50</b> is transmitted to the data encryption module <b>26</b> in the form of blocks <b>52</b>, as described previously. The blocks <b>52</b> may be compressed if the system <b>10</b> includes the data compression module <b>24</b>. In other embodiments, the blocks <b>52</b> may be uncompressed if the system <b>10</b> does not include the data compression module <b>24</b>, or if the blocks <b>52</b> are not prescribed to be compressed based on the prescribed compression criteria. If the LOB file <b>50</b> is passed to the data encryption module <b>26</b> in the form of blocks <b>52</b>, the data encryption module <b>26</b> then performs data encryption for the LOB file <b>50</b> on a block-by-block basis to create a plurality of encrypted blocks. Alternatively, the LOB file <b>50</b> may be passed to the data encryption module <b>26</b> as a single file. In such cases, the data encryption module <b>26</b> performs data encryption for the LOB file <b>50</b> to create a single encrypted LOB file.
0094Various techniques may be used by the data encryption module <b>26</b> to perform data encryption. In the illustrated embodiments, upon determining that the LOB file <b>50</b>, or a block of the LOB file <b>50</b>, needs to be encrypted, the data encryption module <b>26</b> obtains an encryption key from a database. The database providing the encryption key may be another database (not shown) or the database <b>14</b>. The data encryption module <b>26</b> then encrypts the LOB file <b>50</b> or the block(s) of the LOB file <b>50</b> using a prescribed encryption algorithm and the encryption key. In some embodiments, the encryption algorithm used is based on a type of the LOB file <b>50</b>. For example, an administrator may prescribe that encryption algorithm “EA2” be used by the data encryption module <b>26</b> to encrypt “.gif” file.
0095In the above embodiments, the data encryption is performed after the data compression. In other embodiments, instead of performing data encryption (Step <b>704</b>) after data compression (Step <b>404</b>), the system <b>10</b> may be configured to perform data compression (Step <b>404</b>) after data encryption (Step <b>704</b>).
0096<figref idref="DRAWINGS">FIG. 16A</figref> illustrates a table <b>300</b> of metadata for the LOB file <b>50</b> that may be stored in the database <b>14</b> after the LOB file data have been processed by the data encryption module <b>26</b>. The metadata are similar to those discussed with reference to <figref idref="DRAWINGS">FIG. 1A</figref>, except that the metadata of <figref idref="DRAWINGS">FIG. 16A</figref> also includes an encryption flag <b>800</b> and an encryption key address <b>802</b>. In the illustrated example, the encryption flag <b>800</b> is set to “ON,” indicating that the compression units “CU1,” “CU2,” and “CU3” are encrypted. The encryption key address <b>802</b> indicates the location at which the encryption key can be obtained. In some embodiments, the encryption key address <b>802</b> may be implemented using a pointer.
0097Assuming again, that the client <b>12</b><i>a </i>wishes to access a portion of the LOB file <b>50</b> “XYZ,” wherein the portion is from the 260th kb to the 750th kb of the LOB file <b>50</b>. The client <b>12</b><i>a </i>sends a request to the system <b>10</b>, which looks up the metadata for the LOB file <b>50</b> “XYZ.” Based on the encrypted block sizes <b>610</b><i>a, </i><b>610</b><i>b, </i>from table <b>300</b>, which cover the range of data requested by the client <b>12</b><i>a, </i>the system <b>10</b> determines that the requested portion of the LOB file <b>50</b> can be obtained by accessing encrypted compression units <b>500</b><i>a </i>“CU1,” and encrypted compression unit <b>500</b><i>b </i>“CU2.” The system <b>10</b> then accesses the compression maps <b>502</b><i>a, </i><b>502</b><i>b </i>of the encrypted compression units <b>500</b><i>a, </i><b>500</b><i>b. </i>From the compression maps <b>502</b><i>a, </i>the system <b>10</b> determines that the first block <b>52</b><i>a </i>“B1” does not contain any requested data because it only covers data from 0 kb to 250 kb, and that the second block <b>52</b><i>b </i>“B2” contains at least a portion of the requested data because it covers data from 251 kb to 500 kb. Using the address <b>656</b><i>b </i>“SUA2” for the second sub-unit <b>504</b><i>b, </i>the system <b>10</b> then retrieves the encrypted compressed data for the second sub-unit <b>504</b><i>b. </i>The system <b>10</b> then obtains the encryption key using the encryption key address <b>802</b>, and uses the encryption key to decrypt the data from sub-unit <b>504</b><i>b. </i>After the data from the sub-unit <b>504</b><i>b </i>is decrypted, the system <b>10</b> then decompresses the data (having a size of 20 kb) from the sub-unit <b>504</b><i>b </i>to obtain the uncompressed data (having a size of 250 kb), and transmits it to the client <b>12</b><i>a. </i>
0098In the example, since sub-unit <b>504</b><i>b </i>only provides data up to the 500th kb, in order to provide the remaining requested data (the 501st kb to the 750th kb) to the client <b>12</b><i>a, </i>the system <b>10</b> accesses the next data compression map <b>502</b><i>b. </i>According to the compression map <b>502</b><i>b, </i>the next sub-unit <b>504</b><i>c </i>can provide the 501st kb to the 748th kb. As such, the system <b>10</b> uses the sub-unit address <b>656</b><i>c </i>to retrieve the encrypted compressed data for the sub-unit <b>504</b><i>c. </i>The system <b>10</b> then uses the encryption key to decrypt the data from sub-unit <b>504</b><i>c. </i>After the data from the sub-unit <b>504</b><i>c </i>is decrypted, the system <b>10</b> then decompresses the data (having a size of 5 kb) from the sub-unit <b>504</b><i>c </i>to obtain the uncompressed data (having a size of 248 kb), and transmits it to the client <b>12</b><i>a. </i>
0099In the example, since sub-units <b>504</b><i>a, </i><b>504</b><i>b </i>only provide data up to the 748th kb, in order to provide the remaining requested data (the 749th kb to the 750th kb) to the client <b>12</b><i>a, </i>the system <b>10</b> accesses the next sub-unit <b>504</b><i>d. </i>According to the compression map <b>502</b><i>b, </i>the next sub-unit <b>504</b><i>d </i>can provide the 749th kb to the 750th kb. As such, the system <b>10</b> uses the sub-unit address <b>656</b><i>d </i>to retrieve the encrypted compressed data for the sub-unit <b>504</b><i>d. </i>The system <b>10</b> then uses the encryption key to decrypt the data from sub-unit <b>504</b><i>d. </i>After the data from the sub-unit <b>504</b><i>d </i>is decrypted, the system <b>10</b> then decompresses the data (having a size of 30 kb) from the sub-unit <b>504</b><i>d </i>to obtain the uncompressed data (having a size of 252 kb), and transmits it to the client <b>12</b><i>a. </i>
0100In the above embodiments, the same encryption key is used to encrypt and decrypt sub-units <b>504</b> from different compression units <b>500</b>. In other embodiments, different keys may be used to encrypt and decrypt different sub-units <b>504</b> associated with the same compression unit <b>500</b>. In such cases, the compression map <b>502</b> of a compression unit <b>500</b> may include information regarding the encryption keys (e.g., locations of the encryption keys) for decrypting different sub-units <b>504</b> of the compression unit <b>500</b>. In further embodiments, different compression units <b>500</b> may use different encryption keys, but sub-units <b>504</b> associated with the same compression unit <b>500</b> use the same encryption key.
0101<figref idref="DRAWINGS">FIG. 16B</figref> illustrates a variation of the example of the metadata shown in <figref idref="DRAWINGS">FIG. 16A</figref>, particularly showing that the compression units “CU1,” “CU2,” and “CU3” have different encryption criteria. Compression unit “CU1” has an encryption flag that is set to “ON,” indicating that the compression unit “CU1” has been encrypted, wherein the encryption key is located at address “K1.” Compression unit “CU2” (which may be, for example, a non-secure portion of the LOB file) has an encryption flag that is set to “OFF,” indicating that the compression unit “CU2” has not been encrypted. Compression unit “CU3” has an encryption flag that is set to “ON,” indicating that the compression unit “CU3” has been encrypted, wherein the encryption key is located at address “K2.” In the illustrated example, the system <b>10</b> is configured to use the encryption key located at “K1” to decrypt different sub-unit(s) <b>504</b> of the compression unit “CU1,” and the encryption key located at “K2” to decrypt different sub-unit(s) <b>504</b> of the compression unit “CU3.”
0102In other embodiments, instead of storing the encryption key address(es) in the database <b>14</b>, the system <b>10</b> may be configured to store the encryption key identifier(s) in the database <b>14</b>. In such cases, the encryption key identifier(s) may be used by the system <b>10</b> to obtain encryption key(s) from the database <b>14</b>, or from another location.
0103In further embodiments, the system <b>10</b> may use an encryption key management system, in which the encryption key is encrypted to provide an additional level of security. In such cases, the encrypted encryption key is stored in another system (e.g., a second database). If the system <b>10</b> needs to obtain the encryption key, the system <b>10</b> then sends a request to the system, which in turn, obtains a master key from yet another system (e.g., a third database) <b>10</b>. The second system then uses the master key to decrypt the encrypted encryption key, and transmits the decrypted encryption key to the system <b>10</b>. Encryption key management system has been described in U.S. patent application Ser. No. 11/084,346, entitled “METHOD AND APPARATUS FOR EXPIRING ENCRYPTED DATA”, the disclosure of which is hereby expressly incorporated by reference in its entirety.
0104In any of the embodiments described herein, the system <b>10</b> may be configured to allow a data processing threshold to be inputted (e.g., via a user interface) for the data encryption module <b>26</b>. The data processing threshold for the data encryption module <b>26</b> is the prescribed size of a portion of the LOB file <b>50</b> to be processed by the data encryption module <b>26</b>. For example, the data encryption module <b>26</b> may be configured to keep track a size of the portion of the LOB file that has been processed by the data compression module <b>24</b>. When the size reaches or exceeds the prescribed data processing threshold for the data encryption module <b>26</b>, the data encryption module <b>26</b> then performs the various functions associated with data encryption described herein. The data processing threshold for the data encryption module <b>26</b> may be the same as, or different from, that for the data de-duplication module <b>22</b> and/or the data compression module <b>24</b>.
0105As described in the above embodiments, the system <b>10</b> of <figref idref="DRAWINGS">FIG. 14</figref> includes the de-duplication module <b>22</b>, the data compression module <b>24</b>, and the data encryption module <b>26</b>. However, in other embodiments, the system <b>10</b> may not include the data compression module <b>24</b>. In such cases, the data encryption module <b>26</b> is not configured to receive LOB file data from the data compression module <b>24</b>, but from the de-duplication module <b>22</b>. In such cases, the data compression module <b>24</b> is configured to encrypt uncompressed data from the LOB file <b>50</b>. In other embodiments, the system <b>10</b> may not include the de-duplication module <b>22</b>. In further embodiments, the system <b>10</b> may not include both the de-duplication module <b>22</b> and the data compression module <b>24</b>.
0106In any of the embodiments described herein, the metadata may further include a state indicator for tracking a state of a LOB file. The state indicator may be used for different purposes. <figref idref="DRAWINGS">FIGS. 17A-17B</figref> illustrate an example of using state indicators <b>800</b> to keep track of which LOB file(s) <b>50</b> has been re-keyed. As shown in <figref idref="DRAWINGS">FIG. 17A</figref>, the LOB files <b>50</b><i>a</i>-<b>50</b><i>f </i>initially all have the same encryption key “K1,” and their corresponding state indicators <b>800</b><i>a</i>-<b>800</b><i>f </i>are “0.” In some cases, for added security or maintenance, it may be desirable to change the encryption key (re-key) for the LOB files <b>50</b> after a certain prescribed period. To re-key for LOB file <b>50</b><i>a, </i>the system <b>10</b> first obtains the original encryption key “K1” to decrypt the LOB file <b>50</b><i>a. </i>The system <b>10</b> then obtains a new encryption key “K2” to encrypt the LOB file <b>50</b><i>a. </i>After the re-key process has been performed form the LOB file <b>50</b><i>a, </i>the state indicator <b>800</b><i>a </i>for the LOB file <b>50</b><i>a </i>is then changed from “0” to “1,” indicating that the LOB file <b>50</b><i>a </i>has been re-keyed. The system <b>10</b> then performs the re-key process for the next LOB file <b>50</b><i>b, </i>and so forth, until the last LOB file <b>50</b><i>f </i>has been re-keyed.
0107Using a state indicator <b>800</b> to keep track of a state of a LOB file is advantageous in that it allows the system <b>10</b> to determine where in the table <b>300</b> to resume a re-key process if there is a system failure or error. For example, if there is a system failure or error that occurs after the fourth LOB file <b>50</b><i>d </i>has been re-keyed, the state indicators <b>800</b><i>a</i>-<b>800</b><i>d </i>would be “1” for the LOB files <b>50</b><i>a</i>-<b>50</b><i>d. </i>In such cases, the system <b>10</b> looks for the next state indicator that is “0” (which is the one for LOB file <b>50</b><i>e</i>), and determines that the re-key process needs to be resumed starting from LOB file <b>50</b><i>e. </i>In existing systems, maintenance of the files stored in a table is performed on a table-by-table basis. As such, if an error occurs before all of the items in the table are processed, the maintenance would need to be re-started from the beginning. This results in a waste of system resources.
0108In other embodiments, instead of using a state indicator <b>800</b> for each LOB file <b>50</b>, a state indicator may be used for each compression unit <b>500</b> to indicate a state of the compression unit <b>500</b> (e.g., whether the compression unit <b>500</b> has been re-keyed). In further embodiments, a state indicator <b>800</b> may be used for each sub-unit <b>504</b> to indicate a state of the sub-unit <b>504</b> (e.g., whether the sub-unit <b>504</b> has been re-keyed).
0109In further embodiments, instead of using the state indicator <b>800</b> to determine which LOB file <b>50</b>, compression unit <b>500</b>, or sub-unit <b>504</b>, has been re-keyed or not, the state indicator <b>800</b> may be used to determine which LOB file <b>50</b>, compression unit <b>500</b>, or sub-unit <b>504</b> has been re-compressed using a different compression algorithm.
0110It should be noted that the system <b>10</b> needs not perform all of the steps described previously, and that the system <b>10</b> can be configured to perform only one or some of the steps in <figref idref="DRAWINGS">FIG. 15</figref>. For example, in other embodiments, the stored data may not be encrypted, and the system <b>10</b> does not perform data decryption. In other embodiments, the stored data may not be compressed, and the system <b>10</b> does not perform data decompression. In further embodiments, the stored data may not be encrypted and compressed. In such case, the system <b>10</b> does not perform data decryption and data decompression.
0111Computer System Architecture
0112<figref idref="DRAWINGS">FIG. 18</figref> is a block diagram illustrating an embodiment of a computer system <b>1000</b> that can be used to perform various functions described herein. In some embodiments, the computer system <b>1000</b> may be used to implement the system <b>10</b>. In other embodiments, the computer system <b>1000</b> may be used to implement any of the components of the system <b>10</b>, such as, the data receiving module <b>20</b>, the data de-duplication module <b>22</b>, the data compression module <b>24</b>, or the data encryption module <b>26</b>. In further embodiments, the computer system <b>1000</b> may be used to implement the database <b>14</b>.
0113Computer system <b>1000</b> includes a bus <b>1002</b> or other communication mechanism for communicating information, and a processor <b>1004</b> coupled with the bus <b>1002</b> for processing information. The processor <b>1004</b> may be a processor in the system <b>10</b> of <figref idref="DRAWINGS">FIG. 1</figref> that is used to perform the various functions described herein. The computer system <b>1000</b> also includes a main memory <b>1006</b>, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus <b>1002</b> for storing information and instructions to be executed by the processor <b>1004</b>. The main memory <b>1006</b> also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor <b>1004</b>. The computer system <b>1000</b> further includes a read only memory (ROM) <b>1008</b> or other static storage device coupled to the bus <b>1002</b> for storing static information and instructions for the processor <b>1004</b>. A data storage device <b>1010</b>, such as a magnetic disk or optical disk, is provided and coupled to the bus <b>1002</b> for storing information and instructions.
0114The computer system <b>1000</b> may be coupled via the bus <b>1002</b> to a display <b>1012</b>, such as a cathode ray tube (CRT), for displaying information to a user. An input device <b>1014</b>, including alphanumeric and other keys, is coupled to the bus <b>1002</b> for communicating information and command selections to processor <b>1004</b>. Another type of user input device is cursor control <b>1016</b>, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor <b>1004</b> and for controlling cursor movement on display <b>1012</b>. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. The display <b>1012</b>, input device <b>1014</b>, and the cursor control <b>1016</b> may be used to implement various user interfaces described herein.
0115In some embodiments, the computer system <b>1000</b> can be used to perform various functions described herein. According to some embodiments of the invention, such use is provided by computer system <b>1000</b> in response to processor <b>1004</b> executing one or more sequences of one or more instructions contained in the main memory <b>1006</b>. Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read into the main memory <b>1006</b> from another computer-readable medium, such as storage device <b>1010</b>. Execution of the sequences of instructions contained in the main memory <b>1006</b> causes the processor <b>1004</b> to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in the main memory <b>1006</b>. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.
0116The term “computer-readable medium” as used herein refers to any medium that participates in providing instructions to the processor <b>1004</b> for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as the storage device <b>1010</b>. Volatile media includes dynamic memory, such as the main memory <b>1006</b>. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise the bus <b>1002</b>. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
0117Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
0118Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor <b>1004</b> for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system <b>1000</b> can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector coupled to the bus <b>1002</b> can receive the data carried in the infrared signal and place the data on the bus <b>1002</b>. The bus <b>1002</b> carries the data to the main memory <b>1006</b>, from which the processor <b>1004</b> retrieves and executes the instructions. The instructions received by the main memory <b>1006</b> may optionally be stored on the storage device <b>1010</b> either before or after execution by the processor <b>1004</b>.
0119The computer system <b>1000</b> also includes a communication interface <b>1018</b> coupled to the bus <b>1002</b>. The communication interface <b>1018</b> provides a two-way data communication coupling to a network link <b>1020</b> that is connected to a local network <b>1022</b>. For example, the communication interface <b>1018</b> may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface <b>1018</b> may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface <b>1018</b> sends and receives electrical, electromagnetic or optical signals that carry data streams representing various types of information.
0120The network link <b>1020</b> typically provides data communication through one or more networks to other devices. For example, the network link <b>1020</b> may provide a connection through local network <b>1022</b> to a host computer <b>1024</b> or to equipment/device <b>1026</b>, or a switch operatively coupled to any of the devices described herein. The data streams transported over the network link <b>1020</b> can comprise electrical, electromagnetic or optical signals. The signals through the various networks and the signals on the network link <b>1020</b> and through the communication interface <b>1018</b>, which carry data to and from the computer system <b>1000</b>, are exemplary forms of carrier waves transporting the information. The computer system <b>1000</b> can send messages and receive data, including program code, through the network(s), the network link <b>1020</b>, and the communication interface <b>1018</b>.
0121Although particular embodiments have been shown and described, it will be understood that it is not intended to limit the claimed inventions, and it will be obvious to those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the application. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense. The present inventions are intended to cover alternatives, modifications, and equivalents, which may be included within the spirit and scope of the present inventions as defined by the claims.
Contents6
18 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US2010258441A1 | Cited by | United States of America | Pre-grant |
| US2019129888A1 | Cited by | United States of America | Search report |
| US8892907B2 | Cited by | United States of America | Applicant |
| US10348693B2 | Cited by | United States of America | Applicant |
| US7519635B1 | Cited by | United States of America | Search report |
| US2008098083A1 | Cited by | United States of America | Pre-grant |
| US8775479B2 | Cited by | United States of America | Applicant |
| US9992020B1 | Cited by | United States of America | Search report |
| US2016117518A1 | Cited by | United States of America | Pre-grant |
| US11281642B2 | Cited by | United States of America | Applicant |
| US10922006B2 | Cited by | United States of America | Applicant |
| US2010031086A1 | Cited by | United States of America | Pre-grant |
| US2012311327A1 | Cited by | United States of America | Pre-grant |
| US9774572B2 | Cited by | United States of America | Search report |
| US2008144079A1 | Cited by | United States of America | Pre-grant |
| US10032038B2 | Cited by | United States of America | Search report |
| US8635194B2 | Cited by | United States of America | Applicant |
| US10348700B2 | Cited by | United States of America | Applicant |
| US7913114B2 | Cited by | United States of America | Search report |
| US2016337320A1 | Cited by | United States of America | Pre-grant |
| US11455212B2 | Cited by | United States of America | Applicant |
| US9465823B2 | Cited by | United States of America | Applicant |
| WO2011081738A3 | Cited by | World Intellectual Property Organization (WIPO) | International search |
| US2011145580A1 | Cited by | United States of America | Pre-grant |
| US9483486B1 | Cited by | United States of America | Search report |
| US11803860B2 | Cited by | United States of America | Applicant |
| US11709739B2 | Cited by | United States of America | Applicant |
| US2009037495A1 | Cited by | United States of America | Pre-grant |
| US2010199106A1 | Cited by | United States of America | Pre-grant |
| US2013054894A1 | Cited by | United States of America | Pre-grant |
| US2009169001A1 | Cited by | United States of America | Pre-grant |
| US8898138B2 | Cited by | United States of America | Search report |
| US7886124B2 | Cited by | United States of America | Applicant |
| US7920700B2 | Cited by | United States of America | Applicant |
| US2012041957A1 | Cited by | United States of America | Pre-grant |
| US8965856B2 | Cited by | United States of America | Search report |
| US10956274B2 | Cited by | United States of America | Search report |
| US2016004873A1 | Cited by | United States of America | Pre-grant |
| US9760719B2 | Cited by | United States of America | Search report |
| US10275603B2 | Cited by | United States of America | Applicant |
| US8621241B1 | Cited by | United States of America | Search report |
| US9910010B2 | Cited by | United States of America | Applicant |
| US2011145593A1 | Cited by | United States of America | Pre-grant |
| US10789207B2 | Cited by | United States of America | Search report |
| US9537650B2 | Cited by | United States of America | Applicant |
| US8095803B1 | Cited by | United States of America | Search report |
| US2011055471A1 | Cited by | United States of America | Pre-grant |
| US10977231B2 | Cited by | United States of America | Applicant |
| US2003065656A1 | Cites | United States of America | Pre-grant |
| US2004010621A1 | Cites | United States of America | Pre-grant |
| US2004172336A1 | Cites | United States of America | Pre-grant |
| US2007083473A1 | Cites | United States of America | Pre-grant |
| US2007088912A1 | Cites | United States of America | Pre-grant |
| US2008098083A1 | Cites | United States of America | Pre-grant |
| US2008144079A1 | Cites | United States of America | Pre-grant |
| US2008281846A1 | Cites | United States of America | Pre-grant |
| US2009024578A1 | Cites | United States of America | Pre-grant |
| US2009030956A1 | Cites | United States of America | Pre-grant |
| US2009037366A1 | Cites | United States of America | Pre-grant |
| US2009037495A1 | Cites | United States of America | Pre-grant |
| US2009037498A1 | Cites | United States of America | Pre-grant |
| US2009037499A1 | Cites | United States of America | Pre-grant |
| US2009106281A1 | Cites | United States of America | Pre-grant |
| US6343293B1 | Cites | United States of America | Pre-grant |
| US6349308B1 | Cites | United States of America | Pre-grant |
| US6377993B1 | Cites | United States of America | Pre-grant |
| US6460044B1 | Cites | United States of America | Pre-grant |
| US6567928B1 | Cites | United States of America | Pre-grant |
| US6598161B1 | Cites | United States of America | Pre-grant |
| US6910094B1 | Cites | United States of America | Pre-grant |
| US6947556B1 | Cites | United States of America | Pre-grant |
| US6957236B1 | Cites | United States of America | Pre-grant |
| US6976022B2 | Cites | United States of America | Pre-grant |
| US6981004B2 | Cites | United States of America | Pre-grant |
| US7036149B2 | Cites | United States of America | Pre-grant |
| US7062515B1 | Cites | United States of America | Pre-grant |
| US7146501B2 | Cites | United States of America | Pre-grant |
| US7200604B2 | Cites | United States of America | Pre-grant |
| US7418544B2 | Cites | United States of America | Pre-grant |
| US7467163B1 | Cites | United States of America | Pre-grant |
| US7496586B1 | Cites | United States of America | Pre-grant |
| US7546364B2 | Cites | United States of America | Pre-grant |
2 members in 1 office
Priority claims2
| Document | Office | Kind | Date |
|---|---|---|---|
| 58414406 | United States of America | A | |
| US20060584144 | – | – | – |
Members2
| Document | Office | Kind | |
|---|---|---|---|
| US2008098236A1 | United States of America | A1 | |
| US7920700B2 | United States of America | B2 |
59 transactions on the USPTO file
Allowed after 1 non-final rejection.
- Non-final rejections
- 1
- Final rejections
- 0
- RCEs
- 0
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Payment of Maintenance Fee, 8th Year, Large EntityM1552 | M1552 | |
| Recordation of Patent Grant MailedPGM/ | PGM/ | |
| Patent Issue Date Used in PTA CalculationAllowedPTAC | PTAC | |
| Issue Notification MailedAllowedWPIR | WPIR | |
| Dispatch to FDCD1935 | D1935 | |
| Application Is Considered Ready for IssuePILS | PILS | |
| Response to Reasons for AllowanceREAS | REAS | |
| Issue Fee Payment VerifiedN084 | N084 | |
| Issue Fee Payment ReceivedIFEE | IFEE | |
| Mail Miscellaneous Communication to ApplicantMM327 | MM327 | |
| Miscellaneous Communication to Applicant - No Action CountM327 | M327 | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Reference capture on IDSRCAP | RCAP | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Mail Examiner's AmendmentMEX.A | MEX.A | |
| Mail Notice of AllowanceAllowedMN/=. | MN/=. | |
| Notice of Allowance Data Verification CompletedAllowedN/=. | N/=. | |
| Examiner's Amendment CommunicationEX.A | EX.A | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Electronic Information Disclosure StatementEIDS. | EIDS. | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Non-Final ActionA... | A... | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Miscellaneous Incoming LetterLET. | LET. | |
| Electronic Information Disclosure StatementEIDS. | EIDS. | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Electronic Information Disclosure StatementEIDS. | EIDS. | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response to Election / Restriction FiledELC. | ELC. | |
| Mail Restriction RequirementMCTRS | MCTRS | |
| Restriction/Election RequirementCTRS | CTRS | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Electronic Information Disclosure StatementEIDS. | EIDS. | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| PG-Pub Issue NotificationPG-ISSUE | PG-ISSUE | |
| Withdraw Flagged for 5/25W525 | W525 | |
| Flagged for 5/25F525 | F525 | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Receipt of all Acknowledgement LettersL130 | L130 | |
| Receipt of Acknowledgment LetterL197 | L197 | |
| IFW TSS Processing by Tech Center CompleteTSSCOMP | TSSCOMP | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Application Is Now CompleteCOMP | COMP | |
| Additional Application Filing FeesADDFLFEE | ADDFLFEE | |
| A statement by one or more inventors satisfying the requirement under 35 USC 115, Oath of the ApplicOATHDECL | OATHDECL | |
| Notice Mailed--Application Incomplete--Filing Date AssignedINCD | INCD | |
| Agency Referral Letter MailedML196 | ML196 | |
| Referred by L&R for Third-Level Security Review. Agency Referral Letter GeneratedL196 | L196 | |
| Referred to Level 2 (LARS) by OIPE CSRL198 | L198 | |
| IFW Scan & PACR Auto Security ReviewSCAN | SCAN | |
| Initial Exam Team nnIEXX | IEXX |
6 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Maintenance fee paymentMAFP | MAFP | |
| Maintenance fee paymentMAFP | MAFP | |
| Fee paymentFPAY | FPAY | |
| Information on status: patent grantGrantedPATENTED CASESTCF | STCF | |
| AssignmentAS | AS | |
| AssignmentAS | AS |
Numbers
- Publication
- 20080098236
- Publication, DOCDB
- 2008098236
- Publication, EPODOC
- US2008098236
- Application
- 11584144
- Application, DOCDB
- 58414406
- Application, EPODOC
- US20060584144
Titles
- English
- System and method for data encryption
Patent term adjustment
- A delay
- +820 daysthe office missed an examination deadline
- B delay
- +533 dayspendency past three years
- Overlap
- −150 daysdelays counted once
- Applicant delay
- −79 days
- Net adjustment
- 1,124 days
Classification
- CPC, 5
- G06F11/1464
- H04L9/0891
- H04L9/0894
- H04L2209/30
- H04L2209/60
- IPC, 4
- H04L9 00
- G06F12 14
- H04L9 32
- G06F11 30
- USPC, 5
- 713189000
- 713165000
- 713167000
- 713193000
- 714E11207