Management of I/O operations in data storage systems

BACKGROUND

0001The present invention relates to managing I/O operations in data storage systems.

0002This application incorporates herein by reference as follows:

0003U.S. application Ser. No. 10/264,603, Systems and Methods of Multiple Access Paths to Single Ported Storage Devices, filed on Oct. 3, 2002 (Attorney Docket Pillar 701);

0004U.S. application Ser. No. 10/354,797, Methods and Systems of Host Caching, filed on Jan. 29, 2003 (Attorney Docket No. Pillar 709);

0005U.S. application Ser. No. 10/397,610, Methods and Systems for Management of System Metadata, filed on Mar. 26, 2003 (Attorney Docket No. Pillar 707);

0006U.S. application Ser. No. 10/440,347, Methods and Systems of Cache Memory Management and Snapshot Operations, filed on May 16, 2003 (Attorney Docket No. Pillar 713);

0007U.S. application Ser. No. 10/600,417, Systems and Methods of Data Migration in Snapshot Operations, filed on Jun. 19, 2003 (Attorney Docket No. Pillar 711);

0008U.S. application Ser. No. 10/616,128, Snapshots of File Systems in Data Storage Systems, filed on Jul. 8, 2003 (Attorney Docket No. Pillar 714);

0009U.S. application Ser. No. 10/677,560, Systems and Methods of Multiple Access Paths to Single Ported Storage Devices, filed on Oct. 1, 2003 (Attorney Docket No. Pillar 716);

0010U.S. application Ser. No. 10/696,327, Data Replication in Data Storage Systems, filed on Oct. 28, 2003 (Attorney Docket No. Pillar 715); and

0011U.S. application Ser. No. 10/837,322, Guided Configuration of Data Storage Systems, filed on Apr. 30, 2004 (Attorney Docket No. Pillar 720).

0012In data storage systems, the hosts will make I/O requests (i.e., reads and writes) of the data storage subsystems. Each user of the data storage system may need a different priority of service for the I/O request. The system adminstrator typically assigns the priority of service based on the type of work and/or the identity of the user.

0013For efficiency each host can accumulate a batch of I/O requests from users and transmit them to the data storage subsystem. If a host receives ten I/O requests, it will be desirable to process more high priority requests than low priority requests.

0014One problem is how to ensure that there is fairness between multiple hosts accessing a data storage subsystem. For example, if one host has all high priority requests and a second host has all low priority requests, each request will receive equal priority at a given data storage subsystem. We would like a host transmitting high priority I/O requests to be given more of the data storage subsystem's capacity than a host transmitting low priority I/O requests.

0015However, all the host I/O requests that arrive at the data storage subsystem for processing get intermixed without regard to priority. Since each host may have its own file system and logical unit number (LUN) of a storage area network (SAN), yet share the same data storage subsystem, there may be a contention problem.

0016For example, the data storage system should ensure a host with a higher quality of service (QoS) file system is not given lower priority than another host with a lower QoS file system and retain the ability to configure file systems and SAN LUNs by different QoS. At the same time, the data storage system should ensure that all I/O requests are completed in a reasonable time.

0017In an attempt to solve this problem, the hosts could communicate between each other on a regular basis to exchange information about pending I/O requests, but communications need to be frequent to manage the I/O requests and maintain a sense of priority. In addition, this will not scale well as additional hosts will add significant processing overhead.

0018It would be desirable if a solution existed that did not require such communication between the hosts since overhead would not increase as hosts increased. It would be also desirable if the solution ensured high priority I/O requests were guaranteed a certain amount of I/O resources, while still not starving out lower priority I/O requests. It would be helpful if the number of priority levels could be easily modified to allow for different priorities (e.g., two or more) to allow for better tuning of the system. The maximum number of I/O requests allowed per priority level could be then determined through testing and some qualitative analysis of different workloads.

SUMMARY OF THE INVENTION

0019The invention relates to management of I/O operations in data storage systems. In an embodiment, the invention provides a data storage subsystem processing I/O requests each having a priority, comprising a processor, a memory coupled to the processor, a disk array, an array controller coupled to the processor and the disk array, a network interface, coupled to the processor, to receive an I/O request with a priority, and a program in the memory for managing the I/O request based on the priority, a clip level of the priority, the total workload in the data storage subystem, and comparing the clip level to the total workload.

0020In another embodiment, the invention determines if the total workload is greater than the clip level or even if the total workload is above the clip level if the priority workload is below the fairness level and accepts the I/O request for processing such as incrementing the total workload and the priority workload, such as a read or a write to the disk array, either according to a write-back or a write-through scheme. Thus, the invention processes I/O requests based on priority, workload, clip levels, and fairness levels.

0021The invention also contemplates use of static and dynamic adjusted clip levels. In the case of dynamic clip levels, each priority of workload includes ranges such that if a priority of workload is in a lower range, the clip levels of the other priorities adjust to fully utilize I/O resources.

0022The invention also relates to a method of managing I/O requests, comprising receiving an I/O request, determining priority of the I/O request, comparing the total workload to the clip level for the priority, and processing the I/O request if the total workload is below the clip level or the priority workload is below a fairness level.

BRIEF DESCRIPTION OF THE DRAWINGS

0023<figref idref="DRAWINGS">FIG. 1</figref> illustrates a data storage system and provides details of a first host and of a first data storage subsystem.

0024<figref idref="DRAWINGS">FIG. 2</figref> represents a data structure in memory of a data storage subsystem to manage I/O requests according to clip levels, and fairness levels.

0025<figref idref="DRAWINGS">FIG. 3</figref> illustrates a flow chart of a method implemented in a data storage subsystem to handle I/O requests using clip levels and fairness levels.

0026<figref idref="DRAWINGS">FIG. 4</figref> is a diagram representing high, medium, and low priority workloads with respect to time, and high, medium, and low priority static clip levels.

0027<figref idref="DRAWINGS">FIG. 5</figref> illustrates a data structure in memory of a data storage subsystem to manage I/O requests according to dynamic clip levels and fairness levels.

0028<figref idref="DRAWINGS">FIG. 6</figref> illustrates a flow chart of a method implemented in a data storage subsystem to handle I/O requests using dynamic clip levels.

0029<figref idref="DRAWINGS">FIG. 7</figref> is a diagram that relates the low, medium, and high priority workloads with respect to time and dynamic clip levels.

0030<figref idref="DRAWINGS">FIG. 8</figref> illustrates how a host handles I/O requests from users and I/O requests rejected by a data storage subsystem.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

0031The following description includes the best mode of carrying out the invention, illustrates the principles of the invention, uses illustrative values, and should not be taken in a limiting sense. The scope of the invention is determined by reference to the claims. Each part or step is assigned its own number in the specification and drawings.

0032<figref idref="DRAWINGS">FIG. 1</figref> illustrates a data storage system <b>100</b> that includes first through Nth hosts <b>18</b>, <b>19</b> and <b>20</b>, and first through Nth data storage subsystems <b>44</b>, <b>46</b> and <b>48</b>. Each host is a computer that can connect to clients, data storage subsystems and other hosts using software/hardware interfaces such as network interface cards and software drivers to implement Ethernet, Fibre Channel, ATM, SCSI, InfiniBand, etc. Hennessy and Patterson, <i>Computer Architecture: A Quantitative Approach </i>(2003), and Patterson and Hennessy, <i>Computer Organization and Design: The Hardware/Software Interface </i>(2004) describe computer hardware and software, storage systems, memory, caching and networks and are incorporated herein by reference.

0033Each host runs an operating system such as Linux, UNIX, a Microsoft OS, or another suitable operating system. Tanenbaum, <i>Modern Operating Systems </i>(2001) describes operating systems in detail and is incorporated herein by reference.

0034<figref idref="DRAWINGS">FIG. 1</figref> shows the first host <b>18</b> includes a CPU-memory bus <b>14</b> that communicates with the processors <b>13</b> and <b>16</b> and a memory <b>15</b>. The processors <b>13</b> and <b>16</b> used are not essential to the invention and could be any suitable general-purpose processor such as an Intel Pentium processor, an ASIC dedicated to perform the operations described herein, or a field programmable gate array (FPGA).

0035Each host includes a bus adapter <b>22</b> between the CPU-memory bus <b>14</b> and an interface bus <b>24</b>, which in turn interfaces with network adapters <b>17</b>, <b>25</b> and <b>26</b>.

0036The first host <b>18</b> communicates through the network adapter <b>25</b> over a link <b>40</b> with a second host <b>19</b>. The first host <b>18</b> can also communicate through the network adapter <b>17</b> over link <b>28</b> with the local area network (LAN) <b>30</b>. The first host <b>18</b> also communicates through the network adapter <b>26</b> over a link <b>21</b> with a storage interconnect network <b>29</b>. Similarly, the second host <b>19</b> communicates over links <b>38</b> and <b>39</b> with the LAN <b>30</b> and the storage interconnect network <b>29</b>, respectively. The storage interconnect network <b>29</b> also communicates over links <b>32</b>, <b>34</b>, and <b>36</b> with the data storage subsystems <b>44</b>, <b>46</b>, and <b>48</b>, respectively.

0037In sum, the hosts <b>18</b>, <b>19</b> and <b>20</b> communicate with each other, the LAN <b>30</b> and storage interconnect network <b>29</b> and data storage subsystems <b>44</b>, <b>46</b>, and <b>48</b>.

0038The LAN <b>30</b> and the storage interconnect network <b>29</b> can be separate networks as illustrated or combined in a single network, and may be any suitable known bus, SAN, LAN, or WAN technology such as Fibre Channel, SCSI, InfiniBand, or Ethernet, and the type of interconnect is not essential to the invention. See Kembel, The FibreChannel Consultant, <i>A Comprehensive Introduction </i>(1998), Kembel, The FibreChannel Consultant, <i>Arbitrated Loop </i>(1996-1997) The FibreChannel Consultant, <i>Fibre Channel Switched Fabric </i>(2001), Clark, <i>Designing Storage Area Networks </i>(2003), Clark, <i>IP SANs: A Guide to iSCSI, iFCP, and FCIP Protocols for Storage Area Networks </i>(2002) and Clark, <i>Designing Storage Area Networks </i>(1999), which are incorporated herein by reference.

0039<figref idref="DRAWINGS">FIG. 1</figref> shows the first data storage subsystem <b>44</b> includes a CPU-memory bus <b>33</b> that communicates with the processor <b>31</b> and a memory <b>35</b>. The processor <b>31</b> used is not essential to the invention and could be any suitable general-purpose processor such as an Intel Pentium processor, an ASIC dedicated to perform the operations described herein, or a field programmable gate array (FPGA). The CPU-memory bus <b>33</b> also communicates through an adapter <b>41</b> and link <b>32</b> with the storage interconnect network <b>29</b> and an array controller <b>42</b>, such as a RAID controller, interfacing with an array of storage devices (e.g., a disk array <b>43</b>).

0040U.S. application Ser. No. 10/677,560, Systems and Methods of Multiple Access Paths to Single Ported Storage Devices, filed on Oct. 1, 2003 (Attorney Docket No. Pillar 716) describes suitable data storage subsystems, each containing a disk array, and is incorporated by reference herein. In an alternative embodiment, any other suitable array of storage devices can replace the disk arrays (e.g. an array of tape drives or an array of nonvolatile semiconductor memory). Massiglia, <i>The RAID Book: A Storage System Technology Handbook </i>(6th Edition, 1997) describing RAID technology is incorporated herein by reference.

0041A host may access secondary storage devices (e.g., disk drives) through a VLUN (virtual logical unit) that abstracts the storage device(s) as a linear array of fixed-size blocks. A logical block address (LBA) identifies each fixed-sized block. The data storage system constructs a VLUN from all or parts of several physical storage devices such as disk drives. To make a large VLUN, a data storage system may concatenate space allocated from several storage devices. To improve performance, the data storage system maps adjacent regions of VLUN space onto different physical storage devices (striping). To improve reliability, the system holds multiple copies of a VLUN on different storage devices (mirroring).

0042Users request write and read operations of the data storage system <b>100</b>. A system administrator can assign a low, medium, or high priority level for each type of work (e.g., backup, document production, and transaction processing). The priority can be set in an I/O request such as a SCSI command that provides a command descriptor block (CDB). For example, a three-bit field in the CDB can set up to eight levels of command priority. The width of the bit field determines the possible levels of priority.

0043In operation, a user requests an I/O operation of one of the hosts <b>18</b>, <b>19</b>, or <b>20</b> which will transmit the request on the LAN <b>30</b> or the storage interconnect network <b>29</b> to one of the data storage subsystems <b>44</b>, <b>46</b>, or <b>48</b>.

0044If a write is received, the data storage subsystem <b>44</b> can use a write-through scheme and not acknowledge the write until the data is written to nonvolatile memory (e.g., disk array <b>43</b>). This ensures data consistency between the host and data storage subsystem in the event of a power failure, etc.

0045In a write-back scheme, the data storage subsystem <b>44</b> can acknowledge the write before data is written to a disk array <b>43</b> as long as the data is stored in another form of nonvolatile memory (e.g., battery backed RAM) until written to the disk array to again ensure data consistency.

0046<figref idref="DRAWINGS">FIG. 2</figref> illustrates a memory of each data storage subsystem that contains values in a data structure (e.g., a table) for the I/O requests, the workload according to priorities (e.g., high, medium, and low), and the total workload clip level and the fairness level for each priority. In <figref idref="DRAWINGS">FIGS. 2, 4</figref>, <b>5</b>, <b>7</b>, and <b>8</b>, the values are for illustration purposes only; the actual values are determined on a system by system basis.

0047The total workload is a number representing the sum of the high, medium, and low priority I/O requests accepted for processing in the data storage subsystem. If an I/O request is accepted for processing, the data storage subsystem increments the total workload by one. After the I/O request is processed, e.g., the write or read is completed, the data storage subsystem decrements the total workload by one and acknowledges completion to the host.

0048Each I/O request has a priority that relates to a clip level and a fairness level that is used to determine if the I/O request will be accepted for processing by the data storage subsystem. The fairness levels ensure sufficient I/O resources so each I/O request is processed within a reasonable time.

0049When an I/O request arrives at the data storage subsystem, the data storage subsystem will process the I/O request in one of the following ways: (1) the data storage subsystem will reject the I/O request with a busy status to the host if the total workload is above the maximum that the data storage subsystem can process; (2) the data storage subsystem will reject the I/O request with an insufficient priority status to the host if the priority of the request is insufficient; and (3) the data storage subsystem will accept the I/O request for processing if it is not too busy and the priority is sufficient. After processing is complete, the data storage subsystem will transmit completion status to the host.

0050<figref idref="DRAWINGS">FIGS. 2-3</figref> illustrate a method in the data storage subsystem for management of I/O requests. Preferably, the method is implemented in a higher level language, e.g., the C programming language, which instructs the the data storage subsystem.

0051Referring to <figref idref="DRAWINGS">FIG. 3</figref>, the method starts at step <b>50</b> when the data storage subsystem receives an I/O request, for example, from a host. At step <b>54</b>, the data storage subsystem determines if the total workload exceeds a max value. The max value represents the maximum number I/O requests the data storage subsystem can process. If the total workload exceeds the max, the data storage subsystem rejects the I/O request at step <b>56</b> and at step <b>600</b> returns to the main program, and if not, the data storage subsystem proceeds to step <b>60</b>. At step <b>60</b>, the data storage subsystem determines the priority of the I/O request (e.g., high).

0052Let's first assume the data storage subsystem determines the I/O request is high priority at step <b>60</b>. At step <b>62</b>, the data storage subsystem will determine if the total workload is greater than a high priority clip level. In <figref idref="DRAWINGS">FIG. 2</figref>, the high priority clip level=500. If the total workload does not exceed the high priority clip level, the data storage subsystem will process the I/O request as follows:

00531) At step <b>65</b>, the data storage subsystem increments by one the total workload and the high priority workload, that is, the high priority I/O requests and the total of I/O request that will be processed in the data storage subsystem;

00542) At step <b>66</b>, the data storage subsystem processes the I/O request either by a read or a write using a write-through or write-back scheme;

00553) At step <b>67</b>, the data storage subsystem decrements by one the high priority workload and the total workload, since the I/O request was processed at step <b>66</b>; and

00564) At step <b>89</b>, the data storage subsystem acknowledges the I/O request was processed, that is, it sends a I/O request complete response to the host. Finally, the method returns to the main program at step <b>608</b>.

0057Returning to step <b>62</b>, even if the total workload is greater than the clip level, the data storage subsystem will process a certain number of high priority I/O requests as a matter of fairness. Thus, at step <b>63</b> the data storage subsystem will process a high priority I/O request if the number of high priority workload falls below the fairness level. Otherwise, a burst of medium and low priority I/O requests could prevent high priority I/O requests from getting timely processed by the data storage subsystem.

0058<figref idref="DRAWINGS">FIGS. 2-3</figref> illustrate the data storage subsystem will process a high priority I/O request even if the total workload is greater than the clip level (e.g., 500 total I/O requests) as long as a certain number (e.g., 200) of high priority I/O requests is not being processed. Thus, the data storage subsystem will again execute steps <b>65</b>, <b>66</b>, <b>67</b>, and <b>89</b> as described earlier. Conversely, if the fairness level is met, the high priority I/O request will be rejected at step <b>64</b>. Finally, the method returns to the main program at step <b>602</b>. In short, the fairness level ensures high priority I/O requests are handled to give excellent response time.

0059Let's now assume the data storage subsystem determined the I/O request was medium priority at step <b>68</b>:

0060At step <b>70</b>, the data storage subsystem will determine if the total workload is greater than a clip level (e.g., 250) associated with the medium priority I/O requests.

0061If the total workload does not exceed the medium priority clip level, the data storage subsystem will process the medium priority I/O request as follows:

00621) At step <b>72</b>, the data storage subsystem increments by one the total workload and the medium priority workload in the data storage subsystem;

00632) At step <b>66</b>, the data storage subsystem processes the I/O request either by a read or a write using a write-through or write-back scheme;

00643) At step <b>74</b>, the data storage subsystem decrements by one the total workload and the medium priority workload in the data storage subsystem, since the I/O request is no longer using data storage subsystem capacity; and

00654) At step <b>89</b>, the data storage subsystem sends a I/O request complete response to the host. Finally, the method returns to the main program at step <b>608</b>.

0066Returning to step <b>70</b>, even if the total workload is greater than the clip level, the data storage subsystem processes a certain number of medium priority I/O requests as a matter of fairness. Thus, if the data storage subsystem determines that it is not processing a minimum number (e.g., 100) of medium priority I/O requests set by the fairness level at step <b>76</b>, the data storage subsystem will process the medium priority I/O request by executing steps <b>72</b>, <b>66</b>, <b>74</b>, and <b>89</b>. Conversely, if the clip level is exceeded and the fairness level met, the medium priority I/O request will be rejected at step <b>78</b> and the method returns to the main program at step <b>604</b>.

0067Now let's assume, the data storage subsystem determined that the I/O request is low priority at step <b>80</b>:

0068At step <b>80</b>, the data storage subsystem determines if the total workload is greater than a clip level (e.g., 75) associated with the low priority I/O requests.

0069If the total workload does not exceed the low priority clip level, the data storage subsystem will process the low priority I/O request as follows:

00701) At step <b>86</b>, the data storage subsystem increments by one the number of total workload and low priority workload in the data storage subsystem;

00712) At step <b>66</b>, the data storage subsystem processes the I/O request either by a read or a write using a write-through or write-back scheme;

00723) At step <b>88</b>, the data storage subsystem decrements by one the total workload and low priority workload in the data storage subsystem, since the I/O request is no longer using data storage subsystem capacity; and

00734) At step <b>89</b>, the data storage subsystem sends a I/O request complete response to the host. Finally, the method returns to the main program at step <b>608</b>.

0074Returning to step <b>80</b>, even if the total workload is greater than the clip level, the data storage subsystem processes some low priority I/O requests as a matter of fairness. The data storage subsystem processes a low priority I/O request by executing steps <b>86</b>, <b>66</b>, <b>88</b>, and <b>89</b> if the data storage subsystem is not processing a minimum number (e.g., 15) of low priority I/O requests set by the fairness level as determined at step <b>82</b>. Conversely, if the clip level is exceeded and the fairness level met, the low priority I/O request will be rejected at step <b>84</b>. Finally, the method returns to the main program at step <b>606</b>.

0075Thus, each priority has a fairness level that determines if an I/O request of a given priority will be accepted by the data storage subsystem. This ensures the data storage subsystem reserves a minimum amount of resources for each priority of I/O request.

0076<figref idref="DRAWINGS">FIG. 4</figref> is a diagram of each priority of workload with respect to time when the data storage subsystem has static clip levels. The data storage subsystem has a high priority clip level of 500, medium priority clip level of 250, and low priority clip level of 75. Initially, the data storage subsystem processes the high, medium, and low priority clip levels below their respective clip levels. As time proceeds, the high priority I/O requests drops from level <b>93</b> along the curve <b>96</b> to level <b>97</b>. In the meantime, the medium priority and low priority workloads remain constant at levels <b>94</b> and <b>95</b> that is below the illustrative medium and low priority clip levels of 250 and 75.

0077<figref idref="DRAWINGS">FIG. 5</figref> illustrates another embodiment of values held in a data structure in the memory of each data storage subsystem to manage I/O requests according to dynamic clip levels. Each I/O request has a high, medium, or low priority.

0078<figref idref="DRAWINGS">FIG. 5</figref> depicts an upper range (e.g., 375-500) and a lower range (e.g., 0-374) for high priority workload. The data storage subsystem use the ranges to dynamically adjust the clip levels. For example, if the data storage subsystem has a high priority workload in the upper range (e.g., 375-500), the high, medium, and low priority clip levels stay at their default values. In contrast, if the data storage subsystem has a high priority workload in the lower range (e.g., 0-374) the data storage subsystem increases the medium priority clip level upward from a default value (e.g., 250) to an adjusted clip level (e.g., 400), and/or the low priority clip level upward from a default value (e.g., 75) to an adjusted clip level (e.g., 160). This dynamic clip adjustment ensures that the data storage subsystem is more fully utilized for processing medium priority and low priority I/O requests, when the high priority workload falls into lower range.

0079The data structure shows an upper range (e.g., 150-250) and a lower range (e.g., 0-149) for medium priority workload. The data storage subsystem will use these ranges to dynamically adjust the high priority clip level and/or the low priority clip level. For example, if the data storage subsystem has a medium priority workload in the upper range (e.g., 150-250), the high, medium, and low priority clip levels stay at their default values. In contrast, if the data storage subsystem has a medium priority workload in the lower range (e.g., 0-149) the data storage subsystem increases the high priority clip level from the default value (e.g., 500) to an adjusted clip level (e.g., 700) and/or the low priority clip level from the default value (e.g., 75) to an adjusted clip level (e.g., 160). This dynamic clip adjustment again ensures the data storage subsystem is better used for processing high and low priority I/O requests, when the medium priority workload falls into the lower range.

0080<figref idref="DRAWINGS">FIG. 6</figref> illustrates a method that instructs a data storage subsystem to process I/O requests using dynamic clip levels for three priorities with two ranges per priority.

0081The method starts at step <b>110</b>. At step <b>112</b>, the data storage subsystem determines if the high priority workload is in the upper range (e.g., 375-500). If in the upper range, the data storage subsystem sets (or maintains) the high, medium, and low priority clip levels (e.g., 500, 250, and 75) to their default values at step <b>114</b>. If the data storage subsystem determines the high priority workload is in the lower range (e.g., 0-374), the data storage subsystem increases the medium and low priority clip levels from their default values (e.g., 250 and 75) to adjusted clip levels (e.g., 400 and 160) at step <b>120</b>. At step <b>300</b>, the method returns to the main program.

0082At step <b>122</b>, the data storage subsystem determines if medium priority workload is in an upper range (e.g., 150-250). If in the upper range, the data storage subsystem sets (or maintains) the high, medium, and low priority clip levels at their default values (e.g., 500, 250, and 75) at step <b>130</b> and then returns to the main program at step <b>302</b>. If the data storage subsystem determines the medium priority workload is in the lower range (e.g., 0-149), the data storage subsystem increases the low priority clip level from its default value (e.g., 75) to its adjusted clip level (e.g., 160) at step <b>128</b>. At step <b>302</b>, the method returns to the main program.

0083In alternative embodiments, the method of <figref idref="DRAWINGS">FIG. 6</figref> can be implemented with two or more priorities with two or more ranges per priority. If higher priority workload is in a lower range, the data storage subsystem increases a clip level associated with lower priority workload to best use the capacity of data storage subsystem.

0084<figref idref="DRAWINGS">FIG. 7</figref> is a diagram that relates the workload of each priority with respect to time for dynamic clip levels. The data storage subsystem initially processes high, medium, and low priority I/O requests at levels <b>93</b>, <b>94</b>, and <b>95</b>. This is not actual experimental results but represents the workload for the data storage subsystem and high, medium, and low priority default values of the clip levels (e.g., 500, 250, and 75). The high priority workload drops along the curve <b>148</b>, then flattens at point <b>158</b> then rises at point <b>161</b>. As the high priority workload falls from the upper range (e.g., 375-500) the data storage subsystem increases the medium and low priority clip levels from their default values 250 and 75 to adjusted clip levels <b>153</b> and <b>155</b> (e.g., 400 and 160). As a result of the increased medium priority clip level, the medium priority workload begins to rise at point <b>157</b>, following curve <b>154</b> to point <b>162</b>. Similarly, the low priority workload begins to rise at point <b>152</b>, following curve <b>156</b> to point <b>164</b>. However, when the high priority workload begins to rise again at point <b>161</b>, the medium priority and low priority clip levels go back to their default values (e.g., 250 and 75), and the medium priority workload falls from point <b>162</b> to <b>166</b>, and the low priority workload from point <b>164</b> to <b>168</b>.

0085<figref idref="DRAWINGS">FIG. 8</figref> illustrates values held in memory of each host to handle I/O requests from users and rejected I/O requests from the data storage subsystem. The table enables batching of I/O requests and resending of I/O requests rejected by the data storage subsystem. Each priority of I/O request has its own values for batch size (i.e., number of I/O requests), maximum dwell time (milliseconds), and minimum backoff time to resend (milliseconds). The host transmits the I/O requests to the data storage subsystem when the I/O requests in the host meet the batch size. The host sorts the batch of I/O requests to reduce the seek time on the disk array. However, if the I/O requests count does not reach the batch size by a maximum dwell time, the host will transmit I/O requests to the data storage subsystem to avoid delay. The host will also wait for a minimum backoff time before resending a previously rejected I/O request to the data storage subsystem to reduce the likelihood of another rejection.