Data center energy management system

FIELD OF THE INVENTION

[0001] This invention relates generally to data centers. More particularly, the invention pertains to energy management of data centers.

BACKGROUND OF THE INVENTION

[0002] Computers typically include electronic packages that generate considerable amounts of heat. Typically, these electronic packages include one or more components such as CPUs (central processing units) as represented by MPUs (microprocessor units) and MCMs (multi-chip modules), and system boards having printed circuit boards (PCBs) in general. Excessive heat tends to adversely affect the performance and operating life of these packages. In recent years, the electronic packages have become more dense and, hence, generate more heat during operation. When a plurality of computers are stored in the same location, as in a data center, there is an even greater potential for the adverse effects of overheating.

[0003] A data center may be defined as a location, e.g., room, that houses numerous electronic packages, each package arranged in one of a plurality of racks. A standard rack may be defined as an Electronics Industry Association (EIA) enclosure, 78 in. (2 meters) wide, 24 in. (0.61 meter) wide and 30 in. (0.76 meter) deep. Standard racks may be configured to house a number of computer systems, e.g., about forty (40) to eighty (80). Each computer system having a system board, power supply, and mass storage. The system boards typically include PCBs having a number of components, e.g., processors, micro-controllers, high-speed video cards, memories, semi-conductor devices, and the like, that dissipate relatively significant amounts of heat during the operation of the respective components. For example, a typical computer system comprising a system board, multiple microprocessors, power supply, and mass storage may dissipate approximately 250 W of power. Thus, a rack containing forty (40) computer systems of this type may dissipate approximately 10 KW of power.

[0004] In order to substantially guarantee proper operation, and to extend the life of the electronic packages arranged in the data center, it is necessary to maintain the temperatures of the packages within predetermined safe operating ranges. Operation at temperatures above maximum operating temperatures may result in irreversible damage to the electronic packages. In addition, it has been established that the reliabilities of electronic packages, such as semiconductor electronic devices, decrease with increasing temperature. Therefore, the heat energy produced by the electronic packages during operation must thus be removed at a rate that ensures that operational and reliability requirements are met. Because of the sheer size of data centers and the high number of electronic packages contained therein, it is often expensive to maintain data centers below predetermined temperatures.

[0005] The power required to remove the heat dissipated by the electronic packages in the racks is generally equal to about 10 percent of the power needed to operate the packages. However, the power required to remove the heat dissipated by a plurality of racks in a data center is generally equal to about 50 percent of the power needed to operate the packages in the racks. The disparity in the amount of power required to dissipate the various heat loads between racks of data centers stems from, for example, the additional thermodynamic work needed in the data center to cool the air. In one respect, racks are typically cooled with fans that operate to move cooling fluid, e.g., air, across the heat dissipating components; whereas, data centers often implement reverse power cycles to cool heated return air. The additional work required to achieve the temperature reduction, in addition to the work associated with moving the cooling fluid in the data center and the condenser, often add up to the 50 percent power requirement. As such, the cooling of data centers presents problems in addition to those faced with the cooling of racks.

[0006] Data centers are typically cooled by operation of one or more air conditioning units. The compressors of the air conditioning units typically require a minimum of about thirty (30) percent of the required cooling capacity to sufficiently cool the data centers. The other components, e.g., condensers, air movers (fans), etc., typically require an additional twenty (20) percent of the required cooling capacity. As an example, a high density data center with 100 racks, each rack having a maximum power dissipation of 10 KW, generally requires 1 MW of cooling capacity. Air conditioning units with a capacity of 1 MW of heat removal generally requires a minimum of 300 KW input compressor power in addition to the power needed to drive the air moving devices, e.g., fans, blowers, etc.

[0007] Conventional data center air conditioning units do not vary their cooling fluid output based on the distributed needs of the data center. Typically, the distribution of work among the operating electronic components in the data center is random and is not controlled. Because of work distribution, some components may be operating at a maximum capacity, while at the same time, other components may be operating at various power levels below a maximum capacity. Conventional cooling systems operating at 100 percent, often attempt to cool electronic packages that may not be operating at a level that may cause its temperature to exceed a predetermined temperature range. Consequently, conventional cooling systems often incur greater amounts of operating expenses than may be necessary to sufficiently cool the heat generating components contained in the racks of data centers.

SUMMARY OF THE INVENTION

[0008] According to an embodiment, the invention pertains to an energy management system for one or more data centers. The system includes a system controller and one or more data centers. According to this embodiment, each data center has a plurality of racks, and a plurality of electronic packages. Each rack contains at least one electronic package and a cooling system. The system controller is interfaced with each cooling system and interfaced with the plurality of the electronic packages, and the system controller is configured to distribute workload among the plurality of electronic packages based upon energy requirements.

[0009] According to another embodiment, the invention relates to an arrangement for optimizing energy use in one or more data centers. The arrangement includes system controlling means, and one or more data facilitating means, with each data facilitating means having a plurality of processing and electronic means. Each data facilitating means also includes cooling means. According to this embodiment, the system controlling means is interfaced with the plurality of processing and electronic means and also with the cooling means. The system controlling means is configured to distribute workload among the plurality of processing and electronic means.

[0010] According to yet another embodiment, the invention pertains to a method of energy management for one or more data centers, with each data center having a cooling system and a plurality of racks. Each rack has at least one electronic package. According to this embodiment, the method includes the steps of determining energy utilization, and determining an optimal workload-to-cooling arrangement. The method further includes the step of implementing the optimal workload-to-cooling arrangement.

BRIEF DESCRIPTION OF THE DRAWINGS

P-0011[0011] The present invention is illustrated by way of example and not limitation in the accompanying figures in which like numeral references refer to like elements, and wherein:

P-0012[0012]FIG. 1A illustrates an exemplary schematic illustration of a data center system in accordance with an embodiment of the invention;

P-0013[0013]FIG. 1B is an illustration of an exemplary cooling system to be used in a data center room in accordance with an embodiment of the invention;

P-0014[0014]FIG. 2 illustrates an exemplary simplified schematic illustration of a global data center system in accordance with an embodiment of the invention; and

P-0015[0015]FIG. 3 is a flowchart illustrating a method according to an embodiment of the invention.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

P-0016[0016] According to an embodiment of the present invention, an energy management system is configured to distribute the workload and to manipulate the cooling in one or more data centers, according to desired energy requirements. This may involve the transference of workload from one server to another or from one heat-generating component to another. The system is also configured to adjust the flow of cooling fluid within the data center. Thus, instead of applying cooling fluid throughout the entire data center, the cooling fluid may solely be applied to the locations of working servers or heat generating components.

P-0017[0017]FIG. 1A illustrates a simplified schematic illustration of a data center energy management system <b>100</b> in accordance with an embodiment of the invention. As illustrated, the energy management system <b>100</b> includes a data center room <b>101</b> with a plurality of computer racks <b>110</b><i>a</i>-<b>110</b><i>p </i>and a plurality of cooling vents <b>120</b><i>a</i>-<b>120</b><i>p </i>associated with the computer racks. Although FIG. 1A illustrates sixteen computer racks <b>110</b><i>a</i>-<b>110</b><i>p </i>and associated cooling vents <b>120</b><i>a</i>-<b>120</b><i>p</i>, the data center room <b>101</b> may contain any number of computer racks and cooling vents, e.g., fifty computer racks and fifty cooling vents <b>120</b><i>a</i>-<b>120</b><i>p</i>. The number of cooling vents <b>120</b><i>a</i>-<b>120</b><i>p </i>may be more or less than the number of computer racks <b>110</b><i>a</i>-<b>110</b><i>p</i>. The data center energy management system <b>100</b> also includes a system controller <b>130</b>. The system controller <b>130</b> controls the overall energy management functions.

P-0018[0018] Each of the plurality of computer racks <b>110</b><i>a</i>-<b>110</b><i>p </i>generally houses an electronic package <b>112</b><i>a</i>-<b>112</b><i>p</i>. Each electronic package <b>112</b><i>a</i>-<b>112</b><i>p </i>may be a component or a combination of components. These components may include processors, micro-controllers, high-speed video cards, memories, semi-conductor devices, or subsystems such as computers, servers and the like. The electronic packages <b>112</b><i>a</i>-<b>112</b><i>p </i>may be implemented to perform various processing and electronic functions, e.g., storing, computing, switching, routing, displaying, and like functions. In the performance of these processing and electronic functions, the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p </i>generally dissipate relatively large amounts of heat. Because the computer racks <b>110</b><i>a</i>-<b>110</b><i>p </i>have been generally known to include upwards of forty (40) or more subsystems, they may require substantially large amounts of cooling to maintain the subsystems and the components generally within a predetermined operating temperature range.

P-0019[0019]FIG. 1B is an exemplary illustration of a cooling system <b>115</b> for cooling the data center <b>101</b>. FIG. 1B illustrates an arrangement for the cooling system <b>115</b> with respect to the data center room <b>101</b>. The data center room <b>101</b> includes a raised floor <b>140</b>, with the vents <b>120</b> in the floor <b>140</b>. FIG. 1B also illustrates a space <b>160</b> beneath the raised floor <b>140</b>. The space <b>160</b> may function as a plenum to deliver cooling fluid to the plurality of racks <b>110</b>. It should be noted that although FIG. 1B is an illustration of the cooling system <b>115</b>, the racks <b>110</b> are represented by dotted lines to illustrate the relationship between the cooling system <b>115</b> and the racks <b>110</b>. The cooling system <b>115</b> includes the cooling vents <b>120</b>, a fan <b>121</b>, a cooling coil <b>122</b>, a compressor <b>123</b>, and a condenser <b>124</b>. As stated above, although the figure illustrates four racks <b>110</b> and four vents <b>120</b>, the number of vents may be more or less than the number of racks <b>110</b>. For instance, in a particular arrangement, there may be one cooling vent <b>120</b> for every two racks <b>110</b>.

P-0020[0020] In the cooling system <b>115</b>, the fan <b>121</b> supplies cooling fluid into the space <b>160</b>. Air is supplied into the fan <b>121</b> from the heated air in the data center room <b>101</b> as indicated by arrows <b>170</b> and <b>180</b>. In operation, the heated air enters into the cooling system <b>115</b> as indicated by arrow <b>180</b> and is cooled by operation of the cooling coil <b>122</b>, the compressor <b>123</b>, and the condenser <b>124</b>, in any reasonably suitable manner generally known to those of ordinary skill in the art. In addition, based upon the cooling fluid required by the heat loads in the racks <b>110</b>, the cooling system <b>115</b> may operate at various levels. The cooling fluid generally flows from the fan <b>121</b> and into the space <b>160</b> (e.g., plenum) as indicated by the arrow <b>190</b>. The cooling fluid flows out of the raised floor <b>140</b> through a plurality of cooling vents <b>120</b> that generally operate to control the velocity and the volume flow rate of the cooling fluid there through. It is to be understood that the above description is but one manner of a variety of different manners in which a cooling system <b>115</b> may be arranged for cooling a data center room <b>101</b>.

P-0021[0021] As outlined above, the system controller <b>130</b>, illustrated in FIG. 1A, controls the operation of the cooling system <b>115</b> and the distribution of work among the plurality of computer racks <b>110</b>. The system controller <b>130</b> may include a memory (not shown) configured to provide storage of a computer software that provides the functionality for distributing the work load among the computer racks <b>110</b> and also for controlling the operation of the cooling arrangement <b>115</b>, including the cooling vents <b>120</b>, the fan <b>121</b>, the cooling coil <b>122</b>, the compressor <b>123</b>, the condenser <b>124</b>, and various other air-conditioning elements. The memory (not shown) may be implemented as volatile memory, non-volatile memory, or any combination thereof, such as dynamic random access memory (DRAM), EPROM, flash memory, and the like. It should be noted that a data room arrangement is further described in co-pending application: “Data Center Cooling System”, Ser. No. 09/139,843, assigned to the same assignee as the present application, the disclosure of which is hereby incorporated by reference in its entirety.

P-0022[0022] The operation of the system controller <b>130</b> is further explained using the illustration of FIG. 1A. In operation, the system controller <b>130</b>, via the associated software, may monitor the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p</i>. This may be accomplished by monitoring the workload as it enters the system and is assigned to a particular electronic package <b>112</b><i>a</i>-<b>112</b><i>p</i>. The system controller <b>130</b> may index the workload of each electronic package <b>112</b><i>a</i>-<b>112</b><i>p</i>. Based on the information pertaining to the workload of each electronic package <b>112</b><i>a</i>-<b>112</b><i>p</i>, the system controller <b>130</b> may determine the energy utilization of each working electronic package. Controller software may include an algorithm that calculates energy utilization as a function of the workload.

P-0023[0023] Temperature sensors (not shown) may also be used to determine the energy utilization of the electronic packages. Temperature sensors may be infrared temperature measurement means, thermocouples, thermisters or the like, positioned at various positions in the computer racks <b>110</b><i>a</i>-<b>110</b><i>p</i>, or in the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p </i>themselves. The temperature sensors (not shown) may also be placed in the aisles, in a non-intrusive manner, to measure the temperature of exhaust air from the racks <b>110</b><i>a</i>-<b>110</b><i>p</i>. Each of the temperature sensors may detect temperature of the associated rack <b>110</b><i>a</i>-<b>110</b><i>p </i>and/or electronic package <b>112</b><i>a</i>-<b>112</b><i>p</i>, and based on this detected temperature, the system controller <b>130</b> may determine the energy utilization.

P-0024[0024] Based on the determination of the energy utilization among the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p</i>, the system controller <b>130</b> may determine an optimal workload-to-cooling arrangement. The “workload-to-cooling” arrangement refers to the arrangement of the workload among the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p</i>, with respect to the arrangement of the cooling system. The arrangement of the cooling system is defined by the number and location of fluid distributing cooling vents <b>120</b><i>a</i>-<b>120</b><i>p</i>, as well as the rate and temperature at which the fluids are distributed. The optimal workload-to-cooling arrangement may be one in which energy utilization is minimized. The optimal workload-to-cooling arrangement may also be one in which energy cost are minimized.

P-0025[0025] Based on the above energy requirements, i.e., minimum energy utilization, or minimum energy cost, the system controller <b>130</b> determines the optimum workload-to-cooling arrangement. The system controller <b>130</b> may include software that performs optimizing calculations. These calculations are based on workload distributions and cooling arrangements.

P-0026[0026] In one embodiment, the optimizing calculations may be based on a constant workload distribution and a variable cooling arrangement. For example, the calculations may involve permutations of possible workload-to-cooling arrangements that have a fixed workload distribution among the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p</i>, but a variable cooling arrangement. Varying the cooling arrangement may involve varying the distribution of cooling fluids among the vents <b>120</b><i>a</i>-<b>120</b><i>p</i>, varying the rate at which the cooling fluids are distributed, and varying the temperature of the cooling fluids.

P-0027[0027] In another embodiment, the optimizing calculations may be based on a variable workload distribution and a constant cooling arrangement. For example, the calculations may involve permutations of possible workload-to-cooling arrangements that vary the workload distribution among the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p</i>, but keep the cooling arrangement constant.

P-0028[0028] In yet another embodiment, the optimizing calculations may be based on a variable workload distribution and a variable cooling arrangement. For example, the calculations may involve permutations of possible workload-to-cooling arrangements that vary the workload distribution among the electronic packages <b>112</b><i>a</i>-<b>112</b><i>p</i>. The calculations may also involve variations in the cooling arrangement, which may include varying the distribution of cooling fluids among the vents <b>120</b><i>a</i>-<b>120</b><i>p</i>, varying the rate at which the cooling fluids are distributed, and varying the temperature of the cooling fluids.

P-0029[0029] Although permutative calculations are outlined as examples of calculations that may be utilized in the determination of optimized energy usage, other methods of calculations may be employed. For example, initial approximations for an optimized workload-to-cooling arrangement may be made, and an iterative procedure for determining an actual optimized workload-to-cooling arrangement may be performed. Also, stored values of energy utilization for known workload-to-cooling arrangements may be tabled or charted in order to interpolate an optimized workload-to-cooling arrangement. Calculations may also be based upon approximated optimized energy values, from which the workload-to-cooling arrangement is determined.

P-0030[0030] The optimal workload-to-cooling arrangement may include grouped workloads. Workload grouping may involve shifting a plurality of dispersed server workloads to a single server, or it may involve shifting different dispersed server workloads to grouped or adjacently located servers. The grouping makes it possible to use a reduced number of the cooling vents <b>120</b><i>a</i>-<b>120</b><i>p </i>for cooling the working servers <b>112</b><i>a</i>-<b>112</b><i>p</i>. Therefore, the amount of energy required to cool the servers may be reduced.

P-0031[0031] The optimizing process is further explained in following examples. In a first example, the data center energy management system <b>100</b> of FIG. 2 contains servers <b>112</b><i>a</i>-<b>112</b><i>p </i>and corresponding cooling vents <b>120</b><i>a</i>-<b>120</b><i>p</i>. Servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m </i>are all working at a maximum capacity of 10 KW. In this example, the maximum working capacity of each of the plurality of servers <b>112</b><i>a</i>-<b>112</b><i>p </i>is 10 KW. In addition, each cooling vent <b>120</b><i>a</i>-<b>120</b><i>p </i>of the cooling system <b>115</b> is blowing cooling fluids at a temperature of 55° F. and at a low throttle. The system controller <b>130</b> determines the energy utilization of each working server, <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m</i>. An algorithm associated with the controller <b>130</b> may estimate the energy utilization of the servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m </i>by monitoring the workloads of the servers <b>112</b><i>a</i>-<b>112</b><i>p</i>, and performing calculations that estimate the energy utilization as a function of the workload.

P-0032[0032] The heat energy dissipated by <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m </i>may also be determined from measurements by sensing means (not shown) located in the servers <b>112</b><i>a</i>-<b>112</b><i>p</i>. Alternatively, the system controller <b>130</b> may use a combination of the sensing means (not shown) and calculations based on the workload, to determine the energy utilization of the electronic packages <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m. </i>

P-0033[0033] After determining the energy utilization of the servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m</i>, the system controller <b>130</b> may determine an optimal workload-to-cooling arrangement. The optimal workload-to-cooling arrangement may be one in which energy utilization is minimized, or one in which energy cost are minimized. In this example, the energy utilization is to be minimized, therefore the system controller <b>130</b> performs calculations to determine the most energy efficient workload-to-cooling arrangement.

P-0034[0034] As outlined above, the optimizing calculations may be performed using different permutations of sample workload-to-cooling arrangements. The optimizing calculations may be based on permutations that have a varying cooling arrangement whilst maintaining a constant workload distribution. The optimizing calculations may alternatively be based on permutations that have a varying workload distribution and a constant cooling arrangement. The calculations may also be based on permutations having varying workload distributions and varying cooling arrangements.

P-0035[0035] In this example, the optimizing calculations use permutations of sample workload-to-cooling arrangements in which both the workload distribution and the cooling arrangements vary. The system controller <b>130</b> includes software that performs optimizing calculations. As stated above, the optimal arrangement may involve the grouping of workloads. These calculations therefore use permutations in which the workload is shifted around from dispersed servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m </i>to servers that are adjacently located or grouped. The permutations also involve different sample cooling arrangements, i.e., arrangements in which some of the cooling vents <b>120</b><i>a</i>-<b>120</b><i>p </i>are closed, or in which the cooling fluids are blown in reduced or increased amounts. The cooling fluids may also be distributed at increased or reduced temperatures.

P-0036[0036] After performing energy calculations of the different sample workload-to-cooling arrangements, the most energy efficient arrangement is selected as the optimal arrangement. For instance, in accessing the different permutations, the two most energy efficient workload-to-cooling arrangements may include the following groups of servers: A first group of servers <b>112</b><i>f</i>, <b>112</b><i>g</i>, <b>112</b><i>j</i>, and <b>112</b><i>k</i>, located substantially in the center of the data center room <b>101</b>, and a second group of servers <b>112</b><i>a</i>, <b>112</b><i>b</i>, <b>112</b><i>e</i>, and <b>112</b><i>f</i>, located at a corner of the data center room <b>101</b>. Assuming that these two groups of servers utilize a substantially equal amount of energy, then the more energy efficient of the two workload-to-cooling arrangements is dependent upon which cooling arrangement for cooling the servers, is more energy efficient.

P-0037[0037] The energy utilization associated with the use of the different vents may be different. For instance, some vents may be located in an area in the data center room <b>101</b> where they are able to provide better circulation throughout the entire data center room <b>101</b>, than vents located elsewhere. As a result, some vents may be able to more efficiently maintain, not only operating electronic packages, but also the inactive electronic packages <b>112</b><i>a</i>-<b>112</b><i>p</i>, at predetermined temperatures. Also, the cooling system <b>115</b> may be designed in such a manner that particular vents involve the operation of fans that utilize more energy than fans associated with other vents. Differences in energy utilization associated with vents may also occur due to mechanical problems such as clogging etc.

P-0038[0038] Returning to the example, it may be more efficient to cool the center of the room <b>101</b> because the circulation at this location is generally better than other areas in the room. Therefore, the first group, <b>112</b><i>f</i>, <b>112</b><i>g</i>, <b>112</b><i>j</i>, and <b>112</b><i>k </i>would be used. Furthermore, the centrally located cooling vents <b>120</b><i>f</i>, <b>120</b><i>g</i>, <b>120</b><i>j</i>, and <b>120</b><i>k </i>are the most efficient circulators, so these vents should be used in combination with the first group of servers, <b>112</b><i>f</i>, <b>112</b><i>g</i>, <b>112</b><i>j</i>, and <b>112</b><i>k</i>, to optimize energy efficiency. Other vents that are not as centrally located may have a tendency to produce eddies and other undesired circulatory effects. In this example, the optimized workload-to-cooling arrangement involves the use of servers <b>112</b><i>f</i>, <b>112</b><i>g</i>, <b>112</b><i>j</i>, and <b>112</b><i>k </i>in combination with cooling vents <b>120</b><i>f</i>, <b>120</b><i>g</i>, <b>120</b><i>j</i>, and <b>120</b><i>k</i>. It should be noted that although the outlined example illustrates a one-to-one ratio of cooling vents to racks, it is possible to have a smaller or larger number of cooling vents as compared to racks. Also, the temperature and the rate at which the cooling fluids are distributed may be altered.

P-0039[0039] In a second example, the system <b>100</b> of FIG. 2 contains servers <b>112</b><i>a</i>-<b>112</b><i>p </i>and corresponding cooling vents <b>120</b><i>a</i>-<b>120</b><i>p</i>. Servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, and <b>112</b><i>h </i>are all working at a capacity of 3 KW. Server <b>112</b><i>m </i>is operating at a maximum capacity of 10 KW. The maximum working capacity of each of the plurality of servers <b>112</b><i>a</i>-<b>112</b><i>p </i>is 10 KW. In addition, the cooling arrangement <b>115</b> is performing with each of the cooling vents <b>120</b><i>a</i>-<b>120</b><i>p </i>blowing cooling fluids at a low throttle at a temperature of 55° F. In a manner as described in the first example, the system controller <b>130</b> determines the energy utilization of each working server, <b>112</b><i>a</i>, <b>112</b><i>e</i>, <b>112</b><i>h</i>, and <b>112</b><i>m</i>, i.e., by means of, calculations that determine energy utilization as a function of workload, sensing means, or a combination thereof.

P-0040[0040] After determining the energy utilization, the system controller <b>130</b> may optimize the operation of the system <b>100</b>. According to this example, the system may be optimized according to a minimum energy requirement. As in the first example, the system controller <b>130</b> performs optimizing energy calculations for different permutations of workload-to-cooling arrangements. In this example, calculations may involve permutations that vary the workload distribution and the cooling arrangement.

P-0041[0041] As stated above, the calculations of sample workload-to-cooling arrangements may involve grouped workloads in order to minimize energy requirements. The system controller <b>130</b> may perform calculations in which, the workload is shifted around from dispersed servers to servers that are adjacently located or grouped. Because the servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, and <b>112</b><i>h</i>, are operating at 3 KW, and each of the servers <b>112</b> have a maximum operating capacity of 10 KW, it is possible to combine these workloads to a single server. Therefore, the calculations may be based on permutations that combine the workloads of servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, and <b>112</b><i>h</i>, as well as shift the workload of server <b>112</b><i>m </i>to another server.

P-0042[0042] After performing energy calculations of the different sample workload-to-cooling arrangements, the most energy efficient arrangement is selected as the optimal arrangement. In this example, the workload-to-cooling arrangement may be one in which the original workload is shifted to servers <b>112</b><i>f </i>and <b>112</b><i>g </i>with server <b>112</b><i>f </i>operating at 9 KW and <b>112</b><i>g </i>operating at 10 KW. The optimizing calculations may show that the operation of these servers <b>112</b><i>f </i>and <b>112</b><i>g</i>, in combination with the use of cooling vents <b>120</b><i>f </i>and <b>120</b><i>g</i>, may utilize the minimum energy. Again, as outlined above, although the outlined example illustrates a one-to-one ratio of cooling vents to racks, it is possible to have a smaller or larger number of cooling vents as compared to racks.

P-0043[0043] As stated above, the permutative calculations outlined in the above examples, is but one manner of determining optimized arrangements. Other methods of calculations may be employed. For example, initial approximations for an optimized workload-to-cooling arrangement may be made, and an iterative procedure for determining the actual optimized workload-to-cooling arrangements may be determined. Also, stored values of energy utilization for known workload-to-cooling arrangements may be tabled or charted in order to interpolate an optimized workload-to-cooling arrangement. Calculations may also be based upon approximated optimized energy values, from which the workload-to-cooling arrangement is determined.

P-0044[0044] It should be noted that the grouping of the workloads might be performed in a manner to minimize the switching of workloads from one server to another. For instance, in the second example, the system controller <b>130</b> may allow the server <b>112</b><i>m </i>to continue operating at 10 KW. The workload from the other servers <b>112</b><i>a</i>, <b>112</b><i>e</i>, and <b>112</b><i>h </i>may be switched to the server <b>112</b><i>n</i>, so that cooling may be provided primarily by the vents <b>120</b><i>m </i>and <b>120</b><i>n</i>. By not switching the workload from server <b>112</b><i>m</i>, the server <b>112</b><i>m </i>is allowed to perform its functions without substantial interruption.

P-0045[0045] Although the examples illuminate situations in which workloads are grouped in order to ascertain an optimal workload-to-cooling arrangement, optimal arrangements may be obtained by separating workloads. For instance, server <b>112</b><i>d </i>may be operating at a maximum capacity of 20 KW, with associated cooling vent <b>120</b><i>d </i>operating at full throttle to maintain the server at a predetermined safe temperature. The use of the cooling vent <b>120</b><i>d </i>at full throttle may be inefficient. In this situation, the system controller <b>130</b> may determine that it is more energy efficient to separate the workloads so that servers <b>112</b><i>c</i>, <b>112</b><i>d</i>, <b>112</b><i>g</i>, and <b>112</b><i>h </i>all operate at 5 KW because it is easier to cool the servers with divided workloads. In this example, vents <b>120</b><i>c</i>, <b>120</b><i>d</i>, <b>120</b><i>g</i>, and <b>120</b><i>h </i>may be used to provide the cooling fluids more efficiently in terms of energy utilization.

P-0046[0046] It should also be noted that the distribution of workloads and cooling may be performed on a cost-based analysis. According to a cost-based criterion, the system controller <b>130</b> utilizes an optimizing algorithm that minimizes energy cost. Therefore in the above example in which the server <b>112</b><i>d </i>is operating at 20 KW, the system controller <b>130</b> may distribute the workload among other servers, and/or distribute the cooling fluids among the cooling vents <b>120</b><i>a</i>-<b>120</b><i>p</i>, in order to minimize the cost of the energy. The controller <b>130</b> may also manipulate other elements of the cooling system <b>115</b> to minimize the energy cost, e.g., the fan-speed may be reduced.

P-0047[0047]FIG. 2 illustrates an exemplary simplified schematic illustration of a global data center system. FIG. 2 shows an energy management system <b>300</b> that includes data centers <b>101</b>, <b>201</b>, and <b>301</b>. The data centers <b>101</b>, <b>201</b>, and <b>301</b> may be in different geographic locations. For instance, data center <b>101</b> may be in New York, data center <b>201</b> may be in California, and data center <b>301</b> may be in Asia. Electronic packages <b>112</b>, <b>212</b>, and <b>312</b> and corresponding cooling vents <b>120</b>, <b>220</b>, and <b>320</b> are also illustrated. Also illustrated is a system controller <b>330</b>, for controlling the operation of the data centers <b>101</b>, <b>201</b>, and <b>301</b>. It should be noted that each of the data centers <b>101</b>, <b>201</b>, and <b>301</b> may each include a respective system controller without departing from the scope of the invention. In this instance, each system controller may be in communication with each other, e.g., networked through a portal such as the Internet. For simplicity sake, this embodiment of the invention will be described with a single system controller <b>330</b>.

P-0048[0048] The system controller <b>330</b> operates in a similar manner to the system controller <b>130</b> outlined above. According to one embodiment, the system controller <b>330</b> operates to optimize energy utilization. This may be accomplished by minimizing the energy cost, or by minimizing energy utilization. In operation, the system controller <b>330</b> may monitor the workload and determine the energy utilization of the electronic packages <b>112</b>, <b>212</b>, and <b>312</b>. The energy utilization may be determined by calculations equating the energy utilization as a function of the workload. The energy utilization may also be determined by temperature sensors (not shown) located in and/or in the vicinity of the electronic packages <b>112</b>, <b>212</b>, and <b>312</b>.

P-0049[0049] Based on the determination of the energy utilization of servers <b>112</b>, <b>212</b>, and <b>312</b>, the system controller <b>330</b> optimizes the system <b>300</b> according to energy requirements. The optimizing may be to minimize energy utilization or to minimize energy cost. When optimizing according to a minimum energy cost requirement, the system controller <b>330</b> may distribute the workload and/or cooling according to energy prices.

P-0050[0050] For example, if the only active servers are in the data center <b>201</b>, which for example is located in California, the system controller <b>330</b> may switch the workload to the data center <b>301</b> or data center <b>101</b> in other geographic locations, if the energy prices at either of these locations are cheaper than at data center <b>201</b>. For instance, if the data center <b>301</b> is in Asia where energy is in less demand and cheaper because it is nighttime, the workload may be routed to the data center <b>301</b>. Alternatively, the climate where a data center is located may have an impact on energy efficiency and energy prices. If the data center <b>101</b> is in New York, and it is winter in New York, the system controller <b>330</b> may switch the workload to the data center <b>101</b>. This switch may be made because cooling components such as the condenser (element <b>124</b> in FIG. 2B) are more cost efficient at lower temperatures, e.g. 50° F. in New York winter.

P-0051[0051] The system controller <b>330</b> may also be operated in a manner to minimize energy utilization. The operation of the system controller <b>330</b> may be in accordance with a minimum energy requirement as outlined above. However, the system controller <b>330</b> has the ability to shift workloads (and/or cooling operation) from electronic packages in one data center to electronic packages in data centers at another geographic location. For example, if the only active servers are in the data center <b>201</b>, which for example is located in California, the system controller <b>330</b> may switch the workload to the data center <b>301</b> or data center <b>101</b> in other geographic locations, if the energy utilization at either of these locations is more efficient than at data center <b>201</b>. If the data center <b>101</b> is in New York, and it is winter in New York, the system controller <b>330</b> may switch the workload to the data center <b>101</b>, because cooling components such as the condenser (element <b>124</b> in FIG. 2B) utilize less energy at lower temperatures, e.g. 50° F. in New York winter.

P-0052[0052]FIG. 3 is a flowchart illustrating a method <b>400</b> according to an embodiment of the invention. The method <b>400</b> may be implemented in a system such as system <b>100</b> illustrated in FIG. 1A or system <b>300</b> illustrated in FIG. 3. Each data center has a cooling arrangement with cooling vents and racks, and electronic packages in the data center racks. It is to be understood that the steps illustrated in the method <b>400</b> may be contained as a routine or subroutine in any desired computer accessible medium. Such medium including the memory, internal and external computer memory units, and other types of computer accessible media, such as a compact disc readable by a storage device. Thus, although particular reference is made to the controller <b>130</b> as performing certain functions, it is to be understood that any electronic device capable of executing the above-described function may perform those functions.

P-0053[0053] At step <b>410</b>, energy utilization is determined. In making this determination, the electronic packages <b>112</b> are monitored. The step of monitoring the electronic packages <b>112</b> may involve the use of software including an algorithm that calculates energy utilization as a function of the workload. The monitoring may also involve the use of sensing means attached to, or in the general vicinity of the electronic packages <b>112</b>.

P-0054[0054] At step <b>420</b>, an optimal workload-to-cooling arrangement is determined. The optimal arrangement may be one in which energy utilization is minimized. The optimal arrangement may also be one in which energy cost are minimized. This may be determined with optimizing energy calculations involving different workload-to-cooling arrangements. In performing the calculations, the workload distribution and/or the cooling arrangement may be varied.

P-0055[0055] At step <b>430</b>, the optimal workload-to-cooling arrangement is implemented. Therefore, the workload may be distributed among the electronic packages <b>112</b> and the cooling arrangements may be changed for example, by opening and closing vents. The temperature of the cooling may also be adjusted, and the speed of circulating fluids may be changed. After performing step <b>430</b>, the system may go into an idle state.

P-0056[0056] It should be noted that, the data, routines and/or executable instructions stored in software for enabling certain embodiments of the present invention may also be implemented in firmware or designed into hardware components.

P-0057[0057] What has been described and illustrated herein is a preferred embodiment of the invention along with some of its variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention, which is intended to be defined by the following claims—and their equivalents—in which all terms are meant in their broadest reasonable sense unless otherwise indicated.