Dynamically modifying the resources of a virtual server
Claim Score by NHIP
Abstract
A system and a method dynamically adjusts the quality of service guarantees for virtual servers based upon the resource demands experienced by the virtual servers. Virtual server resource denials are monitored to determine if a virtual server is overloaded based upon the resource denials. Virtual server resources are modified dynamically to respond to the changing resource requirements of each virtual server. Occasionally, a physical host housing a virtual server may not have additional resources to allocate to a virtual server requiring increased resources. In this instance, a virtual server hosted by the overloaded physical host is transferred to another physical host with sufficient resources.
Term
Term ended
Expired 11 May 2020, 6.4 years ago.
- Priority
- Filed
- Granted
- Expired
- Today
7 claims: 5 independent, 2 dependent
- 1A network system for dynamically modifying the computer resources allocated to a virtual server, the network system comprising a plurality of physical hosts, the virtual server operating in a first physical host, the computer resources allocated to the virtual server being specified as a quality of service guarantee, the network system comprising:a virtual server resource monitor communicatively coupled to the first physical host and configured to monitor resource denials and to send a virtual server overloaded signal in response to the resource denials;a virtual server resource modifier communicatively coupled to the first physical host and configured to receive the virtual server overloaded signal and, in response to the virtual server overloaded signal, to modify a resource allocation for the virtual server and to send a virtual server resource modification signal;a load balancing module communicatively coupled to the plurality of physical hosts and configured to receive the virtual server resource modification signal and to determine whether the first physical host is overloaded and, in response to a determination that the first physical host is overloaded, to send a physical host transfer signal that indicates a second physical host;and a dynamic virtual server mover communicatively coupled to the plurality of physical hosts and configured to receive the physical host transfer signal and, in response to the physical host transfer signal, to transfer the virtual server from the first physical host to the second physical host.
- 4A computer program product to be executed in a computer for dynamically modifying the computer resources allocated to a virtual server operating in a first physical host in a network system, the network system comprising a plurality of physical hosts, the computer resources allocated to the virtual server being specified as a quality of service guarantee, the computer program product comprising:program code for creating a virtual server resource monitor communicatively coupled to the first physical host and configured to monitor resource denials and, in response to the resource denials, to send a virtual server overloaded signal;program code for creating a virtual server resource modifier communicatively coupled to the first physical host and configured to receive the virtual server overloaded signal and, in response to the virtual server overloaded signal, to modify a resource allocation for the virtual server and to send a virtual server resource modification signal;program code for creating a load balancing module communicatively coupled to the plurality of physical hosts and configured to receive the virtual server resource modification signal and to determine whether the first physical host is overloaded and, in response to a determination that the first physical host is overloaded, to send a physical host transfer signal that indicates a second physical host;and program code for creating a dynamic virtual server mover communicatively coupled to the plurality of physical hosts and configured to receive the physical host transfer signal and, in response to the physical host transfer signal, to transfer the virtual server from the first physical host to the second physical host.
- 5Broadest claimClaim Score 56, average(NHIP)A method performed by a computing device, having a processor and memory, for modifying the computer resources allocated to a virtual server operating in a first physical host of multiple physical hosts, comprising:receiving an indication that a first physical host is overloaded, wherein the indication is based on a determination that a virtual server is overloaded and wherein the determination that a virtual server is overloaded is based on one or more resource unavailable messages resulting from denied requests to modify a resource allocation;determining that a second physical host can accommodate the requested modified resource allocation;and generating a physical host transfer signal that indicates the second physical host and transferring the virtual server from the first physical host to the second physical host.
- 6A computer-readable storage device storing instructions that, when executed by a computing device, cause the computing device to perform operations configured to modify computer resources allocated to a virtual server operating in a first physical host of multiple physical hosts, the operations comprising:receiving an indication that a first physical host is overloaded, wherein the indication is based on a determination that a virtual server is overloaded and wherein the determination that a virtual server is overloaded is based on one or more resource unavailable messages resulting from denied requests to modify a resource allocation;determining that a second physical host can accommodate the requested modified resource allocation;and generating a physical host transfer signal that indicates the second physical host and transferring the virtual server from the first physical host to the second physical host if the first physical host is overloaded.
- 7A system for modifying the computer resources allocated to a virtual server operating in a first physical host of multiple physical hosts, the system comprising:one or more processors and one or more memories;a component configured to receive an indication that a first physical host is overloaded, wherein the indication is based on a determination that a virtual server is overloaded and wherein the determination that a virtual server is overloaded is based on one or more resource unavailable messages resulting from denied requests to modify a resource allocation;a component configured to determine that a second physical host can accommodate the requested modified resource allocation;and a component configured to generate a physical host transfer signal that indicates a second physical host and to transfer the virtual server from the first physical host to the second physical host if the first physical host is overloaded.
Independent claims5
83 paragraphs in 5 sections, as filed
CROSS REFERENCE TO RELATED APPLICATIONS
0001This patent application is a continuation of U.S. patent application Ser. No. 11/971,778, filed Jan. 9, 2008, entitled “DYNAMICALLY MODIFYING THE RESOURCES OF A VIRTUAL SERVER”, now U.S. Pat. No. Re. 42,726 issued Sep. 20, 2011, which is a reissue of U.S. patent application Ser. No. 09/569,371 filed May 11, 2000, entitled “DYNAMICALLY MODIFYING THE RESOURCES OF A VIRTUAL SERVER”, now U.S. Pat. No. 6,985,937 issued on Jan. 10, 2006, all of which are incorporated by reference herein in their entireties.
0002This application is related to U.S. patent Ser. No. 09/499,098, entitled “Selective Interception of System Calls,” by Borislav D. Deianov et al., filed Feb. 4, 2000, now U.S. Pat. No. 6,546,546 and commonly assigned with the present application. The subject matter of this related application is incorporated by reference herein in its entirety.
BACKGROUND
00031. Field of Invention
0004The present invention relates generally to resource allocation for a virtual server, and more particularly, to monitoring and dynamically modifying the resource allocation for a virtual server based upon usage.
00052. Background of the Invention
0006Networked computer resources are growing more popular as the benefits of sharing computing resources become evident. One of the fastest-growing segments of the Internet is the network market. Network systems contain common elements, generally including a dedicated local server to maintain the shared network data, and a communications system for providing data communication services between devices on the network. Data communications services and servers are not easy to configure, manage, and maintain. Thus, there is an incentive for Internet Service Providers (ISPs) to provide such network services and servers, thereby relieving corporations of the burden of providing these services directly.
0007It is not economically feasible for an ISP to remotely manage servers located on a customer's premises, and support many different customers in this fashion. Rather, an ISP would prefer to offer network services to multiple customers while keeping all of the server host computers within a central location of the ISP for ease of management. Accordingly, ISPs typically dedicate one or more physical host computers as each individual customer's server(s), and maintain each host computer in the centralized facility. This means the ISP will have to own and maintain potentially large numbers of physical host computers, at least one for each customer's server or private network. However, most customers will neither require nor be amenable to paying for the user of an entire host computer. Generally, only a fraction of the processing power, storage, and other resources of a host computer will be required to meet the needs of an individual customer.
0008Different customers have different virtual server needs. For example, a company A providing large quantities of data and information to its employees and customers will want to ensure that its virtual servers are always available to perform a large number of tasks. Company A may be willing to pay a premium for a guaranteed high quality of service, with high server availability and large amounts of processing power always on-call. By contrast, a small individual B who merely uses his virtual server for back-up file storage space has very different quality of service requirements. Customer B needs (and wishes to pay for) only a limited amount of storage space to be available on an intermittent basis.
0009When servicing the needs of multiple customers having different needs, it is desirable to provide a virtual server that is dynamic, not static, in its allocation of resources. A customer's virtual server is typically assigned a fixed level of resources, corresponding to either a fixed percentage of the capacity of a particular physical host (for example, the operating system may be instructed to allocate twenty percent of the central processing unit cycles to process A and two percent to process B) or a fixed number of units (for example, the operating system may be instructed to allocate X cycles per second to process A and Y cycles per second to process B). However, customers may be unable to anticipate the exact amount of resources they will require, and a static assignment of a particular resource allocation limit may not allow the virtual server system to adapt to changing customer needs.
0010Instead of requiring customers to select a static level of resources, a better resource allocation model is structured along the lines of electricity pricing—a customer receives what he needs, and he pays for what he receives. Referring back to a previous example, small customer B may initially request a very low level of resources. However, should his new home business suddenly expand, he may quickly bump up against the limit of the server resources he originally requested. In this case, it would be preferable if customer B's virtual server resources were able to automatically, dynamically adjust to his increased resource needs.
0011Thus it is desirable to provide a system and method for a virtual server capable of providing quality of service guarantees for a customer, which is also capable of adjusting the quality of service based upon changing customer demand. It is desirable for such a system to dynamically adjust the physical host resources allocated to a virtual server.
SUMMARY OF THE INVENTION
0012The present invention dynamically adjusts the quality of service guarantees for virtual servers based upon the resource demands experienced by the virtual servers. Virtual servers having individual quality of service guarantees are distributed among a group of physical hosts. Each physical host's resources are allocated among the physical host's resident virtual servers. The resources allocated to a particular virtual server may be dynamically adjusted in response to changing virtual server resource needs.
0013Occasionally, a physical host executing a virtual server may not have additional resources to allocate to a virtual server requiring increased resources. In this instance, a virtual server hosted by the overloaded physical host is transferred to another physical host with sufficient resources.
0014In one embodiment, a dynamic resource configuration module monitors resource denials received by virtual servers and determines if a virtual server is overloaded based upon the resource denials. A resource denial may refer to any request by the virtual server that cannot be immediately serviced, such as a denial of a request to create a file or a network packet delay. If the resource denials received by a particular virtual server exceed a pre-specified limit, the virtual server is considered overloaded and a request is made for additional resources.
0015The resource usage of the physical hosts within the system is monitored. A load-balancing function is performed to select the appropriate physical host when a virtual server transfer becomes necessary. A virtual server is transferred between physical hosts with minimal impact upon the operation of the virtual server.
0016The features and advantages described in the specification are not all-inclusive, and particularly, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims hereof. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter.
BRIEF DESCRIPTION OF THE DRAWINGS
0017<figref idref="DRAWINGS">FIG. 1</figref> is an illustration of a system for dynamically modifying the resources of a virtual server.
0018<figref idref="DRAWINGS">FIG. 2A</figref> is a flowchart of a process for dynamically modifying the resources of a virtual server.
0019<figref idref="DRAWINGS">FIG. 2B</figref> is a flowchart of another process for dynamically modifying the resources of a virtual server.
0020<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram of a process for determining whether an individual resource in a virtual server has reached its limit.
0021<figref idref="DRAWINGS">FIG. 4</figref> is a flowchart of a process for determining when to increase or decrease a virtual server resource allocation.
0022<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of one process for performing resource load balancing among physical hosts.
0023<figref idref="DRAWINGS">FIG. 6</figref> is a flowchart of one process for transferring a virtual server from one physical host to another physical host.
0024The figures depict a preferred embodiment of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
0025Reference will now be made in detail to several embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever practicable, the same reference numbers will be used throughout the drawings to refer to the same or like parts. The term “virtual server” as used herein refers to a virtual server capable of receiving a quality of service guarantee from a physical host. Multiple virtual servers may reside in a single physical host, and different virtual servers on the same physical host may receive different quality of service guarantee.
0026<figref idref="DRAWINGS">FIG. 1</figref> shows an embodiment of a system for dynamic resource configuration in virtual servers. A dynamic resource configuration module <b>100</b> is coupled via a network to a group of physical host machines <b>160</b> (<b>160</b>A, <b>160</b>B and <b>160</b>C), or may be resident on any of these hosts <b>160</b>. The physical host machines <b>160</b> may be any kind of computer adapted to support virtual servers. The module <b>100</b> may be implemented in a software driver. It is to be understood that the dynamic resource configuration module <b>100</b> will typically support more than one physical host machine <b>160</b>. However, in one embodiment, the dynamic resource configuration module <b>100</b> may support a single physical host <b>160</b>.
0027The group of physical hosts <b>160</b> contains a group of virtual servers <b>162</b>. Physical host <b>160</b>A contains virtual servers <b>162</b>A and <b>162</b>B; physical host <b>160</b>B contains virtual servers <b>162</b>C, <b>162</b>D and <b>162</b>E; and physical host <b>160</b>C contains virtual servers <b>162</b>F and <b>162</b>G.
0028In one embodiment, each individual virtual server <b>162</b> has a different quality of service guarantee. Different quality of service guarantees are implemented by allocating different amounts of the resources of each physical host machine <b>160</b> to servicing each of the virtual servers <b>162</b>. Physical host <b>160</b> resources may be allocated as percentages of the resources of a particular physical host <b>160</b>, or as a particular number of units within a physical host <b>160</b> (for example, the operating system may be instructed to allocate X cycles per second to process A and Y cycles per second to process B). In the embodiment shown in <figref idref="DRAWINGS">FIG. 1</figref>, physical host <b>160</b> resources are allocated to individual virtual servers <b>162</b> as percentages of each physical host <b>160</b>. Table 1 lists the resource allocations of each virtual server <b>162</b> as shown in <figref idref="DRAWINGS">FIG. 1</figref>:
0029<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="center" /><thead><row><entry namest="1" nameend="1" rowsep="1">TABLE 1</entry></row></thead><tbody valign="top"><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row><row><entry>Virtual Server Resource Allocation in FIG. 1</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="offset" colwidth="28pt" align="left" /><colspec colname="1" colwidth="77pt" align="left" /><colspec colname="2" colwidth="112pt" align="left" /><tbody valign="top"><row><entry /><entry>Virtual Server</entry><entry>Resource Allocation</entry></row><row><entry /><entry namest="offset" nameend="2" align="center" rowsep="1" /></row><row><entry /><entry>162A</entry><entry>15% of physical host 160A</entry></row><row><entry /><entry>162B</entry><entry>60% of physical host 160A</entry></row><row><entry /><entry>162C</entry><entry>10% of physical host 160B</entry></row><row><entry /><entry>162D</entry><entry>10% of physical host 160B</entry></row><row><entry /><entry>162E</entry><entry>10% of physical host 160B</entry></row><row><entry /><entry>162F</entry><entry>20% of physical host 160C</entry></row><row><entry /><entry>162G</entry><entry>30% of physical host 160C</entry></row><row><entry /><entry namest="offset" nameend="2" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>
0030The virtual servers <b>162</b> each may consume a different amount of the resources of the physical host machines <b>160</b>. The resources of a physical host machine comprise the set of functions and features the physical host machine uses in implementing tasks for each virtual server. Examples of resources include disk space, memory, network capacity and processing cycles (CPU resources). As shown in <figref idref="DRAWINGS">FIG. 1</figref>, virtual server <b>162</b>A consumes 15% of the physical host <b>160</b>A resources. This means that 15% of physical host <b>160</b>A's disk space, memory, network bandwidth, and CPU processing will be dedicated to servicing the needs of virtual server <b>162</b>A. A variety of other types of physical host resources will be evident to one of skill in the art.
0031A resource allocation for a virtual server is specified as a “quality of service guarantee” for that particular server. Each physical host stores quality of service guarantees for the virtual servers it hosts. As a physical host performs processes associated with a particular virtual server, the physical host accesses the stored quality of service information to enable the physical host to request the correct quality of service from the operating system kernel of the physical host.
0032One implementation for storing quality of service guarantee information is a quality of service parameter table. A quality of service parameter table in each physical host <b>160</b> associates each virtual server <b>162</b> resident in the particular physical host <b>160</b> with quality of service parameters. These parameters are used to allocate physical host <b>160</b> resources for each resident virtual server <b>162</b>. For example, physical host <b>160</b>A includes a quality of service parameter table, which lists resident virtual servers <b>162</b>A and <b>162</b>B. The parameter table lists whatever virtual servers are resident in the physical host. As virtual server resource allocations are changed, and as virtual servers are transferred between physical hosts, the corresponding quality of service parameter tables are updated to reflect these changes and transfers. In another embodiment, a single master quality of service parameter table can coordinate multiple slave tables associated with each physical host.
0033Dynamic resource configuration module <b>100</b> includes a virtual server resource monitor <b>110</b>, a virtual server resource modifier <b>120</b>, a physical host load balancer <b>130</b>, a dynamic virtual server mover <b>140</b>, and a file system <b>150</b>. In one embodiment, these modules are portions of the software code implementing the dynamic resource configuration module <b>100</b>. The dynamic resource configuration module <b>100</b> is further communicatively coupled to each physical host <b>160</b>.
0034The virtual server resource monitor <b>110</b> monitors the resource usage of the virtual servers <b>162</b> to determine if they are overloaded. The virtual server resource modifier <b>120</b> dynamically modifies the resource allocations of the virtual servers <b>162</b> on an as-needed basis. The physical host load-balancer <b>130</b> periodically monitors the resource usage of the physical hosts <b>160</b>, and uses the dynamic virtual server mover <b>140</b> to transfer virtual servers <b>162</b> between physical hosts <b>160</b> as needed to balance the loads of the physical hosts <b>160</b>. The file system <b>150</b> is used for storing state information associated with a particular virtual server <b>162</b> when transferring the particular virtual server <b>162</b> to a different physical host <b>160</b>. In another embodiment, the file system <b>150</b> is not used, and state information is copied directly from one physical host to another physical host to transfer a virtual server.
0035<figref idref="DRAWINGS">FIG. 2A</figref> is a flowchart of an embodiment of the overall process for dynamically modifying the resources of a virtual server. Virtual server resource denials are monitored <b>210</b>. Resource monitoring is performed using the selective interception of system calls. One embodiment of selectively intercepting system calls is disclosed in the related application, the subject matter of which is incorporated herein by reference. Each resource (e.g., disk space, memory, network bandwidth, or CPU cycles) used by a virtual server is monitored to determine the time at which the resource is fully used, that is, the point at which a request for more resources is either implicitly or explicitly denied. Examples of resource denials include a memory allocation request denial and a network packet delay signal.
0036A determination is made <b>220</b> as to whether a particular virtual server resource is overloaded. The number of times a particular resource denial is received in a time window is averaged using one of a number of well-known techniques. If the average number of denials is beyond a pre-configured threshold, the virtual server is determined <b>220</b> to be overloaded for the corresponding resource. If the virtual server is not determined to be overloaded, the method continues to monitor <b>210</b> virtual server resource denials.
0037If the virtual server is determined to be overloaded, a determination is made <b>230</b> as to whether the corresponding resource of the physical host hosting the virtual server resource is also overloaded. For example, referring to <figref idref="DRAWINGS">FIG. 1</figref>, if it was determined that a resource for virtual server <b>162</b>B was overloaded, module <b>100</b> would then check to see if that same resource was overloaded for physical host <b>160</b>A which contains virtual server <b>162</b>B. A physical host <b>160</b> resource is determined to be overloaded if the physical host <b>160</b> does not have enough of the particular resource unallocated to the resident individual virtual servers <b>162</b> to service the resource increase request. The physical host resource is overloaded if: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0038">Resource request>Resource available; where Resource available≧0</li></ul>
0039For example, assume virtual server <b>162</b>B requests an additional memory allocation of 1 megabyte. If physical host <b>160</b>A has only 100 kilobytes of memory available (the rest already having been allocated to virtual servers <b>162</b>A and <b>162</b>B), then physical host <b>160</b>A cannot service virtual server <b>162</b>B's request and physical host <b>160</b>A is considered overloaded. This same principle may be extended to other types of resources.
0040If the particular physical host resource is not determined to be overloaded the virtual server resource allocation within the physical host is increased <b>240</b>. The method then continues to monitor <b>210</b> virtual server resource denials.
0041However, if the physical host is determined to be overloaded, a new physical host is selected <b>250</b> to accommodate the overloaded virtual server and its required resource increases. A variety of different fitting heuristic methods may be used to select a new physical host to execute the virtual server. For example, a first fit method may be used, wherein the first physical host <b>160</b> determined to have enough extra resources to accommodate the overloaded virtual server <b>162</b> is selected. In a best fit method, the physical host <b>160</b> with available resources most closely matching the resource needs of the overloaded virtual server <b>162</b> is selected. In an easiest fit method, the physical host <b>160</b> with the most available resources is selected to accommodate the overloaded virtual server <b>162</b>. For the following discussion, assume that physical host <b>160</b>A is overloaded, and new physical host <b>160</b>B has been selected to receive virtual server <b>162</b>B.
0042Once the new physical host <b>160</b>B has been selected <b>250</b>, the virtual server <b>162</b>B is moved <b>260</b> to the new physical host <b>160</b>B. The virtual server <b>162</b>B is also allocated its required resource increase. In one embodiment, the old overloaded physical host <b>160</b>A places state information for the virtual server <b>162</b>B being transferred into a common file system <b>150</b>, e.g. in a configuration file or other system file. The new physical host <b>160</b>B accesses the state information and restarts the virtual server <b>162</b>B as resident in the new physical host <b>160</b>B. In another embodiment, the virtual server <b>162</b>B files are copied directly from the old physical host <b>160</b>A to the new physical host <b>160</b>B.
0043Once the virtual server information transfer is complete, the old physical host <b>160</b>A has one fewer virtual server, and the new physical host <b>160</b>B has one additional virtual server. The quality of service tables for both the old and new physical hosts are modified <b>260</b> to reflect this change. The quality of service table entries for virtual server <b>162</b>B will also reflect the virtual server's resource increase.
0044The virtual server user is transferred <b>270</b> from the old physical host (<b>160</b>A) to the new physical host (<b>160</b>B) by transferring the virtual server address. The transfer process may use either “break, then make” timing, or “make, then break” timing. The timing of the transfer process determines whether all processes and configuration information associated with the virtual server to be transferred are first shut down in the old physical host, or first started up in the new physical host, before the virtual server address is transferred. Transferring the virtual server address transfers the virtual server user from one virtual server location to another. For example, using “break, then make” timing, the virtual server <b>162</b>B is first shut down in the old physical host <b>160</b>A, a new virtual server is created in new physical host <b>160</b>B and started up, and the virtual server <b>162</b>B address is then transferred over to the new physical host <b>160</b>B. In another embodiment using “make, then break” timing, a new virtual server is created in new physical host <b>160</b>B and started up, the virtual server <b>162</b>B address is transferred over to the new physical host <b>160</b>B, and the virtual server <b>162</b>B is then shut down in old physical host <b>160</b>A.
0045As used herein, the terms “customer,” “user,” and “virtual server user” refer to individuals or groups of individuals accessing the same virtual server. Typically, a virtual server “user” is a group of individuals with a shared association. For example, “user” may collectively refer to the employees of a company, or to certain employees within a division of a company. One company (a “customer”) may have several different users, each corresponding to a different group within the company, and each having many different individuals. Additionally, a “user” may also refer to a single individual.
0046The process for virtual server resource configuration is dynamic and ongoing during the operation of the virtual servers. After the virtual server user transfer <b>270</b> is completed, the process continues to monitor <b>210</b> virtual server resource denials.
0047<figref idref="DRAWINGS">FIG. 2B</figref> is another embodiment of a flowchart of the process for dynamically modifying the resources of a virtual server. The method shown in <figref idref="DRAWINGS">FIG. 2B</figref> is similar to the method shown in <figref idref="DRAWINGS">FIG. 2A</figref>. However, the method of <figref idref="DRAWINGS">FIG. 2B</figref> includes three additional steps, steps <b>242</b>, <b>244</b> and <b>246</b>, which together provide a method for reclaiming unused virtual server resources.
0048As before, virtual server resource denials are monitored <b>210</b>. If a determination <b>220</b> is made that a particular virtual server resource is overloaded, and a determination <b>230</b> is made that the corresponding physical host resources are not overloaded, the virtual server resource allocation is increased <b>240</b>.
0049Next, a timer is set <b>242</b> for a pre-specified interval. Upon timer expiry, the method determines <b>244</b> whether the newly increased virtual server resource is currently operating at its resource limit. If one or more resource denial signals corresponding to the newly increased virtual server resource are received during the timer period, the virtual server is assumed to be operating at its resource limit.
0050If the virtual server is determined <b>244</b> to be operating at its limit for a particular resource, the method continues <b>210</b> to monitor resource denials. However, if the virtual server is not operating at its limit for a particular resource, the method decreases <b>246</b> the virtual server resource allocation by a pre-specified amount. Steps <b>242</b>, <b>244</b>, and <b>246</b> allow the dynamic resource configuration module <b>100</b> to reclaim unused resources within the virtual server system, by temporarily increasing resources allocated to a virtual server as needed.
0051In another embodiment, a recently transferred virtual server <b>162</b> may also allow unused resources to be reclaimed by the virtual server <b>162</b>'s new physical host. In this embodiment, step <b>270</b> would be followed by steps <b>242</b>, <b>244</b> and <b>246</b>.
0052<figref idref="DRAWINGS">FIG. 3</figref> shows an embodiment of one process for determining whether an individual resource in a virtual server has reached its resource limit. A virtual server resource monitor <b>110</b> receives a set of input signals <b>312</b> from a virtual server <b>162</b>B. The virtual server resource monitor <b>110</b> processes these signals to determine if any resources from virtual server <b>162</b>B are overloaded. If an overloaded resource is found, the virtual server resource monitor <b>110</b> sends a “resource overloaded” signal <b>350</b> to the virtual server resource modifier <b>120</b>.
0053Many different types of input signals <b>312</b> may be processed to determine if a resource is overloaded. The virtual server resource monitor <b>110</b> monitors different types of resource denials, which are instances wherein a request for additional resources is either implicitly or explicitly denied. <figref idref="DRAWINGS">FIG. 3</figref> shows four examples of resource denial signals: a create file denial signal <b>312</b>A generated, for example, by a lack of disk space, a memory allocation denial signal <b>312</b>B, a network packet delay signal <b>312</b>C generated by a lack of network bandwidth, and a central processing unit (CPU) process scheduling delay signal <b>312</b>D generated by exceeding CPU usage limits. It is to be understood that there may be many other types of signals indicating an implicit or explicit denial of resources. The examples shown herein are used purely for illustrative purposes.
0054In order to associate resource request denials with a particular virtual server executing in a physical host computer, certain selected system calls are intercepted. For example, not all CPU scheduling within the physical host computer is associated with a virtual sewer. The monitor <b>110</b> must be able to distinguish between resource requests made from virtual servers, and other resource requests. The monitor <b>110</b> must also be able to distinguish between resource requests made by different virtual servers within the same physical server.
0055A system call performs some system operation, such as the access of a system hardware or software resource, when the system call is executed. In order to make a system call, arguments are programmatically loaded into specific registers of the central processing unit on which the operating system is executing. One of these arguments identifies the specific system call that is being made. This argument is typically in the form of a number that is an offset into the operating system interrupt vector table, which contains pointers to the actual executable code of the system calls. The other loaded arguments include parameters to be passed to the system call.
0056Once the arguments have been loaded, a software interrupt is generated, signaling to the operating system that a process is requesting execution of a system call. The operating system reads the registers, and executes the requested system call with the specified parameters. The system call executes and performs the desired functionality. If the system call generates a return value, it places the generated return value (or a pointer thereto) in a pre-designated register where it can be accessed by the calling process.
0057In order to intercept a system call, a pointer in an interrupt vector table to a system call is replaced with a pointer to alternative object code to be executed instead of the system call. Then, when the system call is made, the alternative object code will execute instead. The alternative object code is known as a system call wrapper.
0058The method of the related application may be used to selectively intercept system calls such that a system call wrapper only executes when a system call is made by a select process associated with one of the virtual servers being monitored. When a system call is made by a non-select process, the default system call is executed. Furthermore, only certain types of system calls relating to resource allocation, as described above, are selectively intercepted.
0059The system call wrapper for the intercepted system call allows the resource request by a particular virtual server and the resulting response to be monitored. Request denial responses are monitored by the virtual server resource monitor <b>110</b>. As will be evident to one of skill in the art, the specific system calls to be monitored will be system-dependent, and may vary based upon the type of operating system and physical server machine being used.
0060Each resource denial signal <b>312</b> is input into an individual resource denial table <b>320</b> for tracking purposes. Create file denial signals <b>312</b>A are recorded in a disk denial table <b>320</b>A; memory allocation denial signals <b>312</b>B are recorded in a memory denial table <b>320</b>B; network packet delay signals <b>312</b>C are recorded in a network denial table <b>320</b>C; and CPU process scheduling delay signals <b>312</b>D are recorded in a CPU denial table <b>320</b>D. A calculation <b>330</b> is performed on the signals stored in each table to determine the mean number of times a particular resource denial occurs in a pre-specified time window. Different time windows may be specified for each type of resource denial. The calculation of mean resource denials is performed individually for each different type of resource denial being monitored (<b>330</b>A, <b>330</b>B, <b>330</b>C and <b>330</b>D).
0061The mean number of resource denials may be calculated using one of several well-known techniques for averaging a signal rate over a period of time. Each technique determines whether the number of received resource denial signals a received in a particular time window t exceeds a certain threshold T: <br />a(t)>T?
0062In one embodiment, a “jumping-window” technique is used. The jumping-window technique measures the number of resource denials a received in consecutive windows of time length t. A new time interval t starts immediately after the end of the last time interval t. In another embodiment, a “moving-window” technique is used. The moving-window technique measures the number of resource denials a received in a continuously moving window of time length t. In the moving-windows technique, all windows of time length t are measured.
0063The virtual server resource monitor <b>110</b> checks <b>340</b> if the metric a(t) calculated is beyond the pre-specified threshold T. This determination is made individually for each type of resource denial signal (<b>340</b>A, <b>340</b>B, <b>340</b>C and <b>340</b>D), and need not be made simultaneously. Each different type of resource denial signal <b>312</b> may have a different pre-specified threshold T.
0064If the metric a(t) representing the average resource denial rate does not exceed the threshold T, the method continues to calculate a(t) <b>330</b> so that resource denials are continuously monitored. Using the jumping-window technique, after the next consecutive time interval t passes, the method will again check <b>340</b> if a(t)>T. Using the moving-windows technique, a continuous loop of steps <b>330</b> and <b>340</b> is used to measure each continuously-moving window of time t. In another embodiment, a pre-specified schedule for repeating calculating mean resource denials <b>330</b> and checking <b>340</b> if the threshold T has been exceeded can be established to limit the amount of processing required by the virtual server resource monitor <b>110</b>.
0065However, if the metric a(t) does exceed the threshold T, a “resource overloaded” signal is sent <b>350</b> to the virtual server resource modifier <b>120</b>. Each type of resource denial signal <b>312</b> has an associated resource overloaded signal. <figref idref="DRAWINGS">FIG. 3</figref> shows four examples of resource overloaded signals: disk resource overloaded signal <b>350</b>A, memory resource overloaded signal <b>350</b>B, network resource overloaded signal <b>350</b>C, and CPU resource overloaded signal <b>350</b>D. It is to be understood that there may be many other types of signals indicating an overloaded resource. The examples shown herein are used purely for illustrative purposes.
0066<figref idref="DRAWINGS">FIG. 4</figref> shows a flowchart of an embodiment of a method for determining when to increase or decrease a particular resource allocation within a virtual server. The virtual server resource modifier <b>120</b> performs the method shown in <figref idref="DRAWINGS">FIG. 4</figref>. A separate analysis using the method of <figref idref="DRAWINGS">FIG. 4</figref> is performed for each type of resource being monitored.
0067The modifier <b>120</b> waits <b>410</b> to receive a resource overloaded signal <b>350</b> from the virtual server resource monitor <b>110</b>. When a resource overloaded signal <b>350</b> is received, the modifier <b>120</b> checks <b>420</b> to determine whether the signal <b>350</b> falls within a pre-specified “hysteresis time window” H. The hysteresis time window H check <b>420</b> damps the modifier <b>120</b> system to avoid rapid changes in the system state. For example, in a situation in which a virtual server has overloaded its existing memory resource allocation, the virtual server may attempt to access memory repeatedly before the memory resource allocation is increased. Each memory access attempt may generate a memory resource overloaded signal <b>350</b>B. The modifier <b>120</b> only needs to respond to one of these signals. The hysteresis time window H check <b>420</b> avoids repetitive responses to resource overloaded messages. Thus, the modifier <b>120</b> checks <b>420</b> whether the most recently received resource overloaded signal <b>350</b> (received at T<sub>1</sub>) is close in time (within the hysteresis time window H) to a previously received resource overloaded signal <b>350</b> (received at T<sub>0</sub>) for a particular resource: <br />T<sub>1</sub>−T<sub>0</sub><H?
0068If the recent and previous resource overloaded signals have occurred close enough in time to fall within the pre-specified hysteresis time window H, no further action will be taken and the modifier <b>120</b> returns and waits <b>410</b> to receive another resource overloaded signal <b>350</b>. If the current resource overloaded message is not received within the hysteresis time window H, the modifier <b>120</b> proceeds to increase <b>430</b> the virtual server resource allocation.
0069The resource allocation for a particular overloaded resource is increased <b>430</b> by a pre-specified amount i. Amount i may be specified as a certain percentage of the resources of a physical host, or alternatively amount i may be specified as a certain number of resource units. Amount i may also be specified as a certain percentage of each particular virtual server's current resource allocation, e.g. increase a resource by 5% of its current value. After a particular resource has been increased the modifier <b>120</b> sets <b>440</b> a timer for a pre-specified time period.
0070When the timer expires, the modifier <b>120</b> determines <b>450</b> if the recently increased resource is being fully utilized. In one embodiment, a resource is fully utilized if a corresponding resource denial signal has been received within the timer period <b>440</b> after the resource was increased.
0071If the resource is determined <b>450</b> to be fully utilized, the modifier <b>120</b> returns and waits <b>410</b> for an overloaded signal. However, if it is determined that the resource is not being fully utilized, the modifier <b>120</b> decreases <b>460</b> the resource by a pre-specified amount d. Amount d may be specified as a certain percentage of the resources of a physical host, or amount d may be specified as a certain number of resource units. Amount d may also be specified as a certain percentage of each particular virtual server's current resource allocation, e.g. decrease a resource by 10% of its current value.
0072In one embodiment, d (the resource decreases amount) is larger than i (the resource increase amount). This allows unused resources to be decreased aggressively, but overloaded resources to be increased cautiously. In another embodiment, d and i are set such that the resource allocation is increased and decreased by equal amounts. For example, assume that the increase in virtual server resources i is specified as a percentage of each virtual server's current resource allocation. The decrease in virtual server resources d is specified as d=1−(1/1+i), which returns the resource allocation to its previous level. Once the resource reaches a fully utilized state, the modifier <b>120</b> then returns to waiting <b>410</b>.
0073<figref idref="DRAWINGS">FIG. 5</figref> shows a block diagram of an embodiment of a process for performing resource load balancing among physical hosts, in the context of a working example of overloaded physical host <b>160</b>A. The physical host load balancer <b>130</b> periodically monitors the resource usage of a group of physical hosts <b>160</b> (<b>160</b>A, <b>160</b>B and <b>160</b>C) and transfers virtual servers to different ones of these physical hosts <b>160</b> in order to balance the resource loads between the physical hosts <b>160</b>. Requests to increase virtual server resource allocations are also sent to the physical host load balancer <b>130</b> in order to assist in the balancing of physical host <b>160</b> resource loads. This process is next explained by example.
0074In this example, physical host load balancing module <b>130</b> receives a signal <b>510</b> from the virtual server resource modifier <b>120</b> indicating that virtual server <b>162</b>B requires an increased resource allocation. This signal is used as an input <b>520</b> into the load-balancing calculator <b>530</b>. The load-balancing calculator <b>530</b> also requests and receives as input the current physical host resource loads <b>535</b> from the physical host resource monitor <b>540</b>.
0075The physical host resource monitor <b>540</b> performs periodic physical host resource checks <b>545</b> upon the group of physical hosts <b>160</b> (<b>160</b>A, <b>160</b>B and <b>160</b>C). Resource checks <b>545</b> monitor the current virtual server resource guarantees in each quality of service table for each physical host <b>160</b>.
0076The load-balancing calculator <b>530</b> determines whether a virtual server's request for additional resources <b>510</b> will overload the particular physical host currently hosting the virtual server. Using the example shown in <figref idref="DRAWINGS">FIG. 5</figref>, the load-balancing calculator <b>530</b> determines whether physical host <b>160</b>A is capable of supporting the request for additional virtual server <b>162</b>B resources <b>510</b>. If the resource request <b>510</b> exceeds the available resources of physical host <b>160</b>A, the load-balancing calculator <b>530</b> determines that physical host <b>160</b>A is overloaded.
0077In one embodiment, the load-balancing calculator <b>530</b> uses an easiest fit heuristic to find the physical host that has the most available resources. Each different type of resource is associated with an ordinal and a weight. The i<sup>th </sup>resource R<sub>i </sub>has ordinal i and weight w<sub>i</sub>. For example, resource R<sub>1 </sub>represents disk resources, R<sub>2 </sub>represents memory resources, R<sub>3 </sub>represents network resources and R<sub>4 </sub>represents CPU resources. The weights for each respective resource are determined by the system operator.
0078Let R<sub>i</sub>(V) denote the resource requirement of the virtual server under consideration, e.g. virtual server <b>162</b>B, including the requested resource increase from signal <b>510</b>. Let R<sub>i</sub>(S<sub>j</sub>) denote the resource availability at the j<sup>th </sup>physical host. The load-balancing calculator <b>530</b> computes the weighted resource availability of physical host j as the sum over i:
0079<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><munder><mo>∑</mo><mi>i</mi></munder><mo></mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo></mo><mrow><msub><mi>w</mi><mi>i</mi></msub><mo>*</mo><mrow><mo>(</mo><mrow><mrow><msub><mi>R</mi><mi>i</mi></msub><mo></mo><mrow><mo>(</mo><msub><mi>S</mi><mi>j</mi></msub><mo>)</mo></mrow></mrow><mo>-</mo><mrow><msub><mi>R</mi><mi>i</mi></msub><mo></mo><mrow><mo>(</mo><mi>V</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></math></maths><img file="USRE44686E_D0001.tif" />
0080Using the easiest fit heuristic, the load-balancing calculator <b>530</b> will select the physical host with the largest weighted resource availability to receive the virtual server <b>162</b>B (in the example of <figref idref="DRAWINGS">FIG. 5</figref>, physical host <b>160</b>B). The choice of physical host <b>160</b>B is subject to the constraint that the selected physical host <b>160</b>B has sufficient resources to meet the resource demands of virtual server <b>162</b>B. The load-balancing calculator <b>530</b> sends <b>550</b> a signal <b>560</b> to the dynamic virtual server mover <b>140</b> indicating that virtual server <b>162</b>B is to be transferred to physical host <b>160</b>B.
0081It will be understood by one of skill in the art that load-balancing calculator <b>530</b> may use other criteria for selecting which virtual server to transfer out of an overloaded physical host. In the embodiment given above, the load balancing calculator <b>530</b> transfers the virtual server that has most recently requested additional resources. However, in another embodiment, the load balancing calculator could select, for example, the smallest virtual server within an overloaded physical host for transfer, regardless of which virtual server has recently made a request for increased resources.
0082<figref idref="DRAWINGS">FIG. 6</figref> is a flowchart of an embodiment of the process for transferring a virtual server from one physical host to another physical host. The dynamic virtual server mover <b>140</b> directs the process of <figref idref="DRAWINGS">FIG. 6</figref>. This process is next explained by example.
0083In this example, virtual server <b>162</b>B is transferred from old physical host <b>160</b>A to new physical host <b>160</b>B. The mover <b>140</b> waits <b>610</b> to receive a transfer virtual server signal <b>560</b>. The mover <b>140</b> receives a signal <b>560</b> directing the transfer of virtual server <b>162</b>B from physical host <b>160</b>A to physical host <b>160</b>B. The mover <b>140</b> directs physical host <b>160</b>A to store <b>620</b> local state information associated with virtual server <b>162</b>B in the file system <b>150</b>. As shown in <figref idref="DRAWINGS">FIG. 1</figref>, file system <b>150</b> is commonly accessible from physical hosts <b>160</b>A, <b>160</b>B and <b>160</b>C.
0084Mover <b>140</b> next directs physical host <b>160</b>A to stop <b>630</b> local processes associated with the virtual server being moved, e.g. virtual server <b>162</b>B. Mover <b>140</b> directs physical host <b>160</b>B to access <b>640</b> the virtual server <b>162</b>B state information stored in file system <b>150</b>. Mover <b>140</b> directs physical host <b>160</b>B to start <b>650</b> processes associated with virtual server <b>162</b>B locally. This enables virtual server <b>162</b>B to begin running locally in physical host <b>160</b>B. The user of virtual server <b>162</b>B is then transferred <b>660</b> from physical host <b>160</b>A to physical host <b>160</b>B by transferring the virtual server <b>162</b>B address to the new physical host <b>160</b>B. As explained previously, the mover <b>140</b> may use either “make, then break” timing or “break, then make” timing for the transfer process. Although the invention has been described in considerable detail with reference to certain embodiments, other embodiments are possible. As will be understood by those of skill in the art, the invention may be embodied in other specific forms without departing from the essential characteristics thereof. For example, the dynamic resource configuration module may support different numbers of physical hosts. Additionally, different fitting heuristic methods may be used to select physical hosts for receiving transferred virtual servers during load balancing among the physical hosts. Accordingly, the present invention is intended to embrace all such alternatives, modifications, and variations as fall within the spirit and scope of the appended claims and equivalents.
Contents5
Every citation, both waysCites: the store holds 101 of 102
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US12107849B2 | Cited by | United States of America | Search report |
| US11194620B2 | Cited by | United States of America | Applicant |
| US9405581B2 | Cited by | United States of America | Applicant |
| US10601740B1 | Cited by | United States of America | Search report |
| US11188368B2 | Cited by | United States of America | Applicant |
| US10819702B2 | Cited by | United States of America | Search report |
| US9880884B2 | Cited by | United States of America | Applicant |
| US9680920B2 | Cited by | United States of America | Applicant |
| US9400689B2 | Cited by | United States of America | Applicant |
| US9686347B2 | Cited by | United States of America | Applicant |
| US11671421B2 | Cited by | United States of America | Applicant |
| US2023269245A1 | Cited by | United States of America | Search report |
| US2018288028A1 | Cited by | United States of America | Search report |
| US2009282101A1 | Cites | United States of America | Search report |
| US3377624A | Cites | United States of America | Applicant |
| US4177510A | Cites | United States of America | Applicant |
| US5189667A | Cites | United States of America | Applicant |
| US5212793A | Cites | United States of America | Applicant |
| US5226160A | Cites | United States of America | Applicant |
| US5249290A | Cites | United States of America | Applicant |
| US5263147A | Cites | United States of America | Applicant |
| US5325530A | Cites | United States of America | Applicant |
| US5437032A | Cites | United States of America | Applicant |
| US5528753A | Cites | United States of America | Applicant |
| US5572680A | Cites | United States of America | Applicant |
| US5584023A | Cites | United States of America | Applicant |
| US5603020A | Cites | United States of America | Applicant |
| US5623492A | Cites | United States of America | Applicant |
| US5636371A | Cites | United States of America | Applicant |
| US5640595A | Cites | United States of America | Applicant |
| US5692047A | Cites | United States of America | Applicant |
| US5706097A | Cites | United States of America | Applicant |
| US5706453A | Cites | United States of America | Applicant |
| US5708774A | Cites | United States of America | Applicant |
| US5719854A | Cites | United States of America | Applicant |
| US5727203A | Cites | United States of America | Applicant |
| US5748614A | Cites | United States of America | Applicant |
| US5752003A | Cites | United States of America | Applicant |
| US5761477A | Cites | United States of America | Applicant |
| US5764889A | Cites | United States of America | Applicant |
| US5781550A | Cites | United States of America | Applicant |
| US5799173A | Cites | United States of America | Applicant |
| US5809527A | Cites | United States of America | Applicant |
| US5828893A | Cites | United States of America | Applicant |
| US5838686A | Cites | United States of America | Applicant |
| US5838916A | Cites | United States of America | Applicant |
| US5842002A | Cites | United States of America | Applicant |
| US5845129A | Cites | United States of America | Applicant |
| US5850399A | Cites | United States of America | Applicant |
| US5860004A | Cites | United States of America | Applicant |
| US5864683A | Cites | United States of America | Applicant |
| US5889956A | Cites | United States of America | Applicant |
| US5889996A | Cites | United States of America | Applicant |
| US5892968A | Cites | United States of America | Applicant |
| US5905730A | Cites | United States of America | Applicant |
| US5905859A | Cites | United States of America | Applicant |
| US5913024A | Cites | United States of America | Applicant |
| US5915085A | Cites | United States of America | Applicant |
| US5915095A | Cites | United States of America | Applicant |
| US5918018A | Cites | United States of America | Applicant |
| US5920699A | Cites | United States of America | Applicant |
| US5933603A | Cites | United States of America | Applicant |
| US5937159A | Cites | United States of America | Applicant |
| US5956481A | Cites | United States of America | Applicant |
| US5961583A | Cites | United States of America | Applicant |
| US5978373A | Cites | United States of America | Applicant |
| US5982748A | Cites | United States of America | Applicant |
| US5987524A | Cites | United States of America | Applicant |
| US5991812A | Cites | United States of America | Applicant |
| US5999963A | Cites | United States of America | Applicant |
| US6016318A | Cites | United States of America | Applicant |
| US6018527A | Cites | United States of America | Applicant |
| US6023721A | Cites | United States of America | Applicant |
| US6038608A | Cites | United States of America | Applicant |
| US6038609A | Cites | United States of America | Applicant |
| US6047325A | Cites | United States of America | Applicant |
| US6055617A | Cites | United States of America | Applicant |
| US6061349A | Cites | United States of America | Applicant |
| US6065118A | Cites | United States of America | Applicant |
| US6075791A | Cites | United States of America | Applicant |
| US6075938A | Cites | United States of America | Applicant |
| US6078929A | Cites | United States of America | Applicant |
| US6078957A | Cites | United States of America | Applicant |
| US6086623A | Cites | United States of America | Applicant |
| US6092178A | Cites | United States of America | Applicant |
| US6094674A | Cites | United States of America | Applicant |
| US6101543A | Cites | United States of America | Applicant |
| US6108701A | Cites | United States of America | Applicant |
| US6108759A | Cites | United States of America | Applicant |
| US6122673A | Cites | United States of America | Applicant |
| US6154776A | Cites | United States of America | Applicant |
| US6154778A | Cites | United States of America | Applicant |
| US6161139A | Cites | United States of America | Applicant |
| US6167520A | Cites | United States of America | Applicant |
| US6172981B1 | Cites | United States of America | Applicant |
| US6189046B1 | Cites | United States of America | Applicant |
| US6192389B1 | Cites | United States of America | Applicant |
| US6192512B1 | Cites | United States of America | Applicant |
| US6230203B1 | Cites | United States of America | Applicant |
| US6240463B1 | Cites | United States of America | Applicant |
3 members in 1 office
Priority claims10
| Document | Office | Kind | Date |
|---|---|---|---|
| 56937100 | United States of America | A | |
| 56937100 | United States of America | A | |
| 97177808 | United States of America | A | |
| 97177808 | United States of America | A | |
| 201113236527 | United States of America | A | |
| 09569371 | – | – | – |
| 11971778 | – | – | – |
| US20000569371 | – | – | – |
| US20080971778 | – | – | – |
| US201113236527 | – | – | – |
Members3
| Document | Office | Kind | |
|---|---|---|---|
| US6985937B1 | United States of America | B1 | |
| USRE42726E | United States of America | E | |
| USRE44686EThis record | United States of America | E |
52 transactions on the USPTO file
Allowed after 1 non-final rejection.
- Non-final rejections
- 1
- Final rejections
- 0
- RCEs
- 0
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Post Issue Communication - Certificate of CorrectionN423 | N423 | |
| Recordation of Patent Grant MailedPGM/ | PGM/ | |
| Email NotificationEML_NTR | EML_NTR | |
| Issue Notification MailedAllowedWPIR | WPIR | |
| Dispatch to FDCD1935 | D1935 | |
| Email NotificationEML_NTR | EML_NTR | |
| Printer Rush- No mailingTCPB | TCPB | |
| Mail Response to 312 Amendment (PTO-271)MN271 | MN271 | |
| Response to Amendment under Rule 312N271 | N271 | |
| Pubs Case Remand to TCPUBTC | PUBTC | |
| Application Is Considered Ready for IssuePILS | PILS | |
| Amendment after Notice of Allowance (Rule 312)AllowedA.NA | A.NA | |
| Miscellaneous Incoming LetterLET. | LET. | |
| Issue Fee Payment VerifiedN084 | N084 | |
| Issue Fee Payment ReceivedIFEE | IFEE | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Notice of AllowanceAllowedMN/=. | MN/=. | |
| Notice of Allowance Data Verification CompletedAllowedN/=. | N/=. | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Paralegal or electronic terminal disclaimer approvedP574 | P574 | |
| Terminal Disclaimer FiledDIST | DIST | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Non-Final ActionA... | A... | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Email NotificationEML_NTF | EML_NTF | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Application Is Now CompleteCOMP | COMP | |
| Email NotificationEML_NTR | EML_NTR | |
| Filing Receipt - UpdatedFLRCPT.U | FLRCPT.U | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Miscellaneous Incoming LetterLET. | LET. | |
| Reference capture on IDSRCAP | RCAP | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Miscellaneous Incoming LetterLET. | LET. | |
| Additional Application Filing FeesADDFLFEE | ADDFLFEE | |
| A statement by one or more inventors satisfying the requirement under 35 USC 115, Oath of the ApplicOATHDECL | OATHDECL | |
| Notice of Reissue Published in Official GazetteNRE. | NRE. | |
| Electronic ReviewELC_RVW | ELC_RVW | |
| Cleared by OIPE CSRL194 | L194 | |
| Email NotificationEML_NTR | EML_NTR | |
| Email NotificationEML_NTF | EML_NTF | |
| Filing ReceiptFLRCPT.O | FLRCPT.O | |
| Notice Mailed--Application Incomplete--Filing Date AssignedINCD | INCD | |
| Miscellaneous Incoming LetterLET. | LET. | |
| Preliminary AmendmentA.PE | A.PE | |
| IFW Scan & PACR Auto Security ReviewSCAN | SCAN | |
| Initial Exam Team nnIEXX | IEXX |
6 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| AssignmentAS | AS | |
| Fee paymentFPAY | FPAY | |
| AssignmentAS | AS | |
| Certificate of correctionCC | CC | |
| AssignmentAS | AS | |
| AssignmentAS | AS |
Numbers
- Publication
- RE044686
- Publication, DOCDB
- RE44686
- Publication, EPODOC
- USRE44686E
- Application
- 13236527
- Application, DOCDB
- 201113236527
- Application, EPODOC
- US201113236527
Titles
- English
- Dynamically modifying the resources of a virtual server
Classification
- CPC, 2
- G06F9/505
- G06F9/5077
- IPC, 3
- G06F15 173
- G06F9 46
- G06F11 00
- USPC, 8
- 709223000
- 370231000
- 370235000
- 709224000
- 709226000
- 709238000
- 714035000
- 718105000