Maintaining high availability during n-node failover

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

0001This application claims priority to U.S. Provisional Application No. 62/434,398, filed on Dec. 14, 2016, the entirety of which is incorporated by reference herein.

BACKGROUND

0002A virtual machine (“VM”) may refer to a specific software-based implementation of a machine in a virtualization environment, in which the hardware resources of a real computer (e.g., CPU, memory, etc.) are virtualized or transformed into the underlying support for the fully functional virtual machine that can run its own operating system and applications on the underlying physical resources just like a real computer.

0003Virtualization works by inserting a thin layer of software directly on the computer hardware or on a host operating system. This layer of software contains a virtual machine monitor or “hypervisor” that allocates hardware resources dynamically and transparently. Multiple operating systems run concurrently on a single physical computer and share hardware resources with each other. By encapsulating an entire machine, including CPU, memory, operating system, and network devices, a virtual machine is completely compatible with most standard operating systems, applications, and device drivers. Most modern implementations allow several operating systems and applications to safely run at the same time on a single computer, with each having access to the resources it needs when it needs them.

0004Virtualization allows one to run multiple virtual machines on a single physical machine, with each virtual machine sharing the resources of that one physical computer across multiple environments. Different virtual machines can run different operating systems and multiple applications on the same physical computer.

0005One reason for the broad adoption of virtualization in modern business and computing environments is because of the resource utilization advantages provided by virtual machines. Without virtualization, if a physical machine is limited to a single dedicated operating system, then during periods of inactivity by the dedicated operating system the physical machine is not utilized to perform useful work. This is wasteful and inefficient if there are users on other physical machines which are currently waiting for computing resources. To address this problem, virtualization allows multiple VMs to share the underlying physical resources so that during periods of inactivity by one VM, other VMs can take advantage of the resource availability to process workloads. This can produce great efficiencies for the utilization of physical devices, and can result in reduced redundancies and better resource cost management.

0006Furthermore, there are now products that can aggregate multiple physical machines, running virtualization environments to not only utilize the processing power of the physical devices to aggregate the storage of the individual physical devices to create a logical storage pool wherein the data may be distributed across the physical devices but appears to the virtual machines to be part of the system that the virtual machine is hosted on. Such systems operate under the covers by using metadata, which may be distributed and replicated any number of times across the system, to locate the indicated data. These systems are commonly referred to as clustered systems, wherein the resources of the group are pooled to provide logically combined, but physically separate systems.

SUMMARY

0007In accordance with one aspect of the present disclosure, a system is disclosed. The system includes a plurality of nodes in a virtualization environment, with each of the plurality of nodes having a plurality of virtual machines. The system also includes a scheduling module in the virtualization environment. The scheduling module is configured to reserve resources for each of the plurality of virtual machines of a first node on at least one candidate node. The at least one candidate node is part of the plurality of nodes and separate from the first node, and to reserve the resources for each of the plurality of virtual machines, the scheduling module is configured to identify a list of candidate nodes based upon a qualifier function, rank the candidate nodes based upon a priority function, and determine the candidate nodes that satisfy a high availability threshold. The scheduling module is also configured to select a highest-ranked candidate node that satisfies the high availability threshold and reserve the resources on the highest-ranked candidate node for the each of the plurality of virtual machines for operation during failure of the first node.

0008In accordance with other aspects of the present disclosure, a method is disclosed. The method includes identifying, by a scheduling module, candidate nodes for allocating resources for a virtual machine that resides on a first node. The first node is separate from the candidate nodes, and the scheduling module, the candidate nodes, and the first node are part of a virtualization environment. The scheduling module identifies the candidate nodes based upon a qualifier function. The method also includes ranking, by the scheduling module, the candidate nodes based upon a priority function, determining, by the scheduling module, the candidate nodes that satisfy a high availability threshold, and reserving, by the scheduling module, the resources on a highest-ranked candidate node that satisfies the high availability threshold, such that the virtual machine is configured to operate on the reserved resources during failure of the first node.

0009In accordance with yet other aspects of the present disclosure, a non-transitory computer readable media is disclosed. The non-transitory computer readable media has computer-executable instructions embodied thereon that, when executed by a processor of a virtualization environment, causes the processor to identify a list of candidate nodes based upon a qualifier function to reserve resources for a virtual machine of a first node, and the candidate nodes are separate from the first node and part of the virtualization environment. The instructions also cause the process to rank the candidate nodes based upon a priority function, determine the candidate nodes that satisfy a high availability threshold, and select a highest-ranked candidate node that satisfies the high availability threshold and reserve the resources on the highest-ranked candidate node for the virtual machine for operation during failure of the first node.

0010The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the following drawings and the detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

<figref idref="DRAWINGS">FIG. 1A</figref> is an example block diagram of a virtualization environment, in accordance with some embodiments of the present disclosure.

<figref idref="DRAWINGS">FIG. 1B</figref> is a block diagram showing an example of data flow within the virtualization environment of <figref idref="DRAWINGS">FIG. 1A</figref>, in accordance with some embodiments of the present disclosure.

<figref idref="DRAWINGS">FIGS. 2A-2C</figref> are block diagrams showing examples of resource reservations, in accordance with some embodiments of the present disclosure.

<figref idref="DRAWINGS">FIG. 3</figref> is an example flowchart outlining operations for reserving resources within the virtualization environment of <figref idref="DRAWINGS">FIG. 1A</figref>, in accordance with some embodiments of the present disclosure.

<figref idref="DRAWINGS">FIG. 4</figref> is an example state diagram for using the reserving resources during a failure of a node in the virtualization environment of <figref idref="DRAWINGS">FIG. 1A</figref>, in accordance with some embodiments of the present disclosure.

<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of a computing system suitable for implementing particular embodiments disclosed herein.

0017The foregoing and other features of the present disclosure will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only several embodiments in accordance with the disclosure and are, therefore, not to be considered limiting of its scope, the disclosure will be described with additional specificity and detail through use of the accompanying drawings.

DETAILED DESCRIPTION

0018This disclosure is directed to a system and method for allocating resources in case of one or more node failures in a virtualization environment. As discussed in greater detail below, the virtualization environment includes one or more clusters, with each cluster having one or more nodes (also called host machines) and with each node having one or more virtual machines. Since the virtual machines on a particular node use the various hardware and software resources of the underlying node, during a failure of a particular node, the virtual machines on the failed node may be rendered non-operational until the failed node recovers from the failure. To minimize such disruptions on the operations of the virtual machines due to failures of an underlying node, the present disclosure provides a scheduling module that reserves resources for each virtual machine in the virtualization environment for use during failure of the underlying node.

0019The resources are reserved on other nodes within the virtualization environment. In some embodiments, the scheduling module may reserve resources for each virtual machine on other nodes of the same cluster, nodes of another cluster, or a combination thereof. By reserving resources for a particular virtual machine on another node, the virtual machines of a failed node may be restarted on the node having the reserved resources, thereby minimizing disruptions to the operations of that virtual machine. For example, if Node <b>1</b> has 2 virtual machines (VM<b>1</b> and VM<b>2</b>), the scheduling module may reserve resources for each of VM<b>1</b> and VM<b>2</b> on other nodes (e.g., Node <b>2</b>) in the virtualization environment. During failure of the Node <b>1</b>, the scheduling module may restart VM<b>1</b> and VM<b>2</b> on Node <b>2</b> instead of Node <b>1</b>. It is to be understood that all the virtual machines on a particular node need not have reserved resources on the same node. Thus, VM<b>1</b> may have reserved resources on Node <b>2</b> and VM<b>2</b> may have reserved resources on Node <b>3</b>. In some embodiments, to account for multiple node failures, the scheduling module may reserve resources for some or all virtual machines on multiple nodes. Thus, the scheduling module may reserve resources for both VM<b>1</b> and VM<b>2</b> on both Nodes <b>2</b> and <b>3</b>. In other embodiments, the scheduling module may reserve resources for VM<b>1</b> on both Nodes <b>2</b> and <b>3</b>, and reserve resources for VM<b>2</b> on only Node <b>2</b>.

0020The scheduling module takes into account multiple factors in determining which nodes to reserve resources on and how many nodes to reserve resources on. For example, in some embodiments, the scheduling module looks at the amount and type of resources (e.g., CPU, Memory, etc.) that a particular virtual machine for which resources have to reserved needs to operate properly. The scheduling module may also look at the resources that are consumed by virtual machines on other nodes on which resources may potentially be reserved and whether those nodes have resources available to accommodate other virtual machines if needed. The scheduling module may look at other factors (also called qualifier functions) in identifying a list of candidate nodes on which resources may potentially be reserved. For example, if a particular node only has enough resources to operate the virtual machines that are originally on that node, such a node may not be a good candidate for reserving resources. The scheduling module may use other or additional qualifier functions to identify the list of the candidate nodes. The qualifier functions may be pre-determined and pre-programmed within the scheduler module.

0021Upon identifying the list of the candidate nodes on which resources may be reserved, the scheduling module may rank the candidate nodes in the list based on certain criteria (also called priority functions). The scheduling module may take various factors into account in ranking the various candidate nodes. For examples, the candidate nodes that have more resources to spare than other nodes may be ranked higher. Candidate nodes that are less likely to fail may also be ranked higher. The priority functions may be pre-defined and pre-programmed within the scheduling module. In some embodiments, each priority function may be assigned a weight, such that candidate nodes having higher weighted priority functions may be ranked higher. Thus, based upon the desired requirements for operating the virtualization environment, the priority functions may vary.

0022From the ranked candidate nodes, the scheduling module may determine which candidate nodes are “highly available” or in other words, satisfy a high availability threshold. In some embodiments, “highly available” candidate nodes are those nodes that have sufficient resources to satisfy the resource requirements of another virtual machine and still function without impact or significant impact to the existing virtual machines on that node. For example, the scheduling module may determine that VM<b>1</b> needs a specific type of a graphic card to operate properly. Then, to reserve resources for VM<b>1</b> on other ranked candidate nodes, the scheduling module may find candidate nodes that have the same graphic card as needed by VM<b>1</b>. If only Node <b>2</b> has that graphic card and Node <b>3</b> does not, then Node <b>2</b> may be considered “highly available” (and therefore potentially satisfies the high availability threshold) and the scheduling module may attempt to reserve resources for VM<b>1</b> on Node <b>2</b> subject to any additional conditions being satisfied. These additional conditions may look at other factors (e.g., hardware software policy constraints, etc.) in determining which nodes to reserve resources on.

0023Once the scheduling module identifies one or more “highly available” candidate node(s) (e.g., a candidate node that satisfies the high availability threshold), the scheduling module may reserve parcels or segments of resources on one or more of the highest-ranked candidate nodes that are “highly available” for a particular virtual machine. That particular virtual machine is intended to use the reserved resources only during failure of its underlying node. However, the virtual machines on the candidate node on which the resources have been reserved may continue to use the reserved resources until those resources are needed in failure conditions. For example, if VM<b>1</b> has reserved resources on Node <b>2</b>, the virtual machines on Node <b>2</b> may continue to use the reserved resources until VM<b>1</b> needs the reserved resources due to failure of Node <b>1</b> on which VM<b>1</b> originally resides. In some embodiments, the scheduling module may use a look up table to map each virtual machine with the nodes having their reserved resources. Upon detecting a failure of an underlying node, the scheduling module may consult the look-up table and find the virtual machines from the failed node and map those virtual machines to the nodes having their reserved resources. The scheduling module may then restart those virtual machines on the nodes with the reserved resources. In some embodiments, once the failed node is back up again, the scheduling module may move the virtual machines back to that node, thereby freeing up the reserved resources for use again. In other embodiments, the scheduling module may use other mechanisms for mapping the virtual machines to their respective reserved resources.

0024As noted above, some virtual machines may have resources reserved on multiple nodes. Likewise, in some embodiments, certain resources may be reserved for multiple virtual machines. For example, a resource X may be reserved for both VM<b>1</b> on Node <b>1</b> and VM<b>2</b> on Node <b>2</b>. Specifically, the scheduling module may be configured to handle a particular number of node failures. For example, in some embodiments, the scheduling module may be configured to handle one node failure. In such cases, the scheduling module may be configured to guarantee availability of resources for virtual machines on any one node. For example, if the scheduling module is configured for handling a single node failure, if Node <b>1</b> fails, the scheduling module guarantees that all virtual machines on Node <b>1</b> have available reserved resources on other nodes (e.g., Node <b>2</b>) and can be restarted on Node <b>2</b>. In these cases, even if the virtual machines on Node <b>3</b> also have the same resources reserved on Node <b>2</b>, since Node <b>3</b> is still functioning, those reserved resources are not needed by Node <b>3</b>. Thus, both Nodes <b>1</b> and <b>3</b> may have reserved resources on Node <b>2</b>, but only one of Nodes <b>1</b> and <b>3</b> may use the reserved resources at a time if the scheduling module is configured to handle a single node failure. When Node <b>1</b> recovers from the failure, the virtual machines of Node <b>1</b> are migrated back from Node <b>2</b> to Node <b>1</b>, thereby freeing up the reserved resources for use again in the future by either Node <b>1</b> or Node <b>3</b>. Thus, by overlapping reserved resources, the scheduling module is able to reserve resources effectively for all virtual machines in the virtualization environment and maintain availability of those reserved resources when needed.

0025In other embodiments, the scheduling module may be configured to handle more than one node failure. In such cases, the scheduling module may reserve resources such that the virtual machines on any two nodes within the virtualization environment are guaranteed resources during failure of their underlying nodes. Thus, the configuration of the scheduling module may vary from one embodiment to another.

0026Furthermore, the scheduling module reserves resources for a particular virtual machine at the time of creating that virtual machine. Thus, when a virtual machine is created on a particular node, the scheduling node may analyze the requirements of that virtual machine, the availability of resources on the other nodes in the virtualization environment, as well any other factors or policies, and reserve resources for the new virtual machine. As part of reserving resources for the new virtual machine, the scheduling module may need to evaluate the resources already reserved for the existing virtual machines and reassign or reallocate some of those already reserved resources to ensure high availability during node failures. The reserving of resources to plan for a potential node failure is termed herein as a “failover plan,” “failure plan,” or the like. Furthermore, when a particular virtual machine is moved to a reserved resource, the scheduling module is configured to not accept any new virtual machines until the failed node(s) are received.

0027By virtue of reserving resources for virtual machines, the present disclosure improves the reliability and robustness of the virtualization environment and increases user satisfaction by minimizing the impact from node failures. Computer functionality is also improved because existing resources are more efficiently used and shared between various components. Operation of virtual machines is improved since the virtual machines are not solely dependent upon the resources of the underlying node.

0028<figref idref="DRAWINGS">FIG. 1A</figref> illustrates a clustered virtualization environment <b>100</b> according to particular embodiments. The architecture of <figref idref="DRAWINGS">FIG. 1A</figref> can be implemented for a distributed platform that contains multiple host machines (a.k.a. nodes) <b>101</b><i>a</i>-<i>c </i>that manage multiple tiers of storage. The multiple tiers of storage may include storage that is accessible through network <b>140</b>, such as, by way of example and not limitation, cloud storage <b>126</b> (e.g., which may be accessible through the Internet), network-attached storage (NAS) <b>128</b> (e.g., which may be accessible through a LAN), or a storage area network (SAN). Unlike the prior art, the present embodiment also permits local storage <b>122</b><i>a</i>-<i>c </i>that is incorporated into or directly attached to the host machine and/or appliance to be managed as part of storage pool <b>160</b>. Examples of such local storage include Solid State Drives <b>125</b> (henceforth “SSDs”), Hard Disk Drives <b>127</b> (henceforth “HDDs” or “spindle drives”), optical disk drives, external drives (e.g., a storage device connected to a host machine via a native drive interface or a serial attached SCSI interface), or any other direct-attached storage. These storage devices, both direct-attached and network-accessible, collectively form storage pool <b>160</b>. Virtual disks (or “vDisks”) may be structured from the physical storage devices in storage pool <b>160</b>, as described in more detail below. As used herein, the term vDisk refers to the storage abstraction that is exposed by a Controller/Service VM (CVM) <b>110</b> to be used by a user VM <b>105</b>. In particular embodiments, the vDisk may be exposed via iSCSI (“internet small computer system interface”) or NFS (“network file system”) and is mounted as a virtual disk on the user VM. In particular embodiments, vDisks may be organized into one or more volume groups (VGs).

0029Each host machine <b>101</b><i>a</i>-<i>c </i>may run virtualization software, such as VMWARE ESX(I), MICROSOFT HYPER-V, or REDHAT KVM. The virtualization software includes hypervisor <b>130</b><i>a</i>-<i>c </i>to create, manage, and destroy user VMs <b>105</b><i>a</i>-<i>c </i>(collectively called “VMs <b>105</b>”), as well as managing the interactions between the underlying hardware and user VMs <b>105</b>. User VMs <b>105</b> may run one or more applications that may operate as “clients” with respect to other elements within virtualization environment <b>100</b>. Though not depicted in <figref idref="DRAWINGS">FIG. 1A</figref>, a hypervisor may connect to network <b>140</b>. In particular embodiments, a host machine <b>101</b> may be a physical hardware computing device; in particular embodiments, a host machine <b>101</b> may be a virtual machine.

0030CVMs <b>110</b><i>a</i>-<i>c </i>(collectively called “CVMs <b>110</b>”) are used to manage storage and input/output (“I/O”) activities according to particular embodiments. These special VMs act as the storage controller in the currently described architecture. Multiple such storage controllers may coordinate within a cluster to form a unified storage controller system. CVMs <b>110</b> may run as virtual machines on the various host machines <b>122</b>, and work together to form a distributed system <b>110</b> that manages all the storage resources, including local storage <b>122</b>, NAS <b>128</b>, and cloud storage <b>126</b>. The CVMs may connect to network <b>140</b> directly, or via a hypervisor. Since the CVMs run independent of hypervisors <b>130</b><i>a</i>-<i>c</i>, this means that the current approach can be used and implemented within any virtual machine architecture, since the CVMs of particular embodiments can be used in conjunction with any hypervisor from any virtualization vendor.

0031A host machine may be designated as a leader node within a cluster of host machines. For example, host machine <b>101</b><i>b</i>, as indicated by the asterisks, may be a leader node. A leader node may have a software component designated to perform operations of the leader. For example, CVM <b>110</b><i>b </i>on host machine <b>101</b><i>b </i>may be designated to perform such operations. A leader may be responsible for monitoring or handling requests from other host machines or software components on other host machines throughout the virtualized environment. If a leader fails, a new leader may be designated. In particular embodiments, a management module (e.g., in the form of an agent) may be running on the leader node.

0032Each CVM <b>110</b><i>a</i>-<i>c </i>exports one or more block devices or NFS server targets that appear as disks to user VMs <b>105</b><i>a</i>-<i>c</i>. These disks are virtual, since they are implemented by the software running inside CVMs <b>110</b><i>a</i>-<i>c</i>. Thus, to user VMs <b>105</b><i>a</i>-<i>c</i>, CVMs <b>110</b><i>a</i>-<i>c </i>appear to be exporting a clustered storage appliance that contains some disks. All user data (including the operating system) in the user VMs <b>105</b><i>a</i>-<i>c </i>and reside on these virtual disks.

0033Significant performance advantages can be gained by allowing the virtualization system to access and utilize local storage <b>122</b> as disclosed herein. This is because I/O performance is typically much faster when performing access to local storage <b>122</b> as compared to performing access to NAS <b>128</b> across a network <b>140</b>. This faster performance for locally attached storage <b>122</b> can be increased even further by using certain types of optimized local storage devices, such as SSDs. Further details regarding methods and mechanisms for implementing the virtualization environment illustrated in <figref idref="DRAWINGS">FIG. 1A</figref> are described in U.S. Pat. No. 8,601,473, which is hereby incorporated by reference in its entirety.

0034<figref idref="DRAWINGS">FIG. 1B</figref> illustrates data flow within an example clustered virtualization environment <b>100</b> according to particular embodiments. As described above, one or more user VMs and a CVM may run on each host machine <b>101</b> along with a hypervisor. As a user VM performs I/O operations (e.g., a read operation or a write operation), the I/O commands of the user VM may be sent to the hypervisor that shares the same server as the user VM. For example, the hypervisor may present to the virtual machines an emulated storage controller, receive an I/O command and facilitate the performance of the I/O command (e.g., via interfacing with storage that is the object of the command, or passing the command to a service that will perform the I/O command). An emulated storage controller may facilitate I/O operations between a user VM and a vDisk. A vDisk may present to a user VM as one or more discrete storage drives, but each vDisk may correspond to any part of one or more drives within storage pool <b>160</b>. Additionally or alternatively, CVM <b>110</b><i>a</i>-<i>c </i>may present an emulated storage controller either to the hypervisor or to user VMs to facilitate I/O operations. CVM <b>110</b><i>a</i>-<i>c </i>may be connected to storage within storage pool <b>160</b>. CVM <b>110</b><i>a </i>may have the ability to perform I/O operations using local storage <b>122</b><i>a </i>within the same host machine <b>101</b><i>a</i>, by connecting via network <b>140</b> to cloud storage <b>126</b> or NAS <b>128</b>, or by connecting via network <b>140</b> to local storage <b>122</b><i>b</i>-<i>c </i>within another host machine <b>101</b><i>b</i>-<i>c </i>(e.g., via connecting to another CVM <b>110</b><i>b </i>or <b>110</b><i>c</i>). In particular embodiments, any suitable computing system <b>500</b> may be used to implement a host machine <b>101</b>.

0035When initially placing (e.g., creating) a user VM into a virtualization environment (e.g., the virtualization environment <b>100</b>), a scheduling module in the hypervisor may account for resource (e.g., CPU or memory) allocation planning in order to prioritize high availability of resources across the virtualization environment. High availability of resources means planning for the possibility of failure of one or more nodes in a cluster and reserving parcels of available resources on other (presumably available) nodes in the cluster to support the option of moving VMs from failed nodes to the other available nodes. As used herein, “resources” may include memory, CPU, virtual graphics cards, or any other hardware, software, or firmware element(s) that a VM may need in performing a certain operation or task. The scheduling module may take into account consumption of multiple types of resources (e.g., CPU, as well as memory) by VMs running on nodes in the cluster. In some embodiments, a scheduling module may account for failure of one or more nodes in the cluster. The scheduling module may consider other factors in addition to or instead of the factors mentioned above in determining the resource allocation for VMs during failure of nodes. For example, the scheduling module may consider factors such as, by way of example and not limitation: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0036">individual and collective resource requirements (e.g., any individual user VM <b>105</b> may have its own resource requirements; in another example, the CVMs <b>110</b> may collectively have resource requirements);</li><li id="ul0002-0002" num="0037">affinity and anti-affinity specifications (e.g., there may be a preference for keeping certain VMs together on the same host machine; conversely, there may be a preference for maintaining no more than one CVM <b>110</b> on any host machine);</li><li id="ul0002-0003" num="0038">Hardware, software, or other policy constraints (certain VMs may need to be placed on host machines other than the leader node, in order to reduce the risk of crashing the leader node);</li><li id="ul0002-0004" num="0039">Quality of Service (“QoS”) requirements (certain high-priority VMs may need to be placed on host machines that are guaranteed to have high resource availability);</li><li id="ul0002-0005" num="0040">data locality (e.g., certain VMs may have preference for placement on certain host machines in order to maintain access to data stored in local storage); and/or</li><li id="ul0002-0006" num="0041">inter-workload interference.</li></ul></li></ul>

0042The scheduling module may also account for detected events that affect the availability of resources, such as current CPU or memory consumption, historical/average CCPU or memory consumption, CPU or memory consumption patterns (e.g., regular fluctuations over a day or a week), VM power-off, VM failover, host machine failover. Certain types of detected events may cause a divergence between the actual state of the cluster and the scheduling module's view of the state of the system (e.g., when a host machine is disconnected then reconnected, when a leader node crashes, or when a host machine crashes in the middle of a VM power-off or VM migration); in order to reconcile the scheduling module's view with the actual state of the system, particular embodiments may remove VMs that have been powered off in the meantime.

0043In order to help with maintaining high availability during VM placement, the scheduling module may be consulted to determine optimal resource allocation prior to taking certain actions such as, by way of example and not limitation: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0044">placement of a newly created VM;</li><li id="ul0004-0002" num="0045">migration of a VM;</li><li id="ul0004-0003" num="0046">removal of a host machine from the cluster;</li><li id="ul0004-0004" num="0047">addition of a host machine to the cluster;</li><li id="ul0004-0005" num="0048">creation of a failover plan; or</li><li id="ul0004-0006" num="0049">pinning a VM to a host machine.</li></ul></li></ul>

0050<figref idref="DRAWINGS">FIG. 2A</figref> is a schematic illustrating consumption of parcels of resources, such as memory, by six VMs running on three host machines. As shown, VMs <b>205</b><i>a</i><b>1</b>, <b>205</b><i>a</i><b>2</b>, and <b>205</b><i>a</i><b>3</b> running on host machine <b>101</b><i>a </i>respectively consume 18.8%, 22.5%, and 12.4% of total available memory on host machine <b>101</b><i>a</i>; VMs <b>205</b><i>b</i><b>1</b>, <b>205</b><i>b</i><b>2</b>, and <b>205</b><i>b</i><b>3</b> running on host machine <b>101</b><i>b </i>respectively consume 15.6%, 21.9%, and 25% of total available memory on host machine <b>101</b><i>b</i>; and VMs <b>205</b><i>c</i><b>1</b>, <b>205</b><i>c</i><b>2</b>, and <b>205</b><i>c</i><b>3</b> running on host machine <b>101</b><i>c </i>respectively consume 21.9%, 18.8%, and 15.6% of total available memory on host machine <b>101</b><i>c. </i>

0051<figref idref="DRAWINGS">FIG. 2B</figref> is a schematic illustrating resource parcel reservation in a failover plan for host machine <b>101</b><i>a</i>, wherein VMs <b>205</b><i>a</i><b>1</b>, <b>205</b><i>a</i><b>2</b>, and <b>205</b><i>a</i><b>3</b> running on host machine <b>101</b><i>a </i>are moved to host machines <b>101</b><i>b </i>and <b>101</b><i>c</i>. Under this failover plan, parcels of resources for VM <b>205</b><i>a</i><b>1</b> are reserved on host machine <b>101</b><i>b </i>(see shaded box for VM <b>205</b><i>a</i><b>1</b>′) and parcels of resources for VMs <b>205</b><i>a</i><b>2</b> and <b>205</b><i>a</i><b>3</b> are reserved on host machine <b>101</b><i>c </i>(see shaded boxes for VMs <b>205</b><i>a</i><b>2</b>′ and <b>205</b><i>a</i><b>3</b>′). As is apparent from <figref idref="DRAWINGS">FIG. 2B</figref>, this cluster is only able to handle failure of one host machine at a time (at least with respect to memory usage), since, after failure of host machine <b>101</b><i>a</i>, there is not sufficient available memory on either host machine <b>101</b><i>b </i>(which is at 81.3% estimated usage after reserving resources for VM <b>205</b><i>a</i><b>1</b>′) or host machine <b>101</b><i>c </i>(which is at 91.2% estimated usage after reserving resources for VMs <b>205</b><i>a</i><b>2</b>′ and <b>205</b><i>a</i><b>3</b>′) to be able to withstand failure of the other host machine.

0052<figref idref="DRAWINGS">FIG. 2C</figref> is a schematic illustrating redistribution of VMs <b>205</b> after a new node (host machine <b>101</b><i>d</i>) is added in order to maintain a high availability cluster that can support two-node failover plan for host machines <b>101</b><i>b </i>and <b>101</b><i>d</i>. As shown, VMs <b>205</b><i>a</i><b>1</b> and <b>205</b><i>a</i><b>2</b> (which are running on host machine <b>101</b><i>a</i>) respectively consume 18.8% and 22.5% of total available memory on host machine <b>101</b><i>a</i>. VMs <b>205</b><i>b</i><b>1</b>, <b>205</b><i>c</i><b>3</b>, and <b>205</b><i>a</i><b>3</b> running on host machine <b>101</b><i>b </i>respectively consume 15.6%, 15.6%, and 12.4% of total available memory on host machine <b>101</b><i>b</i>. VMs <b>205</b><i>c</i><b>1</b> and <b>205</b><i>b</i><b>2</b> running on host machine <b>101</b><i>c </i>respectively consume 21.9% and 21.9% of total available memory on host machine <b>101</b><i>c</i>. VMs <b>205</b><i>b</i><b>3</b> and <b>205</b><i>c</i><b>2</b> running on host machine <b>101</b><i>d </i>respectively consume 25% and 18.8% of total available memory on host machine <b>101</b><i>d. </i>

0053The failover plan for host machines <b>101</b><i>b </i>and <b>101</b><i>d </i>comprises reservation of parcels of resources for VMs <b>205</b><i>b</i><b>1</b>, <b>205</b><i>c</i><b>3</b>, and <b>205</b><i>a</i><b>3</b> on host machines <b>101</b><i>a </i>(see shaded box for VM <b>205</b><i>b</i><b>1</b>′) and <b>101</b><i>c </i>(see shaded boxes for VMs <b>205</b><i>c</i><b>3</b>′ and <b>205</b><i>a</i><b>3</b>′), as well as reservation of parcels of resources for VMs <b>205</b><i>b</i><b>3</b> and <b>205</b><i>c</i><b>2</b> on host machines <b>101</b><i>a </i>(see shaded box for VM <b>205</b><i>b</i><b>3</b>′) and <b>101</b><i>c </i>(see shaded box for VM <b>205</b><i>c</i><b>2</b>′). As shown <figref idref="DRAWINGS">FIG. 2C</figref>, with the addition of host machine <b>101</b><i>d</i>, the cluster is now able to handle failure of two host machines at a time (at least with respect to memory usage).

0054<figref idref="DRAWINGS">FIG. 3</figref> is a flowchart illustrating operations for an example method of determining whether a cluster can withstand N-node failover if a proposed change is implemented. In operation <b>310</b>, the scheduling module receives a request to assess the impact of a proposed change (e.g., such as adding a new VM to a host machine in the cluster) to the virtualization environment <b>100</b> on maintaining high availability of resources for VMs in the cluster. In order to respond to the request, the scheduling module may access one or more factors discussed above. The scheduling module may also assess the ability to continue normal operations if one or more nodes fail (e.g., crash, become unresponsive, or otherwise become unavailable). Operations <b>320</b>-<b>328</b> may assess failover plans for every permutation of pairs of failed nodes in order to assess the ability to place VMs from the failed nodes onto other nodes.

0055In operation <b>320</b>, the scheduling module identifies candidate nodes for post-failover placement of the VM by filtering available nodes based on one or more qualifier functions. For example, a qualifier function may determine whether potential candidate nodes have sufficient quantities of the resource in question available. In another example, a qualifier function may identify potential candidate nodes based on affinity rules. In another example, a qualifier function may identify potential candidate nodes based on whether other resources (e.g., particular required and/or preferred ports) are available.

0056In operation <b>322</b>, the scheduling module ranks the candidate nodes based on one or more priority functions, such as, by way of example and not limitation: identifying nodes that are lightly loaded, identifying nodes with which the VM has an affinity, or identifying nodes that have a balanced CPU or memory usage rate. In particular embodiments, the final ranking score for each of the candidates nodes may comprise a sum of each of the priority functions. In particular embodiments, each of the priority functions may be weighted.

0057In operation <b>324</b>, the scheduling module determines admission of high availability for the ranked candidate nodes. In particular embodiments, if placement of a given VM on a given candidate node does not support high availability, the candidate node may be dropped from eligibility for placement of the VM (but not necessarily dropped from eligibility for placement of all VMs). In particular embodiments, when assessing whether placement of the VM on the candidate node would support high availability, a parcel of resources may be reserved as overhead (e.g., for a hypervisor running on the node, for other processes running on the node, or in order to prevent a single process from growing to consume most or all available resources on the node (e.g., sshd/login, bash, top/kill). In some embodiments, the scheduling module may determine which of the ranked candidate nodes satisfy a high availability threshold (also referred to herein as a high availability admission threshold). The high availability threshold may be based at least upon the resources needed by a given VM and the resources available on the ranked candidate nodes. For example, if a given VM needs a particular resource that is not available on a specific candidate node, then that candidate node is said to not satisfy the high availability threshold, and is not considered for reserving resources for the given VM.

0058In operation <b>326</b>, the scheduling module selects the highest-ranked node that satisfies the high-availability admission threshold. In operation <b>328</b>, the scheduling module reserves a parcel of resources on the selected node for failover of the VM. During the process of reservation of parcels on selected nodes for VMs in the cluster, the scheduling module may identify whether there is a lack of sufficient resources on available nodes to reserve parcels of resources for all the VMs.

0059In operation <b>330</b>, the scheduling module determines whether one or more nodes need to be added to the cluster in order to reserve parcels of resources for all the VMs in anticipation of the possibility of N-node failover. If not, in operation <b>335</b>, the scheduling module sends a response to the original request indicating that the proposed change does not violate the goal of maintaining high availability of resources within the cluster.

0060In operation <b>340</b>, if one or more nodes need to be added to the cluster, the scheduling module may determine whether the required number of nodes are available to be added to the cluster. In operation <b>345</b><i>a</i>, if nodes are not available to be added, the scheduling module may send a response to the original request indicating that the proposed change would violate the goal of maintaining high availability of resources within the cluster. In operation <b>345</b><i>b</i>, if nodes are available to be added, the scheduling module may add the required number of nodes to the cluster. The scheduling module may also redistribute amongst the nodes VMs as appropriate in order to rebalance VMs across the cluster. Finally, in operation <b>350</b>, the scheduling module may send a response to the original request indicating that the proposed change does not violate the goal of maintaining high availability of resources within the cluster.

0061<figref idref="DRAWINGS">FIG. 4</figref> is an example state diagram <b>400</b> for illustrating an example resource allocation during a failover event of one node, in accordance with some embodiments of the present disclosure. The state diagram <b>400</b> may include four states: a highly available state <b>405</b>, a first healing state <b>410</b>, a second healing state <b>415</b>, and a critical state <b>420</b>. The state diagram <b>400</b> is implemented by the scheduling module and may be invoked upon a detected failure of a node. Before any node failure, the scheduling module remains on the highly available state <b>405</b>. When a node failure occurs, the scheduling module moves from the highly available state <b>405</b> to the first healing state <b>410</b>. As part of transitioning to the first healing state <b>410</b>, the scheduling module moves (e.g., migrates and restarts) the VMs from the failed node to their respective reserved parcels or segments. As discussed above, the reserved parcel or segment of each of the VM on each of the nodes in the virtualization environment <b>100</b> is pre-determined at the time of the creation of that VM. Thus, at the time of failure of a node, the scheduling node already knows the reserved parcels or segments for each VM on the failed node.

0062Once the VMs from the failed node are restarted on the reserved parcel or segment, the scheduling module determines whether the virtualization environment <b>100</b> has capability for handling additional node failures. For example and as discussed above, in some embodiments, the scheduling module may have allocated reserved parcels to handle a single node failure. In such cases, if a second node fails, the scheduling module may not have reserved segments for the VMs on the second failed node. In other embodiments, the scheduling module may have allocated reserved parcels to handle more than one (e.g., two) node failures.

0063Thus, at the first healing state <b>410</b>, the scheduling module determines if the scheduling module is capable of handling additional node failures. For example, if the scheduling module is configured to handle a single node failure, then the scheduling module may not be able to move VMs to their reserved parcels in the event of another node failure. In such a case, the scheduling module may transition from the first healing state <b>410</b> to the critical state <b>420</b>. On the other hand, if the scheduling module determines at the first healing state <b>410</b> that additional node failures may be handled (e.g., by guaranteeing reserved parcels of additional VMs), the scheduling module goes back to the highly available state <b>405</b>. The scheduling module then remains in the highly available state <b>405</b> until another node failure is detected.

0064If the scheduling module is at the critical state <b>420</b>, the scheduling module remains at this state until the failed node has recovered or has been fixed. When the failed node comes back up, the VMs that had migrated to the reserved segments are moved back to the recovered node, thereby recovering the reserved parcels or segments. These recovered parcels or segments can again be used for another node failure. Along with moving the VMs back, the scheduling module transitions to the second healing state <b>415</b>. If at the second healing state <b>415</b>, the scheduling module determines that not enough reserved segments have been recovered (e.g., due to one or more VMs not being able to move back to the recovered node), the scheduling module goes back to the critical state <b>420</b>. The scheduling module keeps moving between the critical state <b>420</b> and the second healing state <b>415</b> until enough number of reserved segments or parcels have been recovered. What constitutes “enough” may vary from one embodiment to another.

0065For example, in some embodiments, “enough” may mean all VMs that had moved to their respective reserved parcels. In other embodiments, “enough” may be defined in terms of a percentage, number, or other function of the reserved parcels that have been recovered. Once enough reserved parcels have been recovered, the scheduling module goes back to the highly available state <b>405</b> and waits for another node failure.

0066<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of an illustrative computing system <b>500</b> suitable for implementing particular embodiments. In particular embodiments, one or more computer systems <b>500</b> perform one or more operations of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems <b>500</b> provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems <b>500</b> performs one or more operations of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems <b>500</b>. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

0067This disclosure contemplates any suitable number of computer systems <b>500</b>. This disclosure contemplates computer system <b>500</b> taking any suitable physical form. As example and not by way of limitation, computer system <b>500</b> may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a mainframe, a mesh of computer systems, a server, a laptop or notebook computer system, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system <b>500</b> may include one or more computer systems <b>500</b>; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems <b>500</b> may perform without substantial spatial or temporal limitation one or more operations of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems <b>500</b> may perform in real time or in batch mode one or more operations of one or more methods described or illustrated herein. One or more computer systems <b>500</b> may perform at different times or at different locations one or more operations of one or more methods described or illustrated herein, where appropriate.

0068Computer system <b>500</b> includes a bus <b>502</b> (e.g., an address bus and a data bus) or other communication mechanism for communicating information, which interconnects subsystems and devices, such as processor <b>504</b>, memory <b>506</b> (e.g., RAM), static storage <b>508</b> (e.g., ROM), dynamic storage <b>510</b> (e.g., magnetic or optical), communication interface <b>514</b> (e.g., modem, Ethernet card, a network interface controller (MC) or network adapter for communicating with an Ethernet or other wire-based network, a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network), input/output (I/O) interface <b>512</b> (e.g., keyboard, keypad, mouse, microphone). In particular embodiments, computer system <b>500</b> may include one or more of any such components.

0069In particular embodiments, processor <b>504</b> includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor <b>504</b> may retrieve (or fetch) the instructions from an internal register, an internal cache, memory <b>506</b>, static storage <b>508</b>, or dynamic storage <b>510</b>; decode and execute them; and then write one or more results to an internal register, an internal cache, memory <b>506</b>, static storage <b>508</b>, or dynamic storage <b>510</b>. In particular embodiments, processor <b>504</b> may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor <b>504</b> including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor <b>504</b> may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory <b>506</b>, static storage <b>508</b>, or dynamic storage <b>510</b>, and the instruction caches may speed up retrieval of those instructions by processor <b>504</b>. Data in the data caches may be copies of data in memory <b>506</b>, static storage <b>508</b>, or dynamic storage <b>510</b> for instructions executing at processor <b>504</b> to operate on; the results of previous instructions executed at processor <b>504</b> for access by subsequent instructions executing at processor <b>504</b> or for writing to memory <b>506</b>, static storage <b>508</b>, or dynamic storage <b>510</b>; or other suitable data. The data caches may speed up read or write operations by processor <b>504</b>. The TLBs may speed up virtual-address translation for processor <b>504</b>. In particular embodiments, processor <b>504</b> may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor <b>504</b> including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor <b>504</b> may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors <b>502</b>. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

0070In particular embodiments, I/O interface <b>512</b> includes hardware, software, or both, providing one or more interfaces for communication between computer system <b>500</b> and one or more I/O devices. Computer system <b>500</b> may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system <b>500</b>. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces <b>512</b> for them. Where appropriate, I/O interface <b>512</b> may include one or more device or software drivers enabling processor <b>504</b> to drive one or more of these I/O devices. I/O interface <b>512</b> may include one or more I/O interfaces <b>512</b>, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

0071In particular embodiments, communication interface <b>514</b> includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system <b>500</b> and one or more other computer systems <b>500</b> or one or more networks. As an example and not by way of limitation, communication interface <b>514</b> may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface <b>514</b> for it. As an example and not by way of limitation, computer system <b>500</b> may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system <b>500</b> may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system <b>500</b> may include any suitable communication interface <b>514</b> for any of these networks, where appropriate. Communication interface <b>514</b> may include one or more communication interfaces <b>514</b>, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

0072One or more memory buses (which may each include an address bus and a data bus) may couple processor <b>504</b> to memory <b>506</b>. Bus <b>502</b> may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor <b>504</b> and memory <b>506</b> and facilitate accesses to memory <b>506</b> requested by processor <b>504</b>. In particular embodiments, memory <b>506</b> includes random access memory (RAM). This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory <b>506</b> may include one or more memories <b>506</b>, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

0073Where appropriate, the ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. In particular embodiments, dynamic storage <b>510</b> may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Dynamic storage <b>510</b> may include removable or non-removable (or fixed) media, where appropriate. Dynamic storage <b>510</b> may be internal or external to computer system <b>500</b>, where appropriate. This disclosure contemplates mass dynamic storage <b>510</b> taking any suitable physical form. Dynamic storage <b>510</b> may include one or more storage control units facilitating communication between processor <b>504</b> and dynamic storage <b>510</b>, where appropriate.

0074In particular embodiments, bus <b>502</b> includes hardware, software, or both coupling components of computer system <b>500</b> to each other. As an example and not by way of limitation, bus <b>502</b> may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus <b>502</b> may include one or more buses <b>506</b>, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

0075According particular embodiments, computer system <b>500</b> performs specific operations by processor <b>504</b> executing one or more sequences of one or more instructions contained in memory <b>506</b>. Such instructions may be read into memory <b>506</b> from another computer readable/usable medium, such as static storage <b>508</b> or dynamic storage <b>510</b>. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement particular embodiments. Thus, particular embodiments are not limited to any specific combination of hardware circuitry and/or software. In particular embodiments, the term “logic” shall mean any combination of software or hardware that is used.

0076The term “computer readable medium” or “computer usable medium” as used herein refers to any medium that participates in providing instructions to processor <b>504</b> for execution. Such a medium may take many forms, including but not limited to, nonvolatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as static storage <b>508</b> or dynamic storage <b>510</b>. Volatile media includes dynamic memory, such as memory <b>506</b>.

0077Common forms of computer readable media include, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

0078In particular embodiments, execution of the sequences of instructions may be performed by a single computer system <b>500</b>; in alternative embodiments, two or more computer systems <b>500</b> coupled by communication link <b>516</b> (e.g., LAN, PTSN, or wireless network) may perform the sequence of instructions in coordination with one another.

0079Computer system <b>500</b> may transmit and receive messages, data, and instructions, including program, i.e., application code, through communication link <b>516</b> and communication interface <b>514</b>. Received program code may be executed by processor <b>504</b> as it is received, and/or stored in static storage <b>508</b> or dynamic storage <b>510</b>, or other non-volatile storage for later execution. A database <b>520</b> may be used to store data accessible by the system <b>500</b> by way of data interface <b>518</b>.

0080Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDS), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

0081It is also to be understood that in some embodiments, any of the operations described herein can be implemented at least in part as computer-readable instructions stored on a computer-readable memory. Upon execution of the computer-readable instructions by a processor, the computer-readable instructions can cause a node to perform the operations.

0082The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are merely exemplary, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable and/or physically interacting components and/or wirelessly interactable and/or wirelessly interacting components and/or logically interacting and/or logically interactable components.

0083With respect to the use of substantially any plural and/or singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations can be expressly set forth herein for sake of clarity.

0084It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.). It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims can contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” and/or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations). Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word and/or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.” Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent.

0085The foregoing description of illustrative embodiments has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or can be acquired from practice of the disclosed embodiments. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.