Maintainable grid managers
Abstract
A network of grid managers includes a first computer linked to a second computer, the first computer having a first grid manager and the second computer having a second grid manager, the first and second grid managers handling at least locating, reserving, allocating, monitoring, and deallocating one or more computational resources for an application, the grid manager, upon receipt of a command, loads new instructions to modify current instructions residing in the service that modifies a behavior of the service without restarting the service.

Term
Term ended
Expired 27 July 2024, 2.2 years ago.
- Priority
- Filed
- Granted
- Expired
- Today
6 claims: 3 independent, 3 dependent
- 1A network comprising:a first computer (12) linked to a second computer (14);the first computer (12) having a first service (152) and the second computer (14) having a second service (154), wherein the first service (152) is associated with the first computer (12) and the second service (154) is associated with the second computer (14);the first computer (12) having an application manager (118) configured to communicate with the first service (152) and the second service (154);each of the first and second services handling at least locating, reserving, allocating, monitoring, and deallocating one or more computational resources for an application (120, 122);each service (152, 154), upon receipt of a command, sent as a string by an Application Program Interface, loads new instructions to modify current instructions residing in the service (152, 154) that modifies a behavior of the service (152, 154) without restarting the service (152, 154);wherein the first service (152) and the second service (154) are each configured to: modify a relationship between the first service (152) and the second service (154) upon receipt of the command;receive the application (120, 122) and information about installing and customizing the application (120, 122);install, customize, and execute the application (120, 122);log all changes made to the associated computer (12, 14) during the installation, customization, and execution of the application (120, 122);stop the application (120, 122);deinstall the application (120, 122) after waiting a prespecified amount of time and after the application manager (118) has made a determination that a current resource utilization does not indicate a need to restart the application (120, 122);and revert the associated computer (12, 14) to an original state by removing the application (120, 122) and removing, based on the logged changes, modifications made to the associated computer (12,14) during the installation, customization and execution of the application (120, 122).
- 3A method comprising:sending a command as a string by an Application Program Interface;receiving a message having the command for a first service (152) that handles locating, reserving, allocating, monitoring, and deallocating one or more computational resources for an application (120, 122) running on a first computer (12) in a network (8);loading a first set of instructions from a location remote from the first service (152) in response to the command;replacing a portion of instructions for the first service (152) with the first set of instructions;executing the first service (152) according to the first set of instructions;modifying a relationship between the first service (152) and a second service (154) in response to the command, wherein the second service (154) comprises locating, reserving, allocating, monitoring, and deallocating one or more computational resources for the application (120, 122) running on a second computer (14) in the network;receiving, by the first service (152), the application (120, 122) and information about installing and customizing the application (120, 122);installing, customizing, and executing the application (120, 122);logging all changes made to the first computer (12) during the installation, customization, and execution of the application (120, 122);stopping the application (120, 122);deinstalling the application (120, 122) after waiting a prespecified amount of time and after an application manager (118) has made a determination that a current resource utilization does not indicate a need to restart the application (120, 122), wherein the application manager (118) is on the first computer and is configured to communicate with the first service (152) and the second service (154);and reverting the first computer (12) to an original state by: removing the application (120, 122), and removing, based on the logged changes, modifications made to the first computer (12) during the installation, customization and execution of the application (120, 122).
- 5A computer program product, the computer program product being operable to cause a data processing apparatus to:send a command as a string by an Application Program Interlace;receive a message having the command for a first service (152) comprising locating, reserving, allocating, monitoring, and deallocating one or more computational resources for an application (120, 122) running on a computer (12) in a network (8);load a first set of instructions from a location remote from the first service (152) in response to the command;replace a portion of instructions for the first service (152) with the first set of instructions;execute the first service (152) according to the first set of instructions;modify a relationship between the first service (152) and a second service (154) in response to the command, wherein the second service (154) comprises locating, reserving, allocating, monitoring, and deallocating one or more computational resources for the application (120, 122) running on a second computer (14) in the network;receive the application (120, 122) and information about installing and customizing the application (120, 122);install, customize, and execute the application (120, 122);log all changes made during the installation, customization, and execution of the application (120, 122);stop the application (120, 122);deinstall the application (120, 122) after waiting a prespecified amount of time and after an application manager (118) has made a determination that a current resource utilization does not indicate a need to restart the application (120, 122), wherein the application manager (118) is on the first computer and is configured to communicate with the first service (152) and the second service (154);and revert to an original state by removing the application (120, 122), and removing, based on the logged changes, modifications made during the installation, customization and execution of the application (120, 122).
Independent claims3
56 paragraphs in 5 sections, as filed
TECHNICAL FIELD
0001The present invention relates to data processing by digital computer, and more particularly to application process management for grid computing.
BACKGROUND
0002In today's data centers, the clusters of servers in a client-server network that run business applications often do a poor job of managing unpredictable workloads. One server may sit idle, while another is constrained. This leads to a "Catch-22" where companies, needing to avoid network bottlenecks and safeguard connectivity with customers, business partners and employees, often plan for the highest spikes in workload demand, then watch as those surplus servers operate well under capacity most of the time.
0003In grid computing, all of the disparate computers and systems in an organization or among organizations become one large, integrated computing system. That single integrated system can then handle problems and processes too large and intensive for any single computer to easily handle in an efficient manner.
0004More specifically, grid computing is a form of distributed system wherein computing resources are shared across networks. Grid computing enables the selection, aggregation, and sharing of information resources resident in multiple administrative domains and across geographic areas. These information resources are shared, for example, based upon their availability, capability, and cost, as well as a user's quality of service (QoS) requirements. Grid computing can mean reduced cost of ownership, aggregated and improved efficiency of computing, data, and storage resources, and enablement of virtual organizations for applications and data sharing.
0005The document authored by I. Foster et al., entitled "The Physiology of the Grid", discloses a well-defined set of basic interfaces from which can be constructed systems to align Grid technologies with Web services technologies. The architecture represents a Grid service as a Web service that conforms to specific conventions and provides a set of well-defined interfaces for such purposes as lifetime management, notification and manageability. Each interface provided by a Grid service defines a set of operations that are invoked by exchanging a defined sequence of messages.
0006The document authored by I. Foster et al., entitled The Anatomy of the Grid", discloses an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling resource sharing. The document describes requirements that such mechanisms must satisfy and the central role played by the intergrid protocols that enable interoperability among different Grid systems.
0007The document authored by K. Czajkowski et al., entitled " Grid Information Services for Distributed Resource Sharing", discloses an information services architecture that addresses performance, security, scalability, and robustness requirements. The architecture defines low-level enquiry and registration protocols that make it easy to incorporate individual entities into information structures, including aggregate directories supporting different query languages and discovery strategies.
0008Thus, according to an aspect, it is a problem to improve the management of computational resources for an application in a network. This problem is solved by the network according to claim 1, the method according to claim 3, and the computer program product according to claim 5. Preferred embodiments are defined in the dependent claims.
SUMMARY
0009In one aspect, the invention features a network that includes a first computer linked to a second computer, the first computer having a first service and the second computer having a second service, the first and second services handling at least locating, reserving, allocating, monitoring, and deallocating one or more computational resources for an application, the service, upon receipt of a command, sent as a string by an Application Program Interface, loads new instructions to modify current instructions residing in the service that modifies a behavior of the service without restarting the service.
0010Embodiments may include one or more of the following. The command causes a modification of a behavior of the application without restarting the application.
0011The invention features a method that includes sending a command as a string by an Application Program Interface, and receiving a message having a command for a service that handles locating, reserving, allocating, monitoring, and deallocating one or more computational resources for an application running on a computer in a network. The method also includes loading a first set of instructions from a location remote from the service in response to the command, replacing a portion of instructions for the service with the first set of instructions, and executing the service according to the first set of instructions.
0012The method may further include replacing a portion of instructions for the application with the first set of instructions and executing the application according to the first set of instructions. The method further includes modifying a relationship between the service and a second service in response to the command, wherein the second service comprises locating, reserving, allocating, monitoring, and deallocating one or more computational resources for an application running on a computer in the network.
0013These and other embodiments may have one or more of the following advantages. A network for grid computing can be extended and maintained with less effort by using grid managers.
0014The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.
DESCRIPTION OF DRAWINGS
0015<ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">FIG 1</figref> is a block diagram of a grid computing environment.</li><li><figref idref="f0002">FIG. 2</figref> is a flow diagram for discovering and reserving resources in the grid computing environment of <figref idref="f0001">FIG. 1</figref>.</li><li><figref idref="f0003">FIG 3</figref> is a flow diagram for installing, running, and removing applications in the grid computing environment of <figref idref="f0001">FIG. 1</figref>.</li><li><figref idref="f0004">FIG 4</figref> is a block diagram of a computer device in the grid computing environment of <figref idref="f0001">FIG. 1</figref>.</li><li><figref idref="f0004">FIG 4A</figref> is a flow diagram for starting up an application in the computer device of <figref idref="f0004">FIG 4</figref>.</li><li><figref idref="f0005">FIG 5</figref> is a flow diagram for starting up grid managers in the grid computing environment of <figref idref="f0001">FIG. 1</figref>.</li><li><figref idref="f0006">FIG 5A</figref> is a block diagram of the grid computing environment of <figref idref="f0001">FIG 1</figref> that is augmented with another computer device.</li><li><figref idref="f0007">FIG 6</figref> is a block diagram of an exemplary a grid graphical user interface (GUI) component for visualization of a grid computing environment.</li><li><figref idref="f0008">FIG 7</figref> is a block diagram of a grid browser component.</li></ul>
0016Like reference symbols in the various drawings indicate like elements.
DETAILED DESCRIPTION
0017As shown in <figref idref="f0001">FIG. 1</figref>, services in a grid computing environment <b>100</b> manage computational resources for applications. The grid computing environment <b>100</b> is a set of distributed computing resources that can individually be assigned to perform computing or data retrieval tasks for the applications. The computational resources include computer devices <b>12, 14, 16, 18, 20,</b> and <b>22.</b> The computer devices communicate using a network <b>8.</b> The applications have scalable computational requirements. For example, an example application that uses computer devices <b>12, 14, 16, 18, 20,</b> and <b>22</b> in the grid computing environment <b>100</b> is an internet pricing configurator. The computer device <b>12</b> provides network access to pricing information to users via web browsers on computer devices that are connected to the internet. The web browsers can be any application able to display content and/or execute applications such as web pages, media files, and programs, such as Netscape Navigator®, Microsoft Internet Explorer®, and similar applications.
0018In this example, a web server on computer device <b>12</b> provides pricing information to the users. Calculation parameters for each price to be calculated are passed by an IPC dispatcher <b>116</b> to IPC servers <b>120, 122, 124,</b> and <b>126</b> that execute on computer devices <b>12, 14,16,</b> and <b>18,</b> respectively. Due to the flexibility of the web server and applications on the internet, the number of users can vary. This generates dynamic computational requirements for the internet pricing configurator. An IPC manager <b>118</b> communicates with services in the grid computing environment <b>100</b> so that the services can allocate and deallocate computational resources (e.g., processors in computer devices <b>12, 14, 16, 18, 20, 22</b>) based on the dynamic computational requirements of the internet pricing configurator. Allocating and deallocating computational resources in this manner allows computer devices <b>12, 14, 16, 18, 20,</b> or <b>22</b> to be designated as general-purpose computational resources and not solely dedicated to handling peak demands of the internet pricing configurator application. The IPC manager <b>118</b> coordinates with the IPC dispatcher <b>116</b> so that the IPC dispatcher <b>116</b> has access to resources in network <b>8.</b>
0019This capability to allocate and deallocate the resources in the grid computing environment <b>100</b> enables the IPC manager <b>118</b> to locate and use available computational resources on an "as needed" basis. Once resources are located, the IPC manager <b>118</b> can use services in the grid computing environment <b>100</b> to install the IPC servers <b>120, 122, 124,</b> and <b>126</b> as applications on computer devices in the grid computing environment <b>100.</b> The IPC dispatcher <b>116</b> uses Web Service Definition Language (WSDL) interfaces defined in the Open Grid Services Infrastructure (OGSI) Version 1.0 by Tuecke <i>et al</i> to manage and exchange the information flow between the IPC dispatcher <b>116</b> and IPC servers <b>120, 122, 124,</b> and <b>126.</b> For example, the OGSI WSDL interfaces can be used to pass computation parameters for pricing calculations from the IPC dispatcher <b>116</b> and the IPC servers <b>120, 122, 124</b>, and <b>126.</b> The OGSI WSDL interfaces can also be used to pass completed results from the IPC servers <b>120, 122, 124,</b> and <b>126</b> back to IPC, dispatcher <b>116.</b> The OGSI WSDL interfaces enable the controlled, fault-resilient, and secure management of the grid computing environment <b>100</b> and applications such as the internet pricing configurator.
0020While the IPC dispatcher <b>116</b> uses IPC servers <b>120, 122, 124</b>, and <b>126</b> to perform calculations for users, services in the grid computing environment <b>100</b> monitor resource utilization on computer devices in the grid computing environment <b>100</b> running the IPC servers <b>120, 122, 124,</b> and <b>126.</b> The services also send this utilization information to the IPC manager <b>118</b>. Based on a comparison between utilization requirements and current resource loading, the IPC manager <b>118</b> can dynamically inform services in the grid computing environment <b>100</b> to allocate more resources for IPC servers <b>120, 122, 124,</b> and <b>126</b> or deallocate resources to keep utilization of resources in the grid computing environment <b>100</b> at a desired level. Grid managers <b>152, 154, 156, 160, 162,</b> and <b>164</b> are resident in computer devices <b>12, 14, 16, 18, 20,</b> and <b>22,</b> respectively. Within the grid computing environment 100, pairs of grid managers can have directional relations that classify one grid manager as superior to another grid manager. A grid manager can have more than one superior relations with other grid managers. For example, grid manager <b>152</b> has a superior relation with grid managers <b>154</b> and <b>156.</b> A grid manager can also have more than one inferior relations with other grid managers. Through these hierarchical relations, IPC manager <b>118</b> does not need access to a list of all computer devices in network <b>8</b> to use the computational resources in the grid computing environment <b>100.</b> IPC manager <b>118</b> is only required to have access to a network address of one computer device running a grid manager (e.g., computer device <b>12</b> running grid manager <b>152)</b> and this grid manager uses its relations with other grid managers running on other computer devices to provide IPC dispatcher <b>116</b> with indirect access to other computer devices in the grid computing environment <b>100.</b>
0021A grid manager (e.g., <b>152, 154, 156, 160, 162,</b> and <b>164</b>) maintains a first list of all superior relations with other grid managers and a second list of all inferior relations with other grid managers. Each grid manager maintains an "always open" communications channel to all the grid managers in these lists over network 8 using, for example, the aforementioned OGSI WSDL interfaces on transmission control protocol (TCP), hypertext transfer protocol (HTTP), and simple object access protocol (SOAP). These lists and corresponding communication channels can be modified, allowing a dynamic reconfiguration of the grid hierarchy during runtime. This also allows a failing grid manager to be dynamically replaced in the hierarchy. For example, referring to <figref idref="f0001">FIG. 1</figref>, if grid manager <b>154</b> fails, then grid manager <b>152</b> loses its connection to grid managers <b>160</b> and <b>162.</b> In this case, relations between grid managers can be modified so that grid manager <b>152</b> has new superior relations to grid managers <b>160</b> and <b>162.</b> Likewise, grid managers <b>160</b> and <b>162</b> have new inferior relations to grid manager <b>152.</b>
0022As shown in <figref idref="f0002">FIG. 2</figref>, an application start process <b>200</b> is designed so applications (e.g., internet pricing configurator) get necessary resources allocated in the network <b>8</b> before executing on a computer device (e.g., <b>12, 14, 16, 18, 20,</b> or <b>22</b>). Process <b>200</b> also guarantees if similar applications are trying to start at the same time on the same resource on a computer device that the two or more applications do not collide or interfere with each other. For example, the IPC manager <b>118</b> can require that an IPC server (e.g., <b>120</b>) be the only application executing on a processor in computer device <b>14</b> for quality of service (QoS). In this case, another application would interfere if the other application simultaneously attempted to execute on the processor in computer device <b>14.</b>
0023Process <b>200</b> includes IPC manager <b>118</b> (or some other application) sending <b>(202)</b> requirements for computational resources to query a grid manager (e.g., <b>154</b>) to determine if there are resources matching these requirements available in the grid computing environment <b>100.</b> These requirements specify information pertaining to resources in a computer device such as required number of processors, required percentage of utilization for those processors, main memory, and network speed. The query can also include information to which hierarchy level (in the grid computing environment <b>100</b>) the query should be propagated. Process <b>200</b> includes grid manager <b>154</b> receiving <b>(204)</b> the requirements.
0024To respond to the query for available resources from IPC manager <b>118,</b> process <b>200</b> includes grid manager <b>154</b> matching <b>(206)</b> the requirements against resources known to grid manager <b>154.</b> These resources include resources (e.g., processor <b>40</b>) in computer device <b>14</b> that are directly managed by grid manager <b>154.</b> Resources directly managed by grid manager <b>154</b> that are currently available and meet the requirements are added to a resource-query list maintained by grid manager <b>154.</b>
0025Grid manager <b>154</b> also sends the query to grid managers <b>160</b> and <b>162</b> having inferior relations with grid manager <b>154.</b> Process <b>200</b> includes grid managers <b>160</b> and <b>162</b> responding <b>(208)</b> to the query by sending to grid manager <b>154</b> lists of resources (e.g., processors on computer devices <b>18, 20)</b> that meet the requested requirements and are available and known to grid managers <b>160</b> and <b>162,</b> respectively. These resource-query lists of resources that are known to grid managers <b>160</b> and <b>162</b> can also include resources managed by grid managers (not shown) with inferior relations to grid managers <b>160</b> and <b>162.</b> Grid manager <b>154</b> adds these resource-query lists of available resources from grid managers <b>160</b> and <b>162</b> to its resource-query list of available resources meeting the requested requirements. If process <b>200</b> determines <b>(210)</b> that there is at least one resource (e.g., processor <b>40</b>) in this resource-query list, then grid manager <b>154</b> sends <b>(214)</b> this resource-query list to IPC manager <b>118.</b> Otherwise, if process <b>200</b> determines <b>(212)</b> that grid manager <b>154</b> has a relation with a superior grid manager (e.g., grid manager <b>152</b>), grid manager <b>154</b> sends <b>(202)</b> the query for available resources to grid manager <b>152.</b> In response to this query, grid manager <b>152</b> does not send a redundant query back to grid manager <b>154</b> having an inferior relation with grid manager <b>152.</b>
0026Process <b>200</b> includes grid manager <b>154</b> sending <b>(214)</b> the list of available resources along with addresses of their corresponding grid managers in the network <b>8</b> that match the requirements. The IPC manager <b>118</b> selects a resource (e.g., on computer device <b>16</b>) from the list and requests <b>(216)</b> a reservation of the resource on computer device <b>16</b> to the grid manager <b>154</b> managing the resource on computer device <b>16.</b> If the resource in computer device <b>16</b> is still available for reservation <b>(218)</b> and the reservation succeeds, grid manager <b>154</b> sends <b>(220)</b> a reservation number to the IPC manager <b>118.</b> This reservation means that the IPC manager <b>118</b> is guaranteed and allocated the requested resource on the computer device <b>16</b> in the grid computing environment <b>100.</b> The grid manager <b>154</b> handles queries for available resources from applications such as IPC manager <b>118</b> using independent processing threads of execution. Thus, the grid manager <b>154</b> uses a semaphore to ensure that the same resource (e.g., processor <b>40</b>) is not assigned multiple reservation numbers for different applications simultaneously requesting the same resource.
0027If the grid manager determines that the requested resource in computer device <b>16</b> is not available for reservation and the reservation fails, the IPC manager <b>118</b> selects the next available resource in the list and requests <b>(216)</b> the reservation of this next available resource. If the IPC manager <b>118</b> receives a registration number and a timeout measured from the sending of the registration number does not expire <b>(222),</b> the IPC manager <b>118</b> starts <b>(224)</b> the IPC server <b>122</b> on the processor <b>40</b> resource in computer device <b>16.</b> Starting the IPC server <b>122</b> is initiated by passing the reservation number and an application file to the grid manager <b>156</b> and then grid manager <b>156</b> reads the application file to install and execute the IPC server <b>122</b> on computer device <b>16.</b>
0028As shown in <figref idref="f0003">FIG. 3</figref>, process <b>250</b> installs an application (e.g., IPC server <b>122</b>) on a computer device (e.g., <b>14</b>) in the grid computing environment <b>100</b> to set up an available resource for the application, using the available resource, and removing or deinstalling the application to free up the resource for use by subsequent applications when the resource is no longer needed. Process <b>250</b> includes IPC manager <b>118</b> transferring <b>(252)</b> an application file containing code for IPC server <b>122</b> in addition to instructions on how to install, customize, track and remove the application from computer device <b>14</b> so that the grid manager <b>154</b> can return computer device <b>14</b> to an original state after executing the application.
0029IPC manager <b>118</b> transfers the application file using a file transfer protocol (FTP), hypertext transfer protocol (HTTP), or a file copy from a network attached storage (NAS) for example, to computer device <b>14</b> as a single file, such as a compressed zip file. Within this zip file there is information about installing and customizing the application IPC server <b>122.</b> This information is represented by a small executable program or extended markup language (XIVIL) document that is extracted and interpreted <b>(254)</b> by an installation and customizing engine (not shown) in grid manager <b>154.</b> Process <b>250</b> includes grid manager <b>154</b> installing <b>(256)</b> and running <b>(258)</b> the application. During installation <b>(256),</b> customization and execution <b>(258)</b> of the application, all changes to the computer device <b>14</b> are logged so that when the application is terminated <b>(260)</b> or deinstalled by grid manager <b>154</b> upon request by IPC manager <b>118,</b> grid manager <b>154</b> removes the application from the computer device <b>14</b> and also removes <b>(262)</b> any other changes to computer device <b>14</b> that were done when installing and running the application. Thus, the computer device <b>14</b> reverts to its original state prior to execution of the application and all of the resources of computer device <b>14</b> are again available for use by a subsequent application. This allows the resources to become available after running the application without rebooting computer device <b>14.</b> These changes include space in memory (e.g., <b>32</b>) allocated to store and run application code in addition to other changes such as allocation of communication ports.
0030In some examples, multiple applications can simultaneously run on resources in a single computer device (e.g., <b>14</b>). Applications for the grid computing environment <b>100</b> are classified in part based on their resource requirements. Some changes to a computer device to run an application are only required for the first execution of an application of its class and subsequent executions do not require these changes. In these examples, grid manager <b>154</b> only does the changes for the first execution. Furthermore, when deinstalling the applications, grid manager <b>154</b> only removes the changes for the last application that was executed and terminated.
0031After installing applications on computer devices in the grid computing environment <b>100,</b> grid managers are configured to start or stop the processes of these applications upon request. In the example of the internet pricing configurator (IPC) application, grid manager <b>154</b> is configured to start or stop IPC server <b>122</b> on computer device <b>14</b> after installing IPC server <b>122</b> on computer device <b>14.</b> The IPC manager <b>118</b> requests grid managers to start or stop IPC servers in the grid computing environment <b>100</b> based on current utilization of resources in the grid computing environment <b>100.</b> After stopping IPC server <b>122</b> on computer device <b>14,</b> IPC manager <b>118</b> waits a prespecified amount of time and then requests grid manager <b>154</b> to deinstall IPC server <b>122</b> if current resource utilization does not indicate a need to start IPC server <b>122</b> again. Furthermore, as mentioned previously, grid managers monitor resource utilization on computer devices such as computer device <b>14</b> running applications (e.g. IPC servers <b>120, 122, 124,</b> and <b>126</b>) and send this utilization information to IPC manager <b>118.</b>
0032In many examples, control of application processes on resources in a computer device is specific to the operating system (OS). The grid computing environment <b>100</b> is configured to handle different operating systems on computer devices. Furthermore, grid computing environment <b>100</b> is designed to handle different applications (e.g., internet pricing configurator) that do not have to be redesigned to execute on the grid computing environment <b>100.</b> A grid manager controls an application process in a general manner that decreases interdependence between development of grid manager code and application code. An interface is provided to application code to enable grid managers to discover, control (e.g., start, stop, halt, resume) and inspect or monitor a state of application processes. The interface is provided for operating system processes that are exposed by the operating system or hosting environment and includes three aspects. One aspect of the interface is process data, such as process identification, states, degree of resource consumption (such as Central Processing Unit (CPU), memory, socket bindings, or other resources that an application can use), and application specific data defined by a process data scheme.
0033A second aspect of the interface is managing operations, such as start, stop, wait, resume, change priority, and other operations defined by supported managing operations.
0034A third aspect of the interface is control bindings and definitions, such as process data scheme, supported managing operations, and communication bindings. Since not all applications running in the grid computing environment <b>100</b> have access to the same information and capabilities in these three aspects, the applications provide to grid managers a list of queries and commands that each application supports.
0035The interface provided to application code is an Application Program Interface (API). The API is a set of methods (embedded in software code) prescribed by the grid manager software by which a programmer writing an application program (e.g., internet pricing configurator) can handle requests from the grid manager.
0036As shown in <figref idref="f0004">FIG. 4</figref>, IPC server <b>122</b> includes an API <b>302</b> and a document <b>304.</b> Since the API <b>302</b> is adapted to different types of applications, the document <b>304</b> describes how grid manager <b>154</b> communicates with the IPC server <b>122</b> and what requests through the API <b>302</b> are supported by the IPC server <b>122.</b> Grid manager <b>154</b> reads document <b>304</b> before starting up IPC server <b>122.</b> In some examples, document <b>304</b> is written in XML and includes a Document Type Description (DTD) <b>306.</b> A DTD is a specific definition that follows the rules of the Standard Generalized Markup Language (SGML). A DTD is a specification that accompanies a document and identifies what the markups are that separate paragraphs, identify topic headings, and how each markup is to be processed. By including the DTD <b>306</b> with document <b>304,</b> grid manager <b>154</b> having a DTD "reader" (or "SGML compiler") is able to process the document <b>304</b> and can correctly interpret many different kinds of documents <b>304</b> that use a range of different markup codes and related meanings.
0037As shown in <figref idref="f0004">FIG. 4A</figref>, grid manager <b>154</b> uses process <b>350</b> to install applications such as IPC server <b>122.</b> Grid manager <b>154</b> reads <b>(352)</b> DTD <b>306</b> in document <b>304</b> to identify markups in document <b>304.</b> Grid manager <b>154</b> reads <b>(354)</b> document <b>304</b> using markups to identify communication parameters for communicating with IPC server <b>122.</b> Grid manager <b>154</b> sets up <b>(356)</b> communications with IPC server <b>122</b> based on the specifications of the communication parameters. Grid manager <b>154</b> communicates <b>(358)</b> with IPC server <b>122</b> using the communication parameters to send requests such as "Start", "Stop", and "Are you idle?".
0038Before any applications (e.g., internet pricing configurator) can be executed on network 8, grid managers <b>152, 154, 156, 160, 162,</b> and <b>164</b> are asynchronously started up on computer devices <b>12, 14, 16, 18, 20,</b> and <b>22,</b> and relations to other grid managers are established. As shown in <figref idref="f0005">FIG. 5</figref>, process <b>400</b> initializes relations among grid managers. For each grid manager (e.g., grid manager <b>154</b>), the grid manager <b>154</b> starts up on computer device <b>14</b> by reading <b>(402)</b> a properties file. The properties file contains a list of addresses of computer devices with grid managers having superior relations to grid manager <b>154.</b> This list was described earlier as a first list of all superior relations with other grid managers. If <b>(404)</b> a superior grid manager (e.g., grid manager <b>152</b>) is specified in this list of addresses, grid manager <b>154</b> requests <b>(406)</b> to open a communication channel to the superior grid manager (e.g., <b>152</b>). If grid manager <b>152</b> is already started, then grid manager <b>152</b> responds by accepting the request of the opening of the communication channel from grid manager <b>152.</b> Process <b>400</b> includes grid manager <b>154</b> detecting <b>(408)</b> any requests for communication channels from grid managers (e.g., grid managers <b>160, 162</b>) identified as having inferior relations with grid manager <b>154.</b> If process 400 determines <b>(410)</b> that there are some requests, grid manager 154 allows communication channels from the inferior grid managers (e.g., <b>160, 162</b>). Process <b>400</b> includes grid manager <b>154</b> checking <b>(414)</b> if there are any pending requests for communication to grid managers having superior relations. If there are any pending requests, grid manager <b>154</b> requests <b>(406)</b> communication channels to grid managers. These communication channels are used for resource queries between grid managers (as described previously) and "heart beat" messages between grid managers to ensure that each grid manager in the grid computing environment <b>100</b> is functioning.
0039Once grid managers <b>152, 154, 156, 160, 162,</b> and <b>164</b> are running with established relations, the grid managers are used for the proper operation of the grid computing environment <b>100.</b> Often during the lifecycle of the grid computing environment <b>100</b> the functionality of the grid managers are enhanced. It is often not possible or convenient to shut down the grid computing environment <b>100</b> and start the grid computing environment <b>100</b> up with the enhancements. Grid managers <b>152, 154, 156, 160, 162,</b> and <b>164</b> are configured so that there is only a minimal impact on users of the grid computing environment <b>100</b> when a change happens. To enable this transparency, an API is provided for user interfaces to enable an administrator of grid computing environment <b>100</b> to access each of the grid managers <b>152, 154, 156, 160, 162,</b> and <b>164</b> individually or all together. The API is static in that it includes only one method, i.e., a string that contains a command typed by the administrator. The API is dynamic because the string can contain many different commands.
0040In some cases, the grid managers are developed using the Java programming language. In these cases, new commands issued to the grid managers can be supported by loading new or revised Java classes dynamically via classloaders. This dynamic access to code can be done without shutting down grid managers in the grid computing environment 100. Using Java classloaders, each time an instance of a class for a grid manager is generated, the definition and behavior of the class can be updated to provide new functionality to the grid computing environment <b>100.</b>
0041Another way to modify the functionality of the grid computing environment <b>100</b> dynamically without shutting down the grid computing environment <b>100</b> is to change the hierarchical relations between grid managers, remove grid managers, or add new grid managers. The API provided for administration of the grid computing environment <b>100</b> is also configured to send strings to individual grid managers with commands to delete existing relations or add new relations.
0042For administrators of grid computing environment <b>100,</b> it is useful to visualize the applications and a grid manager on one computer device in the grid computing environment <b>100</b> as well as other computer devices running part of the grid management hierarchy in the form of grid managers with one or more levels of inferior relations to the grid manager. The view of these computer devices is referred to as a grid landscape. As shown in <figref idref="f0007">FIG. 6</figref>, a grid graphical user interface (GUI) <b>500</b> for visualization of a grid landscape, such as the grid computing environment <b>100,</b> includes GUI-elements visualizing an organization of services running on computer devices. The GUI <b>500</b> provides a grid-like structure with columns and rows. Rows represent services, which in turn are structured hierarchically with respect to the application where a service belongs to, the type of the service, and the specific service instances. Each service instance row is associated with a place in the grid computing environment <b>100</b> representing where it is instantiated. In this context, columns represent the computer devices in the grid landscape. Specifically, GUI <b>500</b> has three columns representing three computer devices <b>12, 14,</b> and <b>16.</b> GUI <b>500</b> shows that grid manager <b>152</b> runs on computer device <b>12</b> with inferior grid managers <b>154</b> and <b>156</b> running on computer devices <b>14</b> and <b>16,</b> respectively. GUI <b>500</b> also shows internet pricing configurator services running on computer device <b>12.</b> These internet pricing configurator services include IPC dispatcher <b>116,</b> IPC server <b>120,</b> and IPC manager <b>118.</b>
0043The GUI <b>500</b> is dynamically refreshed with feedback from the grid managers and internet pricing configurator (or other application) services so that new services appear in GUI <b>500</b> to an administrator. Similarly, services that are shut down are removed in GUI <b>500.</b>
0044As shown in <figref idref="f0008">FIG. 7</figref>, a grid browser component <b>600</b> is a composite graphical user interface (GUI) for browsing grid managers on computer devices in the grid computing environment <b>100.</b> The component <b>600</b> displays a graph with curved edges and vertices. Vertices represent computer devices in the grid computing environment <b>100</b> and curved edges represent the directional association of grid managers on two computer devices (vertices) in the grid computing environment <b>100.</b> This association is hierarchical (i.e., superior/inferior). Each vertex displays the network address of a computer device as well as applications currently running on the computer device. For example, component <b>600</b> shows computer devices <b>12, 14, 16, 18, 20,</b> and <b>22</b> with IPC servers <b>118, 120, 122,</b> and <b>124.</b> In other examples (not shown), the grid browser component <b>600</b> shows non-hierarchical, peer to peer associations of grid managers with non-directional edges representing the associations.
0045The grid browser component <b>600</b> is context sensitive. Depending on the relationship among the grid managers on the computer devices (e.g., superior/inferior), computer devices are traversed in respect to a user's browsing history.
0046By clicking on a vertex representing a computer device in GUI <b>600</b> (e.g., computer device <b>14</b>), a user can automatically view a grid manager and applications running on the computer device and grid managers having inferior relations to the grid manager using GUI <b>500.</b> The user can pick a computer device and see relations between its grid manager and other grid managers. This connection between GUIs <b>500</b> and <b>600</b> is done using software that generates GUIs <b>500</b> and <b>600.</b>
0047The network <b>8</b> can be implemented in a variety of ways. The network <b>8</b> includes any kind and any combination of networks such as an Internet, a local area network (LAN) or other local network, a private network, a public network, a plain old telephone system (POTS), or other similar wired or wireless networks. Communications through the network <b>8</b> may be secured with a mechanism such as encryption, a security protocol, or other type of similar mechanism. Communications through the network 8 can include any kind and any combination of communication links such as modem links, Ethernet links, cables, point-to-point links, infrared connections, fiber optic links, wireless links, cellular links, Bluetooth®, satellite links, and other similar links.
0048The network <b>8</b> is simplified for ease of explanation. The network <b>8</b> can include more or fewer additional elements such as networks, communication links, proxy servers, firewalls or other security mechanisms, Internet Service Providers (ISPs), gatekeepers, gateways, switches, routers, hubs, client terminals, and other elements.
0049Computer devices <b>12, 14, 16, 18, 20,</b> and <b>22</b> communicate over medium <b>10</b> using one of many different networking protocols. For instance, one protocol is Transmission Control Protocol/Internet Protocol (TCP/IP) combined with SOAP (Simple Object Access Protocol).
0050Embodiments of the invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Embodiment of the invention can be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a node-readable storage device or in a propagated signal, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
0051Method steps of embodiments of the invention can be performed by one or more programmable processors executing a computer program to perform functions of the invention by operating on input data and generating output. Method steps can also be performed by, and apparatus of the invention can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
0052Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.
0053To provide for interaction with a user, embodiments of the invention can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
0054Embodiments of the invention can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of embodiments of the invention, or any combination of such back-end, middleware, or front-end components, The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), e.g., the Internet.
0055The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
0056A number of embodiments of the invention have been described. Other embodiments are within the scope of the following claims.
Contents5
8 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| WO0113227A | Cites | World Intellectual Property Organization (WIPO) |
| US2003033369A1 | Cites | United States of America |
| I. FOSTER, C. KESSELMAN, J.M. NICK, S. TUECKE: "The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration" 22 June 2002 (2002-06-22), , XP002308622 Retrieved from the Internet: URL:http://www.globus.org/research/papers/ ogsa.pdf> [retrieved on 2004-11-25] page 7, paragraph 3.1 - page 8 page 10 - page 12 page 17 - page 19 page 21, paragraph 6.4 - page 22 page 10 - page 12 | Non-patent | – |
| ALLEN G ET AL: "The Cactus Worm: experiments with dynamic resource discovery and allocation in a grid environment" INTERNATIONAL JOURNAL OF HIGH PERFORMANCE COMPUTING APPLICATIONS SAGE SCIENCE PRESS USA, vol. 15, no. 4, 2001, pages 345-358, XP002308620 ISSN: 1078-3482 | Non-patent | – |
| BERMAN F ET AL: "The GrADS project: software support for high-level grid application development" INTERNATIONAL JOURNAL OF HIGH PERFORMANCE COMPUTING APPLICATIONS SAGE SCIENCE PRESS USA, vol. 15, no. 4, 31 July 2001 (2001-07-31), pages 327-344, XP002308621 ISSN: 1078-3482 | Non-patent | – |
62 members in 7 offices
Priority claims5
| Document | Office | Kind | Date |
|---|---|---|---|
| 490818P | United States of America | – | |
| 49081803 | United States of America | P | |
| 706066 | United States of America | – | |
| 70606603 | United States of America | A | |
| 2004008406 | European Patent Office (EPO) | W |
Members62
| Document | Office | Kind | |
|---|---|---|---|
| US2005027785A1 | United States of America | A1 | |
| US2005027812A1 | United States of America | A1 | |
| US2005027813A1 | United States of America | A1 | |
| US2005027843A1 | United States of America | A1 | |
| US2005027864A1 | United States of America | A1 | |
| US2005027865A1 | United States of America | A1 | |
| WO2005015392A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2005015393A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2005015394A2 | World Intellectual Property Organization (WIPO) | A2 | |
| WO2005015395A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2005015396A2 | World Intellectual Property Organization (WIPO) | A2 | |
| WO2005015397A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2005015398A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US2005044251A1 | United States of America | A1 | |
| WO2005015396A3 | World Intellectual Property Organization (WIPO) | A3 | |
| WO2005015394A3 | World Intellectual Property Organization (WIPO) | A3 | |
| EP1649364A1 | European Patent Office (EPO) | A1 | |
| EP1649365A2 | European Patent Office (EPO) | A2 | |
| EP1649366A1 | European Patent Office (EPO) | A1 | |
| EP1649367A2 | European Patent Office (EPO) | A2 | |
| EP1649368A1 | European Patent Office (EPO) | A1 | |
| EP1649369A1 | European Patent Office (EPO) | A1 | |
| EP1649370A1 | European Patent Office (EPO) | A1 | |
| CN1829963A | China | A | |
| CN1829965A | China | A | |
| CN1829966A | China | A | |
| CN1829967A | China | A | |
| CN1829968A | China | A | |
| CN1829969A | China | A | |
| CN1829970A | China | A | |
| JP2007500382A | Japan | A | |
| JP2007500383A | Japan | A | |
| JP2007500384A | Japan | A | |
| JP2007500385A | Japan | A | |
| JP2007500386A | Japan | A | |
| JP2007500387A | Japan | A | |
| JP2007500888A | Japan | A | |
| CN100378669C | China | C | |
| CN100422937C | China | C | |
| CN100422938C | China | C | |
| CN100440152C | China | C | |
| CN100472455C | China | C | |
| US2009083425A1 | United States of America | A1 | |
| US7546553B2 | United States of America | B2 | |
| US7568199B2 | United States of America | B2 | |
| US7574707B2 | United States of America | B2 | |
| US7594015B2 | United States of America | B2 | |
| US7631069B2 | United States of America | B2 | |
| US7673054B2 | United States of America | B2 | |
| US7703029B2 | United States of America | B2 | |
| CN1829963B | China | B | |
| EP1649366B1This record | European Patent Office (EPO) | B1 | |
| AT488801T | Austria | T | |
| ATE488801T1 | Austria | T1 | |
| DE602004030126D1 | Germany | D1 | |
| CN1829970B | China | B | |
| US8135841B2 | United States of America | B2 | |
| EP1649365B1 | European Patent Office (EPO) | B1 | |
| JP4914717B2 | Japan | B2 | |
| AT553431T | Austria | T | |
| ATE553431T1 | Austria | T1 | |
| EP1649364B1 | European Patent Office (EPO) | B1 |
60 legal events, as 7 offices reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | Office | |
|---|---|---|---|
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Patent expired after termination of 20 yearsExpiredPE20 | PE20 | GB | |
| Expiry of rightR071 | R071 | DE | |
| Annual fee paid to national office [announced via postgrant information from national office to epo]GrantedPGFP | PGFP | EP | |
| Annual fee paid to national office [announced via postgrant information from national office to epo]GrantedPGFP | PGFP | EP | |
| Annual fee paid to national office [announced via postgrant information from national office to epo]GrantedPGFP | PGFP | EP | |
| Fee paymentPLFP | PLFP | FR | |
| Fee paymentPLFP | PLFP | FR | |
| Fee paymentPLFP | PLFP | FR | |
| Change of applicant/patenteeR081 | R081 | DE | |
| Change of representativeR082 | R082 | DE | |
| Change of representativeR082 | R082 | DE | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Patent lapsedLapsedMM4A | MM4A | IE | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Patent ceasedCeasedPL | PL | CH | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| No opposition filed against granted patent, or epo opposition proceedings concluded without decisionGrantedR097 | R097 | DE | |
| No opposition filedOpposition26N | 26N | EP | |
| No opposition filed within time limitOppositionORIGINAL CODE: 0009261PLBE | PLBE | EP | |
| Information on the status of an ep patent application or granted ep patentGrantedSTATUS: NO OPPOSITION FILED WITHIN TIME LIMITSTAA | STAA | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Lapsed in a contracting state [announced via postgrant information from national office to epo]LapsedPG25 | PG25 | EP | |
| Discontinued in the netherlands as no translation has been filedVDEP | VDEP | NL | |
| Corresponds to:REF | REF | EP | |
| European patents granted designating irelandGrantedFG4D | FG4D | IE | |
| European patent takes effect as a national patent in ch/liEP | EP | CH | |
| Designated contracting statesAK | AK | EP | |
| European patent grantedGrantedFG4D | FG4D | GB | |
| (expected) grantORIGINAL CODE: 0009210GRAA | GRAA | EP | |
| Grant fee paidORIGINAL CODE: EPIDOSNIGR3GRAS | GRAS | EP | |
| Despatch of communication of intention to grant a patentORIGINAL CODE: EPIDOSNIGR1GRAP | GRAP | EP | |
| First examination report despatched17Q | 17Q | EP | |
| Information on inventor provided before grant (corrected)RIN1 | RIN1 | EP | |
| Information on inventor provided before grant (corrected)RIN1 | RIN1 | EP | |
| Request for extension of the european patent (deleted)DAX | DAX | EP | |
| Request for examination filed17P | 17P | EP | |
| Designated contracting statesAK | AK | EP | |
| Public reference made under article 153(3) epc to a published international application that has entered the european phaseORIGINAL CODE: 0009012PUAI | PUAI | EP |
Numbers
- Publication
- 1649366
- Application
- 47635370
Titles3
- German
- WARTUNGSFREUNDLICHE GRID-MANAGER
- English
- MAINTAINABLE GRID MANAGERS
- French
- GESTIONNAIRES DE GRILLES APTES A L'ENTRETIEN
Classification
- CPC, 1
- G06F9/5072
- IPC, 2
- G06F9 46
- G06F15 16
Designated states28
- Contracting states, 28
- Austria
- Belgium
- Bulgaria
- Switzerland
- Cyprus
- Czechia
- Germany
- Denmark
- Estonia
- Spain
- Finland
- France
- United Kingdom
- Greece
- Hungary
- Ireland
- Italy
- Liechtenstein
- Luxembourg
- Monaco
- Netherlands (Kingdom of the)
- Poland
- Portugal
- Romania
and 4 moreShow fewer
- Sweden
- Slovenia
- Slovakia
- Türkiye