System and method providing automatic policy enforcement in a multi-computer service application

RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No. 09/696,752, filed on Oct. 24, 2000 now U.S. Pat. No. 6,915,338, titled “System and Method Providing Automatic Policy Enforcement in a Multi-Computer Service Application.”

TECHNICAL FIELD

This invention relates to distributed computer systems, such as server data centers or Websites. More particularly, this invention pertains to a way to automatically enforce a policy in a multi-computer service application.

BACKGROUND

It is no secret that Internet usage has exploded over the past few years and continues to grow rapidly. People have become very comfortable with many services offered on the World Wide Web (or simply “Web”), such as electronic mail, online shopping, gathering news and information, listening to music, viewing video clips, looking for jobs, and so forth. To keep pace with the growing demand for Internet-based services, there has been tremendous growth in the computer systems dedicated to hosting Websites, providing backend services for those sites, and storing data associated with the sites.

One type of distributed computer system is an Internet data center (IDC), which is a specifically designed complex that houses many computers for hosting Internet-based services. IDCs, which also go by the names “Webfarms” and “server farms”, typically house hundreds to thousands of computers in climate-controlled, physically secure buildings. These computers are interconnected to run one or more programs supporting one or more Internet services or Websites. IDCs provide reliable Internet access, reliable power supplies, and a secure operating environment.

<figref idref="DRAWINGS">FIG. 1</figref> shows an Internet data center <b>100</b>. It has many server computers <b>102</b> arranged in a specially constructed room. The computers are general-purpose computers, typically configured as servers. An Internet data center may be constructed to house a single site for a single entity (e.g., a data center for Yahoo! or MSN), or to accommodate multiple sites for multiple entities (e.g., an Exodus center that host sites for multiple companies).

The IDC <b>100</b> is illustrated with three entities that share the computer resources: entity A, entity B, and entity C. These entities represent various companies that want a presence on the Web. The IDC <b>100</b> has a pool of additional computers <b>104</b> that may be used by the entities at times of heavy traffic. For example, an entity engaged in online retailing may experience significantly more demand during the Christmas season. The additional computers give the IDC flexibility to meet this demand.

While there are often many computers, the Internet service or Website may only run a few programs. For instance, one Website may have 2000–3000 computers that run only 10–20 pieces of software. Computers can be added daily to provide scalability as the Website receives increasingly more visitors, but the underlying programs change less frequently. Rather, there are simply more computers running the same software in parallel to accommodate the increased volume of visitors.

Managing the physical resources of an Internet service is difficult today. Decisions such as when to add (or remove) computers to carry out functionality of the Internet service are made by human operators. Often, these decisions are made based on the operators' experience in running the Internet service. Unfortunately, with the rapid growth of services, there is a shortage of qualified operators who can make real-time decisions affecting the operation of a Website. Accordingly, it would be beneficial if some of the managerial, or policy aspects of running a Internet service could be automated.

Today, there is no conventional way to automate policy aspects of running an Internet service in a way that abstracts the functionality of the policy from the underlying physical deployment of the application. Perhaps this is because the industry has grown so fast that everyone's focus has been simply to keep up with the exploding demand by adding computers to Website applications. Not much thought has gone into how to model a policy mechanism that is scale-invariant.

At best, most distributed applications rely on human intervention and/or manual control for: (a) installing application components, (b) configuring the components, (c) monitoring the health of the overall application and individual components, and (d) taking reactive measures to maintain good overall application and component health. Where these tasks are automated, they operate in context of isolated physical components, without a big picture of how the components of the application relate to one another.

The downside with such traditional procedures to implementing policy is that Website operators must maintain constant vigilance over the operation of the application. Upon detecting a change in the application's operation that is contrary to the overall health of the application, Website operators are typically required to consult one or more documents that essentially detail the actions the Web Operators should take upon the occurrence of the condition. Moreover, such documents must continually be updated as the Website grows in physical resources and/or as the policy changes. Accordingly, it would be beneficial if some of the managerial, or policy aspects of running an Internet service could be automated.

SUMMARY

Systems and methods to enforce policy in a multi-computer service application are described. In one aspect, a scale-independent logical model of an application is generated. The application is for implementation in a distributed computing system. The scale-independent logical model includes multiple components representing logical functions of the application and intercommunication protocols. The model components are converted into one or more instances representative of physical resources used to implement the logical functions. The instances specify information such as communication ports on the physical resources and communication paths that link the physical resources.

BRIEF DESCRIPTION OF THE DRAWINGS

<figref idref="DRAWINGS">FIG. 1</figref> illustrates a conventional Internet data center (IDC).

<figref idref="DRAWINGS">FIG. 2</figref> illustrates a set of model components that form the building blocks for modeling an application, along with the associated schema.

<figref idref="DRAWINGS">FIG. 3</figref> illustrates a database application for an IDC that is modeled in terms of the components.

<figref idref="DRAWINGS">FIG. 4</figref> illustrates an Internet-based email application for an IDC.

<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of a computer that may be used to implement the modeling software for modeling the IDC.

<figref idref="DRAWINGS">FIG. 6</figref> is a flow diagram of a process for modeling an IDC.

<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram of a deployment system that converts a logical model to a fully functioning physical implementation

<figref idref="DRAWINGS">FIG. 8</figref> illustrates a translation of the logical model into real-world instances.

<figref idref="DRAWINGS">FIG. 9</figref> illustrates exemplary data records of an instance database used to store the real-world instances.

<figref idref="DRAWINGS">FIG. 10</figref> is a flow diagram of a process for deploying resources for the application based on the logical model.

<figref idref="DRAWINGS">FIG. 11</figref> illustrates a system providing automatic policy enforcement in a multi-computer application.

<figref idref="DRAWINGS">FIG. 12</figref> illustrates a procedure that provides automatic policy enforcement in a multi-computer application.

DETAILED DESCRIPTION

A system and procedure to automatically enforce policy in a multi-computer service application includes a modeling system and a deployment system. The modeling system permits developers of applications for distributed computer systems (e.g., server data centers, Internet data centers (IDCs), Web farms, and the like) to architect the hardware and software in an abstract manner. The modeling system defines a set of components used to describe the functionality of an application in a logical, scale-independent manner. An “application” within this context refers to an entire service hosted on the distributed computer system.

For instance, an Internet data center may host a Website for an online retailer, where the application entails the entire software and hardware configuration that implements the online retailer's Internet presence. The application might include, for example, a front end to handle client requests, an order processing system, a billing system, an inventory system, a database system, and a policy module to respond to changes in the operation of the application.

The model components are arranged and interconnected to form a scale-independent model of the application. Each component specifies some functionality of the application. The model can then be used to construct a scalable physical blueprint in terms of which machines run which pieces of software to form the application.

The deployment system uses the logical model to deploy various computer/software resources in real-time as the applications need them. The deployment system converts each of the model components into one or more instances that correspond to physical resources. The deployment system tracks the instances and all available resources. The deployment system decides when resources should be added (or removed) and monitors the current state of implementation. The deployment system installs the application and then dynamically and automatically modifies the resources used to implement the application in an ongoing basis as the operating parameters of the application change.

One of the components deployed by the deployment system is a policy module. The policy module is one of the model components in the logical model. The policy module responds to changes in the operation of the application by enforcing a policy. The policy module receives notifications from the deployed resources. In response to receiving such notifications, the policy module evaluates the notifications against a policy, which results in one or more of data, a request, or notification being sent to one or destination resources according to the configured logical data connections.

The system is described in the context of Website designs, such as applications for Internet data centers. However, the design system may be implemented to model other large size and scalable applications for computer systems. Accordingly, the design system can be implemented in a wide variety of ways, including Internet-based implementations and non-Internet-based implementations.

Model Components and Schema

The modeling system defines several model components that form the building blocks of a logical, scale-independent application: a module, a port, and a wire. It also defines a set of model extensions including, but not limited to: a store, an event source, an event sink, and an event path. In a design tool, the components are represented pictorially as graphical elements or symbols that may be arranged and interconnected to create scale-independent models of Website applications. The graphical elements have an associated schema that dictates how the functional operations being represented by the graphical elements are to be specified.

<figref idref="DRAWINGS">FIG. 2</figref> illustrates a set of model components <b>200</b> that a module, as represented by modules <b>202</b>(A)–<b>202</b>(C), a store <b>204</b>, ports <b>206</b>, wires <b>208</b>, event sources <b>210</b>, event sinks <b>212</b>, and event paths <b>214</b>. The components <b>200</b> are arranged in a no particular manner other than to foster discussion of their individual traits.

A module <b>202</b> represents a basic unit of functionality for the application and is depicted as a block. It is a logical entity that represents some portion of the application as implemented at the IDC, but has no physical manifestation. The module often corresponds to a software program that handles a logical set of tasks for the application. For instance, one module might represent a front end for a Website, another module might represent a login database, and another module might represent an electronic mail program.

Each module <b>202</b> is a container of behavior. A simple module is atomic and has associated a unique identifier. Modules can be nested into a hierarchy of modules to form more complex behaviors. In a module hierarchy, the leaf modules are simple modules, and the non-leaf modules are compound modules.

Each module <b>202</b> defines a unit of scaling. While one module logically represents a functional operation of the application, the module may translate to any number of computers when actually implemented. In this way, the module is scale-independent, allowing the number of underlying computers used to implement the module to change at any time. When converted to a physical implementation, “instances” are created from the modules. The module instances are assigned a unique identifier and maintain ancestral data regarding which module created them. The instances of simple modules are called “engines”, which correspond to software programs that run on individual computer nodes.

A store <b>204</b> is the most basic unit of storage and is depicted graphically as a disk storage icon. It represents a logical partitioning, which may be implemented by any number of physical disks or other storage media.

A port <b>206</b> is a service access point for a module <b>202</b> or store <b>204</b> and is depicted as spherical knobs projecting from the module or store. All service-related communications into and out of the module go through the port <b>206</b>. Each port <b>206</b> has a “type”, which is a set of attributes describing format, semantics, protocol, and so forth. At runtime, the port represents a set of physical ports associated with the instantiated engines of the modules.

A wire <b>208</b> is the logical binding that defines a communication route between two ports <b>206</b> and is depicted as a bold line. Each wire <b>208</b> can be type-checked (i.e., protocols, roles) and defines protocol configuration constraints (e.g., HTTP requires TCP, TCP requires IP, etc.).

Event sources <b>210</b> and event sinks <b>212</b> are used for discrete semantic messaging between modules and stores. An event source <b>210</b> is pictorially shown as a triangle pointing away from the module or store, while an event sink <b>212</b> is depicted as a triangle pointing toward the module or store. An event path <b>214</b> is a logical connection between sources and sinks, and carries event messages used to inform modules/stores and implement policy (e.g., scaling, fail-over, monitoring, application processes, etc.). It is depicted as a dashed line.

<figref idref="DRAWINGS">FIG. 2</figref> also illustrates the schema underlying the graphical elements as exemplary data structures associated with the model components. Module <b>202</b>(A) has an associated structure <b>220</b> that contains various characteristics for the module, such as functionality, processing requirements, software, and so forth. Thus, a module for a database server function might have characteristics pertaining to the kind of database (e.g., relational), the data structure (e.g., tables, relationships), software (e.g., SQL), software version, and so forth. Modules <b>202</b>(B) and <b>202</b>(C) have similar structures (not shown).

The store <b>204</b> has a corresponding structure <b>222</b> that defines the requirements for storage. The store schema structure <b>222</b> might include, for example, the kind of storage (e.g., disk), the storage format, and so on. Each port <b>206</b> has a schema structure, as represented by structure <b>224</b>, which dictates the port's type. Each wire <b>208</b> also is also associated with a structure, such as structure <b>226</b>, which outlines the protocols implemented by the connection. Similar schema structures may also be provide for event sources, event sinks, and paths.

Using the model components, an application developer can logically configure scale-independent applications prior to physically laying them out in the Internet data centers. The developer drafts a model using a user interface to select and interconnect the model components. Once constructed, the modeling software generates the application based on the depicted model and the underlying schema. The application may subsequently be converted into a physical blueprint that details the computers and software needed to implement the application for a specified number of client visitors.

The scale-invariant nature of the modeling system allows application developers to focus only on designing software for a specific functional task (e.g., front end, login database, email program, etc.). All external communications can then be expressed in terms of transmitting to and receiving from one or more associated ports. In this manner, the application developers need not worry about how many machines will be used to run the module, or how other modules of the scale-independent application are being configured.

Exemplary Module and Application

<figref idref="DRAWINGS">FIG. 3</figref> shows a fault-tolerant SQL (structure query language) database module <b>300</b> to demonstrate how the model components may be organized and connected to represent a portion of an application. In this example, the database module <b>300</b> represents a SQL database that may be used independently or as a component in a larger application. The SQL database module <b>300</b> has a module interface composed of a single port <b>302</b> that implements a TDS (Tabular Data Stream) protocol.

The SQL database module <b>300</b> is a compound module made up of three simple modules: a fail-over policy module <b>310</b>, a primary SQL module <b>312</b>, and a secondary SQL module <b>314</b>. The primary and secondary SQL modules represent dual programs that operate in parallel so that, in the event that the primary module <b>312</b> crashes, the secondary module <b>314</b> can assume the role without loss of service. The database module <b>300</b> also has a data store <b>316</b> that represents the memory storage for the SQL database module.

The primary SQL module <b>312</b> has a module interface that includes a first port <b>320</b> for communicating with the compound module port <b>302</b> and a second port <b>322</b> for communicating with the store <b>316</b>. The primary SQL module <b>312</b> also has an event source <b>324</b> and an event sink <b>326</b> for handling event messages from the fail-over policy module <b>310</b>. Similarly, the secondary SQL module <b>314</b> has a module interface with a first port <b>330</b> for communicating with the compound module port <b>302</b>, a second port <b>332</b> for communicating with the store <b>316</b>, and an event sink <b>334</b> for receiving events from the fail-over policy module <b>310</b>. A wire <b>336</b> interconnects the external compound module port <b>302</b> with the ports <b>320</b> and <b>330</b> of the primary and secondary SQL modules, respectively.

The store <b>316</b> has a port <b>340</b> to communicate with the primary and secondary SQL modules <b>312</b> and an event sink <b>342</b> to receive event messages from the fail-over policy module <b>310</b>. A wire <b>344</b> interconnects the store port <b>340</b> with the ports <b>322</b> and <b>332</b> of the primary and secondary SQL modules, respectively.

The fail-over policy module <b>310</b> has a module interface that includes three event sources and one event sink. An event sink <b>350</b> receives a “fail” event from the event source <b>324</b> of the primary SQL module <b>312</b> via an event path <b>352</b> when the primary SQL module experiences some failure. In response to receiving a “fail” event, the fail-over policy module <b>310</b> concurrently issues a first event to stop the failed primary module <b>312</b>, another event to assign the secondary module <b>314</b> as the new owner of the store <b>316</b>, and a third event to start the secondary module <b>314</b>. The “stop” event is issued via an event source <b>354</b> over an event path <b>356</b> to the event sink <b>326</b> of the primary SQL module <b>312</b>. The “stop” event directs the primary SQL module <b>312</b> to halt operation.

The fail-over policy module <b>310</b> issues an “assign owner” (AO) event from an event source <b>358</b>, over the event path <b>360</b> to the event sink <b>342</b> of the store <b>316</b>. The assign owner event directs the storage mechanisms to switch to allowing access by the secondary SQL module <b>314</b>, rather than the primary SQL module <b>312</b>. The fail-over policy module <b>310</b> also issues a “start” event from event source <b>362</b> over event path <b>364</b> to the event sink <b>334</b> of the secondary module <b>314</b>. The start event directs the secondary SQL module to start operation in place of the primary SQL module.

The SQL database module <b>300</b> illustrates how the model components—modules, stores, ports, wires, sources, sinks, and paths—may be arranged and interconnected to form a complex module. The developer specifies the characteristics associated with each component according to the prescribed schema. The complex module may in turn be added to other simple or complex modules to form other complex modules. Eventually, the largest complex module becomes the application, which may then be used to form a blueprint for constructing the data center.

<figref idref="DRAWINGS">FIG. 4</figref> shows a simplified application <b>400</b> for an online retailer. The application <b>400</b> includes a front end module <b>402</b>, a catalog module <b>404</b>, an order processing module <b>406</b>, and a fulfillment module <b>408</b>. The application <b>400</b> also includes a customer database <b>410</b> and the fault-tolerant SQL database module <b>300</b>. Notice that the SQL database module <b>300</b> is the same as that shown in <figref idref="DRAWINGS">FIG. 3</figref> to illustrate how complex modules can be nested into even greater complex modules to form an application.

The front end module <b>402</b> handles requests from clients who wish to shop with the online retailer. The front end module <b>402</b> has a port <b>420</b> that accommodates communications with external clients using the TCP/IP protocol over the Internet. The front end module <b>402</b> also has an order port <b>422</b> to define a communication exchange with the order processing module <b>406</b> and a catalog port <b>424</b> for communication flow to the catalog module <b>404</b>. The ports <b>422</b> and <b>424</b> may be configured according to any of a variety of types, which support any one of a number of protocols including SOAP, TCP, or UDP. An event sink <b>426</b> is also provided to receive a “new product” message from the catalog module <b>404</b> when a new product has been added to the catalog.

The catalog module <b>404</b> provides catalog information that may be served by the front end to the requesting clients. The catalog module <b>404</b> has a front end port <b>430</b> connected via a wire <b>432</b> to the catalog port <b>424</b> of the front end module <b>402</b>. The front end port <b>430</b> has a type that matches the catalog port <b>424</b>. The catalog module <b>404</b> also has an event source <b>434</b> that for communicating the “new product” messages over path <b>436</b> to the event sink <b>426</b> of the front end module <b>402</b>.

A SQL port <b>438</b> interfaces the catalog module <b>404</b> with the SQL database module <b>300</b>. The SQL port <b>438</b> has a type that utilizes the TDS protocol for the communication exchange with the external port <b>302</b> of the SQL database <b>300</b>.

The order processing module <b>406</b> has a front end port <b>440</b> to define a communication interface with the front end module <b>402</b> via a wire <b>442</b>. The order processing module <b>406</b> also has a fulfillment port <b>444</b> to facilitate communication with the fulfillment module <b>408</b> over wire <b>446</b> and a database port <b>448</b> to facilitate communication with the customer database <b>410</b> via wire <b>450</b>.

An event source <b>452</b> is provided at the order processing module <b>406</b> to pass “order complete” events to the fulfillment module <b>408</b> via path <b>454</b>. These events inform the fulfillment module <b>408</b> that an order is complete and ready to be filled. A second event source <b>456</b> passes “new account” events to the customer database <b>410</b> via path <b>458</b> whenever a new customer orders a product.

The fulfillment module <b>408</b> has an order port <b>460</b> to provide access to the wire <b>446</b> to the order processing module <b>406</b> and a database port <b>462</b> to interface with the customer database <b>410</b>. The fulfillment module <b>408</b> also has an event sink <b>464</b> to receive the “order complete” events from the order processing module <b>406</b>.

The customer database <b>410</b> has an order port <b>470</b> to provide access to wire <b>450</b> and a fulfillment port <b>472</b> to facilitate communication with the fulfillment module <b>408</b> via wire <b>474</b>. The customer database <b>410</b> further has an event sink <b>476</b> to receive the “new account” events from the order processing module <b>406</b>.

The modeling approach illustrated in <figref idref="DRAWINGS">FIGS. 3 and 4</figref> is tremendously beneficial because it allows developers and IDC operators to view the entire operation in terms of abstract functional pieces that are scale-independent. The online retailer application <b>400</b>, for example, requires a front end unit, a catalog unit, an order processing unit, and a fulfillment unit regardless of whether the retailer is handling 100 hits a day or 10 million hits per day.

The scale-independent nature frees the developer to focus on his/her little piece of the application. For instance, a developer assigned the task of building the front end module <b>402</b> need only be concerned with writing software code to facilitate response/reply exchanges. Any communication to and from the module is defined in terms of order-related data being passed to the order processing module <b>406</b> via the order port <b>422</b> and product data being received from the catalog module <b>404</b> via the catalog port <b>424</b>. The developer defines the data flow to and from the order port <b>422</b> and the catalog port <b>424</b> according to their respective associated protocol types.

The application <b>400</b> can then be used to construct a computer system that hosts the online retailer. Initially, the online retailer may not receive very much traffic, especially if launched away from the Christmas season. So, perhaps the front end module <b>402</b> translates initially to only a few computers to handle the light traffic from the Internet. But, suppose that over time the site becomes more popular and the Christmas season is fast approaching. In this situation, the online retailer may authorize the IDC operator to add many more computers for the front end tasks. These computers are equipped the software and configured to accept HTTP requests for product information and to serve Web pages containing the product information. The computers are added (or removed) as needed, without altering the basic application <b>400</b>.

Computer-Based Modeling System and Method

<figref idref="DRAWINGS">FIG. 5</figref> shows an exemplary computer system <b>500</b> that implements modeling software used to design Website applications. The modeling computer may be implemented as one of the nodes in a Website, or as a separate computer not included as one of the nodes. The modeling computer has a processor <b>502</b>, volatile memory <b>504</b> (e.g., RAM), and non-volatile memory <b>506</b> (e.g., ROM, Flash, hard disk, optical, RAID memory, etc.). The modeling computer <b>500</b> runs an operating system <b>510</b> and modeling system <b>512</b>.

For purposes of illustration, operating system <b>510</b> and modeling system <b>512</b> are illustrated as discrete blocks stored in the non-volatile memory <b>506</b>, although it is recognized that such programs and components reside at various times in different storage components of the computer <b>500</b> and are executed by the processor <b>502</b>. Generally, these software components are stored in non-volatile memory <b>506</b> and from there, are loaded at least partially into the volatile main memory <b>504</b> for execution on the processor <b>502</b>.

The modeling system <b>512</b> includes a user interface <b>514</b> (e.g., a graphical UI) that presents the pictorial icons of the model components <b>516</b> (e.g., modules, ports, sources, sinks, etc.), a component schema database <b>518</b>, and a logical-to-physical converter <b>520</b>. The modeling system <b>512</b> allows a developer to design an application for an IDC by defining modules, ports, and event message schemes. The user interface <b>514</b> presents symbols of the components <b>516</b>, such as the symbols shown in <figref idref="DRAWINGS">FIGS. 2–4</figref>, and permits the developer to arrange and interconnect them. The UI <b>514</b> may even support conventional UI techniques as drag-and-drop operations.

The symbols depicted on the screen represent an underlying schema <b>518</b> that is used to define the model. For instance, a block-like module symbol is associated with the characteristics of the functionality that the module is to represent in the application. Thus, the developer may define a database module that has characteristics pertaining to the kind of database (e.g., relational), the data structure (e.g., tables, relationships), software (e.g., SQL), software version, and so forth. Accordingly, by drafting the model on the UI, the developer is architecting the entire schema that will be used to design the scale-independent application.

Once the application is created, the logical-to-physical converter <b>520</b> converts the application to a physical blueprint that details the number of computers, software components, physical ports, and so forth. The converter takes various parameters—such as how many site visitors are expected, memory requirements, bandwidth requirements, processing capabilities, and the like—and scales the application according to the schema <b>518</b> represented by the application. The converter <b>520</b> specifies the number of computers needed to implement each module, the number of disks to accommodate the stores, and the types of communications protocols among the modules and stores.

In this manner, the modeling system changes the development effort from a node-centric approach to architecting IDCs to an application-centric approach. Within conventional node-centric methodology, the focus was on the computers and how they were laid out. The application was then loaded onto the nodes in a somewhat ad hoc manner. With the new application-centric approach, the focus is initially on the application itself. The physical nodes used to implement the application are derived once the application is specified.

<figref idref="DRAWINGS">FIG. 6</figref> shows a method for modeling a scale-independent application for an Internet data center. The method <b>600</b> may be implemented, for example, by the modeling system <b>512</b> executing on the modeling computer <b>500</b>. In such an implementation, the method is implemented in software that, when executed on computer <b>500</b>, performs the operations illustrated as blocks in <figref idref="DRAWINGS">FIG. 6</figref>.

At block <b>602</b>, the modeling system <b>512</b> allows the developer to define the modules and stores that form the functional elements of the applications. The UI <b>514</b> enables the developer to create modules and stores and to define their characteristics as prescribed by a predetermined schema. This entry process begins to construct the logical building blocks of the application.

At block <b>604</b>, the modeling system <b>512</b> enables the developer to define the ports for the modules and stores. The developer selects the type of ports and ensures compatibility of ports that are connected to one another. At block <b>606</b>, the developer also defines any events that may be passed among modules and stores. The developer creates event sources and event sinks to accommodate the various events. At block <b>608</b>, the developer uses the modeling system <b>512</b> to interconnect the ports with wires and the event sources/sinks with event paths. By joining the various modules and stores, the developer effectively forms a logical representation of the application.

At block <b>610</b>, the modeling system <b>512</b> generates an application using the graphical representation constructed by the developer. The modeling system <b>512</b> generates the logical specifications associated with the graphical model, including the characteristics of the modules and stores as well as the types of the ports and event sources/sinks. The application provides a complete logical representation of the service that will eventually be implemented at the Internet data center. The application may be stored on disk or some other form of computer-readable medium (block <b>612</b>).

At block <b>614</b>, the modeling system <b>512</b> converts the application to a physical blueprint that specifies the computers, the software run by each of the computers, and the interconnections among the computers. This physical blueprint may be used by the operator to install and manage the service afforded by the application.

Computer-Based Deployment System

Once a logical model is created, an automatic computer-based deployment system uses the logical model to deploy various computer/software resources to implement the application. The deployment system converts each of the model components into one or more instances that correspond to physical resources, such as nodes of a distributed computer system that are loaded with specific types of software to implement the function represented by the model components. The deployment system initially installs an application on the physical resources according to the logical model. It then dynamically and automatically modifies the resources used to implement the application in an ongoing basis as the operating parameters of the application change.

<figref idref="DRAWINGS">FIG. 7</figref> shows a deployment system <b>700</b> that converts the logical model to a fully functioning physical implementation. The deployment system <b>700</b> includes a policy module <b>702</b>, a core runtime logical-to-physical converter <b>704</b>, and hardware/software resources <b>706</b> that are all interconnected via a wireless and/or wire-based communication network <b>708</b> (e.g., a LAN, a WAN, intranet, Internet, combinations thereof, etc.). In this example, the hardware/software resources are illustrated as computer nodes of a distributed computer system, as represented by computers <b>706</b>(<b>1</b>), <b>706</b>(<b>2</b>), . . . , <b>706</b>(N). The policy module <b>702</b> and core runtime converter <b>704</b> may be implemented on one or more computers, which may or may not be part of the nodes in the distributed computer system.

For purposes of discussion, the deployment system <b>700</b> is described in the context of an Internet service that is executed at an Internet data center having an abundance of generic computer nodes. The nodes can be allocated to one or more Internet services from a reserve pool of nodes, as illustrated in <figref idref="DRAWINGS">FIG. 1</figref>.

The policy module <b>702</b> implements one or more policies devised by the developer or operator of the Internet service. The policies specify when instances derived from the logical model should be created, manipulated, and destroyed. The policy module monitors various events generated by the nodes and implements policy decisions regarding how to handle the events. By specifying when and what instances should be created (or destroyed), the policy module <b>702</b> effectively dictates when hardware/software resources <b>706</b> should be added (or removed) to support the changing demands of the Internet service.

The core runtime converter <b>704</b> implements the policy decisions made by the policy module <b>702</b>. The runtime converter <b>704</b> has a service running state <b>710</b> that tracks all instances of the model components currently in existence. That is, the service running state <b>710</b> tracks the elements in the physical world with respect to the logical model. The service running state <b>710</b> maintains a copy of the logical model, such as online retailing model <b>400</b> of <figref idref="DRAWINGS">FIG. 4</figref>. The logical model is created by the modeling system described above with respect to <figref idref="DRAWINGS">FIGS. 5 and 6</figref>. The current instances are maintained in a database <b>714</b>. The records in instance database <b>714</b> include such information as identify of the instance, the name of the logical component from which it is derived, the node on which it is running, the network addresses representing the ports of the modules and stores, type of software loaded on the node, various protocols supported by the instance, and so forth.

The instances are derived from the logical model. The policy module <b>702</b> articulates the number of instances of each model component used to implement the Internet service at any given time. For instance, suppose the Internet service requires one hundred computers to effectively implement a front end that handles site traffic at 99.9% efficiency with each computer running at 70% utilization. The policy module might further specify that more computers should be added if some policy threshold is met (e.g., efficiency rating drops below some threshold or computer utilization rises above some threshold) or removed if another threshold is met.

A resource manager <b>716</b> tracks all of the physical resources available to the Website. These resources include computer nodes, storage, software, and so forth. Records identifying all of the resources are kept in the resource database <b>718</b>. For instance, there might be one record for each computer node, storage device, and software module in the Internet data center. The records contain such information the identity of the allocated nodes, computing characteristics and capabilities, the application(s) to which they are allocated, the date and time of allocation, and so forth.

The resource manager <b>716</b> allocates the resources as needed or requested by the Internet service according to the policy implemented by the policy module <b>702</b>. The allocation depends upon the availability of resources at the time of request. The resource manager <b>716</b> may also recover resources that are no longer needed by the Internet service and return the resources to the pool of available resources.

Upon allocation (or recovery) of a resource, the resource manager <b>716</b> post a record to the resource database <b>718</b> reflecting which resource is allocated to (or recovered from) which application. As an example, when an Internet service desires more nodes for the front end tasks, the resource manager <b>716</b> allocates one or more free nodes from the pool of resources to the Internet service.

Another unit, referred to as “new” <b>720</b>, manages the creation of new instances in the physical world from the model components specified in the logical model <b>400</b>. A loader <b>722</b> carries out the configuration of newly allocated resources to the functions dictated by the new instances. In this manner, when another instance of a component is desired, the new program <b>720</b> communicates with the resource manager <b>716</b> to allocate a node from the resource pool and with the loader <b>722</b> to load the appropriate software programs onto the node.

Each computer node <b>706</b>(<b>1</b>)–<b>706</b>(N) has a policy engine <b>730</b>(<b>1</b>)–<b>730</b>(N) and a node loader <b>732</b>(<b>1</b>)–<b>732</b>(N). The policy engine <b>730</b> locally implements the policies dictated by the policy module <b>702</b>. Depending upon the policies, the policy engine <b>730</b> monitors various parameters and performance metrics of the local node and generates events that may be sent to inform the policy module <b>702</b> as to how the node is operating within the policy framework.

For instance, suppose a policy specifies that nodes used to instantiate a particular module of the logical model should not operate at more than 80% utilization. The policy engine <b>730</b> enforces this policy by monitoring utilization. If utilization rises above 80%, the policy engine <b>730</b> sends an event to the policy module <b>702</b> to notify of the utilization increase. With this information, the policy module <b>702</b> may decide to create another instance of the module to relieve the over-utilized node.

The node loader <b>732</b> performs the actual loading tasks specified by the loader <b>722</b> in the core runtime converter <b>704</b>. It is the local code on an otherwise generic node to which the runtime loader <b>722</b> may communicate when configuring the node.

<figref idref="DRAWINGS">FIG. 8</figref> shows an example of a portion of logical model <b>400</b> being converted into actual instances. To illustrate the conversion, the front end module <b>402</b> and the order processing module <b>406</b> are extracted from the online retailer service application <b>400</b> (<figref idref="DRAWINGS">FIG. 4</figref>). A wire <b>442</b> interconnects the two modules by logically coupling the ports <b>422</b> and <b>440</b>.

The front end module <b>402</b> in the logical model translates to one or more computer nodes in the physical world that runs software for handling client queries. These physical instances are represented diagrammatically by the rectangles with ovals. Here, based on a policy, the front end module <b>402</b> is converted into multiple front end instances <b>800</b>(<b>1</b>), <b>800</b>(<b>2</b>), <b>800</b>(<b>3</b>), . . . , <b>800</b>(J), which each corresponds in one-to-one fashion with a computer node loaded with software used to implement the front end of the service. The order processing module <b>406</b> translates to one or more computer nodes that run a program for processing client orders. In <figref idref="DRAWINGS">FIG. 8</figref>, the order processing module <b>406</b> is converted into plural order processing instances <b>802</b>(<b>1</b>), <b>802</b>(<b>2</b>), . . . , <b>802</b>(K).

The ports <b>422</b> and <b>440</b> represent protocols and roles within protocols in the logical world. They translate to the physical world as a set of network addresses used to communicate with the software at the respective modules. For instance, the physical translation of a logical port might be IP Port <b>80</b>, using HTTP (hypertext transport protocol). In <figref idref="DRAWINGS">FIG. 8</figref>, one logical port in the logical model is converted to a port address for each instance of the module. Logical port <b>422</b> converts into physical address ports <b>804</b>(<b>1</b>)–<b>804</b>(J) and logical port <b>440</b> converts into physical ports <b>806</b>(<b>1</b>)–<b>806</b>(K). Furthermore, the logical ports might correspond to particular objects and their interfaces. For example, port A might correspond to an object A and interface on a first computer, while port B might correspond to an object B and interface on a second computer.

The wire <b>442</b> represents logical connections between ports. It translates to a physical mesh of all possible communication paths between the instances of each module. The number of wires is determined as the cross product of the number of instances of each module. In <figref idref="DRAWINGS">FIG. 8</figref>, the wire <b>442</b> converts to a physical connection between every instance <b>800</b>(<b>1</b>)–<b>800</b>(J) of the front end module <b>402</b> and every instance <b>802</b>(<b>1</b>)–<b>802</b>(K) of the order processing module <b>406</b>.

<figref idref="DRAWINGS">FIG. 9</figref> illustrates exemplary records <b>900</b> in the instance database <b>714</b> that track the instances derived from the logical model <b>400</b>. In this example, the database is a relational database that stores records in tables, and the records may be linked to one another via relationships. Here, there are three tables: a module table <b>902</b>, a port table <b>904</b>, and a wire table <b>906</b>. Each table holds one record for a corresponding instance of the logical model. Each record has a number of fields relating to the type of information that is being tracked for each instance.

The module table <b>902</b> tracks instances of modules in the logical model. There is one record for each module instance. Thus, with respect to the front end module of <figref idref="DRAWINGS">FIG. 8</figref>, there are “J” records in the module table <b>902</b> corresponding to the “J” front end instances <b>800</b>(<b>1</b>)–<b>800</b>(J). Each record in the module table <b>902</b> contains an instance ID to identify the individual instance, an identity of the module component from which the instance is derived, a node ID to identify the computer node to which the instance is associated, the type of software loaded on the node to implement the module functionality, a software ID, an identity of the various ports to the modules, and various protocols supported by the module instance. It is noted that other implementations may include additional fields or fewer fields than those illustrated.

The port table <b>904</b> tracks instances of the port in the logical model, such as ports <b>422</b> and <b>440</b> in <figref idref="DRAWINGS">FIG. 8</figref>. A record from this table includes such information as the port ID, the model component identity, a node ID, the network address represented by the port, the instance ID of the corresponding instance, the protocol used by the port, and an ID of the wire to which the port is connected. Again, more or less fields may be used in other implementations.

The wire table <b>906</b> tracks instances of the wires in the logical model, such as wire <b>442</b> in <figref idref="DRAWINGS">FIG. 8</figref>. A record in the wire table includes a wire ID, a model component identity, a node ID, a port ID, an instance ID, and the protocol supported by the wire.

Notice that the three tables can be correlated with one another via various relationships. For example, the module table <b>902</b> and the port table <b>904</b> are related by various data fields, such as the port ID field and the instance ID field. The wire table <b>906</b> correlates with the port table <b>904</b> via wire ID and port ID and with the module table <b>902</b> via instance ID. Notice also that the tables have a node ID field that provides a reference into the resource database by identifying which node the instance is associated with.

It is noted that the illustrated arrangement of the database is merely for discussion purposes. Many other table arrangements with more or fewer tables than illustrated.

Realtime Deployment Method

<figref idref="DRAWINGS">FIG. 10</figref> shows a method <b>1000</b> for deploying resources for an application based on the logical model. The method <b>1000</b> is implemented by the deployment system <b>700</b> of <figref idref="DRAWINGS">FIG. 7</figref> and hence can be embodied as software that, when executed on one or more computers, performs the operations illustrated as blocks in <figref idref="DRAWINGS">FIG. 12</figref>.

At block <b>1002</b>, the policy module <b>702</b> monitors various operating parameters and listens for events from the policy engines <b>730</b> at the individual nodes <b>706</b>. The policy module <b>702</b> evaluates the parameters and events against the policy backdrop to determine whether the current physical implementation is satisfactorily supporting the application, or whether a new instance of some component should be added (block <b>1004</b>). It is noted that the continuing process is described in the context of adding a new instance. However, the policy may alternatively dictate that one or more instances should be removed. Removal of instances is somewhat easier in that instances are deleted from the instance database <b>714</b> and the computer nodes are returned to the additional pool for reallocation.

Assuming that a new instance is desired (i.e., the “yes” branch from block <b>1004</b>), the policy module <b>702</b> consults the service running state <b>710</b> to understand the current number and arrangement of instances (block <b>1006</b>). The policy module <b>702</b> can request various types of information of the service running state <b>710</b>, such as: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0104">How many instances of a given module?</li><li id="ul0001-0002" num="0105">What nodes are associated with a given model component?</li><li id="ul0001-0003" num="0106">Which nodes are attached to a given wire?</li><li id="ul0001-0004" num="0107">What is the network address of a node?</li><li id="ul0001-0005" num="0108">What wire is attached to a port on a given component?</li><li id="ul0001-0006" num="0109">What components does a given component own?</li></ul>

Depending upon the event or operating condition, the policy module <b>702</b> can request information on a particular module in the logical model <b>400</b>. For example, assume that an event has been received from a front end node indicating that the utilization has risen to above 90%. In response, a policy specifying the addition of another instance at such utilization levels is triggered. The policy module <b>702</b> asks the service running state <b>710</b> how many instances of the front end module currently exist, and information regarding how to specify an instance for the front end module.

At block <b>1008</b>, the policy module <b>702</b> calls the program “new” <b>720</b> to create a new instance of a module in the logical model. At block <b>1010</b>, the new program <b>720</b> calls the resource manager <b>716</b> to request allocation of a new node (assuming one is available). The resource manager <b>716</b> examines the resources using data from the resource database <b>718</b> and allocates a new node that is currently available from a pool of free nodes. The resource manager <b>716</b> records the allocation in the resource database <b>718</b>, identifying which node is allocated, what application it is being allocated to, the date and time that it is allocated, and so forth.

At block <b>1012</b>, the new program <b>720</b> calls the loader <b>722</b> to install software onto the allocated node and configure the node to perform the functions represented by the logical module from which the instance is created. In response, the loader <b>722</b> initializes the node by communicating with the node loader <b>730</b> of the new node via the network <b>708</b> to install an image of an operating system (e.g., Windows NT server operating system from Microsoft Corporation). The loader <b>722</b> then loads the appropriate software and configures the node to perform the functions represented by the logical model component from which the instance is derived. For example, for an instance of an SQL module in the logical model <b>400</b>, the node loader <b>732</b> loads SQL server software. The loader <b>722</b> registers the physical port addresses with the service running state <b>710</b>, which records the new instance in the instances database <b>714</b>.

At block <b>1014</b>, the service running state <b>710</b> is notified when the newly allocated and configured node is up and running. The service running state <b>710</b> records a new instance of the logical model in the instances database <b>714</b>. The record reflects an ID of the instance, the name of the logical component from which it is derived, the allocated node, the network addresses of the node, software type and IDs, various protocols supported by the instance, and so on.

At block <b>1016</b>, the policy module <b>702</b> is notified by the core runtime converter <b>704</b> that a new instance is up and running. The new instance should relieve the event or operating condition to bring operation back into compliance with the policy.

Policy Enforcement Mechanism

<figref idref="DRAWINGS">FIG. 11</figref> illustrates a system <b>1100</b> to enforce policy in a multi-computer service application having a plurality of software programs that execute on a plurality of computers. The service application includes a communications medium that allows data communications between different ones of the computers. In this example, the application is a distributed copying service, where copy modules <b>1102</b> send electronic documents to front ends <b>1104</b>.

The application is designed and implemented using the modeling system and deployment system discussed above. The model components of the application include the logical counterparts of the policy module <b>1101</b>, one or more copy modules <b>1102</b>, and one or more front ends <b>1104</b>. The modeling system allows the model components to be arranged and interconnected to form a scale-independent model of the application. The deployment system converts each of the model components into one or more instances that correspond to physical resources. In this example, the physical instances of the system's model components are represented diagrammatically by rectangles with ovals.

Each module includes respective logical input ports <b>1106</b> and logical output ports <b>1108</b>. The logical input ports <b>1106</b> and logical output ports <b>1108</b> are configured on different modules in accordance with a logical model <b>1124</b> of the distributed copying service. Each logical input and output port is defined by port software.

Logical wires or data connections are configured between the logical output and input ports in accordance with the logical model. For example, data flow arrows <b>1110</b>, <b>1112</b>, and <b>1114</b> represent logical data connections. Each module can include other communication paths other than those represented by the data flow arrows. Upon each module's instantiation, each respective port is configured to communicate through different numbers of logical data connections without modifying the port software.

The policy module <b>1101</b> implements one or more policies devised by the developer or operator of the service application. The policy module monitors various events generated by the computer nodes and implements policy decisions regarding how to handle the events. In this example, the front end module <b>1104</b> represents a newly instantiated module that is ready to receive electronic data from copy modules <b>1102</b>. The front end module provides a notification <b>1110</b> to output port <b>1108</b>-<b>2</b>, which is configured according to the configured logical data connections to send the notification to the policy module's input port <b>1106</b>-<b>1</b>. The policy module accepts the notification and in response, formulates a request for one or more destination modules. As noted above, such a request includes, data or notifications.

The request is provided to the output port <b>1108</b>-<b>1</b> of the policy module. The output port forwards the request <b>1112</b> to respective input ports <b>1106</b> of one or more copy modules in accordance with the configured logical data connections. In this example, the request notifies the copy modules that they can begin sending data to the newly instantiated front end module.

In response to receiving the request, the respective output ports <b>1108</b>-<b>3</b> through <b>1108</b>-N are reconfigured, during runtime, according to the logical connections <b>1114</b> described in the logical model to send data to the new front end.

In this manner, the policy enforcement system changes distributed application management from a manual process requiring human intervention and control, to an automatic process. With conventional management procedures, a sending module would have had to have been recoded with the new addresses of any added components. With the new automatic policy enforcement, scale-invariant policies can be developed that do not require constant human intervention and control to respond to changes in application operation.

<figref idref="DRAWINGS">FIG. 12</figref> illustrates an exemplary procedure <b>1200</b> to automate policy enforcement in a multiple-computer application. At step <b>1202</b>, the procedure configures logical output ports and logical input ports on different modules according to a logical model (logical model) of the multi-computer service application. Each logical input and output port is defined by port software.

At step <b>1204</b>, the procedure configures logical data connections between the logical output and input ports according to the logical model. At step <b>1206</b>, the procedure configures each port to communicate through different numbers of logical data connections. This is done without modifying the port software.

At step <b>1208</b>, upon occurrence of a condition, the procedure sends a notification from a particular module to a policy module. At step <b>1210</b>, the policy module receives the notification. At step <b>1212</b>, the policy module responds to the notification by determining a request for one or more destination modules. This request is determined based on a policy developed by the designer of the application. At step <b>1214</b>, the policy module provides the request to an output port of the policy module to forward the request to input ports of multiple modules according to the configured logical data connections. In this way, the response to the notification is automatically forwarded to the proper modules without a need to determine who or where those modules are.

CONCLUSION

Although the description above uses language that is specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the invention.