System and method for server configuration control and management

SYSTEM AND METHOD FOR SERVER CONFIGURATION CONTROL AND MANAGEMENT

FIELD

0003[0001] The invention relates to managing data processing system configurations. More specifically, the invention relates to managing system configurations in a network storage system.

BACKGROUND

0005[0002] Large-scale data processing systems frequently rely on banks of similar or identical machines to provide infrastructure services, rather than requiring each computer in the system to operate autonomously by including all of its own infrastructure. For example, data storage functionality is often consolidated into a server or data server array; this can facilitate efficient backup strategies and permit an arbitrary number of data processing systems to access and operate on a common data set. (An alternative to this arrangement might involve providing some of the overall data storage space required at each processing system, but this approach may complicate application design by requiring software to distinguish between data stored locally and data stored on other peer systems. Successful backup strategies may be more difficult to implement, and the failure of an individual system may impact the work of other systems that needed access to the data stored on the failed system.) [0003] Of course, shifting an infrastructure function such as data storage onto a bank of special-purpose machines gives rise to a different set of challenges, even as it alleviates some difficulties of managing a large-scale system. Important among these challenges is the task of configuring the bank of machines so that each operates as intended to provide the infrastructure function. Even special-purpose machines that only provide a limited range of services may have complex configuration requirements, particularly when the machines are installed at diverse geographical locations to provide redundancy and/or to take advantage of services or facilities with locality-dependent aspects. [0004] GroupsTof servers that are to work together to provide a basic data processing service with redundancy and high availability may require consistent and coordinated, but not identical, configurations. As a simple example, consider two data storage servers (e.g., "fileservers") that are to store data for client systems. These servers may need identical configurations to control access from remote clients, but they may need different network communication configurations because they are connected to a distributed data network through two different circuits.

0006[0005] Current system management procedures generally take an ad hoc approach to managing similar systems. For example, an administrator may keep copies of various generic configurations in a library, and prepare a configuration for a new system based on the closest generic configuration. However, once a generic configuration is customized and deployed, there is often no way to update a common parameter in all the configurations of servers in the farm without connecting to each machine in turn and making the modification — a time-consuming and error-prone task.

0007[0006] Therefore, a better method of configuring and managing many similar (but not necessarily identical) data processing systems efficiently may be of value in this field.

SUMMARY

0009[0007] Some embodiments of the invention obtain data processing system configurations, each configuration to include an element to control access from a remote client to a storage facility of a data processing system, and compare the configurations. Configuration differences detected during the comparison may be reported to a system administrator or management center for further investigation and possible correction. Other embodiments obtain a base configuration for a data processing system and override some elements of the base configuration to produce a locally-adapted configuration that may be used to control interactions between the data processing system and a data storage client. Still other embodiments permit attributes to be attached to elements of a data processing system configuration, the attributes to control the subsequent use of the configuration.

BRIEF DESCRIPTION OF DRAWINGS

0011[0008] Embodiments of the invention are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to

0012"an" or "one" embodiment in this disclosure are not necessarily to the same embodiment, and such references mean "at least one."

0013[0009] Figure 1 shows an environment where an embodiment of the invention operates.

0014[0010] Figure 2 shows some subsystems of an infrastructure server including an embodiment of the invention.

0015[0011] Figure 3 shows a logical flow of operations in a file server.

0016[0012] Figure 4 illustrates configuration element import and export.

0017[0013] Figure 5 outlines the operations of an embodiment of the invention.

0018[0014] Figure 6 is a block diagram of file server subsystems according to another embodiment of the invention.

0019[0015] Figure 7 shows an example user interface of another embodiment of the invention.

0020[0016] Figure 8 shows a second example user interface of the third embodiment of the invention.

0021[0017] Figure 9 shows an overall combination logical and physical diagram of a distributed system.

DETAILED DESCRIPTION

0023[0018] As described in greater detail below, the solution introduced here provides methods and apparatus to improve the accuracy and efficiency of configuring a large number of similar systems. For simplicity, these techniques are often described in scenarios limited to a small number of systems, but it will be clear to those of skill in the art that extending the ideas to an arbitrarily large number of systems is straightforward and proportionately beneficial. [0019] Figure 1 shows an environment where embodiments of the invention may usefully be deployed. A number of data centers 110, 120, 130 are shown; each data center contains several data processing systems 111, 112; 121, 122; 131, 132 connected to arrays of mass storage devices 113, 114; 123, 124; 133, 134 and configured to provide data storage services. Control instructions and data may be transferred between data centers across a public distributed data network 140 such as the Internet, or across one or more private distributed data networks 150, or some combination of these types of networks. Data storage services may be provided directly to a client 160, or may be used by a server 170 that provides a derivative service to client 160. The distributed data processing system may be managed from a network operations center ("NOC") or administrative console, shown here as computer system 180. [0020] Figure 2 shows some subsystems that may be present in an infrastructure server such as the data storage servers shown in Figure 1, whose configuration can be managed by an embodiment of the invention. The server may include mass storage control logic 210 to control an array of storage devices 113; communication logic 210 to drive a network interface 225 that allows the system to communicate over a distributed data network 140; backup logic 230 to operate a backup device 235, and system management logic 240 to coordinate configuration and operation of the various subsystems. In the server depicted in this figure, the infrastructure services are provided to clients by several servers 250, which respond to requests according to various protocols. Specifically shown are Network File System ("NFS") server 252, Common Internet File System ("CIFS") server 254, Hypertext Transfer Protocol ("HTTP" or "web") server 256, and File Transfer Protocol ("FTP") server 258. The servers 252, 254, etc. can be software modules or layers implemented within the same physical computing platform.

0024[0021] Figure 3 illustrates an example of the logical structure of the operating system of a storage server, such as any of servers 111, 112, 121, 122, 131, 132 in Figure 1, and also indicates the logical sequence that the subsystems of Figure 2 may perform to respond to a data storage or retrieval request from a client 160. The request 314 may arrive at the server 300 via a network data connection and be processed by a network access layer 220. The request includes a payload containing instructions and data from the client to the server and a network header. The header may specify, among other things, source and destination addresses and the particular protocol to which the request conforms (e.g., NFS). The payload 324 is passed up through a protocol- handling layer 320, which implements one or more of the different types of servers/protocols represented in Figure 2, and eventually to a file system layer 330, which processes data 334 to determine what action the client is requesting, and how the filesystem can accommodate the request. If there is data to be stored, the file system layer 330 may pass it to a RAID (Redundant Arrays of Independent Disks) storage access manager 340, where it may be separated into portions 344 according to a data protection and error recovery scheme, then passed to storage drivers 350 and written on an array of mass storage devices 113.

0025[0022] If the client's request is to retrieve information, the data flow proceeds in the reverse direction: data fragments 358 may be retrieved from the array of mass storage devices 113 and put together 348 to prepare a client reply 338. The reply may be encapsulated within a protocol-compliant packet 328, then placed in an "envelope" 318 for transmission over the distributed data network to client 160.

0026[0023] The subsystems shown in Figure 2 generally require configuration to be able to perform their portions of the logical flow described with reference to Figure 3. For example, a network communication subsystem may require information about the addresses it is to respond to and the network routers it is to use; and an NFS or CIFS server may need information to control access from remote clients to data storage resources. Various subsystems may obtain their configuration information through different channels. For example, some subsystems may read a text file when starting, while others may refer to arguments passed on a command line, retrieve information from a database or registry, or query a central repository of configuration data through a network connection. Text configuration files may have a predetermined structure required by the subsystem to which they apply. Text files may contain lines of "keys" and "values;" free-form configuration directives; hierarchical settings in a representation such as extensible Markup Language ("XML"); Hypertext Markup Language ("HTML") data, or the like.

0027[0024] A basic configuration management tool might include the elements shown in Figure 4: import logic 410 may collect configuration information for various subsystems from the location where the subsystem expects to find it: for example, a configuration file in a predetermined directory 420, a command line given in a system startup script 430, or a database 440. The configuration information may be separated into "elements" 450, where an element may include an identifying name 452, a value 454, and other useful information such as the subsystem or subsystems affected by the configuration element 456 or a textual help message 458 to explain how the element is used. The set of configuration elements imported from various sources may be manipulated as a single consolidated configuration 460.

0028[0025] The configuration management tool, which may run on a management workstation 180, presents a consolidated view of configuration elements for many different subsystems and interact with a system administrator to select appropriate configuration values. When the administrator is satisfied with the configuration, export logic 470 writes the configuration elements back to the appropriate locations (Le. the same configuration files, startup scripts, and registry entries that the elements came from in the first place) and in the appropriate formats. For example, a text-based configuration file may have lines of "keys" and "values," or may be formatted as a hierarchical extensible Markup Language ("XML") file. The configuration management tool may provide a button or control to reset the newly-reconfigured system or otherwise make the new configuration effective.

0029[0026] Some embodiments may use XML as a common or intermediate format (i.e. the import logic may produce an XML document containing configuration elements drawn from various sources, and the export logic may convert an XML document back into native formats such as text files or database records). XML is a widely-adopted text-based format that permits hierarchical data to be represented effectively for convenient machine analysis and processing.

0030[0027] The tool described with reference to Figure 4 can begin to address the problem of configuring a server that is to provide infrastructure data processing services to clients by providing a consolidated view of configuration elements that affect various storage server subsystems. However, embodiments of the solution introduced here go beyond this basic capability to assist in the management of groups or "farms" of infrastructure servers. As mentioned earlier, groups of servers that are to work together to provide a basic data processing service with redundancy and high availability may require consistent and coordinated, but not identical, configurations. (Of course, identical configurations can also be managed by the same methods.) A configuration management tool that provides some of the following functional capabilities may be useful in this environment.

0031[0028] Figure 5 is a flow chart outlining a first functional capability of an embodiment of the invention: identifying differences between system configurations. A first data processing system configuration is retrieved (510), perhaps by querying a remote system ("pulling" the configuration from the remote system). Retrieving the configuration may involve importing the configuration elements from a number of different sources, as described earlier. The importing may be performed on the remote system, so that the management station receives commonly-formatted elements; or on the management system, after raw configuration data (files, command lines, and database tables) are transmitted from the remote system. [0029] A second data processing system configuration is also retrieved (520). The second configuration may come from a second remote system or from a data repository containing a previously-stored copy of the first system's configuration.

0032[0030] Next, the two configurations are compared, element by element (530) to detect differences between the configurations. Some differences between the configurations may be expected (for example, when two similar but not identical servers' configurations are compared, their hostnames and Internet Protocol ("IP") addresses will probably differ). An embodiment may maintain a list of identifiers of configuration elements that are expected to be different, and remove expected differences from the list of all differences found (540), leaving only unexpected differences. Finally, the differences detected (except for those that may have been filtered out in 540) are reported to a user (550). Reports may list all differences or only unexpected differences in a plain text format, in XML or HTML, or in a database-friendly format such as comma- separated values ("CSV") or records for the Excel™ database by Microsoft Corporation of Redmond, Washington ("XLS records"). [0031] An embodiment of the invention may repeat this procedure on a configurable schedule to produce a periodic report of changes between a system's actual configuration and its expected configuration. Differences may be reported by printing them as a list, sending them as electronic mail, or generating a notification event at a network management station. Periodic configuration monitoring may help guard against inadvertent or malicious tampering with the configurations of a plurality of servers that may be detected as unexpected configuration element differences. For example, an unexpected change in the identities of users who are authorized to administer the server, or an unexpected disk partition allocation, may indicate a security breach. [0032] Another embodiment of the invention can assist in deploying similar (but not necessarily identical) configurations to a number of data processing systems. As shown in Figure 6, a server may include (among other subsystems such as those shown in Figure 2) communication logic 220 to receive a base (or "generic") configuration 610 from a management workstation 180, override logic 620 to alter a configuration element of the base configuration 610 according to a local condition 630 and produce a localized configuration 660, and data exchange logic 650 to interact with a data storage client according to the altered configuration 660. In such a system, unpacking logic 670 may be used to extract configuration elements from the localized version 660 of the consolidated configuration 610 received from the management station 180 and prepare a text file 682, a database entry 684, an XML file 686, and/or a registry entry 688, as appropriate, to configure the various subsystems of the server. Packing logic 690 may be used to perform the reverse operation, preparing a consolidated configuration from various text files, database entries, XML files, and/or registry entries. Information about configuration elements to override and the appropriate values to use may be stored in a local database 640. [0033] The "override" logic described above with reference to Figure 6 can also be implemented at the management workstation. Consider the following example: an administrator wishes to manage ten storage servers in a remote data center. The servers are to be configured similarly and cooperate closely, so the administrator establishes a group containing the servers and creates a generic configuration for them. The generic configuration may be transmitted ("pushed") to (or received - pulled - by) the servers periodically, as described above.

0033[0034] Now, the administrator is asked to enable a new feature on one of the servers so that it can be evaluated before being enabled for all ten servers. A configuration change to enable the feature can be entered and stored at the management workstation, and the change can be marked to override the generic configuration (which leaves the new feature disabled) for only one of the ten servers. Subsequently, only that server will receive the non-standard configuration, and only that server will activate the feature. Later, if the evaluation period is successful, the change can be set to override the generic configuration for all ten servers, or the generic configuration itself can be modified to enable the feature.

0034[0035] Note that the preceding example involves various actions by a system administrator only to explain a possible usage scenario and the motivations behind it. Embodiments of the invention can operate without human interaction.

0035[0036] Configuration elements in a generic configuration may contain a flag or permission indicator to control whether they are allowed to be changed or overridden by override logic on a server system. In some embodiments, several layers of permissions may be implemented, so that a generic configuration may be distributed to a regional data center containing a plurality of server systems (e.g., storage servers), and a first group of generic configuration elements may be overridden to adapt the configuration to the common needs of the servers at the regional data center. Then, the region- appropriate configuration may be forwarded to individual servers, which may apply server-specific overrides to a different (though possibly overlapping) set of configuration elements.

0036[0037] In a manner similar to the periodic configuration retrieval and check described with reference to Figure 5, systems like those shown in Figure 6 may periodically retrieve (or a management station may periodically send) a generic configuration which will be adapted according to local overrides and applied to the system. The periodic reconfiguration may be used to ensure that inadvertent or malicious changes do not remain in effect for an extended period of time. For example, the system could be reconfigured every hour, every day, or every week, as appropriate to the threat environment. In some embodiments, a configuration change alerter may detect configuration changes and transmit the current operating configuration to a central management station whenever a change occurs. System administrators at the central management station may review the change and decide whether to permit it (and possibly to ignore the difference between the system's configuration and a template configuration in subsequent comparison cycles) or to reject it, push an earlier configuration, and restrict local changes to certain configuration elements.

0037[0038] Figure 7 shows a sample user interface screen 700 of a software program to assist system managers in administering a plurality of similar servers. This user interface can be implemented by a tool such as described with reference to Figures 5-6. The interface has a field 710 which identifies the name of a file server, the configuration of which has been compared with a generic configuration identified in field 720. A portion 730 of the configuration elements (fields) are shown in a hierarchical "tree" view, and a particular configuration element 740 is highlighted to indicate that a difference has been detected between the two configurations. "Detail" window 750 shows more information about the difference; in this example, the system configuration has been altered from the generic configuration to remove "execute" access from a remote user to a storage facility of the data processing system. This alteration is described in further detail in "Help" sub-window 760. The user interface permits a manager (e.g., a storage network administrator) to identify and examine such differences, and to take several different actions when a difference is encountered. First, the manager may activate control 772 to reject the change and revert to the generic configuration. Second, the manager may activate control 775 to accept the change and ignore the difference in future configuration comparisons. Finally, the manager may forbid local changes at the remote file server by activating control 777. [0039] Once the manager is satisfied with the configuration displayed by the software program, he may activate control 793 to cause the configuration to be pushed to one or more remote systems, or control 796 to cause the configuration to be saved for later reference. A scheduling control or menu selection (not shown) may lead to an interface screen such as that shown in Figure 8, element 800, where the manager can arrange for periodic configuration action 810 {e.g. compare or update) of a target server 820 at a particular period and time 830, as described with reference to other embodiments.

0038[0040] Figure 9 is a combination logical/physical system diagram, showing where parts of embodiments may be deployed in a complete system. A system administration console 900, which can be implemented in software and installed on a computer in a network operations center ("NOC"), includes a user interface 910, configuration manipulation logic 920, and update scheduling logic 930. Reference configuration data may be stored in a local database 940. The administration console 900 communicates with infrastructure servers in data centers 110 and 130 over a distributed data network 140. Within each data center, similar servers 111 and 112 or dissimilar servers 131 and 132 are configured to provide data storage services to clients (not shown). User data is stored on mass storage devices 113, 114, 133 and 134, which may be configured as Redundant Arrays of Independent Disks. Each infrastructure server includes logical subsystems as shown at element 950: system management logic 240, configuration manipulation logic 960 (which complements the system administration console's configuration manipulation logic 920), and protocol handlers 970 to interact with clients and fulfill client requests.

0039[0041] An embodiment of the invention may be a machine-readable medium having stored thereon instructions which cause a programmable processor to perform operations as described above. In other embodiments, the operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed computer components and custom hardware components. [0042] A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to Compact Disc Read-Only Memory (CD-ROMs), Read-Only Memory (ROMs), Random Access Memory (RAM) and Erasable Programmable Read-Only Memory (EPROM).

0040[0043] The term "logic," as used herein, can include, for example, hardwired circuitry, programmable circuitry (e.g. Field-Programmable Gate Arrays ("FPGAs")), software instructions executed by a programmable processor, or any combination thereof.

0041[0044] The applications of the present invention have been described largely by reference to specific examples and in terms of particular allocations of functionality to certain hardware and/or software components. However, those of skill in the art will recognize that flexible fileserver configuration management can also be achieved by software and hardware that distribute the functions of embodiments of this invention differently than herein described. Such variations and implementations are understood to be captured according to the following claims.