Approach for managing state information by a group of servers that services a group of clients
Claim Score by NHIP
Abstract
An approach for managing state information by a group of servers that services a group of clients is disclosed. One server is designated as the primary server and is responsible for generating state information to be used by both the servers and the clients. The remaining servers are designated as secondary servers that help to manage the group, but which do not generate the state information. When the primary server fails or is not available due to a network partition event, one of the secondary servers changes role to become the primary server. With a network partition event, each partition can have a primary server, and when the network partition heals, one of the primary servers changes role back to being a secondary server. As a result, the group of servers maintains a consistent set of state information without being vulnerable to the single failure of a server.
Term
Projected expiry 12 August 2029.
- Priority and filed
- Published
- Today
- Projected expiry
36 claims: 6 independent, 30 dependent
- 1A method for managing state information by a first group of data processing servers that services a second group of clients, comprising:electronically causing one server of the first group of data processing servers to be designated as a primary server that generates the state information to be used by both the first group of data processing servers and the second group of clients;electronically causing the remaining servers of the first group of data processing servers to be designated as secondary servers that receive the state information from the primary server but that do not generate the state information;and in response to detecting that the primary server cannot communicate with at least one secondary server, electronically causing one of the secondary servers to be designated as the primary server that generates additional state information to be used by both the first group of data processing servers and the second group of clients.
- 9Broadest claimClaim Score 58, broad(NHIP)A method for managing state information to be used by a first group of data processing servers and a second group of clients, comprising:electronically receiving the state information from a first server at a second server, wherein: the first server is included in the first group of data processing servers and is responsible for generating the state information;the second server is included in the first group of data processing servers and is not responsible for generating the state information;the state information is to be used by both the first group of data processing servers and the second group of clients;the second server electronically detecting that the first server is not able to communicate with the second server;in response to said detecting, the second server determining that the second server should be responsible for generating the state information in place of the first server;and the second server generating the state information.
- 13An apparatus for managing state information by a first group of data processing servers that services a second group of clients, comprising:means for electronically causing one server of the first group of data processing servers to be designated as a primary server that generates the state information to be used by both the first group of data processing servers and the second group of clients;means for electronically causing the remaining servers of the first group of data processing servers to be designated as secondary servers that receive the state information from the primary server but that do not generate the state information;and means for in response to detecting that the primary server cannot communicate with at least one secondary server, electronically causing one of the secondary servers to be designated as the primary server that generates additional state information to be used by both the first group of data processing servers and the second group of clients.
- 21An apparatus for managing state information to be used by a first group of data processing servers and a second group of clients, comprising:means for electronically receiving the state information from a first server at a second server, wherein: the first server is included in the first group of data processing servers and is responsible for generating the state information;the second server is included in the first group of data processing servers and is not responsible for generating the state information;the state information is to be used by both the first group of data processing servers and the second group of clients;means for the second server electronically detecting that the first server is not able to communicate with the second server;in response to said detecting, means for the second server determining that the second server should be responsible for generating the state information in place of the first server;and means for the second server generating the state information.
- 25An apparatus for managing state information by a first group of data processing servers that services a second group of clients, comprising:a network interface that is coupled to a data network for receiving one or more packet flows therefrom;a processor;one or more stored sequences of instructions which, when executed by the processor, cause the processor to perform the steps of: electronically causing one server of the first group of data processing servers to be designated as a primary server that generates the state information to be used by both the first group of data processing servers and the second group of clients;electronically causing the remaining servers of the first group of data processing servers to be designated as secondary servers that receive the state information from the primary server but that do not generate the state information;and in response to detecting that the primary server cannot communicate with at least one secondary server, electronically causing one of the secondary servers to be designated as the primary server that generates additional state information to be used by both the first group of data processing servers and the second group of clients.
- 33An apparatus for managing state information to be used by a first group of data processing servers and a second group of clients, comprising:a network interface that is coupled to a data network for receiving one or more packet flows therefrom;a processor;one or more stored sequences of instructions which, when executed by the processor, cause the processor to perform the steps of: electronically receiving the state information from a first server at a second server, wherein: the first server is included in the first group of data processing servers and is responsible for generating the state information;the second server is included in the first group of data processing servers and is not responsible for generating the state information;the state information is to be used by both the first group of data processing servers and the second group of clients;the second server electronically detecting that the first server is not able to communicate with the second server;in response to said detecting, the second server determining that the second server should be responsible for generating the state information in place of the first server;and the second server generating the state information.
Independent claims6
274 paragraphs in 7 sections, as filed
FIELD OF THE INVENTION
0001The present invention generally relates to managing state information for a group of devices, and more specifically, to managing state information by a group of servers that services a group of clients.
BACKGROUND
0002The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
0003For large organizations, there are often a number of separate and geographically dispersed sites that the organization wants to connect through a network. For example, a company may have manufacturing sites, development sites, distribution sites, and a number of sales locations that are located throughout a region, country, or the world. The company wants to interconnect the sites via a network so that the sites can share information and personnel spread out among the sites can communicate with each other. A typical solution is for the company to establish a network among the sites or to purchase network service from a service provider with multi-protocol label switching (MPLS) capability that interconnects the company's various sites together in a private network. Because the company typically wants each site to be able to communicate with any other site, such a network arrangement is described as an “any to any” solution that allows any site to send packets across the private network to any other site.
0004In such a network of geographically dispersed locations, the organization generally wants to add confidentiality to the communications between the sites, such as by using one or more cryptographic techniques to encrypt and decrypt the network traffic between the sites. For example, a group key management system, such as the group domain of interpretation (GDOI) protocol defined in RFC 3547, can be used to provide cryptographic keys and policy to a group of devices in the network. As a specific example, Internet Protocol Security (IPsec) defined in RFCs 2401, 2404, and 2406 can be used to provide security associations (SAs) that define the cryptographic keys and encryption methods to be used for communications between the sites. The communications between sites can be just between two particular sites or between any number of sites, such as in the form of secure multicasts among the virtual private network (VPN) gateways that interconnect each site to the network.
0005<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram that depicts a set of sites <b>110</b>, <b>112</b>, and <b>114</b> and a key server <b>120</b> that are interconnected through a network <b>100</b> and a group key management system. For example, the GDOI group key management protocol can be used. For communications involving multiple sites, a group can be formed to include the participating sites. For example, in <figref idref="DRAWINGS">FIG. 1</figref>, sites <b>110</b>, <b>112</b>, and <b>114</b> can participate in a secure multicast, and therefore, sites <b>110</b>, <b>112</b>, and <b>114</b> are referred to as the group members.
0006Key server <b>120</b> is responsible for generating group keys and group policy, such as by establishing SAs based on IPsec. Each of sites <b>110</b>, <b>112</b>, and <b>114</b> registers with key server <b>120</b> using the group key management protocol and by providing the required authentication information. Then sites <b>110</b>, <b>112</b>, and <b>114</b> receive the current security association, denoted in <figref idref="DRAWINGS">FIG. 1</figref> as SA-<b>1</b>, with the current IPsec keys and policy from key server <b>120</b>, as depicted by arrows <b>130</b>, <b>132</b>, and <b>134</b>. As a result, sites <b>110</b>, <b>112</b>, and <b>114</b> can securely communicate with each other based on SA-<b>1</b>.
0007Because SAs are set to expire after a specified amount of time or need to be replaced if a member of the group leaves, key server <b>120</b> periodically pushes updates to the group policy in the form of new SAs, such as SA-<b>2</b> and SA-<b>3</b> depicted in <figref idref="DRAWINGS">FIG. 1</figref>. As a specific example, key server <b>120</b> sends rekey messages to sites <b>110</b>, <b>112</b>, and <b>114</b> that transmit the new SA to be used by sites <b>110</b>, <b>112</b>, and <b>114</b>, as depicted by arrows <b>130</b>, <b>132</b>, and <b>134</b>.
0008One problem with using a single group server, such as key server <b>120</b> in <figref idref="DRAWINGS">FIG. 1</figref>, is that the single group server represents a single point of failure for communications among the members of the group. For example, if key server <b>120</b> fails, sites <b>110</b>, <b>112</b>, and <b>114</b> will not receive new group keys when the current SAs expire or when a member leaves the group that would typically require generation and distribution of a new SA to preclude the leaving member from being able to read the communications for the group.
0009One approach for addressing the single point of failure problem when using a single key server is to use multiple independent key servers. However, if one group member registers with key server A and another group member registers with key server B, the two group members will receive different SAs. As a result, group members registering with different key servers cannot communicate with each other. Instead, only group members that register with the same key server can communicate using the SAs from that key server.
0010Another approach for addressing the single point of failure problem is to employ multiple groups with each group having a single key server. In order for members of the different groups to communicate, the group members must register with each key server of each group to receive the SA for each group. By having the SA from each key server, any group member can communicate with any other group member using one of the SAs. However, as the number of groups increases, the number of SAs that must be obtained and maintained by each group member increases, which represents a significant scaling problem for a large number of sites that are served by many key servers. For example, in some implementations, the number of sites can number in the hundreds or even thousands, and there can be dozens of key servers that each group member must register with and obtain the different SAs. Thereafter, while each group member has all the different SAs, each group member must identify which SA is being used for each group communication.
0011Another problem with multiple groups having different key servers is that network partitions can occur, resulting in some group members being unable to communicate with some key servers. A network partition occurs when network interconnections are unavailable resulting in the members of one group being unable to communicate with the key server for another group and possibly some members of the other group.
0012For example, if a network partition occurs, members of group A are unable to communicate with the key server for group B while members of group B cannot communicate with the key server for group A. Even if the individual members of groups A and B can communicate (even though the members of group A cannot communicate with the key server for group B and vice versa), then as new SAs are generated by each group's key server, the members of the different groups will not share the same SA, and therefore will be unable to communicate with each other.
0013Yet another approach for addressing the single point of failure problem is to employ a hierarchical arrangement of key servers, such as with the Kerberos authentication system that employs a number of key distribution centers (KDCs). With Kerberos, one KDC is specified to be the master server that maintains and modifies a database of key information. The remaining KDCs are the slave servers, each of which includes a read-only copy of the database from the master server.
0014Having multiple slave key servers with the hierarchical approach addresses the single failure problem if another slave server fails, since other slave servers can be used to obtain the keys from the database. However, the Kerberos approach is still susceptible to a single failure of the master key server, since the slave key servers are unable to create new objects or to modify current objects in their copies of the database from the master server.
0015Still another approach for addressing the single point of failure problem is to use a distributed database that allows the same copy of the database to be stored on multiple servers. Each database server acts a master that can update the copy of the database stored on that database server. However, to ensure consistency across the multiple copies of the database, changes to each object in the database must be tracked so that the changes to each object can be applied to all copies of the database in a consistent manner.
0016A distributed database is not susceptible to a single point of failure since any copy of the database is considered to be a master copy. However, ensuring consistency among multiple changes the objects within the distributed database by the multiple masters requires significantly more complexity through the use of the transaction identifiers to track multiple changes to the same object, in addition to other protocol complexities such as the use of locks and acknowledgement messages to prevent conflicting changes to an object by multiple masters.
0017Based on the foregoing, there is a clear need for improved techniques for maintaining a unified state among a group of servers. In particular, there is a need for maintaining security associations among a group of servers that distributes group keys and policy to a group of clients serviced by the group of servers.
BRIEF DESCRIPTION OF THE DRAWINGS
0018The present invention is depicted by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
0019<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram that depicts a set of sites and a key server that are interconnected through a network and a group key management system;
0020<figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref> are block diagrams that depict an overview of an arrangement for managing state information by a group of servers that services a group of clients, according to an embodiment;
0021<figref idref="DRAWINGS">FIG. 3A</figref> and <figref idref="DRAWINGS">FIG. 3B</figref> are flow diagrams that depict an overview for an approach for managing state information by a group of servers that services a group of clients, according to an embodiment;
0022<figref idref="DRAWINGS">FIG. 4A</figref> and <figref idref="DRAWINGS">FIG. 4B</figref> are block diagrams that depict an overview of an arrangement for managing state information when a network partition occurs and heals, respectively, according to an embodiment;
0023<figref idref="DRAWINGS">FIG. 5A</figref> and <figref idref="DRAWINGS">FIG. 5B</figref> are flow diagrams that depict an overview for an approach for managing state information when a network partition occurs and is healed, respectively, according to an embodiment;
0024<figref idref="DRAWINGS">FIG. 6</figref> is a block diagram depicting the format of a message, according to an embodiment;
0025<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram depicting a high level state machine, according to an embodiment;
0026<figref idref="DRAWINGS">FIG. 8</figref> is a block diagram depicting an example of a initialization state machine, according to an embodiment;
0027<figref idref="DRAWINGS">FIG. 9</figref> is a block diagram depicting an example of a secondary state machine, according to an embodiment;
0028<figref idref="DRAWINGS">FIG. 10</figref> is a block diagram depicting an example of a primary state machine, according to an embodiment; and
0029<figref idref="DRAWINGS">FIG. 11</figref> is a block diagram that depicts a computer system upon which embodiments of the invention may be implemented.
DETAILED DESCRIPTION
0030A method and apparatus for managing state information by a group of servers that services a group of clients is described. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are depicted in block diagram form in order to avoid unnecessarily obscuring the present invention.
0031In the following description, the various functions shall be discussed under topic headings that appear in the following order:
1.0 GENERAL OVERVIEW
00332.0 STRUCTURAL AND FUNCTIONAL OVERVIEW <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0034"> 2.1 Introduction </li><li id="ul0002-0002" num="0035"> 2.2 Structural Overview </li><li id="ul0002-0003" num="0036"> 2.3 Functional Overview <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0037"> 2.3.1 Initial Server Configuration </li><li id="ul0003-0002" num="0038"> 2.3.2 Primary Server Fails </li><li id="ul0003-0003" num="0039"> 2.3.3 Failed Server Returns </li></ul></li></ul></li></ul>
00403.0 PRIMARY SERVER AND SECONDARY SERVERS <ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0000"><ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0041"> 3.1 Initializing a Server </li><li id="ul0005-0002" num="0042"> 3.2 Changing Role from Secondary Server to Primary Server </li><li id="ul0005-0003" num="0043"> 3.3 Changing Role from Primary Server to Secondary Server </li><li id="ul0005-0004" num="0044"> 3.4 Functions of a Primary Server </li><li id="ul0005-0005" num="0045"> 3.5 Functions of Secondary Servers </li><li id="ul0005-0006" num="0046"> 3.6 Load Balancing Clients Among Servers </li></ul></li></ul>
00474.0 FAILURE OF A PRIMARY SERVER <ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0000"><ul id="ul0007" list-style="none"><li id="ul0007-0001" num="0048"> 4.1 Designating a New Primary Server </li><li id="ul0007-0002" num="0049"> 4.2 Return of the Failed Primary Server </li></ul></li></ul>
00505.0. HANDLING NETWORK PARTITIONS <ul id="ul0008" list-style="none"><li id="ul0008-0001" num="0000"><ul id="ul0009" list-style="none"><li id="ul0009-0001" num="0051"> 5.1 Designating a Primary Server When a Network Partition Occurs </li><li id="ul0009-0002" num="0052"> 5.2 Joining Network Partitions and Demoting a Primary Server </li><li id="ul0009-0003" num="0053"> 5.3 A Secondary Server as Intermediary Between Two Primary Servers </li></ul></li></ul>
00546.0 USING TIMERS TO DETECT AN UNAVAILBLE PRIMARY SERVER <ul id="ul0010" list-style="none"><li id="ul0010-0001" num="0000"><ul id="ul0011" list-style="none"><li id="ul0011-0001" num="0055"> 6.1 Primary Periodic Timer </li><li id="ul0011-0002" num="0056"> 6.2 Dead Primary Timer </li><li id="ul0011-0003" num="0057"> 6.3 Re-Evaluation Role Timer </li><li id="ul0011-0004" num="0058"> 6.4 New Per-User Policy Timer </li></ul></li></ul>
00597.0 PROTOCOL MESSAGES <ul id="ul0012" list-style="none"><li id="ul0012-0001" num="0000"><ul id="ul0013" list-style="none"><li id="ul0013-0001" num="0060"> 7.1 Announcement Messages </li><li id="ul0013-0002" num="0061"> 7.2 Message Format </li></ul></li></ul>
00628.0 STATES AND STATE MACHINES <ul id="ul0014" list-style="none"><li id="ul0014-0001" num="0000"><ul id="ul0015" list-style="none"><li id="ul0015-0001" num="0063"> 8.1 Local Stored State </li><li id="ul0015-0002" num="0064"> 8.2 High Level State Machine </li><li id="ul0015-0003" num="0065"> 8.3 Initialization State Machine </li><li id="ul0015-0004" num="0066"> 8.4 Secondary State Machine </li><li id="ul0015-0005" num="0067"> 8.5 Primary State Machine </li></ul></li></ul>
9.0 IMPLEMENTATION MECHANISMS AND HARDWARE OVERVIEW
10.0 EXTENSIONS AND ALTERNATIVES
1.0 General Overview
0070Techniques are provided for managing state information by a group of servers that services a group of clients. In one embodiment, an approach for managing state information by a group of data processing servers that services a group of clients includes electronically causing one server of the group of data processing servers to be designated as a primary server. The primary server generates the state information to be used by both the group of data processing servers and the group of clients. The approach also includes electronically causing the remaining servers of the group of data processing servers to be designated as secondary servers. The secondary servers receive the state information from the primary server but do not generate the state information. In response to detecting that the primary server cannot communicate with at least one of the secondary servers, the approach further includes electronically causing one of the secondary servers to be designated as the primary server. The newly designated primary server generates additional state information to be used by both the group of data processing servers and the group of clients.
0071In other aspects, the primary server distributes the state information to the secondary servers. The primary server and the secondary servers are capable of adding or removing a client from the group of clients and distributing the state information to any client of the group of clients that contacts the primary server or one of the secondary servers. Electronically causing one of the secondary servers to be designated as the primary server includes electronically causing priority information to be compared between at least two secondary servers, and based on the comparison, causing the secondary server with the highest priority to be designated as the primary server.
0072In yet other aspects, the primary server that is not able to communicate with at least one secondary server includes the primary server not being able to communicate with a subgroup of secondary servers and a subgroup of clients. One secondary server in the subgroup of secondary servers is then designated as the primary server generates additional state information to be used by the subgroup of secondary servers and the subgroup of clients, while the original primary server that is not able to communicate with the subgroup of secondary servers and the subgroup of clients can still communicate with another subgroup of secondary servers and another subgroup of clients. When the original primary server is later able to communicate with both subgroups of secondary servers and both subgroups of clients, the state information from the two primary servers is synchronized, and one primary server is designated as a secondary server while the other primary server remains the primary server that generates additional state information for both the servers and the clients.
0073In still other aspects, each client includes an ordered listing of the data processing servers that each client uses to select a particular server from which to obtain state information. The ordered listings of clients can be arranged with a different ordering for different clients so that client requests are distributed among the data processing servers. The state information includes one or more objects, and the primary server can create and destroy existing objects but cannot modify the existing objects. The primary server and the secondary server are active servers, and each server is included in a different local area network (LAN). The state information includes one or more security associations based on Internet Protocol Security (IPsec) as defined in RFCs 2401, 2404, and 2406. The data processing servers use a group key management system based on Group Domain of Interpretation (GDOI) as defined in RFC 3547. The data processing servers provide IPsec keys to the clients based on GDOI as part of either a secure multicast feature or a group virtual private network (VPN) feature.
0074In another embodiment, an approach for managing state information to be used by data processing servers and clients includes electronically receiving the state information at one server from another server. Both servers are data processing servers, with one responsible for generating the state information while the other does not generate the state information to be used by the servers and clients. When the server not responsible for generating the state information detects that the other server is not able to communicate, then the server not responsible for generating the state information determines that that server should now be responsible for generating the state information, and then that server generates the state information.
0075In other aspects, the server that is now generating the state information detects that the other server is again able to communicate, and the server sends that other server state information. The server provides priority information to the other server and receives designation information from the other server that indicates that the other server is not responsible for generating the state information.
2.0 Structural and Functional Overview
2.1 Introduction
0076An approach for managing state information generally involves a group of servers that service a group of clients, such as key servers providing security associations to group members of a secure multicast. Among the group of servers, one server is designated as the primary server and is responsible for generating the state information, such as the security associations (SAs) based on the applicable security policy. The other servers in the group of servers are designated as secondary servers who receive the state information from the primary server, but who do not generate that state information on their own. Any of the clients can communicate with any of the servers to obtain the state information, such as the security association that includes the group keys for the group members to securely communicate among each other.
0077When the primary server fails or is otherwise not able to communicate with the secondary servers, one of the secondary servers changes role from a secondary server to the primary server, and thereafter generates the state information for use by the remaining secondary servers and the clients. The original primary server may fail on its own, or the original primary server may not be able to communicate with one or more of the secondary servers due to a network partition or other communications problem. In either case, from among the remaining secondary servers, a new primary server is designated.
0078If and when the server previously designated as the primary server comes back up, the server is initialized as a secondary server, and upon seeing that another of the servers is now the primary server, remains as a secondary server. If the server did not fail but was unable to communicate (e.g., as with a network partition event) is now able to again communicate after the communications problem is fixed (e.g., as with the healing of a network partition), then two primary servers would exist at the same time. In this situation, the two primary servers synchronize their state information and exchange priority information. Then the primary server with the highest priority remains in the role of the primary server while the other primary server changes role to become a secondary server.
0079By having multiple servers with one server in the role of the primary server and the rest of the servers in the role of secondary servers, the problem of having a single point of failure is addressed since any of the servers can change role and become the primary server. However, through the designation of only one server at a time as the primary server that is responsible for maintaining the state information for use by both the servers and the clients, all servers and clients use the same state information, and the problem of trying to track and coordinate changes by multiple creators of the state information is avoided. Also, because any secondary server can potentially become the primary server, at least one server is available to create state information for use by both the servers and the clients.
0080In the event of a network partition among geographically dispersed servers, the partition that does not include the original primary server can have a secondary server change role to become the primary server for that partition and thereafter generate the state information for that partition, while the other partition continues to operate with the original primary server. Because a primary server can change role to become a secondary server if more than one primary server is identified, such as when a network partition is healed, conflicts that might arise from having multiple primary servers are avoided.
0081The techniques described herein can be used to support multiple servers that are located on different local area networks (LANs), such as when the servers are geographically dispersed across a state, region, county, hemisphere, or the world. Geographically distributing the servers reduces the changes of one problem rendering multiple servers unable to service the clients. Also, while any client can generally interact with any of the servers, one or more load balancing approaches, such as those described herein, can be used to distribute the work load in servicing the clients among the servers based on the relative proximity of the clients to the servers (e.g., each server services those clients that are geographically closest to the server).
2.2 Structural Overview
0082<figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref> are block diagrams that depict an overview of an arrangement for managing state information by a group of servers that services a group of clients, according to an embodiment. <figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref> are described in terms of key servers and group members of secure multicast, but any type of servers servicing any type of clients can be used. Also, <figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref> are described with reference to keys and security associations for securing the multicast, although other types of objects within a set of state information can be used. Finally, <figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref> are depicted as including only two key servers and two group members for simplicity, but any number of key servers and group members can be used.
0083<figref idref="DRAWINGS">FIG. 2A</figref> depicts key servers <b>210</b>, <b>220</b> and group members <b>240</b>, <b>260</b> that are interconnected through a network <b>200</b> in which key servers <b>210</b>, <b>220</b> service group members <b>240</b>, <b>260</b> as part of a secure multicast group. Thus, key servers <b>210</b>, <b>220</b> are examples of data processing servers and group members <b>240</b>, <b>260</b> are examples of clients that operate according to the techniques described herein.
0084Network <b>200</b> can be a private network from an MPLS provider or any other type of network, such as a wide area network or the Internet. The elements of <figref idref="DRAWINGS">FIG. 2A</figref> identified as group members <b>240</b>, <b>260</b> represent clients that are serviced by the group servers, such as the members of a secure multicast. In practice such elements are implemented using a “customer edge (CE) device,” typically a router, that is located at the location of the customer of an MPLS provider, and the CE device connects to a “provider edge (PE) device,” typically another router, that is part of the MPLS network. The CE device connects to the customer's network at the particular location, through which one or more end users interact, such as by using general-purpose computers that receive communications from other group members or the key servers through the CE device.
0085Typically, pair-wise connections are established between each pair of key servers in the group of key servers to facilitate communications between the key servers, such as for the exchange of state information such as the keys for the secure multicast. For example, connection <b>250</b> is a pair-wise connection between key server <b>210</b> and key server <b>220</b>. Use of pair-wise connections between the key servers does not use a significant amount of network resources since the number of key servers is generally small compared to the number of group members (e.g., tens of servers versus hundreds or thousands of clients).
0086As depicted in <figref idref="DRAWINGS">FIG. 2A</figref>, key server <b>210</b> is the primary server and therefore is responsible for generating the state information, such as by generating a key K<b>1</b> as part of a security association based on the selected IPsec policy. Key server <b>220</b> is a secondary server and therefore does not generate keys for use by the group but can distribute the keys for the group to the members of the secure multicast group.
0087For example, key server <b>210</b> is capable of distributing the key K<b>1</b> to group members <b>240</b>, <b>260</b> as depicted by arrows <b>214</b>, <b>216</b>, respectively. For the particular example depicted in <figref idref="DRAWINGS">FIG. 2A</figref>, key server <b>210</b> distributes the key K<b>1</b> to group member <b>240</b>, as depicted by arrow <b>214</b>, but key server <b>210</b> does not distribute the key K<b>1</b> to group member <b>260</b> because group member <b>260</b> obtains key K<b>1</b> from key server <b>220</b> via arrow <b>226</b>, as described below.
0088Key server <b>210</b> also distributes the key K<b>1</b> to key server <b>220</b> over a connection <b>250</b>. Although not depicted in <figref idref="DRAWINGS">FIG. 2A</figref>, if additional key servers were included, there would be additional pair-wise connections between each pair of key servers.
0089Key server <b>220</b> is capable of distributing the key K<b>1</b> to group members <b>240</b>, <b>260</b> as depicted by arrows <b>224</b>, <b>226</b>, respectively. For the particular example depicted in <figref idref="DRAWINGS">FIG. 2A</figref>, key server <b>220</b> distributes the key K<b>1</b> to group member <b>260</b>, as depicted by arrow <b>226</b>, but does not distribute key K<b>1</b> to group member <b>240</b> via arrow <b>224</b> since group member <b>240</b> obtains key K<b>1</b> from key server <b>210</b>, as described above with reference to arrow <b>214</b>.
0090The situation depicted in <figref idref="DRAWINGS">FIG. 2A</figref> represents an early time in the secure multicast involving group members <b>240</b>, <b>260</b> and key servers <b>210</b>, <b>220</b>, of which key server <b>210</b> is designated as the primary server and key server <b>220</b> is designated as the secondary server. Group members <b>240</b>, <b>260</b> engage in the secure multicast based on using key K<b>1</b> that group members <b>240</b>, <b>260</b> received from key servers <b>210</b>, <b>220</b>, respectively.
0091<figref idref="DRAWINGS">FIG. 2B</figref> represents the situation in which key server <b>210</b> has failed at a later time than the time depicted in <figref idref="DRAWINGS">FIG. 2A</figref>, and therefore the elements of <figref idref="DRAWINGS">FIG. 2B</figref> are the same as in <figref idref="DRAWINGS">FIG. 2A</figref>, except for the differences discussed herein.
0092When key server <b>210</b> fails, there is no longer a server designated as the primary server within the group of data processing servers that servers group members <b>240</b>, <b>260</b>. Upon detecting that key server <b>210</b> has failed, key server <b>220</b> becomes the primary server with the responsibility for generating state information, such as additional keys for group members <b>240</b>, <b>260</b>, such as when the current policy and group key is about to expire, as specified by the IPsec protocol.
0093For example, as depicted in <figref idref="DRAWINGS">FIG. 2B</figref>, key server <b>220</b> generates a new key, K<b>2</b>, based on the upcoming expiration of key K<b>1</b>, and distributes K<b>2</b> to members of the secure multicast group, such as by sending a rekey message to group members <b>240</b>, <b>260</b>, as depicted by arrows <b>224</b>, <b>226</b>. Following the expiration of key K<b>1</b>, group members <b>240</b>, <b>260</b> continue communicating via the secure multicast based on key K<b>2</b>. Although key server <b>220</b> and group members <b>240</b>, <b>260</b> may retain knowledge of key K<b>1</b>, because K<b>1</b> has expired (as depicted by the “X”'s that cross out K<b>1</b> within key server <b>220</b> and group members <b>240</b>, <b>260</b>), key K<b>2</b> is now the current group key. Therefore, key K<b>2</b> is subsequently used instead for secure communications among the members of the secure multicast group.
0094<figref idref="DRAWINGS">FIG. 2C</figref> represents the situation in which key server <b>210</b> has recovered from the prior failure and is now capable of servicing the group members of the secure multicast. Therefore, the elements of <figref idref="DRAWINGS">FIG. 2C</figref> are the same as in <figref idref="DRAWINGS">FIG. 2B</figref>, except for the differences discussed herein.
0095When key server <b>210</b> recovers from the prior failure, key server <b>210</b> returns to service as a secondary server. Because key server <b>220</b> is now the primary key server with the responsibility of generating new policy and keys for the secure multicast group, key server <b>210</b> is no longer responsible for key generation. Thus, when key K<b>2</b> expires, key server <b>220</b> generates a new key K<b>3</b> that is shared with key server <b>210</b> via connection <b>250</b> and then distributed to group members <b>240</b>, <b>260</b> by key servers <b>210</b>, <b>220</b>, respectively. However, key server <b>210</b> is still capable of providing keys to group members when requested, and therefore, key server <b>210</b> obtains the current key, K<b>2</b>, from the primary server, key server <b>220</b>, over pair-wise connection <b>250</b>.
0096In the example of <figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref>, after key server <b>210</b> recovers from the failure, key server <b>210</b> remains as a secondary server instead of being designated as the primary server, as in the initial situation of <figref idref="DRAWINGS">FIG. 2A</figref>. Key server <b>210</b> does not resume the role of the primary server because only one server of key servers <b>210</b>, <b>220</b> is designated as the primary server. Generally, there is no need to switch the role of the primary server and the secondary server between key servers <b>210</b>, <b>220</b> since either of key servers <b>210</b> and <b>220</b> is capable of fulfilling the responsibility of being the primary key server and generating new policy and group keys for use by the servers and clients.
0097However, in other implementations, upon the recovery of key server <b>210</b>, key servers <b>210</b>, <b>220</b> can determine if key server <b>210</b> should once again regain the responsibility of being the primary server, such as by comparing priority information and determining that key server <b>210</b> has a higher priority than key server <b>220</b>. Such an implementation may be desirable depending on the specific details of the particular implementation, such as that key server <b>210</b> is a higher performance server and therefore the preferred choice to be designated as the primary server. In that situation, key server <b>220</b> can switch role from primary server to secondary server and key server <b>210</b> can switch role from secondary server to primary server.
2.3 Functional Overview
0098<figref idref="DRAWINGS">FIG. 3A</figref> and <figref idref="DRAWINGS">FIG. 3B</figref> are flow diagrams that depict an overview for an approach for managing state information by a group of servers that services a group of clients, according to an embodiment. <figref idref="DRAWINGS">FIG. 3A</figref> and <figref idref="DRAWINGS">FIG. 3B</figref> are described in terms of the key servers and the group members of secure multicast as depicted in <figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref>, but any type of servers servicing any type of clients can be used besides those of <figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref>. Also, <figref idref="DRAWINGS">FIG. 3A</figref> and <figref idref="DRAWINGS">FIG. 3B</figref> are described with reference to keys and security associations for securing the multicast, although other types of objects within a set of state information can be used. And <figref idref="DRAWINGS">FIG. 3A</figref> and <figref idref="DRAWINGS">FIG. 3B</figref> are depicted as including only two key servers and two group members for simplicity, but any number of key servers and group members can be used. Finally, in <figref idref="DRAWINGS">FIG. 3A</figref> and <figref idref="DRAWINGS">FIG. 3B</figref>, the different blocks are grouped under the headings key server <b>210</b>, group members <b>240</b>,<b>260</b>, and key server <b>220</b> to denote which entities in <figref idref="DRAWINGS">FIG. 2A</figref>, <figref idref="DRAWINGS">FIG. 2B</figref>, and <figref idref="DRAWINGS">FIG. 2C</figref> are performing the particular functions.
2.3.1 Initial Server Configuration
0099<figref idref="DRAWINGS">FIG. 3A</figref> begins with block <b>310</b> in which key server <b>210</b> starts up as a secondary server. For example, the data processing servers can be configured to initially be designated as a secondary server upon start-up, as opposed to initially being designated as a primary server since except for the first server to start-up, there will typically already be a server designated as the primary server.
0100In block <b>314</b>, key server <b>210</b> checks for other servers and finds none. For example, following completion of start-up, key server <b>210</b> can check to see whether any other servers that are to cooperate together in servicing the clients are already up and running. Each key server can be configured to include a list of the servers in the group of data processing servers for a particular group of clients, and thus key server <b>210</b> can include a list of servers for this example implementation that identifies itself and key server <b>220</b>. In this example, key server <b>210</b> finds no other servers because key server <b>220</b> has not yet started up.
0101In block <b>318</b>, key server <b>210</b> becomes the primary server. For example, upon finding no other servers, and in particular no other server that is already designated as the primary server for the secure multicast group, key server <b>210</b> determines that key server <b>210</b>'s current role as a secondary server should be changed to designate key server <b>210</b> as the primary server. As a result, key server <b>210</b> updates a designation indicator to reflect that key server <b>210</b> is now the primary server instead of a secondary server. Following the change in role, key server <b>210</b> waits until one or more clients require state information, such as a group key for the secure multicast.
0102In block <b>320</b>, group member <b>240</b> joins the group and requests key information from key server <b>210</b>. For example, group member <b>240</b> includes an ordered list of key servers for the secure multicast group that identifies key server <b>210</b> and key server <b>220</b>. Because key server <b>210</b> is listed first in the list, group member <b>240</b> contacts key server <b>210</b> to request to join the secure multicast group. Upon successful authentication, key server <b>210</b> adds group member <b>240</b> to the group and supplies group member <b>240</b> with the initial group key, K<b>1</b>, as depicted by block <b>324</b>. Once group member <b>240</b> has the current group key, K<b>1</b>, group member <b>240</b> can participate in the multicast. However, since there are no other group members within the multicast yet, group member <b>240</b> waits for other members to join the group.
0103Although not depicted in <figref idref="DRAWINGS">FIG. 3A</figref>, if group member <b>240</b> were unable to successfully contact and interact with key server <b>210</b>, group member <b>240</b> would move on to the next key server in the ordered list, key server <b>220</b>, to attempt to join the group.
0104In block <b>330</b>, key server <b>220</b> starts up as a secondary server. For example, as explained above with respect to block <b>310</b> and key server <b>210</b>, key server <b>220</b> is configured to be designated as a secondary server upon start-up.
0105In block <b>334</b>, key server <b>220</b> checks for other servers and finds key server <b>210</b>. For example, following the completion of start-up, key server <b>220</b> can check the list of key servers for the group to determine that key server <b>210</b> should also be servicing the group members. In this example, key server <b>220</b> finds that key server <b>210</b> is present because key server <b>210</b> has already started up.
0106In block <b>338</b>, key server <b>220</b> sends a request message to key server <b>210</b> with the following information: “role=secondary, priority=20, send key information.” The first item in the message is key server <b>220</b>'s current role, with “secondary” indicating that key server <b>220</b> is designated to fulfill the role of a secondary server. The second item in the message is the priority of key server <b>220</b>, which in this example is 20. The third item in the message is a request for the message recipient to send key information, such as the current and previous group keys for the secure multicast.
0107In block <b>340</b>, key server <b>210</b> sends a reply message to key server <b>220</b> with the following information: “role=primary, priority=10, keys=K<b>1</b>.” The first item in the message is key server <b>210</b>'s current role, with “primary” indicating the key server <b>210</b> is designated to fulfill the role of the primary server. The second item is key server <b>210</b>'s priority, which is 10 in this example. The third item is the list of current keys, which includes the current group key, K<b>1</b>, which is also the only key that has been generated for the group so far.
0108In block <b>344</b>, key server <b>220</b> remains a secondary server. For example, when key server <b>220</b> receives the reply message from key server <b>210</b> that indicates that key server <b>210</b> is the primary server, key server <b>220</b> determines that there is no need for key server <b>220</b> to become the primary server, so key server <b>220</b> remains designated as a secondary server. However, in other implementations, the servers can be configured to have the server with the highest priority be the primary server, and in such an implementation, key server <b>220</b> would determine that key server <b>220</b> should become the primary server since key server <b>220</b>'s priority of 20 is greater than key server <b>210</b>'s priority of 10.
0109In block <b>350</b>, group member <b>260</b> joins the group and requests key information from key server <b>220</b>. For example, group member <b>260</b> includes an ordered list of key servers for the secure multicast group that identifies key server <b>220</b> and key server <b>210</b>. Because key server <b>220</b> is listed first in the ordered list, group member <b>260</b> first contacts key server <b>220</b> to request to join the secure multicast group. Upon successful authentication, key server <b>220</b> adds group member <b>260</b> to the group and supplies group member <b>260</b> with the initial group key, K<b>1</b>, as depicted by block <b>354</b>.
0110In this example, both group members <b>240</b> and <b>260</b> include an ordered list of key servers, but with different key servers identified in the first position in the list. Through the use of the ordered listings, the work required to serve the group members by the key servers can be shared, or load balanced. As in this example, key server <b>210</b> serviced group member <b>240</b> while key server <b>220</b> serviced group member <b>260</b>. Although either of key servers <b>210</b> and <b>220</b> could have authenticated and supplied the current group key K<b>1</b> to one or both of group members <b>240</b> and <b>260</b>, the use of different ordered listings of key servers by the group members allows the workload in servicing the group members to be shared. Therefore the use of ordered lists by the group members represents a form of load balancing of the workload in the servers servicing the clients, although a separate load balancing device is not required.
0111While a key server can easily service two group members as in this example, in other implementations involving hundreds or even thousands of group members with a limited number of key servers, say just ten key servers, dividing up the responsibility for servicing the group members prevents a bottleneck situation that could occur if all of the hundreds or thousands of group members were serviced by just a single key server.
0112As with group member <b>240</b> and although not depicted in <figref idref="DRAWINGS">FIG. 3A</figref>, if group member <b>260</b> were unable to successfully contact and interact with key server <b>220</b>, group member <b>260</b> would move on to the next key server in the list, key server <b>210</b>, to attempt to join the group.
0113In block <b>358</b>, group members <b>240</b> and <b>260</b> participate in the secure multicast using current group key K<b>1</b>. For example, now that both group member <b>240</b> and group member <b>260</b> have joined the group and know the current group key, K<b>1</b>, group members <b>240</b> and <b>260</b> can interact via network <b>200</b> by encrypting and decrypting multicast messages using group key K<b>1</b>.
0114At the end of <figref idref="DRAWINGS">FIG. 3A</figref>, key servers <b>210</b> and <b>220</b> and group members <b>240</b> and <b>260</b> are in the configuration depicted in <figref idref="DRAWINGS">FIG. 2A</figref>, namely that key server <b>210</b> is the primary group server, key server <b>220</b> is a secondary group server, group member <b>240</b> belongs to the group and has obtained key group K<b>1</b> from key server <b>210</b>, and group member <b>260</b> also belongs to the group and has obtained group key K<b>1</b> from key server <b>220</b>.
2.3.2 Primary Server Fails
0115The first portion of <figref idref="DRAWINGS">FIG. 3B</figref> depicts the functions and interactions that occur when a primary key server, such as key server <b>210</b> in <figref idref="DRAWINGS">FIG. 2A</figref>, fails and a secondary key server, such as key server <b>220</b> in <figref idref="DRAWINGS">FIG. 2A</figref>, changes designation to become the primary key server, resulting in the configuration depicted in <figref idref="DRAWINGS">FIG. 2B</figref>.
0116In block <b>360</b>, key server <b>210</b> fails. For example, key server <b>210</b> can crash or lock up, thereby rendering key server <b>210</b> unable to service group members <b>240</b>, <b>260</b>. As another example, a problem can occur in the connection between key server <b>210</b> and network <b>200</b>. As yet another example, a problem can arise within network <b>200</b> that isolates key server <b>210</b> from key server <b>220</b> and group members <b>240</b> and <b>260</b>. These last two examples are more illustrative of network partitioning events, which are described more fully below, instead of being characterized as a “failure” of key server <b>210</b> since key server <b>210</b> has not itself failed, but rather the connections between key server <b>210</b> and one or more other key servers have failed or are otherwise not functioning correctly.
0117In block <b>364</b>, a “primary period timer” pops on key server <b>220</b>. For example, key server <b>220</b> can be configured to include a timer that tracks how much time has passed since the last communication from the primary server, key server <b>210</b>. When the time since the last communication from the primary server exceeds a specified time, the time is said to have been exceeded, or the timer is said to have “popped.” The popping of the primary period timer indicates that there is a potential problem that warrants further action by key server <b>220</b> to resolve.
0118In block <b>368</b>, key server <b>220</b> sends a request message to key server <b>210</b> with the following information: “role=secondary, priority=20, send key information.” For example, the same message that key server <b>220</b> sent in block <b>338</b> is used.
0119In block <b>370</b>, a “dead primary timer” pops on key server <b>220</b>. For example, key server <b>220</b> can be configured to include a timer that tracks how much time has passed since the last communication sent to the primary server, key server <b>210</b>, that requested a reply but for which no reply has been received. When the time since the last communication sent to the primary server without a response exceeds a specified time, the timer pops, indicating that the primary server is “dead” or at least unable to respond.
0120In block <b>374</b>, key server <b>220</b> becomes the primary server. For example, upon the expiration of the “dead primary timer,” key server <b>220</b> determines that there is no other server currently performing the functions of the primary server, and so key server <b>220</b> determines that key server <b>220</b>'s current role as a secondary server should be changed to designate key server <b>220</b> as the primary server. As a result, key server <b>220</b> updates a designation indicator to reflect that key server <b>220</b> is now the primary server instead of a secondary server. Following the change in role, key server <b>220</b> waits until one or more clients require state information, such as a group key for the secure multicast when another group member joins the group, or when a current group key is about to expire and should be replaced with a new group key.
0121In block <b>378</b>, before key K<b>1</b> expires, key server <b>220</b> generates key K<b>2</b> and sends a rekey message to the group members that transmits the new group key, K<b>2</b>. For example, key server <b>220</b> can track the expiration time of the current group key, K<b>1</b>, and when the current group key is going to expire within a certain amount of time, key server <b>220</b> generates a new security association based on the current IPsec policy, including a new group key, K<b>2</b>, and then key server <b>220</b> transmits the new group key to each of group members <b>240</b>, <b>260</b> prior to the expiration of group key K<b>1</b>.
0122In block <b>380</b>, group members <b>240</b> and <b>260</b> participate in the secure multicast, using key K<b>2</b> after key K<b>1</b> expires. For example, group member <b>240</b> and group member <b>260</b> both know the new group key, K<b>2</b>, and after the initial group key, K<b>1</b>, expires, group members <b>240</b> and <b>260</b> can interact via network <b>200</b> by encrypting and decrypting multicast messages using group key K<b>2</b>.
2.3.3 Failed Server Returns
0123The second portion of <figref idref="DRAWINGS">FIG. 3B</figref> depicts the functions and interactions that occur when the failed server, such as key server <b>210</b> in <figref idref="DRAWINGS">FIG. 2B</figref>, recovers from a failure and returns to service the group members of the secure multicast, resulting in the configuration depicted in <figref idref="DRAWINGS">FIG. 2C</figref>.
0124In block <b>382</b>, key server <b>210</b> starts up as a secondary server. For example, if key server <b>210</b> had failed due to a crash or other problem that required key server <b>210</b> to reboot, key server <b>210</b> would start-up as a secondary server, just as in block <b>310</b>.
0125In block <b>384</b>, key server <b>210</b> checks for other servers and finds key server <b>220</b>. For example, just as in block <b>314</b>, key server <b>210</b> uses a list of key servers for the multicast group to check for the presence of any of the other key servers. However, unlike in block <b>314</b>, here key server <b>210</b> finds that key server <b>220</b> is already up and running.
0126In block <b>388</b>, key server <b>210</b> sends a request message to key server <b>220</b> with the following information: “role=secondary, priority=10, send key information.” As in the request message sent by key server <b>220</b> in block <b>338</b>, here key server <b>210</b> is announcing itself to key server <b>220</b> and providing some of key server <b>210</b>'s basic information plus requesting the current state information in the form of the group keys.
0127In block <b>390</b>, key server <b>220</b> sends a reply message to key server <b>210</b> with the following information: “role=primary, priority=20, keys=K<b>2</b>.” This is the same type of reply message as was sent by key server <b>210</b> in block <b>340</b>, except here the key information includes the current group key, K<b>2</b>, but not the expired group key, K<b>1</b>.
0128In block <b>394</b>, key server <b>210</b> remains a secondary server. For example, when key server <b>210</b> receives the reply message from key server <b>220</b> that indicates that key server <b>220</b> is the primary server, key server <b>210</b> determines that there is no need for key server <b>210</b> to become the primary server, so key server <b>220</b> remains designated as a secondary server. However, as noted above, other implementations can be used in which the key server with the highest priority is determined to be the primary server. Yet even in that situation, because key server <b>220</b>'s priority of 20 is greater than key server <b>210</b>'s priority of 10, key server <b>210</b> would still remain designated as a secondary server.
0129At the end of <figref idref="DRAWINGS">FIG. 3B</figref>, key servers <b>210</b> and <b>220</b> and group members <b>240</b> and <b>260</b> are in the configuration depicted in <figref idref="DRAWINGS">FIG. 2C</figref>, namely that key server <b>210</b> is a secondary server, key server <b>220</b> is the primary server, group members <b>240</b> and <b>260</b> both belong to the group and each has obtained key K<b>2</b> from key server <b>220</b> via a rekey message.
0130Subsequently, if key server <b>220</b> fails, key server <b>210</b> can assume the responsibility as the primary server in the same manner as key server <b>220</b> did when key server <b>210</b> fails.
3.0. Primary Server and Secondary Servers
3.1 Initializing a Server
0131In one embodiment, a server is initialized as a secondary server. For example, when a server in a group of data processing servers is initialized to provide service to a group of clients, the server can be configured to have an initial role of a secondary server, instead of an initial role as a primary server. Each server in the group of data processing servers is initialized as a secondary server, which is the role that all the servers except for the primary server will have. By having each server begin as a secondary server, the number of designation changes that are required is minimized because only one server is designated as the primary server.
0132In another embodiment, a server is initialized as a primary server. For example, when a server in a group of data processing servers is initialized to provide service to a group of clients, the server can be configured to have an initial designation of a primary server, instead of an initial designation as a secondary server. Each server in the group of data processing servers is initialized as a primary server, although only one server will remain designated as the primary server. For example, as each server is initialized, a comparison of the newly initialized primary server's priority to that of the current primary server can be made, and the server with the highest priority is determined to be the primary server, with the other server changing role to that of a secondary server.
0133In yet another embodiment, one or more servers are initialized as a primary server, while the remaining servers are initialized as a secondary server. For example, if some of the servers are particularly suited to be the primary server, such as a result of having the best capability to generate state information or otherwise having little or no other processing responsibilities, can be initialized as a primary server, and then use priority information to determine which server should continue as the primary server. The remaining servers are configured to initialize as secondary servers, although if necessary, one or more of the secondary servers can change designation to primary server, if the previous primary server is no longer available.
3.2 Changing Role from Secondary Server to Primary Server
0134A server can change role from being a secondary server to the primary server. For example, the first key server to initialize for a secure multicast group will find no other key servers, and therefore the first key server to initialize determines that the current designation as a secondary server should be change to indicate that the server is the primary server. As another example, if the primary server fails or is otherwise unavailable, a secondary server can determine that the current designation as a secondary server should be change to indicate that the server is the primary server.
0135In general, any server can change role from being a secondary server to being the primary server, although there is typically just one primary server at any given time among the group of data processing servers that are servicing the clients.
0136The determination to change role is generally made by a secondary server itself, although in other implementations, another server, device, mechanism, user, or administrator can instruct a secondary server to change designation to become the primary server.
3.3 Changing Role from Primary Server to Secondary Server
0137A server can change role from being the primary server to a secondary server to the primary server. For example, if the primary server is no longer able to communicate with one or more secondary servers, yet the primary server has not itself suffered a failure, a network partitioning event has likely occurred in which communications from and to the primary server are disrupted, resulting in two partitions, one of which includes the primary server and the other of which includes one or more secondary servers.
0138After the network partitioning problem is resolved and the primary server can again communicate with the other servers and clients, the primary server will typically find that another server is designated as the primary server for those servers and clients that were in the other partition. When the two partitions are joined, there will be two primary servers among the group of data processing servers servicing the clients. As a result, one of the primary servers changes role to become a secondary server.
0139When there are two primary servers for a limited time, as in the case of a network partitioned being fixed and the joining of the two partitions, the two primary servers can synchronize each primary server's state information to determine a consistent set of state information for both primary servers, and then the primary servers can compare priority information to determine which primary server should remain as the primary server (e.g., the server with the highest priority) and which primary server should change role to become a secondary server.
0140For example, if a network partition has existed for some time, each partition will have a primary server that is responsible for generating state information for the clients and secondary servers within the given partition. Over time, additional state information is generated and distributed among the secondary servers and clients. In the context of a secure multicast, each partition will operate as a separate secure multicast using keys generated by the partition's primary server.
0141When the network partitions are later joined together, clients are initially unable to communicate because the state information for the two partitions will typically be different. In the context of a secure multicast, this means that each partition has a different current group key, and therefore the group members in one partition are unable to communicate with the group members of the other partition following the joining of the partitions. Following the join, the two primary key servers synchronize their different key sets, so that each has full knowledge of any keys that were generated by the other partition when the partition existed. The synchronized group keys can then be distributed to the clients by one or more of the key servers.
0142After the state information is synchronized following the joining of the partitions, the two primary servers determine which server should remain as primary. For example, the primary server with the highest priority will remain as primary while the other primary server will change designation to secondary. Thereafter, the primary server is responsible for generating new state information, such as new keys, and distributing the new keys to the secondary servers, from which the clients obtain the new keys either by request or by rekey messages.
3.4 Functions of a Primary Server
0143One function of a primary server that is different than that of a secondary server is the generation of state information by creating one or more objects. For example, in a secure multicast, only the primary key server generates new security associations based on IPsec, which means that only the primary key server generates new group keys for the secure multicast group. Once the primary server has generated the state information, the primary server communicates the state information to the secondary servers. Thereafter, the clients receive state information from either the primary server or the secondary servers.
0144In some implementations, the primary server is also the only server responsible for deleting or destroying state information. For example, if a group member leaves a secure multicast, the group key typically needs to be changed to prevent the leaving group member from later being able to decrypt the multicast communications. In this situation, the primary server is responsible for not only generating the new group key, but also for specifying that the old group key is no longer valid and should be removed from the current keys for the group. Note that destroying the old group key by specifying that the old group key is no longer valid is different than a group key that expires, following which the expired group key is removed from the state information for the group by the primary and secondary servers.
0145In one embodiment, the primary server is capable of creating new objects to include within the state information or destroying existing objects within the state information, but the primary server is not capable of modifying existing objects within the state information. For example, in the multicast context, the primary server can create or delete SAs and group keys, but the primary server does not modify an existing SA or an existing group key.
0146The primary server can be characterized as an “active” server because the primary server is able to respond to requests from clients and other servers and can send messages to clients and servers, in contrast to a server that is “not active” or in “standby” that only acts if the active server is unable to act.
0147The primary server can be characterized as a “master” server because the primary server is the only server able to create or destroy state information, whereas the secondary servers can be characterized as “slave” servers that can only read the state information but cannot create or destroy state information.
0148In general, the remaining functions of the primary server are the same as those for the secondary servers, as described below. However, in some implementations, the primary server can be responsible for only generating the state information and communicating the state information to the secondary servers, and as a result, the primary server does not perform the other functions of the secondary servers, as described below.
3.5 Functions of Secondary Servers
0149The secondary servers can share state information with the primary server, with the other secondary servers, and with the clients serviced by the servers. For example, in a secure multicast, the secondary key servers can obtain security associations and group keys from the primary server or other secondary servers, and then the secondary key servers can provide the security associations and group keys to clients, either in response to clients joining the group or proactively as part of rekey messages prior to a current security association expiring.
0150Secondary servers also manage the membership of the group of clients. For example, in a secure multicast, the secondary key servers can handle members joining a secure multicast group and members leaving the secure multicast group.
0151In general, the secondary servers can provide any type of service to the clients and can interact in any manner with the other servers except that the secondary servers cannot create or destroy the state information to be used by both the servers and the clients. For example, in a secure multicast, if a secondary server is handling the departure of a group member, the secondary server can inform the primary server, the other secondary servers, and the remaining group members of the departure, but only the primary server can generate a new group key to be used after the departing member has left the group.
3.6 Load Balancing Clients Among Servers
0152In one embodiment, clients can be load balanced among a group of data processing servers. For example, a client can include an ordered listing of the servers that are available to service the client, and the client makes requests of the servers based on the ordered listing, working through the list until the client finds a server with which the client can communicate to obtain the desired response.
0153As a specific example, if the client has a request to make of a server, the client sends the request to the server that is listed first in the ordered listing. If that server fails to provide a suitable response, the client sends the request to the next server on the list, and so on, until the client's request receives a suitable response. When making a subsequent request, the client can either start at the top of the ordered listing, or begin with the last server from whom the client received a suitable response.
0154Different clients can have different ordered listings. For example, some clients can have a list such as KS1, KS2, and KS3 to denote key servers <b>1</b>, <b>2</b>, and <b>3</b>. Other clients can have a list such as KS2, KS3, and KS1. Yet other clients can have a list such as KS3, KS2, and KS1. Clients using the first list contact KS1 first, whereas clients using the second list contact KS2 first, and client with the third list contact KS3 first. As a result, over time client requests can be distributed among the three key servers, instead of most going to KS1, as would be the case if each client had the first list, thereby providing for a basic form of load balancing of client request among the group of servers without the use of a traditional load-balancing device.
0155The ordering of the servers on the client ordered lists can be developed based on the particular factors of a given implementation. For example, for a geographically dispersed group of clients in a secure multicast that includes clients in numerous countries and in which there are a limited number of national or regional key servers, the client lists can be ordered to specify the servers based on proximity to the client. As a specific example, each client's list of the servers has the closest server listed first, then the next closest server, and so on.
0156In another embodiment, an anycast address, such as in Internet Protocol version 6 (IPv6) Anycast Address, is used in place of a server address. Routing considerations would determine which server in the group of servers would receive the request, typically based on which server is closest to the client. The degree of load balancing that results would be similar to the geographical distribution of the clients that are making the requests.
0157In yet another embodiment, a round-robin domain name server (DNS) is used to distribute requests. A request would specify a server by hostname instead of an IP address, and the DNS translates the name to a specific IP address. The DNS server can include the addresses of the group of servers so that individual requests are mapped to the server addresses in a round-robin manner, thereby distributing the client requests among the servers.
4.0 Failure of a Primary Server
4.1 Designating a New Primary Server
0158Following the failure of a primary server, one of the secondary servers can be designated as the new primary server using any of a number of approaches. For example, as discussed above with respect to <figref idref="DRAWINGS">FIG. 2B</figref> and <figref idref="DRAWINGS">FIG. 3B</figref>, key server <b>220</b> determines that because key server <b>210</b> has failed, key server <b>220</b> should change role from being a secondary server to the primary server.
0159As another example, if there are multiple secondary servers, the secondary servers determine which secondary server should become the primary server. For example, the secondary servers can compare priority information to determine which secondary server has the highest priority and therefore determine that the secondary server with the highest priority should become the primary server. As yet another example, the first secondary server to detect that the primary server has failed is determined to be the secondary server that should become the primary server. Also, the secondary server that is the least busy can be selected as the secondary server that should become the primary server. Yet another example is that a secondary server can be randomly selected from among the secondary servers to become the primary server. Another example is that the secondary server with the first IP address is determined to be the secondary server that should assume the role of the primary server. In general, any election approach can be used to determine which secondary server should become the primary server when the need for a secondary server to assume the role of the primary server is identified.
0160In general the determination of which secondary server is to become the new primary server following the failure of the current primary server is made by one or more of the secondary servers themselves. However, in other implementations, another device besides the secondary servers themselves can aid in all or part of the election process to determine which secondary server should become the primary server, based on an appropriate set of criteria, such as the priority information or the secondary server workloads. Regardless of how a secondary server is selected, when the need to designate a primary server arises, one secondary server changes role to become the new primary server when the current primary server is no longer able to serve as the primary server.
4.2 Return of the Failed Primary Server
0161If and when a failed primary server recovers from the failure, the server returns to servicing the clients. As discussed above with respect to <figref idref="DRAWINGS">FIG. 2C</figref> and <figref idref="DRAWINGS">FIG. 3B</figref>, the server returns as a secondary server and checks to see if there is a primary server, and if so, the server remains designated as a secondary server.
0162However, if the server is not able to identify another server as the primary server, the returning server determines that the returning server should change designation from secondary server to the primary server, and then does so. This may occur if there are no other secondary servers that are available to service the clients or when the failed server returns so quickly following the failure that the failure went undetected by any of the secondary servers.
0163In some implementations, upon recovery of the failed primary server, the failed server assumes the responsibility of being the primary server, and if a secondary server has become the new primary server in the absence of the failed server, the new primary server changes designation back to being one of the secondary servers. Such an approach may be appropriate when there is a desire to have one particular server always be the primary server, if possible. Also, if the system is configured such that the server with the highest priority is to be the primary server, then when that highest priority server recovers from a failure, that highest priority server will resume being the primary server.
0164When the failed primary server returns, whether or not the failed primary server remains as one of the secondary servers or resumes responsibility as the primary server, the returning server exchanges state information with either the current primary server or one of the secondary servers. As a result, the recovered server has the most current state information, which can be used to respond to requests from any of the clients.
5.0. Handling Network Partitions
5.1 Designating a Primary Server When a Network Partition Occurs
0165When a network partition occurs, the network is effectively divided in two smaller networks or partitions for the purposes of the data processing servers providing service to the clients. In the example of <figref idref="DRAWINGS">FIG. 2</figref>, assume that a network partition occurs such that key server <b>210</b> and group member <b>240</b> are in one partition, while key server <b>220</b> and group member <b>260</b> are in the other partition. While key server <b>210</b> is designated as the primary server and can generate and distribute state information to the group members in the corresponding partition (e.g., group member <b>240</b>), key server <b>220</b> is designated as a secondary server and there is no other server in the other partition designated as the primary server. As a result, key server <b>220</b> becomes the primary server for the other partition, and thereafter can generate state information for the group members belonging to the corresponding partition (e.g., group member <b>260</b>).
0166<figref idref="DRAWINGS">FIG. 4A</figref> and <figref idref="DRAWINGS">FIG. 4B</figref> are block diagrams that depict an overview of an arrangement for managing state information when a network partition occurs and heals, respectively, according to an embodiment. <figref idref="DRAWINGS">FIG. 4A</figref> and <figref idref="DRAWINGS">FIG. 4B</figref> are described in terms of key servers and group members of secure multicast, but any type of servers servicing any type of clients can be used. Also, <figref idref="DRAWINGS">FIG. 4A</figref> and <figref idref="DRAWINGS">FIG. 4B</figref> are described with reference to keys and security associations for securing the multicast, although other types of objects within a set of state information can be used. Finally, <figref idref="DRAWINGS">FIG. 4A</figref> and <figref idref="DRAWINGS">FIG. 4B</figref> are depicted as including only two key servers and two group members for simplicity, but any number of key servers and group members can be used.
0167<figref idref="DRAWINGS">FIG. 4A</figref> depicts the example of <figref idref="DRAWINGS">FIG. 2A</figref> after a network partition occurs, resulting in partition <b>280</b> that includes key server <b>210</b> and group member <b>240</b> and in partition <b>290</b> that includes key server <b>220</b> and group member <b>240</b>. Partition <b>280</b> can function with key server <b>210</b> acting as the primary server, with key server <b>210</b> having the responsibility for generating new keys for use by group member <b>240</b> and any other group members within partition <b>280</b> (although for clarity, no other group members within partition <b>280</b> are depicted in <figref idref="DRAWINGS">FIG. 4A</figref>). For example, <figref idref="DRAWINGS">FIG. 4A</figref> depicts key server <b>210</b> generating a new key K<b>1</b>′ and transmitting key K<b>1</b>′ to group member <b>240</b>, as depicted by arrow <b>214</b>.
0168However, partition <b>290</b> has no primary server as a result of the network partition, and therefore key server <b>220</b> eventually detects that key server <b>210</b> has “failed” (although the apparent failure is not the result of key server <b>210</b> failing but rather from the inability of key server <b>220</b> to communicate with key server <b>210</b> due to the partition through network <b>200</b>). Key server <b>220</b> determines that key server <b>220</b> should become the primary server, and thereafter key server <b>220</b> generates new keys for use by group member <b>260</b> and any other group members within partition <b>290</b> (although for clarity, no other group members within partition <b>290</b> are depicted in <figref idref="DRAWINGS">FIG. 4A</figref>). For example, <figref idref="DRAWINGS">FIG. 4A</figref> depicts key server <b>220</b> generating a new key K<b>2</b>′ and transmitting key K<b>2</b>′ to group member <b>240</b>, as depicted by arrow <b>226</b>.
0169<figref idref="DRAWINGS">FIG. 5A</figref> is a flow diagram that depicts an overview for an approach for managing state information when a network partition occurs, according to an embodiment. <figref idref="DRAWINGS">FIG. 5A</figref> is described in terms of the key servers and the group members of secure multicast as depicted in <figref idref="DRAWINGS">FIG. 4A</figref>, but any type of servers servicing any type of clients can be used besides those of <figref idref="DRAWINGS">FIG. 4A</figref>. Also, <figref idref="DRAWINGS">FIG. 5A</figref> is described with reference to keys and security associations for securing the multicast, although other types of objects within a set of state information can be used. And <figref idref="DRAWINGS">FIG. 5A</figref> is depicted as including only two key servers and two group members for simplicity, but any number of key servers and group members can be used. Finally, in <figref idref="DRAWINGS">FIG. 5A</figref>, the different blocks are grouped under the headings partition <b>280</b> and partition <b>290</b>, under which additional headings for key server <b>210</b> and group member <b>240</b> plus key server <b>220</b> and group member <b>260</b> are provided, to denote which entities in <figref idref="DRAWINGS">FIG. 4A</figref> are performing the stated functions and in which partition each entity is located.
0170In block <b>510</b>, a network partition occurs. For example, if the arrangement depicted in <figref idref="DRAWINGS">FIG. 2A</figref> undergoes a network partition, the result is the arrangement depicted in <figref idref="DRAWINGS">FIG. 4A</figref>.
0171In block <b>520</b>, key server <b>220</b> detects the “failure” of the primary server. For example, key server <b>220</b> has a “primary period timer” pop due to not receiving any messages from key server <b>210</b> in a specified time, as described above with respect to block <b>364</b>. Also, key server <b>220</b> attempts to contact key server <b>210</b>, as in block <b>368</b> above, but is not successful based on a dead primary timer popping, as in block <b>370</b> above. Observe that while key server <b>220</b> believes that key server <b>210</b> has failed, key server <b>210</b> has not failed but is only unable to communicate with key server <b>220</b> due to the network partition. Thus, the network partition results in key server <b>210</b> appearing to have failed from the viewpoint of key server <b>220</b>.
0172In block <b>524</b>, key server <b>220</b> becomes the primary server for partition <b>290</b>. For example, upon detecting that key server <b>210</b> has “failed,” key server <b>220</b> determines that there is no other server currently performing the functions of the primary server, and so key server <b>220</b> determines that key server <b>220</b>'s current role as a secondary server should be changed to designate key server <b>220</b> as the primary server. As a result, key server <b>220</b> updates a designation indicator to reflect that key server <b>220</b> is now the primary server instead of a secondary server.
0173In block <b>530</b>, key server <b>220</b> sends a rekey message with key K<b>2</b>′. For example, prior to the expiration of key K<b>1</b>, key server <b>220</b> generates a new group key K<b>2</b>′ and sends a rekey message to the group members of partition <b>290</b>, such as group member <b>260</b>.
0174In block <b>534</b>, group member <b>260</b> and any other group member within partition <b>290</b> uses the new key K<b>2</b>′. For example, group member <b>260</b> can interact with other group members within partition <b>290</b> (although no other group members are depicted in partition <b>290</b> in <figref idref="DRAWINGS">FIG. 4A</figref>) via the portion of network <b>200</b> that is within partition <b>290</b> by encrypting and decrypting multicast messages using key K<b>2</b>′. However, group member <b>260</b> cannot communicate with group member <b>240</b> or any other group members within partition <b>280</b> due to the network partition.
0175In partition <b>280</b>, which includes key server <b>210</b> and group member <b>240</b>, key server <b>210</b>, which was designated as the primary server prior to the partition, continues serving as the primary server for partition <b>280</b>.
0176In block <b>540</b>, key server <b>210</b> sends a rekey message with key K<b>1</b>′. For example, prior to the expiration of key K<b>1</b>, key server <b>210</b> generates a new key K<b>1</b>′ and sends a rekey message to the group members of partition <b>280</b>, such as group member <b>240</b>.
0177In block <b>544</b>, group member <b>240</b> and any other group member within partition <b>280</b> uses the new key K<b>1</b>′. For example, group member <b>240</b> can interact with other group members within partition <b>280</b> (although no other group members are depicted within partition <b>280</b> in <figref idref="DRAWINGS">FIG. 4A</figref>) via the portion of network <b>200</b> that is within partition <b>280</b> by encrypting and decrypting multicast messages using key K<b>1</b>′. However, group member <b>240</b> cannot communicate with group member <b>260</b> or any other group members within partition <b>290</b> due to the network partition.
0178As long as the network partition exists, key server <b>210</b> continues to serve as the primary server for partition <b>280</b> while key server <b>220</b> continues to serve as the primary server for partition <b>290</b>. Although not depicted in <figref idref="DRAWINGS">FIG. 4A</figref> and <figref idref="DRAWINGS">FIG. 5A</figref>, if each partition included additional key servers, the responsibility for acting as the primary server within each partition can change among the servers, such as when a primary server fails, as described above in Section 4.0. However, within each partition, there is generally one server at any given time that acts as the primary server. Also, each of partitions <b>280</b> and <b>290</b> could experience additional network partitioning events, resulting in more than three or more network partitions, any of which can be healed and joined as described herein.
5.2 Joining Network Partitions and Demoting a Primary Server
0179After a network partition occurs, in most case the problem that causes the network partition is eventually fixed, resulting in the healing of the network partition and subsequent joining back together of the previously separate network partitions. In this situation, because each separate network partition includes a primary server, there are two primary servers immediately following the network join. For example, in the example of <figref idref="DRAWINGS">FIG. 4A</figref>, both key servers <b>210</b> and <b>220</b> are designated as primary servers for partitions <b>280</b> and <b>290</b>, so that when the network partition is removed, there are two primary servers.
0180Because both primary servers could create new state information, yet only one is needed and only one is desired to avoid having to coordinate the two different servers creating state information, one primary server is demoted back to being a secondary server. For example, one of key servers <b>210</b> and <b>220</b> changes role back to that of a secondary server, leaving the other primary server as the only primary server among the group of data processing servers that services the clients for the group. Therefore, only one primary server is designated and has responsibility for generating state information for use by the other secondary servers and the clients.
0181<figref idref="DRAWINGS">FIG. 4B</figref> depicts the example of <figref idref="DRAWINGS">FIG. 4A</figref> after the network partition is healed, resulting in key servers <b>210</b> and <b>220</b> and group member <b>240</b> and <b>260</b> being able to communicate among each other, as in <figref idref="DRAWINGS">FIG. 2A</figref>. However, immediately after the network partition heals, there are two primary servers, key server <b>210</b> that was the primary server for partition <b>280</b> and key server <b>220</b> that was the primary server for partition <b>290</b>, yet the arrangement only needs and should only have one primary server.
0182Also, because the current group key being used by group member <b>240</b> and the other group members that were in partition <b>280</b> is K<b>1</b>′, and because the current group key being sued by group member <b>260</b> and the other group members that were in partition <b>290</b> is K<b>2</b>′ (recalling that the old group key K<b>1</b> has expired), group members <b>240</b> and <b>260</b> cannot communicate due to the use of different group keys. Thus, a new group key is needed that can be distributed to all group members so that all group members can communicate securely as part of the multicast group.
0183Upon being able to communicate, key servers <b>210</b> and <b>220</b> synchronize the state information that each has by exchanging each server's list of keys. For example, key server <b>210</b> communicates key K<b>1</b>′ over connection <b>250</b> to key server <b>220</b>, and likewise key server <b>220</b> communicates key K<b>2</b>′ over connection to key server <b>210</b>. In addition to synchronizing the keys, each key server transmits the server's current role and priority information. In this example, both of key servers <b>210</b> and <b>220</b> are designated as primary servers, but key server <b>210</b>'s priority is 10 while key server <b>220</b>'s priority is 20.
0184Based upon the priority information, key server <b>220</b> determines that no change is role is necessary because key server <b>220</b>'s priority is greater than key server <b>210</b>'s priority. However, key server <b>210</b> determines that because key server <b>220</b> has a higher priority, key server <b>210</b> should change role to become a secondary server. Thus, as depicted in <figref idref="DRAWINGS">FIG. 4B</figref>, key server <b>210</b> is a secondary server while key server <b>220</b> is the primary server.
0185Comparing <figref idref="DRAWINGS">FIG. 4B</figref>, which depicts the arrangement after the joining of network partitions <b>280</b> and <b>290</b>, to <figref idref="DRAWINGS">FIG. 2A</figref>, which depicts the arrangement prior to the network partition occurring, shows that the designation of primary server has moved from key server <b>210</b> to key server <b>220</b>. Yet in both cases, both key server <b>210</b> and <b>220</b> have the same keys based on synchronizing the key information following the joining of the network partitions.
0186<figref idref="DRAWINGS">FIG. 5B</figref> is a flow diagram that depicts an overview for an approach for managing state information when a network partition is healed, according to an embodiment. <figref idref="DRAWINGS">FIG. 5B</figref> is described in terms of the key servers and the group members of secure multicast as depicted in <figref idref="DRAWINGS">FIG. 4B</figref>, but any type of servers servicing any type of clients can be used besides those of <figref idref="DRAWINGS">FIG. 4B</figref>. Also, <figref idref="DRAWINGS">FIG. 5B</figref> is described with reference to keys and security associations for securing the multicast, although other types of objects within a set of state information can be used. And <figref idref="DRAWINGS">FIG. 5B</figref> is depicted as including only two key servers and two group members for simplicity, but any number of key servers and group members can be used. Finally, in <figref idref="DRAWINGS">FIG. 5B</figref>, the different blocks are grouped under the headings key server <b>210</b>, group members <b>240</b>, <b>260</b>, and key server <b>220</b> to denote which entities in <figref idref="DRAWINGS">FIG. 4B</figref> are performing the stated functions.
0187In block <b>550</b>, the network partition heals. For example, the underlying network outage or other problem that resulted in partitions <b>280</b> and <b>290</b> is fixed, thereby allowing key servers <b>210</b> and <b>220</b> and group members <b>240</b> and <b>260</b> to again be able to communicate with each other.
0188In block <b>554</b>, key server <b>210</b> sends an announcement message to key server <b>220</b> with the following information: “role=primary, priority=10, keys=K1′.” For example, upon detecting that key server <b>220</b> is available via connection <b>250</b>, key server <b>210</b> sends a message similar to the reply messages of blocks <b>340</b> and <b>390</b> to key server <b>220</b>.
0189Similarly, in block <b>558</b>, key server <b>220</b> sends an announcement message to key server <b>210</b> with the following information: “role=primary, priority=20, keys=K2′.” For example, upon detecting that key server <b>210</b> is available via connection <b>250</b>, key server <b>220</b> sends a message similar to the reply messages of blocks <b>340</b> and <b>390</b> to key server <b>210</b>.
0190As part of blocks <b>554</b> and <b>558</b>, upon receipt of the announcement messages, key servers <b>210</b> and <b>220</b> synchronize the keys each server originally had plus the keys received from the other server. For example, key server <b>210</b> that receives keys K<b>1</b> and K<b>2</b>′ from key server <b>220</b> determines that while key server <b>210</b> already knew about key K<b>1</b>, key server <b>210</b> did not know about key K<b>2</b>′, and therefore, key server <b>210</b> adds key K<b>2</b>′ to the list of keys maintained by key server <b>210</b>. Similarly, key server <b>220</b> determines that while K<b>1</b> is an old key, key K<b>1</b>′ is new and therefore is added to the list of keys maintained by key server <b>220</b>.
0191In block <b>560</b>, key server <b>220</b> remains the primary server. For example, key server <b>220</b> compares the priority information from key server <b>210</b> (e.g., “priority=10”) to key server <b>220</b>'s own priority information (e.g., “priority=20”), and because key server <b>210</b>'s priority is less, key server <b>220</b> remains as the primary server. As a result, key server <b>220</b> remains responsible for generating key information for the group.
0192However, in block <b>564</b>, key server <b>210</b> becomes secondary. For example, key server <b>210</b> compares the priority information from key server <b>220</b> (e.g., “priority=20”) to key server <b>210</b>'s own priority information (e.g., “priority=10”), and because key server <b>220</b>'s priority is greater, key server <b>210</b> changes role from primary server to secondary server. As a result, key server <b>210</b> no longer generates new keys for the group members, and instead key server <b>210</b> distributes the key information generated by key server <b>220</b> to the group members, as required.
0193In block <b>570</b>, key server <b>220</b> sends a rekey message with key K<b>2</b>″. For example, after synchronizing keys with key server <b>210</b>, key server <b>220</b> determines that some group members do not know about key K<b>2</b>′, and therefore key server <b>220</b> determines that a new key should be generated, even if the current keys being used, keys K<b>1</b>′ and K<b>2</b>′, have not yet expired. As a result, key server <b>220</b> generates a new key K<b>2</b>″, and distributes new key K<b>2</b>″ to key server <b>210</b> via connection <b>250</b>. In addition, key server <b>220</b> distributes the new key K<b>2</b>″ to group member <b>260</b>, such as is depicted in <figref idref="DRAWINGS">FIG. 4B</figref> by arrow <b>226</b>.
0194In block <b>574</b>, key server <b>210</b> updates the keys to include new key K<b>2</b>″. For example, key server <b>210</b> adds key K<b>2</b>″ to the list of keys maintained by key server <b>210</b>, and thereafter, key server <b>210</b> can distribute new key K<b>2</b>″ to group member <b>240</b>, such as is depicted in <figref idref="DRAWINGS">FIG. 4B</figref> by arrow <b>214</b>.
0195In block <b>580</b>, group members <b>240</b> and <b>260</b> participate in the secure multicast using new key K<b>2</b>″. For example, group member <b>240</b> and group member <b>260</b> both know the new group key, K<b>2</b>″, and thereafter can interact via network <b>200</b> by encrypting and decrypting multicast messages using key K<b>2</b>″.
5.3 A Secondary Server as Intermediary Between Two Primary Servers
0196While some network partitions effectively split a network into two separate partitions, such as partitions <b>280</b> and <b>290</b> depicted in <figref idref="DRAWINGS">FIG. 4A</figref>, other network partitions are not necessarily so widespread that there are two completely separate partitions. For example, network <b>200</b> in <figref idref="DRAWINGS">FIG. 4A</figref> could undergo network failures that cripple connection <b>250</b> and some other connections within network <b>200</b>, leaving key server <b>210</b> and key server <b>220</b> unable to communicate, but there are sufficient remaining connections that group members <b>240</b> and <b>260</b> and possibly other servers can communicate.
0197For example, assume that there is a third key server in an arrangement, such as that in <figref idref="DRAWINGS">FIG. 4A</figref>, and that the network partition prevents key server <b>210</b> and <b>220</b> from communicating due to the failure of connection <b>250</b>, but that a third key server remains able to communicate with both key server <b>210</b> and <b>220</b>. If the third key server is the primary server, there is no problem, as the third key server remains able to distribute the state information generated to both secondary servers.
0198However, if the third key server is a secondary server, then the primary server, say key server <b>210</b>, cannot communicate directly with the key server <b>220</b> that is designated as a secondary server, along with the third key server. In this situation, the third key server can exchange state information with both key servers <b>210</b> and <b>220</b>, thereby effectively acting as an intermediary between key servers <b>210</b> and <b>220</b>. As a result, any new state information that is created by key server <b>210</b> can be communicated to both the third key server directly and to key server <b>220</b> indirectly via the third key server.
6.0 Using Timers to Detect an Unavailble Primary Server
0199An unavailable primary server can be detected by one or more secondary servers using any suitable mechanism, such as a primary periodic timer and/or a dead primary timer. In addition, a re-evaluation role timer can be used to delay changing role from being a secondary server to the primary server, so as to allow time to receive announcements from any other secondary servers that are also about to change designation to the primary server.
6.1 Primary Periodic Timer
0200In some implementations, a primary periodic timer is used to determine whether or not a primary server is functioning normally or has either failed or is otherwise unreachable, such as the result of a failure of the primary server itself, a network partitioning event, or some other cause. For example, each time a secondary server receives a message from the primary server, the primary periodic timer is reset.
0201If the primary periodic timer satisfies a specified relationship with a specified value (e.g., the timer pops when the time is equal to greater than the specified time or when the time exceeds the specified time), then the secondary server is alerted to take one or more actions. For example, the secondary server can immediately assume the role of the primary server. As another example, the secondary server can contact one or more other secondary servers to determine whether any of the other secondary servers have received a message from the primary server within the specified time. As yet another example, the secondary server can attempt to verify the failure of the primary server, such as by sending a message to the primary server and waiting for a response.
6.2 Dead Primary Timer
0202In some implementations, a dead primary timer is used to determine whether or not a primary server is functioning normally or is unable to perform the functions of the primary server, such as the result of a failure of the primary server itself, a network partitioning event, or some other cause. For example, secondary servers can periodically send announcement or request messages to the primary server, or a secondary server can use a dead primary timer in response to the pop of a primary periodic timer, in order to see if the primary server responds to a message sent from the secondary server.
0203If the dead primary timer satisfies a specified relationship with a specified value (e.g., the time pops when the time is equal to greater than the specified time or when the time exceeds the specified time), then the secondary server can determine that based on the lack of a timely response from the primary server, that the primary server is dead or at least unavailable to fulfill the responsibilities of being the primary server by being unable to create new state information.
6.3 Re-Evaluation Role Timer
0204In some implementations, a re-evaluation role timer is used to delay the change in role for a server from being a secondary server to the primary server. The delay allows time for the secondary server to receive any announcements form other secondary servers that are about to assume the role of the primary server. If such announcements are received, the secondary server can re-evaluate whether to assume the role of the primary server, such as by comparing priority information. As a result, repeated changes by the secondary servers to and from the role of primary server, and the resulting changes in and distribution of state information, can be minimized or avoided.
6.4 New Per-User Policy Timer
0205In some implementations, a new per-user policy timer is used, such as when the logical key hierarchy (LKH) approach as defined in RFC 2627 is in effect for the group. LKH is an approach for efficiently re-keying a large group of clients for the purpose of excluding one or more group members, who have left the group, been compromise, or for some other reason should no longer be able to securely interact with the group. LKH typically employs a logical key tree in which each group member has a unique key, plus the logical keys above the group member in the logical key tree.
0206Per-user keys are the unique user specific keys that are used to effectively eject group members from the group. The new per-user policy timer will periodically pop so that the server will send any new per-user LKH keys that have been recently generated, so that the other servers know those recently generated LKH keys. Thus, such per-user keys become part of the group policy and are therefore examples of the types of objects that can be created by secondary servers and included in the state information for the group. However, such objects are only for use by a limited number of servers and group members, based on the LKH logical key tree, and therefore are not the type of objects within a set of state information that is only generated by the primary server for use by all the other servers and the clients for the group. Thus, in those implementations, the generation and distribution of such new per-user keys is the only data flow (e.g., state information) that does not originate at a primary server.
7.0 Protocol Messages
7.1 Announcement Messages
0207In one embodiment, interactions among the data processing servers, whether designated as the primary server or as one of the secondary servers, is accomplished using one type of message. For example, an announcement message is used that specifies the priority information for the sender (e.g., the server that sends the message), the sender's role (e.g., whether primary or secondary), a flag for a return message from the recipient, and the state information for the group.
0208Priority information can be specified by the system administrator as part of the configuration of each server. The priority information can be changed after initial configuration, such as by the system administrator making adjustments based on past performance of the group.
0209Priority information can be used for any suitable purpose, such as for deciding which secondary server among two or more secondary servers should assume the role of the primary server when there is no other server designated as the primary server or when a previously designated primary server is no longer available, such as due to the primary server failing or as a result of a network partition. For example, in choosing which secondary server should become the primary server, the secondary server with the largest priority is determined to be the proper choice. If the priority of two secondary servers is the same and no other secondary server has a larger priority, then another criteria can be used to decide among the secondary servers with the highest priority, such as based on the servers respective IP addresses or some other suitable criteria.
0210The role of the sender is used to determine which server is responsible for generating state information (e.g., which server is designated as the primary server) and which servers help to manage the group, such as by distributing state information to the clients and managing group membership, but which do not generate state information (e.g., the servers designated as the secondary servers). By sharing role information, the servers can determine whether a primary server is currently available for the group, so as to allow for one secondary server to assume the role of the primary server when necessary. Also, following a network partition that is later healed, there could be two primary servers initially, and by sharing role information, the two primary servers can recognize such a situation so that one primary server can change role back to being a secondary server (e.g., by comparing priority information).
0211The flag for indicating that the sender desires a return packet for the recipient allows for the sender to request and receive state information from the recipient, which the sender can use to update the locally stored state information by the sender. By using a flag to request a return message, all message can be implemented as one-way messages, thereby simplifying the interactions among the servers. Yet by requesting and receiving a response from the recipient, the servers can more quickly determine together whether a secondary server should assume the role of the primary server, or whether which server of two primary servers should revert back to the secondary server role, such that there remains only one primary server. The request flag can be set so that the sender can obtain the recipient's role, priority, and state information, regardless of whether or not the sender already has any of that information since any of the different pieces of information can potentially change over time, such as when a system administrator makes changes based on current or past performance.
0212The state information allows the sender to specify the state information for the group that is known by the sender. For example, the sender can specify the current IPsec security association and keys that are known by the sender. Upon receipt of the state information from another server, a server can compare and update the server's own state information, thereby ensuring consistency of the state information between the servers.
0213In another embodiment, two or more message types are used instead of one type of message. For example, a request message is used that includes the sender's priority, role, and that asks for the recipient's state information, but does not include the sender's state information. A reply message is then send in response to the request message, and the reply message includes that sender's priority, role, and state information, but does not ask for the recipient's state information.
0214In other implementations, more or less information can be included in a particular type of message. For example, instead of providing priority information to each other in the messages, each data processing server can already include a list of servers for the group that identifies the priority of each server.
0215Some implementations can include one or more security features for use with protocol messages that are sent among the data processing servers. For example, authentication can be used to verify that messages are sent from a legitimate peer server. As a specific example, when using IPsec, keys that fit into the Internet Security Association and Key Management Protocol (ISAKMP) authentication framework can be used, such as pre-shared secret keys, pre-shared Rivest-Shamir-Adleman (RSA) public keys, or Public Key Infrastructure (PKI) certificates containing RSA public keys. The use of public keys is beneficial for providing true source origin authentication.
0216As another example, each server can be pre-configured, and therefore is authorized to potentially be a server for the group. Also, announcements can employ encryption, such as the level of encryption employed during GDOI registration or with the sending of rekey messages. As yet another example, messages can include a monotonically increasing sequence number to provide replay protection, with the sequence number in the message itself or as part of an encapsulation protocol, such as IPsec.
0217Lastly, prior to sending state information to another server, the sender can require the recipient server to prove that the recipient is alive and functioning properly (e.g., “liveliness proof”), such as by employing a periodic liveliness check between the servers.
7.2 Message Format
0218<figref idref="DRAWINGS">FIG. 6</figref> is a block diagram depicting the format for a message, according to an embodiment of the invention. <figref idref="DRAWINGS">FIG. 6</figref> is described with respect to a system employing GDOI and IPsec, although other types of state information can be used. Although <figref idref="DRAWINGS">FIG. 6</figref> depicts certain items within the message format, other messages can be used with fewer or more items than those of <figref idref="DRAWINGS">FIG. 6</figref> or with a different arrangement or organization of the items than that depicted in <figref idref="DRAWINGS">FIG. 6</figref>.
0219<figref idref="DRAWINGS">FIG. 6</figref> depicts an announcement message <b>600</b> that includes the following: a ISAKMP header <b>610</b>; a hash payload <b>614</b>; a coop key server header <b>620</b>; an ID) payload <b>644</b>; a sequence number payload <b>648</b>; a policy creator payload <b>650</b>; an SA payload <b>680</b>; a KD payload <b>682</b> a policy creator payload <b>684</b>; an SA payload <b>686</b>; and a KD payload <b>688</b>. As depicted in <figref idref="DRAWINGS">FIG. 6</figref>, state information <b>690</b> includes policy creator payload <b>650</b>, SA payload <b>680</b>, KD payload <b>682</b>, policy creator payload <b>684</b>, SA payload <b>686</b>, and KD payload <b>688</b>, which can include the current SA and keys for the group, such as an SA that is about to expire, and the next SA and keys for the group, to be used when the current SA expires.
0220ISAKMP header <b>610</b> is the ISAKMP header as defined in RFC 2409. Hash payload <b>614</b> is the result of applying a hash function to the payload of announcement message <b>600</b>.
0221Coop key server header <b>620</b> is depicted in <figref idref="DRAWINGS">FIG. 6</figref> as including the following: a next payload <b>622</b> (to identify the payload type of the next payload in the message), reserved <b>624</b> (reserved for later use); a payload length <b>626</b> (the length of the current payload); an MJVER <b>628</b> and an MNVER <b>630</b> (for specify the major and minor version of the cooperative server implementation, respectively); a message type <b>632</b> (to specify the type of message, such as 0 for an announcement message in some implementations, or 0 for a request message and 1 for a reply message in other implementations); one or more flags <b>634</b> (such as to specify whether or not a reply message is desired); a role <b>638</b> (whether the sender is primary—“1”—or secondary—“2”); and a priority <b>640</b> (the priority value for the sender).
0222ID payload <b>644</b> identifies the group identifier for which the message is sent. Sequence number payload <b>648</b> provides the monotonically increasing sequence number for anti-replay protection.
0223Policy creator payload <b>650</b> is depicted in <figref idref="DRAWINGS">FIG. 6</figref> as including the following: a next payload <b>652</b> (to identify the payload type of the next payload in the message); a reserved <b>654</b> (reserved for later use); a payload length <b>656</b> (the length of the current payload); an IP type <b>658</b> (the identification type of the policy creator, such as “1” for Internet Protocol version 4 (IPv4) and “2” for IPv6); and an ID payload <b>660</b> (the ID payload length, such as “4” for IPv4 and “16” for IPv6, and an identification value such as an IPv4 or IPv6 address corresponding to the identification type).
0224SA payload <b>680</b> includes the security association information, while KD payload <b>682</b> includes the key distribution information.
0225Policy creator payload <b>684</b>, SA payload <b>686</b>, and KD payload <b>688</b> are analogous to policy creator payload <b>650</b>, SA payload <b>680</b>, and KD payload <b>682</b>. Each set of policy creator payload, SA payload, and KD payload represents policy from the key server identified in the ID payload of the Policy Creator Payload.
8.0 States and State Machines
8.1 Local Stored State
0226The servers, whether the primary server or one of the secondary servers, maintain a small amount of state information locally as part of the server. In one embodiment, the locally stored state information for a server includes the pre-configured priority value for the server and a data structure for each known peer server (e.g., each of the other known data processing servers for the group).
0227The data structure for each peer server includes the following: the identity of the server (e.g., the server's IP address), the role of the server in the most recent message (e.g., primary or secondary), and the priority of the server in the most recent message.
0228The set of data structures for the known peer servers can be referred to as the peer key server database, which summarizes the known state of all other servers and can be used for a particular server to decide whether the serve should change designation (e.g., from secondary to primary if there is no known primary server or from primary to secondary if there is more than one primary server).
8.2 High Level State Machine
0229<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram depicting a high level state machine <b>700</b>, according to an embodiment. High-level state machine <b>700</b> includes an initialize state <b>710</b>, a secondary state <b>720</b>, and a primary state <b>730</b>. Example implementations of initialize state <b>710</b>, secondary state <b>720</b>, and primary state <b>730</b> are described in more detail below with respect to <figref idref="DRAWINGS">FIG. 8</figref>, <figref idref="DRAWINGS">FIG. 9</figref> and <figref idref="DRAWINGS">FIG. 10</figref>, respectively.
0230In the example of <figref idref="DRAWINGS">FIG. 7</figref>, each server begins in initialize state <b>710</b> at the startup of the system or when the system is configured. After transitioning from initialize state <b>710</b> to secondary state <b>720</b>, the server determines whether or not a primary server is already available. If there is no other primary server available, the server transitions from secondary state <b>720</b> to primary state <b>730</b>. Later the server can transition from primary state <b>730</b> to secondary state <b>720</b>, such as following a network partition event that is later healed, resulting in two primary servers for the group for which one primary server is demoted to being a secondary server.
8.3 Initialization State Machine
0231<figref idref="DRAWINGS">FIG. 8</figref> is a block diagram depicting an example of an initialization state machine <b>800</b>, according to an embodiment. Initialization state machine <b>800</b> begins with the start-up of the server, as represented by oval <b>810</b>. The server then undergoes subsystem initialization <b>820</b> so that the server is able to send an announcement message to each peer server and request a reply (to obtain the state information known to each peer server) state <b>830</b>.
0232Finally, as a result of the announcement message being sent <b>840</b>, the server enters the secondary state machine <b>900</b> of <figref idref="DRAWINGS">FIG. 9</figref>, as represented by oval <b>850</b>.
8.4 Secondary State Machine
0233<figref idref="DRAWINGS">FIG. 9</figref> is a block diagram depicting an example of a secondary state machine <b>900</b>, according to an embodiment. Secondary state machine <b>900</b> begins with entry into the secondary state as indicated by oval <b>910</b>, which can be reached from oval <b>850</b> of the initialization state machine <b>800</b> of <figref idref="DRAWINGS">FIG. 8</figref> or from oval <b>1060</b> of the primary state machine <b>1000</b> of <figref idref="DRAWINGS">FIG. 10</figref>.
0234Secondary state machine <b>900</b> then enters the wait for event state <b>920</b>. Any of several events can cause the secondary server to leave wait for event state <b>920</b>, following which the secondary server will re-enter wait for event state <b>920</b>, except when the secondary server determines that the secondary server's role should change to become the primary server.
0235The secondary server leaves wait for event state <b>920</b> if the primary periodic timer pops or the new per-user policy timer pops, as depicted by arrow <b>922</b>. The primary periodic timer pops when messages have not been received from the primary server in a specified amount of time, and the new per-user policy timer pops periodically to facilitate the distribution of any recently generated per-user keys to the other servers for the group.
0236Arrow <b>922</b> leads to the send announcement state <b>930</b>, in which the secondary server sends an announcement message to the other servers for the group. When the announcement message is sent, as depicted by arrow <b>934</b>, the secondary server transitions back to wait for event state <b>920</b>. When the announcement is sent, the primary periodic timer or new per-user policy timer is reset, depending on which timer popped that lead to send announcement state <b>930</b>.
0237The secondary server leaves wait for event state <b>920</b> if an announcement message is received, as depicted by arrow <b>924</b>, following which the secondary server enters install policy state <b>940</b>. For example, upon receipt of an announcement message, the secondary server compares the received state information to the locally stored state information, and updates the latter based on the former, as appropriate. If the announcement message specifies that a reply message is desired, an announcement message is sent to the sending server that describes the current policy. When policy installation is complete, the secondary server transitions from install policy state <b>940</b> back to wait for event state <b>920</b>, as depicted by arrow <b>944</b>.
0238The secondary server leaves wait for event state <b>920</b> if the dead primary timer pops or the re-evaluate role timer pops, as depicted by arrow <b>926</b>. The dead primary timer pops when the primary server has been non-responsive for more than a specified length of time, while the re-evaluate role timer pops when a specified amount of time has passed after the secondary server has determined that the secondary server should become the primary server. Delaying the actual move from secondary server to primary server allows time to receive announcement messages from other secondary servers that are also planning to change role to become the primary server, and if such messages are received, the secondary server can re-evaluate whether the change to become the primary server is still appropriate (e.g., whether that secondary server has the highest priority among the servers that are planning on becoming the primary server). If the secondary server determines that changing role to become the primary server is not appropriate, the primary periodic timer and the dead periodic timers are reset.
0239Arrow <b>926</b> leads to evaluate role state <b>950</b>. If evaluate role state <b>950</b> is reached because the dead primary timer pops, the primary server is removed from the list of known peer servers. If the resulting list of peers shows another peer that is designated as the primary server, then the secondary server determines that changing role to become the primary server is not appropriate, and the primary periodic timer and dead primary timer are reset. However, if no other server is designated as the primary server, the secondary server identifies the secondary server with the highest priority and updates the local state to show that that secondary server as the primary server. In either case, the secondary server has determined to stay in the secondary role and returns to the wait for event state <b>920</b>, as depicted by arrow <b>952</b>.
0240However, if in evaluate role state <b>950</b>, the secondary server determines that the priority of the secondary server is higher than the priority of any other secondary server, then the secondary server sends an announcement to the other servers that the secondary serve is the primary server. But the secondary server only resets the re-evaluate role timer and will remain in the secondary role, as depicted by arrow <b>952</b>, and then returns to the wait for event state <b>920</b>. When the re-evaluate role timer pops, as depicted by arrow <b>926</b>, the secondary server returns to evaluate role state <b>950</b>. Here the peer list is checked again to determine if another secondary server with a higher priority has announced a change to become the primary server, as described above.
0241If after the expiration of the re-evaluate role timer, the secondary server determines that changing role from secondary to primary is appropriate because the secondary server is either the only server that has announced becoming the primary server or is the server with the highest priority among those announcing becoming the primary server, the secondary server decides to switch to assume the primary role, as depicted by arrow <b>954</b>, resulting in the server entering the primary state machine <b>1000</b> of <figref idref="DRAWINGS">FIG. 10</figref>, as depicted by oval <b>960</b>.
0242However, if after the expiration of the re-evaluate role timer, the secondary server determines that another server should become the primary server, the secondary server decides to change role back to being a secondary server, resets the primary periodic timer and the dead primary timer, and then returns to wait for event state <b>920</b>.
0243Although not depicted in <figref idref="DRAWINGS">FIG. 9</figref>, a secondary server can enter the primary state at the request of the system administrator, in which case the secondary state machine <b>900</b> transitions from the wait for event state <b>920</b> to the evaluate role state <b>950</b> and then on to switching to the primary role as depicted by oval <b>960</b>, without the need to wait for a timer to pop or to evaluate whether the secondary servers' role should be changed. At the same time, if a server is currently designated as the primary server, then that primary server changes back to being the secondary server to avoid having two primary servers, although such a transition is also not depicted in either <figref idref="DRAWINGS">FIG. 9</figref> or <figref idref="DRAWINGS">FIG. 10</figref>.
8.5 Primary State Machine
0244<figref idref="DRAWINGS">FIG. 10</figref> is a block diagram depicting an example of a primary state machine <b>1000</b>, according to an embodiment. Primary state machine <b>1000</b> beings with entry into the primary state as indicated by oval <b>1010</b>, which can be reached from oval <b>960</b> of the secondary state machine <b>900</b> of <figref idref="DRAWINGS">FIG. 9</figref>.
0245Primary state machine <b>1000</b> then determines whether there is a current group policy in block <b>1020</b>. If so, then the primary server enters the wait for event state <b>1030</b>. If not, then the primary server proceeds to the create group policy state <b>1040</b>. For example, if primary state machine <b>1000</b> was entered previously by another server that subsequently failed or was rendered unavailable due to a network partition, current group policy would typically still be in use, so the primary server enters wait for event state <b>1030</b>. However, if the primary state machine <b>1000</b> is being entered into for the first time, then there is no existing group policy and the primary server must first create policy for the group, as discussed below for create group policy state <b>1040</b>.
0246Any of several events can cause the primary server to leave wait for event state <b>1030</b>, following which the primary server re-enters wait for event state <b>1030</b>, except when the primary server determines that the primary server's role should change to become a secondary server.
0247The primary server leaves wait for event state <b>1030</b> if the refresh policy timer pops, as indicated by arrow <b>1034</b>. The refresh policy timer pops on a regular basis, since the SA and keys expire after a specified amount of time. Typically, the refresh policy timer is set to pop prior to the expiration of the SA and keys so that the primary server can create new policy and distribute the new policy prior to the expiration of the current policy.
0248Arrow <b>1034</b> leads to create group policy state <b>1040</b>, where the primary server enters the create group policy state. Recall that create group policy state <b>1040</b> is also reached from block <b>1020</b> when primary state machine <b>1000</b> is entered for the first time. Here the primary server generates any needed GDOI and/or IPsec policy and associated keys, and then the primary server includes the new policy in an announcement message that is sent to the secondary servers, as depicted by arrow <b>1044</b>, following which the primary server returns to wait for event state <b>1030</b>.
0249The primary server also leaves wait for event state <b>1030</b> when the primary server receives an announcement message, as depicted by arrow <b>1038</b>, following which the primary server enters the install policy state <b>1050</b>. If a request for a reply message is specified in the announcement message that is received, the primary server sends an announcement message with the current policy that is known by the primary server. Also, if the announcement message includes policy information, the primary server's policy databases are updated with the new policy information (e.g., new per-user LKH keys).
0250After the policy items are handled, the primary server decides whether to remain as the primary server and return to the wait for event state <b>1030</b>, or whether to yield the primary role, as depicted by arrow <b>1058</b>, and therefore change back to secondary state machine <b>900</b> of <figref idref="DRAWINGS">FIG. 9</figref>, as depicted by oval <b>1060</b>. For example, if the announcement message indicates that the peer server from which the announcement message is received is also designated as the primary server, then primary state machine <b>1000</b> compares priority information. If the other server has a lower priority, the primary server returns to wait for event state <b>1030</b>, since the other server with the lower priority should change back to being a secondary server.
0251However, if the other server has a higher priority, then the primary state machine <b>1000</b> sends an announcement message with the current policy so that the other primary server has the current policy, and then the current primary server yields the primary role, as depicted by arrow <b>1058</b>, and changes role back to being a secondary server.
9.0 Implementation Mechanisms and Hardware Overview
0252The approach for managing state information by a group of servers that services a group of clients described herein may be implemented in a variety of ways and the invention is not limited to any particular implementation. The approach may be integrated into a network system or a router device, or may be implemented as a stand-alone mechanism. Furthermore, the approach may be implemented in computer software, hardware, or a combination thereof.
0253<figref idref="DRAWINGS">FIG. 11</figref> is a block diagram that illustrates a computer system <b>1100</b> upon which an embodiment of the invention may be implemented. The preferred embodiment is implemented using one or more computer programs running on a network element such as a router device. Thus, in this embodiment, the computer system <b>1100</b> is a router.
0254Computer system <b>1100</b> includes a bus <b>1102</b> or other communication mechanism for communicating information, and a processor <b>1104</b> coupled with bus <b>1102</b> for processing information. Computer system <b>1100</b> also includes a main memory <b>1106</b>, such as a random access memory (RAM), flash memory, or other dynamic storage device, coupled to bus <b>1102</b> for storing information and instructions to be executed by processor <b>1104</b>. Main memory <b>1106</b> also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor <b>1104</b>. Computer system <b>1100</b> further includes a read only memory (ROM) <b>1108</b> or other static storage device coupled to bus <b>1102</b> for storing static information and instructions for processor <b>1104</b>. A storage device <b>1110</b>, such as a magnetic disk, flash memory or optical disk, is provided and coupled to bus <b>1102</b> for storing information and instructions.
0255A communication interface <b>1118</b> may be coupled to bus <b>1102</b> for communicating information and command selections to processor <b>1104</b>. Communication interface <b>1118</b> is a conventional serial interface such as an RS-232 or RS-422 interface. An external terminal <b>1112</b> or other computer system connects to the computer system <b>1100</b> and provides commands to it using the interface <b>1114</b>. Firmware or software running in the computer system <b>1100</b> provides a terminal interface or character-based command interface so that external commands can be given to the computer system.
0256A switching system <b>1116</b> is coupled to bus <b>1102</b> and has an input interface <b>1114</b> and an output interface <b>1119</b> to one or more external network elements. The external network elements may include a local network <b>1122</b> coupled to one or more host computers <b>1124</b>, or a global network such as Internet <b>1128</b> having one or more servers <b>1130</b>. The switching system <b>1116</b> switches information traffic arriving on input interface <b>1114</b> to output interface <b>1119</b> according to pre-determined protocols and conventions that are well known. For example, switching system <b>1116</b>, in cooperation with processor <b>1104</b>, can determine a destination of a packet of data arriving on input interface <b>1114</b> and send it to the correct destination using output interface <b>1119</b>. The destinations may include host computer <b>1124</b>, server <b>1130</b>, other end stations, or other routing and switching devices in local network <b>1122</b> or Internet <b>1128</b>.
0257The invention is related to the use of computer system <b>1100</b> for managing state information by a group of servers that services a group of clients. According to one embodiment of the invention, a method and apparatus for managing state information by a group of servers that services a group of clients are provided by computer system <b>1100</b> in response to processor <b>1104</b> executing one or more sequences of one or more instructions contained in main memory <b>1106</b>. Such instructions may be read into main memory <b>1106</b> from another machine-readable medium, such as storage device <b>1110</b>. Execution of the sequences of instructions contained in main memory <b>1106</b> causes processor <b>1104</b> to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in main memory <b>1106</b>. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.
0258The term “machine-readable medium” as used herein refers to any medium that participates in providing instructions to processor <b>1104</b> for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device <b>1110</b>. Volatile media includes dynamic memory, such as main memory <b>1106</b>. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus <b>1102</b>. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
0259Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
0260Various forms of machine-readable media may be involved in carrying one or more sequences of one or more instructions to processor <b>1104</b> for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system <b>1100</b> can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector coupled to bus <b>1102</b> can receive the data carried in the infrared signal and place the data on bus <b>1102</b>. Bus <b>1102</b> carries the data to main memory <b>1106</b>, from which processor <b>1104</b> retrieves and executes the instructions. The instructions received by main memory <b>1106</b> may optionally be stored on storage device <b>1110</b> either before or after execution by processor <b>1104</b>.
0261Communication interface <b>1118</b> also provides a two-way data communication coupling to a network link <b>1120</b> that is connected to a local network <b>1122</b>. For example, communication interface <b>1118</b> may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface <b>1118</b> may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface <b>1118</b> sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
0262Network link <b>1120</b> typically provides data communication through one or more networks to other data devices. For example, network link <b>1120</b> may provide a connection through local network <b>1122</b> to a host computer <b>1124</b> or to data equipment operated by an Internet Service Provider (ISP) <b>1126</b>. ISP <b>1126</b> in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the “Internet” <b>1128</b>. Local network <b>1122</b> and Internet <b>1128</b> both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link <b>1120</b> and through communication interface <b>1118</b>, which carry the digital data to and from computer system <b>1100</b>, are exemplary forms of carrier waves transporting the information.
0263Computer system <b>1100</b> can send messages and receive data, including program code, through the network(s), network link <b>1120</b> and communication interface <b>1118</b>. In the Internet example, a server <b>1130</b> might transmit a requested code for an application program through Internet <b>1128</b>, ISP <b>1126</b>, local network <b>1122</b> and communication interface <b>1118</b>. In accordance with the invention, one such downloaded application provides for managing state information by a group of servers that services a group of clients as described herein.
0264The received code may be executed by processor <b>1104</b> as it is received, and/or stored in storage device <b>1110</b>, or other non-volatile storage for later execution. In this manner, computer system <b>1100</b> may obtain application code in the form of a carrier wave.
10.0 Extensions And Alternatives
0265In the foregoing description, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, although examples have illustrated the use of security associations and group keys in conjunction with GDOI and IPsec in the context of a secure multicast group, the use of such examples of objects that are part of the state information for use by the servers and clients are used for explanation purposes only, and embodiments of the invention are not limited to any particular type of object that is included in the state information for use by the servers and the clients. Thus, the specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The invention includes other contexts and applications in which the mechanisms and processes described herein are available to other mechanisms, methods, programs, and processes.
0266For example, the techniques described herein can be applied to a protocol for providing security alerts or for policy updates to network devices such as routers and firewalls. For example, an alert may consist of a security gateway determining that there is an ongoing attack against the network on port “X,” as in a distributed denial of service (DDOS) attack. Therefore, the alert is like an IPsec policy that is to be distributed by multiple group servers, including one primary group server and one or more secondary group servers, to all other security gateways in the network to shut down port “X.” After the DDOS attack is over, the primary group server deletes the alert, and the alert deletion is passed on to the security gateways in the network by the primary group server and the secondary group servers, as in the examples above in which new policy is transmitted via a rekey message.
0267In addition, in this description, certain process steps are set forth in a particular order, and alphabetic and alphanumeric labels are used to identify certain steps. Unless specifically stated in the disclosure, embodiments of the invention are not limited to any particular order of carrying out such steps. In particular, the labels are used merely for convenient identification of steps, and are not intended to imply, specify or require a particular order of carrying out such steps. Furthermore, other embodiments may use more or fewer steps than those discussed herein.
Contents7
15 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US2007220215A1 | Cited by | United States of America | Pre-grant |
| US11502816B2 | Cited by | United States of America | Applicant |
| CN108234631A | Cited by | China | Search report |
| US10506032B2 | Cited by | United States of America | Search report |
| US2013107882A1 | Cited by | United States of America | Pre-grant |
| US8874960B1 | Cited by | United States of America | Search report |
| US2006090003A1 | Cited by | United States of America | Pre-grant |
| US9294270B2 | Cited by | United States of America | Search report |
| US11570161B2 | Cited by | United States of America | Search report |
| US8275873B2 | Cited by | United States of America | Applicant |
| US2008175387A1 | Cited by | United States of America | Pre-grant |
| US7953978B2 | Cited by | United States of America | Search report |
| WO2010098914A1 | Cited by | World Intellectual Property Organization (WIPO) | International search |
| US2015205617A1 | Cited by | United States of America | Pre-grant |
| US9009302B2 | Cited by | United States of America | Search report |
| US7653662B2 | Cited by | United States of America | Search report |
| US8891770B2 | Cited by | United States of America | Search report |
| EP2077499A1 | Cited by | European Patent Office (EPO) | Search report |
| US8964744B2 | Cited by | United States of America | Search report |
| US2010262717A1 | Cited by | United States of America | Pre-grant |
| US10904217B2 | Cited by | United States of America | Applicant |
| US8644159B2 | Cited by | United States of America | Search report |
| US2014082348A1 | Cited by | United States of America | Pre-grant |
| US2007133539A1 | Cited by | United States of America | Pre-grant |
| US2009080657A1 | Cited by | United States of America | Pre-grant |
| US9183163B2 | Cited by | United States of America | Search report |
| US10116637B1 | Cited by | United States of America | Applicant |
| US8037303B2 | Cited by | United States of America | Search report |
| CN113169865A | Cited by | China | Search report |
| US2006282547A1 | Cited by | United States of America | Pre-grant |
| US2011164752A1 | Cited by | United States of America | Pre-grant |
| US2015195353A1 | Cited by | United States of America | Pre-grant |
| US2022333930A1 | Cited by | United States of America | Search report |
| US11349741B2 | Cited by | United States of America | Applicant |
| US9948621B2 | Cited by | United States of America | Search report |
| CN104796251A | Cited by | China | Search report |
| US2016277418A1 | Cited by | United States of America | Pre-grant |
| US2016182477A1 | Cited by | United States of America | Search report |
| US2007162750A1 | Cited by | United States of America | Pre-grant |
| US11101999B2 | Cited by | United States of America | Applicant |
| US10243928B2 | Cited by | United States of America | Search report |
| US9683858B2 | Cited by | United States of America | Applicant |
| US7930427B2 | Cited by | United States of America | Applicant |
| US2016043926A1 | Cited by | United States of America | Pre-grant |
| US2009232153A1 | Cited by | United States of America | Pre-grant |
| US2009222582A1 | Cited by | United States of America | Pre-grant |
| US2011235551A1 | Cited by | United States of America | Pre-grant |
| US10288433B2 | Cited by | United States of America | Applicant |
| US2010223458A1 | Cited by | United States of America | Pre-grant |
| US2014075017A1 | Cited by | United States of America | Search report |
| US10571288B2 | Cited by | United States of America | Applicant |
| US2008288646A1 | Cited by | United States of America | Pre-grant |
| US9912735B2 | Cited by | United States of America | Search report |
| US8090880B2 | Cited by | United States of America | Search report |
| US2014025945A1 | Cited by | United States of America | Pre-grant |
| US2007211735A1 | Cited by | United States of America | Pre-grant |
| US2015199220A1 | Cited by | United States of America | Pre-grant |
| US2009063847A1 | Cited by | United States of America | Pre-grant |
| US8458298B2 | Cited by | United States of America | Search report |
| US2007214359A1 | Cited by | United States of America | Pre-grant |
| US8625610B2 | Cited by | United States of America | Search report |
| US2009097417A1 | Cited by | United States of America | Pre-grant |
| US2013219035A1 | Cited by | United States of America | Pre-grant |
| US2007248225A1 | Cited by | United States of America | Pre-grant |
| US7926095B1 | Cited by | United States of America | Search report |
| US2009222584A1 | Cited by | United States of America | Pre-grant |
| US2015205618A1 | Cited by | United States of America | Pre-grant |
| US7386851B1 | Cited by | United States of America | Search report |
| US8095600B2 | Cited by | United States of America | Applicant |
| US7958262B2 | Cited by | United States of America | Applicant |
| US8582468B2 | Cited by | United States of America | Applicant |
| US10326678B2 | Cited by | United States of America | Applicant |
| US8959170B2 | Cited by | United States of America | Search report |
| US7978848B2 | Cited by | United States of America | Search report |
| US10630663B1 | Cited by | United States of America | Applicant |
| US2016182477A1 | Cited by | United States of America | Search report |
| US9531618B2 | Cited by | United States of America | Search report |
| US2009172799A1 | Cited by | United States of America | Pre-grant |
| US2009222581A1 | Cited by | United States of America | Pre-grant |
| US8447039B2 | Cited by | United States of America | Search report |
| US2008031246A1 | Cited by | United States of America | Pre-grant |
| US8385552B2 | Cited by | United States of America | Applicant |
| US2015010152A1 | Cited by | United States of America | Pre-grant |
| US2008168549A1 | Cited by | United States of America | Pre-grant |
| US10541814B2 | Cited by | United States of America | Applicant |
| US2008065889A1 | Cited by | United States of America | Pre-grant |
| US10778432B2 | Cited by | United States of America | Applicant |
| US8612134B2 | Cited by | United States of America | Applicant |
| US9813516B2 | Cited by | United States of America | Applicant |
| US8160255B2 | Cited by | United States of America | Applicant |
| US2016164897A1 | Cited by | United States of America | Pre-grant |
| US10536361B2 | Cited by | United States of America | Applicant |
| US10565021B2 | Cited by | United States of America | Search report |
| US2007271451A1 | Cited by | United States of America | Pre-grant |
| US10135612B1 | Cited by | United States of America | Search report |
| US8561166B2 | Cited by | United States of America | Search report |
| US10768983B2 | Cited by | United States of America | Search report |
| US10169090B2 | Cited by | United States of America | Applicant |
| US2023308263A1 | Cited by | United States of America | Search report |
| US12320650B2 | Cited by | United States of America | Search report |
4 members in 1 office; this record represents the family
Members4
| Document | Office | Kind | |
|---|---|---|---|
| US2007016663A1 | United States of America | A1 | |
| US7827262B2 | United States of America | B2 | |
| US2010318605A1 | United States of America | A1 | |
| US7991836B2 | United States of America | B2 |
54 transactions on the USPTO file
Allowed after 1 non-final rejection and 1 final rejection.
- Non-final rejections
- 1
- Final rejections
- 1
- RCEs
- 0
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Expire PatentEXP. | EXP. | |
| Maintenance Fee Reminder MailedREM. | REM. | |
| Email NotificationEML_NTR | EML_NTR | |
| Change in Power of Attorney (May Include Associate POA)PA.. | PA.. | |
| Correspondence Address ChangeC.AD | C.AD | |
| Payment of Maintenance Fee, 8th Year, Large EntityM1552 | M1552 | |
| Recordation of Patent Grant MailedPGM/ | PGM/ | |
| Patent Issue Date Used in PTA CalculationAllowedPTAC | PTAC | |
| Issue Notification MailedAllowedWPIR | WPIR | |
| Dispatch to FDCD1935 | D1935 | |
| Application Is Considered Ready for IssuePILS | PILS | |
| Issue Fee Payment VerifiedN084 | N084 | |
| Issue Fee Payment ReceivedIFEE | IFEE | |
| Mail Response to 312 Amendment (PTO-271)MN271 | MN271 | |
| Response to Amendment under Rule 312N271 | N271 | |
| Amendment after Notice of Allowance (Rule 312)AllowedA.NA | A.NA | |
| Response to Reasons for AllowanceREAS | REAS | |
| Mail Notice of AllowanceAllowedMN/=. | MN/=. | |
| Notice of Allowance Data Verification CompletedAllowedN/=. | N/=. | |
| Examiner Interview Summary Record (PTOL - 413)EXIN | EXIN | |
| Mail Advisory Action (PTOL - 303)MCTAV | MCTAV | |
| Advisory Action (PTOL-303)CTAV | CTAV | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Final ActionA.NE | A.NE | |
| Mail Examiner Interview Summary (PTOL - 413)MEXIN | MEXIN | |
| Examiner Interview Summary Record (PTOL - 413)EXIN | EXIN | |
| Mail Final Rejection (PTOL - 326)Final rejectionMCTFR | MCTFR | |
| Final RejectionFinal rejectionCTFR | CTFR | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response after Non-Final ActionA... | A... | |
| Mail Non-Final RejectionNon-final rejectionMCTNF | MCTNF | |
| Non-Final RejectionNon-final rejectionCTNF | CTNF | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response to Election / Restriction FiledELC. | ELC. | |
| Mail Restriction RequirementMCTRS | MCTRS | |
| Restriction/Election RequirementCTRS | CTRS | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Withdraw Flagged for 5/25W525 | W525 | |
| Flagged for 5/25F525 | F525 | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| PG-Pub Issue NotificationPG-ISSUE | PG-ISSUE | |
| IFW TSS Processing by Tech Center CompleteTSSCOMP | TSSCOMP | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Cleared by L&R (LARS)L128 | L128 | |
| Application Is Now CompleteCOMP | COMP | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Referred to Level 2 (LARS) by OIPE CSRL198 | L198 | |
| IFW Scan & PACR Auto Security ReviewSCAN | SCAN | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Reference capture on IDSRCAP | RCAP | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Initial Exam Team nnIEXX | IEXX |
8 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Lapsed due to failure to pay maintenance feeLapsedFP | FP | |
| Lapse for failure to pay maintenance feesLapsedPATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITYLAPS | LAPS | |
| Information on status: patent discontinuationPATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362STCH | STCH | |
| Fee payment procedureMAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITYFEPP | FEPP | |
| Maintenance fee paymentMAFP | MAFP | |
| Fee paymentFPAY | FPAY | |
| Information on status: patent grantGrantedPATENTED CASESTCF | STCF | |
| AssignmentAS | AS |
Numbers
- Publication
- 20070016663
- Application
- 11183278
Titles
- English
- Approach for managing state information by a group of servers that services a group of clients
Patent term adjustment
- A delay
- +944 daysthe office missed an examination deadline
- B delay
- +841 dayspendency past three years
- Overlap
- −275 daysdelays counted once
- Applicant delay
- −20 days
- Net adjustment
- 1,490 days
Classification
- CPC, 6
- G06F11/2041
- G06F11/2028
- H04L41/0654
- G06F11/1425
- H04L41/0894
- H04L41/0893
- IPC, 2
- G06F15 173
- G06F11 00