System and method for automatic negotiation of a security protocol

<u>FIELD OF THE INVENTION</u>

The invention relates to the field of networked computing, and more particularly to the automatic negotiation of security protocols between a security-enabled domain and one or more external nodes.

<u>BACKGROUND OF THE INVENTION</u>

Advances in networking technology have permitted network administrators and others to maintain greater and more sophisticated security controls on their networks and other installations. Microsoft Windows™ NT, 2000 and related products for instance permit administrators to deploy security-enabled network domains using the Active Directory™ (AD) structure. The publicly known Kerberos network standard likewise permits nodes within a network to authenticate each other, using a key/authentication platform. With these operating technologies, a network administrator may be able, for instance, to push rules, applications, patches, drives and other resources from a network server to individual workstations or other clients for uniform installation, on a secure basis. All machines within the security-enabled domain may be able to identify and authenticate the transmission of those and other types of data, transparently.

However, the ability to deliver rules, applications or other resources to and from a workstation becomes more difficult when that node lies outside the security-enabled domain. For instance, a company may have a collection of computers located on a local area network (LAN) but also interact with computers in a remote location which are not part of the Active Directory™ or other security-enabled domain. Communicating across the boundary of a secure domain becomes more complicated, in part because establishing a connection between a machine internal to the domain a machine outside the domain requires that an agreement be reached on a mutually supported security protocol.

Systems administrators and others are therefore forced to attempt to arrange for the entry of an external agent or node into the security-enable domain by identifying a compatible protocol between the internal and external machines, before the session takes place. For instance, an external node may be configured to communicate via a transport layer security (TLS) protocol, a Kerberos-based protocol, a secure socket layer (SSL) or other protocol with an administrative server within the security-enabled domain. That machine may in turn may in that protocol its default protocol, indicate a protocol failure, request that the protocol be switched, or make other responses to the external node or agent. Manual setting or adjusting of the security, transport and other protocols may therefore be required, a process which may be time consuming and prone to error. Other problems exist.

<u>SUMMARY OF THE INVENTION</u>

The invention overcoming these and other problems in the art relates in one regard to a system and method for automatic negotiation of a security protocol, in which secure communications with an external agent or node may be established and identities authenticated, on an automated basis without a need for administrator intervention. According to the invention in one regard, a network manager or other agent or node within a security-enabled domain may initiate an attempt to establish a secure connection with an external agent or node. That request may contain a data field indicating a set of security protocols available for use by the manager. The external agent may receive the request and compare the protocols available to the internal agent or manager to a set of protocols supported by the external agent. If a match between available protocols is found, communications may proceed based on that selected protocol. In embodiments, each of the external agent and internal agent may authenticate each other, via a key, certificate, or other authentication mechanism.

<u>BRIEF DESCRIPTION OF THE DRAWINGS</u>

<ul id="ul0001" list-style="none" compact="compact"><li>Fig. 1 illustrates a network architecture in which an embodiment of the invention may operate.</li><li>Fig. 2 illustrates a negotiation process between an internal node and an external node, according to an embodiment of the invention.</li><li>Fig. 3 illustrates a comparison between protocol tables, according to an embodiment of the invention.</li><li>Fig. 4 illustrates overall protocol negotiation processing, according to an embodiment of the invention.</li></ul>

<u>DETAILED DESCRIPTION OF EMBODIMENTS</u>

Fig. 1 illustrates an architecture in which a protocol negotiation platform and method may operate, according to an embodiment of the invention. As illustrated, in the illustrated embodiment a set of clients, servers, agents or other nodes or machines may operate in a security-enabled domain 102. Security-enabled domain may in embodiments be or include, for instance, Microsoft Windows™ Active Directory™, a Kerberos or other certificate-based or key-based domain, or other closed or secure distributed directory or other environment. Illustratively shown within security-enabled domain are an internal manager 104, which in embodiments may be or include a server or other node, as well as a set of internal agents 106 (illustrated as A1, A2 ... AN, N arbitrary).

In embodiments the set of internal agents 106 may consist of or include additional servers, workstations or other clients, or other internal agents or nodes operating within the security-enabled domain 102 and communicating with internal manager 104. In embodiments the internal manager 104 may schedule or perform network administrative functions, such as transmitting or "pushing" network rules or other data to the set of internal agents 106, such as operating guidelines for storage (e.g. RAID policies, failover criteria, memory limits), bandwidth utilization or other rules or data. When communicating these or other types of data, the internal manager 104 and set of internal agents 106 may take advantage of the security resources of security-enabled domain to ensure the integrity of the network and the distribution of rules and other data.

As illustrated, in embodiments the security-enabled domain 102 may provide authentication services, for instance using certificates such as certificate 108, which may in embodiments be or include as a certificate configured according to X.509 or other standards or formats. In embodiments keys or other mechanisms may likewise be used. As illustrated, certificate 108 may be associated with and provide authentication data for the internal manager 104. Any one of the set of internal agents 106 may authenticate the rules, instructions or other data received from the internal manager 104 by communicating certificate 108 to a certificate authority 110 for verification. Certificate authority 110 may itself be located within security-enabled domain 102, or as illustrated be located outside the security-enabled domain 102.

In embodiments, the certificate authority 110 may be or include a server or other node configured to read and decode certificate 108 or other authentication mechanisms, and return results to the set of internal agents 106 or other nodes. Each of the nodes in the set of internal agents 106 may likewise have associated with them a certificate, key or other authentication data compatible with the security-enabled domain 102. Nodes in the set of internal agents 106 may likewise communicate with and mutually authenticate each other, using certificate or other mechanisms.

In the embodiment illustrated in Fig. 1, an external agent 114 may likewise be configured to communicate with internal manager 104 via communications network 112. The external agent 114 may also be or include a server, workstation or other node or resource. The external agent 114 may likewise have associated with it a certificate 116 identifying the external agent 114 for authentication. The communications network 112 through which external agent 114 may communicate with internal manager 104 or other internal nodes in embodiments may be, include or interface to any one or more of, for instance, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a storage area network (SAN), a frame relay connection, an Advanced Intelligent Network (AIN) connection, a synchronous optical network (SONET) connection, a digital T1, T3, E1 or E3 line, Digital Data Service (DDS) connection, an ATM (Asynchronous Transfer Mode) connection, an FDDI (Fiber Distributed Data Interface), CDDI (Copper Distributed Data Interface) or other wired, wireless or optical connection. The external agent 114 may in embodiments be or include a workstation, server, wireless network-enabled device, or other node, agent or platform configured for networked communications.

Unlike prior implementations of cross-domain communication, according to embodiments of the invention the external agent 114 may initiate contact with the internal manager 104 to establish a secure connection based on a mutually compatible protocol with manually selecting a compatible protocol, in an automatic and transparent fashion. As illustrated for instance in Fig. 2, an external application 130 executing on external agent 114 may initiate contact with internal manager 104 via external negotiation engine 126. External application 130 may be or include a systems utility, productivity or other application, such as, for instance, a data backup scheduler, a firewall, virus protection or other application. External application 130 may for example require user profiles, updates or other data to perform various tasks and therefore initiate such communication with internal manager 104.

The external negotiation engine 126 may process and manage the communication requested by the external application 130, to establish a mutually compatible communications link to the internal manager 104 in security-enabled domain 102. As illustrated, in embodiments the external negotiation engine 126 may initiate and manage a negotiation module 118, illustrated as an implementation of the publicly known Simple and Protected GSS-API Negotiation (SPNEGO) protocol. Other protocols may be used. In embodiments, negotiation module 118 may be accessed, initiated or generated via an operating system of external agent 114, for instance via an application programming interface (API) or other mechanisms.

The external negotiation engine 126 may likewise include or generate an external transport specifier 120 indicating a message-based or other channel which external agent 114 may employ to execute the protocol negotiation process. For instance, in embodiments the external transport specifier 120 may specify a Security Support Provider Interface (SSPI) protocol, as part of the Microsoft .NET architecture, permitting external application 130 or other software or modules to access for instance dynamic link libraries (dlls) or other resources supporting standard cryptographic or other encoding schemes. Other protocols may be used or specified in external transport specifier 120. The external negotiation engine 126 may consequently communicate a datagram indicating that or other data to an internal negotiation engine 128 associated with internal manager 104, as illustrated in Fig. 2.

Internal negotiation engine 128 may likewise include or interface to a negotiation module 122 and internal transport specifier 124. Internal negotiation engine 128 may in turn communicate with an internal application 132 executing on or accessed by internal manager 104. Internal application 132 may, for example, be or include a systems administration, productivity or other application. Upon receipt of a request to establish communication with the internal manager 104, the internal negotiation engine 128 may establish a message-based or other channel with external agent 114 via internal transport specifier 124, for instance confirming channel communications using the SSPI protocol.

With a preliminary channel established between external agent 114 and the internal manager 104, the external negotiation engine 126 and external negotiation engine 128 may initiate protocol negotiation and reduction. In embodiments, the external agent 114 may transmit an external protocol table 134 as illustrated in Fig. 3 to the internal manager 104. The external protocol table 134 may specify which protocols external agent 114 may be configured to use. When received by the internal manager 104, the external protocol table 134 may be compared to an internal protocol table 136, indicating a set of security protocols available for use by internal manager 104. Either one or external protocol table 134 and internal protocol table 136 may include fields indicating, for example, transport layer security (TLS), secure socket layer (SSL), Kerberos, secure IP (IPSec) or other available protocols or standards. The negotiation engine 128 associated with the internal manager 104 may identify one or more protocols mutually supported by external agent 114 and internal manager 104, as illustrated in Fig. 3.

Negotiation engine 128 may in embodiments likewise communicate internal protocol table 136 to the negotiation engine 126 associated with external agent 114, for similar protocol comparison. Negotiation engine 126 and negotiation engine 128 may consequently negotiate the selection of a mutually available protocol to establish secure communications across security-enabled domain. For instance, if only a single common protocol is available to both external agent and internal manager 104, the external agent 114 and the internal manager 104 may agree to set up a session using that protocol, such as TLS or another protocol. If the negotiation engine 126 and negotiation engine 128 agree that no common protocol may be found, the attempt to establish cross-domain communications may be terminated. Conversely, if the negotiation engine 126 and negotiation engine 128 identify multiple protocols in common, a protocol may be selected based on network criteria, such as transfer speed, bit depth of keys or other security mechanisms, or other factors.

With a mutually compatible protocol in place, a secure session between external agent 114 and internal manager 104 may be established. In embodiments, for added security each of external agent 114 and internal manager may likewise perform authentication steps to verify the identity, privilege level or other security details of the opposite node. As illustrated in Fig. 1, this may be performed using certificates or other security mechanisms. External agent 114 may authenticate internal manager 104 by communicating certificate 108 to certificate authority 110. Internal manager 104 may conversely authenticate external agent 114 by communicating certificate 116 to certificate authority 110. Other security mechanisms may be used.

The type or content of data exchanged between the external agent 114 and internal manager 104 may in embodiments depend on the mutual authentication between the two nodes. For instance, access to network administrative rules or parameters may be reserved for internal or external nodes only indicating a given level of access privilege. Other authentication rules or criteria may be used. After the operational security protocol has been established and any authentication processing is complete, the external agent 114 and internal manager 104 may exchange data, applications, rules or other information. When the traffic is complete, negotiation engine 126 and negotiation engine 128 may release or terminate the communications link.

Overall network negotiation processing according to an embodiment of the invention is illustrated in Fig. 4. In step 402, processing may begin. In step 404, a request to establish a secure connection across the security-enabled network 102 may be generated in either of external agent 114, internal manager 104 or other clients, agents or nodes. In step 406, the request to establish a secure connection may be transmitted to the recipient node, whether internal manager 104, external agent 114 or another client, agent or node, the request incorporating a first protocol set compatible with the transmitting node. In step 408, the request may be received by the recipient node. In step 410, the recipient node whether internal manager 104, external agent 114 or another client, agent or node may compare the first protocol set with a second protocol set of the recipient node, to determine if a match may be found amongst available protocols.

If a match is found between the first protocol set and the second protocol set, processing may proceed to step 412 where a determination may be made whether more than one matching protocol has been found. If more than one matching protocol set has been found, processing may proceed to step 414 where one of the matching protocols may be selected for use based on protocol criteria, such as transfer speed, bit depth of keys or other security mechanisms, or other factors. Processing may then proceed to step 416, where a secure connection or session may be initiated between external agent 114 and the internal manager 104, based on the selected protocol. Likewise, if in step 412 only one matching protocol is found, processing may proceed to step 416 where a secure connection or session may be initiated. For instance, in embodiments specified ports may be opened under the TCP/IP or other communication or other protocols.

In step 418, a protocol-specific exchange may be initiated between the external agent 114 and internal manager 104, with handshaking and other steps proceeding according to the matching protocol employed. In step 420, either one of external agent 114 and internal manager 104 or both may authenticate the corresponding other node by transmitting the corresponding certificate 116 (of the external agent 114) or certificate 108 (of the internal manager) to certificate authority 108, as appropriate. In embodiments, the certificate 116 or certificate 108 or other security data may be or include certificate objects conforming to the X.509 standard, or other standards or formats. With appropriate authentication complete, processing may proceed to step 422, in which a secure connection or session may be conducted between external agent 114 and internal manager 104. For instance, network or other rules may be communicated between the two nodes, for systems administration or other purposes.

When the secure session is complete, processing may proceed to step 424 where the secure connection between the external agent 114 and internal manager 104 may be terminated or released. In step 426, processing may terminate, repeat, return to a prior processing point or take other action. Likewise if no matching protocol may be identified in the determination of step 410, processing may proceed to step 426 to terminate, repeat, return to a prior processing point or take other action.

The foregoing description of the invention is illustrative, and modifications in configuration and implementation will occur to persons skilled in the art. For instance, while the invention has generally been described in terms of a single external agent 114, in embodiments multiple external agents or nodes may be configured to automatically negotiate a matching protocol with internal manager 104 or other clients or nodes within security-enabled domain 102. Similarly, while an authentication mechanism has generally been described as being supported by a single authentication entity 110 using X.509 or other standards, in embodiments multiple authentication entities or other authentication or authorization platforms may be used. Other hardware, software or other resources described as singular may in embodiments be distributed, and similarly in embodiments resources described as distributed may be combined.

Moreover, while instances in which one or the other of nodes or agents external to the security-enabled domain 102 and nodes or agents internal to that domain have been described at times as initiating the negotiation of a secure protocol, it will be understood that any node or agent configured according to the invention, external or internal to the domain, may initiate protocol processing. Likewise either one or both of internal and external agents may initiate authentication of the opposite agent or node. The scope of the invention is accordingly intended to be limited only by the following claims.