Chassis controllers for converting universal flows

The present invention relates to a chassis controller for converting universal flow.

Many of today's enterprises have large, sophisticated networks that include switches, hubs, routers, servers, workstations and other network equipment that support a variety of connections, applications and systems. Further sophistication of computer networking, including virtual machine movement, dynamic workload, multi-tenancy, and customer-specific quality of service and security configurations, requires a more appropriate paradigm for network control. Networks have traditionally been managed with low-level settings for individual components. In many cases, the network configuration depends on the underlying network. For example, to block a user's access using an access control list (ACL) entry, you need to know the user's current IP address. More complex tasks require broader network knowledge. To force a guest user's port 80 traffic to traverse the HTTP proxy, you need to know the current network topology and location of each guest. This process becomes increasingly difficult when the network switching element is shared among a large number of users.

Correspondingly, there is a growing movement towards a new network control paradigm called Software Defined Networking (SDN). In the SDN paradigm, a network controller running on one or more servers in a network controls, maintains, and implements control logic that manages the forwarding behavior of shared network switching elements for each user. In many cases, it is necessary to be aware of the network state in order to make network management decisions. To facilitate management decisions, the network controller creates and maintains a view of the network state, on which the management application provides an application programming interface that allows access to the view of the network state.

Some of the main objectives of maintaining a large network (including both data centers and corporate networks) are scalability, mobility and multi-tenancy. Many methods taken to address one of these objectives result in interfering with at least one of the other methods. For example, one method can easily provide network mobility to virtual machines within the L2 region, but cannot extend the L2 region. Also, the continued isolation of users makes mobility very complex. Therefore, there is a need for improved solutions that meet the goals of scalability, mobility and multi-tenancy.

Some embodiments of the present invention allow several different logics for different users by one or more shared forwarding elements without even allowing several different users to control or even view each other's transfer logics. Provides a network control system that allows you to specify a data path (LDP) set. Since these shared forwarding elements are managed by the network control system to realize the LDP set, they are hereinafter referred to as managed switching elements or managed forwarding elements.

In some embodiments, a network control system is one or more controllers (also referred to below as controller instances) that allow the system to accept LDP sets from users and configure switching elements to implement these LDP sets. ). With these controllers, the system is defined by the connections between the shared switching elements in such a way that different users share the same switching element while preventing them from browsing or controlling each other's LDP sets and logical networks. The control of these shared switching elements and logical networks can be virtualized.

In some embodiments, each controller instance translates user input from logical control plane (LCP) data to logical forwarding plane (LFP) data and then LFP data to physical control plane (PCP) data. A device (eg, a general purpose computer) that executes one or more modules. These modules in some embodiments include control modules and virtualization modules. The control module allows the user to specify and populate the logical data path set (LDPS), and the virtualization module implements the specified LDPS by mapping the LDPS to the physical switching infrastructure. The control module and the virtualization module are two separate applications in some embodiments, but are part of the same application in other embodiments.

In some of the embodiments, the controller's control module receives LCP data that describes the LDPS (eg, data that describes the connection associated with the logical switching element) from the user or another source. The control module then converts this data into LFP data that will later be supplied to the virtualization module. The virtualization module then generates PCP data from the LFP data. PCP data is propagated to the managed switching element. In some embodiments, the control module and virtualization module use the nLog engine to generate LFP data from LCP data and PCP data from LFP data.

Some embodiments of network control systems use different controllers to perform different tasks. For example, in some embodiments, the network control system uses three types of controllers. The first type of controller is an application protocol interface (API) controller. The API controller receives the configuration data and the user query from the user by calling the API, and is involved in the response of the user query. The API controller further disseminates the received configuration data to other controllers. Therefore, the API controller of some embodiments is an interface between the user and the network control system.

The second type of controller is a logical controller involved in the realization of an LDP set by computing a universal flow entry, which is a general representation of the flow entry for a managed switching element that realizes the LDP set. The logical controller in some embodiments does not interact directly with the managed switching element, but pushes the universal flow entry to the physical controller, which is a third controller type.

Physical controllers in different embodiments have different mechanisms. In some embodiments, the physical controller generates customized flow entries from universal flow entries and pushes these customized flow entries down to the managed switching element. In another embodiment, the physical controller identifies, for a particular managed physical switching element, a chassis controller, which is the type of fourth controller involved in generating customized flow entries for a particular switching element, from the logical controller. Transfer the received universal flow entry to the chassis controller. The chassis controller then generates customized flow entries from the universal flow entries and pushes these customized flow entries to the managed switching element. In yet another embodiment, the physical controller generates such flow entries for some managed switching elements while instructing the chassis controller to generate customized flow entries for other managed switching elements. ..

The above-mentioned outline of the invention is a brief outline for some embodiments of the present invention and is not intended to be an outline of all the subjects of the present invention disclosed herein. The following brief description of the embodiments for carrying out the invention and the drawings referenced in the embodiments for carrying out the invention further describes the outline of the invention and the embodiments described in other embodiments. Therefore, in order to understand all the embodiments described herein, it is necessary to consider the whole of the outline of the invention, the embodiments for carrying out the invention and the brief description of the drawings. Also, since the claimed subject matter can be implemented in other specific forms without departing from the gist of the subject matter, it is limited by the detailed description in the outline of the invention, the form for carrying out the invention and the brief description of the drawings. It is not specified by the scope of the attached claims.

The novel features of the invention are defined in the appended claims, but for purposes of explanation, some embodiments of the invention will be described with reference to the following drawings.<figref num="1">The figure which shows the virtualized network system in embodiment of this invention.</figref><figref num="2">Diagram showing the switch infrastructure of a multi-user server hosting system.</figref><figref num="3">The figure which shows the network controller which manages an edge switching element.</figref><figref num="4">The figure which shows an example of a plurality of logical switching elements realized over a set of switching elements.</figref><figref num="5">The figure which shows the propagation of the instruction which controls a switching element managed through various processing layers of a controller instance.</figref><figref num="6">The figure which shows the multi-instance, distributed network control system in embodiment.</figref><figref num="7">The figure which shows an example which specifies a master controller instance for a switching element.</figref><figref num="8">The figure which shows the example of the operation of a controller instance.</figref><figref num="9">The figure which conceptually shows the software architecture for an input conversion application.</figref><figref num="10">The figure which shows the control application in embodiment of this invention.</figref><figref num="11">The figure which shows the virtualization application in embodiment of this invention.</figref><figref num="12">A diagram conceptually showing different tables in the RE output table.</figref><figref num="13">The simplified figure which shows the table mapping operation of the control application and the virtualization application in embodiment of this invention.</figref><figref num="14">The figure which shows an example of the integrated application.</figref><figref num="15">The figure which shows another example of the integrated application.</figref><figref num="16">The figure which conceptually shows an example of the architecture of a network control system.</figref><figref num="17">The figure which conceptually shows an example of the architecture of a network control system.</figref><figref num="18">The figure which shows an example of the architecture for a chassis control application.</figref><figref num="19A">、</figref><figref num="19B">A diagram showing an example of creating a tunnel between two managed switching elements based on universal physical control plane data.</figref><figref num="20">The figure which conceptually shows the process to perform to generate the customized physical control plane data from the universal physical control plane data in embodiment.</figref><figref num="21">The figure which conceptually shows the process which generates the customized tunnel flow instruction and executes to send the customized instruction to the managed switching element in embodiment.</figref><figref num="22A">、</figref><figref num="22B">The figure which conceptually shows an example of the operation of the chassis controller which converts a universal tunnel flow instruction into a customized instruction in seven different stages.</figref><figref num="23">The figure which conceptually shows the electronic system used to realize the embodiment of this invention.</figref>

Hereinafter, various details, examples and embodiments of the present invention will be described in the detailed description of the present invention. However, it will be apparent to those skilled in the art that the present invention is not limited to the embodiments described below and may be practiced without the use of the following specific details and examples.

Some embodiments of the present invention allow for different users by one or more shared forwarding elements without even allowing several different users to control or even view each other's forwarding logic. Provides a network control system that allows you to specify several different sets of LDP. Shared forwarding elements in some embodiments include virtual or physical network switches, software switches (eg, Open vSwitch), routers and / or other switching devices, and these switches and routers. And and / or any other network element (eg, load balancer, etc.) that establishes a connection with and / or other switching equipment. Such forwarding elements (eg, physical switches or routers) are referred to below as switching elements. Also called elements). In contrast to off-the-shelf switches, a software forwarding element is, in some embodiments, a switching element formed by storing its switching table and switching logic in the memory of a stand-alone device (eg, a stand-alone computer). However, in other embodiments, it is formed by storing the switching table and switching logic in the memory of a hypervisor and a device (eg, a computer) that further executes one or more virtual machines in addition to the hypervisor. It is a switching element.

Since these management / shared switching elements are managed by the network control system to realize the LDP set, they are hereinafter referred to as managed switching elements or managed forwarding elements. In some embodiments, the control system manages them by pushing PCP data to these switching elements, as described further below. In general, a switching element receives data (eg, a data packet), drops the received data packet, for example, passes a packet received from one source device to another destination device, processes the packet to the destination device. Perform one or more processing actions on the data, such as passing it to. In some embodiments, the PCP data pushed to the switching element is a data packet received by the switching element (eg, by the switching element's general purpose processor) and by the switching element (eg, the switching element's dedicated switching circuit). Is converted to physical forwarding plane data that specifies how to handle.

In some embodiments, a network control system is one or more controllers (also referred to below as controller instances) that allow the system to accept LDP sets from users and configure switching elements to implement these LDP sets. ). These controllers allow the system to connect between shared switching elements in such a way that different users share the same managed switching element while preventing them from browsing or controlling each other's LDP sets and logical networks. Control of these defined shared switching elements and logical networks can be virtualized.

In some embodiments, each controller instance is a device (eg, a general purpose computer) that executes one or more modules that convert user input from LCP to LFP and then convert LFP data to PCP data. .. These modules in some embodiments include control modules and virtualization modules. The control module allows the user to specify and populate the LDPS, and the virtualization module implements the specified LDPS by mapping the LDPS to the physical switching infrastructure. In some embodiments, the control module and the virtualization module represent data specified or mapped with respect to records written to a relational database data structure. That is, the relational database data structure stores both the logical data path input received via the control module and the physical data to which the logical data path input is mapped by the virtualization module. In some embodiments, the control application and the virtualization application are two separate applications in some embodiments, but are part of the same application in other embodiments.

So far, some examples of network control systems have been described. Hereinafter, a more detailed embodiment will be described. Chapter I describes some embodiments of network control systems. The following Chapter II describes the conversion of the universal forwarding state by the network control system. Chapter III describes the electronic systems used to implement some embodiments of the present invention.

I. Network control system A. External layer to push the flow to the control layer FIG. 1 shows a virtualized network system 100 of some embodiments of the present invention. This system allows a large number of users to create and control a large number of different sets of LDPs on a shared set of network infrastructure switching elements (eg, switches, virtual switches, software switches, etc.). When allowing a user to create and control a set of a user's logical data path (LDP) set (ie, a user's switching logic), the system allows the user to view or modify the switching logic of another user. Do not allow direct access to another user's set of LDP sets. However, the system does not allow such communication to users if different users want to pass packets to each other through virtualization switching logic.

As shown in FIG. 1, system 100 includes one or more switching elements 105 and a network controller 110. The switching element has N switching devices (N is a number of 1 or more) forming the network infrastructure switching element of the system 100. In some embodiments, network infrastructure switching elements include virtual network switches or physical network switches, software switches (eg, Open vSwitch), routers and / or other switching devices, and these switches and routers. , And / or any other network element (eg, load balancer, etc.) that establishes a connection with other switching devices. All such network infrastructure switching elements are referred to below as switching elements or forwarding elements.

The virtual switching device or physical switching device 105 generally includes a control switching logic 125 and a forwarding switching logic 130. In some embodiments, the control switching logic 125 specifies (1) the rules applied to the input packets, (2) the packets to be dropped, and (3) the packet processing methods applied to the input packets. The virtual switching element or the physical switching element 105 uses control logic 125 to populate the table that manages the transfer logic 130. The forwarding logic 130 performs a lookup operation on the input packet and forwards the input packet to the destination address.

As further shown in FIG. 1, the network controller 110 includes a control application 115. By control application 115, switching logic is specified for one or more users (eg, by one or more administrators or users) for an LDP set. The network controller 110 further includes a virtualization application 120 that translates the LDP set into control switching logic pushed to the switching device 105. In this application, the control application and the virtualization application are referred to as a "control engine" and a "virtualization engine" in some embodiments.

In some embodiments, the virtualization system 100 has two or more network controllers 110. Each network controller has a logic controller involved in designating control logic for a set of switching devices for a particular LDPS. The network controller further comprises a physical controller, and each physical controller pushes control logic to a set of switching elements that it is responsible for managing. In other words, the logical controller specifies the control logic only for the set of switching elements that realize a specific LDPS, and the physical controller controls the switching elements that it manages regardless of the LDP set realized by the switching elements. To push.

In some embodiments, the network controller virtualized application uses a relational database data structure to store a copy of the state of the switch element that is tracked by the virtualized application with respect to a data record (eg, a data tuple). These data records represent graphs of all physical or virtual switching elements and their interconnects within the physical network topology, as well as their forwarding tables. For example, in some embodiments, each switching element in the network infrastructure is represented by one or more data records in a relational database data structure. However, in other embodiments, the relational database data structure for the virtualized application stores state information about only some of the switching elements. For example, as described in more detail below, virtualization applications in some embodiments only track switching elements at the edge of the network infrastructure. In yet another embodiment, the virtualization application stores state information about the edge switching elements in the network and some non-edge switching elements in the network that facilitate communication between the edge switching elements.

In some embodiments, the relational database data structure is at the core of the control model in the virtualized network system 100. Under one method, an application controls a network by reading from a relational database data structure and writing to a relational database data structure. Specifically, in some embodiments, application control logic (1) reads the current state associated with network entity records in a relational database data structure and (2) acts on these records to network. The state can be changed. Under this model, when a virtualization application 120 needs to modify a record in a table of switching element 105 (for example, a control plane flow table), it first puts one or more records that represent the table in a relational database data structure. Write to. The virtualization application then propagates this change to the table of switching elements.

In some embodiments, the control application further uses a relational database data structure to store the logical configuration and logical state for each user-specified LDPS. In these embodiments, the information in the relational database data structure that represents the state of the actual switching element describes only a subset of all the information stored in the relational database data structure.

In some embodiments, the control and virtualization applications use a secondary data structure to store the logical configuration and logical state for a user-specified LDPS. This secondary data structure in these embodiments serves as a communication medium between different network controllers. For example, if a user specifies a particular LDPS using a logical controller that is not involved in a particular LDPS, then the logical controller is another that is involved in the particular LDPS through the secondary data structures of these logical controllers. Pass the logical configuration for a specific LDPS to the logical controller. In some embodiments, the logical controller that receives the logical configuration for a particular LDPS from the user passes the configuration data to all other controllers in the virtualized network system. Thus, the secondary storage structure in all logical controllers includes logical configuration data for all LDP sets for all users in some embodiments.

In some embodiments, the controller instance operating system (not shown) provides different embodiments of control and virtualization applications, as well as different sets of communication structures (not shown) for the switching element 105. For example, in some embodiments, the operating system is used to (1) perform physical switching for any one user, and (2) push switching logic for the user to the switching element. It provides a managed switching element with a communication interface (not shown) to and from the virtualization application 120. In some of these embodiments, a virtualized application is commonly known to specify a set of APIs that allow an external application (eg, a virtualized application) to control the control plane functionality of a switching element. It manages the control switching logic 125 of the switching element through the switch access interface. Specifically, the managed switching element communication interface implements a collection of APIs that allows a virtualized application to send records stored in a relational database data structure to the switching element using the managed switching element communication interface. ..

Two examples of such known switch access interfaces are the OpenFlow interface and the open virtual switch communication interface. They can be found in the following two references, McKeown, N. (2008), OpenFlow: Enabling Innovation in Campus Networks (http://www.openflowswitch.org//documents/openflow-wp-latest.pdf). (Yes) and Pettit, J. (2010), Virtual Switching in an Era of Advanced Edges (searchable from http://openvswitch.org/papers/dccaves2010.pdf), respectively. These two documents are incorporated herein by reference.

Note that in the case of these embodiments described above and below where a relational database data structure is used to store data records, a data structure capable of storing object-oriented data object format data can be used instead or connectedly. .. An example of such a data structure is the NIB data structure. Some examples of using NIB data structures are described in U.S. Patent Application Nos. 13 / 177,529 and U.S. Patent Application Nos. 13 / 177,533, both filed July 6, 2011. U.S. Patent Application No. 13 / 177,529 and U.S. Patent Application No. 13 / 177,533 are incorporated herein by reference.

Figure 1 conceptually illustrates the use of the switch access API by drawing an enclosure 135 around the control switching logic 125. Through these APIs, virtualized applications can read and write entries in the control plane flow table. The connectivity of virtualized applications to the control plane resources of the switching element (eg, the control plane table) is achieved in-band (ie, with network traffic controlled by the operating system) in some embodiments. However, in other embodiments it is implemented out of band (ie, via an independent physical network). Standard IGP protocols such as IS-IS or OSPF are sufficient when using independent networks, requiring minimal requirements for choices that avoid concentration of failures and basic connectivity to the operating system. is there.

In order to specify a control switching logic 125 for a switching element when the switching element is a physical switching element (as opposed to a software switching element), some virtualization applications have a control of the switching element. Use the open virtual switch protocol to create one or more control tables in the plane. Generally, the control plane is created and executed by a general-purpose CPU of a switching element. When the system creates the control table, the virtualization application uses the OpenFlow protocol to write flow entries to the control table. A general-purpose CPU for a physical switching element uses internal logic to convert an entry written to populate one or more forwarding tables in the switching element's forwarding plane into a control table. The forwarding table is generally created and executed by a dedicated switching chip of the switching element. By performing a flow entry in the forwarding table, the switching chip of the switching element can process and route packets of data it receives.

In some embodiments, the virtualized network system 100 is a chassis controller (chassis) in addition to a logical controller and a physical controller. It has a controller). In these embodiments, the chassis controller implements a switch access API to manage specific switching elements. That is, it is the chassis controller that pushes the control logic to a particular switching element. The physical controller in these embodiments functions as an aggregation point for relaying from the logical controller to the chassis controller interface-connected to the set of switching elements in which the physical controller is involved. The physical controller distributes control logic to the chassis controller that manages the set of switching elements. In these embodiments, the network controller operating system has a communication channel (eg, for example) between the physical controller and the chassis controller so that the physical controller can send control logic stored in the relational database data structure as data records to the chassis controller. A managed switching element communication interface that establishes a remote procedure call (RPC) channel. As a result, the chassis controller pushes control logic to the switching element using the switch access API or other protocol.

Communication structures provided by some embodiments of the operating system transfer data records to another network controller (eg, from a logical controller to another logical controller, from a physical controller to another physical controller, from a logical controller to a physical controller. In addition, it has an exporter (not shown) that can be used by the network controller to send data from the physical controller to the logical controller, etc.). Specifically, network controller control and virtualization applications can use exporters to export data records stored in relational database data structures to one or more other network controllers. In some embodiments, the exporter establishes a communication channel (eg, an RPC channel) between the two network controllers so that one network controller can send data records over the channel to another.

The operating system of some embodiments further comprises an importer that can be used by the network controller to receive data records from the network controller. The importer of some embodiments acts as the equivalent of an exporter of another network controller. That is, the importer is on the receiving side of the communication channel established between the two network controllers. In some embodiments, the network controller follows a publish-subscribe model in which the receiving controller subscribes to the channel to receive data only from the network controller that supplies the data of interest.

B. Pushing the flow to the edge switching element As mentioned above, relational database data structures store data for each switching element in the system's network infrastructure in some embodiments, but in other embodiments for switching elements at the edge of the network infrastructure. Stores only status information. 2 and 3 show an example of distinguishing between the two different methods. Specifically, Figure 2 shows the switch infrastructure of a multi-user server hosting system. In this system, six switching elements are employed to interconnect the six machines of two users A and B. Four of these switching elements 205-220 are edge switching elements that directly connect users A and B's machines 235-260, and two of the switching elements 225 and 230 connect the edge switching elements to each other. Internal switching elements that connect and connect to each other (ie, non-edge switching elements). All of the illustrated switching elements described above and below may be software switching elements in some embodiments, but in other embodiments a mixture of software switching elements and physical switching elements. For example, edge switching elements 205-220, and non-edge switching elements 225 and 230 are software switching elements in some embodiments. Further, the "machine" described in this application includes a virtual machine such as an arithmetic unit and a physical machine.

FIG. 3 shows a network controller 300 that manages edge switching elements 205 to 220. The network controller 300 is similar to the network controller 110 described above with reference to FIG. As shown in FIG. 3, controller 300 includes control application 305 and virtualization application 310. The operating system for controller instance 300 maintains a relational database data structure (not shown) that contains data records for edge switching elements 205-220 only. Applications 305 and 310 running on the operating system also allow users A and B to change the switching element configuration for the edge switching element they use. The network controller 300 then propagates these changes to the edge switching element as needed. Specifically, in this example, the two edge switching elements 205 and 220 are used by both user A and B machines, while the edge switching element 210 is used only by user A's machine 245, edge switching. Element 215 is used only by User B's Machine 250. Thus, FIG. 3 shows a network controller 300 that modifies the records for users A and B at switching elements 205 and 220, but updates only the records for user A at switching element 210 and the records for user B at switching element 215.

The controller 300 in some embodiments controls only the edge switching elements (ie, maintains only the data in the relational database data structure for the edge switching elements) for several reasons. Sufficient to maintain the required separation between machines (eg, arithmetic units), as opposed to maintaining the separation between all unnecessary switching elements by controlling the edge switching elements. Provide means to the controller. The internal switching element transfers data packets between the switching elements. Edge switching elements forward data packets between the machine and other network elements (eg, other switching elements). Thus, since the edge switching element is the last switching element in a line to forward the packet to the machine, the controller can maintain user isolation simply by controlling the edge switching element.

In addition to controlling edge switching elements, network controllers in some embodiments further include non-edge switching elements that are inserted into the switch network hierarchy to simplify and / or facilitate the operation of controlled edge switching elements. Use and control. For example, in some embodiments, the controllers interact with each other in a hierarchical switching architecture in which the switching elements they control have several edge switching elements that are leaf nodes and one or more non-edge switching elements that are non-leaf nodes. Request to be connected. In some such embodiments, each of the edge switching elements is connected to one or more non-leaf switching elements and such non-leaf switching elements to facilitate communication with other edge switching elements. To use.

The above description relates to the control of edge switching elements and non-edge switching elements by network controllers of some embodiments. In some embodiments, edge switching elements and non-edge switching elements (leaf nodes and non-leaf nodes) may be referred to as managed switching elements. This is because these switching elements are managed by the network controller (as opposed to unmanaged switching elements in a network that is not managed by the network controller) in order to achieve an LDP set with managed switching elements. ..

The network controller of some embodiments implements a logical switching element across managed switching elements based on the physical and logical data described above. A logical switching element (also referred to as a "logical forwarding element") may be specified to function in one or more different ways in which the switching element may function (eg, layer 2 switching, layer 3 routing, etc.). The network controller realizes the specified logical switching element by controlling the managed switching element. In some embodiments, the network controller implements a large number of logical switching elements across managed switching elements. This allows many different logical switching elements to be implemented across managed switching elements regardless of the network topology of the network.

The managed switching elements of some embodiments may be configured to route network data based on different routing criteria. In this way, the flow of network data through the switching elements in the network can be controlled to implement a large number of logical switching elements across the managed switching elements.

C. Logical switching element and physical switching element FIG. 4 shows an example of a large number of logical switching elements realized over a set of switching elements. In particular, FIG. 4 conceptually shows the logical switching elements 480 and 490 realized across the managed switching elements 410 to 430. As shown in FIG. 4, network 400 includes managed switching elements 410-430 and machines 440-465. As shown in FIG. 4, machines 440, 450 and 460 belong to user A and machines 445, 455 and 465 belong to user B.

The managed switching elements 410 to 430 of some embodiments route network data (eg, packets, frames, etc.) between network elements in the network connected to the managed switching elements 410 to 430. As shown, the managed switching element 410 routes network data between the machines 440 and 445 and the switching element 420. Similarly, network element 420 routes network data between machine 450 and managed switching elements 410 and 430, and switching element 430 routes network data between machines 455-465 and switching element 420. ..

Also, each of the managed switching elements 410 to 430 routes network data, which in some embodiments is in the form of a table, based on the transfer logic of the switch. In some embodiments, the forwarding table determines where network data should be routed (eg, a port on the switch) according to routing criteria. For example, the forwarding table of the Layer 2 switching element may determine where network data should be routed based on the MAC address (eg, source MAC address and / or destination MAC address). As another example, the forwarding table of the Layer 3 switching element may determine where network data should be routed based on the IP address (eg, source IP address and / or destination IP address). Many other types of routing criteria are possible.

As shown in FIG. 4, the forwarding table in each of the managed switching elements 410 to 430 contains several records. In some embodiments, each record specifies an action for routing network data based on routing criteria. Records may be referred to as flow entries in some embodiments because they control the "flow" of data through managed switching elements 410-430.

FIG. 4 further shows a conceptual representation of each user's logical network. As shown, user A's logical network 480 includes a logical switching element 485 to which user A's machines 440, 450 and 460 are attached. User B's logical network 490 includes a logical switching element 495 to which User B's machines 445, 455 and 465 are connected. Therefore, from the point of view of user A, user A has a switching element to which only the machine of user A is connected, and from the point of view of user B, user B has a switching element to which only the machine of user B is connected. .. In other words, for each user, the user has his own network with only the user's machines.

The conceptual flow entries that realize the flow of network data to machine 450 originating from machine 440 and network data to machine 460 originating from machine 440 will be described below. The flow entries "A1 to A2" in the forwarding table of the managed switching element 410 instruct the managed switching element 410 to route the network data destined for the machine 450 originating from the machine 410 to the switching element 420. The flow entries "A1 to A2" in the forwarding table of the switching element 420 instruct the switching element 420 to route the network data destined for the machine 450 originating from the machine 410 to the machine 450. Thus, when machine 440 sends network data destined for machine 450, managed switching elements 410 and 420 route network data along the data path 470 based on the corresponding records in the switching element's forwarding table.

Further, the flow entries "A1 to A3" in the forwarding table of the managed switching element 410 instruct the managed switching element 410 to route the network data addressed to the machine 460 transmitted from the machine 440 to the switching element 420. The flow entries "A1 to A3" in the forwarding table of the switching element 420 instruct the switching element 420 to route the network data destined for the machine 460 originating from the machine 440 to the switching element 430. The flow entries "A1 to A3" in the forwarding table of the switching element 430 instruct the switching element 430 to route the network data destined for the machine 460 originating from the machine 440 to the machine 460. Therefore, when machine 440 sends network data destined for machine 460, managed switching elements 410 through 430 route network data along data paths 470 and 475 based on the corresponding records in the switching element's forwarding table. ..

A conceptual flow entry for routing network data originating from machine 440 to machine 450 and network data originating from machine 440 to machine 460 has been described above, but a similar flow entry is in user A's logical network 480. It will be included in the forwarding table of managed switching elements 410-430 that route network data between other machines. Further similar flow entries will be included in the forwarding table of managed switching elements 410-430 that route network data between machines in User B's logical network 490.

The conceptual flow entry shown in FIG. 4 contains both source and destination information about the managed switching element to estimate the next hop switching element to which the packet should be sent. However, the source information does not need to be in the flow entry because the managed switching element of some embodiments can infer the next hop switching element using only the destination information (eg, context identifier, destination address, etc.). ..

In some embodiments, tunneling protocols (eg, control and provisioning of radio access points (CAPWAP)) (eg, control and provisioning of) to facilitate the implementation of logical switching elements 485 and 495 across managed switching elements 410-430. wireless access point), GRE (generic routing encapsulation), GRE Internet Protocol Security (IPsec) Etc.) may be used. With tunneling, a packet is sent through the switch and router as the payload of another packet. That is, the tunnel packet needs to expose its address (eg, source MAC address and destination MAC address) because the packet is forwarded based on the address contained in the header of the other packet that encapsulates the tunnel packet. There is no. Therefore, tunnel packets can have meaningful addresses in the logical address space, but other packets are forwarded / routed based on the addresses in the physical address space, thus separating the logical address space from the physical address space by tunneling. it can. In this way, the tunnel may be viewed as a "logical wire" connecting the managed switching elements in the network to implement the logical switching elements 485 and 495.

By configuring the switching elements in the various ways described above to implement a large number of logical switching elements across a set of switching elements, a large number of users can enjoy an independent network and / or switching element from each user's point of view. In fact, some or all of the same set of switching elements and / or connections are shared between sets of switching elements (eg, tunnels, physical wires).

II. Universal forwarding state A. Controller instance tier FIG. 5 shows the propagation of instructions that control a managed switching element through various processing layers of a controller instance of some embodiments of the present invention. FIG. 5 shows a control data pipeline 500 that transforms and propagates control plane data to managed switching element 525 through four processing layers of the same controller instance or different controller instances. These four layers are an input conversion layer 505, a control layer 510, a virtualization layer 515 and a customization layer 520.

In some embodiments, these four layers are in the same controller instance. However, other configurations of these layers exist in other embodiments. For example, in other embodiments, only control layer 510 and virtualization layer 515 are in the same controller instance, but the functionality of propagating customized physical control plane (CPCP) data is not shown in another controller instance (eg, not shown). It is in the customization layer of the chassis controller). In these other embodiments, the universal physical control plane (UPCP) data is the relational database data structure of one controller instance (not shown) before another controller instance generates CPCP data and pushes it to a managed switching element. ) Transfers to the relational database data structures of other controller instances. The former controller instance may be a logical controller that generates UPCP data, and the latter controller instance may be a physical controller or chassis controller that customizes UPCP data into CPCP data.

As shown in FIG. 5, the input transform layer 505 in some embodiments has an LCP 530 that can be used to represent the output of this layer. In some embodiments, an application (eg, a web-based application (not shown) is provided to the user to provide input for specifying the LDP set. This application sends input in the form of an API call to input transformation layer 505. The input conversion layer 505 converts the API call into LCP data in a format that can be controlled by the control layer 510. For example, the input is transformed into a set of input events that can be fed to the control layer's nLog table mapping engine. The nLog table mapping engine and its operation will be further described below.

The control layer 510 in some embodiments has LCP530 and LFP535 that can be used to represent inputs and outputs to this layer. The LCP contains a collection of high-level structures that allow the control layer and its users to specify one or more LDPs in the LCP for one or more users. LFP535 represents a user's LDP set in a format that can be processed by virtualization layer 515. Thus, the two logical planes 530 and 535 are virtualized spatial analogs of the control plane 555 and the forwarding plane 560, which are generally found in the general managed switching element 525, as shown in FIG.

In some embodiments, the control layer 510 defines and exposes the LCP structure. The layer itself or the user of the layer uses an LCP structure to define different sets of LDPs within the LCP. For example, in some embodiments, the LCP data 530 includes logical ACL data and the like. Some of this data (eg, logical ACL data) can be specified by the user, while other such data (eg, logical L2 or L3 records) are generated by the control layer and specified by the user. It does not have to be. In some embodiments, control layer 510 responds to certain changes (indicating changes to managed switching elements and management data paths) to the relational database data structures detected by control layer 510, such data. And / or specify.

In some embodiments, the LCP data (ie, the LDP set data represented with respect to the control plane structure) takes into account the current operational data from the managed switching element, and this control plane data is the PCP data. Can be specified first without considering how it is converted to. For example, this control plane data may later be transformed into physical control data for three managed switching elements that achieve the desired switching between the five computers, but the LCP data connects the five computers. Control data for one logical switching element may be specified.

The control layer contains a set of modules (not shown) for converting any LDPS in the LCP to LDPS in the LFP535. In some embodiments, control layer 510 uses the nLog table mapping engine to perform this transformation. It will be further described below that the control layer uses the nLog table mapping engine to perform this transformation. The control layer further includes a set of modules (not shown) for pushing the LDP set from the LFP535 of the control layer 510 to the LFP540 of the virtualization layer 515.

The LFP540 contains one or more LDP sets for one or more users. The LFP540 in some embodiments comprises logical forwarding data for one or more LDP sets of one or more users. Some of this data is pushed to the LFP540 by the control layer, while other such data is in relational database data structures, as further described below for some embodiments. Pushed to the LFP by the virtualization layer that detects the event.

In addition to LFP540, virtualization layer 515 includes UPCP545. UPCP545 contains UPCP data for the LDP set. The virtualization layer contains a set of modules (not shown) for converting the LDP set in the LFP540 into UPCP data in the UPCP545. In some embodiments, the virtualization layer 515 uses the nLog table mapping engine to perform this transformation. The virtualization layer further includes a set of modules (not shown) for pushing UPCP data from the UPCP 545 of the virtualization layer 515 to the relational database data structure of the customization layer 520.

In some embodiments, the UPCP data sent to customization layer 515 allows the managed switching element 525 to process data packets according to the LDP set specified by control layer 510. However, in contrast to CPCP data, UPCP data does not represent location-specific information for managed switching elements and / or managed switching elements in some embodiments, and thus is logical data specified by the control layer. Is not completely realized.

The UPCP data must be converted to CPCP data for each of the managed switching elements in order to fully implement the LDP set in the managed switching elements. For example, when an LDP set specifies a tunnel that spans several managed switching elements, the UPCP data uses the specific network address (eg, IP address) of the managed switching element that represents one end of the tunnel. To express. However, each of the other managed switching elements across the tunnel uses a port number local to the managed switching element to refer to the terminal managed switching element with a particular network address. That is, a particular network address must be translated into a local port number for each of the managed switching elements in order to fully implement the LDP set that specifies the tunnel in the managed switching element.

Assuming that the customization layer 520 is operating in a different controller instance than the controller instance that generates the UPCP data, the UPCP data, which is the intermediate data that is converted to the CPCP data, will cause the control system of some embodiments to be You will be able to change the magnification. This is because the virtualization layer 515 does not need to convert the LFP data that specifies the LDP set into CPCP data for each of the managed switching elements that implement the LDP set. Instead, the virtualization layer 515 converts the LFP data to UPCP data only once for all managed switching elements that implement the LDP set. In this way, the virtualized application saves the computational resources that would be spent converting the LDP set to CPCP data as many times as the number of managed switching elements that implement the LDP set.

Customization layer 520 includes UPCP546 and CPCP550 that can be used to represent inputs and outputs to this layer. The customization layer contains a set of modules (not shown) for converting UPCP data in UPCP546 to CPCP data in CPCP550. In some embodiments, the customization layer 520 uses the nLog table mapping engine to perform this transformation. The customization layer further includes a set of modules (not shown) for pushing CPCP data from the CPCP 550 of the customization layer 520 to the managed switching element 525.

The CPCP data pushed to each of the managed switching elements is specific to the managed switching element. CPCP data, referred to as "physical" data, allows managed switching elements to perform physical switching operations in both the physical data processing area and the logical data processing area. In some embodiments, the customization layer 520 operates in a separate controller instance for each of the managed switching elements.

In some embodiments, customization layer 520 does not operate on the controller instance. The customization layer 515 in these embodiments is on the controlled switching element 525. Therefore, in these embodiments, the virtualization layer 515 sends UPCP data to the managed switching element. Each of the managed switching elements customizes the UPCP data to CPCP data specific to the managed switching element. In some of these embodiments, the controller daemon is running on each of the managed switching elements and transforms the universal data into customized data for the managed switching elements. The controller daemon will be further described below.

In some embodiments, the customized physical control plane data propagated to the managed switching element 525 allows this switching element to with respect to network data (eg, packets) based on the logical values specified in the logical domain. You will be able to perform physical forwarding operations. Specifically, in some embodiments, the customized physical control plane Ndeta specifies a flow entry containing the logical value. These logical values include a logical address, a logical port number, and the like used for transferring network data in the logical area. These flow entries bring the logical value to the specified physical value in the physical domain so that the managed switching element can perform the logical forwarding operation on the network data by performing the physical forwarding operation based on the logical value. Further mapping. In this way, physical control plane data facilitates the implementation of logical switching elements across managed switching elements. Some examples of using propagated physical control plane data to achieve logical data processing in controlled switching elements are further described in US Patent Application No. 13 / 177,535, filed July 6, 2011. Will be done. U.S. Patent Application No. 13 / 177,535 is incorporated herein by reference.

The control plane data processed by the layers of the control data pipeline 500 becomes more global as the layers get higher. That is, the logical control plane data in the control layer 510 spans the entire set of managed switching elements that realize the logical switching elements defined by the logical control plane data. In contrast, the customized physical control plane data in customization layer 520 is local and specific to each of the managed switching elements that implement the logical switching element.

B. Multi-controller instance FIG. 6 shows a multi-instance, distributed network control system 600 of some embodiments. This distributed system uses three controller instances 605, 610 and 615 to control a large number of switching elements 690. In some embodiments, the distributed system 600 allows different controller instances to control the operation of the same switching element or different switching elements. As shown in FIG. 6, each instance has an input module 620, a control module 625, a record 635, a secondary storage structure (eg PTD) 640, an inter-controller communication interface 645, and managed switching element communication. Includes interface 650.

The input module 620 of the controller instance is similar to the input transform layer 505 described above with reference to FIG. 5 in that it takes input from the user and transforms that input into LCP data that the control module 625 understands and processes. To do. As mentioned above, the input is in the form of an API call in some embodiments. The input module 620 sends LCP data to the control module 625.

The control module 625 of the controller instance is similar to the control layer 510 in that it converts the LCP data into LFP data and pushes the LFP data to the virtualization module 630. Further, the control module 625 determines whether the received LCP data belongs to the LDPS managed by the controller instance. If the controller instance is the master of LDPS for the LCP data (ie, the logical controller that manages the LDPS), the virtualization module of the controller instance further processes the data. If the controller instance is not the master of LDPS for LCP data, the control module 625 of some embodiments stores the LCP data in secondary storage 640.

The virtualization module 630 of the controller instance is similar to the virtualization layer 515 in that it transforms the LFP data into UPCP data. The virtualization module 630 of some embodiments sends UPCP data to another controller instance via the inter-controller communication interface 645 or to a managed switching element via the managed switching element communication interface 650.

If the other controller instance is a physical controller involved in managing at least one of the managed switching elements that implement LDPS, the virtualization module 630 sends UPCP data to another instance. This is when the controller instance in which the virtualization module 630 is generating UPCP data is just a logical controller involved in a particular LDPS, but not a physical controller or chassis controller involved in a managed switching element that implements LDPS. Is.

If the managed switching element is configured to transform the UPCP data into CPCP data specific to the managed switching element, the virtualization module 630 sends the UPCP data to the managed switching element. In this case, the controller instance does not have a customization layer or module that performs the conversion from UPCP data to CPCP data.

In some embodiments, record 635 is a collection of records stored in the relational database data structure of the controller instance. In some embodiments, some or all of the input modules, control modules and virtualization modules use, update and manage records stored in relational database data structures. That is, the inputs and / or outputs of these modules are stored in a relational database data structure.

System 600 maintains the same switching element data records in the relational database data structures of each instance in some embodiments, but in other embodiments it is based on the LDPS managed by each controller instance. Allows relational database data structures of different instances to store different sets of switching element data records.

The PTD 640 of some embodiments is a secondary storage structure for storing user-specified network configuration data (eg, LCP data converted from input in the form of API calls). In some embodiments, the PTD for each controller instance uses system 600 to store configuration data for all users. Since the controller instance that receives the user input propagates the configuration data to the PTDs of the other controller instances, all PTDs of all controller instances have all configuration data for all users in these embodiments. However, in other embodiments, the PTD of the controller instance stores only the configuration data for a particular LDPS managed by the controller instance.

By allowing different controller instances to store the same or overlapping configuration data and / or secondary storage structure records, the system can fail any network controller (or relational database data structure and / or secondary storage structure instance). Improve overall recovery by being wary of data loss due to failure). For example, replicating a PTD across a controller instance allows a failed controller instance to quickly reload its PTD from another instance.

Inter-controller communication interface 645 is used to establish a communication channel (eg, RPC channel) with another controller instance (eg, by an exporter not shown). As shown, the inter-controller communication interface facilitates the exchange of data between different controller instances 605-615.

The managed switching element communication interface 650 facilitates communication between the controller instance and the managed switching element, as described above. In some embodiments, a managed switching element communication interface is used to propagate the UPCP data generated by the virtualization module 630 to each of the managed switching elements that can convert universal data to customized data.

System 600 uses Coordination Manager (CM) 655 for some or all of the communication between the distributed controller instances. CM655 in each instance allows an instance to coordinate certain behaviors with other instances. Different embodiments use CMs to coordinate sets of different behaviors between instances. Examples of such operations include writing to relational database data structures, writing to PTDs, controlling switching elements, facilitating communication between controllers related to the fault tolerance of controller instances, and the like. Also, CM is used to find the master of LDPS and the master of managed switching elements.

As mentioned above, different controller instances in the system 600 can control the operation of the same switching element or the operation of different switching elements. By distributing control of these behaviors across several instances, the system can more easily scale up to handle additional switching elements. Specifically, the system can distribute the management of different switching elements to different controller instances in order to benefit from the efficiencies that can be achieved by using a large number of controller instances. In such a distributed system, the number of calculations that each controller must perform to generate and distribute flow entries across the switching elements can be reduced by reducing the number of switching elements that each controller instance has under its control. Decrease. In other embodiments, a large number of controller instances can be used to create a scale-out network management system. Calculating the best way to distribute network flow tables in large networks is a CPU-intensive task. By partitioning the process by controller instances, the system 600 can use a larger but less sophisticated set of computer systems to create a scale-out network management system that can handle large networks.

In order to distribute the workload and avoid competing with controller instances that behave differently, the system 600 of some embodiments LDPSs one controller instance (eg, 605) within the system 600 and / or any predetermined controller instance. It is called the master of the managed switching element (that is, the logical controller or the physical controller). In some embodiments, each master controller instance stores only the data associated with the managed switching element being processed by the master in a relational database data structure.

In some embodiments, as mentioned above, the CM facilitates communication between controllers related to the recovery of controller instances. For example, the CM realizes communication between controllers by the above-mentioned secondary storage device. A controller instance in a control system can fail for many reasons (eg, hardware failure, software failure, network failure, etc.). Different embodiments may use different techniques to determine if the controller instance has failed. In some embodiments, a consensus protocol is used to determine if a controller instance in the control system has failed. Some of these embodiments may use Apache Zookeeper to implement the consensus protocol, while others may implement the consensus protocol in other ways. ..

Some embodiments of CM655 may utilize a defined timeout to determine if the controller instance has failed. For example, if the CM of a controller instance does not respond to communication (eg, sent from another CM of another controller instance in the control system) within a certain length of time (ie, the specified timeout length), it will respond. A controller instance that does not is determined to have failed. Other techniques may be utilized to determine if the controller instance has failed in other embodiments.

If the master controller instance fails, a new master for the LDP set and switching elements needs to be determined. Some embodiments of the CM655 do so by performing a master election process that elects the master controller instance (eg, to partition the management of the LDP set and / or to partition the management of the switching element). Make a good judgment. The CM655 in some embodiments may perform a master election process to elect a new master controller instance for both the LDP set and the switching element for which the failing controller instance was the master. However, in other embodiments, the CM655 has (1) a master selection process for selecting a new master controller instance for the LDP set for which the failed controller instance was the master, and (2) a failed controller instance. You may perform another master election process to elect a new master controller instance for the switching element for which was the master. In these cases, the CM655 is a CM655 with two different controller instances, one for the LDP set where the failed controller instance was the master and another for the switching element where the failed controller instance was the master. May be determined as a new controller instance.

Alternatively or connectively, the controllers in some embodiments clusters execute a consensus algorithm to determine the leader controller as described above. The leader controller separates the tasks involving each controller instance in the cluster by assigning a master controller and possibly a hotstand-by controller to a particular work item to take over in the event of a master controller failure.

In some embodiments, the master election process is also to separate the management of LDP sets and / or the management of switching elements when controller instances are added to the control system. In particular, when the control system 600 detects a change in a member of a controller instance in the control system 600, some embodiments of the CM655 perform a master election process. For example, if the control system 600 detects that a new network controller has been added to the control system 600, the CM655 will transfer some of the LDP set management and / or switching element management from the existing controller instance to the new controller instance. A master selection process may be performed for redistributing. However, in other embodiments, if the control system 600 detects that a new network controller has been added to the control system 600, then part of the management of the LDP set and / or the management of the switching element is from the existing controller instance. Not redistributed to new network controllers. Instead, the control system 600 in these embodiments is an unassigned LDP set and / or switching element (eg, a new LDP set and / or switching element, or an LDP set and / or from a failed network controller. If a switching element) is detected, an unallocated LDP set and / or switching element is assigned to the new controller instance.

C. Separate management of LDP sets and managed switching elements FIG. 7 shows an example of designating a master controller instance for a switching element (ie, a physical controller) in a distributed system 700 similar to the system 600 of FIG. In this example, the two controllers 705 and 710 control three switching elements S1, S2 and S3 for two different users A and B. Through the two control applications 715 and 720, the two users have a large number of records similarly stored in the two relational database data structures 755 and 760 of the two controller instances 705 and 710 by the controller virtualization applications 745 and 750. Specify two different LDP sets 725 and 730 that are converted to.

In the example shown in FIG. 7, both control applications 715 and 720 of both controllers 705 and 710 can modify the record of switching element S2 for both users A and B, but only controller 705 is the master of this switching element. Is. This example shows two different scenarios. The first scenario involves controller 705 updating record S2b1 at switching element S2 for user B. The second scenario includes a controller 705 in which the control application 720 updates record S2a1 in switching element S2 after updating record S2a1 for switching element S2 and user A in relational database data structure 760. In the example shown in FIG. 7, this update is routed from the relational database data structure 760 of controller 710 to the relational database data structure 755 of controller 705 and then to the switching element S2.

Different embodiments use different techniques to propagate changes to the relational database data structure 760 of controller instance 710 to the relational database data structure 755 of controller instance 705. For example, to propagate this update, the virtualization application 750 of controller 710 in some embodiments sends a set of records directly to the relational database data structure 755 (using an inter-controller communication module or exporter / importer). By). In response, the virtualization application 745 sends changes to the relational database data structure 755 to the switching element S2.

System 700 in some embodiments records S2a1 in switching element S2 in response to a request from control application 720, rather than propagating changes in the relational database data structure to the relational database data structure of another controller instance. Use other techniques to change. For example, some embodiments of distributed control systems use a secondary storage structure (eg, PTD) as a communication channel between different controller instances. In some embodiments, the PTD is replicated across all instances and some or all of the relational database data structures are pushed from one controller instance to another via the PTD storage layer. Thus, in the example shown in FIG. 7, changes to the relational database data structure 760 are replicated to the PTD of controller 710, from which it can be replicated to the PTD of controller 705 or the relational database data structure 755.

In addition to calling the controller instance the master of the switching element, some embodiments refer to one controller instance as the master of the LDPS, so there may be other variations on the sequence of operations shown in FIG. In some embodiments, the different controller instances may be the switching element and the master of the corresponding record for that switching element in the relational database data structure, while in other embodiments the controller instance switches in the relational database data structure. Requires to be the master of all records for the element and its switching elements.

In an embodiment where the system 700 allows the nomination of a master for switching elements and relational database data structure records, the example shown in FIG. 7 shows that controller instance 710 is the master for relational database data structure records S2a1. Controller instance 705 is the master for switching element S2. If a controller instance other than controller instances 705 and 710 was the master of relational database data structure record S2a1, the request for modification of the relational database data structure record from control application 720 must be propagated to the other controller instances. Wouldn't have been. Other controller instances modify relational database data structure records. With this change, the relational database data structure 755 and switching element S2 update their records by one or more means of propagating the change to controller instance 705.

In other embodiments, the controller instance 705 may be the master of the relational database data structure record S2a1 or the master of all records of the switching element S2 and its relational database data structure. In these embodiments, the request for modification of the relational database data structure record from the control application 720 would have to be propagated to the controller instance 705 that modifies the record and switching element S2 in the relational database data structure 755.

As mentioned above, different embodiments employ different techniques to facilitate communication between different controller instances. Also, different embodiments implement controller instances in different ways. For example, in some embodiments, a stack of control applications (eg, 625 or 715 in FIGS. 6 and 7) and virtualization applications (eg, 630 or 745) are installed and run on a single computer. Also, in some embodiments, a large number of controller instances can be installed and run in parallel on a single computer. In some embodiments, the controller instance may also have a stack of components divided among several computers. For example, within an instance, the control application (eg, 625 or 715) may be on the first physical machine or virtual machine, and the virtualization application (eg, 630 or 745) may be on the second physical machine or virtual machine. You can.

Figure 8 shows an example of the behavior of several controller instances that act as controllers for distributing inputs, LDPS master controllers (also known as logical controllers), and managed switching element master controllers (also known as physical controllers). .. In some embodiments, not all controller instances include a full stack of different modules and interfaces as described above with reference to FIG. Alternatively, not all controller instances perform all full stack functions. For example, the controller instances 805, 810 and 815 shown in FIG. 8 do not have a full stack of modules and interfaces.

The controller instance 805 in this example is a controller instance for distributing the input. That is, the controller instance 805 of some embodiments gets input from the user in the form of an API call. By calling the API, the user can query for requests or information (eg, the user's logic) to configure a particular LDPS (eg, to configure a logical switching element or logical router implemented in a set of managed switching elements). You can specify a request for network traffic statistics for the logical port of the switch. Input module 820 of controller instance 805 receives calls to these APIs in some embodiments and puts them in a form that can be stored in PTD825 and sent to another controller instance (eg, a data tuple or record). Convert.

The controller instance 805 in this example then sends these records to another controller instance involved in managing the records for a particular LDPS. In this example, controller instance 810 is involved in the LDPS record. The controller instance 810 receives the record from the PTD 825 of the controller instance 805 and stores the record in the PTD 845 which is the secondary storage structure of the controller instance 810. In some embodiments, the PTDs of different controller instances can communicate directly with each other and therefore do not need to rely on the inter-controller communication interface.

The control application 810 then detects adding these records to the PTD and processes them to generate or modify other records in the relational database data structure 842. In particular, control applications generate LFP data. As a result, the virtualized application detects changes and / or additions to these records in the relational database data structure and modifies and / or generates other records in the relational database data structure. These other records represent UPCP data in this example. These records are then sent through the controller-to-controller communication interface 850 of controller instance 810 to another controller instance that manages at least one of the switching elements that implements a particular LDPS.

The controller instance 815 in this example is the controller instance that manages the switching element 855. The switching element implements at least a portion of a particular LDPS. Controller instance 815 receives a record representing UPCP data from controller instance 810 via the inter-controller communication interface 865. In some embodiments, the controller instance 815 has a control application and a virtualization application for converting UPCP data into CPCP data. However, in this example, controller instance 815 only identifies a set of managed switching elements to which UPCP data is sent. In this way, the controller instance 815 acts as an aggregation point for collecting data to be sent to the managed switching element that the controller is responsible for managing. In this example, the managed switching element 855 is one of the managed switching elements by the controller instance 815.

D. Input conversion layer Figure 9 conceptually shows the software architecture for the input conversion application 900. The input conversion application of some embodiments functions as the input conversion layer 505 described above with reference to FIG. In particular, input conversion applications receive input from user interface applications that allow users to enter input values. The input conversion application translates the input into a request and dispatches the request to one or more controller instances to process the request. The input conversion application operates in the same controller instance in which the control application operates in some embodiments, but operates as an independent controller instance in other embodiments. As shown in FIG. 9, the input conversion application includes an input parser 905, a filter 910, a request generator 915, a request repository 920, a dispatcher 925, a response manager 930, and an inter-controller communication interface 940. ..

In some embodiments, the input transformation application 900 supports a set of LDPs and a set of API calls to specify queries for information. In these embodiments, a user interface application that allows the user to enter an input value is implemented to send an input in the form of an API call to the input conversion application 900. Therefore, these API calls specify LDPS (eg, user-specified logical switching element configuration) and / or query for user information (eg, network traffic statistics for the logical port of the user's logical switching element). To do. Further, the input conversion application 900 may obtain input from another input conversion application of a logical controller, a physical controller and / or another controller instance in some embodiments.

The input parser 905 of some embodiments receives input in the form of API calls from the user interface application. In some embodiments, the input parser extracts the user input value from the API call and passes the input value to the filter 910. Filter 910 excludes input values that do not meet certain requirements. For example, filter 910 excludes input values that specify invalid network addresses for logical ports. In response to an API call containing a non-conforming input value, the response manager 930 sends a response to the user indicating that the input is non-conforming.

Request generator 915 generates a request that is sent to one or more controller instances that process the request that produces a response to the request. These requests may include LDPS data for receiving inquiries about the controller instance to be processed and / or information. For example, the request may ask for statistical information on the logical port of the logical switching element managed by the user. The response to this request will include the requested statistics prepared by the controller instance responsible for managing the LDPS associated with the logical switching element.

Request generators 915 in different embodiments generate requests according to different formats, depending on the implementation of the controller instance that receives and processes the request. For example, the request generated by the request generator 915 in some embodiments is in the form of a record (eg, a data tuple) suitable for storage in the relational database data structure of the controller instance that receives the request. In some of these embodiments, the receiving controller instance uses the nLog table mapping engine to process the records that represent the request. In other embodiments, the request is in the form of an object-oriented data object that can interact with the NIB data structure of the controller instance that receives the request. In these embodiments, the receiving controller instance processes the data object directly on the NIB data structure without going through the nLog table mapping engine.

The request generator 915 of some embodiments deposits the request in the request repository 920 so that the dispatcher 925 can send the generated request to the appropriate controller instance. Dispatcher 925 identifies the controller instance to which each request should be sent. In some cases, the dispatcher examines the LDPS associated with the request and identifies the controller instance that is the master of that LDPS. In some cases, the dispatcher is specific as a controller instance to send a request when the request is specifically related to the switching element (for example, when it comes to logical port statistics where the request maps to a port on the switching element). Identify the master of the switching element (ie, the physical controller). The dispatcher sends the request to the identified controller instance. The receiving controller instance returns a response if the request contains a query for information.

The inter-controller communication interface 940 is similar to the inter-controller communication interface 645 described above with reference to FIG. 6 in that it establishes a communication channel (eg, an RPC channel) with another controller instance from which a request can be sent. The communication channel is bidirectional in some embodiments, but unidirectional in other embodiments. When the channel is unidirectional, the inter-controller communication interface establishes a number of channels with another controller instance so that the input transform application can send requests and receive responses over different channels.

When the receiving controller instance receives a request that specifies a query for information, the controller instance processes the request and produces a response containing the queried information. The response manager 930 receives a response from the controller instance that processed the request over the channel established by the inter-controller communication interface 940. In some cases, two or more responses may be sent back in response to the request sent. For example, a request for statistics from all logical ports of a user-managed logical switching element would return a response from each controller. Responses from multiple physical controller instances to a number of different switching elements whose ports are mapped to logical ports may be sent back to the input transformation application 900 either directly or through the master of the LDPS associated with the logical switch. In such cases, the response manager 930 of some embodiments merges these responses and sends a single merged response to the user interface application.

As described above, the control application running on the controller instance converts the data record representing the LCP data into the data record representing the LFP data by performing the conversion operation. Specifically, in some embodiments, the control application populates an LDP set into an LDPS table (eg, a logical forwarding table) created by the virtualization application.

E.nLog engine The controller instance in some embodiments performs mapping operations by using an nLog table mapping engine that uses variations in datalog table mapping techniques. Data logs are used in the field of database management to map one set of tables to another. Data logs are not a good tool for performing table mapping operations in network control system virtualization applications, as their current implementations are often slow.

Therefore, some embodiments of the nLog engine are custom designed to operate quickly so that LDPS data tuples can be mapped to managed switching element data tuples in real time. This custom design is based on several custom design options. For example, some embodiments compile the nLog table mapping engine from a set of high-level declaration rules represented by the application developer (eg, by the developer of the control application). In some of these embodiments, one custom-designed option made to the nLog engine is to allow application developers to use only the AND operator to represent declarative rules. It is a thing. By preventing developers from using other operators (eg, OR, XOR, etc.), these embodiments relate to AND operations where the resulting nLog engine rules perform faster at runtime. Guarantee to be represented.

Another custom design option pertains to the join operations performed by the nLog engine. Join operations are a common database operation for creating associations between records in different tables. In some embodiments, performing an outer join operation (also called an outer join operation) is impractical for the real-time operation of the engine because it can be time consuming, so the nLog engine makes the join operation an inner join operation. Limited to (also called inner join operation).

Yet another custom design option is to implement the nLog engine as a distributed table mapping engine run by several different controller instances. Some embodiments implement a distributed nLog engine by partitioning the management of the LDP set. Separating the management of an LDP set involves designating only one controller instance for each of that particular LDPS as the instance responsible for designating the records associated with that particular LDPS. For example, if the control system uses three switching elements to specify five LDP sets for five different users with two different controller instances, one controller instance will be two of the LDP sets. It may be the master for one related record and the other controller instance may be the master for the records for the other three LDP sets.

Separating the management of the LDP set further allocates the table mapping behavior for each LDPS to the nLog engine of the controller instance involved in the LDPS in some embodiments. Distributing nLog table mapping operations across several nLog instances offloads the load on each nLog instance, resulting in an increase in the speed at which each nLog instance can complete the mapping operation. In addition, such distribution reduces the memory size requirement for each machine running the controller instance. Some embodiments segment nLog table mapping operations across different instances by instructing the first join operation performed by each nLog instance to be based on LDPS parameters. Such instructions ensure that if an instance initiates a set of LDPS-related join operations that are not managed by the nLog instance, the join operation for each nLog instance will immediately fail and end. Some examples of using the nLog engine are described in the previously incorporated US Patent Application No. 13 / 177,533.

F. Control layer FIG. 10 shows a control application 1000 of some embodiments of the present invention. This application 1000 is used in some embodiments as the control module 625 of FIG. This application 1000 uses the nLog table mapping engine to map an input table containing an input data tuple representing LCP data to a data tuple representing LFP data. This application is on top of a virtualization application 1005 that receives a data tuple specifying an LDP set from control application 1000. Virtualization application 1005 maps data tuples to UPCP data.

More specifically, the control application 1000 allows different users to specify different sets of LDPs that specify the desired configuration of the logical switching elements that they manage. The control application 1000 uses a mapping operation to convert the data for each LDPS of each user into a set of data tuples that specify the LFP data for the logical switching element associated with the LDPS. In some embodiments, the control application runs on the same host on which the virtualization application 1005 runs. The control application and the virtualization application need not run on the same machine in other embodiments.

As shown in FIG. 10, the control application 1000 includes a set of rule engine input tables 1010, a set of function tables and constant tables 1015, an importer 1020, a rule engine 1025, and a set of rule engine output tables 1045. Includes translator 1050, exporter 1055, PTD 1060, and compiler 1035. Compiler 1035 is a component of an application that runs in a different time instance than the other components of the application. The compiler works when the developer needs to specify a rules engine for a particular control application and / or virtualized environment, while the modules of the remaining applications are LDPs specified by one or more users. Runs at runtime when the application interfaces with the virtualized application to deploy the set.

In some embodiments, the compiler 1035 gets a fairly small set (eg, hundreds of rows) of declarative instructions 1040 specified in the declarative language and uses them in the rule engine 1025 to perform table mapping for the application. Convert to a large set (eg, thousands of lines) of code that specifies the behavior (ie, object code). Therefore, the compiler greatly simplifies the process of control application developers who specify and update control applications. It uses a high-level programming language that allows developers to compactly define complex mapping behaviors for control applications through a compiler, followed by one or more changes (eg, logic supported by the control application). This is because this mapping behavior can be updated in response to changes in networking functionality, changes to the desired behavior of the control application, etc.). In addition, the compiler eliminates the need for developers to consider the order in which events arrive at the control application when defining mapping behavior.

In some embodiments, the rules engine (RE) input table 1010 is a logical data and / or switching configuration specified by the user and / or control application (eg, access control list configuration, dedicated virtual network configuration, port security configuration). Etc.) including tables with. The input table 1010 further includes a table containing physical data from the switching element managed by the network control system in some embodiments. In some embodiments, such physical data includes data about managed switching elements and other data about network configurations adopted by network control systems to deploy different sets of LDPs for different users.

The RE input table 1010 is partially populated with user-provided LCP data. The RE input table 1010 also includes LFP and UPCP data. In addition to the RE input table 1010, the control application 1000 includes other miscellaneous tables 1015 used by the rules engine 1025 to collect inputs for table mapping operations. These tables 1015 contain constant tables that store constant values for the constants for which the rules engine 1025 must perform table mapping operations. For example, in the constant table 1015, the constant "zero" specified as the value 0, the constant "dispatch_port_no" specified as the value 4,000, and the constant "broadcast_MAC_addr" specified as the value 0xFF: FF: FF: FF: FF: FF. And may be included.

When the rule engine 1025 references a constant, the corresponding value specified for the constant is actually searched and used. Also, the values specified for the constants in the constant table 1015 may be changed and / or updated. Thus, the constant table 1015 provides the ability for the rule engine 1025 to change the specified values for the referenced constants without having to rewrite or recompile the code that specifies the behavior of the rule engine 1025. Table 1015 further contains a function table that contains the functions that the rules engine 1025 needs to use to calculate the values needed to populate the output table 1045.

The rules engine 1025 performs a table mapping operation that specifies a method for converting LCP data to LFP data. Rule Engine (RE) Whenever one of the input tables changes, the rules engine performs a set of table mapping actions that may result in changing one or more data tuples in one or more RE output tables. To do.

As shown in FIG. 10, the rules engine 1025 includes an event processor 1022, some query plans 1027, and a table processor 1030. Each query plan is a set of rules that specifies a set of join operations to be performed when one of the RE input tables changes. In the following, such changes will be referred to as input table events. In this example, each query plan is generated by compiler 1035 from one declarative rule in the set of declarations 1040. In some embodiments, two or more query plans are generated from one declarative rule. For example, a query plan is created for each table joined by one declarative rule. That is, if you specify that the declarative rule joins four tables, then four different query plans are created from that one declaration. In some embodiments, the query plan is specified by using the nLog declarative language.

The event processor 1022 of rule engine 1025 detects the occurrence of each input table event. Event processing programs of different embodiments detect the occurrence of input table events in different ways. In some embodiments, the event processor registers a callback in the RE input table to notify changes to the records in the RE input table. In such an embodiment, event processing program 1022 detects an input table event when one of the records receives a notification from the modified RE input table.

In response to the detected input table event, the event processor 1022 tells the table processor 1303 to (1) select the appropriate query plan for the detected table event and (2) execute the query plan. Instruct. To execute a query plan, the table processor 1030, in some embodiments, records one or more records representing one or more sets of data values from one or more input tables 1010 and miscellaneous tables 1015. Performs the join operation specified by the query plan to generate. The table processor 1030 of some embodiments then performs a selection operation to (1) select a subset of data values from the records generated by the join operation, and (2) of the selected data values. Write a subset to one or more RE output tables 1045.

In some embodiments, the RE output table 1045 stores both logical network element data attributes and physical network element data attributes. Table 1045 is called the RE output table to store the output of the rule engine 1025 table mapping operation. In some embodiments, the RE output table can be grouped into several different categories. For example, in some embodiments, these tables may be RE input tables and / or control application (CA) output tables. If the rules engine detects an input event that requires the execution of a query plan by modifying the table, then the table is an RE input table. The RE output table 1045 may be the RE input table 1010 that generates an event that causes the rules engine to execute another query plan. Such an event is called an internal input event and is contrasted with an external input event, which is an event that occurs when the control application 1000 or importer 1020 modifies the RE input table.

As described further below, if the table changes cause the exporter 1055 to export the changes to the virtualization application 1005, the table is a CA output table. The table of the RE output table 1045 may be both the RE input table, the CA output table or the RE input table and the CA output table in some embodiments.

Exporter 1055 detects changes to the CA output table in the RE output table 1045. Exporters of different embodiments detect the occurrence of CA output table events in different ways. In some embodiments, the exporter registers a callback in the CA output table to notify changes to the records in the CA output table. In such an embodiment, exporter 1055 detects an output table event when one of the records receives a notification from the modified CA output table.

In response to the detected output table event, exporter 1055 retrieves some or all of the modified data tuples in the modified CA output table and uses this modified data tuple as the input table for virtualized application 1005. Propagate to (not shown). In some embodiments, instead of the exporter 1055 pushing the data tuple to the virtualization application, the virtualization application 1005 pulls the data tuple from the CA output table 1045 to the input table of the virtualization application. In some embodiments, the CA output table 1045 of control application 1000 and the input table of virtualization 1005 may be identical. In yet another embodiment, the CA output table is essentially a virtualization application (VA) input table because the control application and the virtualization application use one set of tables.

In some embodiments, the control application does not hold data for the LDP set that the control application is not responsible for managing in output table 1045. However, such data can be stored in the PTD and is converted by the translator 1050 into a format stored in the PTD. The PTD in Control Application 1000 controls this data from one or more other controls on the other controller instance so that some of the other controller instances responsible for managing the LDP set associated with the data can process the data. Propagate to the application instance.

In some embodiments, the control application further brings the data stored in the output table 1045 (ie, the data that the control application holds in the output table) to the PTD for data recovery. In addition, such data is transformed by translator 1050, stored in the PTD, and propagated to other control application instances of other controller instances. Therefore, in these embodiments, the PTD of the controller instance has all the configuration data for all the LDP sets managed by the network control system. That is, each PTD includes a global view of the logical network configuration in some embodiments.

The importer 1020 interfaces with many different sources of input data and uses the input data to modify or create the input table 1010. The importer 1020 of some embodiments receives input data from the input conversion application 1070 via an inter-controller communication interface (not shown). The importer 1020 also interfaces with the PTD 1060 so that the data received via the PTD from other controller instances can be used as input data to modify or create the input table 1010. In addition, the importer 1020 further detects changes by using the RE input table &amp; CA output table of the RE input table and the RE output table 1045.

G. Virtualization layer As mentioned above, the virtualization applications of some embodiments specify how different sets of LDPs of different users of the network control system can be realized by the managed switching element by the network control system. In some embodiments, the virtualization application specifies an implementation of an LDP set within the infrastructure of a managed switching element by performing a transformation operation. These conversion operations are switched to first store the LDP set data record in a managed switching element and then generate forwarding plane data (eg, flow entry) to define the forwarding behavior of the switching element. Convert to the control data record used by the element (eg UPCP data). The conversion operation further generates other data (eg, in a table) that specifies the network structure (eg, tunnels, queues, queue collections, etc.) to be defined within and between managed switching elements. The network structure further includes management software switching elements that are dynamically deployed or preset management software switching elements that are dynamically added to the set of managed switching elements.

FIG. 11 shows a virtualization application 1100 of some embodiments of the present invention. This application 1100 is used in some embodiments as the virtualization module 630 of FIG. The virtualization application 1100 uses the nLog table mapping engine to map an input table that contains LDPS data tuples that represent UPCP data. This application is under control application 1105, which generates LDPS data tuples. The control application 1105 is similar to the control application 1000 described above with reference to FIG. Virtualization application 1100 is similar to virtualization application 1005.

As shown in FIG. 11, the virtualization application 1100 includes a set of rule engine input tables 1110, a set of function and constant tables 1115, an importer 1120, a rule engine 1125, and a set of rule engine output tables 1145. Includes translator 1150, exporter 1155, PTD 1160, and compiler 1135. Compiler 1135 is similar to compiler 1035 described above with reference to FIG.

In order for the virtualization application 1100 to map LDPS data tuples to UPCP data tuples, developers in some embodiments declare that they include instructions to map LDPS data tuples to UPCP data tuples for some managed switching elements. Specifies type instruction 1140 in a declarative language. In some such embodiments, these switching elements include UPCP for converting UPCP data to CPCP data.

For other managed switching elements, the virtualization application 1100 maps LDPS data tuples to CPCP data tuples that are specific to each managed switching element that does not have an UPCP. In some embodiments, when the virtualization application 1100 receives UPCP data from a virtualized application on another controller instance, it converts the universal physical control plane data tuple to a physical data pathset data tuple. Further map the CPCP data tuples in output table 1140 to the CPCP data tuples for some managed switching elements that they do not have.

In some embodiments, if there is a chassis controller for converting UPCP tuples to CPCP data specific to a particular managed switching element, the virtualization application 1100 will convert the input UPCP data to the particular managed switching element. Do not convert to CPCP data. In these embodiments, the controller instance with the virtualization application 1100 identifies a set of managed switching elements whose master is the controller instance and delivers UPCP data to the set of managed switching elements.

The RE input table 1110 is similar to the RE input table 1010. In addition to the RE input table 1110, the virtualization application 1100 includes other miscellaneous tables 1115 used by the rules engine 1125 to collect inputs for table mapping operations. These tables 1115 are similar to table 1015. As shown in Figure 11, the rules engine 1125 is a table processor that functions in the same way that event processor 1122, some query plans 1127, and event processor 1022, query plan 1027, and table processing 1030 run. Including 1130.

In some embodiments, the RE output table 1145 stores both logical network element data attributes and physical network element data attributes. Table 1145 is called the RE output table to store the output of the rule engine 1125 table mapping operation. In some embodiments, the RE output table can be grouped into several different categories. For example, in some embodiments, these tables may be RE input tables and / or virtualized application (VA) output tables. If the rules engine detects an input event that requires the execution of a query plan by modifying the table, then the table is an RE input table. The RE output table 1145 may be the RE input table 1110 that generates an event that causes the rules engine to execute another query plan after it has been modified by the rules engine. Such an event is called an internal input event and is contrasted with an external input event, which is an event caused by a change in the RE input table by the control application 1105 via the importer 1020.

When a table change causes exporter 1155 to export the change to a managed switching element or other controller instance, the table is a VA control table. As shown in FIG. 12, the table of the RE output table 1145 may be both the RE input table 1110, the VA output table 1205 or the RE input table 1110 and the VA output table 1205 in some embodiments.

Exporter 1155 detects changes in RE output table 1145 to VA output table 1205. Exporters of different embodiments detect the occurrence of VA output table events in different ways. In some embodiments, the exporter registers a callback in the VA output table to notify changes to the records in the VA output table. In such an embodiment, exporter 1155 detects an output table event when one of the records receives a notification from the modified VA output table.

In response to the detected output table event, exporter 1155 gets each of the modified data tuples in the modified VA output table and uses this modified data tuple on another controller instance (eg, chassis controller). Propagate to one or more of or one or more of the managed switching elements. In doing so, the export completes deploying the LDPS (eg, one or more logical switching configurations) to one or more managed switching elements as specified by the record.

When the VA output table stores both logical network element data attributes and physical network element data attributes in some embodiments, the PTD1160 in some embodiments is a logical network element data attribute and physical network element in output table 1145. Stores both logical network element attributes and physical network element attributes that are the same as or derived from the data attributes. However, in other embodiments, the PTD 1160 stores only physical network element attributes that are the same as or derived from the physical network element data attributes in output table 1145.

In some embodiments, the virtualized application does not hold data for the LDP set that the virtualized application is not responsible for managing in output table 1145. However, such data can be stored in the PTD and is converted by the translator 1150 into a format stored in the PTD. The PTD of Virtualization Application 1100 uses this data on one or more of the other Controller Instances so that some of the other Virtualization Application Instances responsible for managing the LDP set associated with the data can process the data. Propagate to other virtualization application instances.

In some embodiments, the virtualization application further brings the data stored in the output table 1145 (ie, the data that the virtualization application holds in the output table) to the PTD for data recovery. In addition, such data is transformed by translator 1150, stored in the PTD, and propagated to other virtualization application instances of other controller instances. Therefore, in these embodiments, the PTD of the controller instance has all the configuration data for all the LDP sets managed by the network control system. That is, each PTD includes a global view of the logical network configuration in some embodiments.

Importer 1120 interfaces with many different sources of input data and uses the input data to modify or create input table 1110. The importer 1120 of some embodiments receives input data from the input conversion application 1170 via the inter-controller communication interface. The importer 1120 also interfaces with the PTD 1160 so that the data received via the PTD from other controller instances can be used as input data to modify or create the input table 1110. In addition, the importer 1120 further detects changes by using the RE input table &amp; VA output table of the RE input table and the RE output table 1145.

H. network controller FIG. 13 is a simplified diagram showing the table mapping operation of the control application and the virtualization application of some embodiments of the present invention. As shown in the upper half of FIG. 13, control application 1305 maps LCP data to LFP data that virtualization application 1310 of some embodiments then maps to UPCP data or CPCP data.

The lower half of FIG. 13 shows the table mapping operation of the control application and the virtualization application. As shown in the lower half, in some embodiments to generate LFP data from the input LCP data, the nLog engine 1320 of the control application of all these data along with the constant and function table (not shown) data. Because collections are used, the control application's input table 1315 stores LCP data, LFP (LFP) data, and UPCP data.

FIG. 13 shows that the importer 1350 receives LCP data from the user (eg, through an input conversion application) and updates the control application's input table 1315 with the LCP data. FIG. 13 shows that the importer 1350 detects or receives changes in the PTD 1340 (eg, changes in LCP data originating from other controller instances) in some embodiments and in response to such changes in the input table 1315. Further indicates that may be updated.

The lower half of FIG. 13 further shows the table mapping behavior of virtualization application 1310. As shown, because the nLog engine 1320 of the virtualization application in some embodiments uses LFP data along with data in constant and function tables (not shown) to generate UPCP and / or CPCP data. , The virtual application input table 1355 stores LFP data. In some embodiments, exporter 1370 sends the generated UPCP data to one or more other controller instances (eg, chassis controllers) to transform the UPCP data into CPCP data specific to the managed switching element. Generate CPCP data before pushing this data to a managed switching element or one or more managed switching elements. In other embodiments, exporter 1370 sends the generated CPCP data to one or more managed switching elements to specify the forwarding behavior of those managed switching elements.

In some embodiments, if there is a chassis controller for converting UPCP data to CPCP data specific to a particular managed switching element, virtualization application 1310 translates the input UPCP data for a particular managed switching element. Do not convert to CPCP data. In these embodiments, the controller instance with the virtualization application 1310 identifies a set of managed switching elements whose master is the controller instance and delivers UPCP data to the set of managed switching elements.

FIG. 13 shows that the importer 1375 receives the LFP data from the control application 1305 and updates the virtualized application's input table 1355 with the LFP data. FIG. 13 shows that the importer 1375 detects or receives changes in the PTD 1340 (eg, changes in LCP data originating from other controller instances) in some embodiments and in response to such changes in the input table 1355. Further indicates that may be updated. FIG. 13 further shows that importer 1375 may receive UPCP data from another controller instance.

As mentioned above, some of the logical or physical data that the importer pushes into the input table of the control or virtualization application is related to the data generated by the other controller instance and passed to the PTD. For example, in some embodiments, logical data about logical structures (eg, logical ports, logical queues, etc.) associated with a large number of LDP sets can change, and translators (eg, translator 1380 for controller instances) This change may be written to the input table. Another example of such logical data generated by another controller instance in a multi-controller instance environment occurs when the user provides LCP data for LDPS on a first controller instance that is not involved in LDPS. This change is added to the PTD of the first controller instance by the translator of the first controller instance. This change is then propagated across the PTDs of other controller instances by the replication process performed by the PTD. The importer of the second controller instance, which is the master of LDPS or the logical controller involved in LDPS, finally changes in this way and writes the change to the input table of the application (for example, the input table of the control application). Therefore, in some cases, the logical data that the importer writes to the input table may originate from the PTD of another controller instance.

As mentioned above, the control application 1305 and the virtualization application 1310 are two separate applications running on the same machine or different machines in some embodiments. However, other embodiments represent these two applications as two modules of one integrated application, the control application module 1305 that generates logical data in LFP and the virtualization application that generates physical data in UPCP or CPCP. Realize two applications.

Yet another embodiment integrates the control and virtualization behaviors of these two applications within one integrated application without separating them into two separate modules. FIG. 14 shows an example of such an integrated application 1400. This application 1400 uses the nLog table mapping engine 1410 to map data from the table input set 1415 to the table output set 1420, as in the embodiment described above. It may contain one or more tables in the input set of. The input set of tables in this integrated application may include LCP data that needs to be mapped to LFP data, or LFP data that needs to be mapped to CPCP or UPCP data. The input set of the table may further include UPCP data that needs to be mapped to CPCP data. UPCP data is delivered to a set of chassis controllers for a set of managed switching elements without being mapped to CPCP data. The mapping is that the controller instance running the integrated application 1400 is a logical controller or a physical controller, and the managed switching element of the physical controller has a chassis controller for mapping UPCP data to CPCP data for the managed switching element. It depends on whether you are a master.

In this integrated control / virtualization application 1400, the importer 1430 gets input data from the user or another controller instance. Importer 1430 further detects or receives changes in PTD1440 that are replicated relative to PTD. Exporter 1425 exports output table records to another controller instance (eg, chassis controller).

When sending an output table record to another controller instance, the exporter has an inter-controller communication interface (non-controller) so that the data contained in the record is sent to the other controller instance over a communication channel (eg, RPC channel). (Shown) is used. When sending an output table record to a managed switching element, the exporter provides a managed switching element communication interface (not shown) so that the data contained in the record is sent to the managed switching element over two channels. use. One channel is established using a switch control protocol (eg, OpenFlow) to control the forwarding plane of the managed switching element, and the other channel uses the configuration protocol to send configuration data. Established.

When sending an output table record to the chassis controller, the exporter 1425 in some embodiments uses a single channel of communication to send the data contained in the record. In these embodiments, the chassis controller accepts data through this single channel, but communicates with the managed switching element through the two channels. The chassis controller will be described in more detail below with reference to FIG.

Figure 15 shows another example of such an integrated application 1500. The integrated application 1500 uses the network information base data structure 1510 to store some of the input and output data for the nLog table mapping engine 1410. As mentioned above, NIB data structures store data in the form of object-oriented data objects. In the integrated application 1500, the output table 1420 is the primary storage structure. PTD1440 and NIB1510 are secondary storage structures.

The integrated application 1500 uses the nLog table mapping engine 1410 to map data from the table input set 1415 to the table output set 1420. In some embodiments, some of the data in the table output set 1420 is exported by the exporter 1425 to one or more other controller instances, or one or a managed switching element. Such exported data includes UPCP data or CPCP data that define the flow behavior of the managed switching element. These data may be backed up in the PTD by the translator 1435 in the PTD 1440 for data recovery.

Some of the data in the table output set 1420 is published by NIB publisher 1505 to NIB 1510. These data include configuration information for logical switching elements that users manage using the integrated application 1500. The data stored in the NIB 1510 is replicated by the Coordination Manager 1520 to other NIBs in other controller instances.

The NIB monitor 1515 receives change notifications from NIB1510 and pushes changes to input table 1415 via importer 1430 for some notifications (eg, notifications related to the LDP set for which the integrated application is the master). To do.

The query manager 1525 uses an inter-controller communication interface (not shown) to interface with an input conversion application (not shown) to receive queries on configuration data (eg, queries for information). As shown in FIG. 15, the manager 1525 of some embodiments queries the NIB to provide state information (eg, logical port statistics) about the logical network elements that the user manages. Also connect to the interface. However, in other embodiments, the query manager 1525 queries the output table 1420 to get state information.

In some embodiments, application 1500 uses a secondary storage structure (not shown) other than PTD and NIB. These structures include a persistent non-transactional database (PNTD) and hash table. In some embodiments, these two types of secondary storage structures provide different query interfaces for storing different types of data, storing data in different ways, and / or processing different types of queries. To do.

A PNTD is a persistent database stored on a disk or other non-volatile memory. Some embodiments use this database to store data about one or more switching element attributes or switching element operations (eg, statistics, calculations, etc.). For example, this database is used in some embodiments to store the number of packets routed through a particular port of a particular switching element. Other examples of data types stored in PNTD are error messages, log files, warning messages and billing data.

The PNTD in some embodiments has a database query manager (not shown) that can handle database queries, but this query manager cannot handle complex conditional transaction queries because it is not a transactional database. In some embodiments, access to the PNTD is faster than access to the PTD, but slower than access to the hash table.

Unlike PNTD, hash tables are not databases stored on disk or other non-volatile memory. Instead, the hash table is a storage structure stored in volatile system memory (eg, RAM). Hash tables use a hashing technique that uses a hash index to quickly identify the records stored in the table. This structure, combined with the placement of the hash table in system memory, allows access to this table very quickly. To facilitate this rapid access, a simplified query interface is used in some embodiments. For example, in some embodiments, the hash table has only two queries: a Put query to write a value to the table and a Get query to retrieve the value from the table. Some embodiments use hash tables to store rapidly changing data. Examples of such rapidly changing data are network entity status, statistics, status, uptime, link configuration and packet processing information. Also, in some embodiments, the integrated application uses a hash table as a cache to store repeatedly queried information such as flow entries written to a large number of nodes. Some embodiments employ a hash structure in the NIB for quick access to records in the NIB. Therefore, in some of these embodiments, the hash table is part of the NIB data structure.

PTD and PNTD improve controller recovery by storing network data on the hard disk. If the controller system fails, network configuration data is stored on disk in PTD and log file information is stored on disk in PNTD.

I. Network control system hierarchy FIG. 16 conceptually shows an example of the architecture of the network control system 1600. In particular, FIG. 16 shows that different elements of the network control system generate CPCP data from the inputs. As shown, the network control system 1600 of some embodiments includes an input conversion controller 1605, a logical controller 1610, physical controllers 1615 and 1620, and three managed switching elements 1625-1635. FIG. 16 further shows five machines 1640 to 1660 connected to managed switching elements (written as "MSE" in FIG. 16) and exchanging data between them. The architectural details such as the number of controllers, the number of managed switching elements and machines in each layer in the hierarchy shown in FIG. 16, and the relationship between the controllers, the managed switching elements and the machines are for illustration purposes only. Absent. It will be appreciated by those skilled in the art that many other different combinations of controllers, switching elements and machines are possible for the network control system 1600.

In some embodiments, each controller in a network control system has a full stack of different modules and interfaces described above with reference to FIG. However, each controller does not have to use all the modules and interfaces to perform the functionality given to the controller. Alternatively, in some embodiments, the controller in the system has only the modules and interfaces necessary to perform the functionality given to the controller. For example, the logical controller 1610, which is the master of the LDPS, does not have an input module (eg, an input conversion application) to generate UPCP data from the input LCP data, but a control module and a virtualization module (eg, a control application or It has a virtualized application or an integrated application).

Also, different combinations of different controllers may be operating on the same machine. For example, the input conversion controller 1605 and the logical controller 1610 may operate in the same arithmetic unit. Also, one controller may work differently for different LDP sets. For example, a single controller may be the master of the first LDPS and the master of the managed switching elements that implement the second LDPS.

Input conversion controller 1605 includes an input conversion application (not shown) that generates LCP data from input received from a user who specifies a particular LDPS. The input conversion controller 1605 identifies the LDPS master from the configuration data for system 1605. In this example, the master of LDPS is the logical controller 1610. In some embodiments, the two or more controllers may be masters of the same LDPS. Also, one logical controller may be the master of two or more LDP sets.

Logical controller 1610 is involved in a particular LDPS. Therefore, the logical controller 1610 generates UPCP data from the LCP data received from the input conversion controller. Specifically, the control module (not shown) of the logical controller 1610 generates LFP data from the received LCP data, and the virtualization module (not shown) of the logical controller 1610 generates UPCP data from the LFP data.

The logical controller 1610 identifies the physical controller that is the master of the managed switching elements that implement LDPS. In this example, the logical controller 1610 identifies the physical controllers 1615 and 1620 because the managed switching elements 1625 to 1635 are configured to implement LDPS in this example. The logical controller 1610 sends the generated UPCP data to the physical controllers 1615 and 1620.

Each of the physical controllers 1615 and 1620 may be the master of one or more managed switching elements. In this example, the physical controller 1615 is the master of the two managed switching elements 1625 and 1630, and the physical controller 1620 is the master of the managed switching elements 1635. As a master of a set of managed switching elements, the physical controller of some embodiments will generate specific CPCP data for each of the managed switching elements from the received UPCP data. Therefore, in this example, the physical controller 1615 generates customized PCP data for each of the managed switching elements 1625 and 1630. Physical controller 1320 generates customized PCP data for the managed switching element 1635. The physical controller sends CPCP data to a managed switching element whose master is the controller. In some embodiments, many physical controllers may be masters of the same managed switching element.

In addition to sending CPCP data, the physical controller of some embodiments receives data from the managed switching element. For example, the physical controller receives the configuration information of the managed switching element (for example, the VIF identifier of the managed switching element). The physical controller maintains the configuration information and further sends the information to the logical controller so that the logical controller has the configuration information of the managed switching element that realizes the LDP set in which the logical controller is the master.

Each of the managed switching elements 1625 to 1635 generates physical forwarding plane data from the CPCP data received by the managed switching element. As mentioned above, the physical forwarding plane data defines the forwarding behavior of the managed switching element. In other words, the managed switching element uses CPCP data to populate the forwarding table. Managed switching elements 1625 to 1635 transfer data between machines 1640 to 1660 according to the forwarding table.

FIG. 17 conceptually shows an example of the architecture of the network control system 1700. As shown in FIG. 16, FIG. 17 shows that different elements of the network control system generate CPCP data from the inputs. In contrast to the network control system 1600 of FIG. 16, the network control system 1700 has chassis controllers 1725 to 1735. As shown, the network control system 1700 of some embodiments includes an input conversion controller 1705, a logical controller 1610, a physical controller 1715 and 1720, a chassis controller 1725 to 1735, and three managed switching elements 1740 to. Including 1750. FIG. 17 further shows five machines 1755 to 1775 connected to and exchanged data between managed switching elements 1740 to 1750. The architectural details such as the number of controllers, managed switching elements and machines in each layer in the hierarchy shown in FIG. 17, and the relationship between the controllers, the managed switching elements and the machines are for illustration purposes only. Absent. It will be appreciated by those skilled in the art that many other different combinations of controllers, switching elements and machines are possible for the network control system 1700.

The input conversion controller 1705 is similar to the input conversion controller 1605 in that it includes an input conversion application that generates LCP data from the input received from a user who specifies a particular LDPS. The input conversion controller 1705 identifies the master of LDPS from the configuration data for system 1705. In this example, the master of LDPS is the logical controller 1710.

The logical controller 1710 is similar to the logical controller 1610 in that it generates UPCP data from the LCP data received from the input conversion controller 1705. Logical controller 1710 identifies the physical controller that is the master of the managed switching elements that implement LDPS. In this example, the logical controllers 1710 identify the physical controllers 1715 and 1720 because the managed switching elements 1740 to 1750 are configured to implement LDPS in this example. The logical controller 1710 sends the generated UPCP data to the physical controllers 1715 and 1720.

Each of the physical controllers 1715 and 1720, such as the physical controllers 1615 and 1620, may be the master of one or more managed switching elements. In this example, the physical controller 1715 is the master of the two managed switching elements 1740 and 1745, and the physical controller 1730 is the master of the managed switching elements 1750. However, the physical controllers 1715 and 1720 do not generate CPCP data for managed switching elements 1740 to 1750. As the master of the set of managed switching elements, the physical controller sends UPCP data to the chassis controllers involved in each of the managed switching elements for which the physical controller is the master. That is, the physical controller of some embodiments identifies a chassis controller that interfaces to a managed switching element on which the physical controller is the master. In some embodiments, the physical controllers identify these chassis controllers by determining if they are subscribed to the channels of the physical controllers.

The chassis controller of some embodiments has a one-to-one relationship with the managed switching element. The chassis controller receives UPCP data from the physical controller that is the master of the managed switching element and generates CPCP data specific to the managed switching element. An example of the chassis controller architecture will be described in more detail below with reference to FIG. In some embodiments, the chassis controller operates on the same machine on which the managed switching element managed by the chassis controller operates, whereas in other embodiments, the chassis controller and managed switching element operate on different machines. To do. In this example, the chassis controller 1725 and the managed switching element 1740 operate in the same arithmetic unit.

Each of the managed switching elements 1740 to 1750, such as the managed switching elements 1625 to 1635, generates physical forwarding plane data from the CPCP data received by the managed switching elements. Managed switching elements 1740 to 1750 use CPCP data to populate their respective forwarding tables. Managed switching elements 1740 to 1750 transfer data between machines 1755 to 1775 according to the flow table.

As mentioned above, the managed switching element may implement two or more LDPSs in some cases. In such cases, the physical controller that is the master of such managed switching elements receives UPCP data for each LDP set. Therefore, the physical controller in the network control system 1700 may function as an aggregation point for relaying UPCP data for different LDP sets to the chassis controller for a specific managed switching element that realizes the LDP set.

Although the chassis controller shown in FIG. 17 is at a level above the managed switching element, the chassis controller is generally a chassis controller because some embodiments are in or adjacent to the managed switching element. It operates at the same level as the managed switching element operates.

In some embodiments, the network control system may have a hybrid of network control systems 1600 and 1700. That is, in this hybrid network control system, a part of the physical controller generates CPCP data for some of the managed switching elements, and a part of the physical controller generates CPCP data for some of the managed switching elements. Do not generate. For the latter managed switching element, the hybrid system has a chassis controller that produces CPCP data.

As mentioned above, the chassis controller of some embodiments is a controller for managing a single managed switching element. The chassis controller of some embodiments does not have a full stack of the different modules and interfaces described above with reference to FIG. One of the modules that a chassis controller has is a chassis control application that generates CPCP data from UPCP data that it receives from one or more physical controllers. FIG. 18 shows an example of the architecture for chassis control application 1800. This application 1800 uses the nLog table mapping engine to map an input table containing an input data tuple representing UPCP data to a data tuple representing LFP data. The application 1800 manages the managed switching element 1885 in this example by exchanging data with the managed switching element 1885. In some embodiments, application 1800 (ie, chassis controller) operates on the same machine on which the managed switching element 1885 is operating.

As shown in FIG. 18, the chassis control application 1800 includes a set of rule engine input tables 1810, a set of function and constant tables 1815, an importer 1820, a rule engine 1825, and a set of rule engine output tables 1845. , Exporter 1855, managed switching element communication interface 1865, and compiler 1835. FIG. 18 further shows the physical controller 1805 and the managed switching element 1885.

Compiler 1835 is similar to compiler 1035 in FIG. In some embodiments, the rules engine (RE) input table 1810 is the UPCP data and / or switching configuration (eg, access control list) sent by physical controller 1805, which is the master of managed switching element 1885, to chassis control application 1800. Includes tables with configurations, dedicated virtual network configurations, port security configurations, etc.). Input table 1810 further includes a table containing physical data from managed switching element 1885. In some embodiments, such physical data includes data about managed switching element 1885 (eg, CPCP data, physical forwarding data) and other data about the configuration of managed switching element 1885.

The RE input table 1810 is similar to the RE input table 1010. Input table 1810 is partially populated by UPCP data provided by physical controller 1805. Physical controller 1805 in some embodiments receives UPCP data from one or more logical controllers (not shown).

In addition to the input table 1810, chassis control application 1800 includes other miscellaneous tables 1815 used by the rules engine 1825 to collect inputs for table mapping operations. These tables 1815 are similar to table 1015. As shown in FIG. 18, the rules engine 1825 is a table processor that functions in the same way that event processor 1822, some query plans 1827, and event processor 1022, query plan 1027, and table processing 1030 run. Including 1830.

In some embodiments, the RE output table 1845 stores both logical network element data attributes and physical network element data attributes. Table 1845 is called the RE output table to store the output of the rule engine 1825 table mapping operation. In some embodiments, the RE output table can be grouped into several different categories. For example, in some embodiments, these tables may be RE input tables and / or chassis control application (CCA) output tables. If the rules engine detects an input event that requires the execution of a query plan by modifying the table, then the table is an RE input table. The RE output table 1845 may be the RE input table 1810 that generates an event that causes the rules engine to execute another query plan after it has been modified by the rules engine. Such an event is called an internal input event and is contrasted with an external input event, which is an event caused by a change in the RE input table by the control application 1805 via the importer 1820. When a table change causes exporter 1855 to export the change to a managed switching element or other controller instance, the table is a CA output table.

Exporter 1855 detects changes to the CCA output table in RE output table 1845. Exporters of different embodiments detect the occurrence of CCA output table events in different ways. In some embodiments, the exporter registers a callback in the CCA output table to notify changes to the records in the CCA output table. In such an embodiment, exporter 1855 detects an output table event when one of the records receives a notification from the modified CCA output table.

In response to the detected output table event, exporter 1855 gets each of the modified data tuples in the modified output table and uses this modified data tuple for another controller instance (eg, a physical controller). Propagate to one or more or managed switching elements 1885. Exporter 1855 uses an inter-controller communication interface (not shown) to send modified data tuples to other controller instances. The inter-controller communication interface establishes a communication channel (eg, RPC channel) with other controller instances.

Exporter 1855 in some embodiments uses managed switching element communication interface 1865 to send modified data tuples to managed switching element 1885. The managed switching element communication interface of some embodiments establishes two channels of communication. The managed switching element communication interface uses a switching control protocol to establish the first channel of the two channels. An example of a switching control protocol is the OpenFlow protocol. The OpenFlow protocol is, in some embodiments, a communication protocol for controlling the forwarding plane (eg, forwarding table) of a switching element. For example, the OpenFlow protocol has a command to add a flow entry to a managed switching element 1885, a command to remove a flow entry from a managed switching element 1885, and a command to change a flow entry in a managed switching element. And provide.

The managed switching element communication interface establishes a second channel of two channels using a configuration protocol to send out configuration information. In some embodiments, the configuration information includes information for configuring the managed switching element 1885, such as information for input ports, output ports, QoS configurations for ports, and the like.

The managed switching element communication interface 1865 receives updates on the managed switching element 1885 from the managed switching element 1885 via two channels. If there is a flow entry or configuration change in the managed switching element 1885 that is not initiated by the chassis control application 1800, the managed switching element 1885 in some embodiments sends an update to the chassis control application. Examples of such changes include failure of the machine connected to the port of managed switching element 1885, VM migration to managed switching element 1885, and so on. The managed switching element communication interface 1865 sends updates to the importer 1820 that modifies one or more input tables 1810. If there is output generated by the rules engine 1825 from these updates, exporter 1855 sends this output to physical controller 1805.

J. Flow entry generation Figure 19A-B shows an example of creating a tunnel between two managed switching elements based on UPCP data. Specifically, Figure 19A-B shows a series of operations performed by different components of the network management system 1900 to establish a tunnel between two managed switching elements 1925 and 1930 in four different stages. Shown in 1901-1904. Figures 19A-B further show the logical switching elements 1905 and VM1 and VM2. Each of the four stages 1901-1904 shows the network control system 1900 and the managed switching elements 1925 and 1930 at the bottom, and the VMs connected to the logical switching elements 1905 and 1905 at the top. VMs are shown both at the top and bottom of each stage.

As shown in the first stage 1901, the logical switching element 1905 transfers data between VM1 and VM2. Specifically, data reaches or originates from VM1 through logical port 1 of logical switching element 1905 and reaches or originates from VM2 through logical port 2 of logical switching element 1905. The logical switching element 1905 is implemented by the managed switching element 1925 in this example. That is, logical port 1 is mapped to port 3 of managed switching element 1925, and logical port 2 is mapped to port 4 of managed switching element 1925.

The network control system 1900 in this example has a controller cluster 1910 and two chassis controllers 1915 and 1920. The controller cluster 1910 includes an input conversion controller (not shown) that collectively generates UPCP data based on the inputs received by the controller cluster 1910, a logical controller (not shown), and a physical controller (not shown). The chassis controller receives the UPCP data and customizes the universal data to PCP data specific to the managed switching element managed by each chassis controller. The chassis controllers 1915 and 1920 are CPCPed so that managed switching elements 1925 and 1930 can generate the physical forwarding plane data used by the managed switching elements to transfer data between the managed switching elements 1925 and 1930. Pass the data to managed switching elements 1925 and 1930, respectively.

In the second stage 1902, the administrator of the network including the managed switching element 1930 creates VM3 on the host (not shown) on which the managed switching element 1930 operates. The administrator creates port 5 of the managed switching element 1930 and connects VM3 to the port. When port 3 is created, managed switching elements 1930 in some embodiments send information about the newly created port to controller cluster 1910. In some embodiments, the information may include a port number, a network address (eg, an IP address and a MAC address), a transmission zone to which the managed switching element belongs, a machine connected to the port, and the like. As described above, this configuration information passes through the chassis controller that manages the managed switching element, and then through the physical and logical controllers to the user who manages the logical switching element 1905. For this user, a new VM is available to be added to the user-managed logical switching element 1905.

At stage 1903, the user in this example decides to use VM3 and connect VM3 to the logical switching element 1905. As a result, logical port 6 of logical switching element 1905 is created. Therefore, data that reaches or originates from VM3 passes through logical port 6. In some embodiments, controller cluster 1910 implements a logical switching element to create a tunnel between each pair of managed switching elements having a pair of ports to which a pair of logical ports of the logical switching element are mapped. Instruct all managed switching elements to do. In this example, the tunnel is the data between logical port 1 and logical port 6 (ie, between VM1 and VM3) and between logical port 2 and logical port 6 (ie, between VM2 and VM3). Can be established between managed switching elements 1925 and 1930 to facilitate communication. That is, data exchanged between port 3 of managed switching element 1925 and port 5 of managed switching element 1930 and exchange between port 4 of managed switching element 1925 and port 5 of managed switching element 1930. The data to be generated can pass through a tunnel established between the managed switching elements 1925 and 1930.

Because logical port 1 and logical port 2 are mapped to two ports on the same managed switching element 1925, the tunnel between the two managed switching elements is between logical port 1 and logical port 2 (ie, that is). Not required to facilitate the exchange of data between VM1 and VM2).

The third stage, 1903, further demonstrates that the controller cluster 1910 sends UPCP data from the managed switching element 1925 to the managed switching element 1930 that specifies the instructions to create the tunnel. In this example, the UPCP data is sent to chassis controller 1915, which customizes the UPCP data to PCP data specific to the managed switching element 1925.

The fourth stage 1904 indicates that the chassis controller 1915 sends PCP data to the tunnel specifying instructions to create the tunnel and to forward the packet to the tunnel. Managed switching element 1925 creates a tunnel to managed switching element 1930 based on CPCP data. More specifically, the managed switching element 1925 creates port 7 and establishes a tunnel (eg, a GRE tunnel) to port 8 of the managed switching element 1930. A more detailed operation of creating a tunnel between two managed switching elements is described below.

FIG. 20 conceptually illustrates the process 2000 that some embodiments perform to generate CPCP data from UPCP data that specifies the creation and use of tunnels between two managed switching elements. In some embodiments, process 2000 is performed by a chassis controller that interfaces with the managed switching element or a physical controller that interfaces directly with the managed switching element.

Process 2000 starts by receiving UPCP data from the logical controller or the physical controller. In some embodiments, the UPCP data have different types. One type of UPCP data is a universal tunnel flow instruction that specifies the creation and use of tunnels in managed switching elements. In some embodiments, universal tunnel flow instructions include instructions for ports created on managed switching elements in the network. This port is the port of the managed switching element to which the user maps the logical port of the logical switching element. This port is also the destination port that the tunneled data needs to reach. Information about the port is that (1) the transmission zone to which the managed switching element with the port belongs and (2) the tunnel used to build the tunnel to the managed switching element with the destination port in some embodiments. The type of tunnel based on the protocol (eg GRE, CAPWAP, etc.) and (3) the network address (eg IP address) of the managed switching element that has the destination port (eg, acting as one end of the tunnel to establish) The IP address of the VIF to be used) is included.

Processing 2000 then determines if the received UPCP data is a universal tunnel flow instruction (in 2010). In some embodiments, the UPCP data specifies the type so that processing 2000 can determine the type of universal plane data received. If determined that the universal data processing 2000 received is not a universal tunnel flow instruction (in 2010), the process proceeds to 2015, and generates a CPCP data, the generated data processing managed switching the management 2000 manages Process UPCP data to send to the element. After that, the process 2000 ends.

If process 2000 determines that the received UPCP data is a universal tunnel flow instruction (in 2010), it proceeds to 2020 and parses the data to obtain information about the destination port. Process 2000 then determines if the managed switching element with the destination port is in the same transmission zone as the managed switching element with the source port (at 2025). The managed switching element having the source port is a managed switching element managed by the chassis controller or the physical controller that executes the process 2000. In some embodiments, transmission zones include a group of machines that can communicate with each other without the use of second level managed switching elements such as pool nodes.

In some embodiments, the logical controller determines if the managed switching element with the destination port is in the same transmission zone as the managed switching element with the source port. The logical controller takes this decision into account when preparing the universal tunnel flow instruction to send (via the physical controller) to the chassis controller performing process 2000. Specifically, the universal tunnel flow instruction contains different information to create different tunnels. After the description of FIG. 21, examples of these different tunnels will be described below. In these embodiments, process 2000 skips 2025 and proceeds to 2015.

Process 2000 proceeds to 2015 as described above if it determines (in 2025) that the managed switching element including the source port and the managed switching element including the destination port are not in the same transmission zone. If process 2000 determines (in 2025) that the managed switching element, including the source port, and the managed switching element, including the destination port, are in the same transmission zone, proceed to 2030 to customize and customize the universal tunnel flow instruction. The information is sent to a managed switching element that has a source port. Customization of the universal tunnel flow instruction is described in detail below. After that, the process 2000 ends.

In Figure 21, some embodiments generate customized tunnel flow instructions and manage switching the customized instructions so that the managed switching element creates a tunnel and sends data to the destination through the tunnel. The process to be executed to send to the element is conceptually shown. In some embodiments, processing 2100 is performed by a controller instance that interfaces with the managed switching element or a physical controller that interfaces directly with the managed switching element. In some embodiments, the process 2100 is managed by the controller executing the process 2100, receiving a universal tunnel flow instruction, parsing the port information about the destination port, and managing the managed switching element having the destination port. It starts when it is determined that it is in the same transmission zone as the target switching element.

Process 2100 begins by generating an instruction (at 2105) to create a tunnel port. In some embodiments, process 2100 generates an instruction to create a tunnel port at a managed switching element managed by the controller based on the port information. The instruction includes the type of tunnel to be established, the IP address of the NIC that is the destination end of the tunnel, and the like. Managed by the controller The tunnel port of the managed switching element becomes the other end of the tunnel.

Process 2100 then sends the generated instruction to create the tunnel port to the managed switching element managed by the controller (at 2110). As mentioned above, the chassis controller of some embodiments or the physical controller that directly interfaces with the managed switching element uses two channels to communicate with the managed switching element. One channel is a configuration channel that exchanges configuration information with the managed switching element, and the other channel is a switching element control channel for exchanging flow entry and event data with the managed switching element (for example, the OpenFlow protocol). Channels established using). In some embodiments, the process uses a configuration channel to send generated instructions that create a tunnel port to a managed switching element managed by the controller. Upon receiving the generated instruction, the managed switching element of some embodiments creates a tunnel port at the managed switching element using the tunnel protocol specified by the tunnel type, and the tunnel port and destination port. Establish a tunnel to and from the port of the managed switching element that has. Once the tunnel port and tunnel are created and established, the managed switching element of some embodiments returns the value of the tunnel identifier (eg, 4) to the controller instance.

The process 2100 of some embodiments then receives the value of the tunnel port identifier (eg, "tunnel_port = 4") via the configuration channel (at 2115). Processing 2100 then uses this received value to modify the flow entry contained in the universal tunnel flow instruction. When this flow entry is sent to the managed switching element, it causes the managed switching element to perform an operation. However, because it is universal data, this flow entry identifies the tunnel port by the universal identifier (eg tunnel_port) rather than the actual port number. For example, this flow entry in the received universal tunnel flow instruction is "If destination = destination machine's UUID, send to tunnel_port (If destination = destination machine's UUID, send to tunnel_port) ". Process 2100 creates a flow entry (at 2120) that contains the value of the tunnel port identifier. Specifically, process 2100 replaces the identifier for the tunnel port with the actual value of the identifier that identifies the created port. For example, a modified flow entry would look like "If destination = destination machine's UUID, send to 4".

Process 2100 then sends this flow entry to the managed switching element (at 2125). In some embodiments, the process sends this flow entry to a managed switching element via a switching element control channel (eg, an OpenFlow channel). The managed switching element uses this flow entry to update the flow entry table. After that, the managed switching element transfers the data through the tunnel by sending the data directed to the destination machine to the tunnel port. After that, the process ends.

Figures 22A-B conceptually show an example of the operation of chassis controller 2210 to convert universal tunnel flow instructions into customized instructions received and used by the managed switching element 2215 in seven different stages 2201-2207. Chassis controller 2210 is similar to chassis controller 1800 described above with reference to FIG. However, for ease of explanation, Figures 22A-B do not show all the components of chassis controller 2210.

As shown, the chassis controller 2210 includes an input table 2220, a rule engine 2225, and an output table 2230, which are similar to the input table 1820, the rule engine 1825, and the output table 1845. Chassis controller 2210 manages managed switching element 2215. Two channels 2235 and 2240 are established between the chassis controller and the managed switching element 2215 in some embodiments. Channel 2235 is for exchanging configuration data (eg, data on ports, current status of ports, creation of queues associated with managed switching elements, etc.). Channel 2240 is an OpenFlow channel (OpenFlow control channel) through which flow entries are exchanged in some embodiments.

Stage 1 2201 indicates that chassis controller 2210 has updated input table 2220 with universal tunnel flow instructions received from a physical controller (not shown). As shown, the universal tunnel flow instruction includes instruction 2245 to create a tunnel and flow entry 2250. As shown, instruction 2245 contains the type of tunnel created and the IP address of the managed switching element with the destination port. Flow entry 2250 specifies what to do with universal data that is not specific to managed switching element 2215. The rules engine performs table mapping operations on instruction 2245 and flow entry 2250.

The second stage 2202 shows the result of the table mapping operation performed by the rule engine 2225. Instruction 2260 is due to instruction 2245. Instructions 2245 and 2260 may be identical in some embodiments, but may not be identical in other embodiments. For example, the values in instructions 2245 and 2260 that represent the type of tunnel may be different. Instruction 2260 includes the IP address and the type of tunnel created, among other things that may be included in instruction 2260. Flow entry 2250 stays at input table 2220 because it did not trigger any table mapping operation.

The third stage 2203 indicates that instruction 2260 is being pushed to managed switching element 2215 via configuration channel 2235. The managed switching element 2215 creates a tunnel port and establishes a tunnel between the managed switching element 2215 and another managed switching element that has a destination port. In some embodiments, one end of the tunnel is the created tunnel port and the other end of the tunnel is the port associated with the destination IP address. The managed switching element 2215 of some embodiments uses the protocol specified by the tunnel type to establish the tunnel.

Stage 4 2204 indicates that the managed switching element 2215 created a tunnel port (port 1 in this example) and tunnel 2270. This stage further indicates that the managed switching element returns the actual value of the tunnel port identifier. The managed switching element 2215 sends this information through the OpenFlow channel 2240 in this example. The information enters the input table 2220 as input event data. The fifth stage 2205 indicates that the input table 2220 is updated with the information from the managed switching element 2215. This update triggers Rule Engine 2225 to perform a table mapping operation.

The sixth stage 2206 shows the result of the table mapping operation performed in the previous stage 2204. Output table 2230 has a flow entry 2275 that specifies what happens here with respect to information specific to managed switching element 2215. Specifically, flow entry 2275 specifies that the managed switching element 2215 should be sent to the packet through port 1 if the destination of the packet is the destination port. The seventh stage 2207 indicates that the flow entry 2275 is being pushed to the managed switching element 2215 that forwards the packet using the flow entry 2275.

The data exchanged between the instruction 2245 and the chassis controller 2210 and the managed switching element 2215 as shown in FIG. 22 is a conceptual representation of the universal tunnel flow instruction and the customized instruction, and is an actual expression and. It does not have to be in the form.

The example of FIG. 22 will also be described with respect to the operation of chassis controller 2210. This example is also applicable to some embodiments of the physical controller that transform the UPCP data into CPCP data for a managed switching element whose master is the physical controller.

19-22 show a tunnel between two management edge switching elements to facilitate the exchange of data between a pair of machines (eg, VMs) using the two logical ports of the logical switching element. Indicates the creation of. This tunnel covers one of the possible uses of the tunnel. In some embodiments of the invention, many other uses of the tunnel are possible in network control systems. Examples of tunnel uses are (1) a tunnel between a managed switching element and a pool node, and (2) one is an edge switching element and the other provides L3 gateway services (ie, the network layer (ie, network layer). A tunnel between two managed switching elements (managed switching elements) that are connected to the router to receive routing services in L3), and (3) a logical port and another logical port connected to the L2 gateway service. There is a tunnel between two managed switching elements.

Next, a series of events that create a tunnel will be described in each of the three examples. For a tunnel between a managed switching element and a pool node, the pool node is provided first, followed by the managed switching element. The VM is connected to the port of the managed switching element. This VM is the first VM to be connected to a managed switching element. This VN is then coupled to the logical port of the logical switching element by mapping the logical port to the port of the managed switching element. When the logical port is mapped to the port of the managed switching element, the logical controller sends a universal tunnel flow instruction to (or to the physical controller) the chassis controller that interfaces with the managed switching element (eg, via the physical controller). hand).

The chassis controller then commands the managed switching element to create a tunnel to the pool node. Once the tunnel is created, another VM that is subsequently provided and connected to the managed switching element exchanges data with the pool node when this new VM is joined to a logical port on the same logical switching element. Share the same tunnel. When a new node is joined to a logical port on a different logical switch, the logical controller sends out the same universal tunnel flow instruction that was transmitted when the first VM was connected to the managed switching element. However, the universal tunnel flow instruction does not cause a new tunnel to be created to the pool node, for example because the tunnel has already been created and is operational.

If the established tunnel is a one-way tunnel, another one-way tunnel is established from the pool node side. If the logical port to which the first VM is attached is mapped to the port of the managed switching element, the logical controller further sends universal tunnel flow instructions to the pool node. Based on the universal tunnel flow instruction, the chassis controller interface-connecting to the pool node instructs the pool node to create a tunnel to the managed switching element.

In the case of a tunnel between a managed edge switching element and a managed switching element that provides L3 gateway service, a logical switching element containing some of the user's VMs is provided, and the logical router provides the L3 gateway service. It is assumed that it is realized in the transmission node. A logical patch port is created in a logical switching element to link a logical router to a logical switching element. In some embodiments, the order in which the logical patches are created and the VMs are delivered does not make a difference to the tunnel creation. By creating a logical patch port, the logical controller has all the managed switching elements that implement the logical switching element (ie, all managed switching elements each having at least one port to which the logical port of the logical switching element is mapped). ), Sends a universal tunnel flow command to the chassis controller (or physical controller) connected to the interface. Each chassis controller for each of these managed switching elements commands the managed switching elements to create a tunnel to the transmission node. Each of the managed switching elements creates a tunnel to the transmission node, resulting in as many tunnels as there are managed switching elements that implement the logical switching element.

When these tunnels are unidirectional, the transmission node is for creating tunnels to each of the managed switching elements that implement the logical switching elements. When a logical patch port is created and connected to a logical router, the logical switching element pushes universal tunnel flow instructions to the transmission node. The chassis controller interfaced to the transmission node instructs the transmission node to create a tunnel, and the transmission node creates a tunnel to the managed switching element.

In some embodiments, every machine connected to one of the managed switching elements and every machine connected to another managed switching element uses the same logical switching element or the logical ports of two different switching elements. A tunnel established between two managed switching elements, whether or not it is, can be used to exchange data between these two machines. It is an example of tunneling that allows different users managing different sets of LDPs to share managed switching elements while they are isolated.

Creation of a tunnel between two managed switching elements that has a logical port and another logical port connected to the L2 gateway service begins when the logical port is connected to the L2 gateway service. Upon connection, the logical controller sends universal tunnel flow instructions to all managed switching elements that implement the other logical ports of the logical switching element. Based on the instructions, a tunnel is established from these managed switching elements to the managed switching elements that implement the logical ports connected to the L2 gateway service.

III. Electronic system Many of the features and applications described above are implemented as software processing designated as an instruction set recorded on a computer-readable storage medium (also called a computer-readable medium). When these instructions are executed by one or more processing units (eg, one or more processors, processor cores or other processing units), they cause the processing units to perform the operations indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, and the like. Computer-readable media do not include carrier and electronic signals that travel wirelessly or over a wired connection.

As used herein, the term "software" is intended to include firmware or applications stored in magnetic storage in read-only memory that can be read into memory for processing by a processor. Also, in some embodiments, many software inventions can be realized as subparts of a larger program, leaving separate software inventions. In some embodiments, numerous software inventions can also be realized as independent programs. Finally, any combination of independent programs that together realize the inventions of the software described herein are within the scope of the invention. In some embodiments, the software program, when installed to operate on one or more electronic systems, specifies implementations of one or more specific machines that perform and perform the operation of the software program.

FIG. 23 conceptually illustrates the electronic system 2300 used to implement some embodiments of the present invention. An electronic system 2300 may be used to run any of the control applications, virtualization applications or operating system applications described above. The electronic system 2300 may be a computer (eg, desktop computer, personal computer, tablet computer, server computer, mainframe, blade computer, etc.), telephone, PDA or some other type of electronic device. Such electronic systems include interfaces for various computer-readable media and various other computer-readable media. The electronic system 2300 includes a bus 2305, a processing unit 2310, a system memory 2325, a read-only memory 2330, a fixed storage device 2335, an input device 2340, and an output device 2345.

Bus 2305 collectively represents a chipset bus that communicatively connects all systems, peripherals and many internal devices of the electronic system 2300. For example, bus 2305 connects the processing unit 2310 communicably with read-only memory 2330, system memory 2325, and fixed storage device 2335.

From these various memory units, the processing unit 2310 searches for instructions to be executed and data to be processed in order to execute the processing of the present invention. The processing unit may be a single processor or, in different embodiments, a multi-core processor.

Read-only memory (ROM) 2330 stores static data and instructions required by the processing unit 2310 of the electronic system and other modules. On the other hand, the fixed storage device 2335 is a read / write memory element. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 2300 is off. Some embodiments of the present invention use a large capacity storage device (eg, a magnetic disk or optical disk and a corresponding disk drive) as the fixed storage device 2335.

Other embodiments use removable storage devices (eg, floppy disks, flash drives, etc.) as fixed storage devices. Like the fixed storage device 2335, the system memory 2325 is a read / write memory element. However, unlike storage device 2335, system memory is a volatile read / write memory element such as random access memory. System memory stores some of the instructions and data that the processor needs at run time. In some embodiments, the processing of the present invention is stored in system memory 2325, fixed storage 2335 and / or read-only memory 2330. From these various memory units, the processing unit 2310 retrieves the instructions to be executed and the data to be processed to perform the processing of some embodiments.

Bus 2305 also connects to input device 2340 and output device 2345. The input device allows the user to communicate information to the electronic system and select commands. The input device 2340 includes an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). Output device 2345 displays an image generated by the electronic system. Output devices include printers and display devices such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs). Some embodiments include devices such as touch screens that function as both input and output devices.

Finally, as shown in FIG. 23, the bus 2305 further connects the electronic system 2300 to the network 2365 via a network adapter (not shown). As such, the computer may be part of a network of networks such as a computer network (eg, a local area network (LAN), a wide area network (WAN) or an intranet, or the Internet. Any or all of the 2300 components may be used in connection with the present invention.

Some embodiments are electronic such as microprocessors, storage devices and memory that store computer program instructions on a machine-readable or computer-readable medium (also referred to as a computer-readable medium, machine-readable medium or machine-readable storage medium). Includes parts. Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROMs), writable compact discs (CD-R), and rewritable compact discs (CD-RW). Read-only general-purpose discs (eg DVD-ROM, dual-layer DVD-ROM), various writable / rewritable DVDs (eg DVD-RAM, DVD-RW, DVD + RW, etc.), flash memory (eg DVD-RAM, DVD-RW, DVD + RW, etc.) , SD cards, mini SD cards, micro SD cards, etc.), magnetic hard drives and / or solid hard drives, read-only and writable Blu-Ray (Blu-Ray®) discs, ultra-high density optical discs, and anything else. There are optical or magnetic media, as well as floppy discs. A computer-readable medium may contain a computer program that can be executed by at least one processing unit and includes a set of instructions that perform various operations. Examples of computer programs or computer code include, for example, machine code generated by a compiler, files containing high-level code executed by a computer, electronic component or microprocessor using an interpreter.

The above description has primarily referred to microprocessors or multi-core processors running software, but some embodiments may be one or more, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). It is executed by an integrated circuit. In some embodiments, such an integrated circuit executes instructions stored in the circuit itself.

As used herein, the terms "computer," "server," "processor," and "memory" all refer to electronic devices or other technical devices. These terms exclude users or groups of users. For the purposes of this specification, the term "display" means to display on an electronic device. As used herein, the terms "one computer-readable medium," "multiple computer-readable media," and "machine-readable medium" are tangible objects that store information in a computer-readable format. Completely limited to physical objects. These terms exclude any radio signal, wired download signal and any other transient signal.

Although the present invention has been described with reference to many specific details, it will be appreciated by those skilled in the art that the invention may be practiced in other particular embodiments without departing from the gist of the invention. Also, many drawings (including FIGS. 20 and 21) conceptually illustrate the process. Certain actions of these processes do not have to be performed in the exact order shown and described. A particular action may not be performed in a series of successive actions, and different specific actions may be performed in different embodiments. Further processing can be implemented using some sub-processing or as part of a larger macro processing.

Also, some embodiments have been described above in which the user provides an LDP set for LCP data. However, in other embodiments, the user may provide an LDP set for LFP data. Also, some embodiments where the controller instance provides PCP data to the switching element to manage the switching element have been described above. However, in other embodiments, the controller instance may provide physical forwarding plane data to the switching element. In such an embodiment, the relational database data structure stores physical forwarding plane data and the virtualization application produces such data.

Also, in some of the above examples, the user specifies one or more logical switching elements. In some embodiments, the user may provide a physical switching element configuration along with such a logical switching element configuration. Also, in some embodiments, controller instances individually formed by several application layers running on an arithmetic unit will be described, such instances being one or more layers of their operation. It will be appreciated by those skilled in the art that it is formed by a dedicated arithmetic unit or other machine in some embodiments to be performed.

Also, some of the examples above show that LDPS is associated with a single user. It will then be appreciated by those skilled in the art that the user may be associated with one or more sets of LDP sets in some embodiments. That is, the relationship between LDPS and the user is not always one-to-one because the user may be associated with many LDP sets. Therefore, it will be appreciated by those skilled in the art that the present invention is not limited by the above exemplary details.