Untitled record

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation application of U.S. patent application Ser. No. 16/043,926, now published as U.S. Patent Publication 2018/0332124, entitled TRANSPARENT NETWORK SERVICE MIGRATION ACROSS SERVICE DEVICES filed Jul. 24, 2018. U.S. patent application Ser. No. 16/043,926 is a continuation of U.S. patent application Ser. No. 14/195,415, now issued as U.S. Pat. No. 10,110,684, entitled TRANSPARENT NETWORK SERVICE MIGRATION ACROSS SERVICE DEVICES filed Mar. 3, 2014, which claims priority to U.S. Provisional Patent Application No. 61/866,476, entitled TRANSPARENT NETWORK SERVICE MIGRATION ACROSS SERVICE DEVICES filed Aug. 15, 2013. U.S. Provisional Patent Application No. 61/866,476, U.S. patent application Ser. No. 14/195,415, now issued as U.S. Pat. No. 10,110,684, and U.S. patent application Ser. No. 16/043,926, now published as U.S. Patent Publication 2018/0332124, are incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

Many data centers (both enterprise data centers and service provider data centers) are adopting cloud-based infrastructures. In such data centers, a large number of computers connect through a network to provide distributed computing services. One of the primary requirements of a cloud-based computing platform (both public and private) is agility and flexibility to place and move any infrastructure element to anywhere else in the infrastructure.

In a data center environment, it is often necessary to migrate an application from one device to another or otherwise shut down one application and start a new instance. For example, it is sometimes difficult to know resource requirements (e.g., bandwidth, memory, etc.) of applications prior to deployment. A device on which multiple applications are installed may run out of capacity. At this point, the system and/or its administrator would need to migrate a certain application to a different device that has extra capacity. As another example, during operation, applications may need to be migrated among different devices so that resources are more evenly distributed to service the applications. As yet another example, whenever a new version of an application needs to be installed on the devices, the old version is shutdown. When an existing application migrates from one device to another or an old version is shut down, existing flows with clients are typically interrupted. To minimize the impact of the migration, the process should be transparent to the application and its clients. In other words, the application and the clients serviced by the application should be uninterrupted during the migration. Most data centers today are unable to support such uninterrupted migration.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

<figref idref="DRAWINGS">FIG. <b>1</b></figref> is a block diagram illustrating an embodiment of a programmed computer system configured to perform one or more networking services and allow migration of the one or more networking services in accordance with some embodiments.

<figref idref="DRAWINGS">FIG. <b>2</b></figref> is a block diagram illustrating an example of a distributed service platform in its initial state.

<figref idref="DRAWINGS">FIG. <b>3</b></figref> is a block diagram illustrating an example of the distributed service platform in its migrated state.

<figref idref="DRAWINGS">FIG. <b>4</b></figref> is a flowchart illustrating an embodiment of a migration process.

<figref idref="DRAWINGS">FIG. <b>5</b></figref> is a flowchart illustrating an embodiment of a process for handling a packet that is forwarded by the upstream network device to D<b>2</b>.

<figref idref="DRAWINGS">FIG. <b>6</b></figref> is a flowchart illustrating an embodiment of a process for handling a packet that is forwarded by the upstream network device to D<b>1</b> in an embodiment where D<b>2</b> is configured to send the gratuitous ARP after the state information is completely migrated.

<figref idref="DRAWINGS">FIG. <b>7</b></figref> is a messaging diagram illustrating an example set of data that is sent between different components before the network devices undergo the migration process.

<figref idref="DRAWINGS">FIG. <b>8</b></figref> is a messaging diagram illustrating an example set of data that is sent between different components during the migration process.

<figref idref="DRAWINGS">FIG. <b>9</b></figref> is a messaging diagram illustrating another example set of data that is sent between different components during the migration process.

<figref idref="DRAWINGS">FIG. <b>10</b></figref> is a flowchart illustrating an embodiment of a process for migrating state information associated with a TCP flow from D<b>1</b> to D<b>2</b>.

<figref idref="DRAWINGS">FIG. <b>11</b></figref> is a flowchart illustrating another embodiment of a process for migrating state information associated with a TCP flow from D<b>1</b> to D<b>2</b>.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

Migration of a network service between devices is disclosed. A network service that is currently being performed on a first device is migrated so that it can be performed on a second device. In some embodiments, the second device is instructed to notify an upstream network device to forward traffic that is to be serviced by the network service to the second device instead of to the first device, the network service being associated with an Internet Protocol (IP) address. The first device is instructed to migrate an existing flow that is currently serviced by the first device to be serviced by the second device.

As used herein, a flow refers to traffic associated with a connection between two points on a network, such as data packets that are exchanged between two different applications operating on different devices. A flow can be bidirectional or unidirectional. Here, the existing flow that is migrated is an existing flow between a client and a network application.

<figref idref="DRAWINGS">FIG. <b>1</b></figref> is a block diagram illustrating an embodiment of a programmed computer system configured to perform one or more networking services and allow migration of the one or more networking services in accordance with some embodiments. As will be apparent, other computer system architectures and configurations can be used to perform network services and allow migration of the networking services. Computer system <b>100</b>, which includes various subsystems as described below, includes at least one microprocessor subsystem (also referred to as a processor or a central processing unit (CPU)) <b>102</b>. For example, processor <b>102</b> can be implemented by a single-chip processor or by multiple processors. In some embodiments, processor <b>102</b> is a general purpose digital processor that controls the operation of the computer system <b>100</b>. Using instructions retrieved from memory <b>110</b>, the processor <b>102</b> controls the reception and manipulation of input data, and the output and display of data on output devices (e.g., display <b>118</b>). In some embodiments, processor <b>102</b> includes and/or is used to perform the processes described below with respect to <figref idref="DRAWINGS">FIGS. <b>4</b>-<b>6</b></figref>.

Processor <b>102</b> is coupled bi-directionally with memory <b>110</b>, which can include a first primary storage, typically a random access memory (RAM), and a second primary storage area, typically a read-only memory (ROM). As is well known in the art, primary storage can be used as a general storage area and as scratch-pad memory, and can also be used to store input data and processed data. Primary storage can also store programming instructions and data, in the form of data objects and text objects, in addition to other data and instructions for processes operating on processor <b>102</b>. Also as is well known in the art, primary storage typically includes basic operating instructions, program code, data and objects used by the processor <b>102</b> to perform its functions (e.g., programmed instructions). For example, memory <b>110</b> can include any suitable computer-readable storage media, described below, depending on whether, for example, data access needs to be bi-directional or uni-directional. For example, processor <b>102</b> can also directly and very rapidly retrieve and store frequently needed data in a cache memory (not shown).

A removable mass storage device <b>112</b> provides additional data storage capacity for the computer system <b>100</b>, and is coupled either bi-directionally (read/write) or uni-directionally (read only) to processor <b>102</b>. For example, storage <b>112</b> can also include computer-readable media such as magnetic tape, flash memory, PC-CARDS, portable mass storage devices, holographic storage devices, and other storage devices. A fixed mass storage <b>120</b> can also, for example, provide additional data storage capacity. The most common example of mass storage <b>120</b> is a hard disk drive. Mass storage <b>112</b>, <b>120</b> generally store additional programming instructions, data, and the like that typically are not in active use by the processor <b>102</b>. It will be appreciated that the information retained within mass storage <b>112</b> and <b>120</b> can be incorporated, if needed, in standard fashion as part of memory <b>110</b> (e.g., RAM) as virtual memory.

In addition to providing processor <b>102</b> access to storage subsystems, bus <b>114</b> can also be used to provide access to other subsystems and devices. As shown, these can include a display monitor <b>118</b>, a network interface <b>116</b>, a keyboard <b>104</b>, a pointing device <b>106</b>, as well as an auxiliary input/output device interface, a sound card, speakers, and other subsystems as needed. One or more subsystems of each type can be included, and some subsystems can be omitted.

The network interface <b>116</b> allows processor <b>102</b> to be coupled to another computer, computer network, or telecommunications network using a network connection as shown. For example, through the network interface <b>116</b>, the processor <b>102</b> can receive information (e.g., data objects or program instructions) from another network or output information to another network in the course of performing method/process steps. Information, often represented as a sequence of instructions to be executed on a processor, can be received from and outputted to another network. An interface card or similar device and appropriate software implemented by (e.g., executed/performed on) processor <b>102</b> can be used to connect the computer system <b>100</b> to an external network and transfer data according to standard protocols. For example, various process embodiments disclosed herein can be executed on processor <b>102</b>, or can be performed across a network such as the Internet, intranet networks, or local area networks, in conjunction with a remote processor that shares a portion of the processing. Additional mass storage devices (not shown) can also be connected to processor <b>102</b> through network interface <b>116</b>.

An auxiliary I/O device interface (not shown) can be used in conjunction with computer system <b>100</b>. The auxiliary I/O device interface can include general and customized interfaces that allow the processor <b>102</b> to send and, more typically, receive data from other devices such as microphones, touch-sensitive displays, transducer card readers, tape readers, voice or handwriting recognizers, biometrics readers, cameras, portable mass storage devices, and other computers.

In addition, various embodiments disclosed herein further relate to computer storage products with a computer readable medium that includes program code for performing various computer-implemented operations. The computer-readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of computer-readable media include, but are not limited to, all the media mentioned above: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks; and specially configured hardware devices such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and ROM and RAM devices. Examples of program code include both machine code, as produced, for example, by a compiler, or files containing higher level code (e.g., script) that can be executed using an interpreter.

The computer system shown in <figref idref="DRAWINGS">FIG. <b>1</b></figref> is but an example of a computer system suitable for use with the various embodiments disclosed herein. Other computer systems suitable for such use can include additional or fewer subsystems. In addition, bus <b>114</b> is illustrative of any interconnection scheme serving to link the subsystems. Other computer architectures having different configurations of subsystems can also be utilized.

<figref idref="DRAWINGS">FIG. <b>2</b></figref> is a block diagram illustrating an example of a distributed service platform in its initial state.

In this example, client devices <b>214</b> interact with network applications <b>204</b> and <b>206</b> which execute on devices on network <b>200</b>. A client device can be a laptop computer, a desktop computer, a tablet, a mobile device, a smart phone, or any other appropriate computing device. In various embodiments, a web browser, a special purpose application, or other appropriate client application is installed at the client device, enabling a user to access network applications <b>204</b> and <b>206</b> via a network (e.g., the Internet). A network application (also referred to as a target application) is an application that is provided over a network. Examples of network applications include web applications, shopping cart applications, user authentication, credit card authentication, email, file sharing, virtual desktops, voice/video streaming, online collaboration, etc. The network applications may execute on application servers

Network <b>200</b> can be a data center network, an enterprise network, or any other appropriate network. On network <b>200</b>, device <b>201</b> (also referred to as D<b>1</b> or the original device) is configured to provide a first network service <b>210</b> to a first set of network applications <b>204</b> (“App1”), and a second network service <b>212</b> to a second set of network applications <b>206</b> (“App2”). As used herein, a device refers to an entity with one or more network interfaces through which networking traffic (e.g., packets) is sent and received. A device can be implemented using hardware, software, or a combination thereof. A device can be a physical device (e.g., a physical server computer), a virtual device (e.g., a virtual machine such as VMWare™), or a combination thereof. A network interface can be implemented as a physical port (e.g., an Ethernet port, a wireless interface, etc.), a virtual port (e.g., software emulation of a physical port), or a combination thereof. A network service processes traffic between one or more clients and one or more network applications, providing services on behalf of the applications. Examples of network services include load balancing, authorization, security, content acceleration, analytics, application management, etc. Each network service can be implemented as a set of software code (e.g., a software process or a part of a process) that executes on hardware. In this example, the first network service <b>210</b> is a load balancing service that balances processing loads among applications <b>204</b>, and the second network service <b>212</b> is a firewall service that filters traffic sent to applications <b>206</b>.

In this example, each network service provides service to its corresponding network applications under a unique Fully Qualified Domain Name (FQDN), which is translated into a unique Internet Protocol (IP) address (also referred to as a virtual IP address). In this example, network applications <b>204</b> and network applications <b>206</b> are configured to be accessible to clients <b>214</b> at corresponding domain names. Each domain name corresponds to a unique IP address that is resolved by a DNS server. Specifically, the DNS server stores the mappings of FQDNs to IP addresses, and can be used to look up the virtual IP address of the network service that corresponds to a particular FQDN. In this example, applications <b>204</b> are accessible at the URL of “App1.avinetworks.com,” which corresponds to the IP address of 123.7.8.9. Applications <b>206</b> are accessible at the URL of “App2.avinetworks.com,” which corresponds to the IP address of 123.4.5.6.

Instances of network applications <b>204</b> and <b>206</b> can operate on device <b>201</b> and/or one or more other devices. The configuration is flexible and can be different in various embodiments. In this example, traffic (e.g., packets associated with traffic flows) from client devices <b>214</b> is sent to device <b>201</b> and processed by network service <b>210</b> or <b>212</b>. For example, traffic designated for applications <b>204</b> (e.g., requests for the URL of “App1.avinetworks.com” sent by the clients) is load balanced by service <b>210</b> and each flow is sent to a selected one of applications <b>204</b> to be further processed. Traffic designated for applications <b>206</b> (e.g., requests for the URL of “App2.avinetworks.com” sent by the clients) is filtered by firewall service <b>212</b> and sent to an application <b>206</b> to be further processed.

An upstream network device <b>216</b> is configured to forward traffic from client devices <b>214</b> destined for the network applications to network devices such as D<b>1</b> and/or D<b>2</b>. Examples of an upstream network device include a router, a switch, a bridge, etc. While the same upstream network devices can also be configured to forward packets from network <b>200</b> to client devices <b>214</b>, for purposes of discussion, network <b>200</b> is referred to as the end of the network traffic stream from the client, and points on the network path before the packet reaches network <b>200</b> are said to be upstream from the network.

In this example, the upstream network device maintains configuration information that includes routing information (e.g., a routing table) specifying the routes for certain network destinations. The configuration information further includes the mapping of IP addresses to hardware identification information (e.g., an ARP table). Prior to the migration, upstream network device <b>216</b> maintains an ARP table that maps the IP address of 123.4.5.6 and 123.7.8.9 to one or more MAC addresses on D<b>1</b>.

Later, it is determined that D<b>1</b> does not have sufficient resources to provide both network services <b>210</b> and <b>212</b>, therefore network service <b>212</b> needs to be migrated to another device <b>202</b> (also referred to as D<b>2</b> or the replacement device). During migration, state information associated with existing flows are copied or moved from D<b>1</b> to D<b>2</b>, and the flows themselves are uninterrupted. The determination that a migration should take place can be made by D<b>1</b> itself or by a controller <b>220</b>. For example, a monitoring application on D<b>1</b> or the controller may determine that the number of flows handled by network service <b>212</b> has been exceeded. According to preconfigured policies, D<b>1</b> or the controller will initiate the migration process. The migration process is described in greater detail below.

<figref idref="DRAWINGS">FIG. <b>3</b></figref> is a block diagram illustrating an example of the distributed service platform in its migrated state. In this example, the migration process has been successfully completed. Network service <b>212</b> on D<b>1</b> has been migrated to D<b>2</b>, which executes another instance of the same network service, also labeled as network service <b>212</b>. New flows from clients <b>214</b> are serviced by network service <b>212</b> operating on D<b>2</b>, and then sent to an application <b>206</b>. Existing flows that used to be serviced by network service <b>212</b> on D<b>1</b> have also been migrated, and are now serviced by network service <b>212</b> on D<b>2</b>. As will be discussed in greater detail below, the existing flows are uninterrupted, making the migration process transparent to the existing flows.

In some embodiments, a controller initiates the migration process. The controller can be implemented as a software application, a hardware device or component, or a combination. The controller is configured to control the operations of the network services and/or the network devices. In some embodiments, the controller also monitors the health and performance of the network services and/or the network devices. The controller can be a stand-alone device, or be integrated with another device such as D<b>1</b>.

<figref idref="DRAWINGS">FIG. <b>4</b></figref> is a flowchart illustrating an embodiment of a migration process. Process <b>400</b> can be performed by a controller such as <b>220</b> of <figref idref="DRAWINGS">FIG. <b>2</b></figref>. In some embodiments, process <b>400</b> is invoked when the controller, based on its observation of the performance of network services on D<b>1</b>, determines that D<b>1</b> has run out of resources (e.g., connections, memory, etc.) and cannot adequately provide the network services. In some embodiments, process <b>400</b> is invoked when the controller receives diagnostic or other data sent by D<b>1</b> indicating that the one can no longer adequately provide existing network services.

Referring to <figref idref="DRAWINGS">FIG. <b>2</b></figref>, an upstream network device (e.g., router <b>216</b>) is configured to forward packets from clients <b>214</b> to network devices such as D<b>1</b> and/or D<b>2</b>. Initially, according to routing and ARP information, the upstream network device is configured to forward traffic destined for the IP address associated with service <b>212</b> to D<b>1</b>.

At <b>402</b>, D<b>2</b> is instructed to notify an upstream network device to forward traffic that is to be serviced by the network service to be migrated (in this case, network service <b>212</b>) to D<b>2</b> to instead of D<b>1</b>. Some examples of how to accomplish this are described more fully below.

In some embodiments, the upstream network device maintains an ARP table that maps IP addresses to hardware addresses (e.g., network interface addresses). The upstream network device uses the ARP table to determine, for each packet that is destined for a certain IP address, the corresponding device/network interface to which the packet should be forwarded. To perform <b>402</b>, D<b>2</b> is instructed to notify the upstream network device by sending a gratuitous Address Resolution Protocol (ARP) that associates a network interface identifier of D<b>2</b> (e.g., D<b>2</b>'s MAC address) with the network service's virtual IP address. Upon receiving the notification, the upstream network device updates its ARP table, changing the IP address from being mapped to the MAC address of D<b>1</b> to being mapped to the MAC address of D<b>2</b>. Thereafter, the upstream network device forwards packets designated for the IP address to D<b>2</b>. The format of the gratuitous ARP message is specified by existing ARP protocol, and the mechanism for sending the message is known to those skilled in the art. Depending on implementation, the gratuitous ARP can be sent as soon as D<b>2</b> is instructed to notify the upstream network device, or at a later time after existing flows are migrated.

At <b>404</b>, D<b>1</b> is instructed to migrate the network service to D<b>2</b>. Migration of the network service includes copying or moving state information associated with one or more existing flows currently serviced by D<b>1</b> to D<b>2</b>. In various embodiments, an indication of the instruction to migrate is sent in a packet, a message, an inter-process communication (IPC) call, or any other appropriate format. D<b>1</b> can copy or move the state information according to a preconfigured message or other data format.

Typically, a flow is established when a connection request is made by one device at one point, and is ended when the connection is closed by either devices at either points or due to inactivity/timeout. State information associated with existing flows serviced by D<b>1</b> is maintained in a flow table using known techniques. In some embodiments, the state information includes identification information of the flow, and information about the status and/or processing of the flow. In various embodiments, what is included in the state information depends on the connection and/or the requirements of the network service. For example, the state information can include sequence information used to reconstruct the TCP state, information about the load balanced server selected by the load balancing network service to handle the flow, whether a flow is permitted according to the firewall network service, security/encryption state of a flow according to a security network service, etc.

To migrate an existing flow, state information associated with the existing flow is moved or copied from D<b>1</b> to D<b>2</b>, ensuring that D<b>2</b> has the relevant information to process incoming packets associated with existing flow in the same way D<b>1</b> would, so that the existing flow is uninterrupted. Specifically, the TCP connection for the flow is not broken; the flow states are maintained with respect to the client; and the client's interactions with its corresponding network application remain the same as before the migration. For example, if the state information recorded by load balancing service <b>212</b> on D<b>1</b> indicates that an existing flow is sent to a specific server running a specific instance of network application <b>206</b>, then the migration process copies the state information to D<b>2</b> to ensure that packets associated with this flow are still sent to the same server/network application instance and that the flow is uninterrupted. In some embodiments, the migration of state information is conducted using a proprietary protocol implemented based on inter-process communication (IPC) methods, where a function call is invoked by the first device to send, to the second device, state information associated with the one or more existing flows being migrated.

As will be described in greater detail below, depending on implementation, <b>402</b> and <b>404</b> can be performed in different order, including in parallel.

In some embodiments, D<b>2</b> sends the gratuitous ARP as soon as possible. For example, the controller can send an indication (e.g., a predefined message) indicating that the network service is to be migrated from D<b>1</b> to D<b>2</b>, where the indication includes the virtual IP address of the network service to be migrated. In response, D<b>2</b> sends the gratuitous ARP. Thus future packets designated for the IP address (including new flows and for existing flows) are immediately forwarded by the upstream network device to D<b>2</b> instead of D<b>1</b>. In such embodiments, D<b>1</b> will not receive further flows designated for the IP address once the upstream network device's ARP table is updated based on the gratuitous ARP sent by D<b>2</b>. <figref idref="DRAWINGS">FIG. <b>5</b></figref> is a flowchart illustrating an embodiment of a process for handling a packet that is forwarded by the upstream network device to D<b>2</b>. In this example, process <b>500</b> is performed by D<b>2</b>, which is configured to send the gratuitous ARP as soon as possible. In other words, prior to process <b>500</b>, D<b>2</b> has received an indication that migration has begun, and sent the gratuitous ARP to the upstream network device.

At <b>502</b>, a packet is received at D<b>2</b>.

At <b>504</b>, it is determined whether the packet is associated with a new flow or an existing flow. In a TCP flow, the determination is made by checking a SYN flag in the packet. The SYN flag is set if the packet is associated with a new flow (i.e., it is the first packet in a flow), and is not set if the packet is associated with an existing flow.

If the packet is associated with a new flow, at <b>505</b>, state information is created for the new flow and stored. For example, the source and destination addresses and port number are determined based on the header of the packet. Any other appropriate state information can also be determined based on the packet. The state information is stored in a table or other appropriate data structure.

At <b>506</b>, the packet is processed by D<b>2</b>. In other words, the network service is applied on the packet by D<b>2</b>, using the state information as appropriate, and the resulting packet is forwarded to the appropriate applications.

Packets can be received on an existing flow while the migration of state information from D<b>1</b> to D<b>2</b> is still in process. A packet associated with an existing flow should be handled by the device that has the state information associated with the existing flow. Thus, if the packet is associated with an existing flow, at <b>508</b>, it is determined whether state information associated with the flow is available locally (i.e., whether the state information is available to D<b>2</b>). In some embodiments, D<b>2</b> maintains state information locally in a table that is indexed according to a combination of the source address, the destination address, and the port of the flow. Other formats and data structures (e.g., a tree, a list, etc.) are possible. To determine whether state information associated with the flow is available locally, the source address, destination address, and port of the packet are used to construct an index to conduct a lookup in the table. If an entry is found, it means that D<b>2</b> has state information associated with the existing flow, and therefore is capable of handling traffic associated with the flow. Control is therefore transferred to <b>506</b>, where the packet is processed.

If, however, no state information is available locally, it indicates that the state information for this flow has not yet migrated from D<b>1</b> to D<b>2</b>. D<b>2</b> is therefore not yet capable of handling traffic associated with the flow. Accordingly, at <b>510</b>, the packet is forwarded to D<b>1</b> to be processed. Specifically, D<b>2</b> will rewrite the destination MAC address of the packet to be the MAC address associated with D<b>1</b> and send the packet to D<b>1</b> via either the network interface on which the packet is received or another appropriate network interface.

D<b>1</b> continues to send state information associated with existing flows until state information associated with all existing flows that need to be migrated are copied to D<b>2</b>. In some embodiments, D<b>1</b> notifies D<b>2</b> when the former has completely migrated its flow state information to the latter, so that D<b>2</b> will discontinue process <b>500</b> but instead process all packets received.

In some embodiments, D<b>2</b> waits to send the gratuitous ARP after state information associated with the flows is completely migrated from D<b>1</b> to D<b>2</b>. Accordingly, future packets designated for the IP address are still forwarded by the upstream network device to D<b>1</b> until the ARP is sent and the upstream network device's ARP table is updated. <figref idref="DRAWINGS">FIG. <b>6</b></figref> is a flowchart illustrating an embodiment of a process for handling a packet that is forwarded by the upstream network device to D<b>1</b> in an embodiment where D<b>2</b> is configured to send the gratuitous ARP after the state information is completely migrated.

Process <b>600</b> is performed by D<b>1</b>. At <b>602</b>, a packet is received. At <b>604</b>, it is determined if the packet is associated with a new flow or an existing flow. In some embodiments, the SYN flag of the packet is examined to determine whether the packet is associated with a new flow. If the packet is a first packet in a new flow with the SYN flag set, it should be processed by D<b>2</b> since D<b>2</b> is supposed to be eventually in charge of handling all flows destined for the IP address. Accordingly, at <b>606</b>, the packet is forwarded to D<b>2</b>. Specifically, the destination MAC address of the packet is rewritten to be the MAC address of D<b>2</b>, and the rewritten packet is forwarded to D<b>2</b>. D<b>2</b> will process the packet and add an entry to its state table. If the packet is associated with an existing flow, at <b>608</b>, whether state information is available locally (i.e., whether the state information is available to D<b>1</b>) is checked. As discussed above, the check can be performed by looking up in a state table maintained by D<b>1</b> using index information such as the source and destination addresses and the port number of the packet. If the state information is found to be available locally, then D<b>1</b> processes the packet at <b>610</b>. If, however, the state information is no longer available locally, it indicates that the state information associated with this flow has already been migrated to D<b>2</b>. Thus, at <b>606</b>, the packet is forwarded to D<b>2</b>.

A specific example of migration is given in connection with <figref idref="DRAWINGS">FIG. <b>2</b></figref> and <figref idref="DRAWINGS">FIGS. <b>7</b>-<b>9</b></figref>. In the example, messages are sent between components of a network such as <b>200</b> of <figref idref="DRAWINGS">FIGS. <b>2</b> and <b>3</b></figref>. The arrows indicate the directions of the messages.

<figref idref="DRAWINGS">FIG. <b>7</b></figref> is a messaging diagram illustrating an example set of data that is sent between different components before the network devices undergo the migration process. Initially, as indicated by arrow <b>702</b>, the client sends a packet for a new flow that is destined for the IP address of 123.4.5.6 according to the result of a DNS resolution operation. The packet is received by the upstream networking device, which processes the packet and sends an ARP request for the address of 123.4.5.6. The ARP request is broadcasted to network devices on the network, such as D<b>1</b> and D<b>2</b>, as shown by arrows <b>704</b>. At this point, only D<b>1</b> is configured to service packets destined for the address of 123.4.5.6. Accordingly, as shown by arrow <b>706</b>, D<b>1</b> responds to the ARP request by sending to the upstream networking device an ARP reply that maps the address of 123.4.5.6 to D<b>1</b>'s network interface address, MAC1. In the ARP table of the upstream networking device, an entry that maps 123.4.5.6 to MAC1 is created. Additional data exchange may take place until such time when it is determined that service <b>212</b> should be migrated from D<b>1</b> to D<b>2</b>. A “begin migration” message is sent by the controller to D<b>1</b> and D<b>2</b>, as shown by arrows <b>708</b>. In some embodiments, a protocol is pre-defined for communications between the controller and the network devices, implemented using mechanisms such as inter-process communication (IPC). The “begin migration” message is formatted according to the protocol, and indicates to D<b>1</b> and D<b>2</b> that service <b>212</b> is to be migrated from D<b>1</b> to D<b>2</b>.

<figref idref="DRAWINGS">FIG. <b>8</b></figref> is a messaging diagram illustrating an example set of data that is sent between different components during the migration process. In this example, the “begin migration” message is sent by the controller (arrow <b>802</b>) to D<b>1</b> and D<b>2</b>. D<b>2</b> immediately launches another instance of network service <b>212</b>, and sends a gratuitous ARP message mapping the IP address of 123.4.5.6 to D<b>2</b>'s own network interface address, MAC2, as shown by arrow <b>804</b>. Upon receiving the gratuitous ARP, the upstream networking device will update its ARP table, changing the mapping of 123.4.5.6 to MAC2.

Meanwhile, state information for existing flows is sent from D<b>1</b> to D<b>2</b>, as shown by arrow <b>805</b>. The state information can be transferred one entry at a time or in a batch with multiple entries.

While the state information is being migrated from D<b>1</b> to D<b>2</b>, the clients continue to send traffic. A packet associated with a new flow destined for 123.4.5.6 is sent by a client to the upstream networking device, as shown by arrow <b>806</b>. The upstream networking device looks up in its ARP table the MAC address associated with 123.4.5.6, and forwards the packets to D<b>2</b>, as shown by arrow <b>808</b>. Network service <b>212</b> on D<b>2</b> processes the packet and creates a new state entry in D<b>2</b>'s state table.

A packet associated with an existing flow destined for 123.4.5.6 is sent by a client to the upstream networking device, as shown by arrow <b>810</b>. Again, the upstream networking device looks up in its ARP table the MAC address associated with 123.4.5.6, and forwards the packets to D<b>2</b>, as shown by arrow <b>812</b>. If D<b>2</b> has state information associated with the flow, it will process the packet; otherwise, D<b>2</b> will change the MAC address of the packet from MAC2 to MAC1, and forwards the packet to D<b>1</b>, as shown by arrow <b>814</b>. Since it currently maintains the state information associated with the flow, D<b>1</b> will process the packet.

<figref idref="DRAWINGS">FIG. <b>9</b></figref> is a messaging diagram illustrating another example set of data that is sent between different components during the migration process. In this example, the “begin migration” message is sent by the controller (arrow <b>901</b>) to D<b>1</b> and D<b>2</b>. Upon receiving the message, D<b>2</b> launches another instance of network service <b>212</b>, but does not immediately send the gratuitous ARP to the upstream networking device.

When a packet associated with a new flow destined for 123.4.5.6 is sent by a client to the upstream networking device (arrow <b>902</b>), the upstream networking device will continue to forward the packet to D<b>1</b> since nothing in its ARP table has changed (arrow <b>904</b>). D<b>1</b>, however, having been notified that the migration is in process, will change the network interface address of the packet from MAC1 to MAC2, and forwards the packet to D<b>2</b> (arrow <b>906</b>). D<b>2</b> will process the packet and create a new state entry in its state table.

Similarly, when a packet associated with an existing flow destined for 123.4.5.6 is sent by a client to the upstream networking device (arrow <b>908</b>), the upstream networking device will continue to forward the packet to D<b>1</b> since nothing in its ARP table has changed (arrow <b>910</b>). D<b>1</b> will check whether the state information associated with the flow is available. If so, D<b>1</b> will service the packet. If no state information associated with the flow is available, the state information has already been migrated to D<b>2</b>. Thus D<b>1</b> will change the MAC address of the packet to MAC2, and forward the packet to D<b>2</b> (arrow <b>912</b>).

As discussed above, when D<b>2</b> forwards the received packet to D<b>1</b> (e.g., <b>510</b> of process <b>500</b>) or when D<b>1</b> forwards the received packet to D<b>2</b> (e.g., <b>606</b> of process <b>600</b>), the MAC address needs to be rewritten (e.g., MAC2 needs to be changed to MAC1 or vice versa). In some embodiments, each device implements a bottom layer referred to as a flow dispatch layer in its networking stack. The flow dispatch layer processes a packet that is received on the device's network interface and before the packet enters the TCP/IP stack. The flow dispatch layer performs the rewriting of the MAC address and forwarding of the packet. Since the rewriting and forwarding are performed by the flow dispatch layer before the packet enter into the TCP/IP stack, the amount of work that needs to be performed by each device in association with the forwarding action is reduced.

As discussed above, during the migration of the network service, D<b>1</b> continues to handle existing flows to avoid disruption. These flows need to be moved to the new device in a reasonable amount of time. Some of these flows naturally complete; however, some flows stay for a long period of time.

Each flow has a state associated with it that is stored on D<b>1</b>. In some embodiments, depending on the type of service that is applied to the flow, D<b>1</b> handles differently how the flow state is migrated from D<b>1</b> to D<b>2</b>.

<figref idref="DRAWINGS">FIG. <b>10</b></figref> is a flowchart illustrating an embodiment of a process for migrating state information associated with a TCP flow from D<b>1</b> to D<b>2</b>. Process <b>1000</b> is performed by D<b>1</b> where D<b>1</b> is not configured to terminate the TCP connection from the client, but merely doing cut-through forwarding of the data to the application servers. For example, D<b>1</b> can be configured to implement Network Address Translation (NAT) of the TCP connection using a technique known as TCPFast).

At <b>1002</b>, D<b>1</b> slows down the client from sending data in the flow by stopping or slowing the opening the TCP window associated with the flow. In this example, D<b>1</b> decreases the size of the TCP window available to receive data each time a packet is received from the client, indicating to the client that D<b>1</b> can no longer accept data at the same rate and the client must slow down its transmission.

At <b>1004</b>, state information needed to reconstruct the flow is sent to D<b>2</b>. In some embodiments, the state information includes information needed to reconstruct the TCP state for the TCP flow, such as sequence number, etc. The state information is sent to D<b>2</b> in a control message.

At <b>1006</b>, packets for this flow are forwarded to D<b>2</b>. In some embodiments, the packets are forwarded using D<b>1</b>'s dispatch layer. D<b>2</b> has by now constructed the TCP state for this flow and can handle this TCP flow. D<b>2</b> also opens the TCP window to its normal levels.

At <b>1008</b>, state information (e.g., TCP or proxy state) for the existing flow is cleaned up (e.g., deleted).

<figref idref="DRAWINGS">FIG. <b>11</b></figref> is a flowchart illustrating another embodiment of a process for migrating state information associated with a TCP flow from D<b>1</b> to D<b>2</b>. Process <b>1100</b> is performed by D<b>1</b> where D<b>1</b> is configured to terminate the TCP connection from the client and to open a new TCP connection to the application server (in other words, D<b>1</b> is acting as a full TCP proxy).

At <b>1102</b>, D<b>1</b> stops opening the TCP window to slow down the client. D<b>1</b> decreases the size of the TCP window as data is received from the client, indicating to the client that D<b>1</b>'s buffer is getting full and the client must slow down its transmission.

At <b>1104</b>, D<b>1</b> switches to a mode where in its role as the proxy, D<b>1</b> is not buffering any data. In effect this switches the flow to being in a TCPFast mode. This step is preferably done at a transaction boundary so that the connection to the application server is no longer needed until the next transaction from the client.

At <b>1106</b>, D<b>1</b> sends the state information needed to reconstruct the flow to D<b>2</b>. In some embodiments, the state information includes information needed to reconstruct the TCP state for the TCP flow, such as sequence number, etc. The state information is sent in a control message.

At <b>1108</b>, D<b>1</b> starts forwarding packets for this flow to D<b>2</b>. In some embodiments, the forwarding is performed at D<b>1</b>'s dispatcher layer.

To handle the flow, D<b>2</b> opens a TCP window, switches to a full proxy mode, and performs the network service (e.g., making a new load balancing decision) for the flow from the client.

At <b>1110</b>, D<b>1</b> cleans up (e.g., deletes) any TCP/proxy state for this flow.

Since migration of state information can take place while data is received by the devices, process <b>1000</b> may execute while process <b>500</b> or <b>600</b> is executing. Similarly, process <b>1100</b> may execute white processes <b>500</b> or <b>600</b> is executing.

Migration of a network service between devices has been described. The technique allows the migration to take place in a transparent manner, without interrupting the traffic flow between the client device and a network application accessed by the client device.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.