System, method and computer program product for monitoring and controlling network connections from a supervisory operating system

Description of equivalent WO 2004019162 A2

SYSTEM, METHOD AND COMPUTER PROGRAM PRODUCT FOR MONITORING AND CONTROLLING NETWORK CONNECTIONS FROM A SUPERVISORY

OPERATING SYSTEM

COPYRIGHT NOTIFICATION

[0001 ] Portions of this patent application contain materials that are subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document, or the patent disclosure, as it appears in the Patent and Trademark Office, but otherwise reserves all copyright rights.

COMPUTER PROGRAM LISTING APPENDIX

[0002] A computer program listing appendix on a compact disc incorporating features of the present invention has been submitted with the earlier national application, U.S. Application S.N. 10/226,106, in which this patent application claims priority, and is incorporated herein by reference in its entirety. The files contained on the disc are: sourcecode\apps\-pv4\p-ugins\lnet_ft.c, 7895, Aug 15 14:36; sourcecode\apps \ipv4\plugins\Makefile, 713, Aug 15 14:36; sourcecode\apps\ipv4\plugins .lnet_icmp.c, 13785, Aug 15 14:36; sourcecode\apps\ipv4\plugins\lnet_udp.c, 11309, Aug 15 14:36; sourcecode\apps\ipv4\plugins\lnet_tabldr.c, 999, Aug 15 14:36; sourcecode\apps\ipv4\lnet_ipv4.c, 15626, Aug 15 14:36; sourcecode\apps\ipv4\Makefile, 541, Aug 15 14:36; sourcecode\apps\gpos\lnet_gpos.c, 17258, Aug 15 14:36; sourcecode\apps\gpos\Makefιle, 466, Aug 15 14:36; sourcecode\apps\arp\Makefile, 457, Aug 15 14:36; sourcecode\apps\arp\lnet_arp.c, 10964, Aug 15 14:36; sourcecode\scripts\defconfig, 426, Aug 15 14:36; sourcecode\scripts\ft, 0, Aug 15 14:36; sourcecode\scripts\functions.sh, 7148, Aug 15 14:36; sourcecode\scripts\confιg.in, 1336, Aug 15 14:36; sourcecode\scripts\test_udp, 3300, Aug 15 14:36; sourcecode\scripts\test_ip, 3271, Aug 15 14:36; sourcecode\scripts\Menuconfιg, 30024, Aug 15 14:36; sourcecode\scripts\Configure, 12372, Aug 15 14:36; sourcecode\scripts\mkdep.c, 12136, Aug 15 14:36; sourcecode\scripts\Makefile, 1597, Aug

15 14:36; sourcecode\scripts\unload_arp, 659, Aug 15 14:36; sourcecode\scripts\load_ip,

3008, Aug 15 14:36; sourcecode\scripts\test_arp, 2077, Aug 15 14:36; sourcecode\scripts\load_arp, 1 153, Aug 15 14:36; sourcecode\scripts\test_lnet, 3239, Aug 15 14:36; sourcecode\scripts\inslnet, 3885, Aug 15 14:36; sourcecode\scripts\localinfo,

372, Aug 15 14:36; sourcecode\scripts\hosts, 651, Aug 15 14:36; sourcecode\scripts\rmlnet, 1124, Aug 15 14:36; sourcecode\scripts\ping, 2153, Aug 15

14:36; sourcecode\scripts\addip, 3173, Aug 15 14:36; sourcecode\scripts\unload_ip, 1137,

Aug 15 14:36; sourcecode\scripts\msgbox.c, 2529, Aug 15 14:36; sourcecode\scripts\inputbox.c, 6179, Aug 15 14:36; sourcecode\scripts\yesno.c, 3067, Aug

15 14:36; sourcecode\scripts\colors.h, 5384, Aug 15 14:36; sourcecode\scripts\checklist.c,

9584, Aug 15 14:36; sourcecode\scripts\menubox.c, 12716, Aug 15 14:36; sourcecode\scripts\dialog.h, 5936, Aug 15 14:36; sourcecode . scripts . textbox.c, 15584, Aug

15 14:36; sourcecode . scripts\util.c, 9604, Aug 15 14:36; sourcecode\scripts\lxdialog.c, 6023, Aug 15 14:36; sourcecode\main\lnet.c, 21899, Aug 15 14:36; sourcecode\main\Makef-le, 172, Aug 15 14:36; sourcecode\include\lnet.h, 6253, Aug 15

14:36; sourcecode\include\lnet_udp.h, 3463, Aug 15 14:36; sourcecode\include\lnet_icmp.h, 2856, Aug 15 14:36; sourcecode\include\lne-_arp.h, 1417,

Aug 15 14:36; sourcecode\include\lnet_ipv4.h, 4172, Aug 15 14:36; sourcecode\include\-net_hw.h, 1673, Aug 15 14:36; sourcecode\include\lnet_gpos.h, 1435,

Aug 15 14:36; sourcecode\doc\api.txt, 7841, Aug 15 14:36; sourcecode\doc\ipv4.txt, 6923,

Aug 15 14:36; sourcecode\doc\udp.txt, 4171, Aug 15 14:36; sourcecode\doc\arp.txt, 2664,

Aug 15 14:36; sourcecode\doc\icmp.txt, 4136, Aug 15 14:36; sourcecode\doc\gpos.txt,

5055, Aug 15 14:36; sourcecode\doc\faq.txt, 4855, Aug 15 14:36; sourcecode\doc\getting_started.txt, 3690, Aug 15 14:36; sourcecode\doc\configuration.txt,

1847, Aug 15 14:36; sourcecode\doc\scripts.txt, 2663, Aug 15 14:36; sourcecode\doc\Configure.help, 4154, Aug 15 14:36; sourcecode . GNUmakefile, 4188,

Aug 15 14:36; sourcecode . drivers\lnet_pcnet32.c, 21711, Aug 15 14:36; sourcecode\drivers\lnet_3c905.c, 34753, Aug 15 14:36; sourcecode\drivers\lnet_eeprol00.c, 30847, Aug 15 14:36; sourcecode\drivers\Makefιle,

624, Aug 15 14:36; sourcecode\tests\lnet_arp_test\lnet_arp_test.c, 2039, Aug 15 14:36; sourcecode\tests\lnet_arp_test\Makefile, 488, Aug 15 14:36; sourcecode\tes-s .lnet_ip_test\lnet_ip_test.c, 10396, Aug 15 14:36; sourcecode\-ests\lnet_ip_test\Makefιle, 483, Aug 15 14:36; sourcecode\tests\lnet_ping\lnet_ping.c, 6487, Aug 15 14:36; sourcecode\tests\lnet_ping\Makefile, 465, Aug 15 14:36; sourcecode\tests\lnet_udp_test\lnet_udp_test.c, 10254, Aug 15 14:36; sourcecode\tests\lnet_udp_test\Makefile, 488, Aug 15 14:36; sourcecode\-ests\lnet_test\lnet_test.c, 9744, Aug 15 14:36; sourcecode\tests\lnet_test\Makefile, 181, Aug 15 14:36; sourcecode\skeletons\lnet_ipv4_plugin.c, 4926, Aug 15 14:36; sourcecode\skeletons\lnet_driver.c, 22332, Aug 15 14:36; sourcecode\skeletons\lnet_decoupled_app.c, 5523, Aug 15 14:36; sourcecode\skeletons\lnet_simple_app.c, 4510, Aug 15 14:36; sourcecode\skeletons\Makefile, 284, Aug 15 14:36; sourcecode\Rules.make, 188, Aug 15 14:36; sourcecode\Copyright, 76, Aug 15 14:37.

BACKGROUND OF THE INVENTION

1. Field of the Invention

[0003] The present invention relates to computer networks and data processing systems and, more specifically, to and a system, method, and computer program product for monitoring and controlling network connections from a supervisory operating system

2. Discussion of the Background

[0004] Networked computers cooperating on computations or implementing communication systems, such as SS7, are subject to hardware failures in communication links, switches, hubs, and network hosts, as well as software failures in software implementing or using communication protocols. As network speeds increase and as quality demands increase on service providers,, controlling bandwidth allocation, responding to out of band events, and monitoring performance and security becomes critical. However, most networking protocols do not directly or efficiently allow for this type of functionality. For example, TCP/IP, a widely used networking protocol, is designed to be tolerant of timing fluctuations and therefore does not have a method of rapidly discovering network failures. During the operation of a network stack, handling of timing events or out of band signals may be delayed by stack or operating system scheduling. Other drawbacks and disadvantages exist. [0005] "A Retrospective on the VAX VMM Security Kernel," by Karger et al. describes the development of a virtual-machine monitor (VMM) security kernel for the VAX architecture. The focus is on how the system's hardware, microcode, and software are aimed at meeting A 1 -level security requirements while maintaining the standard interfaces and applications of the VMS and ULTRIX-32 operating systems. The VAX security kernel supports multiple concurrent virtual machines on a single VAX system, providing isolation and controlled sharing of sensitive data. However, computer networking is not discussed.

[0006] Other background references include: U.S. Patent 6,385,643 issued to Jacobs et al.; U.S. Patent 5,958,010 issued to Agarwal et al., U.S. Patent 5,721,922 issued to Dingwall, and "Support For Real-Time Computing Within General Purpose Operating System " by G. Bollella et al.

SUMMARY OF THE INVENTION

[0007] It is an object of the invention to enable a system to monitor and control a networked environment.

[0008] It is anther object of the invention to enable the system to provide high- availability, rapid fault recovery, out of band condition signaling and/or other quality of service assurances and security in a networked environment.

[0009] It is another object of the invention to enable a the system to detect and prevent a network-based attack such as, for example, a denial of service attack.

[00010] These and other object are achieved by the present invention. In one aspect, a method of the present invention includes the step of providing a processing system (e.g., a general purpose computer, a specific purpose computer, a network router, a network switch, or other processing device) with at least two operating systems, which are referred to as a supervisory operating system and a secondary operating system. In one embodiment, the secondary operating system is a task supervised by the supervisory operating system. The supervisory system may be a real-time operating system, but this is not a requirement.

[00011] The method also includes the step of providing a Network Control Software (NCS) in the supervisory operating system. The NCS is an application of the supervisory operating system and is interposed between hardware network device drivers and network clients in the secondary operating system. These network clients may communicate with the NCS via protocol stacks of the secondary operating system or directly, for example, using shared memory or a pseudo-device interface. The NCS is also able to communicate with the clients in the secondary operating system by reading and modifying state information in the secondary operating system and in the client application software.

[00012] Because the NCS is interposed between hardware network device drivers and network clients in the secondary operating system, the NCS may be configured to monitor and control network operations in the secondary operating system. For example, the NCS may be configured to monitor and/or control communication channels of the secondary operating system, provide high speed fail-over, protect against network based attacks, and provide a quality-of-service system that reduces resource contention for critical services.

[00013] In one embodiment, the NCS may monitor and control a networked environment. For example, the NCS may gather information from a network client message stream and from the protocol stacks implemented in the secondary operating system. The NCS may operate across the boundaries of the protocol stacks in the secondary operating system. For example, the NCS can gather information about the timing of a protocol implemented in the secondary operating system, even if the protocol does not itself track this information. The NCS can interpose control information into a data stream and/or capture this information from a data stream, and the NCS may relate and coordinate the operation of different protocols even if those protocols are logically unrelated within the secondary operating system.

[00014] Further, in the embodiments where the supervisory operating system is a realtime operating system, the NCS can operate to impose precise timing on its actions through the real-time capabilities of the supervisory operating system. For example, the NCS may be configured to send periodic updates of state to neighboring computer systems at precise intervals. Further, the NCS can inspect and modify the state of the protocol stacks and network clients in the secondary operating system. For example, the NCS may make use of a sophisticated TCP or T/TCP stack in the secondary operating system, but intervene to prevent waste of resources if the NCS detects a condition that is not detectable by the TCP or T/TCP protocol.

[00015] Advantageously, one of the applications of the NCS is that it can transparently add functionality to enhance existing network protocol stacks and applications in the secondary operating system. For example, instead of one attempting to modify a complex and highly tuned T/TCP protocol stack to prioritize transactions with a certain remote computer, the NCS can be used to impose this prioritization on the T/TCP stack of the secondary operating system by, for example, discarding or delaying messages from lower priority computers transparently to the T/TCP stack.

[00016] The above and other features and advantages of the present invention, as well as the structure and operation of preferred embodiments of the present invention, are described below with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

[00017] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments of the present invention and, together with the description, further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention. In the drawings, like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears.

[00018] FIG. 1 is a functional block diagram of a system according to one embodiment of the invention.

[00019] FIG. 2 is a functional block diagram of an example embodiment of an NCS that can function to perform fast fail-over, monitor TCP connections, and prevent denial of service (DOS) attacks in a system according to one embodiment of the invention.

[00020] FIG. 3 is a flow chart illustrating a process 300 performed by VND when a stack generates a packet for transmission in a system according to one embodiment of the invention.

[00021] FIGS. 4A and 4B are a flow chart illustrating a process 400 performed by one embodiment of event handler 212(a) in a system according to one embodiment of the invention.

[00022] FIGS. 5A and 5B are a flow chart illustrating a process 500 performed by one embodiment of event handler 212(b) in a system according to one embodiment of the invention.

[00023] FIG. 6 is a flow chart illustrating a process 600 performed by one embodiment of thread 213(a) in a system according to one embodiment of the invention.

[00024] FIG. 7 is a flow chart illustrating a process 700 performed by one embodiment of thread 213(b) in a system according to one embodiment of the invention.

[00025] FIG. 8 is a flow chart illustrating a process 800 performed by one embodiment of thread 213(b) in a system according to one embodiment of the invention.

DESCRIPTION OF THE INVENTION

[00026] While the present invention may be embodied in many different forms, there is described herein in detail an illustrative embodiment(s) with the understanding that the present disclosure is to be considered as an example of the principles of the invention and is not intended to limit the invention to the illustrated embodiment(s).

[00027] FIG. 1 is a block diagram illustrating one embodiment of a computer system 101 according to the present invention. Computer system 101 includes a supervisory operating system 104 and a secondary operating system 106. Secondary operating system 106 provides network services to applications 140 (a.k.a., "network clients") through one or more protocol stacks 150. For example, through the network services provided by secondary operating system 106, a network client 140 can transmit data to and receive data from network clients running on other data processing systems, provided, of course, that data processing system 101 and the other data processing systems are connected, directly or indirectly, to the same network (e.g., a network 170).

[00028] In one embodiment, supervisory operating system 104 executes the secondary operating system 106, but this is not a requirement. Additionally, interrupt control operations in secondary operating system 106 may be replaced with software emulation and supervisory operating system 104 may safely preempt the secondary operating system after very limited delays. It is not required that secondary operating system 106 be a traditional operating system, it may be a Java Virtual Machine, for example. A dual-kernel software operating system that can be used with the present invention is described in U.S. Patent No. 5,995,745 (Yodaiken), which is fully incorporated herein by reference. One skilled in the art will appreciate that other multi-kernel operating systems could be used and the invention is not limited to any particular one.

[00029] As shown in FIG. 1, there is also provided a Network Control System (NCS) 110 in the supervisory operating system 104. NCS 110 is used to monitor and control network operations in secondary operating system 106 transparently from the secondary operating systemDs perspective. NCS 1 10 may further monitor and control the network environment. NCS 110 is an application of supervisory operating system 104 and may either be executed within the address space of supervisory operating system 104 or in a protected memory space. Network clients 140 that run on top of secondary operating system 106 may communicate with NCS 110 via one or more protocol stacks 150 of secondary operating system 106 or directly using, for example, shared memory or a pseudo-device interface (not shown). NCS 110 is also able to communicate with the one or more clients 140 by reading and modifying state information in secondary operating system 106 and in client 140 application software. NCS 110 may execute on a periodic schedule or by timeouts, or by the action of triggers from the lower level drivers.

[00030] In one embodiment, secondary operating system 106 is provided with one or more virtual network drivers (VNDs) 120 that emulate a network device driver. That is, a VND 120 appears to secondary operating system 106 and protocol stacks 150 to be a network device driver, such as device driver 133. Virtual network drivers 120 "transmit" and "receive" packets under control of NCS 110. Virtual network drivers 120 present an interface corresponding to a hardware device, for example, an Ethernet driver, or can present a higher level interface. For example, the Message Passing Interface (MPI) may be implemented as a virtual network driver 120 on a supercomputing cluster.

[00031] NCS 110 may operate across the boundaries of protocol stacks 150. For example, NCS 110 can gather information about the timing of a protocol implemented in secondary operating system 106, even if the protocol does not itself track this information, and NCS 110 may relate and coordinate the operation of different protocols even if those protocols are logically unrelated within secondary operating system 106. Further, NCS 1 10 can interpose control information into a message stream flowing through protocol stacks 150 and/or capture this information from the message stream.

[00032] Through the real-time capabilities of supervisory operating system 104, NCS 1 10 is operable to impose precise timing on its actions. For example, NCS 110 can send periodic updates of state (e.g., "keep alive messages") to neighboring computer systems at precise intervals. Further, NCS 110 can inspect and modify the state of the protocol stacks and application programs in the secondary operating system. For example, NCS 110 may make use of a sophisticated T/TCP stack in the secondary operating system, but intervene to prevent waste of resources if NCS 110 detects a condition that is not detectable by the T/TCP protocol.

[00033] In one embodiment, NCS 110 may include event handlers 112, threads 113, and a control database 180. An event handler 1 12 is a set of instructions for performing one or more functions. The event handler 1 12 is invoked upon the occurrence of a pre-defined event. For example, one event handler 1 12 may be invoked in response to device driver 133 receiving a data-packet (e.g., an Ethernet frame) from network device 130, while another event handler 112 is invoked in response to virtual device driver 120 receiving a data-packet from a higher layer protocol. Control database 180 permits NCS 110 to define arbitrary "logical connections". Examples of logical connections are TCP/IP connections, types of TCP/IP services, all communications with packets labeled by a particular hardware address or IP number, or a communication link specific to the real-time driver such as a request/response link with another site or group of sites. Control database 180 may be hard wired into the design of NCS 110, it may be a static data structure or it may be updated dynamically. Control database 180 may be supplemented or created entirely by a program executing under the control of the secondary operating system 106. For example, a SS7 system may include an information system running in the secondary operating system that keeps track of which calls are high priority. The SS7 system may update NCS 110 control database 180 to register calls that get priority for bandwidth being managed by NCS 110.

[00034] One application of NCS 110 is that it can transparently add functionality to existing network protocol stacks and applications. For example, instead of one attempting to modify a complex and highly tuned T/TCP protocol stack (or any other protocol stack) to prioritize transactions with a certain remote computer, NCS 110 can be used to impose this prioritization on the T/TCP stack of the secondary operating system by, for example, discarding or delaying messages from lower priority computers transparently to the T/TCP stack when necessary.

[00035] Another example of functionality that may be provided by NCS 110 includes providing fast fail over in a computing cluster. A computing cluster typically consists of a number of computers connected on a switched network, such as a switched Ethernet, Myranet, or custom "fabric." Common applications of a computing cluster include supercomputing applications and electronic commerce.

[00036] These clusters need to be able to react quickly to failures by shifting tasks to alternate computers not affected by the failure. NCS 110 can enable such quick reactions by detecting failures immediately or soon after they occur and then taking the appropriate corrective action or setting an alarm to indicate that a failure has occurred so that another process or an administrator can take the appropriate actions. In one example, control database 180 lists the address information of some number of other computers in the cluster that would form a "fail-over group." NCS 1 10 might begin by calibrating message delays on the network to these computers and then set up a schedule for regular exchange of packets between members of the fail-over group. As the schedule dictated, NCS 110 would send packets to other computers in the group to indicate that the computer system and secondary operating system with which the NCS is associated are live and making progress. Additionally, NCS 1 10 might monitor secondary operating system 106 by making sure that messages were moving through control stacks within secondary operating system 106, that key processes were being scheduled at appropriate rates, and that no software or hardware panics had been detected. NCS 110 may also operate alarms for other members of the fail-over group, resetting alarms when packets were received from the corresponding member of the fail-over group and taking some specified action when alarms expired.

[00037] In another application of present invention, computer system 101 implements a telephone switching system, such as an SS7 telephone switch. In this embodiment, NCS 110 is configured to detect the receipt of a control signal at the telephone switch and process the control signal as soon as it is received or pass the control signal to the appropriate module for processing. Thus, the present invention ensures that control signals are acted upon in a timely manner while less critical messages can be passed up to a protocol stack of secondary operating system 106. As a concrete example, a control message indicating that a line is not accessible can trigger an immediate action by NCS 110 to redirect data messages via a secondary line, transparently to the protocol stacks in secondary operating system 106.

[00038] The present invention can also be used to prevent theft or denial of service (DOS) attacks. Additionally, the present invention can be used to provide a quality-of- service system that can schedule services and reduce resource contention for critical services. One skilled in the art will appreciate that these uses of the present invention are exemplary only, and that there are other uses of the present invention.

[00039] With reference to FIG. 2, a functional block diagram of an example embodiment of an NCS 210 that can function to perform fast fail-over, monitor TCP connections, and prevent denial of service (DOS) attacks is shown. NCS 210 includes two event handlers (event handler 212(a) and event handler 212(b)) and two threads (thread 213(a) and thread 213(b)). Event handler 212(a) is invoked after network device 130 receives a data-packet from network 170. When network device 130 receives a data-packet from network 170, the received data-packet is passed to network driver 133, which places the received packet in queue 237 (a.k.a., hd_rx_queue 237) and invokes event handler 212(a). Event handler 212(b) is invoked when network device 130 transmits a data-packet (e.g., an Ethernet frame) onto network 170. Thread 213(a) performs fail-over monitoring and thread 213(b) performs TCP monitoring. [00040] FIG. 3 is a flow chart illustrating a process 300 performed, at least in part, by VND 120 when a stack 150 generates a packet for transmission. Process 300 begins in step 301, where the generated packet is placed in queue 225 (a.k.a., vnd_tx_queue 225). In step 302, VND 120 increments a variable called tx_packet_count. In step 304, VND 120 determines whether the packet is a critical packet. That is, VND 120 determines whether the packet is a TCP packet that originated from a TCP port that has been labeled as being "critical." In one embodiment, a list of the critical TCP ports is maintained in control database 180. If the packet is a critical packet, then control passes to step 306, otherwise the process proceeds to step 312.

[00041] In step 306, VND 120 records the current time so that NCS 110 can keep track of how long the TCP packet is queued before it is finally transmitted. In step 308, VND 120 determines whether there is room on queue 235 (a.k.a., hd_tx_queue 235). If there is, VND 120 removes one or more packets from vnd_tx_queue 225 and places those one or more packets onto hd_tx_queue 235 (step 310), otherwise VND 120 sleeps for a configurable amount of time (step 314). After step 314, the process goes back to step 308.

[00042] In step 312, VND 120 determines whether there is room on hd_tx_queue 235. If there is, then the process proceeds to step 310, otherwise the packet is removed from vnd_tx_queue 225 and discarded (step 313).

[00043] FIG. 4 is a flow chart illustrating a process 400 performed by one embodiment of event handler 212(a). As described above, event handler 212(a) is invoked after network device 130 receives a data-packet from network 170. Process 400 begins in step 402, where the event handler determines each site that is in the fail-over group. This information may be stored in control database 180.

[00044] Each site in the fail-over group has an associated site record that includes three fields. The first field is referred to as the "address" field and it stores an address of the site (the address can be a hardware address or network address). The second field is referred to as the "last_tx_timeQ field and it stores the time of day when system 101 last transmitted a data-packet to the site. The third field is referred to as the "last_rx_time" field and it stores the time of day when system 101 last received a data-packet from the site. [00045] In step 404, the event handler determines whether the received packet was transmitted from a site in the fail-over group. The event handler may determine this by comparing the source address information contained in the data-packet to the address field of each site record. If there is a match, then the received packet was transmitted from a site in the fail-over group. If the data-packet was transmitted from a site in the fail-over group, then the process proceeds to step 406, otherwise the process proceeds to step 412.

[00046] In step 406, the event handler determines the current time of day and stores this time in the last rx time field of the site record associated with the site that was the source of the data-packet. In step 408, the event handler determines whether the data-packet is a "reminder-packet." If it is, the process proceeds to step 410, otherwise the process proceeds to step 412. In step 410, the event handler transmits an DlDm aliveD message to the site that was the source of the data-packet.

[00047] In step 412, the event handler examines the packet to determine whether it encapsulates a TCP packet. If it does not, then the packet is passed to VND 120 and the process ends, otherwise the process proceeds to step 414, where the event handler determines whether the encapsulated TCP packet is a SYN packet, which is a packet that is used to initiate a TCP connection. If it is a SYN packet, then the process proceeds to step 416, otherwise the process proceeds to step 426.

[00048] In step 416, the event handler determines whether either a TCP_DOS_WARNING flag is set to TRUE or a TCP_CONGESTION_WARNING flag is set to TRUE. If either flag is set to TRUE, the received SYN packet is discarded (step

418), otherwise, the process proceeds to step 420. In step 420, the event handler passes the received SYN packet to VND 120, which places the packet into queue 227 (a.k.a., vnd_rx_queue 227), and increments open_syn_count by one (i.e., open_syn_count = open_syn_count + 1). In step 422, the event handler determines whether open_syn_count is greater than a predetermined threshold. If it is, then the TCP_DOS_WARNING flag is set to TRUE (step 424), otherwise the process ends.

[00049] In step 426, the event handler determines whether the encapsulated TCP packet is an ACK packet. If it is, then open_syn_count is reduced by one and the received packet is passed to VND 120 (step 428), otherwise the process proceeds to step 430.

[00050] In step 430, the event handler determinations whether the TCP packet is addressed to a TCP port that has been labeled "critical." That is, the thread determines the TCP destination port number of the packet and then may check a list of critical ports to see if the port number is on the list. In one embodiment, a list of the critical TCP ports in maintained in control database 180. IF the TCP packet is addressed to a critical TCP port, then the thread passes the packet to VND 120 and updates port.last_rx_sequence (step 432), otherwise the process proceeds to step 434.

[00051] In step 434, the thread determines whether the TCP_CONGESTION_WARNING flag is set to TRUE. If it is, then the packet is discarded (step 436), otherwise the packet is passed to VND 120 (step 438).

[00052] FIG. 5 is a flow chart illustrating a process 500 performed by one embodiment of event handler 212(b). As described above, event handler 212(b) is executed when a packet is transmitted by network device 130 onto network 170. Process 500 begins in step 502, where the event handler determines how much time the packet spent in the queues 225 and 235 before it was finally transmitted. If the enqueue time is more than a predetermined threshold (a.k.a., allowable TX_ENQUEUE time), then event handler sets a the

TX_ENQUEUE_TOO_SLOW flag to TRUE (step 504), otherwise the process proceeds to step 506. In step 506, the event handler determines whether the packet is addressed to a site in the fail-over group. If it is, then the event handler sets the last_tx_time for the site to the current time (step 508), otherwise the process proceeds to step 510. Also, after step 508, the process proceeds to step 510.

[00053] In step 510, the event handler determines whether the packet originated from a TCP port that has been labeled as critical. If so, then the event handler updates last_tx_sequence for that port (step 511).

[00054] In step 512, the event handler enqueues any packets generated by an NCS thread (e.g., thread 213(a) or 213(b)) that are not on hd_tx_queue 235. In step 514, the event handler determines whether the TX_ENQUEUE_TOO_SLOW flag is set to FALSE. If it is, the process continues to step 516, otherwise the process ends. [00055] In step 516, the event handler determines whether the vnd_tx_queue 225 is empty. If it is, the process ends, otherwise it continues to step 518. In step 518, the event handler selects the packet on vnd_tx_queue that has been on the queue the longest. In step 520, the event handler determines (a) whether the selected packet is a TCP packet that is addressed to a critical TCP port or (b) whether the TCP_CONGESTION_WARNING flag is set to FALSE. If either (a) or (b) is true, then the event handler enqueues the selected packet onto the hd_tx_queue 235 (step 522), otherwise the packet is discarded (step 524). After steps 522 and 524, the process goes back to step 516.

[00056] FIG. 6 is a flow chart illustrating a process 600 performed by one embodiment of thread 213(a). Process 600 begins in step 602, where the thread determines each site that is in the fail-over group. This information may be stored in control database 180.

[00057] Next, in step 604 the thread determines the sites in the fail-over group who have not received a data-packet from system 101 within a pre-determined period of time. The thread can determine this in a number of ways. For example, it can compare the current time to the time of last transmission, which was stored in the variable last_tx_time by event handler 212(b). In step 606, the thread transmits an "I'm alive" packet to each site determined in step 604. So, for example, if the thread determines that no data-packets have been sent to a particular site within the fail-over group within the last 30 seconds, then the thread will send the "I'm alive" data-packet to the site. In this way, the site will know that system 101 is operational because the thread guarantees that the site will, at the least, receive an "I'm alive" data-packet every 30 seconds from system 101.

[00058] In step 608, the thread determines those sites from whom system 101 has not received a data-packet within a pre-determined period of time and adds them to a "watch- list." The thread can determine the sites from whom system 101 has not received a data- packet within a pre-determined period of time in a number of ways. For example, it can compare the current time to the time when the system last received a data-packet from the site; this time was stored in the variable last_rx_time by event handler 212(a). In step 610 the thread transmits a "reminder" packet to each site on the watch-list. So, for example, if system 101 has not received a data-packet from a particular site within the last 30 seconds, then the thread will put the site on the watch-list and send the "reminder" packet to the site. [00059] In step 612, the thread determines the site or sites on the watch list that have been on the watch list for more than a pre-determined amount of time. In step 614, the thread removes the sites determined in step 612 from the watch list and from the fail-over group and notifies a failure-handler that the sites appear to have failed. After step 614, control passes back to step 602.

[00060] FIG. 7 is a flow chart illustrating a process 700 performed by one embodiment of thread 213(b). Process 700 is performed indefinitely, so long as a KEEP_TCP_CONTROL flag is set to TRUE. Process 700 begins in step 701, where the KEEP_TCP_CONTROL flag is checked to see if it is set to TRUE. If it is not set to TRUE, the process ends, otherwise the process proceeds to step 702. In step 702, the thread initializes a variable called "wait_count" to zero. In step 704, the thread scans the TCP control blocks in secondary OS 106. In step 706, thread performs process 800 (see FIG. 8) for each control block. After performing process 800 for each control block, the thread proceeds to step 708.

[00061] Referring now to FIG. 8, process 800 begins in step 802, where the thread determines if the state of the TCP port is set to SYN_RECEIVED, which means that a SYN packet has been received by the port but not yet acknowledged. The state of the TCP port is set to SYN_RECEIVED, then the thread increments wait_count (step 804), otherwise the process proceeds to step 806. In step 806, the thread determines whether the TCP port is a critical TCP port. If it is, then the thread proceeds to step 808, otherwise process 800 ends.

[00062] In step 808, the thread examines the TCP control block to determines the sequence number of the last TCP packet transmitted by the TCP port associated with the TCP control block. This sequence number is referred to as send_max In step 810, the thread compares send_max to port.last_tx_sequence, which is a variable that stores the sequence number of the last TCP packet that was transmitted onto the network 170 and associated with the port. This information can be maintained by event handler 212(b). If send_max is greater than port.last_tx_sequence by more than a threshold value, then the TCP_CONGESTED_WARNING flag is set to TRUE (step 812). The difference between send_max and port.last_tx_sequence provides information about the number of TCP packets that are queued to be transmitted onto network 170. If too many are queued, then the TCP_CONGESTED_WARNING flag should be activated.

[00063] In step 814, the thread examines the TCP control block to determine the next sequence number that the TCP port expects to receive. This sequence number is referred to as rcv_next. In step 816, the thread compares rcv_next to port.last_rx_sequence, which is a variable that stores the sequence number of the last TCP packet that was received from network 170 and is associated with the port. This information can be maintained by event handler 212(a). If rcv_next is less than port.last_rx_sequence by more than a threshold value, then the TCP_CONGESTED_WARNING flag is set to TRUE (step 818), otherwise the process proceeds to step 820. The difference between rcv_next and port.last_rx_sequence provides information about the number of TCP packets that are in queues 227 and 237. If there are too many packets in the queues, then the TCP_CONGESTED_WARNING flag should be activated.

[00064] In step 820, the thread determines whether rcv_next is greater than port.last_rx_sequence by less than a given threshold. If rcv_next is greater than port.last_rx_sequence by less than the given threshold, then the TCP_CONGESTED_WARNING flag is set to TRUE. This step is needed because the sequence numbers wrap.

[00065] Referring back to process 700, in step 708, the thread sets open_syn_count to equal wait_count. Next, the thread determines whether open_syn_count is less than a first threshold value (step 709). If it is, then the TCP_DOS_WARNING flag is set to FALSE (step 710). In step 712, the thread determines whether open_syn_count is greater than a second threshold value, where the second threshold value is greater than the first threshold. If it is, then the TCP_DOS_WARNING flag is set to TRUE (step 714). In step 716, the thread sleeps for a configurable amount of time. After step 716, the process proceeds back to step 701. [00066] While various embodiments/variations of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.