System and methods for generating and distributing alarm and event notifications

The present invention relates to the following in its own following, which are registered with the present specification simultaneously: (1) U.S. Patent Application Serial No. 08 / 916,870 and (2) U.S. Patent Application Serial No. 08 / 920,265. The disclosures of these related patent applications are hereby incorporated by reference for all purposes.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to process control systems, and more particularly to a process control system having an alarm and event notification system.

GENERAL PRIOR ART

Many process equipment (eg, a manufacturing plant, a mineral or crude oil refinery, etc.) are managed with distributed control systems. Typical current control systems include numerous modules adapted to monitor and / or control various processes of the plant. Conventional means connect these modules together to create the distributed nature of the control system. This provides greater performance and the ability to extend or reduce the control system to meet changing plant needs.

Process equipment management providers, such as Honeywell, Inc. are developing control systems that can be adapted to meet a variety of process requirements (e.g., global, local or otherwise) and device types (e.g., manufacturing, warehousing, refinery, etc.). Such providers have two main goals. The first goal is to centralize the control of as many processes as possible to improve the overall efficiency of the facility. The second goal is to support a common interface that communicates data between different process controlling and monitoring modules and also exchanges it with this centralized control or operator center.

Each process or group of associated processes has one or more input characteristics (eg, flow rate, supply, power, etc.) and one or more output characteristics (eg, temperature, pressure, etc.) associated with it , Model Predictive Control ("MPC") techniques have been used to optimize certain processes as a function of such characteristics. An MPC technique uses algorithmic representations of certain processes to estimate characteristic values ​​(represented as parameters, variables, etc.) associated with them that can be used to better control such processes. In recent years, physical, economic and other factors have been integrated into control systems for these associated processes.

Examples of such techniques are described in U.S. Patent No. 5,351,184 entitled "Method of Multivariable Predictive Control Utilizing Range Control", U.S. Patent No. 5,561,599 entitled "Method of Incorporating Independent Feedforward Control in a Multivariable Predictive Controller". US Pat. No. 5,572,420 entitled "Method of Optimal Controller Design of Multivariable Predictive Control Utilizing Range Control" and US Pat. No. 5,574,638, entitled "Method of Optimal Scaling of Variables in a Multivariable Predictive Controller Utilizing Range Control", all of which are owned by the assignee of the present invention, and which are hereby expressly incorporated by reference for all purposes (the above issued patents and US Patent Application Serial No. 08 / 490,499, previously incorporated by reference, are collectively referred to herein as the "Patents and Application of Honeywell").

The distributed control systems used to monitor and control a process are often linked by common communication paths, such as a local area network (LAN) architecture or a wide area network (WAN) architecture. When a requesting node needs a data item from a responding node, it issues a request for the data item over the network, and the responding node returns the data item over the network. Many process control systems use a supervisory control LAN or WAN integrated with one or more process control networks. The process control networks contain the basic unprocessed data needed by the supervisory control network and other process control networks.

An important function in distributed control systems is the generation and distribution of notifications known as events. A notification is an indication of a particular abnormal or exceptional situation regarding a controlled process or its measuring and control devices. A process controller generates notifications that are distributed to a notification client that is an endpoint application that needs the notifications. Notifications may include, for example, alarms, system events, operator messages, and the like associated with a user-visible process, devices, and hardware interruptions.

For example, a first process controller that needs process data is a notification client with respect to a second process controller containing that process data. In the event of an anomaly, such as a communication loss by the second process controller, the second process controller may be required to generate notifications when the anomaly is removed. Typically, the first process controller recognizes that the second process control has cleared the error and requests a notification remedy from the second process controller. The second process controller then re-generates all notifications that may have occurred during the communication failure and sends them to the first process controller. However, this type of notification distribution system has disadvantages. The system is available from the notification client (ie the first process control) requesting the notification removal. This may take some time after the anomaly ends and the second process control is restored. Second, the process controller that generates the notifications may have many notification clients. When each notification client separately requests and receives notification notification from the notification generation process controller, a great deal of network traffic is generated, thereby reducing overall system capacity.

Therefore, what is needed in the art is improved process control systems that can generate and distribute notifications immediately upon the re-establishment of process control without requiring a notification recovery request from a notification client. Also needed are improved process control systems that can quickly dispatch notifications from one network node to multiple notification clients.

BRIEF SUMMARY OF THE INVENTION

The present invention provides a control system according to the following claim 1. The system may include any or several of the features of dependent claims 2 to 9.

The present invention also provides a method according to the following claim 10.

The method may include any one or more of the features of dependent claims 10-16.

An advantage of the present invention is the provision of a high performance notification distribution and remediation scheme that is reliable, deterministic, and flexible. As already established, a typical process facility may include many associated processes, of which various different levels of the overall process are associated (eg, natural resource refining, filtration, gas / oil separation, fabrication, and other similar processes). The present invention introduces systems and methods that optimize the distribution of notification data and the synchronization of notification clients and notification generators using notification remediation techniques that are seamlessly handled by the communications application layer.

In achieving this major advantage, the present invention provides systems and methods for controlling associated processes in process facilities, and more particularly for efficiently distributing notification data between nodes of a real-time process control system that controls a given facility. An exemplary process control system includes sensors, controllable devices, communication paths, a computer system, and notification controls. The sensors and controllable devices are associated with various of the processes of the device, and the communication paths connect the sensors and controllable devices to the computer system. The computer system processes data relating to the process device and distributes the notification data among selected nodes thereof. The nodes are connected by the communication paths, and the computer system further includes notification controls. The notification controllers are associated with the nodes and act to detect the recovery of new or lost communication paths from the first distributed nodes to second distributed nodes, and in response, to transmit notification data from the second distributed nodes to the first distributed nodes.

According to an advantageous embodiment, such notification data includes alarm or event data and the distribution relationship between the second to the first node may suitably exist as any of the relationships 1: n, n: 1 or n: m. These relationships, in abstract terms, represent logical communication links between application and transport layer services provided by the systems and methods of the present invention. More specifically, the notification removal is a function based on hardware, software, firmware, or others, whereby notifications by a notification generator (the second node of the above-introduced exemplary system) for a notification consumer (the first nodes thereof) in response to a communication . Device or other disturbance / - anomaly and a subsequent remedy (such as the restoration of the lost of the communication paths from the first to the second distributed nodes therein) are regenerated.

The principles of the present invention provide, in particular, by notification controls, a suitable means for efficiently utilizing the natural physical constraints of the various components of the process control system and the process control system as a whole, in particular the traffic capacities of the communication paths. Automatically transmitting notification data from a first node (a node controlling a process) to a client node (a server or other consumer node that consumes notification data) in response to detection of a lost communication path recovery from the client to the process node according to the present invention suitably eliminates requirements "Queries" and the like from client-to-process nodes of such notification data, thereby reducing the utilization of traffic capacities of the communication path among the nodes of the control system.

The features and technical advantages of the present invention have been outlined relatively broadly above so that those skilled in the art can better understand the following detailed description of the invention. Additional features and advantages will be described below and form the subject of the claims of the invention. It should be apparent to those skilled in the art that they may readily use the disclosed conception and specific embodiment as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. In addition, it should be apparent to those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the invention in its most general form.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention and its advantages, reference will now be made to the following descriptions taken in conjunction with the accompanying drawings, in which like numerals denote like objects. Show it:

Fig. 1 is a simple block diagram of a process device in which a control system according to the principles of the present invention may be implemented;

FIG. 2 is a block diagram of the notification distribution relationship between a process control module and a supervisory controller according to an embodiment of the present invention; FIG. and

3A and 3B are flowcharts illustrating the general operation of a notification manager according to an embodiment of the present invention.

DETAILED DESCRIPTION

1-3 discussed below and the various embodiments used to describe the principles of the present invention in this specification are merely illustrative and should not be taken in any way to limit the scope of the invention. It will be appreciated by those skilled in the art that the principles of the present invention may be implemented in any suitably arranged process device.

Fig. 1 shows a block diagram of a process device 100 in which a control system according to the principles of the present invention may be implemented. The example process device 100 processes raw materials and includes a control center 105 and six associated processes (elements 110a-110f) arranged in three stages. The term "contain" here means inclusion without limitation. The example control center 105 may include a central area that is usually occupied by an operator (not shown) to monitor and control the three exemplary process stages. A first process stage includes three raw material mills 110a-110c, which receive a "feed" of raw material and grind it, for example, by using a pulverizer or a millstone into smaller particles of raw material. The second process stage includes a washer 110d that receives and cleans the ground raw materials to remove first stage residues. The third process stage includes two separators 110e and 110f which receive the ground and washed raw material and separate it into desired minerals and any remaining raw materials. Since this process device is given by way of illustration only, and the principles of such device are well known, another discussion is beyond the scope of the present specification and is unnecessary.

The exemplary control system includes a supervisory controller 120 and six process nodes or process controllers 125a-125f, each implemented in software and executable by a suitable conventional data processing system (standalone or networked), such as any of the AM K2LCN, AM K4LCN systems , AM HMPU, AxM Honeywell Inc. or similar systems. It will be appreciated by those skilled in the art that such controls may be implemented in hardware, software, or firmware, or any suitable combination thereof. In general, the use of data processing systems in control systems for process equipment is well known.

Each of the process controllers 125 is directly or indirectly associated with a supervisory controller 120 to facilitate the exchange of information. The term "assigned to" and its derivatives may in this case include inclusion in, compound with, abstention, be contained in, connection to or with, coupling to or with, communicable with, interaction with, interleaving, property of, restriction on or with, have, have property or mean the like. The monitoring controller 120 monitors characteristics (eg, status, temperature, pressure, flow rate, current, voltage, energy, utilization, efficiency, cost, and other economic factors, etc.) of the associated processes 110 either directly or indirectly through the process controllers 125 associated with the process 110. Depending on the specific implementation, such monitoring may be performed for a single process, group of processes, or the entire device.

The monitoring controller 120 communicates with the associated processes 100 via the process controllers 125 and generates monitoring data to optimize the process device 100. The term "monitoring data" is defined herein as any numerical, qualitative or other value generated by the monitoring controller 120, for example, a particular process, a group of processes, the entire device, a process step, a group of stages, to control a sequence of processes or stages or the like (e.g. to direct, manage, modify, recommend, regulate, suggest, monitor, collaborate, etc.) to optimize the facility as a whole. In a preferred embodiment, the monitoring data is generated dynamically and is based at least on the efficiency of a given facility, production or economic costs, and most preferably all threes.

The process controllers 125 monitor associated processes 110 and operate to vary according to the monitoring data, to control the associated processes, and in particular to modify one or more processes and improve the monitored characteristics and the device as a whole. The relationship between the supervisory controller 120 and various of the process controllers 125 may be of the master-slave type (full obeying), cooperation (varying obedience, such as by using the monitoring data as a factor in controlling the associated processes), or completely ignoring (non-obeying) , Depending on the specific implementation and needs of a given device, the relationship between the supervisory controller 120 and a specific process controller 125 may be static (ie, only ever either obeying, cooperating, or disobeying), dynamically (ie time-varying, such as in a range between obey and non-obey or a smaller range in between) or can switch between static and dynamic periods.

Fig. 1 shows the process controllers 125a-f as merely logical blocks coupled to the processes 110a-f for illustration purposes only. In reality, the process controllers 125a-f may be implemented in the process device 100 as well as a variety of devices. In the simplest embodiments, an example process controller 125 may be a microcontroller circuit fabricated on a circuit board and stored in one of the processes 110 (ie as part of a separator, washer or mill) which is controlled. In other embodiments, an example process controller 125 may be a stand-alone computer, such as a personal computer (PC), that is remote from the controlled process 110 and coupled thereto by a bus architecture.

In more complex embodiments, an example process controller 125 may be a network node that is coupled to one or more processes 110 through a network architecture. The monitoring controller 120 may then treat the network containing the example process controller 125 and its associated processes 110 as a single functional group. Finally, an example process controller 125 may be a group of process controllers and their associated processes 110 that are networked together. The networked group may then be treated by the monitoring controller 120 as a single functional group.

The process controllers 125a-f generate process data used by the monitoring controller 120 for a variety of purposes, including generating the monitoring data and distributing the process data to one or more client applications. Process data may also be used by the process controller 125 that created it to control the associated process 110. For example, a process controller 125 may read physical parameter data from a process 110, such as temperature, pressure, flow rate, and the like, and use some or all of that process data and possibly some monitoring data to control the process 110. This is especially true for a closed-loop process.

Process data can be transferred directly between process controllers 125a-f in a peer-to-peer relationship, such as in a LAN network. For example, the process controller 4 that controls the washer (element 110d) may request process data from the process controllers 1-3 controlling the mills 1-3 to determine the rate at which the mills 1-3 are dispensing raw material. The washer can thereby adjust the speed at which it washes the milled material. For example, the washer may reduce its power consumption in washing the milled raw material if the amount of milled raw material sent to the washer is relatively low. It may even temporarily shut down to "wait" for a suitable amount of ground raw material to accumulate before resuming washing.

In certain embodiments of the present invention, the monitoring controller 120 may include a LAN, a group of connected LANs, or a WAN architecture. On nodes of the LAN / WAN architecture, one or more client applications are executed. The nodes may be, for example, personal computers. The client applications may all require that the same process data and monitoring be transferred from the process controllers at the same update rate. However, a more likely scenario is that the client applications require different and possibly overlapping subsets of the process data and monitoring data and require that the process data and monitoring data be transferred at different update rates to different client applications.

2 is a block diagram of the notification distribution relationship between a process control module 201 and the monitoring controller 120 according to an embodiment of the present invention. The process control module 201 illustrates the processing and network interface circuitry of an exemplary one of the process controllers 125 in FIG. 2 The solid arrows indicate a physical data path and a notification direction, and the dotted arrows indicate a logical data path and a logical notification direction.

The notification remedy is initiated in the notification manager 256 upon the occurrence of any of the following system operations: server startup, server failover, control startup, control failover, control network communication failure, and the removal and addition (via configuration) of a new process control. In the exemplary embodiment, the monitoring controller 120 is the server with respect to the PCM 201, and both are coupled by a local area network architecture. Notification removal is needed in situations where notification clients and notification generators become unsynchronized, usually due to a particular system or device failure (eg, controller, network, workstation, etc.) and its repair.

In a preferred embodiment of the present invention, the notification removal is performed entirely by the communication application layer serving the client application (s) using the notifications. It is the application layer in the notification client node (the "Notify Attendee") that commands a notification fix. The application layer performs the notification removal on behalf of a notification client when needed by the notification client, so that neither the notification client application nor the function layer of the notification generator is burdened by this function.

Notification generators exist in the PCM 201 as user-configured function blocks managed by a control execution environment (CEE) 202. The CEE 202 may (optionally) receive a notification removal command from the notification manager 256 in the monitoring controller 120 and, in turn, commands each functional block to generate all notifications. The notification manager 256 is responsible for initiating and maintaining connections to all the notification generator nodes. The notification manager 256 is an application layer object that manages all notifications and interfaces with the server event subsystems 252.

The notification detector 203 exists in the user layer of the PCM 201 and detects a notification state or the elimination of a notification state. In response, the notification detector 203 sends a notification regarding the existence or elimination of the condition to a notification generator 204. The notification generator 204 is an object of the user layer responsible for generating a notification packet. The notification generator 204 maintains an association with the notification distribution publisher 205 to facilitate the transport of the notifications to the notification client. Each notification packet is a unique (injective) expression of the notification that caused it.

The notification packet is sent to the notification distribution publisher 205, which is the application layer service responsible for accepting notification packets from the notification generator 204 and transporting them as bundled notification packets to notification distribution agents, such as the notification distribution agent 257 in the supervisory control 120, is responsible. The notification distribution is an application layer service through which notification messages (described below) are transported from notification publishers to notification parties. The notification distribution layer provides the robustness necessary to ensure that notification packets are not lost and provides any necessary notification throttling.

A notification packet comprises one or more notification packets grouped by a notification distribution publisher into an application layer communication packet that can be transmitted to a notification distribution agent. In the example shown in FIG. 2, the notification distribution agent 257 is the application layer endpoint for the notification distribution publisher 205. The notification distribution agent 257 establishes a connection for each notification distribution publisher 205.

The notification packets are translated by a transport layer service 206 into notification messages suitable for transmission by the integrated control protocol and the local network 207. The notification message may be divided into multiple "notification frames" (e.g., MAC packets) on the local area network 207. In the monitoring controller 120, the local network 207 sends the notification frames to the transport layer service 258, which reconverts the frames back into a notification message that is sent to the notification distribution agent 257. The notification messages are translated by the notification distribution agent 257 into notification packages for the notification manager 256.

The notification manager 256 is part of a control data access server (CDA server) 255 and is responsible for sending notifications to the notification clients, which are the endpoint applications that will eventually use (consume) the notifications. In the monitoring controller 120, a server event subsystem 252 including an event journal (or notification journal) 254 and an alarm acknowledgment status register 253 may be provided. includes the alert direction from which notification manager 256 may be used to separately store notifications (events) or alerts.

The notification removal is required by each notification client to synchronize the notification (event) database with that of the online system management. The client builds its alarm and event records based on what it recovers from the PCM 201. In a normal system without failures (steady state), there is no notification fix. When the steady state has been interrupted (mainly a node or network selection), the notification recovery is for synchronizing a notification client with all notification generation nodes. A notification removal from a notification distribution publisher 205 is concurrently processed by all the notification distribution participants 257.

As mentioned earlier, the notification remediation may be initiated by certain scenarios that are recognized by the notification manager 256: server startup, server failover, process control startup, process control failover, control network communication failure and recovery, addition (and configuration) of a new process control, or Startup of a notification client node. The notification removal is initiated by the PCM 201 in response to a command from the notification manager 256.

According to an embodiment of the present invention, the notification distribution relationship may be a 1: n relationship between a notification consumer and a plurality of notification generators. In other embodiments of the present invention, the notification distribution relationship may be an n: 1 relationship between a plurality of notification consumers and a notification generator. In further embodiments of the present invention, the notification distribution relationship may be an n: m relationship between a plurality of notification consumers and multiple notification generators.

The notification removal is triggered by each successful establishment of a transport layer connection in the network serving each notification generation process control. This covers all cases where a notification is needed to handle a device or network failure and correct it. When the network manager 256 detects the establishment or recovery (as the case may be) of a network connection, the notification manager orders a notification remedy from the process controller with which the connection was made or restored.

In a server startup scenario, the notification client portion of the server subscribes to the notification manager 256. The notification manager (NM) 256 turns on when the node is powered up, but remains idle until the client portion subscribes. The NM 256 then polls a system database to determine all currently configured notification generation nodes (process controllers 125a-f). Next, the NM 256 forms a notification transport connection with each notification distribution publisher in each process controller 125a-f and, if successful, commands notification notification from each one. In this case, the notification removal is a lot of regenerated notifications that is between a start notification and an end notification, whereby the final notification client can learn about the notification removal. Advantageously, the final notification client need not request a notification fix.

A server failover is similar to a server startup in that a primary server fails over to a synchronized secondary server that becomes the new primary server. The notification client portion of the new primary server then subscribes to the notification manager 256. This effects the same operation sequence as described above, as a result of which a notification remedy is commanded by all process controllers 125a-f.

In a control startup scenario, a new process controller 125 is powered up and configured on the network, but has not yet established a transport layer connection with the notification manager 256. The NM 256 maintains a list of all messaging distribution publishers based on the system configuration of the network. The NM 256 therefore periodically attempts to form a notification transport layer connection with any notification distribution publisher that is configured but not yet connected. After the transport layer connection is established with the new process controller 125, the NM 256 commands a notification remedy from it. This operation only affects other process controllers if they also startup. The notification client may distinguish between notification removal from a particular process controller 125, as opposed to all process controllers 125a-f, as the boundaries of the notification remedies delimited by a start notification and an end notification are sent process-by-process.

A control failover scenario is very similar to a control startup scenario. When a primary process controller 125 fails over to a secondary process controller 125, the secondary process controller 125 becomes the new primary controller. However, the notification connection with the notification manager 256 is lost. When the NM 256 performs a routine scan of the network based on the system configuration, the NM 256 detects the presence of the new primary process controller 125 and determines that it has the same address as the old primary process controller 125. Since the connection is lost and then cleared by the NM 256, the process controller 125 is commanded by the NM 256 to perform a notification recovery.

A controller failure and recovery scenario is very similar to a controller startup scenario. A process controller 125 fails and is repaired. When turned on again, the following operations are the same as in a control startup scenario. If the notification connection is lost due to the failure of the process controller 125, the notification manager periodically monitors the process controller 125 to be re-operational as before.

In a network outage and recovery scenario, the notification connection is lost and the NM 256 attempts to reconnect to any node on the failed (sub) network. When the network has been repaired and becomes operational again, the NM 256 restores the connections to all affected process control nodes and initiates separate notification remedies for each process control node. Unaffected process controls are not commanded to perform notification fixes.

In the scenario of adding and configuring a new process controller 125, a new process controller 125 is included in the network and configured. The NM 256, which learns from all notification distribution publishers using the system configuration, periodically polls the system database for a complete list of all process control nodes 125a. The NM 256 then compares this list to the dynamic list of process control nodes with which it has established notification connections and attempts to establish a new connection to any new process control node 125. If the connection is successfully established, the NM 256 commands a notification removal. If unsuccessful, NM 256 periodically retries to connect and command the notification.

In a notification client node startup scenario, all notification distribution publishers are commanded to perform a notification removal to send all required notifications to the notification distribution participant in the client node. This is in contrast to the startup of a notification generator node in which only the notification generator node being started is commanded to perform a notification recovery. When a notification creator node is started, the synchronization is further maintained by the other notification creator nodes, and they do not have to perform notification removal.

As can be seen from the above descriptions, the startups of notification client nodes and notification generation nodes may take place in any order. The application layer compensates for the various start-up sequences by using the notification removal to synchronize the notification clients as the notification information becomes available.

Figures 3A and 3B are flowcharts of the general operation of a notification manager 256 according to an embodiment of the present invention. After powering up (step 301), the notification manager 256 remains in an idle state until a subscription request is received from the notification client portion of the server (step 302). In response, the notification manager 256 retrieves the list of currently configured notification distribution public (NDP) process controllers 125 from the system database (step 303). Next, notification manager 256 forms a notification connection with all NDP process controllers 125 (step 304). As each notification connection is made, the notification manager 256 commands the now-connected process controller to notify (step 305).

Then, the notification manager 256 enters a routine scanning mode in which the notification manager 256 repeatedly scans the system configuration list to determine all NDP process controllers 125 configured in the network (step 310). The notification manager 256 compares the list of currently configured NDP process controllers 125a-f with its own list of process controllers 125 with which it has established notification connections (step 311). After determining which configured NDP process controllers are not connected, the notification manager 256 attempts to establish a notification connection with the disconnected NDP process controllers 125 (step 312). When a connection is made, the notification manager 256 commands the newly connected NDP process controller 125 to notify (step 313 and 314). If no connection can be established, the notification manager 256 further scans the network configuration lists, thereby again attempting from time to time to connect to any NDP process controller 125 with which the notification manager 256 has not yet formed a transport layer distribution connection.

When the notification distribution publisher 205 performs a first notification removal and a second notification removal is commanded by the same or another notification distribution agent 257, the notification distribution publisher 205 may, in a preferred embodiment of the present invention, cancel the first notification removal and initiate the second notification removal. The or the notification distribution agent receiving the first notification recovery has the ability to detect the start of the second notification removal and instead to synchronize to the second notification removal.

This is also beneficial for a notification distribution publisher 205 operating in a multi-notification client environment. Notification distributors 257 may initiate multiple notification remedies within a short time but not simultaneously. In such a case, the notification distribution publisher 205 repeatedly aborts the notification removal currently in progress when the next notification removal is commanded instead of sequentially starting and completing all notification remedies. Only the last received notification recovery can be completed, thereby requiring only a minimum amount of time to complete the notification fixes.

In a preferred embodiment of the present invention, a notification client node may at any time request a notification resolution using the same mechanism used by the application layer to perform automatic (background) notification remedies. The application layer that subscribes to the notification client node provides a means by which the notification client node or a surrogate acting on behalf of the notification client node can activate the same mechanism as the application layer to initiate a notification recovery is used.

The present invention is particularly advantageous in cases where notifications are lost as a result of a large flood of notifications overwhelming the ability of a notification client to receive the notifications. Notification removal is used if the time allows to fix the lost notifications and the synchronization is maintained. Since the notification recovery is handled automatically by the application layer, no intervention is required from the notification client node or a human operator.

In a preferred embodiment of the present invention, the notification client node may cause the application layer to \ "push back \" the notification generation nodes during a notification flood, whereby the notification generation nodes hold information until the notification flood has dropped or ended. The notification fix can then be used for resynchronization for any messages lost during the flood of alerts.