Block-oriented control system on high speed Ethernet

<u>TECHNICAL FIELD</u>

0001The present invention relates to control system architecture. More particularly, the present invention relates to an open, interoperable distributed control system in a high performance network environment.

<u>BACKGROUND OF THE INVENTION</u>

0002Automatic control systems are critical to all sectors of industry such as process control, discrete control, batch control (process and discrete combined), machine tool control, motion control, and robotics. One of the strongest needs in modern control systems is development and use of "open" and "interoperable" systems. Open, interoperable systems allow control devices made by different manufacturers to communicate and work together in the same system without the need for custom programming. "Fieldbus" is the common term used to describe these types of control systems.

0003The movement toward open, interoperable fieldbus systems is driven by device manufacturers and end users. Manufacturers want open, interoperable systems because it allows them to sell their products to more end users while reducing development costs. End users want open, interoperable systems so that they can select the best control devices for their system regardless of the device manufacturer.

0004There has also been a trend toward distribution of control functions into intelligent devices. In centralized control systems, a central controller performs all the control functions.

0005In distributed control systems, more than one control device operating in the system takes an active role in the control functions. Although both centralized and decentralized systems use a communication network, decentralized systems reduce overall system costs by reducing or eliminating the centralized controller functions between the control devices and the human-machine interface.

0006In order for distributed control systems to be truly open and interoperable, both the communications system and the user layer (above the communication system layers) must be specified and made open. One of the truly open and interoperable distributed systems is the fieldbus system provided by the Fieldbus Foundation. The FOUNDATION^™ fieldbus user layer is described, e.g., in <patcit id="pcit0001" dnum="US08916178B"><text>US Patent Application 08/916,178</text></patcit> (hereafter the "'178" application) filed August 21, 1997,entitled "BLOCK-ORIENTED CONTROL SYSTEM", and assigned to the assignee of the present application.

0007The lower speed 31.25 kilobits per second fieldbus (H1) used by the FOUNDATION™ fieldbus is described in part by International Electrotechnical Committee (IEC) Standard IEC 61158.

0008While the FOUNDATION™ fieldbus provides the open and interoperable solution for the H1 control capability, there is a great need to provide an open and interoperable solution for distributed control on a very high performance communication system typically called a fieldbus "backbone" network. The backbone network aggregates information from the lower speed control devices, e.g., the H1 and other control devices, which is used in supervisory and advanced control applications. The backbone is also needed for integration of control information into the enterprise's Management Information Systems (MIS).

0009One of the widely accepted standards for high performance communications signaling is Ethernet. Invented by Xerox in the 1970's, Ethernet has progressed from an initial speed of 10 Megabits per second, to 100 Megabits per second, to 1 Gigabit per second and beyond. Ethernet signaling is specified in an Institute of Electrical and Electronics Engineers (IEEE) standard (IEEE 802.3). Ethernet signaling is the underlying technology used by the Internet. The Internet protocols are specified by the Internet Engineering Task Force (IETF) and are issued as Request For Comment (RFC) specifications.

0010Although Ethernet/Internet technology provides the basic services for a high performance fieldbus backbone, it does not provide for all of the functions needed for use in distributed control systems. In particular, IEEE and IETF do not have suitable open and interoperable solutions for integration of distributed control systems (e.g., the H1 subsystem), system time synchronization, and fault tolerance.

0011The method of transferring information from lower speed fieldbuses to the Ethernet used by organizations such as Open DeviceNet^™ Vendor Association, Inc., ("<i><b>Ether</b>Net</i>/<b><i>IP</i>,</b>") and PROFIBUS International, ("<b><i>PROFI</i></b>Net") are not suitable for use in the high performance environment because they encapsulate the lower speed protocol packets in an Ethernet frame. This method, known as "tunneling," is common in centralized control systems, but is inadequate for high performance distributed control systems. Although simpler to specify, tunneling would require too many Transport Control Protocol (TCP) connections with the resulting interrupt processing and memory overhead on the devices connected to the fieldbus backbone. In addition tunneling wastes much of the Ethernet bandwidth because the lower speed protocol packets (e.g., the H1 packets) are small and in many cases the Ethernet packet overhead would be bigger than a lower speed protocol packet.

0012Devices connected to the Ethernet must have a common sense of system time for time stamp and function block scheduling (control) purposes. For high performance distributed control, system time often needs to be accurate to within less than 1 millisecond. Heretofore, there is no known solution that provides this accuracy using the Commercial Off The Shelf (COTS) Ethernet equipment.

0013Fault tolerance of the Ethernet communication media and devices connected to the Ethernet is required for high performance distributed control applications. There is no known solution that provides the required fault tolerance using standard COTS Ethernet equipment. All of the prior attempts in providing the required fault tolerance require special Ethernet/Internet electronic hardware and/or software, and/or a non-standard "redundancy manager" device to be added to the Ethernet.

0014Thus, what is needed is an open, interoperable solution optimized for integration of distributed control systems and other control devices in a high performance fieldbus backbone.

0015What is also needed is an open, interoperable solution that provides system time synchronization suitable for distributed control applications operable over a high performance fieldbus backbone.

0016What is also needed is an open, interoperable solution that provides a fault tolerant high performance fieldbus backbone as well as fault tolerant devices that are connected to the fieldbus backbone.

0017<nplcit id="ncit0001" npl-type="s"><text>J. He et al.: "Clock Synchronisation in Real-Time Distributed Systems based on FIP Field Bus", Second IEEE Workshop on Future Trends of Distributed Computing Systems, 1990, pages 135 to 141</text></nplcit>, disclose a technique for synchronising physical clocks in a field bus system. The precharacterising part of claim 1 starts out from this prior art.

0018<nplcit id="ncit0002" npl-type="s"><text>H. Lönn et al.: "Sychronisation in Safety-Critical Distributed Control Systems", IEEE First International Conference on Algorithms and Architectures for Parallel Processing, 1995, pages 891 to 899</text></nplcit>, disclose a daisy chain method for synchronising clocks of a distributed computer system.

<u>SUMMARY OF THE INVENTION</u>

0019The invention overcomes the shortcomings described above, by an apparatus and a method as defined in claims 1 and 6. The dependent claims relate to preferred embodiments of the invention.

0020Embodiments of the present invention relate to a distributed control system, which operates on a high performance backbone, e.g., the standard COTS Ethernet and Internet technology. The embodiments of the present invention are collectively referred to herein as the "High Speed Ethernet" (HSE). HSE includes the features of the distributed control system described by the '178 application and FOUNDATION™ fieldbus specifications (which are listed in Appendix A as the Reference Set 1), and further includes three new protocols described in the supporting specifications thereof, which are listed in Appendix A as the Reference Set 2. In particular, the new protocols are referred to herein as: the HSE Field Device Access (FDA) Agent, the HSE System Management Kernel (SMK), and the HSE Local Area Network Redundancy Entity (LRE).

0021The HSE FDA Agent allows System Management (SM) and Fieldbus Message Specification (FMS) services used by the H1 devices to be conveyed over the Ethernet using standard Internet User Data Protocol (UDP) and Transport Control Protocol (TCP). This allows HSE Devices on the Ethernet to communicate to H1 devices that are connected via a "HSE Linking Device." The HSE FDA Agent is also used by the local Function Block Application Process (FBAP) in a HSE Device or HSE Linking Device. Thus, the HSE FDA Agent enables remote applications to access HSE Devices and/or H1 devices through a common interface.

0022The HSE SMK ensures that system level functions in each device are coordinated. These functions include system time, addition and removal of devices from the network, and function block scheduling. HSE SMK uses local clock that operates to keep a local time, and keeps the difference between the local time and a system time provided by a time server within a value specified by the time sync class (See Reference Set 1 of Appendix A herein). The local time is used to time stamp events so that event messages from devices may be correlated across the system. Local time is also used to schedule the execution of the local function blocks.

0023HSE fault tolerance is achieved by operational transparency i.e., the redundancy operations are not visible to the HSE applications. This is necessary because HSE applications are required to coexist with standard MIS applications. The HSE LRE coordinates the redundancy function. Each HSE Device periodically transmits a diagnostic message representing its view of the network to the other HSE Devices on its Ethernet interfaces (commonly called Ethernet "Ports"). Each device uses the diagnostic messages to maintain a Network Status Table (NST), which is used for fault detection and Ethernet transmission port selection. There is no central "Redundancy Manager". Instead, each device determines how it should behave in response to faults it detects.

<u>BRIEF DESCRIPTION OF THE DRAWINGS</u>

0024Features and advantages of the present invention will become apparent to those skilled in the art from the following description with reference to the drawings, in which: <ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">Figure 1</figref> is a block diagram showing an exemplary embodiment of a high performance distributed control system in accordance with the principles of the present invention;</li><li><figref idref="f0002">Figure 2</figref> is a block diagram showing an exemplary embodiment of device system architecture of a high performance distributed control system in accordance with the principles of the present invention;</li><li><figref idref="f0003">Figure 3</figref> is a block diagram showing an exemplary embodiment of the structure of the High Speed Ethernet Management Information Base of the device system architecture shown in <figref idref="f0002">Figure 2</figref>;</li><li><figref idref="f0004">Figure 4</figref> is a block diagram showing an exemplary embodiment of the device system architecture shown in <figref idref="f0002">Figure 2</figref>, showing the various local interfaces of the High Speed Ethernet Field Device Access agent;</li><li><figref idref="f0005">Figure 5</figref> is a block diagram showing an exemplary embodiment of the relevant portions of the high performance distributed control system involved in time synchronization process in accordance with the principles of the present invention;</li><li><figref idref="f0006">Figure 6</figref> is a flow diagram illustrative of an exemplary embodiment of the process of time synchronization in accordance with an embodiment of the principles of the present invention;</li><li><figref idref="f0007">Figure 7A</figref> is a timing diagram illustrative of a starting time offset before the time synchronization process in accordance with an embodiment of the principles of the present invention;</li><li><figref idref="f0007">Figure 7B</figref> is a timing diagram illustrative of a starting time offset after the time synchronization process in accordance with an embodiment of the principles of the present invention; and</li><li><figref idref="f0008">Figure 8</figref> is a block diagram showing an exemplary embodiment of the redundant topology of a high performance distributed control system in accordance with the principles of the present invention.</li></ul>

<u>DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT</u>

0025For simplicity and illustrative purposes, the principles of the present invention are described by referring mainly to exemplary embodiments, particularly, with specific exemplary implementations of distributed control system in an Ethernet network. However, one of ordinary skill in the art would readily recognize that the same principles are equally applicable to, and can be implemented in, other implementations and designs using any other high speed networks.

A

:

HSE DISTRIBUTED CONTROL SYSTEM OVERVIEW

0026Referring to <figref idref="f0001">FIG. 1</figref>, an example of a high performance control system <b>100</b> is shown where standard COTS Ethernet equipment <b>130</b> is used to interconnect HSE Linking Devices <b>110</b> and HSE Devices <b>120</b> to an Ethernet Network <b>140.</b> The HSE Linking Devices <b>110</b> in turn connect to H1 Devices <b>170</b> using H1 Networks <b>150.</b> Other types of equipment such as a Personal Computer (PC) <b>160</b> may also be connected to the Ethernet Network <b>140.</b>

0027The actual Ethernet network topology and COTS Ethernet equipment configuration will depend on the particular application needs. However any Ethernet network topology or configuration using standard COTS Ethernet equipment other than the exemplary topology shown in <figref idref="f0001">Fig. 1</figref> may be used.

A.1: HSE SYSTEM ARCHITECTURE

0028The HSE system architecture in accordance with an embodiment of the principles of the present invention is shown in <figref idref="f0002">FIG. 2</figref>. The HSE system architecture is designed to meet the functional needs of the high performance distributed manufacturing and process control environments, e.g., in a high speed Ethernet network. It permits distributed automation systems to be constructed from various control and measurement devices manufactured by different vendors. The HSE system architecture is described by architecture components that have been adapted to the specifics of both the H1 and HSE environments.

0029The various protocols and standards referenced in the following disclosure are described in detail in the manuals and specifications listed in Appendix A herein, which are available from the Fieldbus Foundation, a not-for-profit organization headquartered in Austin, Texas, and the respective current versions as of the filing date of the present invention of all of which are hereby incorporated by reference in their entirety herein. Each of the architecture components of the HSE system architecture (shown in <figref idref="f0002">Fig. 2</figref>) will now be described in more detail.

A.2: Function Block Application Process Virtual Field Device (FBAP VFD)

0030Application Process (AP) is a term defined by the International Standards Organization (ISO) Open Systems Interconnect (OSI) Reference Model (RM), ISO 7498, to describe the portion of a distributed application that is resident in a single device. The term is used in the following description to refer to the entities within a device that performs a related set of functions, such as function block processing, network management, and system management.

0031Virtual Field Device (VFD) is a term defined by the Fieldbus Foundation (See Fieldbus Message Specification FF-870 listed in Reference Set 1 in Appendix A herein). A VFD makes the parameters of an AP visible to a communication network.

0032In accordance with the principles of the present invention, the HSE system architecture (shown in <figref idref="f0002">Fig. 2</figref>) supports the Function Block Application Process Virtual Field Device (FBAP VFD) <b>260.</b> The FBAP VFD <b>260</b> provides a common means for defining inputs, outputs, algorithms, control variables, and behavior of the automation system. There may be multiple FBAP VFDs <b>260</b>, e.g., n FBAP VFDs as shown, in a device in order to satisfy the particular needs an application. The FBAP VFD may or may not be present in an HSE Device or HSE Linking Device. If the HSE FBAP VFD is present, the device is sometimes also referred to as a "HSE Field Device." In the following descriptions, however, the FBAP VFD is to be assumed to be present in the HSE Device and HSE Linking Device, even if the term "HSE Field Device" is not used.

0033A standard set of function block classes and parameters are defined by the Fieldbus Foundation, e.g., in one or more of the specifications listed in Appendix A herein. Manufacturers of control devices may append their own parameters to the standard set of parameters to accommodate additional function block definitions as new requirements are discovered, and as technology advances. A more detailed description of the function block classes and parameters may be found, e.g., in Function Block Application Process-Part 1 Specification FF-890 listed in Reference Set 1 of Appendix A herein.

A.3: H1 Interface

0034Each H1 Network <b>150</b> attached to a HSE Linking Device <b>110</b> (shown in <figref idref="f0001">FIG. 1</figref>) requires a H1 Interface <b>240.</b> The Bridge <b>250</b> is used to convey H1 Network messages directly between other H1 Interfaces <b>240</b> within the same HSE Linking Device <b>110</b> (shown in <figref idref="f0001">FIG. 1</figref>). A HSE Linking Device may comprise, e.g., a HSE Device <b>120</b> (shown in <figref idref="f0001">FIG. 1</figref>) that includes at least one H1 Interface <b>240.</b>

0035A more detailed description of a H1 Interface may be found in the Fieldbus Message Specification FF-870, Fieldbus Access Sublayer Specification FF-821, Data Link Services and Data Link Protocol Specifications FF-821, 822, and Data Link Protocol Specification for Bridge Operation Addendum FF-806, all of which are listed in the Reference Set 1 of Appendix A herein.

A.4: Ethernet/Internet Stack

0036The HSE system architecture uses a standard COTS Ethernet/Internet ("stack") <b>280</b> for communication with other devices on the Ethernet Network <b>140.</b> The Ethernet/Internet stack used by HSE consists of Distributed Host Control Protocol (DHCP) <b>285,</b> Simple Network Time Protocol (SNTP) <b>286,</b> and Simple Network Management Protocol (SNMP) <b>287</b>, which in turn use Transport Control Protocol (TCP) <b>283</b> and User Data Protocol (UDP) <b>284</b> services.

0037TCP <b>283</b> and UDP <b>284</b> in turn use the standard Internet Protocol (IP) <b>282</b> services, which uses the standard IEEE Ethernet 802.3 Media Access Control (MAC) and Physical (PHY) Layers <b>281.</b> The PHY layer in <b>281</b> connects to one or more Ethernet Networks <b>140.</b>

0038The Internet DHCP, SNTP, SNMP, TCP, UDP and IP protocols are specified by the Internet Engineering Task Force (IETF) Request For Comment (RFC) specifications. The IETF RFCs are listed in Appendix B herein. An Institute of Electrical and Electronics Engineers (IEEE) standard (IEEE 802.3) describes the Ethernet MAC and PHY layers. The specific use of each layer and protocol are detailed in the Ethernet Presence Specification FF-586 listed in Reference Set 2 of Appendix A herein.

0039By preserving the standard use of the Ethernet/Internet stack, the HSE system architecture ensures interoperability among the different stack manufacturers.

A.5: HSE Management Agent

0040Again referring to <figref idref="f0002">FIG. 2</figref>, in general, the HSE Management Agent <b>270</b> uses DHCP <b>285</b> for acquiring an IP address for the device, SNTP <b>286</b> for maintaining time synchronization with a time server, and SNMP <b>287</b> for managing the TCP, UDP, and IP protocol layers. HSE Management Agent use of DHCP, SNTP and SNMP is routine and complies with standard practices known to those familiar with the Internet protocols, e.g., according to IEEE 802.3.

0041The HSE Management Agent uses SNMP <b>287</b> for managing the Internet layer protocols. Specifically, the HSE Management Agent <b>270</b> provides Ethernet Network access to the standard Management Information Base-II (MIB II) as defined by SNMPv2 in RFC 1213 and RFC 1643 (see Appendix B), and as defined also by Ethernet Presence FF-586 listed in Reference Set 2 of Appendix A herein.

0042In accordance with an embodiment of the present invention, in order to comply with the ISO standards, the HSE Management Information Base (HSE MIB) <b>271</b> comprises of a standard part, which is the second version of MIB-II as defined in RFC 1213 and a HSE specific part (which is defined under the private enterprises level). For convenience in understanding, the detailed structure of the HSE MIB <b>271</b> is shown in <figref idref="f0003">FIG. 3</figref>. The standardized structure of the HSE MIB <b>271</b> provides a profile allowing interoperability, making the device appear as a well-behaved node.

B: HSE Core

0043Referring again to <figref idref="f0002">FIG. 2</figref>, the HSE core portion <b>200</b> of the HSE system architecture identifies the new HSE capability in accordance with the principles of the present invention. The HSE core <b>200</b> provides the essential capabilities and integration needed to realize the high performance distributed control using HSE Devices, HSE Linking Devices and standard COTS Ethernet equipment.

B.1: Network Management Agent Virtual Field Device

0044The HSE System Architecture includes a Network Management Agent VFD (NMA VFD) <b>210</b> for each HSE Device and each HSE Linking Device. The NMA VFD provides means for configuring, controlling and monitoring HSE Device and HSE Linking Device operation from the network.

0045Management information is contained in the Network Management Information Base (NMIB) <b>213</b> and the System Management Information Base (SMIB) <b>212.</b> Using the configuration management capabilities of the NMA VFD, parameters are set in the NMIB and SMIB to support data exchanges with other devices in the system. This process involves defining the transfers between devices and then selecting the desired communications characteristics to support the transfers.

0046The NMA VFD can also be configured to collect performance and fault related information for selected transfers. This information is accessible during run-time, making it possible to view and analyze the behavior of device communications. If a problem is detected, performance is to be optimized, or device communications are to be changed, then reconfiguration can be performed dynamically while the device is still operating.

0047NMA VFD parameters and behavior are further defined in the HSE Network Management Specification FF-803 listed in the Reference Set 2 of Appendix A herein.

B.2: HSE Field Device Access Agent

0048The HSE Field Device Access (FDA) Agent will now be described with References to <figref idref="f0004">FIG. 4</figref>, which is the same figure as <figref idref="f0002">FIG. 2</figref> except that Local Interactions (<b>291-299</b>) for the HSE Field Device Access (FDA) Agent <b>290</b> are shown. The operation of the HSE FDA Agent will now be described in terms of these local interactions.

0049One of the main functions of the HSE FDA Agent <b>290</b> is to map services already defined for FOUNDATION™ fieldbus System Management (SM) (<i>See</i> FF-880 listed in the Reference Set 1 of Appendix A herein) and Fieldbus Message Specification (FMS) (See FF-870 listed in the Reference Set 1 of Appendix A herein) to and from the standard, COTS Ethernet/Internet <b>280</b> component.

0050Generally, the HSE FDA Agent <b>290</b> emulates the mapping defined by the FOUNDATION™ fieldbus Fieldbus Access Sublayer specification (See FF-875 listed in the Reference Set 1 of Appendix A herein). The HSE FDA Agent <b>290</b> provides the common interface that enables remote applications to access devices of any type on both the H1 Networks <b>150</b> and the HSE Network <b>140.</b>

0051Thus the HSE FDA Agent <b>290</b> in accordance with the principles of the present invention allows systems to be constructed where the control is distributed in into various HSE Devices and/or H1 Devices, and any combinations thereof, as needed by the particular end user application.

B.2.1: HSE FDA Agent Local Interfaces

0052<b>B.2.1(a): Local Interface 291: TCP -</b> The TCP local interface <b>291</b> allows the HSE FDA Agent <b>290</b> to send and/or receive FMS messages using TCP <b>283</b>. TCP <b>283</b> provides interfaces modeled as sockets through which the HSE FDA Agent <b>290</b> submits a buffer that contains one or more messages.

0053<b>B.2.1(b): Local Interface 292: UDP -</b> The UDP local interface <b>292</b> allows the HSE FDA Agent <b>290</b> to send and/or receive SM messages and certain FMS messages using UDP <b>284.</b> UDP <b>284</b> provides interfaces modeled as sockets through which the HSE FDA Agent <b>290</b> submits a buffer that contains one or more messages.

0054<b>B.2.1(c): Local Interface 293: HSE NMIB -</b> The HSE FDA Agent <b>290</b> provides a local interface to the HSE NMIB <b>213</b> in NMA VFD <b>210.</b> The HSE FDA Agent is capable of providing configuration and read-only access to NMA VFD <b>210</b> via the HSE NMIB Local Interface <b>293.</b>

0055<b>B.2.1(d): Local Interface 294: HSE SMIB -</b> The HSE FDA Agent <b>290</b> provides a local interface to the HSE SMIB <b>212</b> in NMA VFD <b>210.</b> The HSE FDA Agent <b>290</b> is capable of providing configuration and read-only access to NMA VFD <b>210</b> via the HSE SMIB Local Interface <b>294.</b>

0056<b>B.2.1(e): Local Interface 295: HSE SMK -</b> The HSE FDA Agent <b>290</b> conveys HSE SM services to and from the HSE SMK <b>220</b> through the HSE SMK local interface <b>295.</b> In accordance with an embodiment of the present invention, in a HSE Linking Device, the HSE SMK <b>220</b> communicates locally with each of the H1 interfaces <b>240,</b> and does not use the HSE FDA Agent <b>290</b>.

0057<b>B.2.1(f): Local Interface 296: HSE LRE -</b> The HSE FDA Agent <b>290</b> maintains a local interface with the HSE LAN Redundancy Entity (HSE LRE) <b>230</b> of the device through the HSE LRE local interface <b>296.</b> Use of the HSE LRE local interface <b>296</b> will be described in more detail later.

0058<b>B.2.1(g): Local Interface 297: H1 Interface -</b> Only HSE FDA Agents <b>290</b> of a HSE Linking Device interact with the H1 Interface(s) <b>240</b> to access H1 Networks <b>150.</b> The H1 local interface provides the HSE FDA Agent with FMS and SM access through the HSE SMK <b>220.</b>

0059The HSE FDA Agent forwards FMS requests and responses received from the TCP Interaction <b>291</b> and UDP Interaction <b>292</b> to H1 Network <b>150</b> through the H1 Interface(s) <b>240.</b> The HSE FDA Agent also forwards H1 requests and responses received from a H1 Network through the H1 Interface Interaction <b>297</b> to the Ethernet Network <b>140</b> using TCP Interaction <b>291</b> and UDP Interaction <b>292.</b>

0060Thus, the HSE FDA Agent <b>290</b> interacts with the services in the H1 Network in the same manner as any other application program would normally interact with the H1 network.

0061<b>B.2.1(h): Local Interface 298: FBAP VFD -</b> The HSE FDA Agent <b>290</b> uses the FBAP VFD local interface <b>298</b> to access the FBAP VFD <b>260.</b> Both FMS and SM messages are communicated using the FBAP VFD local interface <b>298.</b>

0062<b>B.2.1(i): Local Interface 299: HSE Management Agent -</b> The HSE FDA Agent <b>290</b> maintains the HSE Management Agent local interface <b>299</b> with the HSE Management Agent <b>270</b> to access certain Quality of Service parameters associated with its UDP/TCP connections. The use of these parameters by the HSE FDA Agent <b>290</b> is local to the specific UDP/TCP implementation.

B.2.2: HSE FDA Agent Operation

0063Again referring to <figref idref="f0004">FIG. 4</figref>, during the configuration of the system, HSE SMK <b>220</b> uses the local interface <b>295</b> for adding HSE and/or H1 devices to, and deleting the same from, the distributed system. An exchange of SM messages is used to identify new (or to be deleted) HSE and/or H1 Devices in the system.

0064For example, after a new HSE Device receives an Internet Protocol (IP) address, the new HSE Device periodically announces its presence on the Ethernet network <b>140.</b> HSE Linking Devices also announce changes detected on their H1 Network <b>150.</b> In a similar way, HSE SMK uses the local interface <b>295</b> to determine the location of the function block "tags" that might exist in the HSE Devices and/or H1 Devices.

0065During operation of the system, the data acquisition, display and supervisory control functions, which are typically executing on a Personal Computer (PC) connected to the Ethernet Network <b>140,</b> will need to access the data in a HSE Device, a HSE Linking Device and/or H1 devices connected to the H1 Networks <b>150.</b> The data access is typically performed using the "Client/Server" and/or the "Publisher/Subscriber" messages. These data access methods are well known to those familiar with Fieldbus messaging.

0066For Client/Server and Publisher/Subscriber messages originating or terminating in the HSE Device and/or HSE Linking Device, the HSE FDA Agent <b>290</b> sends and receives the Ethernet Network <b>140</b> messages on the local interface <b>291,</b> provides the appropriate mapping to FMS services as previously described above, and uses local interfaces <b>293, 294, 296, 298,</b> and <b>299</b> to access the HSE NMIB <b>213</b>, HSE SMIB <b>212</b>, HSE LRE <b>230</b>, FBAP VFD(s) <b>260</b> and the HSE Management Agent <b>270,</b> respectively. HSE SMK <b>220</b> is not accessed because it has its own SM messages as previously described.

0067For Client/Server, Publisher/Subscriber and/or SM messages originating or terminating in the H1 Network <b>150,</b> the HSE FDA Agent <b>290</b> uses local interface <b>297</b> to send and/or receive messages from H1 Interface(s) <b>240</b>.

0068If the messages from the H1 network <b>150</b> are to/from the Ethernet Network <b>140,</b> and are Client/Server or Publisher/Subscriber messages, HSE FDA Agent <b>290</b> uses the FMS mapping and local interface <b>291.</b> If the H1 messages to/from the Ethernet Network <b>140</b> are SM messages, the HSE FDA Agent uses the SM mapping and local interface <b>292.</b>

0069If the messages to/from H1 Network <b>150</b> are to/from the HSE Linking Device, and are Client/Server or Publisher/Subscriber messages, HSE FDA Agent will use FMS mapping and the appropriate local interface (except the local interfaces <b>291</b> and <b>292</b>).

0070If the messages to/from H1 Network <b>150</b> are to/from the HSE Linking Device, and are SM messages, HSE FDA Agent will use SM mapping and the appropriate local interface (except the local interfaces <b>291</b> and <b>292</b>).

B.3: HSE System Management Kernel

0071Referring again to <figref idref="f0002">FIG. 2</figref>, the HSE system architecture includes a HSE System Management Kernel (SMK) <b>220</b> for each HSE device and/or each HSE linking device. The HSE SMK <b>220</b> maintains information and a level of coordination that provides an integrated network environment for the execution and interoperation of the FBAP VFD <b>260.</b>

0072As previously discussed, HSE SMK <b>220</b> provides for routine configuration of certain basic system information prior to device operation. For example HSE SMK startup takes a device through a set of predefined phases for this purpose. During this procedure a system configuration device recognizes the presence of the device on the network and configures basic information into the HSE SMIB <b>212.</b> Once the device receives its basic configuration information, its HSE SMK brings it to an operational state without affecting the operation of other devices on the network. It also enables the HSE FDA Agent <b>290</b> for use by other functions in the device.

B.3.1: HSE SMK System Time Synchronization

0073Now referring to <figref idref="f0005">FIG. 5</figref>, the HSE Management Agent <b>270</b> in HSE Linking Device <b>110</b> uses SNTP <b>286</b> to interact with remote SNTP Server <b>510</b> in Time Master <b>500</b> to synchronize System Time <b>501'</b> in HSE MIB <b>271'</b> with System Time <b>501</b> in the Time Master <b>500.</b> When System Time <b>501'</b> is synchronized with System Time <b>501,</b> Sync Flag (F) <b>510</b> in the HSE MIB is set to true by the standard SNTP protocol. The Time Master and the HSE Linking Device are interconnected using standard, COTS Ethernet equipment <b>130.</b> This synchronization protocol is defined in IETF RFC 2030.

0074At any moment, Local Time <b>502</b> in HSE SMIB <b>212</b> may or may not be synchronized with System Time <b>501'.</b> In order to coordinate execution of function blocks in a distributed system, and to provide proper time stamping of function block alarms, Local Time <b>502</b> must be Synchronized with System Time <b>501</b>'.

0075All of the function blocks are synchronized with Start of Macrocycle, "T₀" <b>520</b> in HSE SMIB <b>212</b>. Each HSE Linking Device and HSE Device in the system has the same value for To. A function block is executed when HSE SMK <b>220</b> locally issues a Function Block (FB) Start <b>221</b> message for the block. Each FB Start message is generated based on an offset from T₀.

0076At the start of the macrocycle, To and the offset for each block is based on Local Time <b>502.</b> Therefore each device must adjust their Local Time <b>502</b> to equal the System Time <b>501'</b> for the system to function properly. However, because each device has a hardware clock oscillator that is not perfect, Local Time <b>502</b> will eventually drift out of synchronization with System Time <b>501'.</b>

0077<figref idref="f0006">Fig. 6</figref> shows the process of correcting for the drift in accordance with an embodiment of the present invention. In particular, when a macrocycle ends in step <b>601,</b> the HSE SMK <b>220</b> will test the Sync Flag <b>510</b> in HSE MIB <b>271'</b> in step <b>602.</b> If F <b>510</b> is not true, the process ends in step <b>606.</b>

0078If, on the other hand, it is determined in step <b>602</b> above that F <b>510</b> is true, HSE SMK <b>220</b> computes the offset between Local Time <b>502</b> and System Time <b>501'</b> in step <b>603,</b> and sets the Local Time <b>502</b> to equal the System Time <b>501'</b> within a value specified in a desired time sync class (See Reference Set 1 of Appendix A herein)in step <b>604.</b>

0079Once the Local Time <b>502</b> is synchronized, in step <b>605,</b> the start time (To) <b>520</b> (shown in <figref idref="f0005">FIG. 5</figref>) is aligned with start time of other devices.

0080The start time alignment will now be described with references to <figref idref="f0007">FIGs. 7A and 7B. FIG. 7A</figref> shows the macrocycle offset of a device, e.g., device N, before the time synchronization, in which the offset <b>720</b> represents the error that must be corrected in the HSE Device N. As shown, the HSE Device N now has the correct Local Time, but the start time(To) 520' of System Macrocycle <b>700'</b> is not aligned with other devices in the distributed system.

0081<figref idref="f0007">FIG. 7B</figref> shows the macrocycle offset of a device, e.g., device N, after the time synchronization. The HSE SMK <b>220</b> of the Device N delays the start time (T₀) <b>520'</b> of the System Macrocycle 700' by Offset <b>720</b> so that the System Macrocycle begins at the same time(T₀) <b>520</b> as, e.g., System Macrocycle <b>700</b> in HSE Device 1. HSE Device N System Macrocycle is now synchronized with the System Time, and the synchronization process ends at step <b>606</b> (shown in <figref idref="f0006">FIG. 6</figref>).

B.4: Local Area Network Redundancy Entity

0082Referring to <figref idref="f0004">FIG. 4</figref>, each HSE Device and HSE Linking Device has a HSE Local Area Network (LAN) Redundancy Entity (HSE LRE) <b>230.</b> The HSE LRE provides fault tolerance to single failure through the use of redundancy.

0083HSE LRE periodically sends and receives Redundancy Diagnostic Messages over local interface <b>296.</b> HSE FDA Agent <b>290</b> maps the Diagnostic messages on local interfaces <b>291</b> and <b>292</b> (See HSE Redundancy Specification FF-593 listed in the Reference Set 2 of Appendix A herein for the Redundancy Diagnostic Message Formats.)

0084The Redundancy Diagnostic Messages are sent concurrently on Ethernet Network <b>140</b> and Ethernet Network <b>140'.</b> Each device receives the Redundancy Diagnostic Messages on Ethernet Network <b>140</b> and Ethernet Network <b>140'</b> and constructs a local Network Status Table (NST) <b>231.</b> The NST provides detailed status on the condition of every HSE device connected to Ethernet Network <b>140</b> and Ethernet Network <b>140'.</b> The HSE LRE <b>230</b> controls which Ethernet Network <b>140</b> or <b>140'</b> the HSE Device will use for message transmission.

0085With this method, all of the network transmission and device switchover decisions are distributed into the HSE Devices and the system uses standard, COTS Ethernet equipment.

0086<figref idref="f0008">FIG. 8</figref> illustrates the general topology supported by the redundancy aspect of the present invention. The topology shown is only an example, showing one of many possible topologies. Any topology may be used as long as behavior of the equipment providing Ethernet Networks <b>140</b> and <b>140'</b> is standard, COTS Ethernet equipment.

0087HSE redundancy supports both Ethernet Network redundancy and HSE Linking Device redundancy.

B.4.1: Ethernet Network Redundancy

0088Referring to <figref idref="f0008">FIG. 8</figref>, HSE Devices <b>120'</b> and HSE Linking Device Pairs <b>110'</b> have interfaces to both Ethernet Network <b>140</b> and Ethernet Network <b>140'.</b> In this example, Ethernet Network <b>140</b> is provided by COTS Ethernet equipment <b>130</b> and Ethernet Network <b>140'</b> is provided by COTS Ethernet equipment <b>130'.</b> A single failure of any one of the Ethernet Networks or one of the Ethernet interfaces on a HSE device would cause the HSE LRE previously described to force communications on the remaining functional network.

B.4.2: HSE Linking Device Redundancy

0089The HSE LRE <b>230</b> supports HSE Linking Device redundancy. Redundant HSE Linking Device Pair <b>160</b> comprises primary HSE Linking Device <b>110,</b> and standby HSE Linking Device <b>110'.</b> H1 Devices <b>170</b> are connected by H1 Networks <b>150</b> to the Redundant HSE Linking Device Pair <b>160.</b> If primary HSE Linking Device <b>110</b> fails, standby HSE Linking Device <b>110'</b> will assume control. A HSE device <b>120'</b> may be made redundant in the same manner as the HSE linking device <b>110,</b> except in a HSE device H1 interface(s) is (are) not present.

0090The present invention provides the necessary diagnostic message format to allow an open and interoperable switch-over of the redundant high speed Ethernet networks and/or the redundant HSE linking devices (or HSE devices).

0091The redundancy method for backup of each H1 Network is described in the '178 application, and by the specifications listed in Reference Set 1 of Appendix A herein.

0092As can be appreciated, the distributed control system architecture in the foregoing description provides an open, interoperable solution optimized for integration of distributed control systems and other control devices in a high performance backbone, provides an open, interoperable solution that provides system time synchronization suitable for distributed control applications operable over a high performance backbone, and provides an open, interoperable solution that provides a fault tolerant high performance backbone as well as fault tolerant devices that are connected to the backbone.

0093The preferred embodiments set forth above are to illustrate the invention and are not intended to limit the present invention. Additional embodiments and advantages within the scope of the claimed invention will be apparent to one of ordinary skill in the art.

0094Moreover, while the invention has been described with reference to the exemplary embodiments thereof, those skilled in the art will be able to make various modifications to the described embodiments of the invention without departing from the scope of the claims. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method of the present invention has been described by examples, the steps of the method may be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the scope of the following claims and their equivalents.

APPENDIX A

A.1 Reference Set 1

0095<tables id="tabl0001" num="0001"><table frame="none"><tgroup cols="3" colsep="0" rowsep="0"><colspec colnum="1" colname="col1" colwidth="17mm" /><colspec colnum="2" colname="col2" colwidth="18mm" /><colspec colnum="3" colname="col3" colwidth="72mm" /><thead><row><entry valign="top"><b>Number</b></entry><entry valign="top"><b>Revision</b></entry><entry align="center" valign="top"><b>Specification</b></entry></row></thead><tbody><row><entry>FF-801</entry><entry>FS 1.4</entry><entry>Network Management</entry></row><row><entry>FF-806</entry><entry>FS 1.0</entry><entry>Data Link Protocol - Bridge Operation Addendum</entry></row><row><entry>FF-821</entry><entry>FS 1.4</entry><entry>Data Link Services Subset</entry></row><row><entry>FF-822</entry><entry>FS 1.4</entry><entry>Data Link Protocol Subset</entry></row><row><entry>FF-870</entry><entry>FS 1.4</entry><entry>Fieldbus Message Specification</entry></row><row><entry>FF-875</entry><entry>FS 1.4</entry><entry>Fieldbus Access Sublayer</entry></row><row><entry>FF-880</entry><entry>FS 1.4</entry><entry>System Management</entry></row><row><entry>FF-890</entry><entry>FS 1.4</entry><entry>Function Block Application Process-Part 1</entry></row></tbody></tgroup></table></tables>

A.2 Reference Set 2

0096<tables id="tabl0002" num="0002"><table frame="none"><tgroup cols="3" colsep="0" rowsep="0"><colspec colnum="1" colname="col1" colwidth="17mm" /><colspec colnum="2" colname="col2" colwidth="18mm" /><colspec colnum="3" colname="col3" colwidth="49mm" /><thead><row><entry valign="top"><b>Number</b></entry><entry valign="top"><b>Revision</b></entry><entry align="center" valign="top"><b>Specification</b></entry></row></thead><tbody><row><entry>FF-803</entry><entry>FS 1.0</entry><entry>HSE Network Management</entry></row><row><entry>FF-586</entry><entry>FS 1.0</entry><entry>HSE Ethernet Presence</entry></row><row><entry>FF-588</entry><entry>FS 1.0</entry><entry>HSE Field Device Access Agent</entry></row><row><entry>FF-589</entry><entry>FS 1.0</entry><entry>HSE System Management</entry></row><row><entry>FF-593</entry><entry>PS 2.0</entry><entry>HSE Redundancy</entry></row></tbody></tgroup></table></tables>

APPENDIX B

0097<tables id="tabl0003" num="0003"><table frame="all"><tgroup cols="3"><colspec colnum="1" colname="col1" colwidth="24mm" /><colspec colnum="2" colname="col2" colwidth="47mm" colsep="0" /><colspec colnum="3" colname="col3" colwidth="71mm" /><thead><row><entry align="center" valign="top"><b>RFC Number</b></entry><entry namest="col2" nameend="col3" align="center" valign="top"><b>RFC Title</b></entry></row></thead><tbody><row><entry>768</entry><entry namest="col2" nameend="col3" align="left">User Datagram Protocol (UDP)</entry></row><row rowsep="0"><entry>791</entry><entry namest="col2" nameend="col3" align="left">Internet Protocol (IP)</entry></row><row rowsep="0"><entry /><entry namest="col2" nameend="col3" align="left">Amended by:</entry></row><row rowsep="0"><entry /><entry /><entry>IP Subnet Extensions, RFC 950</entry></row><row rowsep="0"><entry /><entry /><entry>IP Broadcast Datagrams, RFC 919</entry></row><row><entry /><entry /><entry>IP Broadcast Datagrams with Subnets, RFC 922</entry></row><row><entry>792</entry><entry namest="col2" nameend="col3" align="left">Internet Control Message Protocol (ICMP)</entry></row><row><entry>793</entry><entry namest="col2" nameend="col3" align="left">Transport Control Protocol (TCP)</entry></row><row><entry>826</entry><entry namest="col2" nameend="col3" align="left">Ethernet Address Resolution Protocol (ARP)</entry></row><row><entry>894</entry><entry namest="col2" nameend="col3" align="left">Ethernet Framing of IP Datagrams over Ethernet</entry></row><row><entry>1042</entry><entry namest="col2" nameend="col3" align="left">IEEE 802.2&amp;3 Framing of IP Datagrams over Ethernet</entry></row><row><entry>1112</entry><entry namest="col2" nameend="col3" align="left">Internet Group Management Protocol (IGMP)</entry></row><row><entry>1122</entry><entry namest="col2" nameend="col3" align="left">Requirements for Internet Hosts -- Communication Layers</entry></row><row><entry>1155</entry><entry namest="col2" nameend="col3" align="left">Structure and Identification of Management Information</entry></row><row><entry>1157</entry><entry namest="col2" nameend="col3" align="left">Simple Network Management Protocol (SNMP)</entry></row><row><entry>1213</entry><entry namest="col2" nameend="col3" align="left">Management Information Base-II (MIB II)</entry></row><row><entry>1533</entry><entry namest="col2" nameend="col3" align="left">DHCP Options and BOOTP Vendor Extensions</entry></row><row><entry>1541</entry><entry namest="col2" nameend="col3" align="left">Dynamic Host Configuration Protocol (DHCP)</entry></row><row><entry>1643</entry><entry namest="col2" nameend="col3" align="left">Definitions of Managed Objects for the Ethernet-like Interface Types</entry></row><row><entry>2030</entry><entry namest="col2" nameend="col3" align="left">Simple Network Time Protocol (SNTP)</entry></row></tbody></tgroup></table></tables>