Communication method and system for transmitting timed and event-driven ethernet messages

The invention relates to a communication method for the transmission of Ethernet messages in a distributed real-time system in which a large number of node computers, e.g. the four node computers (111, 112, 113, 114) which each have at least one communication controller (121, 122, 123, 124) and which are connected via a communication system consisting of one or more communication channels (109), and where in one or more intelligent stem couplers (101, 102) are arranged in each communication channel. According to the invention, the communication system distinguishes between conventional Ethernet (ET) messages and time-controlled Ethernet (TT) messages, and the TT messages are transported between the transmitter and receiver with a constant delay time (Δ) known a priori, with a time conflict between ET and TT messages the transport of the conflicting ET message is delayed or aborted, in order to be able to transport the TT message with the constant delay time (Δ), the constant delay time (Δ) being selected so that the output channels of the stem coupler can be cleared for the transport of the incoming TT message within this delay time (Δ).

<img file="AT411948B_D0001.tif" />

FIG.1

DVR 0078018

AT 41 1 948 B

The invention relates to a communication method for the transmission of Ethernet messages in a distributed real-time system in which a large number of node computers, for example the four node computers, which each have at least one communication controller, and which are connected via a communication system consisting of one or more communication channels, and where one or more intelligent star couplers are arranged in each communication channel.

The invention further relates to a star controller for a communication system for the transmission of Ethernet messages in a distributed real-time system comprising a large number of node computers, for example four node computers, each with at least one communication controller io, the communication system consisting of one or more communication channels via which the node computers are connected to one another, and one or more intelligent star couplers are arranged in each communication channel.

The following text refers to the literature given below:

[1] US 5694542 issued on Dec. 12, 1989: A loosely coupled distributed computer system with 15 node synchronization for precision in real time.

[2] EP 0 658 257 of December 18, 1996: Communication control unit and method for transmitting messages.

[3] US 5887143 issues on March 23, 1999: Time-Triggered Communication Control Unit and Communication.

[4] AT 407 582 from June 15, 2000: Message distribution unit with integrated Guardian to prevent Babbling Idiot errors.

[5] AT 408 383 of March 15, 2001: Method and communication control unit for multi-master clock synchronization in a distributed real-time computer system.

[6] AT patent application 1723/2001 of 10.10.2000: Method for tolerating slightly-off25 specification errors in a distributed, fault-tolerant real-time system.

[7] AT patent application 429/2001 dated March 19, 2001: Communication method for realizing event channels in a time-controlled communication system.

[8] IEEE Ethernet Standard 802.3 at URL: http://standards.ieee.org

[9] Kopetz, H. (1997). Real-Time Systems, Design Principles for Distributed Embedded Applications; ISBN: 0-7923-9894-7. Boston. Kluwer Academic Publishers

[10] Sharon, O., Spratt, M.,, Ä CSMA / CD compatible MAC for real-time transmission based on varying collision intervals. In: INFCOM '98. Seventh Annual Meeting Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE, Volume: 3,1998, pages 1265-1272 vol. 3.

In the past twenty years, the IEEE Ethernet Standard 802.3 [8] has become so widely accepted that the costs for Ethernet-based communication systems have fallen significantly due to the existing mass market for Ethernet controllers in the personal computer sector. For these cost reasons, Ethernet is also increasingly used in real-time data processing, although the existing Ethernet protocol does not have good real-time properties, such as minimal jitter.

A CSMS / CD system is known from [10] in which messages are divided into low and high priority messages, with the high priority message being given preference in the event of a conflict between two messages.

Furthermore, WO 94/06080 A1, for example, which deals with a communication system in which a control and data field is introduced into the message protocol in such a way that the CRC field is a concatenation of these, can also be regarded as general prior art for the invention both and the internal state of the message.

With the procedure proposed here alone, however, the real-time properties of the Ethernet protocol cannot be significantly improved.

It is an object of the invention to enable the transmission of Ethernet messages with good real-time properties.

This object is achieved with a method mentioned at the outset in that, according to the invention, the communication system differentiates between conventional Ethernet (ET) messages and time-controlled Ethernet (TT) messages and that the TT messages have a constant delay time known a priori between transmitter and receiver transported

AT 41 1 948 B, and that in the event of a time conflict between ET and TT messages, the transport of the conflicting ET message is delayed or aborted in order to be able to transport the TT message with the constant delay time, the constant Delay time is selected so that the output channels of the star coupler can be cleared for the transport of the incoming TT message within this delay time.

Furthermore, the above-mentioned object is achieved with a stem controller mentioned above, which is set up according to the invention to distinguish between conventional Ethernet messages (ET messages) and time-controlled Ethernet messages (TT messages), the TT messages with an a priori to transport known, constant delay time between transmitter and receiver, in the event of a time conflict between ET and TT messages, the transport of the conflicting ET message is delayed or aborted in order to be able to transport the TT message with the constant delay time.

In contrast to the “non pre-emptive solution” disclosed in [10], the present invention does not have to wait for the end of the sending of a message with low priority, but rather the message with lower priority is aborted in order to be able to send the message with high priority (“Pre-emptive). As a result, it is also not necessary to wait for the maximum runtime of messages with low priority, and the constant latency can therefore also be kept short.

By guaranteeing a constant delay time, a high level of control technology accuracy can be achieved. The constant delay time is of particular importance because, as is known from the theory of clock synchronization, the variability of the delay time (that is, the difference between the maximum and the minimum delay time) worsens the accuracy of the clock synchronization. A constant delay time known a priori can be taken into account in the clock synchronization algorithm and therefore has no influence on the accuracy of the clock synchronization. Inaccurate clock synchronization leads to a poor time base, since the granularity of the global time must be greater than the accuracy of the clock synchronization. A coarse granularity of the clocks leads to an imprecise temporal resolution of events. Furthermore, the variability of the delay time also determines the accuracy of the synchronization of distributed actions in a distributed one

Computer system.

The present invention makes it possible to significantly improve the real-time properties of a communication system based on Ethernet messages. This new communication system supports the parallel operation of event-controlled and time-controlled Ethernet messages in a single communication system. In the following, the classic Ethernet messages are referred to as ET (event-triggered) messages and the time-controlled Ethernet messages as TT (time-triggered) messages. The TT messages have a constant delay time and minimal jitter.

The invention provides the following significant economic advantages. The minimal jitter of the TT messages enables the construction of control loops with a high level of control technology

Quality. The TT messages allow a global time to be built with good precision. The global time supports the generation of precise local time stamps in the data acquisition and enables the temporal specification of the interfaces to be improved. In addition, conventional Ethernet controllers can be used without changes.

The method according to the invention can be implemented in a particularly simple manner if the constant

Delay time is selected so that the output channels of the stem coupler can be cleared for the transport of the incoming TT message within this delay time.

In one embodiment, a marked field in the message indicates whether the message is a TT message or an ET message.

Furthermore, an optional time field can be included in a TT message, which specifies the time at which the message was sent.

It is advantageous if it is already determined by a priori planning that a time interval of at least the constant delay time is maintained between the transport of two TT messages.

Furthermore, the above-mentioned object is achieved with a system mentioned at the beginning, which according to the invention is set up for this purpose, between conventional Ethernet messages

AT 41 1 948 B (ET messages) and time-controlled Ethernet messages (TT messages) to differentiate and to transport the TT messages with a constant delay time known a priori between sender and receiver, whereby in the event of a time conflict between ET and TT messages the transport of the conflicting ET message is delayed or aborted in order to be able to transport the TT message with the constant delay time.

As already mentioned above, it is advantageous if the constant delay time is selected so that the output channels of the star coupler can be cleared for the transport of the incoming TT message within this delay time.

In addition, it can be provided that it is indicated in a marked field of the message whether the message is a TT message or an ET message.

Furthermore, a TT message can contain an optional time field that specifies the time the message was sent.

It is advantageous here if it has already been determined by a priori planning that a time interval of at least the constant between the transport of two TT messages

Delay time is observed.

In a specific embodiment of the communication system, it is provided that the star coupler is informed via a configuration message of the times at which incoming messages are TT messages.

The stem coupler distinguishes between TT messages and ET messages and transports the TT messages with a constant delay time known a priori, and in the event of a time conflict between ET and TT messages it breaks the transport of the conflicting ET message in order to be able to transport the TT message with the constant delay time.

Provision is then made for the star coupler to re-send the aborted ET message that was in conflict after the timely transmission of a 25 TT message.

It can also be provided that the star coupler synchronizes its local clock using the time field contained in a TT message.

It is particularly advantageous if the star coupler synchronizes its local clock in a fault-tolerant manner using the time fields contained in several TT messages.

In addition, it can be provided that the star coupler is connected to the replicated star couplers within a cluster of node computers via a dedicated unidirectional channel on which all TT messages that the stem coupler transports are output

In addition, it is then also possible for the stem coupler to check for each TT message on the basis of its local time whether the TT message arrives within an a priori known time window around the time of transmission contained in the message, and the stem coupler if it is premature or delayed arrival of a TT message garbled the message so that all correct recipients recognize the message as incorrect.

The star coupler decodes each TT message and encodes it again on the basis of its local timer.

The star coupler reads one or more selected fields of TT messages and checks during the delay time whether the content of these fields matches known criteria that were communicated a priori to the star coupler via a configuration message. If there is no match, the message is garbled in such a way that all correct recipients recognize the message as incorrect.

In addition, it is also provided according to the invention that the communication controller synchronizes its local clock using the time field contained in a TT message.

The communication controller synchronizes its local clock using the time fields contained in several TT messages in a fault-tolerant manner.

In addition, the communication controller autonomously sends a TT message accepted by an application running on a node computer as soon as the sending time specified in the message in the time field is reached.

It is also provided that the communication controller differentiates between ET and TT messages and the communication controller offers the ET messages to the local application software according to the event semantics, with a new message being placed in a waiting line from which the application software reads in a consuming manner , and

AT 411 948 B the communication controller offers TT messages to the local application software in accordance with the state semantics, a new message replacing the old version of the message and reading by the local application software being non-consuming.

Finally, the communication controller has two or more independent communication channels on which identical copies of a TT message are offered and which considers a communication process to be successful if a valid TT message arrives on time on at least one of these redundant channels.

The invention is explained in more detail below with reference to the drawing. In this shows

1 shows the structure of a distributed computer system with a stem coupler,

2 shows the structure of a distributed computer system with two stem couplers,

3 shows the structure of a standardized normal Ethernet message,

4 shows the structure of a standardized extended Ethernet message,

5 shows the structure of a TT Ethernet message, and

6 shows the bit fields of a TT parameter field of the TT Ethernet message

In the following, an implementation of the new method is shown using a possible example with four node computers that are connected via one or two replicated star couplers.

Fig. 1 shows a distributed computer system with a star coupler. It consists of the four node computers 111, 112, 113 and 114, each of which has a communication controller 121, 122, 123 and 124, each with a bidirectional communication channel connection, and which are connected via a communication system consisting of a communication channel 109. In this communication channel there is an intelligent star coupler 101 for the central control of communication. The star coupler 101 can be initialized and monitored via an optional separate communication channel 141.

Fig. 2 shows a distributed fault tolerant computer system with two stem couplers. It consists of the four node computers 111, 112, 113 and 114, each of which has a communication controller 121, 122, 123 and 124, each with two bidirectional communication channel connections. Each of these communication channel connections is connected to an intelligent stem coupler 101 and 102, which carry out the central control of the communication. The star coupler 101 can send its messages to the star coupler 102 via channel 151 and can be initialized and monitored via the separate communication channel 141. The star coupler 102 can send its messages to the stem coupler 101 via channel 152 and can be initialized and monitored via the separate communication channel 142.

3 shows the structure of a normal Ethernet message standardized in accordance with [8]. After a preamble 301 with a length of 7 bytes is the start delimiter field 302, the target address 303, the sender address 304, the message length or the type of the message 307, the variable data field 310, the optional PAD field 311 with the short Messages are extended, and the frame check sequence 312.

4 shows the structure of an extended Ethernet message standardized in accordance with [8]. In addition to the fields described in FIG. 3, there is an identifier for the extended message in field 305 and a tag type field in field 306. In this Tag-Type field the user can define the priority of a message. According to this invention, the highest priority can be used to identify a TT message. Such an identifier conforms to the Ethernet standard [8]. It should be pointed out that in the Ethernet standard the code assignment for field 305 has not yet been fully exhausted and therefore this field could also be used to identify a TT message.

5 shows the structure of a TT Ethernet message. In addition to the fields described in FIG. 4, a TT parameter field is introduced in field 308 and the optional transmission time of the TT message is specified in field 309. A standardized Ethernet controller available on the market sees user-specific data fields in fields 308 and 309. The TT parameter field 308 contains information relating to the structure and the type of the TT message.

6 shows the content of the bit fields of the TT parameter field 308. If the bit in field 601 (low order bit) is set, this means that the time of transmission in field 309 is contained in the TT message. If the bit in field 602 is set, this means that the message comes from a sender with a precise time and can be used for clock synchronization.

If a node computer, for example 111, wants to send a TT message, it sets the message 5

AT 41 1 948 B field 306 the identifier for a TT message and sends the message. Alternatively, the application software running on a node computer can set bit 601 in the message and write the desired transmission time in field 309 of the message. The start of transmission can then take place autonomously by an Ethernet communication controller expanded according to the invention, precisely s at the specified transmission time 309. If the sender sets bit 602 of the message, the message contains a particularly precise time specification that can be used for the clock synchronization of the other controllers.

The star coupler analyzes an incoming message and uses field 306 to determine whether a TT message or an ET message is arriving. In the case of a TT message, the determines

Star coupler on the basis of field 303 the desired output channel, for example to node 114 in FIG constant delay time Δ for the onward transport of the incoming TT message. The delay time Δ must be selected to be long enough that in any case the output channel can be cleared for the transport of the TT message within this delay time Δ. As part of a priori planning of TT communication, it must be ensured that the interval between successive TT messages is greater than the delay time Δ. In individual cases, the star coupler precisely adheres to this constant delay time Δ between the start of receiving a TT message and the start of sending a TT message.

If the star coupler has canceled the transport of a conflicting ET message, it can send the canceled ET message again after the TT message has been transported in a timely manner. The star coupler can also take on guardian functions, as described in [4], to detect and isolate faulty messages and thus prevent errors from spreading. If a TT message receives the time of transmission in field 309, the star coupler can check whether the message arrives according to [6] within a known tolerance interval around the time of transmission and discard the message if this is not the case. Alternatively, the star coupler can be informed via a configuration message transmitted a priori via channel 141 on which input channels and at which times TT messages are expected. In a fault-tolerant system, this information redundancy prevents a faulty computer node from entering an incorrect transmission time in the message. Since the star coupler encodes the message on the basis of its own oscillator and its own power supply at the output, the forwarding of an SOS error from the transmitter to the receiver is prevented [4]. The stem coupler can initially synchronize its local clock by measuring the beginning of the arrival of a TT message and setting its clock so that it would have assumed the value of the global time 309 contained in the message at this time of arrival [5].

A continuous fault-tolerant clock synchronization can be implemented as follows: the star coupler determines the interval between the time of arrival of the synchronization after40 measured with its local clock and the time of transmission contained in field 309 of the message for each synchronization message marked in field 602 [5]. This interval is a measure of the deviation of the receiver's clock from the sender's clock. If a number of such measurements is available, the correction factor for the clock of the star coupler can be calculated using a known fault-tolerant synchronization algorithm, as described in [9] on p. 61. Such a fault-tolerant synchronization method can also be implemented in the hardware of the star coupler [1], in a fault-tolerant system [2, 3] in which replicated communication channels according to FIG. 2 are available, each star coupler can send all TT messages to the other stem coupler via a dedicated connection channel (151 for star coupler 101 and 152 for stem coupler 102) so that it can also synchronize its clock if no message arrives at its own input. In a fault-tolerant system, the star coupler can check the content of the data field 310 of the message within the delay time Δ in accordance with criteria disclosed to it via a configuration message in order to detect data errors of the transmitter. A message recognized as faulty will not be forwarded by the star coupler.

If a receiving communication controller finds the sending time in field 309 of the incoming TT message, it can initially synchronize its local clock by sending the

AT 41 1 948 Β

Measures the beginning of the arrival of this message, and sets its clock so that it would have assumed the value of the global time 309 contained in the message plus the constant delay interval Δ caused by the star coupler at this point of arrival [5]. A continuous, fault-tolerant clock synchronization can be implemented as follows will: For each synchronization message marked in field 602, the communication controller determines the interval between the time of arrival of the synchronization message, measured with its local clock, and the time of transmission contained in field 309 of the message. It shortens this interval by the known delay interval Δ of the stem coupler. This shortened interval is a measure of the deviation of the receiver's clock from the sender's clock. If a number of such measurements are available, the correction factor for the receiver's clock can be calculated using a known, fault-tolerant synchronization algorithm, as described in [9] on p. 61. Such a fault-tolerant synchronization method can also be implemented in the hardware of the receiving communication controller [1]. If the application software of a computer node, e.g. 111, enters the intended transmission time of the message in the message field 309, a communication controller expanded according to the invention can initiate the transmission autonomously at the precisely correct transmission time [2, 3]. At the interface between the receiving communication controller, eg 121, and the application software, a communication controller expanded according to the invention can offer ET messages and TT messages differently. ET messages usually contain information about events and must be processed according to the event semantics [7]. The event semantics require that incoming messages are cached in a queue and passed exactly once to the user process. TT messages normally contain status data that can be offered in a shared memory in accordance with the status semantics. The arrival of a new TT message overwrites the memory value of an older TT message of the same name. The receiving process does not read status data in a consuming way. In a fault-tolerant system which has several independent communication channels, for example two channels as in FIG. 2, messages are sent in a replicated manner. In such a system, communication is successful if at least one of the replicated copy of the message arrives at the recipient.

Finally, it should be noted that the above-described specific implementation of the integration of time-controlled and event-controlled messages in the Ethernet represents only one of many possible implementation variants of the present invention. For example, it is possible to decide whether a message arriving at the star coupler is a TT message, not from the message content in field 306 or in field 305, but from the time at which a message arrived at the star coupler. In such a case, the star coupler must be informed a priori via a configuration message when and on which channel a TT message is to be expected. The same applies to the communication controller.

It is an essential characteristic of this invention that existing commercially available Ethernet controllers can send and receive time-controlled messages without modification.