Method, apparatus, system, and article of manufacture for reliable low-bandwidth information delivery across mixed-mode unicast and multicast networks

(57) Abrégé/Abstract:

A system and method are provided wherein information is disseminated using a reliable IP multicast network implementation coupled with dynamically assigned proxy nodes serving as zonal aggregation points. These nodes share information with each other over the reliable IP multicast network, augmenting this function with a reliable point-to-point communication infrastructure between proxy nodes in those instances where IP multicast is not available or prone to error.

Canada http://opic.gc.ca Ottawa-Hull K1A 0C9 · http://cipo.gc.ca

O PI C

<img file="CA2726887C_D0001.tif" />

CIPO

OPIC-CIPO 191

CA 2726887 C 2017/03/07 (11)(21) 2 726 887 (13) C (72) lnventeurs(suite)/lnventors(continued): BOGART, THOMAS RICHARD, US; PASSINSKY, MICHAEL A., US; KIRYANOV, YURI, US

-2CA 02726887 2010-12-03 (12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)

<img file="CA2726887C_D0002.tif" />

(10) International Publication Number

WO 2010/002844 A3

PCT (19) World Intellectual Property Organization

International Bureau (43) International Publication Date 7 January 2010 (07.01.2010) (51) International Patent Classiflcation:

G06Q 50/00 (2006.01) H04L 29/02 (2006.01) (21) International Application Number:

PCT/US2009/049181 (22) International Filing Date:

June 2009 (30.06.2009) (25) Filing Language: English (26) Publication Language: English (30) Priority Data:

61/077,413 1 July 2008 (01.07.2008) US

61/101,466 30 September 2008 (30.09.2008) US (71) Applicant (for all designated States except US): TWISTED PAIR SOLUTIONS, INC. [US/US]; 3131 Elliott Avenue, Suite 200, Seattle, WA 98121 (US).

(72) Inventors; and (75) Inventors/Applicants (for US only): CLACK, Derick, J. [ZA/US]; 222 W. Comstock Street, Seattle, WA 98119 (US). VERGATO, Bryan, D. [US/US]; 5230 Sw Forest Street, Seattle, WA 98116 (US). BERTOGLIO, Mark, D. [US/US]; 1623 34th Avenue, Seattle, WA 98122 (US). BOTHA, Shaun [US/US]; 7409 Woodlawn Avenue NE, Seattle, WA 98115 (US). BUCHWALTER, Edward, E. [US/US]; 2233 102nd Place Se, Bellevue, WA 98004 (US). RAVA, Guillaume, F. [FR/US]; 12026 10th Avenue Nw, Seattle, WA 98177 (US). BOGART, Thomas, Richard [US/US]; 203 W. Comstock Street, Apt. 5, Seattle, WA 98119 (US). PASSINSKY, Michael, A. [US/US]; 2516 14th Avenue W„ #308, Seattle, WA 98119 (US). KIRYANOV, Yuri [RU/US]; 9910 Ne 137th Street, C309, Seattle, WA 98034 (US).

(74) Agents: TARLETON, E., Russell et al.; Seed Intellectual Property Law Group PLLC, Suite 5400, 701 Fifth Avenue, Seattle, WA 98104-7064 (US).

(81) Designated States (unless otherwise indicated, for every kind of national protection available): AE, AG, AL, AM, AO, AT, AU, AZ, BA, BB, BG, BH, BR, BW, BY, BZ, CA, CH, CL, CN, CO, CR, CU, CZ, DE, DK, DM, DO, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT, HN, HR, HU, ID, IL, IN, IS, JP, KE, KG, KM, KN, KP, KR, KZ, LA, LC, LK, LR, LS, LT, LU, LY, MA, MD, ME, MG, MK, MN, MW, MX, MY, MZ, NA, NG, NI, NO, NZ, OM, PE, PG, PH, PL, PT, RO, RS, RU, SC, SD, SE, SG, SK, SL, SM, ST, SV, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, ZA, ZM, ZW.

(84) Designated States (unless otherwise indicated, for every kind of regional protection available): ARIPO (BW, GH, GM, KE, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European (AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, ΓΓ, LT, LU, LV, MC, MK, MT, NL, NO, PL, PT, RO, SE, SI, SK, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, ML, MR, NE, SN, TD, TG).

[Continued on next page] (54) Title: METHOD, APPARATUS, SYSTEM, AND ARTICLE OF MANUFACTURE FOR RELIABLE LOW-BANDWIDTH = INFORMATION DELIVERY ACROSS MIXED-MODE UNICAST AND MULTICAST NETWORKS

<img file="CA2726887C_D0003.tif" />

Figure 2 - nodes exchange player information

WO 2010/002844 A3 (57) Abstract: A system and method are provided wherein information is disseminated using a reliable IP multicast network implementation coupled with dynamically assigned proxy nodes serving as zonal aggregation points. These nodes share information with each other over the reliable IP multicast network, augmenting this function with a reliable point-to-point communication infrastructure between proxy nodes in those instances where IP multicast is not available or prone to error.

CA 02726887 2010-12-03 wo 2010/002844 A3 lllllllllllllllllllllllllllllllllllll^

Published:

— with international search report (Art. 21(3)) (88) Date of publication of the international search report:

April 2010 — before the expiration of the time limit for amending the claims and to be republished in the event of receipt of amendments (Rule 48.2(h))

CA 02726887 2016-04-26

METHOD, APPARATUS, SYSTEM, AND ARTICLE OF MANUFACTURE FOR RELIABLE LOW-BANDWIDTH INFORMATION DELIVERY ACROSS MIXED-MODE UNICAST AND MULTICAST NETWORKS

TECHNICAL FIELD

The present disclosure generally relates to computer software and/or hardware for computer and communication systems networking, and more particularly but not exclusively relates to information delivery between devices through a communication network for critical applications.

BACKGROUND INFORMATION

Timely and reliable information dissemination and delivery in high user-capacity environments is a key requirement in the successful completion of a mission or other task and, in life-threatening environments, ensuring the safety of personnel conducting the mission. In these kinds of environments, centralized content delivery systems - typically referred to as servers - are used to distribute information in a real-time or near real-time manner.

This type of centralized architecture, while generally functional, suffers from at least two significant drawbacks; 1) the reliance on a central server or cluster of servers that all endpoints directly or indirectly connect to or otherwise communicate with; and 2) inefficient use of available network bandwidth to deliver the information to the endpoints that require such information.

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181

While there are variations on the theme of centralized servers for example dividing the server cluster into multiple layers and clusters to achieve a notion of “divide and conquer” -- the server-based architecture still generally suffers from the same basic issue: endpoints that require information have some sort of subscription in place that causes the server(s) to forward information to each endpoint as that information becomes available or is modified. This leads to the second major drawback of a server-based architecture where oftentimes scarce network resources are stretched to the point where information simply cannot be reliably delivered or where the time to deliver the information in a reliable manner exceeds the useful age of that information. For example: delivery of crucial mission intelligence to mission coordinators (such as rescue workers or mission commanders) on the ground may be delayed to the point where, by the time that intelligence reaches workers, the information is no longer useful - potentially placing lives and the mission at risk.

There are various methods of synchronizing data between serverless networked nodes, a field of study overlapping peer-to-peer networking and distributed networking. There are typically 2 lines of design: high-bandwidth and low-bandwidth. High-bandwidth peer-to-peer architecture aims at utilizing the most bandwidth possible to achieve fast file transfers and is usually not concerned with network congestions: these designs are at the application layer and they assume appropriate network conditions. On the other hand, low-bandwidth solutions are concerned with and interact at the network layer but their implementation and customization of the layers below the Transport layer on the ISO stack often translate into a reduced flexibility in term of network topology and heterogeneity; for instance wireless mesh networking designs often rely on a modified IP stack implemented at every node.

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181

BRIEF SUMMARY

One embodiment described herein provides: first, its capacity to function in a heterogeneous and dynamic network topology; second, its reliability and scalability throughout low-bandwidth intermittent networks and high-bandwidth networks; and third, its original ad-hoc and multi-hop ad-hoc design.

If we call ’’endpoints” (or generally ’’nodes”) nodes attached to a network by opposition to “players” defined as entities coupled to endpoints to access a network, and if we call “channel” a collection of endpoints coupled via:

- an IP multicast group address and Any Sender Multicast (ASM) ;

- point-to-point unicast connections;

- a heterogeneous combination of the 2 topologies above.

And if we call “player properties” an arbitrary key-value table of arbitrary size that is attached to each player, then we shall summarize this design as illustrated by way of example in Figures 1-2:

One aspect provides a method for communication in a network, the method comprising:

providing a first node having a first player with associated first player properties (Figure 1);

providing a second node having a second player with associated second player properties (Figure 1);

providing a third node having a third player with associated third player properties (Figure 1);

coupling said second node to said first node and coupling said third node to said second node, to enable communication between said first, second, and third nodes via an IP multicast network so as to enable a sum of said first, second, and third player properties to form an information base for each said nodes (Figure 2);

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 if one of said nodes is disconnected from other ones of said coupled nodes, updating said information base of each of said other ones of said nodes to remove the associated player properties of said disconnected node, and updating said information base of said disconnected node to remove the associated player properties of said other ones of said coupled nodes; and if said disconnected node reconnects to a particular one of said coupled nodes, updating said information base of said reconnected node and said particular node and any node coupled thereto to include the associated player properties of said reconnected node.

Another aspect provides a communication system, comprising: a plurality of distributed communication devices adapted to communicate with each other in an IP multicast network without use of a centralized server, each of said devices having respective properties that are communicated to other ones of said devices so that each device is aware of properties of all communicatively coupled devices in said IP multicast network; and an information base for each of said devices, said information base being updated to remove properties of any of said devices that are communicatively decoupled and to add properties of any new ones of said devices that communicatively coupled to said IP multicast network.

Another aspect provides an article of manufacture, comprising: a computer-readable medium having computer-readable instructions that are stored thereon and executable by a processor of one of a plurality of distributed communication devices, to:

communicate with other ones of said devices in an IP multicast network without use of a centralized server, each of said devices having respective properties that are communicated to other ones of said devices so that each device is aware of properties of all communicatively coupled devices in said IP multicast network; and

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 update an information base, said information base being updated to remove properties of any of said devices that are communicatively decoupled and to add properties of any new ones of said devices that communicatively coupled to said IP multicast network.

Still another aspect provides a network computing apparatus, comprising:

one of said plurality of distributed communication devices adapted to communicate with each other in the IP multicast network without use of the centralized server, said device including:

a processor adapted to perform said update of said information base;

a storage unit coupled to said processor and adapted to store said information base that includes properties of said one device and properties of coupled other ones of said devices; and a communication interface coupled to said processor and to said storage unit and coupleable to said IP multicast network, and adapted to send and receive said properties and other communication with other ones of said devices via said IP multicast network

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Figure 1 illustrates example nodes with associated players and player properties according to one embodiment.

Figure 2 illustrates the exchange of player properties/information between nodes.

Figure 3 illustrates example nodes, routing tables, and links according to one embodiment.

Figures 4-12 illustrate example implementation of rules according to various embodiments.

Figures 13-14 illustrate example link costs and implementation thereof according to one embodiment.

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181

Figure 15 is a block diagram showing components of an embodiment of a device adapted to communicate with other devices via an IP multicast network.

DETAILED DESCRIPTION

In the following description, numerous specific details are given to provide a thorough understanding of embodiments. The embodiments can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the embodiments.

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

The headings provided herein are for convenience only and do not interpret the scope or meaning of the embodiments.

The embodiment(s) described herein provide an architecture and mode of operation wherein information is disseminated using a reliable IP multicast network implementation coupled with dynamically assigned proxy nodes serving as zonal aggregation points. These nodes share information with each other over the reliable IP multicast network, augmenting this function with a reliable point-to-point communication infrastructure between proxy nodes in those instances where IP multicast is not available or prone to error.

An embodiment employs a number of conceptual entities to provide the features described herein. These entities include “channels”,

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 “players”, “messages”, “properties”, and “nodes”. A “channel” is a notional grouping of like-minded/like-use entities that have a desire or need to share information with each other. These entities are typically used to describe human end-users in an embodiment but may also refer to non-human entities such as computer devices, communications devices and systems, and so forth. A more general term therefore used to describe such an entity is a “player”. Stated in another way, a “player” exists along with other “players” on a “channel” and uses the “channel” to exchange information with each other.

Such information includes “properties’ describing attributes of individual players such as a human player’s name, title, operational role, etc. or, in the case of non-human players, attributes such as an IP address, CPU configuration, bandwidth availability, etc. In short, the properties can include any information that is pertinent to that player identifying itself on a channel to other players.

In an embodiment, in addition to player-specific properties as described above, properties may be applied to the channel itself - e.g., the properties need not pertain to a specific player but may be used to describe information about a group of players or the channel itself. An example of such a property may be the name of the channel, the role of that channel, and specific messages that should be displayed to human end-users when they join that channel. Similarly, a channel-specific property may also be used in a manner so as to coordinate group access to shared resources - essentially operating as a channel-wide flag or semaphore.

In addition to properties, a channel of one embodiment also serves as a virtual conduit of “messages” between players. In an embodiment, such messages may constitute a simple implementation of textual or other data messaging between players or may be utilized in a more sophisticated manner such as implementing a group-wide signaling protocol for purposes of controlling the operation of players or the devices that they represent.

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181

In an embodiment, all players, properties describing those players and the channel, as well as messages on the channel are sent and received by “nodes”. A “node” is a logical interface to the channel provided by the underlying embodiment. This interface provides the conduit by which a computer application communicates on the channel.

In addition to providing this interface to the channel, a node may optionally function as a convergence point of data where the data needs to cross boundaries between channels - essentially “bridging” information across channels - or where the underlying networking infrastructure between instances of the channel is different. For example: in an embodiment, a node may pass data between instances of different channels (“bridging”) so that data may be shared between those channels; or the node may pass data between instances of the same channel where those instances are implemented on different networking topologies - such as bi-directionally relaying data between IP multicast and IP unicast topologies.

Building on these concepts, an embodiment may then be implemented on a variety of networking topologies, be able to scale to many thousands of nodes, properties, and messages, and represent thousands of individual players.

Integrity Assurance Channels

In an embodiment, the ability to scale to thousands of players and the properties associated with those players while keeping bandwidth utilization to a minimum, implements the creation of a special channel to coordinate system-level data exchange between nodes. Such a special channel - known as an “Integrity Assurance Channel” ( IAC) - is responsible for node-level data exchange to ensure the integrity and consistency of the data held at each node. Such data includes the list of players on the channel, their associated properties, properties associated with the channel itself, messages queued for delivery, and so forth. Such coordination is typically only needed in an

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 environment where IP multicast is used as the underlying network topology but it should be noted that such coordination may also exist across more reliable links such as IP unicast utilizing TCP as a delivery protocol. The messages exchanged on the IAC are specific to the IAC and constitute a protocol implemented only by nodes on that IAC. Furthermore, multiple lACs may exist in an embodiment so as to reduce overhead on a particular IAC relative to the traffic generated on the “normal” channels associated with it. In this regard, multiple “normal” channels may be associated with a particular IAC, thereby consolidating their respective information onto a single IAC rather than each propagating their information separately - and potentially in duplicate. While such association may occur on an IAC-per-channel basis, the method for implementation in an embodiment is for multiple normal channels to be associated with a single IAC. Following this method assures the lowest amount of overall bandwidth consumption for the number of “normal” channels in use.

Security

Security is a component of one embodiment. To this end, an embodiment utilizes a variety of encryption technologies including but not limited to the Advanced Encryption Standard or “AES”. In this embodiment, all messages exchanged over the IP network are assumed to be open to threat from outside entities and are therefore encrypted prior to transmission on the network. Further, and in this particular embodiment, the key used for encrypting data is pre-shared in nature - that is to say that all nodes learn about this key when they are first configured. While this is one particular method of key sharing, other methods exist such as Diffie-Hellman key exchange.

“Exhaustive Routing” Algorithm For Peer-To-Peer Networking

In one embodiment, nodes are connected/coupled to each other in a peer-to-peer network. Each node may have several connections or “links” to other nodes, and nodes not connected by a common link can communicate

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 by passing messages through intermediate nodes in a multihop ad-hoc fashion. The routing algorithm allows nodes to find optimum routes between themselves and another node. Each node has a routing table that lists what link to use to reach a specific other node. The routing table lists every single node in the network, hence the name “exhaustive routing”, and what link to use for that node. The routing table of one embodiment need not show a full route, but only a one-hop route to the next node. Since every node has a full list of all nodes in the network, when the next-hop node receives a message, that node knows where to forward the message next. As shown in Figure 3, nodes are circles, and routing tables are notes attached to nodes through a dashed line.

The numbers at the edges of the links between nodes show how nodes refer to these links. For instance, in Figure 3, A calls its link to B “1”. Link numbers are internal to nodes and not communicated to other nodes: for instance, in the Figure 3, B calls its link to C “2”, but C calls its link to B “1”. These numbers reflect the order they were created on the nodes. Still in Figure 3, if A wants to send a message to C, it sees from its routing table that C is reachable through link 1. Thus A will send a message to B, which sees the message is intended for C. B then looks up its routing table and see C is reachable through link 2. Thus B will forward the message onto its link 2. C then receives the message.

Besides providing routes to nodes, the routing table of one embodiment serves another purpose: it lists all nodes currently connected/coupled to the network, providing nodes with “network-wide presence” information. If a node enters or leaves the network, all other nodes are informed of the entry/departure through a mechanism as will be described below. In a specific embodiment, the exhaustive-routing algorithm can be used exclusively to maintain presence information throughout a network.

In one embodiment, routing tables at nodes throughout the network are kept current with the following messaging system: there are 2 basic messages, HELLO and BYE that are exchanged between nodes to maintain

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 accurate routing tables. HELLOs and BYEs are generated, interpreted, and forwarded from node to node according to the following example 8 rules:

Rule#1: Link creation, HELLO messages

The first rule is that when 2 nodes create a link to each other, they send a HELLO message stating their name. An example implementation of this rule is illustrated in Figure 4.

If a node “knows of’ other nodes, that node mentions them in the HELLO message. A node “knows of’ other nodes by reading its routing table and gathering a list of all node names from all links. When a HELLO message contains more than a node’s own name, such a message is called a “composite HELLO”. Implementation of a composite HELLO message is shown in Figure

5.

A composite HELLO message of one embodiment need not specify how to reach the other nodes, but only their names. If a node appears several times in the routing table, that is not repeated in the HELLO message, as for instance in Figure 5, node Z appears twice in X’s routing table but is provided only once in the HELLO message.

So going back to Figure 3, if a standalone node D with no prior links was to form a link with node C, then D would send “HELLO:D” to C and C would send “HELLO:C,B,A” to D, as seen in Figure 6.

Rule #2: Asynchronicity

The second rule explains that the order of events that lead to a new link between 2 nodes, such as in Figure 6, is irrelevant: the only requirement of one embodiment is that the 2 HELLO messages described above are sent to the newly-connected/coupled neighbor node. Depending on context (e.g., hardware architecture, software architecture and operating system, API used), the creation ofthe link between 2 nodes can be a one-sided operation: the link can be created by one node only. In other cases, in order for

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 an inter-node link to be created several messages are exchanged at the network or transport layer, and both nodes may need to initiate the link. In any case, the connection is first fully established, and then the exchange of HELLO happens.

Rule #3: HELLO forwarding

The third basic rule explains how nodes forward incoming HELLO messages. In the example of Figure 6, D would not have to forward its incoming HELLO message since it has no other link, but C would consider forwarding its incoming HELLO message from D onto its other links. The rule (illustrated by the example implementation in Figure 7) is as follows in one embodiment:

On a given node N that has M links to other neighbor nodes, when a HELLO message comes in from a link L-ι, that HELLO message is first decomposed into a list of “simple HELLOs”. A composite HELLO message “HELLO:X,Y,Z...” can be decomposed into a list of simple HELLOs: { “HELLO:X” , “HELLO:Y” , “HELLO:Z”, ... }. For each of these simple HELLO message, the node N needs to iterate through each of its M links except for link Li , and consider if the simple HELLO message needs to be forwarded onto that link or not.

The criteria for a HELLO about a node X, “HELLO:X”, to be forwarded onto a link L, is the following: considering N’s routing table, considering the entries of that routing table, and that each entry lists what nodes are reachable for a given link, in order to forward “HELLO:X” onto a link L,, the entries in the routing table for any link but L, need to show no instance of X. In other words, the node needs to forward a HELLO about X onto a link L, if and only if all entries in the routing table beside the one for L, do not list X.

Rule #4: Routing table update after HELLO forwarding

The fourth rule states that once the third rule is fulfilled on a given node N, that is to say that N is done with forwarding an incoming HELLO

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 message, to the node N updates its routing table before undertaking any other communication. The order of operation - forwarding first, updating routing table second - is used in one embodiment because the forwarding process (that was shown and described above) reads the routing table.

If 2 HELLO messages are received at once, the first one is processed according to rule 3, then rule 4 is applied; and the second HELLO is processed according to rule 3, then rule 4 is applied. The fourth rule of one embodiment provides that: for a HELLO message received from a link L on a node N that lists P nodes : “HELLO:Xi,X2,X3,...,Xp”, N updates the entry for L in its routing table by adding Xi,X₂,X₃,...,Xp to that entry. Routing table entries are sets of unique node names, so if for instance an entry was to list “A,B,C” and “B,D” is to be added, then the entry would be updated to “A,B,C,D”.

Rule #5: Link destruction, BYE messages

Link destruction is operated by BYE messages in one embodiment. When 2 nodes A and B are connected and A decides to destroy the edge, A can send/announce a BYE message to B as shown by way of example in Figure 8.

This kind of BYE message of one embodiment has no payload (unlike forwarded BYE messages with respect to rule #6, described later below): it means the current edge is going to be disconnected/decoupled.

When B receives the BYE message, B replies with a BYE message as shown by way of example in Figure 9.

And both nodes end the connection (link destruction), as shown by way of example in Figure 10.

At this point, A and B would proceed to BYE forwarding (see, e.g., rule #6).

The kind of exchange described above is an “announced link destruction”, meaning that the nodes inform each other the link destruction before actually taking the link down. An alternative is the “unannounced link

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 destruction” where one of the nodes (or both) takes the link down without sending a BYE message. Unless the link comes back up within a defined period, the nodes proceed to BYE forwarding (see, e.g., rule #6) just as if a BYE was received.

Rule #6: BYE forwarding

Once a link is destroyed (whether announced or unannounced : see, e.g., rule #5), the 2 nodes at each end of the severed link inform their remaining neighbors of the link loss by sending BYE messages that list what nodes used to be visible on the destroyed link. Figure 11 shows an example implementation of this rule, wherein the nodes A and B forward BYE messages that list what nodes used to be visible on the (now-destroyed) link(s).

It is noted that in one embodiment, the original BYE message had no payload, but the forwarded BYE messages have a payload: that of the list of nodes that were reachable through that edge.

Just like for HELLO messages, when a BYE message arrives at a node, the BYE message is decomposed into simple BYE messages, and each one is considered for forwarding on each link. Figure 12 illustrates an example implementation.

The difference with HELLO messages is that the routing table is updated first and forwarding happens second. Updating the routing table includes removing references to the nodes listed in the BYE message for (and only for) the link that BYE message was received on. The forwarding logic is the following in one embodiment: once the incoming composite BYE message is decomposed into a list of simple BYE messages , each simple BYE message is considered for forwarding on each link (besides the one it came on). For a given link, a simple BYE is forwarded if (and only if) the node it refers to is not present (in the routing table) in any other link entry but itself. So for instance, if a node has 3 links L1 L2 and L3, and that node received a BYE about node X on L1, then that node will forward the BYE onto L2 if X is not listed (in the

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 routing table) as a visible node from L1 and L3; and the node will forward the BYE onto L3 if X is not listed (in the routing table) as visible node from L1 and L2.

Rule #7: HELLO and BYE recomposition

As described herein for one embodiment, when forwarding composite HELLO messages or composite BYE message, a node is first decomposes into simple messages (“simple” meaning that the message has a payload of 1 node), and then these simple messages are considered one by one for forwarding onto a given link.

In order to limit traffic according to one embodiment, simple messages can be recomposed (re-assembled) into composite messages if and only if they have the same source. HELLOs and BYEs have a field in their header that defines what node originally sent them. When messages are forwarded, this “source” field is not changed.

Rule #8: History-based message pruning

Another field in both HELLO and BYE message headers is the message counter at the source. This message counter is a roll-over counter that a node increments whenever the node sends a message. The combination of the source field and this counter field forms a message ID that is unique within a certain period of time (determined by the size of the roll-over counter).

In one embodiment, these message IDs are stored in a history log per link on each node.

These history logs are used to filter out messages that appear more than once on a same link on a given node. If a message arriving at a node’s link has a message ID that matches the history log, the message is discarded before any processing.

Both the roll-over counter width and the history log size are adjusted to sensible values to implement a fairly efficient history-based

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 message pruning. The pruning of one embodiment only needs to be Za/rand not perfect since this pruning helps lower traffic in case of topologies with loops, but this pruning is not necessary to the routing algorithm. The routing algorithm is resilient to over-sending. Repeated packets only increase traffic (which may lead to longer transient state), but not overall stability.

The above embodiment(s) of 8 rules describe how routing messages are transmitted and how messages alter nodes’ routing tables. Ultimately, the nodes’ routing tables act as a distributed database that establishes global network presence information: each node knows what other nodes are part of the network and how to reach them. The algorithm according to one embodiment allows a completely determined network presence mechanism. Presence information may be sufficient in some scenarios, but another role of the routing table is to provide optimized data routing between nodes.

Optimized Data routing is possible by keeping track of link cost and forwarding cost. The HELLO message header contains a cost field that is initialized to zero at the source node, and is incremented by a determined amount every time the message goes through a link: link cost, and by another determined amount every time the message is forwarded by a node: forwarding cost. Eventually, when a HELLO message is sourced at a node A and arrives at a node B, its cost field reflects the sum of all these increments. These costs are stored in the routing table, and are used to find optimized root. An example implementation is shown in Figure 13.

In Figure 13, costs are represented with the dollar symbol “$’. In this example, all link costs are set to 1 and there are no forwarding costs. The routing tables reflect costs with $ values in parenthesis. For instance, if A wants to send data to B, its routing table shows it can reach B from its link #1 or its link #2, but on link #1 a cost of 1 is associated B , and on link #2 a cost of 2 is associated to B ; therefore a message to B should be sent onto link #1.

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181

When a node wants to send data to another node it does so by prepending a specific message header to it. The message header contains several fields, most current to all network designs (such as the payload length, a CRC of the payload, etc.), but also a value field that is similar to the time to live (TTL) field of IP packets. The difference with TTL is that the value field is decremented by the various link costs and forwarding costs (the TTL is decremented uniformly at each hop). Just like a TTL, when the value of a message reaches 0 the message is dropped.

When certain routes become too busy or unavailable, high-priority messages can be given higher value fields to give the routing algorithm more options. In Figure 14, node D has a forwarding cost, it could be for instance that A, B, and C are linked by a land network and D is a telecommunication satellite. During normal operation when the land network is available, all data messages between A, B and C would use the land network: the routing tables reflect that optimized routes never go through D. During normal operation, if A wants to send data to B, A needs its message value to be set to 2. However if A loses its connection to C, messages with a value of 2 will not even leave A since there is no available route that has a cost of 2 or less. Messages with a value of 3 would use the satellite connection through D and would resume using C as soon as the land network comes back up.

In an embodiment, nodes may become disconnected from the larger system on an often but random basis and may very quickly reconnect either to the same node or one it was not previously connected to. In such an event, the immediate removal of a disconnected node’s information from all other nodes in the system is not desirable as the resulting resynchronization upon reestablishment of connectivity will result in an inefficient use of networking and CPU resources. To mitigate against this, one embodiment employs timers on both ends of the list connection, starting those timers upon disconnection and canceling them upon reconnection. If reconnection does not occur within the timeframe specified by the timer on each end, the information

CA 02726887 2010-12-03

WO 2010/002844

PCT/US2009/049181 is removed from all nodes (using the system of BYE messages described above). If however, reconnection does occur, the reconnecting nodes simply exchange a signature of the data they each contain (instead of the complete system of HELLO messages described above). This signature is represented as a list of the players represented by each of the two nodes involved in the connection and a hash of the combined properties for each of those players. If the signature received by a node from the other node matches the signature the node itself calculates, then no data exchange is required as the data is deemed to be identical on either side. If the signature does not match - either in one direction or both - the nodes initiate a request/response exchange for a list of players and their properties for which the individual signatures differ (using the complete system of HELLO messages described above). This algorithm ensures efficient use of networking and CPU resources, and dramatically reduces the perception of system thrashing on the part of human end-users.

Figure 15 illustrates an embodiment of a network computing device 500 that can form one of more devices in a node. Examples of the network device 500 can include, but not be limited to, a cellular telephone, laptop computer, mobile wireless communication device (such as a PDA, Blackberry, and the like), desktop computer, server, switch, router, or any other network device having communication capability.

One embodiment of the network device 500 includes one or more processors 502 adapted to perform updates of the information base that stores the properties associated with the network device 500, as well as the properties of other devices 512 as explained above. The processor(s) 502 is also adapted to perform other acts associated with operation of the network device 500.

In an embodiment, the network device 500 includes a computerreadable storage medium 504, such as a memory, adapted to store computerreadable instructions that can be executed by the processor 502. For example, such computer-readable instructions can be in the form of a software program executable by the processor 502 to perform the update of properties,

CA 02726887 2016-04-26 communication of properties and other messages, encrypt/decrypt communicated information, and various other features described herein. The storage medium 502 may store an information base that includes the properties 506, such as the properties described above.

A communication interface 508 is provided to enable communication of the properties and other messages to other devices 512 via a network 510, such as an IP multicast network. In one embodiment, the network interface is adapted to communicate the properties via the integrity assurance channel, which can be different than a normal channel used to communicate messages between the devices 512 via the IP multicast network 510.

A bus 514 couples the various components of the network device 500 to each other.

The following applications and patents are owned by the assignee of the present application: U.S. Patent Nos. 8,340,094 and 9,001,826, entitled METHOD, APPARATUS, SYSTEM, AND ARTICLE OF MANUFACTURE FOR RELIABLE LOW BANDWIDTH INFORMATION DELIVERY ACROSS MIXED-MODE UNICAST AND MULTICAST NETWORKS; U.S. Patent No. 8,121,057, entitled WIDE AREA VOICE ENVIRONMENT MULTI-CHANNEL COMMUNICATIONS SYSTEM AND METHOD; U.S. Patent Publication No. 2008-0240096, entitled METHOD, APPARATUS, SYSTEM, AND ARTICLE OF MANUFACTURE FOR PROVIDING DISTRIBUTED CONVERGENCE NODES IN A COMMUNICATION NETWORK ENVIRONMENT’.

The various embodiments described above can be combined to provide further embodiments. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.

These and other changes can be made to the embodiments in light of the abovedetailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the fuil scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.

CA 02726887 2016-04-26