Method and apparatus for performing a software upgrade of a router while the router is online

CROSS-REFERENCE TO RELATED DOCUMENTS

0001The present application is a continuation application of patent application Ser. No. 09/947,285, which was filed on Sep. 5, 2001 now U.S. Pat. No. 6,535,924, and which is incorporated herein in its entirety by reference.

FIELD OF THE INVENTION

0002The present invention is in the field of data-packet-routing over data-packet-networks, in particularly the hardware and software comprising a data packet router. The invention pertains particularly to methods and apparatus for upgrading software running on a data router while the router remains online and operational.

BACKGROUND OF THE INVENTION

0003At the time of the present patent application demand for increased data capacity and efficiency in Internet traffic continues to increase dramatically as more individuals and businesses increase their use of the Internet. The ever-increasing demand also drives development of equipment for use on the Internet such as data packet routers. A number of enterprises are now developing routers that are capable of faster and higher-capacity handling of data packets routed over the Internet.

0004The Internet, operating globally, comprises components from a wide variety of manufacturers. It is, of course, necessary that such equipment conform to certain hardware, software and connection standards, and operate by certain data transfer standards and protocols. These standards are all well-known to the skilled artisan.

0005As new and better equipment for routing data traffic in the Internet continues to be developed, researchers developing such equipment are endeavoring to include such as fault tolerance, diagnostic capabilities, and redundancy into the equipment, links between instances of routing equipment, and in implementation of routing protocols.

0006One of the important developments in Internet technology as of the time of filing of the present patent application is in development of faster and scalable routers utilizing a distributive-processor concept. The present inventors, for example, are involved in development of what are known in the art as Terabit routers, capable of much higher packet transfer rates than for routers currently available in the art. These routers are scalable to higher and higher overall data-transfer rates and capacity, by allowing upwards of seven thousand interfaces to network traffic.

0007An example of a Terabit router as known to the inventor comprises three types of processor-equipped cards adapted for controlling and routing data. Each card has it's own processor lending to a relatively newer concept of distributive-processor data routers. The three types of cards are control cards, line cards, and fabric cards. Generally speaking, a line card functions as an interface for the internal network of the router and the external network to which the router connects. Data packets entering and exiting the Terabit router typically do so through a line card. Fabric cards are a part of an internal data-routing fabric or network of the data router. Fabric cards act as individual routing points within the data router. Control cards are responsible for controlling both line and fabric cards in terms of their functions including initiating routing protocols, maintenance, special processing, control-message distribution, and so on.

0008In the prior art, most routers use a single processor for the processing of data packets. Some routers use more than one processor, however the number of processors is relatively small compared to a distributive data router as known to the inventor. When a data router of the prior art needs updating with new software such as newer versions of routing protocols, configuration software, or the like, the router being upgraded must be shut down and taken off line in order to perform the upgrade. After the upgrade is complete, the router must be re-booted to again join the network topology with its peers.

0009With efficiency of data routing in mind, it is clear that the loss of a router during upgrade procedures affects network performance through the particular portion of the network occupied by the router. Data must be re-routed through peer routers on to its destination, sometimes taking a much longer or even more constricted path or paths. It is desirable, however, that data packets be routed efficiently through a given network and connected routers at all times. Taking one or more routers down for upgrade can be a major inconvenience. The nature of a distributive router as known to the inventor provides processor architecture that may be treated differently than its single-processor counterparts where upgrading is concerned.

0010What is clearly needed is a method and apparatus for enabling a software upgrade to be applied to a distributive-processor data router without having to take the entire router off-line and without significant performance degradation during upgrade. A system such as this would enable, in most cases, automated upgrade of data routing protocol revisions, configuration changes and the like.

SUMMARY OF THE INVENTION

0011In a preferred embodiment of the present invention a method for performing an in-service software upgrade to a data router comprising steps of (a) providing a source node hosting an upgrade software package; (b) causing an upgrade command to be executed; (c) establishing, as a result of the upgrade command, a network session between the data router and the source node; (d) receiving the upgrade software at the router; (e) copying and distributing, within the router, the upgrade software to designated components slated for upgrade; (f) designating one or more components in the router as backups for each component to be upgraded; (g) backing up services and software running on each target element to be upgraded to designated backup element or elements while upgrade proceeds for the target element; (h) causing a switchover at the target element to the new software; and (i) repeating steps (g) and (h) until all the components slated for upgrade are upgraded.

0012In some embodiments the data-packet-network is the Internet network. Also in some embodiments, in step (a), the source location node is an FTP server and the upgrade software package is stored in memory of the server. In other cases the source location node may be a PCMCIA flash card installed in a network-connected computer, the upgrade software package residing on the card. In some embodiments the source location node may be a PCMCIA flash system residing within the router, the upgrade software package residing in the flash system.

0013In some cases execution of the upgrade command is manual and initiated by a user, while in other cases execution of the upgrade command is automated and triggered at a pre-set time. In some cases data network session is conducted over an Ethernet network between the source location node and the receiving component of the router. The location node may be an FTP server and the receiving router component may be a primary control card. In other cases the source location node is a personal computer. In some cases the data network session is conducted over the internet network between source location node and the receiving component for the router.

0014In some embodiments receiving the upgrade software comes after access and request by the router receiving component, while in other embodiments receiving the upgrade software comes after access and request sent to the router receiving component. Designated elements for upgrade may include control cards, line cards, or fabric cards.

0015In some cases the switchover command is automatic and executed after a certain period of time, while in other cases the switchover command is pre-configured to execute at a future designated time, the switchover managed by the primary receiving component of the router. In some cases notification of upgrade status is conducted in a separate and future data session. In some embodiments backup designations are made according to an algorithm for maximizing performance during the upgrade process.

0016In another aspect of the invention a system for upgrading a data router while the router is forwarding data packets over a data-packet-network is provided, comprising a node having access to the router by data link, a software upgrade package stored in memory residing in or connected to the node, a component residing in the router having access to the node over the data link, and a user station having access to the router for displaying upgrade status and results. The system is characterized in that at a specified time, the router component is caused to have access to the software upgrade package, receives and makes copies of image portions of the package, distributes the image portions to router elements within the router identified as targeted for upgrade, backs up functions of targeted elements to designated backup elements, and wherein the targeted elements each install their received image portion, reboot using the new image portion, and report back to the router component, which in turn reports status to the user station.

0017In some embodiments elements are upgraded in groups. Also in some embodiments backups are designated according to an algorithm for maximizing performance during the upgrade process. The data packet network may be the Internet network.

0018In some cases the node having access to the data router is an FTP server, and in some cases may be a personal computer. The data link may be an Ethernet network link.

0019In some cases the memory hosting the software upgrade package is PCMCIA flash memory, and in some the element residing in the router is a primary control card. In other cases the element residing in the router may be a line card. The user station can be a personal computer. In some embodiments the upgrade package includes image portions for control cards, line cards, and fabric cards.

0020In another aspect of the invention a router upgrade application for upgrading at least one targeted computerized element of a router is provided, comprising at least one runtime image, the image including parameters for element boot and operation once booted, an executable command for initiating the upgrade application including distribution and install, and an executable command for applying the at least one runtime image including boot directory reset and reboot instructions. The application is characterized in that the upgrade application functions automatically after a pre-determined or user selected time in cooperation with a primary router element to effect upgrade and reboot to all of the designated target elements of the router.

0021In some embodiments of the router upgrade application the computerized element comprises one of a control card, a line card, or a fabric card. In preferred embodiments the upgrade occurs while the router is forwarding data over a data-packet-network, and the data-packet-network may be the Internet network.

0022In some embodiments the initiation command is automatically executed according to a pre-determined time, and in others the apply command is automatically executed according to a pre-determined time.

0023There may be multiple backup router elements to assume the operational responsibilities of a target element being upgraded, and there may be a backup router element to assume the operational responsibilities of multiple target elements being upgraded simultaneously. In some cases targeted elements include cards of different types, and elements are upgraded in groups by type.

0024In embodiments of the invention described in enabling detail below, for the first time a system and method is provided wherein computerized machines, such as routers, may be upgraded while they continue to perform their on-line functions.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

0025<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram illustrating an upgrade of a data router according to prior art.

0026<figref idref="DRAWINGS">FIG. 2</figref> is a block diagram illustrating an upgrade of a distributed-processor router according to an embodiment of the present invention.

0027<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram illustrating an upgrade distribution process within the router of FIG. <b>2</b>.

0028<figref idref="DRAWINGS">FIG. 4</figref> is a process flow diagram illustrating steps for initiating and completing a software upgrade according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

0029<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram illustrating an upgrade process for a data router <b>100</b> according to prior art. Router <b>100</b> represents a prior-art data router as was described with reference to the background section of this specification. Router <b>100</b> comprises a computer processing unit (CPU) <b>103</b>, a data queue <b>102</b> for temporarily storing packets before they are forwarded, and at least one ingress/egress interface <b>101</b>.

0030Interfaces <b>101</b> illustrated within router <b>100</b> are labeled one as an ingress interface and the other an egress interface for illustrative purpose only. In actual practice, each interface <b>101</b> handles both incoming and outgoing data traffic, although ingress and egress are treated differently in the router. CPU <b>103</b> processes data packets for forwarding as is known in the art.

0031Data packets arriving through ingress of router <b>100</b> are placed in queue <b>102</b>. CPU <b>103</b> performs the required data processing associated with those packets while they are in queue. After processing is complete data packets are downloaded from queue <b>102</b> and sent out through egress of router <b>100</b>. CPU <b>103</b> is also responsible for managing routing protocols, error messaging, and any special processing.

0032A user <b>104</b> (PC icon) is illustrated in this prior-art example as having administrative control over router <b>100</b> via a network control line, logically illustrated herein as three physical paths. User <b>104</b> will typically use a powerful computer in order to upgrade router <b>100</b> with new software periodically or whenever required. An instance of software (SW) illustrated on a graphical user interface (GUI) of user-PC <b>104</b> in this example represents upgrade software that would be used to upgrade router <b>100</b> in this example.

0033Because router <b>100</b> has a single CPU <b>103</b> for processing all data packets that are forwarded therethrough, user <b>104</b> must first shut down router <b>100</b> before performing a major software upgrade. This is illustrated by physical portion of the described network control line labeled Shut Down. The above-described label represents a first command sent to CPU <b>103</b> from user <b>104</b>.

0034At this point router <b>100</b> is off-line and ready to accept an upgrade. Typically interface protocols in the network will notice the absence of router <b>100</b> in the topology, and compensate by upgrading routing tables and the like. While router <b>100</b> is down, user <b>104</b> initiates and installs the upgrade represented by the path labeled Upgrade. After the software upgrade is successfully installed within router <b>100</b>, user <b>104</b> executes a reboot command, illustrated herein by the path labeled Reboot.

0035Router <b>100</b> must remain off-line during the entire upgrade process. In a case where router <b>100</b> might have more than one CPU <b>103</b> sharing processing responsibility, a significant portion of work contributed by router <b>100</b> to a network is still lost temporarily due to the upgrade process.

0036<figref idref="DRAWINGS">FIG. 2</figref> is a block diagram illustrating an upgrade of a distributed-processor router <b>200</b> according to an embodiment of the present invention. Router <b>200</b> represents a state-of-the-art data router as known to the inventor, wherein processing and data-handling responsibility is distributed in the form of computerized router cards. As was described in the background section of this specification, the computerized cards utilized within router <b>200</b> include three categories or types of cards in this example, these being control cards, fabric cards, and line cards. The line cards are illustrated in this embodiment as element # <b>201</b>. Line cards (LC) <b>201</b> comprise the interfacing components between the external network to which router <b>200</b> connects and an internal network for routing data within the router itself.

0037Router <b>200</b> has, in this case, <b>2</b> Ingress/Egress interfaces represented herein by horizontally placed double-arrows associated with either of the two illustrated groups of line cards <b>201</b>. It is noted here that each line card may have a plurality of actual ingress/egress ports. In this embodiment there are eight line cards <b>201</b>, however there may be many more or fewer line cards <b>201</b> installed in router <b>200</b> without departing the spirit of the present invention. It is noted herein also, that the arrangement and groupings of line cards <b>201</b> in this embodiment is logical only does not represent actual implementation of line cards <b>201</b> within router <b>200</b>.

0038Router <b>200</b> further comprises a plurality of fabric cards (FC) <b>202</b> implemented therein and adapted, through interconnected paths, as an internal data network within router <b>200</b>. Each one of fabric cards <b>202</b> comprises a node or hop within the internal fabric of router <b>200</b>. In this example there are <b>16</b> fabric cards illustrated, however there may be many more or fewer fabric cards <b>202</b> without departing from the spirit and scope of the present invention. It is noted herein that the arrangement and groupings of fabric cards <b>202</b> in this example is logical only and does not represent actual physical implementation.

0039Router <b>200</b> further has a plurality of control cards (CC) <b>203</b> implemented therein and adapted to overall control of the router and its elements. Control cards <b>203</b> support, for example, routing protocol software used by router <b>200</b>, manage error and control messaging, perform special processing, and so on. It is noted herein, that the particular arrangement and groupings of control cards <b>203</b> in this example is logical only and does not represent actual physical implementation.

0040Line cards <b>201</b>, fabric cards <b>202</b> and control cards <b>203</b> makeup the internal processing power of router <b>200</b> implemented in a distributed fashion. Because of this distributed architecture wherein all of these cards are interconnected, failure of some of these described processing components can occur without seriously affecting the performance of router <b>200</b> in the routing of data packets. Line cards <b>201</b> comprise the external interfaces of router <b>200</b>, fabric cards <b>202</b> comprise the internal network of router <b>200</b> and control cards <b>203</b> comprise the controlling components of router <b>200</b>.

0041As previously described above, each card within router <b>200</b> has an onboard processor. Each card also has a queue system and a plurality of ingress/egress ports. Therefore, upgrade procedures to router <b>200</b> according to embodiments of the invention may take the form of various configurations. For example, an upgrade procedure may target a single, a number of, or all of the cards within router <b>200</b> that are of a same type. In one embodiment, an upgrade procedure may target specific ones of all of the types of cards within router <b>200</b>. In still another embodiment, an upgrade procedure may target all of the cards operational within router <b>200</b>.

0042The upgrade procedure of the present invention provides for designating backup cards to handle the duties of targeted cards while the targeted cards are being upgraded. Within the illustrated group of line cards <b>201</b> in router <b>200</b>, there are two line cards represented by a dotted border and given the element numbers <b>212</b> and <b>213</b> respectively. For the purpose of discussion, one of these cards can be designated as a backup card while the other is designated as a target card for upgrade. Within the illustrated group of control cards <b>203</b> in router <b>200</b>, there are three control cards represented by a dotted border and given element numbers <b>208</b>,<b>209</b>, and <b>211</b> respectively. For the purpose of discussion, one of these cards, card <b>208</b> in this example, is designated as a global master control card (GMCC). Control card <b>209</b> may be a target card for upgrade, and control card <b>211</b> may be a designated backup card.

0043It should be noted the there doesn't need to be a 1 to 1 relationship between target cards and backup cards. It is essentially running services that are backed up. These services may run on any card or on multiple cards. For example, all target services running on card <b>209</b> may be distributed to card <b>211</b> and <b>208</b>.

0044The relationship between designations as target, backup, and GMCC are described further below.

0045A file transfer protocol node (FTP) <b>204</b>, illustrated herein as a PC icon, is provided in this embodiment as a network source for software required to upgrade one or more of the components of router <b>200</b>. The upgrade software is illustrated in this embodiment as SW <b>210</b> residing on or accessible to node <b>204</b>. Representation of SW <b>210</b> on a GUI of node <b>204</b> is logical only for the purpose of illustration. In actual practice, FTP <b>204</b> may not have a GUI interface or display. A user <b>205</b>, illustrated herein as a PC icon, has a connection to GMCC <b>208</b>, which may be accomplished in any one of several ways. For example, this connection may be connection to Ethernet link <b>206</b> via alternative path <b>214</b>, or directly by an alternative path <b>215</b>. The path of connection is not limiting to embodiments of the invention.

0046The router <b>200</b> has a network connection to FTP node <b>204</b> from which it requests and receives the software package. In one embodiment of the invention this connection may be an Internet connection. In other embodiments any data packet network connection WAN or LAN is applicable. FTP node <b>204</b> is assumed in this example to be a node maintained on the Internet network or other applicable WAN. User <b>205</b> may access the primary control card <b>208</b> for the purpose of receiving upgrade status reports, error notifications, and so on. In one embodiment, user <b>205</b> may manually intervene in the upgrade process directly to GMCC <b>208</b>,

0047FTP node <b>204</b> has a dedicated Ethernet network connection <b>206</b> to GMCC card <b>208</b>. Network connection <b>206</b> in one embodiment of the invention is a high-speed network link capable of over 100 Mbit/sec data transfer rate. In this type of upgrade, one or more cards, in this case, card <b>208</b> within router <b>200</b>, is configured to perform a network boot operation wherein a connection to FTP node <b>204</b> is established for the purpose of downloading SW <b>210</b> to card <b>208</b>. This particular method of getting SW <b>210</b> to router <b>200</b> is not required in order to practice the present invention. In one embodiment, SW <b>210</b> may be distributed to card <b>208</b> from FTP node <b>204</b> via normal Internet path through one of line cards <b>201</b>. In this case, SW <b>210</b> would have to be distributed through fabric <b>202</b> to card <b>208</b>. Using a separate Ethernet connection is simply a convenience. In another embodiment, SW <b>210</b> may be flashed in from a PCMCIA card on a network-connected computer. There are many possibilities.

0048SW <b>210</b> comprises all of the software images and commands required to facilitate upgrade of each type of card within router <b>200</b>. For example, control cards <b>203</b>, line cards <b>201</b>, and fabric cards <b>202</b> boot from and operate according to separate images. These images are termed runtime images by the inventor using familiar object-modeling and programming language that is known in the art. The images are built specifically for the type of card that will boot and operate from them.

0000Each card has a memory for receiving and storing its runtime image. In one preferred embodiment the memory is flashable.

0049Referring again to <figref idref="DRAWINGS">FIG. 2</figref>, in this example, an upgrade is in process from FTP node <b>204</b> through Ethernet link <b>206</b> and GMCC <b>208</b>. GMCC <b>208</b> receives all of the required software images for cards targeted for upgrade. In this example, card <b>209</b> is currently being upgraded and card <b>211</b> is the designated backup card for card <b>209</b> while the upgrade process executes. As a designated backup, card <b>211</b> assumes the operational duties of card <b>209</b> temporarily while card <b>209</b> is being upgraded. If more than one, or all of control cards <b>203</b> are to be upgraded in a single session, then designation for backup card may be rotated from card to card as required. In the case of all of control cards <b>203</b> requiring upgrade in a same session, then GMCC card <b>208</b> will be the last card to reset to the new image.

0050In this example, line cards <b>212</b> and <b>213</b> are currently involved in the upgrade process. Assuming that card <b>213</b> is the target card, then line card <b>212</b> will be a backup. Unlike control cards, backing up a line card is somewhat more involved because it involves re-routing incoming data that would normally use line card <b>213</b>, to a new line interface or that of card <b>212</b>. Line card <b>213</b> cannot send or receive data while its upgrade is being performed.

0051When new versions of an image controlling how the card boots and is configured are part of the upgrade process, the cards will have to perform a reboot to use the new image. A reboot is always required to utilize a new software image which may contain many software enhancements. It is the fact that existing operation of the router does not have to be affected while the system is transitioning to the new enhancements. Once the system has been upgraded, the new enhancements may be turned on and utilized.

0052One with skill in the art will recognize that an upgrade process as described in this specification is performed systematically when upgrading many cards. For example, no more than one or a few cards will be down for any period of time. Furthermore, the upgrade images are built so as to support the last version of the image such that a newly rebooted card may cooperate operationally with a card not yet upgraded. In this way, router <b>200</b> may continue to process and forward data packets as the upgrade procedure is running.

0053This example illustrates upgrade of line and control cards. In the case of fabric cards <b>202</b>, there is a built-in redundancy within the fabric such that when one fabric card is down for upgrade, data destined for that card is re-routed using a next-best shortest path through the fabric that does not include the target fabric card. It is noted herein that the time period that any card is down during an upgrade is quantified by the time it takes to shut down and reboot each card and is relatively small in number of cycles. More detail regarding the upgrade process is provided below.

0054<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram illustrating an upgrade distribution process for router <b>200</b> of FIG. <b>2</b>. In this example, line cards <b>201</b> and control cards <b>203</b> are illustrated in expanded detail for the purpose of clarity in description of the invention. A GMCC <b>305</b> is provided in this example as an interface to the source of the software upgrade. As was described with reference to <figref idref="DRAWINGS">FIG. 2</figref> above, the external link between GMCC <b>305</b> and a software source may be an Ethernet network connection. Interaction between GMCC <b>305</b> and the software source is represented herein by block arrows, one labeled Source, which represents input from a software source and one labeled Status, which represents status data reported back to a user during and after the upgrade process. In the case of an FTP server-sourced upgrade associated with a network boot, status information concerning the progress of an upgrade, such as error reports and so on, can be accessed by a user through the user's connection to the GMCC. In another embodiment status reports may be given directly to a user if there is a dedicated network connection available established for the purpose.

0055In this example, line cards <b>201</b> are illustrated with a plurality of ingress/egress ports <b>302</b>. In this example there are eight ports per card, however there may be more or fewer without departing from the spirit and scope of the invention. One of the illustrated line cards is labeled LC-<b>1</b> and will hereinafter at times be referred to by this label. The other illustrated card is labeled LC-N and will hereinafter at times be referred to by this label. LC-<b>1</b> is designated in this example as a backup line card and is so labeled. LC-N is designated in this example as a target card for upgrade as is so labeled. The protocol of 1-N is used to indicate that there may be essentially any number of line cards.

0056Control cards <b>203</b> are illustrated herein with a plurality of ingress/egress ports <b>304</b> similarly to that described with line cards <b>201</b>. Also similar to the above description there is a designated backup control card and a target control card as is so labeled. Backup card <b>203</b> is labeled CC-N and will hereinafter at times be referred to by this label. Target card <b>203</b> is labeled CC-<b>1</b> and will hereinafter at times be referred to by this label. GMCC <b>305</b> functions as the primary interfacing card between the external software source and other components of router <b>200</b>.

0057Each illustrated card in this example has a flashable memory for storing software images. LC-N has a flash memory <b>306</b> provided therein and LC-<b>1</b> has a flash memory <b>307</b> provided therein. Similarly, CC-N and CC-<b>1</b> have flash memories <b>303</b> and <b>301</b> installed respectively. Each flash memory is capable of storing at least 2 complete software images. It is important to note that an existing SW image is not overwritten or erased during the upgrade process. An image table (not shown) is provided within each flash memory of each card. The table contains the image name and current state of each image on a card.

0058In this example, operational data stored on target cards (LC-N, CC-<b>1</b>) is copied to designated backups (LC-<b>1</b>, CC-N) before each target card is taken offline for upgrade assuming, the nature of the upgrade requires re-boot in each case. For example, CC-<b>1</b> contains current runtime configuration data (CFG-X) that is specific to that card. CFG-X contains all of the pertinent data and component identifications of the components under charge of CC-<b>1</b>. One of these variables is the identification of line cards under control of CC-<b>1</b>. Identification of specific fabric cards under control of CC-<b>1</b> is also a part of CFG-X. CFG-X resides or may reside in flash memory or runtime memory or both in the target card. CFG-X has enough information to allow the backup of CC-<b>1</b> (CC-N) to function on behalf of CC-<b>1</b> when it is offline.

0059GMCC <b>305</b> manages designating a backup to function on behalf of a target card and manages data-transfer operations from the target to the backup before CC-<b>1</b> is taken offline to reboot. In this example, CC-N is illustrated as having it's own configuration data. (CFG) plus that of CC-<b>1</b> (CFG-X). CFG-X in CC-<b>1</b> is illustrated as a dotted rectangle indicating a post transfer state. CGF-X data destined for CC-N is passed through an egress port of CC-<b>1</b> into the router fabric <b>202</b> as is illustrated by directional arrow labeled To Backup. After traveling through fabric <b>202</b>, the data arrives at CC-N from fabric <b>202</b> as illustrated by a directional arrow labeled From Target and is written into memory. In this way CC-N is enabled to perform it's own work plus the original work performed by CC-<b>1</b>. CC-N, in this example has the functional responsibility of 2 cards.

0060In LC-<b>1</b>, the current configuration of CFG-Y is received and stored in memory. LC-N (target card) exhibits a dotted version of CFG-Y in order to illustrate the source. Exact assignments of target cards and dedicated backup cards depends on implementation and enterprise rules. GMCC <b>305</b> distributes the appropriate SW image to each target card. The upgrade software distribution path to both CC and LC type cards is illustrated herein as a plurality of rounded rectangle labeled UPG.

0061GMCC <b>305</b> distributes UPG to target cards through fabric <b>202</b> as illustrated herein by a directional arrow labeled To Targets, after designated backups are in place and operational. The UPG data is distributed to targets through fabric <b>202</b>, more specifically, along paths of fabric cards to each destination. In this case the UPG destinations are CC-<b>1</b> and LC-N. It will be appreciated that the UPG image for LC-N will be different than the UPG image for CC-<b>1</b> because they are markedly different types of cards having different functions to perform within router <b>200</b>.

0062In fabric <b>202</b>, each UPG designation represents a hop or FC fabric card in a path of distribution. In this example, UPG image for CC-<b>1</b> took only one hop through fabric <b>202</b> from GMCC <b>305</b>. UPG image for LC-N takes <b>4</b> hops through fabric <b>202</b>. If any UPG images are destined to fabric cards, then each card will write its own image data to onboard flash and forward that data for other cards.

0063Also present in each flash memory of each upgradeable card is a boot loader and a boot pointer. These components are known in the art and cooperate with the previously-described image table present on each card to designate which image will be used to reboot a card after upgrade and reset to which image will be used to next reboot the card. Only after successful booting using the new image, will an older version be erased from memory. This insures that if an upgraded image is corrupted or otherwise fails, the older image can still be used to boot the card. During the entire upgrade process, messages and responses or acknowledgements are propagated between GMCC <b>305</b> and all target cards for upgrade. Any critical error messages that would force a re-try are passed to an initiating user as indicated by the block arrow labeled Status.

0064It will be apparent to one with skill in the art that GMCC <b>305</b> may itself be upgraded to use a new image and will have to have a backup card to perform it's functions while it is offline. Similarly, other designated backups will have to be upgraded in a process where all or a significant number of cards need the upgrade. The multi-card upgrade process is configured to rotate backup designations as required.

0065In a preferred embodiment the GMCC always has a backup regardless whether it is a target card for upgrade. This ensures that the upgrade task/process itself is fault tolerant in the event that something may happen to the GMCC while the router is being upgraded. The backup GMCC will resume upgrade responsibilities in the event of a GMCC failure.

0066In a case where all cards of a same type in a shelf are to be upgraded, eventually upgrade cards already using the new image will serve as backups. The process is sequential and identifies each card by ID and destination address including slot number in the shelf supporting the cards. Any logical order may be followed. It is noted herein that designating and establishing backup cards to assume responsibilities of a card being upgraded in not specifically required in order to practice the present invention. Router <b>200</b> may, as previously described, lose function of one or even a few cards while still online and forwarding data with performance degradation limited to the data lost as a result of specific card downtime. However, designating and implementing backup cards provides a mechanism to maintain optimum network performance during the upgrade process.

0067In a preferred embodiment, any upgrade requiring reboot to a new version of SW would be performed in one session for an entire shelf of control cards, fabric cards, or line cards. This type of upgrade is termed a router-level upgrade. In one embodiment, it is also possible to perform a router-level upgrade on all of the cards contained in and operational in the router. However, some logic may be applied such as first upgrading all of the control cards, then all of the line cards, and then all of the fabric cards in a single session. The upgrade process in a preferred embodiment is entirely automatic from the point of issuing an upgrade command. The re-boot process is, in preferred embodiments, a timed function wherein the command is executed as part of the runtime image application command structure.

0068<figref idref="DRAWINGS">FIG. 4</figref> is a process flow diagram illustrating steps for initiating and completing a software upgrade according to an embodiment of the present invention. At step <b>401</b>, a user prepares an upgrade session. This pre-session step may include building the runtime images, configuring which cards in a router will be upgraded and so on. A user may set-up a remote FTP server with the upgrade package or a user may set up a PCMCIA Flash delivery, for example.

0069At step <b>402</b>, an upgrade command is executed. This step includes manual execution such as by user input, or automated and timed execution such as is the case of a pre-configured network boot. In a pre-configured network boot, a designated GMCC card analogous to card <b>305</b> described with reference to <figref idref="DRAWINGS">FIG. 3</figref> above would, at a pre-designated time, establish a link to a remote image source such as an FTP server. This step is illustrated herein as step <b>403</b>.

0070In one embodiment, step <b>403</b> is a manual step initiated by a user. A remote session to upgrade a router may be conducted over an Ethernet network, or other suitable network connection, such as the Internet. In one embodiment, router upgrade sessions can be conducted over a LAN in-house. In still another embodiment, an upgrade package resides within the router itself in a common Flash memory file system. In yet another embodiment an upgrade system may work over an Internet connection from a remote location.

0071At step <b>404</b>, a designated GMCC at the system to be upgraded receives image data comprising the upgrade runtime images to be installed. At this step, for a scalable data router, images for all three types of cards (control, line and fabric) may be received. At step <b>405</b>, the GMCC writes the image data to onboard memory and begins propagating the appropriate image data to destination components (typically cards) within the router that will be upgraded. Part of this process is designating backup status for upgrade targets in step <b>405</b>. A backup sequence in one embodiment is configured into the software upgrade package and is controlled by the GMCC. For example, if all of the control cards within a router are to be upgraded with a new image then the GMCC will establish the sequence for designated backups. New backup assignments are executed as individual cards complete the upgrade and reboot. Any logical order may be employed. The exact order will depend on the scope of the upgrade session.

0072It should be remembered that there need not be a one-to-one relationship between backup elements and elements to be upgraded. Further, some elements serving as backups, may subsequently be backed up and upgraded as well. The key to in-service upgrading is that functions of elements to be upgraded must be performed by alternate elements in the system, while the primary elements are down during upgrade.

0073At step <b>406</b>, the appropriate image data is sent to cards (components) targeted for upgrade. It is noted herein that fabric cards may well have maximum receive and send sizes for data packets that are smaller than a complete upgrade image. Therefore, the image data propagated to target cards may be first fragmented into smaller packets and checked for errors before send. Destination cards receiving the image data perform a checksum operation on all of the fragments and the complete image after assembly.

0074Also part of the process in a preferred embodiment is that all target cards receiving data send acknowledgements back to the designated GMCC as to status and error. A reasonable timeout period for each target card may be initiated by the sending GMCC to wait for a response. If no response is received from a particular card then a retry may be initiated. Cards that report back that the upgrade was successful are formally upgraded and do not have to be included in a reattempt to upgrade any cards wherein the initial attempt failed for some reason.

0075At step <b>407</b>, the upgrade process is initiated. At step <b>408</b> an element is backed up and then upgraded. This may be the first element if the upgrade process is first initiated, or any other element in the process as upgrade proceeds.

0076Once a complete image is loaded onto a target card, that card is ready to reboot if necessary to begin using the new image. At step <b>409</b>, it is determined if there are other elements in the schedule to upgrade. If so, another element or group of elements is backed up and upgraded. If not, control proceeds to step <b>410</b> and the upgrade is complete.

0077It is noted that throughout the process, there is a convention for reporting any error states that may occur. This is illustrated by dotted directional arrows emanating from some of the illustrated steps and connecting with a logical line labeled Status Notification drawn from step <b>409</b> to step <b>401</b>.

0078During a multiple card upgrade, the GMCC keeps track of all of the completed, current, and pending states so that if one or more cards do fail to accept the upgrade, a retry may be automatically initiated for just those target cards which failed initially.

0079The following CLI (control line interface)commands can be used to upgrade the different components. <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0080">upgrade [bootldr|image][network<IPADDRESS><filepath>][cc|lc48|fc] [router|<shelfid>[<slotid>]][apply]</li></ul></li></ul>

0081This command upgrades the bootldr or image on the card type specified with the provided arguments. The IP Address of where the image is present is specified when the image has to be retrieved over the network. This particular command configuration is valid only on control card upgrades. Filepath is the location of the software package. It may be on an FTP server or PCMCIA flash card.

0082The [apply] parameter specifies that the card automatically perform all of the more granular operational steps including but not limited to upgrading the boot pointer, converting a configuration database, performing a reboot using the new image, overwriting the old image after successful boot, and backup of running services and configurations, and so on. <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0083">boot network<cc_nodeid>[config parameters]</li></ul></li></ul>

0084This command causes a network boot to happen on the next reboot of the target control card specified by nodeid parameter.

0085It will be apparent to one with skill in the art that many sub-steps may be added to the process taught above without departing from the spirit and scope of the present invention. For example, as image packets are received at a card targeted for upgrade, they are checked for errors and an acknowledgement, either positive or negative, is sent back to the GMCC each time. Also, once a full image is received and assembled, a final checksum of the entire image is performed and an acknowledgement to this effect is sent back to the GMCC. The inventor has chosen not to illustrated all of these more granular steps in order to more simply explain the upgrade process.

0086In one embodiment of the present invention, a mechanism is provided to enable a user to pre-specify a point in time for a router to perform an automated upgrade after a download session to distribute the upgrade images. Once the images are downloaded in to each card, the upgrade time is stored in configuration. A primary upgrade task manager will perform the rolling card resets when the specified time occurs.

0087In one embodiment of the invention, a network timestamp protocol (NTP) client is provided on all cards and on the designated GMCC. Using a timer function, each target card will delay reboot until the end of a self-starting time period. This prevents any cards from resetting before the apply command has executed.

0088In still another embodiment, cards that are introduced into a router as new cards can undergo automatic upgrade to the SW version currently in use within the router. In this implementation, special CLI commands are provided for a user to configure and specify a SW image version and its source location, perhaps at a remote server. When the upgrade task manager on a designated GMCC receives boot confirmation from a new card in the router, it compares its image version against the one stored in configuration for all of the cards. If there is a mismatch, an upgrade will be initiated to the card. It is noted herein that there should be no more than one version delta between the 2 images.

0089An image copy command may also be provided in one embodiment for enabling a user to overwrite the elder of the 2 images stored in flash on any card. For each image type (bootldr, runtime, fpga) there are two images stored in flash. These are designated current and other. The CLI command will be provided to allow the user to copy an image from the “current” to “other”. This is useful once a new image has been qualified and the “other” can then be overwritten.

0090The method and apparatus of the present invention may be practiced over the Internet or other networks such as an Intranet. A WAN or LAN connection may be used as a connection from a router to an image source for an external upgrade.

0091In yet another embodiment, a newly inserted card that has a version of an image that is more than one or two versions older than the other cards in the router may be upgraded successively in an automatic fashion if all of the required images are available at a single source. In this embodiment, a GMCC task manager would perform an image version comparison as previously described above. Noting the existing image on the newly inserted card after boot and the number of upgrades required to attain the current image that the other cards are booting from, the GMCC can establish a session to perform the required number of upgrades until the card has the latest image.

0092In another aspect of the invention, the backup components are distributed based on algorithms which optimize one or both of system load and resources. In this particular embodiment no degradation in router performance will occur during the upgrade.

0093The method and apparatus of the invention enjoys a variety of possible configurations and implementations without departing from the spirit and scope of the present invention. Therefore, the claimed invention should be afforded the broadest possible scope under examination. The scope of the present invention is limited only by the following claims.