Method and system for data delivery with guaranteed quality of service

METHOD AND SYSTEM FOR DATA DELIVERYWITH

GUARANTEED QUALITY OF SERVICE

0003Related Applications

0004The present application claims priority under 35 USC §119(e) from United States provisional application serial number 60/219,884, filed 21 July 2000 and entitled Data Delivery with Guaranteed Quality of Service.

0005The present application also claims priority from United States application serial number 09/657,131 filed on September 7, 2000.

0006Field of the Invention

0007The invention relates to the delivery of content from a server to clients in a manner that provides guaranteed quality of service.

0008Background of the Invention

0009The internet is a collection of public and private networks. Within the United States, the internet includes the interconnected resources of over 5000 local, regional and national Internet Service Providers (ISP), nearly 50 backbone "Tier 1" providers, and an assortment of telephone, cable and other communication companies.

0010Communication over the internet is achieved using the Transfer Control Protocol (TCP) and the Internet Protocol (IP), sometimes referred together as TCP/IP. TCP breaks data into packets and attaches a header that includes source, destination, sequence and checksum information. TCP is responsible for ensuring that all packets are received and that they are assembled in the appropriate order. IP is responsible for routing packets from source to destination. IP attaches a further header to the TCP packets that includes source and destination information. IP routing is a dynamic process in which different packets may follow different routes following complex algorithms and rules depending on routers configurations, weights, network congestion, etc. In general, each system that receives an IP packet makes a determination as to which of the systems to which it is connected is the appropriate next system for getting the packet to its destination. As a result, IP packets often pass through a large number of interconnected systems between their source and destination. At any point a router or gateway can decide to discard packets due to its contention level. The variations among the intermeshed networks and the various alternate routes also result in lost packets (discarded along the path) and interrupted access links. Thus, establishing a steady data flow from the source web server to the client depends on numerous factors that cannot be controlled by either the source or the destination of the data flow. If packets are lost, the source must re-send these packets, further reducing the overall efficiency of the network. The larger the file, the more chances are that an error occurs, thus making downloads of large files over the internet a time consuming and aggravating experience that very often ends up in failure. In addition, since IP protocol tends to evenly distribute resources amongst all active communication channels, the internet's throughput capacity for a specific data flow fully depends on the current load of each of network components involved.

0011Figure 1 provides an illustration of the structure of the internet and the contention points that generate the problems described above. Referring to Figure 1 , when a user requests a file from a source web server, the file is not available at the user's browser until all data comprising the file travel through the following infrastructure:

0012Web server and Web access device

0013Web server connection to ISP point of presence (POP)

0014Web server ISP internal infrastructure (LAN/MAN)

0015Web server ISP uplink to internet

0016Network Access Point (NAP)

0017Internet backbones and network exchange points

0018User ISP uplink to internet

0019User ISP internal infrastructure (LAN/MAN)

0020User ISP access device connection to ISP POP User dial-up connection to ISP POP (last mile connection)

0021User access device (modem)

0022Any of these points can become a contention point depending on its size and utilization rate. A network component or communication link becomes a contention point when its throughput capacity and usage rate do not allow it to support the complete data flow requested by its connected devices and lines. At that point it begins to discard requests and packets that it cannot transport.

0023Although a web server usually has the I/O capacity to fill the communication link that connects it to the Internet (dedicated or shared link), the processing resources it needs to manage a large number of concurrent Internet user's connection can be sometimes insufficient. In addition, contention can appear at the disk level when a large number of user request pieces of information located on the same drive. The server's contention can result from under-sized server I/O capacity, under-sized server computing capacity, under-sized private connection to ISP, and under-sized ISP connection to internet gateway, each of which is becoming increasingly common.

0024Although in most situations ISP LAN infrastructure capacity will be a multiple of the aggregated bandwidth potentially utilized by users, the ISP WAN/MAN infrastructure interconnecting its various POPs is often tightly sized and assumes low user usage rates. Since the WAN/MAN infrastructure is one of the major operating costs of an ISP, it is generally undersized, thus making it a bottleneck when a large number of users are connected. Routing, firewall and proxy devices may also become bottlenecks depending on the type of traffic and number of users connected.

0025The connection of the ISP with its network access points often accounts for most of the ISP operating cost, and consequently ISPs tends to maximize utilization of these connections and have as many users as possible on it, sharing the available bandwidth. Thus this connection becomes another bottleneck.

0026Network Access Points (NAPs), where ISPs connect to the internet backbone, and network exchange points, where backbones of different carriers exchange data, are often major contention points during peak traffic hours. Although numerous carriers are now using alternate exchanges, they remain a location where traffic goes from one infrastructure and technology to the other with all of the configuration and optimization issues involved.

0027Backbones can also become points of contention occurs when carriers have over-subscribed their infrastructure capacity and see large number of users requesting the service they subscribed at the same time.

0028Consequently, the structure of the internet makes it impossible for a content provider to have control over a whole delivery channel, or to guarantee a quality of service for the delivery of content to end users. All of the contentions described above reduce the efficiency of data transfer through the internet, and thus impair the ability of the end user to obtain content from content providers. Thus, while the increasing size and complexity of internet continues to improve the consumer's opportunities to identify desirable content on the internet, these same factors degrade the user's ability to receive that content in an efficient and reliable manner.

0029Summary of the Invention

0030Embodiments of the invention address the contentions that occur upstream from home and business internet users by providing a centralized server that pushes data from a single stored instance to one or multiple clients with a guaranteed quality of service.

0031In one embodiment of the invention, a system for providing a file to an internet user includes a centralized server and virtual circuits of a guaranteed quality of service network that provide communication between the centralized server and internet service provider points of presence (POPs). The centralized server provides files to internet users through the guaranteed quality of service virtual circuits to their POPs. The server may further regulate the rate at which the data is pushed so that it provides a requested degree of saturation of the users' last mile connections to their POPs. Related embodiments of the invention pertain to servers, POP systems and user systems that may be employed in such a system, and processes that are performed in those devices. In another embodiment of the invention, an email system includes a centralized server and virtual circuits of a guaranteed quality of service network that provide communication between the centralized server and internet service provider points of presence (POPs). User systems transfer email attachments to recipients by way of the central server and guaranteed quality of service network. Related embodiments of the invention pertain to servers, POP systems and user systems that may be employed in such a system, and processes that are performed in those devices.

0032In another embodiment of the invention, a video conference system includes a centralized server and virtual circuits of a guaranteed quality of service network that provide communication between the centralized server and internet service provider points of presence (POPs). Video data from video conference participants is routed from internet users through their POPs to the centralized server using the guaranteed quality of service network. From there it is pushed through the guaranteed quality of service network to POPs of the other participants. Related embodiments of the invention pertain to servers, POP systems and user systems that may be employed in such a system, and processes that are performed in those devices.

0033In another embodiment of the invention, an internet cache system includes a centralized server and virtual circuits of a guaranteed quality of service network that provide communication between the centralized server and internet service provider points of presence (POPs). The central server is used by the POPs as a remote cache, and a file cached by any POP is available to all other POPs using the server as a remote cache. Related embodiments of the invention pertain to servers, POP systems and user systems that may be employed in such a system, and processes that are performed in those devices.

0034Description of the Drawings:

0035Figure 1 illustrates the system architecture involved in conventional file transfer through the internet;

0036Figure 2 illustrates a system level implementation of an embodiment of the invention; Figure 3 illustrates a server infrastructure in accordance with an embodiment of the invention;

0037Figure 4 illustrates application level components of the server of Figure 3; and Figure 5 illustrates an ISP POP in accordance with an embodiment of the invention.

0038Description of Embodiments of the Invention

0039Figure 2 illustrates an exemplary implementation of the invention in conjunction with the conventional internet infrastructure. As shown in Figure 2, a number of internet servers are connected to the internet through network access points (NAPs). The servers include internet protocol (IP) routers for enabling communication over the internet. The servers may for example be web servers that make hyper text markup language (html) files available using the hypertext transfer protocol (http).

0040Also connected to the internet are internet service provider (ISP) points of presence (POPs), which are the physical systems of the ISP that provide connectivity to the internet. Internet user systems are connected to POPs through last mile connections. The last mile connections may be serviced by 56k modems, DSL modems or cable modems. In the alternative, the end users may be located at ports on a LAN. Any other conventional last mile connection may be employed as well.

0041Also connected to the internet is a centralized server. The centralized server is connected to ISP POPs, and to network exchange points through point to point virtual circuits of a guaranteed quality of service network. A "guaranteed quality of service" network is one that provides at all times a predetermined level of accuracy, dependability, and speed of data transmission. Examples of parameters that are guaranteed in such networks include maximum cell loss ratio, availability (downtime), throughput, packet delivery delay, jitter, sustained bandwidth, peek bandwidth, and burst capacity. One type a network that provides guaranteed quality of service is an asynchronous transfer mode (ATM) network, which provides permanent virtual circuits having a guaranteed quality of service. Another type of network that provides guaranteed quality of service is a multiprotocol label switching (MPLS) network, which provides switched virtual circuits having a guaranteed quality of service. Other networks such as frame relay and synchronous optical network (SONET) may also be employed to provide delivery of IP packets with guaranteed quality of service. Such delivery is contrasted with that of the internet, in which the quality of service is degraded in proportion to the number of users contending for system resources and therefore cannot be guaranteed.

0042The guaranteed quality of service point to point circuits between the centralized server and the POPs of Figure 2 enables data transfer from the centralized server to the POPs at a reliable speed, accuracy, and dependability. Thus, data transferred from the centralized server to the POP through this network will be reliably delivered. In contrast, the speed, accuracy and dependability of data transfer from the centralized server to the POP through the internet would largely unpredictable. The guaranteed quality of service point to point circuits between the centralized server and the network exchange points of Figure 2 enable data transfer from the centralized server to a POP for which there is no guaranteed quality of service circuit by sending the data to the POP's nearest network exchange. This circumvents many of the internet bottlenecks, thus improving the quality of data transfer available to such POPs.

0043The system illustrated in Figure 2 enables a variety of embodiments of the invention. In one embodiment, content providers promote the availability of various content files, such as downloadable files or files for real time streaming such as audio and video files. While the availability of the files is promoted on web sites hosted on internet web servers, the content files themselves are stored at the centralized server, and are provided to requesters' systems by way of the guaranteed quality of service circuits between the centralized server and the requesters'POPs, thus circumventing internet bottlenecks.

0044The operation of the system in this embodiment of the invention may be illustrated through comparison with conventional internet file download or streaming. Conventionally, an internet user using a web browser to access web pages on a web server may encounter a web page indicating that a content file, such as a video file or an application file, may be transferred to the user by clicking on an icon displayed on the web page. When the user clicks the icon, the web server transfers the file from the web server to the user through the internet as a series of IP packets. Because transfer through the internet is subject to unpredictable and potentially extreme degradation as a result of the bottlenecks discussed above, it is not known whether all of the packets will arrive, or at what time they will arrive. As a result, download times are unpredictable, and the quality streaming applications are unsatisfactory.

0045In contrast, in accordance with this embodiment of the invention, when the user clicks the icon on the web page, a transfer of the content file from the centralized server to the user is initiated. The file is transferred as IP packets routed through the guaranteed quality of service circuit between the centralized server and the user's ISP POP, and from the POP to the user. Because the quality of service available over this circuit is known, the transfer time to the POP is completely predictable, and can be controlled such that, for example, the rate of data transmission is sufficient to consistently saturate the user's last mile connection to the POP. Accordingly, from the user's perspective, the system operates in a manner very similar to that which the user is conventionally accustomed to, in^' that the user locates the file through the internet and initiates the file transfer process in a familiar manner. However, in contrast to the user's conventional experience, the file is received with reliable speed and accuracy, limited only by the bandwidth of the user's last mile connection.

0046In another embodiment of the invention, the system of Figure 2 provides video conferencing services. Particularly, users who desire to establish a video conference may connect to their respective POPs and establish contact with the centralized server through the guaranteed quality of service circuits linking their POPs to the centralized server. Transfer of data between the POPs then takes place through the guaranteed quality of service circuit and the centralized server. This enables video conferencing using typical consumer or business computers through a connection to the internet service provider but with a level of quality that could not be achieved using the internet infrastructure.

0047In a further embodiment of the invention, the system of Figure 2 provides enhanced file transfer between internet users. Particularly, an internet user who wishes to transfer a file to another internet user provides the file to the centralized server through the guaranteed quality of service circuit that links the users POP to the centralized server. The file is subsequently downloaded or streamed to the recipient's POP through the guaranteed quality of service circuit linking the centralized server to the POP. The file transfer is accomplished at a rate and accuracy that could not be achieved using the internet infrastructure. In a preferred embodiment, the file is an attachment to an email message.

0048In another embodiment of the invention, the system of Figure 2 enables an ISP POP uses the centralized server as a remote level of POP cache. Particularly, the POP employs a cache manager for determining the data that should be stored in various available levels of the POP cache. The centralized server is designated as one of the levels of cache, and a file assigned that level of the cache is transferred to the centralized server through the guaranteed quality of service circuit. Subsequently, when a cache hit occurs for a file cached in the centralized server, the file is transferred to the POP through the guaranteed quality of service circuit. Because of the reliable speed and accuracy provided by^" the guaranteed quality of service circuit, the transfer time from the centralized server is not noticeable from the user's perspective. Further, because of the server's ability to serve multiple users from a single instance of a file, a file once cached by any POP is available to all other POPs using the server as a cache. This embodiment vastly increases POP caching capacity without the need for extensive infrastructure investment.

SERVER HARDWARE

0050Figure 3 shows the architecture of a centralized server in accordance with a preferred embodiment of the invention. In the preferred embodiment, the server is implemented on an IBM SP2 supercomputer having a scalable parallel computing architecture. The server includes a plurality of processing nodes, each of which comprises a workstation that may include up to 16 processors. The nodes are connected through a non-blocking switch. Multiple switches may be staged to provide scalability. The nodes of the server are further coupled to a giga-ethernet router. The nodes push IP packets to the router, which converts the packets to appropriate transport level units. The router preferably provides multicast services such that a single packet provided to the router can be transmitted to multiple destinations simultaneously. The router is coupled to a switch. The switch provides connections to one or more internet uplink channels that connect to NAPs for purposes of internet communication. The switch further provides connections to virtual circuits of guaranteed quality of service networks that provide point to point communication with ISP POPs.

0051Each of the nodes is also connected to multiple data storage units such as hard disk drives, and every storage unit is directly accessible by at least two nodes. Data stored in any storage device is accessible by any node through a request that is routed by the non-blocking switch to a node having direct access to the data

0052In general, the server stores a single instance of each file that it makes available. Exceptions may occur where it is required to asynchronously and concurrently provide the same content file to a very large number of users (for example, more than 20,000) at high speeds. An example of such an instance is providing on demand access to a popular movie.

0053Each content file is striped across multiple storage devices so that the physical throughput limitation of a single storage device access mechanism (for example, a hard drive head) does not limit the rate at which the entire content file can be retrieved. Accordingly, a node requesting a file works in collaboration with other nodes having access to the storage devices containing pieces of the content file to retrieve all pieces so that the content file can be reconstructed. It is preferred to distribute content file stripes across storage devices in a manner that evenly distributes the overall retrieval load across the nodes of the server regardless of the specific content file being retrieved. Parameters considered in formulating this distribution may include storage device characteristics (throughput, average access time, storage capacity), the number of different content files stored by the server, the individual file sizes, the number of storage devices and the way they are attached to the nodes that control them, the node bandwidth, and the assumed distribution of content file requests. Content file striping and retrieval functions may be implemented through the IBM GPFS file management application. Reconstruction of corrupted files is preferably provided using the IBM RAID file reconstruction application.

0054Each node is enabled to simultaneously retrieve data and push data to the giga- ethernet router. In the preferred embodiment, data retrieval and pushing tasks are distributed among the nodes of the server to provide balanced utilization of server resources. Such load balancing may be implemented using the IBM END load leveler application.

SERVER APPLICATION LAYER COMPONENTS

0056Figure 5 provides an illustration of application layer components of a preferred embodiment of the server. These components include an access manager, a client manager, a content manager, a resources manager, an account manager, a streaming engine, and a resend engine.

0057The access manager serves as the gateway for communications between the server components and outside devices. All communications directed to the server's IP address are received by the access manager and forwarded to the appropriate component within the server. The access manager further provides interaction with trusted third party systems for determining authorization of access to restricted content.

0058The client manager maintains information concerning user systems. This information may include user system IP addresses and their associated ISP POP IP addresses, user system nominal last mile throughputs, and user system file request histories.

0059The content manager provides services for importing, updating and deleting content files. Content files may be supplied by third party content providers whose content is hosted on the centralized server. Content files may also be provided by internet users for such purposes as video conferencing and routing of email attachments. Content files may be supplied by ISP POPs for purposes of caching at the centralized server. The account manager maintains billing records for allocating charges assessed in relation to content file transfers. These costs may be allocated to either the third party provider of a content file, or to the user system that receives the file. Billing records may include identification of the requested content, and the transfer time, speed, and degree of last mile saturation provided.

0060The resources manager manages controls utilization of server resources and utilization of guaranteed quality of service network resources to ensure that all services provided by the server achieve required quality of service standards. The resources manager reserves resources for use in satisfying content requests, taking into account all previously reserved resources. The resource manager refuses requests for which there are not sufficient resources. In this manner, resources reserved for a given requester cannot be degraded by future granted requests.

0061The streaming engine pushes content file data to the router for delivery to user systems. The streaming engine converts content data into IP packets, and pushes the packets to the router a rate that is paced to achieve a requested quality of service at the recipient user system's last mile connection. The rate at which packets are forwarded to the router is determined in accordance with the user system last mile throughput, the type of delivery service requested, and the availability of buffering at the ISP POP.

0062The resend engine prepares and sends individual IP packets or groups of packets that have been reported as corrupted or lost during initial transfer from a streaming engine. Packets are provided by the resend engine to the user system when requested by the user system and are pushed at a rate calculated to provide a desired degree of saturation.

ISP POP SYSTEM AND FUNCTIONS

0064Figure 5 illustrates a preferred embodiment of an ISP POP system in accordance with the invention. The POP system is an enhancement of a conventional ISP POP system. Like a conventional POP system, the system includes a WAN/MAN interface, a cache manager and cache, and an access manager providing a last mile connection to user systems. However, in accordance with the preferred embodiment of the invention, the WAN/MAN interface provides communication through both an internet uplink, and through a guaranteed quality of service point to point circuit to the centralized server. The routing protocol used in the interface provides routing of packets addressed to the IP address of the centralized server through the guaranteed quality of service circuit, rather than routing such packets through the internet. Packets addressed to other IP addresses are routed in a conventional manner through the internet. Such routing may be accomplished by augmentation of the BGP routing protocol to provide intelligent routing, or by implementation of static routing mechanisms. The POP is preferably designed to provided a guaranteed minimum throughput of packets received from the centralized server so that the guaranteed quality of service provided by the centralized server and guaranteed quality of service circuit are not degraded at the POP.

0065The server of the POP system is augmented with a buffer and a buffer manager. The buffer selectively stores those IP packets that are received from the centralized server, and provides them to appropriate users at rates calculated to provide requested degrees of saturation of their last mile connections. The buffer manager is IP addressable and provides communication with user systems for resending packets requested by the user system. The buffer manager also includes logic for marking packets for which resend has been requested, and maintaining those packets in the buffer as long as possible so that the request for resend can be serviced directly from the buffer rather than from the centralized server. Where requested packets are no longer available in the buffer, the request for the packets is forwarded to the centralized server. Preferably, the buffer may be flushed or bypassed by the centralized server as needed. The use of the buffer and helps to provide saturation of the user's last mile connection by maintaining packets in excess of what is required to saturate the connection. The buffer also detaches the transmission rate calculations in the server from the variability of the last mile connection. The buffer manager preferably provides multicasting services. The POP server is further augmented with a cache manager that uses the centralized server as a remote cache. In this implementation, the cache manager is IP addressable, and the server is used for storage of data that did not originate on the server, but rather originated on other internet servers accessed by internet users connected to the POP, for example, web pages hosted on remote web sites. The centralized server is preferably used as a level of cache storage that is secondary to or further removed from a primary cache. The cache manager monitors user data usage and assigns frequently accessed data to an appropriate cache level. Where data is assigned to the cache level provided by the centralized server, the data is forwarded to the POP router where it is transferred as IP packets to the centralized server. Similarly, where the cache manager determines that requested data is assigned to the cache level provided by the centralized server, the cache manager sends a request to the centralized server for transmission of the data to the POP.

USER SYSTEM AND FUNCTIONS

0067The user system typically is a personal computer or workstation. A user system in accordance with a preferred embodiment of the invention includes a client content manager for communicating directly with the centralized server and the ISP POP buffer manager, thus- enabling the user system to send files to and receive files from the centralized server through the guaranteed quality of service circuit between the centralized server and the user's POP. The client content manager is preferably a stand-alone application, but could be implemented in a variety of other forms such as a plug in for a web browser or email application.

0068The client content manager communicates with the centralized server to request file transfers, and provides control information to the centralized server, or to an intermediate POP buffer manager, for managing data transfer rates. Information preferably communicated for managing data transfer rates includes instantaneous data throughput and packet loss ratio. Other information collected and reported to the centralized server includes file size, file name, duration of transfer, start/end date/time, line interruption and line nominal speed. The client content manager is also responsible for buffering and storing received packets, reconstructing files from packets, and storing the downloaded files or forwarding streamed data to an appropriate application. For downloads, the client content manager preferably employs an enhanced network level protocol enabling it to request resend of individual IP packets from the centralized server, or from the ISP buffer manager.

0069The client content manager provides a user interface for enabling a user to select data transfer parameters. For example, in the case of receiving a file, the user may request immediate (next available) transfer, transfer at a specified time, or overnight transfer at a time to be determined by the server. The use may also specify a desired type of delivery. Available types of delivery may include a specific degree of saturation of the user's last mile connection, for example, total saturation or saturation of a percentage, and interleaved delivery in which the file transfer is suspended at any time that another application such as a web browser needs to use the last mile connection.

0070A variety of other management and housekeeping functions are also provided. These include maintaining a schedule of files to be transferred, providing transfer status, initiating dial-up, disabling other applications during transfer, providing encryption and decryption functions, initiating retries of failed transfers in the event of line interruption, and shutting down the system after a transfer is complete.

DESCRIPTION OF A FILE DOWNLOAD

0072In a preferred embodiment of the invention, the devices described above are utilized to provide file download from the centralized server to a user system. A sequence of operations performed in accordance with this embodiment is now described.

0073Initially, a file is supplied to the centralized server by a content provider. The file may be supplied through a network such as the internet, or may be provided on a physical medium such as an optical disk. The file is loaded onto the server where it is received and processed by the content manager. The content manager assigns a file identifier to the file that identifies the instance of the file stored on the server. The content manager then provides the file identifier to the content provider, and the content provider incorporates the file identifier in a web page where it promotes the availability of the file to internet users. The web page preferably includes an icon that the user can operate to request receipt of the file from the centralized server. The icon preferably initiates a call function for alerting a client content manager in the user's system of the request and the identifier of the requested file.

0074Subsequently, when a user surfs the internet using a conventional web browser and encounters a web page promoting the availability of a file for download, the user can operate the icon to request delivery of the file from the centralized server. The client content manager then generates a user interface that allows the user to specify desired aspects of the file transfer. One aspect of the transfer that the user specifies is the desired time at which the transfer is to occur. The user may request immediate (next available) transfer, transfer at a specified time, or overnight transfer at a time to be determined by the server.

0075Another aspect of the transfer that the user specifies is a desired type of delivery. Available types of delivery include saturation of the user's last mile connection, saturation of a percentage of the user's last mile connection, and interleaved delivery in which the file transfer is suspended whenever another application such as a web browser needs to use the last mile connection. Preferably, where transfer at a later time is requested, specification of the delivery type is deferred until immediately prior to the beginning of the transfer.

0076Upon receiving the user specifications, the client content manager communicates a request for the content file to the centralized server. The request includes the content file identifier, the requested type of delivery and the requested time of delivery. The request further includes user system information including the user system's IP address, the POP's range of addresses, and the user's nominal last mile throughput. While the user system information is preferably already stored at the client manager in the server, it is preferable to send it with each request so that current information is always available. The request is addressed to the centralized server's IP address.

0077The request is routed from the client content manager through the last mile connection to the ISP POP, where the centralized server IP address is recognized and the request is routed to the server over the guaranteed quality of service circuit. In the event that the POP does not have a guaranteed quality of service circuit to the centralized server, the request will be routed in a conventional manner through the internet to the centralized server.

0078The request is received at the centralized server by the server's access manager through the server's router and is forwarded to the resource manager where a time slot for delivery of the content file is reserved. Upon receiving the request, the resource manager first determines whether the sever has a guaranteed quality of service circuit to the user's POP. In the event that there is no such circuit to the POP, the server returns a message to the user indicating that transfer is not available to the user's POP. If the user system employs a dial up connection, it is preferable at this time for the client content manager to prompt the user to initiate a dial-up to an alternate POP to which the server has a circuit so that the transfer may be provided. Alternatively, the user may be offered to receive the file through a guaranteed quality of service connection to the network exchange point nearest the user's POP. Assuming there is a circuit to the POP, the resource manager then determines whether the requested service can be provided at the requested time. First, the network resources needed to provide the transfer of the file are determined from the size of the requested file and the transmission rate needed to provide the requested type of delivery . The required resources can be characterized in terms of a required throughput and a required duration. The nominal throughput of the user's last mile connection may be used in making this determination.

0079After determining the required network resources, the resource manager determines whether those resources are available at the requested time. If the user has requested immediate delivery or delivery at a specific time, the resource manager determines a next available time slot providing at which sufficient resources are available on the server and the virtual circuit to the ISP POP to provided the required transmission rate for the required length of time to satisfy the request at the requested time. An entry is made in a scheduling table to reserve the bandwidth at that time.

0080Where transfer has been requested a specific later time, a similar determination is made as to whether sufficient resources are available at that time. If sufficient resources are available, and entry is made in the scheduling table to reserve the resources at that time. If sufficient resources are not available at the requested time, the resource manager determines a closest available time at which the requested level of service can be provided, and makes a conditional reservation of the resources at that time. A message proposing the alternative time is returned to the client content manager. The user may then confirm the suggested time. Upon confirmation, the resources are unconditionally reserved. If confirmation is not received within a predetermined time, the conditional reservation is voided.

0081Where the user has requested overnight delivery, the resource manager determines, from among times having sufficient available resources, the time that entails minimum cost to the party bearing the cost of the transfer. An entry is then made in the scheduling table to reserve the resources at that time.

0082In all cases, once resources have been reserved in the scheduling table, a token granting rights to the requested resources is returned to the client content manager through the guaranteed quality of service circuit. Transfer may then be initiated by the user system by presenting the token to the centralized server in the form of a message sent to the centralized server. Where the user has requested immediate transfer, the client content manager will immediately present the token. Preferably, at this time the client content manager disables other user system applications that could attempt to use last mile resources.

0083Where the user has requested transfer at a later time, or overnight, the client content manager schedules presentation of the token for the time of requested transfer. In conjunction with such transfers, if the user system employs a dial-up connection to its ISP, it is desirable for the client content manager to be able to initiate dial-up to the ISP at the time at which the token is to be presented, so that the user system does not need to be left on-line continuously prior to the time of transfer. Where overnight delivery is used, it is further desirable for the client content manager to shut down the user system after completion of the delivery.

0084Upon receiving and validating a token, the resource manager requests the streaming engine to perform the file transfer. The streaming engine then retrieves the content file and then begins the processes of constructing IP packets addressed to the user system that contain pieces of the content file. The packets are constructed using a uniform payload size.

0085The manner in which data is pushed by the streaming engine varies in accordance with the requested type of delivery, the availability of buffering at the POP, and feedback provided by the client content manager. Generally, packets are pushed to the router at a rate that is paced to provide the requested type of delivery at the last mile connection. The specific rate at which cells are pushed depends on the maximum last mile throughput and the availability of buffering at the ISP POP. Last mile throughput is determined prior to commencement of transfer by sending a short test file to the client content manager at the user system and receiving back from the client content manager a report of the actual last mile throughput achieved.

0086The availability of buffering is known to the streaming engine through reference to information stored by the client manager. Where buffering is available at the POP, it is assumed that packets received at the POP will not be discarded due to a lack of storage capacity. Accordingly, the streaming engine pushes packets at a rate that is proportional to the desired last mile throughput rate, and the buffer manager pushes the packets to the client content manager at a rate chosen to provide requested degrees of saturation of respective user's last mile connections to the POP. The POP buffer manager thereafter provides the IP packets to the client content manager of the user system at a rate chosen to provide the requested degree of saturation. Where buffering is not available at the POP, the push rate at the server is reduced slightly from an amount proportional to the desired last mile throughput since the risk of discarded packets is much greater. In either case, feedback is provided from the client content manager indicating whether the rate should be increased or decreased to optimize last mile throughput. For example, by reporting a rate of packet loss or corruption, the client content manager effectively instructs the POP buffer manager or server to increase or decrease the rate of packet transmission so that an optimal throughput is achieved and the available last mile bandwidth is most effectively used.

0087Interleaved delivery may be used where buffering is available at the POP. Where interleaved delivery is requested, the server pushes packets at the rate calculated to saturate the last mile throughput. However, transmission of packets from the POP to the client content manager is halted upon detecting that another application is using the last mile connection. Data pushing is resumed upon detection that the last mile connection is available again. Suspension and re-initiation of transfer may be done in response to signals provided by the client content monitor, or may be done through intelligent monitoring of other packets routed to or from the user system IP address.

0088AT the POP, IP packets are received from the centralized server as well as from other servers. IP packets received from the centralized server through the guarantee quality of service circuit are selectively buffered by the POP and are provided to the client content manager of the user system through the last mile connection. Preferably, the client content manager provides a display indicating the progress of the download, such as a percentage completed. The client content manager stores the IP packets in a buffer, and notes missing and corrupted IP packets as the IP packet train is received. Upon detection, missing and corrupted packets are specified in a resend request that indicates the beginning and ending packet of a series of one or more missing packets. Where the POP includes a buffer manager, the request is directed to the POP buffer manager, which locates the requested packet or group of packets and marks them for retention in the buffer. The marked packets are stored for as long as possible in accordance with a buffering priority scheme to enable direct retransmission to the client content manager. If the POP buffer no longer contains the specified packets, or if one or more of the specified packets are missing from the POP buffer or corrupted in the POP buffer, the request is forwarded to the centralized server, where it is serviced by a resend engine. Alternatively, the POP buffer may simply not respond. In this event, the client content manager detects that requested packets have not been received, and sends a request for resend to the centralized server. When ail transmissions of packets are completed, the client content manager determines that the requested packets have not been delivered by the POP buffer manager, and issues a request for the packets to the centralized server.

0089In the centralized server, the resend engine determines the portions of the content file contained in each requested packet based on the uniform payload size, and reconstructs the requested packets. In the case of resend from either the POP buffer manager or the centralized, transmission of the packets is delayed until transmission of all of the originally constructed packets has been completed. Once all packets are correctly received, the client content manager rebuilds the file and saves it to a predetermined location in user system storage. Upon completion of the transfer, the account manager in the server adds an entry to the appropriate party's account noting the transferred content file, the time and rate of transfer. The account may be that of a content provider where the file is one that is intended to be freely accessible, for example, an advertisement video or a demo version of an application. The account may be that of a requester where the file is one that is intended to be available for a fee, such a as a pay-per-view movie or a commercial version of an application.

ASYNCHRONOUS MULTICASTING OF DOWNLOAD FILES

0091An alternative method for providing file download may be employed by the resource manager and streaming engine to facilitate multiple user downloads from a single streaming engine. In this method, when a transfer of a file has been initiated by a first user, and transfer of the file is subsequently requested by a second user, the second user is made a party to a multicast of the transfer to the first user that is already in progress and begins receiving the same IP packets as the first user. Because the packets of the file will be sequenced by the client content manager of the user system when the transfer is complete, they can be received in any order. Accordingly, when the end of the file is reached, the streaming engine cycles back to the beginning of the file and continues to send IP packets to the second user up to the point where the second user joined the multicast. Any necessary resending of packets is initiated by 5 the individual user systems and packets are resent to users on an individual basis.

DESCRIPTION OF FILE STREAMING

0093In another preferred embodiment of the invention, the devices described above are utilized to provide streaming of data for real time use from the centralized server to

009410 a user system. This embodiment may be employed for providing audio or video services using data that is static on the centralized server, or data that is received in real time from live events. The sequence of operations performed in accordance with this embodiment is similar to that of the download embodiment described above. Particularly, the process of scheduling and initiating the transfer is essentially the

009515 same. However, since the requested content file is intended for real time use, as in the case of a video or audio file, it is not possible or necessary for every packet to be received by the user system prior to use of the file. Accordingly, the client content manager does not detect or send requests for lost or corrupted packets. Rather, once the client content manager begins to receive IP packets, their contents are extracted

0096>o and buffered. The buffered data is provided to an appropriate application such as a video player once enough data has been received such that variations in last mile throughput will not cause the buffer to become emptied. The use of the client content manager buffer in this manner provides a smooth data feed to the video player so that no slowing or freezing of the video occurs.

0097>5

DESCRIPTION OF VIDEO CONFERENCING

0099In another preferred embodiment of the invention, the devices described above are utilized to provide video conferencing. In particular, participants in a video conference send data generated at their systems to the centralized server through the so guaranteed quality of service circuits joining the server to their respective POPs. At the centralized server, the streaming engine constructs IP packets as data is received from participants and pushes the packets to the other participants of the video conference. In reserving resources for the video conference, the resource manager of the centralized server determines the availability of resources providing a two way connection between the centralized server and each participant. The client content manager simultaneously receives packets and provides data to a media player, and receives data from a video input device and constructs and sends packets addressed to the centralized server that contain the video data. Each user preferably receives a data stream at a rate calculated to saturate the user's last mile connection to the POP.

FILE TRANSFER BETWEEN INTERNET USERS

0101In another preferred embodiment of the invention, the devices described above are utilized to provide transfer of files between internet users. In this embodiment, the files are preferably transferred as attachments to email messages, and the client content manager is preferably implemented as a plug-in to a client email program.

0102In this embodiment, the attachment to an email of a file greater than a threshold size causes the client content manager to present a user interface enabling the user to transfer the .file through the centralized server and guaranteed quality of service circuits. Upload may be scheduled to occur immediately, at another specified time, or overnight. Once the user has specified scheduling parameters, the client content manager initiates contact with the centralized server to schedule upload of the file. In a manner similar to file download scheduling, the centralized server resources manager determines if resources are available, and provides a token entitling the client content manager to those resources. The client content manager presents the token at the scheduled time for upload and transfers the file by constructing and sending IP packets addressed to the centralized server.

0103Before the email is sent, the client manager also replaces the file identifier attached to the email with a message that indicates to a client content manager at the recipient system that an attachment is being transferred through the centralized server. When the email is received at the recipient system, the recipient client content manager is alerted by the message attached to the email, and generates a user interface enabling the user to schedule download of the file. Scheduling of the download occurs in essentially the manner of the downloading embodiment described above, in that the user requests a time and a type of download, resources are reserved, and a token is provided.

0104While this preferred embodiment utilizes email as a messaging service for indicating to the recipient the availability of a file for download, it will be appreciated that alternative messaging may be implemented completely through the centralized server. For example, a stand-alone client content manager may be utilized to schedule upload of the file to the server, and messaging may be provided by the centralized server.

REMOTE CACHING

0106In another preferred embodiment of the invention, the devices described above are utilized to provide a remote cache for ISP POPs. Particularly, the POP employs a cache manager for determining whether files should be stored in various available levels of the POP cache, and the centralized server is designated as one of the levels of cache. When. a file is assigned to the remote cache level, the cache manager sends a request for uploading the file to a centralized server connected to the POP through a guaranteed quality of service circuit, in the manner of uploads to the server described previously. Similarly, when it is determined that a file requested by a user is stored in the remote cache, the cache manager sends a request for an immediate download of the file from a centralized server, in the manner of downloads described previously. In the event that the file cannot be immediately downloaded from the server, the cache manager treats the file as unavailable and proceeds to retrieve the requested file from the remote internet server where it is hosted.

0107While the above description addresses implementations of the invention that are presently preferred, these implementations are merely illustrative. Those of ordinary skill in the art will recognize from these examples a wide variety of alternative implementations, and all such alternative implementations are considered to be within the scope of the invention as defined by the claims presented below.