Bulk data transfer

Technical field [0001] The present invention relates to network data exchange, and in particular a mass data transfer protocol.

Background Art [0002] With the recent increase in network bandwidth, the widespread connection of users via the global Internet and the increase in the amount of data processed by enterprises and users, the demand for mass data network transfer (files and directories) is still growing. In particular, users want to send larger files over networks with ever-increasing bandwidths and ever-increasing distances.

[0003] Such data transmission paths not only encounter bandwidth bottlenecks and round-trip delay due to geographical distance, but are also exposed to packet loss and variable delays due to the media itself (e.g. wireless), and variable and sometimes excessive overloads.

[0004] Conventional Mass Data Exchange Protocols (TCP) based on Transmission Control Protocol are characterized by a significant reduction in performance compared to global internet paths, due to poor TCP performance in networks with high bandwidth and delay products. Much attention was paid to implementations and alternative transmission protocols to improve performance (transfer speed and bandwidth utilization) in mass data exchange in networks with high bandwidth and long delay. However, the latest approaches offer improved bit rates and better bandwidth utilization mainly on links in the backbone of the Internet, with relatively low bit error rates (BERs) and having large bandwidths, avoiding congestion. However, most user data exchange extends between the ends of the network and not only is subject to long return delays due to geographical distances, but is also not free from packet loss periods and the variable delay characteristics of a typical edge network. In a typical edge network, current approaches are not able to fully utilize bandwidth, suffer from bandwidth variability as congestion increases, and are unable to provide sufficient guarantees of transfer time required by time-sensitive business processes and demanding users. In addition, in limited cases, where current approaches actually improve throughput, they do so at the expense of "fairly" sharing bandwidth with other network applications and do not give the end user any control over bandwidth sharing. The end-user is forced to choose between a standard implementation of the TCP standard, inefficient but "fair", and an alternative new protocol ensuring improved throughput in limited cases, but at the expense of justice in sharing bandwidth. Although this may be acceptable in the backbone of the Internet, it is not accepted in often overcrowded edge networks where data transfers are accepted into networks with limited available bandwidth. One of the existing NACK UDP protocols is the "UDT" protocol, the second generation of SABUL (Yunhong Gu, Robert L. Grossman, "UDT: A Transport Protocol for Data Intensive Applications", draft-gg-udt-01.txt, August 2004). UDT adds a flow control algorithm that uses dynamic-sized windows to control the upload speed and limit the maximum number of unregulated negative acknowledgments. Although this technique imposes a rigid restriction on the number of retransmissions that may accumulate, and protects against collapse due to congestion, it creates an artificial protocol bottleneck due to the poor performance of the UDT retransmission mechanism, and imposes on UDP traffic adapted to arbitrarily high transmission speeds effective Upload speed limit similar to TCP. There is a need in the art for a data transfer system that solves the above problems and provides improved bit rate, predictable transfer speeds independent of network distance or congestion (and related packet delays and losses), automatic full bandwidth utilization and the ability to divide bandwidth proportionally with other traffic, when there is no unused bandwidth, taking into account both current, as well as future implementations of the TCP protocol.

SUMMARY OF THE INVENTION [0005] The above-mentioned problems, as well as others not discussed herein, are solved by the present invention, which will be understood after reading and analyzing the present description.

[0006] The present invention provides a reliable network data transfer system as defined by independent claim 1. The system is useful for transferring data in networks and providing improved data transfer speed in networks using software data transfer applications.

[0007] These and other system components are embedded in the application management software interface that provides full control and monitoring of transfers in the system. Other variants of the implementation include stand-alone applications, operating system software plug-ins, tool applications, hardware components and virtually any other type of software or hardware layout adapted to provide the services of the systems described here.

[0008] The present summary of the essence of the invention is a general summary of a part of the present application and is not exclusive or exhaustive in relation to the subject of the invention. Additional details of the present invention may be found in the detailed description and appended claims. Other aspects will be apparent to specialists after reading and understanding the following detailed description and analysis of the drawing figures forming part thereof, none of which should be construed as a limitation. The scope of the present invention is defined by the appended claims and their legal equivalents.

Description of the drawing figures [0009] The attached drawing figures are provided to provide some aspects and examples regarding the present system, but it is not intended that they be the sole or exhaustive representations of the present invention.

[0010] Fig. 1 is a block diagram of a system according to an illustrative embodiment.

[0011] Fig. 2 is a block diagram of a sender / recipient system according to an illustrative embodiment.

[0012] Fig. 3 is a block diagram of a data sending process according to an illustrative embodiment.

[0013] Fig. 4 is a block diagram of a data reception process according to an illustrative embodiment.

[0014] Fig. 5 is a diagram of a system data flow according to an illustrative embodiment.

[0015] Fig. 6 is a schematic diagram of a system data flow according to an illustrative embodiment.

[0016] Fig. 7 is a schematic diagram of a system 700 data flow according to an illustrative embodiment.

Description of Embodiments [0017] In the description that follows, reference is made to the accompanying drawings, which form part thereof, and in which, for the purposes of illustration, specific embodiments in which the present invention may be used are shown. These embodiments are described in sufficient detail for those skilled in the art to use them, it being noted that other embodiments may be used and that structural, logical and electrical changes may be made without departing from the scope of the present invention. The following description should therefore not be construed as limiting, and the scope of the invention is defined by the appended claims and their legal equivalents.

[0018] It should be noted that the system provided herein in various embodiments can be implemented in hardware, in software, in firmware and in combinations of hardware and / or software and / or firmware. It should be understood that in various embodiments, the system functions may correspond to modules that are implemented in hardware, software, firmware, or any combination thereof. The examples presented herein may combine one or more functions in one module, however, it is assumed that other combinations of functions may be implemented without departing from the scope of the present invention.

[0019] In various embodiments, program portions may be implemented using devices including (without limitation) a signal processor, ASIC, microprocessor, microcontroller, or other type of processor. Environments in which the present invention may be used include (but are not limited to) computing environments with network devices such as computers, servers, routers, and access gates. gateway), LANs, WANs, intranet paths and / or the Internet or other devices for network connections.

[0020] In some embodiments, the functions are implemented in at least two specific connected hardware modules or devices with respective associated control signals and data signals transmitted between and through the modules, or as part of an application-specific integrated circuit. Thus, the sample flowchart of the process is applicable in software, hardware and firmware.

[0021] Various embodiments of the present invention are provided herein. Fig. 1 is a block diagram of a system 100 according to an illustrative embodiment of the present invention. System 100 includes a first node 102 and a second node 132 connected by a network 122.

[0022] The first node 102 includes a processor 104, memory 106 and network interface 112 connected to the bus 114. The first node 102 may optionally include a memory device such as disk 110, output device 116 and input device 118. The second node 132 includes a processor 134, memory 136 and network interface 142 connected to the bus 144. The second node 132 may optionally include a storage device such as disk 140, output device 146 and input device 148. In various examples of memory 106, 136 of first and second node 132, they are shown to include software 108, 138. Such software includes functions for at least transmitting data to a network interface. In various embodiments and applications, such software may be loaded into memory 108, 138 from at least one source including, without limitation, memory devices 110, 140.

[0023] Network 122 in various embodiments includes one or more of the following: Internet, local area network (LAN), intranet, wide area network (WAN) or other type of network.

[0024] The software 108, 138 may operate on the processor 104, 134 of the respective node 102, 132 to allow the nodes 102, 132 to exchange data over the network 122. The software 108, 138 causes the nodes 102, 132 to perform various data exchange operations . These operations involve the exchange of data according to various methods of synchronized acknowledgment as outlined below.

[0025] In the present description, there are references to data transfer in the context of an "end-to-end" transfer path. The transfer path extends from the source host, such as the first node 102, to the destination host, such as the second node 132, through an IP network, such as network 122. The transfer path has a characteristic "bottleneck bandwidth", "round-trip time network "round-trip time", and "round trip time" (ang. path round-trip time).

[0026] The bandwidth of the path bottleneck is the minimum data transmission capacity (in data units per time unit) along the entire length of the transfer path. This includes the bottleneck bandwidth of sending and receiving hosts, such as first node 102 and second node 132, and network bottleneck bandwidth 122, including at least one hop on the network. The bottleneck capacity of nodes 102, 132 is the minimum data rate (data per time unit) of the resources used for the transfer, including media read / write speed 110, 140 and memory 106, 136, bus speed 114, 144 host, processor speed 104, 134 and network interface speed 112, 142. Network bottleneck bandwidth 122 is the minimum bandwidth of specific network links including the network path.

[0027] The round trip path time ("RTT path") is the time required for the flow of the data unit from the data receiver to the source and back. For example, the RTT of the path includes the time of reading the data unit from the carrier 140 or memory 136 of the second node 132, the time of traveling the data unit back through the network 122 to the first node 102 and loading the data unit into the memory 106 and the transmission of the data unit back through the network 122 to the second node 132 and loading the data unit into memory 136. In one example, the time is measured using "markers" in the packet, indicating the start of the transmission and ultimately the time it was received.

[0028] The round-trip time of the network ("RTT of the network") is the time required for the flow of the data unit through the network 122 starting from the time the network sends the receiving host to the data unit until the sending unit reaches the sending host and then back to the host the receiving party, sometimes called network latency.

[0029] In various embodiments, the RTT of the network includes the time the request flow down the communication stack at the destination host (network protocol layer to the network stack in the operating system to the physical interface), the time the request flows through the network to the sending host, time for the sending host to receive the retransmission request and send the next scheduled unit of data (includes moving up the stack to receive the incoming retransmission request (physical interface to the network stack in the operating system to the system protocol layer) and moving down the stack to send the next scheduled data units (the system protocol layer to the network stack in the operating system to the physical interface), plus network flow time to the target host. [0030] The bandwidth-delay product (BDP) of a given transfer path is an expression of the total bandwidth of the path and is equal to the bottleneck bandwidth multiplied by the round trip time. For the purposes of this disclosure, references to BDP are made in terms of round-trip time, but it should be noted that in very high bandwidth networks, bottleneck bandwidth and BDP can in practice be determined by the host bandwidth.

[0031] Data transfer is defined in terms of "data entry speed", "collection speed" and "useful reception speed" which determines "efficiency". The data input speed ("Ri (t)") is the speed at which the sender enters the data into the network for the sending application (it is measured e.g. in bits or bytes per second). The speed of data reception ("Rr (t)") is the speed at which the recipient reads data from the network 122 to the receiving application. Usable reception speed ("Ru (t)") is the speed at which the recipient receives "useful" data, which includes data that has not been previously received (eg duplicate data).

[0032] The terms "duplicate collection speed" and "transfer efficiency" are also used herein. The speed of receiving duplicates ("Rd (t)") is the speed at which the recipient receives data previously received.

[0033] Transfer efficiency is the ratio of useful take-up speed to total take-up speed (Ru / Rr). Maximum transfer efficiency (100%) is obtained when Ru approaches Rr and no duplicates are received (which means that the overhead of the excess protocol data is negligible):

Ru / Rr ~ 1 and Rd ~ 0 The "ideally efficient" protocol enables the transfer of all required data, which may require retransmission of data lost due to packet losses on the transfer path, without redundant transmissions. Note that performance is not the same as using bandwidth. [0034] The stable system 100, according to various embodiments, strives for a constant bit rate for which the use of bandwidth does not oscillate in the presence of packet losses, network delay and packet loss variation and network delay. This allows application 108 in the system 102 to select any data input speed Ri without disturbing the stability of the system 100. If the system 100 uses a fixed target speed, the data is evenly fed into the network 122 and does not create "pulses". In some embodiments where the system 100 uses dynamically adapted speed, the speed evolves to equilibrium velocity in proportion to the distance from the equilibrium rather than the current transfer speed, to ensure stability at high speeds. A stable protocol using dynamically adapted speed does not overfill the buffers intervening in routers on the transfer path and does not disturb small "mouse" traffic.

[0035] Some embodiments include parameters that are used to measure system performance 100, including "predictability", "bandwidth sharing justice" and "independent speed control". Useful data reception speed (Ru) is "predictable" if the bit rate and transfer time are deterministic in variable and unpredictable path conditions, such as variable round trip delay and packet loss.

[0036] A protocol is considered to be "fair bandwidth sharing" for the TCP ("TCP friendly") standard if a single flow competing with TCP is equally aggressive and divides the BW bottleneck bandwidth equally, so that the speed of each flow is BW / N for N competing flows. To ensure high performance and fairness in mass networks, a reliable transfer protocol fairly shares with TCP and is characterized by "max-min" justice: when TCP flow does not use its full proportional bandwidth allocation, the system 100, in some embodiments, uses the remaining bandwidth.

[0037] System 100 offers applications "independent speed control" if the data input speed is not coupled to a reliability mechanism, and system 100 provides the application with an interface used for speed control. Some parameters of the various embodiments that can be manipulated include discrete speed settings such as target speed or max / min ranges, relative aggressiveness, and prioritization rules. System 100, in some embodiments, also provides intelligent feedback for applications, such as performance statistics such as effective speed, bytes transmitted continuously and measured network impact in the form of round trip time, packet loss on the transfer path and overhead Protocol.

[0038] In order to obtain the above-described system 100 features (stability and predictability in the face of packet losses, network latency and packet loss and network latency variability, Ru / Rr ~ 1 performance and independent speed control) proposed embodiments for a reliable mass data transfer system share the following processes:

a. When blocks are lost, retransmission requests are stored at the recipient,

b. The memory for retransmission requests has the following structural properties

i. the time to enter the memory must be at a constant time O (1) ii. the requested search or retransmission must be at a constant time O (1) iii. finding and deleting pending retransmission requests when the retransmitted block is received must be at a fixed time of O (1)

c. Retransmission requests received by the recipient are stored in the sender's memory. The sender's memory cannot grow as packet losses increase.

i. the recipient sends retransmission requests only at the speed at which the sender can send the retransmitted blocks ii. the sender's memory for retransmission requests must allow a constant input time (the proposed embodiment provides a logarithmic input time of O (log (n)), but since the size of the sender's memory does not increase as packet loss increases, the input time is virtually constant) iii. the sender must send retransmitted blocks in an orderly manner (lowest index first) to optimize disk read performance, so that the minimum retransmission index in memory must be found at a fixed time O (1).

d. Retransmission requests must reach the sender without delay. The recipient sends retransmission requests in the smallest possible packets, given the number of retransmission requests to be sent and the speed at which they must be sent.

e. The recipient's system must process the incoming data at the speed at which it is received. If the data must be saved on the recipient's system disk, it must be done in an optimal way.

f. If the recipient's system cannot process the incoming data at the speed at which it receives it, due to system limitations, the incoming data is abandoned and the abandoned blocks are considered lost for the retransmission mechanism.

[0039] Fig. 2 is a block diagram of a system 200 according to an illustrative embodiment. System 200 includes sender system 201 and recipient system 226. Sender system 201 and recipient system 226 are connected to each other via network 224.

[0040] The sender system 201 of the system embodiment 200 includes a set of modules. These modules include transfer initiation module 222, data file source module 218, data application source module 203, block handling module 206 and optional encryption module 208. Sender's system 201 further includes block output module 210, feedback module 216, speed control module 214, retransmission module 212, management interface module 204 and transfer initiation module 222.

[0041] Transfer initiation module 222 supports establishing a control channel with a recipient system 226. The control channel may use reliable or unreliable base transport (e.g., TCP or UDP). The control channel may also be secured using a public private key method, such as SSL (Secure Sockets Layer) or SSH (Secure Shell). Using the control channel, transfer initiation module 222 supports authentication for the sender's application 202 by sending the credentials to the recipient's system 226, and can optionally exchange for each session a symmetric encryption key used in data encryption. The 222 transfer initiation module also supports negotiation of transmission parameters such as block size, target speed etc. and exchanges file or directory metadata for building the target file and restoring partial transfers. The metadata includes attributes such as file name, size, access control parameters and checksum. [0042] The data file system source module 218 provides a data sequence for transfer from disk 220 or memory available to the sender system 201 through the file system 218 of the sender system 201. The sequence can be a file, directory, raw byte sequence, or virtually any other type or form of data.

[0043] The data application source module 203 provides a data sequence for transfer in the sender's application storage space 202.

[0044] The block handling module 206 retrieves data by reading data blocks from the file system or from user storage space 203 when needed for transmission or retransmission. [0045] The encryption module 208 is an optional module in the sender's system 201. The encryption module 208 optionally encrypts data blocks and adds authentication hashes for consistency verification.

[0046] The block output module 210 writes blocks to the network 224.

[0047] Feedback read module 216 reads control feedback from the recipient system 226, including requests to retransmit missing blocks, transfer statistics, and dynamic target speed. Feedback module 216 analyzes the type of message and forwards the content to a suitable module for processing, e.g., retransmission module 212, speed control module 214 or management interface module 204.

[0048] Speed ​​control module 224 schedules the blocks for transmission according to the target speed (e.g., in bits per second).

[0049] Retransmission module 212 stores incoming retransmission requests in a data structure that allows sorting by number in sequence. The retransmission module 212 also designates block numbers for retransmission.

[0050] The management interface module 204 provides a monitoring and control interface from which control commands are sent and the transfer statistics are read.

[0051] The recipient system 226 of the system embodiment 200 includes a set of modules. These modules include transfer initiation module 225, data file system destination module 250, data application destination module 227, block handling module 230 and optional 232 encryption module. The recipient system 200 further includes a block download module 236, feedback record module 248, speed control module 242, retransmission module 246, management interface module 228 and transfer initiation module 225.

[0052] Transfer initiation module 225 supports establishing a control channel with the sender system 201. The control channel may use reliable or unreliable base transport (e.g., TCP or UDP). The control channel can also be secured using a public private key method, such as SSL (Secure Sockets Layer) or SSH (Secure Shell). Using the control channel, transfer initiation module 225 supports authentication for the recipient's application 227 by sending the credentials to the sender's system 201 and can optionally exchange for each session a symmetric encryption key used in data encryption. Transfer initiation module 225 also supports negotiation of transmission parameters such as block size, target speed etc. and exchanges file or directory metadata for building the target file and restoring partial transfers. The metadata includes attributes such as file name, size, access control parameters and checksum. [0053] The block download module 236 reads data blocks from the network 224.

[0054] The encryption module 232 is optional. Embodiments containing the encryption module 232 decrypt encrypted data blocks and verify authentication hashes for consistency. [0055] The block handling module 230 processes incoming data blocks. Processing involves extracting the round-trip timestamp and passing it to the speed calculation module, and extracting the round-trip timestamp and passing it to the expiration prediction module. The processing further includes copying the contents to a disk write module 234 for output.

[0056] The disk write module 234 uses logic to maximize the recipient's input / output ("I / O") speed by minimizing blocking between network read and disk write operations. The write module on disk 234 uses a number of buffers and each time allocates one buffer for reading network 224 and one for writing to disk 252. When the buffer is full by the network reader, it is passed to the write module on disk 234, and a new buffer is determined for the network reader.

[0057] Cache module 238 uses logic to maximize the speed at which blocks are written to disk 252 or to system memory by minimizing non-sequential writes and writing blocks of the optimal size for a given file system.

[0058] The data file system destination module 250 is a file or directory on the disk 252 or system memory available to the local computer through the file system where the received data is stored.

[0059] The data application destination module 229 is a memory sequence in the memory space 229 of application 227 of the receiver 226, where the received data is recorded.

[0060] Retransmission module 246 stores information about missing data blocks for recovery according to the index. The information stored includes a sequence number and a timestamp of the original time the missing data block is sent.

[0061] Feedback recording module 248 sends feedback to the sender's system 201. The feedback may include retransmission requests, statistics, calculated target speed and any other information regarding the exchange of data between sender's system 201 and receiver's system 226.

[0062] The Overdue Prediction Module 240 calculates the waiting time before requesting the retransmission of missing blocks (RTO) using a recursive algorithm that predicts the path-to-path time based on round-trip time measurements.

[0063] The speed control module 242 calculates the target transmission speed according to the configured speed control mechanism determining the constant speed or the speed dynamically adapted as a function of the measured return time.

[0064] Time counter module 244 stores block numbers in the sequence for retransmission according to the absolute time at which retransmission is expected. The absolute time is given by the RTO time calculated by the expiration prediction module. The time counter module sends to the retransmission module a list of block numbers in the sequence whose retransmission is expected at the current time. [0065] The management interface module 228 provides a monitoring and control interface from which control commands are issued and transfer statistics are read.

[0066] The file discrimination module 254 evaluates data already present in the recipient's system 226 and compares it with the data of the sender's system 201 to determine if certain identical data is already present and does not require transmission. In one embodiment, a comparison is made between a recipient's file with the same name as the sender's file based on attributes such as size, modification time, and content checksum. If the files are identical, no data transfer occurs. If the file is partially transferred, the file discrimination module determines the offset at which the transfer should start or resume.

[0067] It should be noted that the exact functions, the way they are grouped and their interconnections, as well as the processes carried out by each of them can change without departing from the scope of the present invention.

[0068] Fig. 3 is a block diagram of a process 300 according to an illustrative embodiment. Process 300 is a computer-executable way of sending a file or data structure from a source system to a destination system. The process 300 includes receiving a transmission command to the destination system of the file or other data structure 302, establishing a connection and exchange of control data with the destination system 304, splitting the file or other data structure into numbered blocks 306. The process 300 further includes determining whether a retransmission request 308 has been received and waits, and retransmission of any requested blocks 310 prior to transmission of any further blocks. The process 300 further includes determining if there are any blocks awaiting transmission 312, transmitting the next block in numbered sequence 314. If there are no blocks awaiting transmission, the process 300 determines whether an indication has been received from the destination system that the last block has been received 316. If such an indication has been received, the process terminates operation 320, otherwise the process retransmits the last block 318.

[0069] Fig. 4 is a block diagram of a receiving process 400 in a destination system from a source system, file or other data structure according to an illustrative embodiment. The process 400 includes receiving a command to receive a file or other data structure 402, establishing a connection and exchanging control data with the source system 404, allocating the memory area and dividing the memory area into blocks according to the number of blocks to receive 406. Process 400 further includes receiving a numbered block 408, determining if the block has already been received 410, rejecting the block if it has previously been received 412, or writing the block to its corresponding memory block corresponding to the number of the received block 414. The process 400 then determines whether there is a gap in blocks 422 received. If the gap exists, the process schedules the retransmission of the missing blocks 416, determines whether the missing blocks should already be received 418 and reports a request to retransmit the missing blocks 420. Then the process 400 determines whether the received block was reported for retransmission 424. If the block was retransmitted, it is removed from the retransmission schedule. Process 400 then determines whether the block was the last block 428. If the block was the last block, process 400 terminates operation 430. Otherwise, the process 400 iterates until the last block is received.

[0070] It should be noted that the operation of the process and the operation of individual procedures may vary without departing from the scope of the present invention.

[0071] Some embodiments of processes 300 and 400, shown in Figs. 3 and 4, respectively, provide computer applications with the ability to reliably transfer data block sequences between them. In some embodiments, process 300 and process 400 are included in a single computer application to provide the system with the ability to act as sender and recipient.

[0072] In operation, the sender's system operates according to the process 300 of Fig. 3 to transfer the data structure in sequence from the source to the destination of the recipient's system at the desired target speed or with the path bottleneck bandwidth whichever is smaller. The recipient system operates according to the process 400 shown in Fig. 4. The transfer is carried out with high efficiency, regardless of the time of the return path before transmission or variability of the return delay and packet loss during transmission.

[0073] When a constant target speed is desired, the transfer speed remains constant minus the packet loss speed, even in the event of congestion. When using fair bandwidth sharing mode, the transfer speed should automatically adjust to use the available bandwidth when the network is lightly loaded, but should match the user-configured proportion of fair speed TCP when the network is congested (not has available bandwidth).

[0074] Fig. 5, Fig. 6 and Fig 7 show different aspects covered by different embodiments of the present invention. These figures show, inter alia, details of round trip time measurements, data retransmission, and calculation and updating of data entry speed according to some examples and should not be interpreted as a comprehensive or restrictive demonstration.

[0075] Fig. 5 is a schematic diagram of a system 500 data flow according to an illustrative embodiment. System 500 includes a sender containing the final data source 502, a network 504 and a recipient containing the final destination of data 506. Transmission requests are sent by the recipient along with a time stamp that is used to calculate the instantaneous return time for reliability methods (i.e. RTT path) ) and congestion measurements (i.e. network RTT). Each time stamp has a type flag (n = RTT of the network, p = RTT of the path). "N" retransmission requests (Trex) are sent at regular intervals from recipient to sender. If there is no retransmission, when the "n" measurement is planned, an empty retransmission request is sent.

[0076] Fig. 5 includes the following "T" reference signals with the following meanings:

T1: time of sending the retransmission request by the recipient

T2: time of retransmission request reaching the sender

T3 (p): time the sender sent a block corresponding to a retransmission request

T3 (n): time to send the first block after receiving a retransmission request marked with the "RTT network" flag

T4: time of arrival of this block to the sender [0077] The following calculations are useful in various embodiments and can be made using measured times T1, T2, T3 (p), T3 (n) and T4:

<td>Treverse_network</td><td>= T2 - T1</td>

<td>Trex_sendqueue</td><td>= T3 (p) - T2</td>

<td>Tforward_network</td><td>= T4 - T3 (n or p)</td>

<td>RTT path</td><td>USED ​​FOR SERIAL RETRANSMISSION REQUESTS (RELIABILITY ALGORITHM):</td>

<td>RTT_path</td><td>= Treverse_network + Trex_sendqueue + Tforward_network = T2 - T1 + T3 (p)</td>

<td>- T2 + T4 - T3 (p) = T4</td><td>- T1</td>

<td>RTT network</td><td>USED ​​FOR NETWORK OVERLOADING (ADAPTED SPEED CONTROL ALGORITHM):</td>

<td>RTT_network</td><td>= Treverse_network + Tforward_network = T2 - T1 + T4 - T3 (n) = T4 - T1 + (T2</td>

- T3 (n)) Fig. 6 is a schematic diagram of a system 500 data flow according to an illustrative embodiment. The illustration of Fig. 6 shows more details of generating a retransmission request, processing and timing according to some embodiments.

[0079] At time T1, the recipient adds the lost blocks required for retransmission to the protocol data unit (PDU) of the retransmission request. In some embodiments, retransmission is required when the "RTO" time is equal to the current estimated RTT time. The current time is saved in the time stamp (TK) in the PDU header of the retransmission request and the flag type flag is set (path "P" or network "N"). Network "N" markers are sent periodically. If this is not the time to send the "N" tag, then the default tag is "P".

[0080] At T2, the retransmission request reaches the sender. The sender enters the request in the queue sorted sequentially by block numbers. Each block is stored with its own TK time stamp.

[0081] When a retransmission request containing a "N" tag is received, the next data PDU (retransmission or original) sent contains a TK tag corrected for sender processing time to measure only network time: TK = TK + (T3 (p) - T2).

[0082] The sender continuously sends blocks at the input speed Ri. All queued retransmissions are sent in order before new data is sent. If there are queued retransmissions, the sender selects the lowest block number and reads that block again from the disk. The retransmitted data block, its TK time stamp and type (P / N) are encapsulated in the PDU.

[0083] When a data block is received by the Recipient at T4, if the block contains a flag, the Recipient updates the predictive estimation of the return path or network (RTO) time. The recipient calculates the sample of the round-trip time (RTT_i) from the built-in tag and enters the sampled round-trip time to the predictive estimator function to calculate the RTO for the path or network.

[0084] Fig. 7 is a schematic diagram of a system 500 data flow according to an illustrative embodiment. This illustration shows both the calculation of the new 506 input speed by the recipient as a function of input parameters such as max / min speed, bandwidth sharing rules such as constant speed or automatically matched speed, and TCP aggressiveness, all of which can be delivered in real time by management interface as well as propagation of new speed to the sender.

[0085] The data sequence for transfer is divided into blocks of equal size. The data block size is calculated so that the protocol data unit (PDU) carrying the data (content + application header + encapsulation packet headers) does not exceed the maximum transmission unit (MTU) for the networks through which the given unit PDUs will probably be sent.

[0086] The system guarantees transmission of all blocks and reconstruction of the source file at the destination. The blocks are sequentially numbered. The system recipient notices the missing blocks and requests the sender of the system to retransmit the missing blocks until all blocks are received and saved in the destination file. The received blocks are saved to disk or memory as they are received, creating a "rare" file that is gradually filled up until it is completed.

[0087] The system sender starts by sending blocks in order, at a target speed specified by the user (either as an absolute value or as a percentage of automatically detected bandwidth), or calculated by the adaptive speed mechanism. In Adapted Speed ​​mode, the sender can optionally use the slow start mechanism, in which the initial upload speed is a fraction of the upload speed, and the Adapted Speed ​​algorithm automatically increases the target speed in a few seconds. Each block contains the block number used to play the file at the recipient's site. The sender can receive block retransmission requests from the recipient. In this case, the sender stores the retransmission requests and re-sends the requested blocks at a user-specified speed or calculated by the adaptive speed mechanism. The server sends all blocks planned for retransmission before sending any new blocks. When there are no more blocks for retransmission or new blocks for transmission, the server enters the termination state, in which it sends the last block of the file repeatedly, until the recipient signals the receipt of the entire file or subsequent retransmission requests are made.

[0088] The recipient of the system is waiting for receiving data blocks. After receiving the block, if the block has not been previously received, the recipient moves the block to memory, such as disk subsystem or system memory. If the block number in the sequence indicates a gap in the block reception, the recipient schedules the retransmissions of all missing blocks with the numbers in the sequence between the last block previously received and the block.

[0089] The retransmission scheduling module is intended to report retransmission requests of missing blocks as a function of the timer, which determines when to send retransmission requests. The retransmission schedule time counter is based on predictive measurements of the return path time. When the appropriate time comes for a retransmission group, the recipient sends retransmission requests for the given blocks to the sender. When the retransmitted blocks are received, their entries are removed from the pending retransmission scheduler. Blocks are moved to memory, disk or other location with correct offset and are saved to a file. When the last block of data is received, any remaining retransmissions are requested according to the fast termination algorithm. When all blocks are received, the recipient sends a completion message to the sender.

[0090] Various other embodiments provide methods for obtaining high data transfer efficiency and predictable transfer speed regardless of round trip delay and packet loss for any high fixed insertion speed.

[0091] Some such embodiments provide block transport providing non-sequential data access to data transfer applications and provide very accurate input speeds that are independent of reliable data reception.

[0092] Embodiments that provide non-sequential data access to a data transfer application include block transport requesting that a data source, such as a disk system, memory, or application, provide data in discrete blocks and not necessarily in sequential ordering.

[0093] Such embodiments determine the size of the data "block" and require the application to deliver blocks individually or in the form of a range. For example, in the ordinary file transfer, such embodiments define a block as a number of bytes. (The size can be configured by the application, it can be a pre-coded value in the implementation or detected by testing the MTU size of the transfer path. For maximum throughput, the block size should be as large as possible without crossing the MTU path and causing packet fragmentation to avoid any unnecessary overhead in completing fragmented packets in the lower IP layer.) The file is divided into blocks, with the first block being block number 1. There is no guarantee as to the order and number of requests for a given block. At each moment of the application, the smallest block number requested is provided. Based on this information, the application can discard previous blocks avoiding the overhead of storing large data buffers in memory and potentially operate on sequential data in parallel.

[0094] Some embodiments include a data entry speed independent of its reliability mechanism. In such embodiments, the input speed does not depend on whether the data has been successfully received and the speed control algorithm is under explicit independent control of the application. This ensures that the transfer quality is independent of network latency and packet loss, as well as providing independent control of transfer speed.

[0095] An application running according to these embodiments uses a target input speed either configured by the application (e.g., absolute value or percentage of the configured or automatically detected bandwidth capacity), or calculated using an equation-based algorithm, and controls the input speed rather using synchronization in timing than confirmation. This speed control maintains the target speed with relative accuracy regardless of system load and ensures CPU friendliness with other applications. Because such embodiments do not require sequential confirmation of the transmitted data to enter new data into the network, the data may be requested again from the application in any order, eliminating the need to maintain excess memory for the transmitted blocks until receiving confirmation of receipt.

[0096] In the "constant speed" mode, the applications according to these embodiments maintain a constant input speed, or, when using the "adapted speed" control algorithm, adjust the target speed according to the measurements made of the available network bandwidth and configurable TCP aggressiveness, which can be explicitly given to the application. The application can set the speed control mode on the fly (e.g. constant or adapted) and boundary parameters including target speed, maximum and minimum transfer speeds and scaling factors for fair bandwidth sharing. (Although for current implementations it may be most useful to express a scaling factor for the calculated target speed relative to the (Reno) TCP standard, the scaling may be relative to any TCP compatible implementation with a constant bit rate as a function of measurable end-to-end network parameters, such as round trip time and package loss).

[0097] In some examples of maximizing the use of the network, it is important that the sender's system enters the data precisely at the required input speed (calculated or predetermined). The problems that need to be overcome to achieve this are first of all the timing mechanism provided by the operating system, and secondly the system load.

[0098] First, in multiprocess operating systems, the granularity of process switching is much greater than the "time between packets" required by high speed network transmissions. Typically, granularity is on the order of 10 to 20 milliseconds, which means that after the CPU is released, the process will not run for at least 10 to 20 milliseconds. Sending one data packet of 1500 bytes every 10 milliseconds provides a transfer speed of 1.2 Mbps. "Spinning" (as opposed to decelerating the CPU) provides highly accurate time synchronization, but is impractical unless the device can be dedicated to a network sender. Some embodiments of the present invention provide data transfer in multi-task commercial systems and cannot afford monopolizing the CPU.

[0099] Secondly, higher system load, in terms of CPU and disk usage, can adversely affect input speed accuracy, causing delays.

[0100] The method used in some embodiments of the present invention provides high accuracy in input speed. Input speeds are "CPU-friendly" and are not affected by system loading as long as sufficient computing power is available.

[0101] Input speeds are CPU friendly because packets are grouped into groups. The group size is calculated in such a way that the inter-packet delay (IPD) is large enough to allow the sender to free the CPU and return from a break due to process switching without affecting the input speed.

[0102] Changes in system load do not affect the input speed, because such embodiments measure the processing time of one packet or group transmission and compensate for the delay caused by this processing time along with the time spent on process switching. The actual inter-packet delay is compensated for within the measured delay so that a constant insertion speed is maintained at varying system loads.

[0103] The following pseudo-code is an example of an algorithm used to introduce packets into the network to obtain very accurate transmission speeds in the network. The limitation of the minimum inter-packet delay ("IPD") to 5000 microseconds (5 milliseconds) when calculating the group size and IPD is selected so that the delay caused by process switching (10-20 milliseconds) can be eliminated during the next 3 to 4 groups.

[0104] Calculate the inter-packet delay (IPD) and group size (batch size) for a given input speed (Ri):

IPD = block_size * 8 / rate [microseconds] if IPD &lt;5000 microseconds batch_size = 5000 / IPD

IPD = block_size * 8 * batch_size / rate [microseconds] otherwise batch_size = 1

Sender's loop: lagbehind = 0 sleeptime = 0

Repeat until transfer is complete / * Sleep procedure * / sleep_spin = sleep_time% 1000 microseconds sleep_yield = sleep_time - sleep_spin sleep (sleep_yield microseconds) / * It may take longer * / if the time has elapsed, spin for the rest of the time / * short for precision * / / * Actual network transmission * /

Send group batch_size blocks delay = current time in microseconds - last_sent last_sent = current time in microseconds / * Calculate sleep time and delay (lagbehind) * / if IPD> delay sleeptime = IPD - delay if lagbehind> sleeptime lagbehind = lagbehind - sleeptime sleeptime = 0 otherwise sleeptime = sleeptime - lagbehind lagbehind = 0 otherwise sleeptime = 0 lagbehind = lagbehind + (delay -IPD) if lagbehind> 100 * IPD lagbehind = 100 * IPD [0105] Some embodiments also provide ways to maintain high transmission performance at high data entry speeds regardless of path delays and packet losses.

[0106] To avoid a transmission rate bottleneck of the reliability algorithm based on positive acknowledgments, some methods of the present invention use an unreliable transmission channel, such as UDP (User Datagram Protocol), which does not throttle input speed or provide recovery of lost data. These methods achieve reliability by implementing their own algorithms for retransmission of lost data. The retransmission algorithm accurately determines when a data block is really "lost" between the original source and final destination, as opposed to a delayed or reordered block, and thus obtains stable and high performance independent of end-to-end delay and packet loss at high insertion speeds. The retransmission algorithm allows a combination of high input speed and usable bit rate, unchanged at the high return time, which occurs in wide intercontinental networks, with high accidental packet losses that occur in some wireless media and with a variable delay and variable level of packet loss, which occur in public Internet connections overloaded by heavy loads.

[0107] Some embodiments of the retransmission algorithm use "negative acknowledgments". Negative confirmation occurs when the recipient notifies the sender of only lost blocks and the sender retransmits them accordingly.

[0108] Some embodiments of the present invention continually sample the round trip path time and use the predictive estimator function to accurately predict the round trip time and determine when the missing block is really lost and should be retransmitted. The retransmission request is neither too early, which maintains stability, nor too late, which would reduce performance. The useful data reception speed is constant and equal to the input speed minus the speed of the loss of path packets at high input speeds. Thus, high transmission efficiency and good use of bandwidth are realized even at high speeds on lines characterized by high losses and variable delay.

[0109] The problem of the optimal retransmission request algorithm may be modeled. For example, for a given path input speed Ri (t) with efficient transmission approaching 100%, the useful data rate Ru (t) should be equal to Ri (t) minus the packet loss speed P (t) times Ri (t):

High efficiency => Ru (t) ~ Ri (t) - P (t) * Ri (t)

To use any fast network approaching 100%, this must be met for Ri in the range from several kilobits per second to any high speed (> 1 gigabit per second).

[0110] To obtain such a generally optimal model, the request for retransmission of the missing block should wait just long enough for the block on the way, potentially delayed or in a changed order to be received, but not longer than after sending the retransmission request to the sender and receiving the reply at the given target input speed Ri of the sender.

[0111] Although the exact waiting time cannot be determined a priori, it can be estimated with a high degree of accuracy by continuous measurements of the path time and the use of a class of predictive estimation equations known as estimation error to predict future path times. recursive or stochastic gradient functions. Some embodiments of the present invention include their use in a block data transmission system to accurately predict the return path time and to calculate the waiting time with a retransmission request. In turn, such embodiments achieve high transmission efficiency.

[0112] In some embodiments, retransmission scheduling includes accurate prediction of round trip path time, accurate sampling of the current round trip time, and high quality retransmission scheduling module based on the expected round trip time.

[0113] For accurate retransmission timeout (RTO) it is very important to accurately predict the change in round trip path time in the time scale used to send retransmission requests and receive retransmitted data blocks. Various embodiments of the present invention calculate RTT prediction by sampling the round trip time for the entire transfer path, which includes sender processing time, e.g. time to search the retransmission data structure and re-read the block from the disk, in addition to the time the data block traveled on the network. In such embodiments, the processing algorithm on the sender's side is constant with the number of retransmission requests and can therefore be safely included in the prediction of round trip time.

[0114] Furthermore, some such embodiments calculate an estimate of the average round trip time ("smooth RTT" or "SRTT") from the sampled round trip time and calculate the delay variance ("RTT variance") from the difference between the RTT sample and the "smooth RTT". The predicted network delay is then calculated based on smooth RTT and RTT variance.

[0115] After receiving and calculating a new RTT sample (RTTi), the smooth RTT ("SRTT") value is calculated as:

SRTT _{i + 1} = SRTT and + γ * (RTT i - SRTT i) where γ is the gain factor that determines the weight of the current RTT sample in the new smooth RTT estimate. The difference between RTTi and SRTTi represents the previous prediction error, consisting of some random measurement error and some error due to wrong previous estimation. Over time, the components of random error cancel out and an error caused by poor prediction shifts the estimate to the "real" average. The low gain factor therefore ensures that the specific SRTT is not overly affected by random error. In one embodiment, a gain factor of γ of 1/8 is used.

[0116] Given the oscillations of the SRTT estimate around the true average, the RTT variance (VRTT) is calculated as:

VRTT _{ί +}ι = VRTT and + η * | RTT t - SRTT ij

Where η is the damping factor. In one embodiment, a damping factor η of ¼ is used.

[0117] The predicted RTT (= RTO) is calculated as:

RTO _{i + 1} - SRTT _{and +}i + 1 / iJ * VRTT _{i + 1} [0118] In one embodiment, the value of RTO is also limited to the practical limits of the round trip time of typical networks.

[0119] Another factor in predicting network latency is RTT sampling frequency. At the transfer rate range used in some embodiments (20 kbps - IGbps), the sampling period is set to 10 milliseconds.

[0120] The accuracy of the predicted RTT (= RTO) depends on the accuracy of the sampled RTT. The recipient generates an accurate "clock marker" using the best clock mechanism offered by the operating system. This tag is generated just before sending the retransmission request and is embedded in the retransmission request PDU. The request goes to the sender on the network. When the sender is ready to retransmit the corresponding block, it embeds the clock marker into the data PDU and sends it to the recipient. After receiving a data PDU containing a clock stamp, the recipient determines the RTT of the track by subtracting the received clock stamp from the current clock indication. This method is accurate because it uses the highest accuracy clock mechanism available in the operating system and takes into account the processing time at the sender.

[0121] The recipient of negative acknowledgment ("NACK") of retransmission requests must handle block retransmission requests. An embodiment that supports block retransmission requests, detection of lost blocks, request for retransmission of lost blocks, and cancellation of pending retransmission requests when blocks are received is described below.

[0122] Some embodiments of the present invention number each block in the data source sequentially from 1 to N, where N = file size / block size [+1 if file size module block size is> 0]. Other embodiments use various other means to identify individual blocks in a sequential manner.

[0123] The sender adds the block number in the sequence to the content in each PDU. The recipient detects a lost block when it receives a block with a sequence number greater than the next expected number in the sequence. Because the block can be received in a different order, the recipient does not immediately request the retransmission of the lost block, but schedules the first retransmission request one RTO. This makes it possible to receive unordered blocks on the way without generating duplicate transmissions with a premature retransmission request.

[0124] The recipient stores all pending retransmission requests along with the exact time they are scheduled. The time is calculated by rounding the planned time to the accuracy of RTT measurements, to ensure that the planned time is not less than the theoretically correct value due to the measurement error.

Planned time [milliseconds] = Loss detection time [milliseconds] +

RTO on detection [milliseconds] + accuracy of RTT measurements [milliseconds] [0125] When the retransmission request is sent by the recipient to the sender, the next retransmission request is scheduled in the scheduled time calculated in the same way. As a result, when a lost block is detected, there is always a pending request for its retransmission until the block is received and the current pending retransmission is deleted.

[0126] Accurate prediction of the round trip path time requires that the overhead of sending and processing retransmission requests be constant with the number of retransmissions and does not intensify data loss. For high-speed transfers in difficult network conditions, the number of retransmissions can be very high. Various embodiments contain a number of elements that ensure an almost constant processing time for the sender and recipient, and that maximize the likelihood of effective delivery of retransmission requests, even with a large number of retransmissions.

[0127] In such embodiments, when the retransmitted block is received, a pending request for its retransmission is taken from the scheduler and deleted. When the losses are large (the number of retransmissions is large), this can be an expensive operation if the collection method is proportional to the number of retransmissions. The download method used in these embodiments ensures constant access time for near optimal useful bit rate in the face of large losses. On the recipient side, when the block is first detected as lost, the request for its retransmission is stored in a linear table. The index at which it is stored is sent in the retransmission request. On the sender's side, this index is stored with the retransmission request. When the sender retransmits the block, the index is added to the containing block, allowing the recipient to check the pending retransmission at a constant time, regardless of the total number of blocks.

[0128] To avoid the accumulation of pending retransmissions, the sender in the present embodiment always retransmits any lost blocks before sending new blocks. Otherwise, the recipient would accumulate more losses and arrange more retransmission requests, thereby generating overload and reducing file transfer performance. In order to retransmit blocks, the sender must read the block data again from the source file. This reverse lookup and read operation can be costly at high transfer rates and particularly aggravating when packet loss is large and file sizes are large. The recipient suppresses retransmission requests to match the speed at which the sender can resend lost blocks, so that the storage of pending retransmissions on the sender's side is almost constant in size (does not increase with network losses).

[0129] In half-duplex media as well as in network devices causing half-duplex behavior due to insufficient queuing resources, large IP packets on the reverse path from recipient to sender may not be able to reach the sender. For this reason, the sender continues to send blocks, which speeds up the rate of loss accumulation on the recipient side and immediately reduces the efficiency of file transfer.

[0130] The recipient in this embodiment takes the following countermeasures:

(a) For a given number of retransmission requests that the recipient must perform per unit of time, and considering that the sender can retransmit no faster than the send speed, the recipient sends the smallest number of blocks for retransmission in the PDU of the retransmission request as determined by the sender's target speed and the speed of retransmission requests.

rexs per request / request interval (s) = MIN (target_rate (bps) / block_size (bits), rex_request rate (rex / s)) * req_interval

The interval between requests is constant and equal to the resolution of the retransmission time counter (10 ms in the current implementation), EXCEPT the following special case:

If the sender's target speed is so low that the minimum request speed would result in less than 1 rex on demand, the interval will be extended to the minimum interval required for 1 rex on demand.

minimum request spacing = block size (bits) / target speed (bits / s) (b) The maximum size of the retransmission request is configurable by the application and is set by default to be smaller than the minimum typical network MTU (1492 bytes).

[0131] In embodiments with a file transfer rate close to the disk read / write bit rate, disk input / output performance may become a bottleneck. In such embodiments, the sender makes searching the disk easier by always sending the blocks nearest to the beginning of the file first. This allows the sender to sequentially read the blocks for retransmission, and the sender to sequentially write the received blocks. On the sender's side, planned transmissions are stored in a sorted data structure: a modified Red Black Tree to store numbers in sequence for planned transmissions sorted by number. The red-black tree is a classic binary tree structure well described in the IT literature and will not be described in this document. The block numbers in the sequence are the keys of the nodes in the tree.

[0132] Due to the fact that only the smallest block number (minimum) needs to be downloaded, the red-black tree has been modified to provide an almost constant download time. The entry time is typical as in a regular red-black tree.

[0133] The modified red-black tree offers the following basic elements: insert_block (block_seq_number) retrieve_minimum_block () [0134] The red-black tree monitors a node containing the minimum number in the sequence, called the current minimum node. When entering, monitoring the minimum node is trivial: knowing the current minimum, if the input block has a sequence number smaller than the current minimum node, it becomes the minimum. If not, the current minimum node remains unchanged.

[0135] On retrieval, the minimum node is removed from the tree and returned to the application, with the new minimum being found and stored. To support the algorithm used to search for the new minimal node, the following statements are true:

- the current minimum node has no left children (or the left child will have a key smaller than the key of the minimum node)

- the current minimum node is the left child of his parent (or his parent will have a key smaller than the minimum node key)

- the subtree rooted in the right child of the current minimal node has all keys smaller than the parent key of the current minimum node and the rest of the tree.

[0136] To find the "next" minimum node, before removing the current minimum node from the tree, the modified red-black tree uses the following algorithm:

If the current minimum node does not have a right child, the next minimum node is its parent.

Otherwise, the next minimum node belongs to a subtree rooted in the right child of the current minimum node.

The next minimum node is taken by the transverse passage of said subtree.

[0137] The modification of the usual red-black tree algorithm used in various embodiments is described below:

minimum_seq_number = -1; insert_block (block_seq_number) getting up like in a regular RBT block_seq_number if minimum_seq_number == -1 minimum_seq_number = block_seq_number otherwise if block_seq_number <minimum_seq_number minimum_seq_number = block_seq_number retrieve / minimal_blocking> minimum has no right child, new minimum = parent of the current minimum node otherwise find a new minimum by crossing the tree starting from the right of the current minimum node remove the minimum node from the tree [0138] Some embodiments of the present invention require free disk access and frequent back and forward searches if retransmissions are required. While some operating systems offer free access to a high performance file system, other operating systems do not cope well with free access and significantly reduce disk read / write speed. The recipient side in this embodiment feels this the most because disk write operations are more expensive.

[0139] In some embodiments, the recipient uses a disk write cache mechanism to minimize random disk access. The cache size is proportional to the target data transfer speed, according to the following equation:

file_cache_size = ((transfer_rate [bps] / 1800 * block_size) / write_size) * write_size

The file cache size is proportional to the "write_size" disk write buffer size. The disk write size buffer is a multiple of the disk cluster, which may be 512 bytes, 1024 bytes, 4096 bytes, 8192 bytes, and more depending on the file system. Some embodiments use a disk size of 64 kilobytes.

[0140] The file cache receives data blocks from the recipient, buffers the blocks and decides when and what data to write to disk. At the end of data transfer, the cache will transfer its contents to the disk. The file cache solves the following problem: when data loss is significant, the cache should delay the actual write to disk as much as possible to ensure that the recipient has the most opportunity to receive retransmitted blocks and to fill the cache gaps caused by packet loss. Ideally, writing to disk occurs when all component blocks in the write buffer have been received. When data loss is insignificant, the file cache is written to disk as early as possible, without caching a large amount of data to reduce the time it takes to end the file transfer.

[0141] Some embodiments of the method of obtaining high disk write cache performance include the use of the upper disk write level indicator. When the cached data exceeds the upper level indicator, the cache is rewritten to disk from the beginning of the cache. The rules for cached storage with significant and insignificant losses are selected by calculating the moving average of the recipient's retransmission table size.

[0142] The moving average is calculated in such a way that its value follows the number of retransmissions in the recipient's table as they increase and adjusts slowly downwards as they decrease. In this way, the recipient closely follows upward trends and lags behind downward trends. Ascending trend:

retransmission_avg i + 1 = retransmission_avg i + 1 * delta i + 1

Downward trend:

retransmission_avg i + 1 = retransmission_avg i + 1/16 * delta i + 1

Where delta i + 1 = retransmission_sample and + 1 - retransmission_avg and [0143] The upper level marker is calculated as a function of the logarithmic step of the moving average retransmission.

high_ watermark = cache_size * 0.1, for retransmission_avg in [0, 100) cache_size * 0.2, for retransmission_avg in [100, 200) cache_size * 0.3, for retransmission_avg in [200, 400) cache_size * 0.4, for retransmission_avg in [400, 900) cache_size * 0.5, for retransmission_avg in [900, 1800) cache_size * 0.6, for retransmission_avg in [1800, 4000) cache_size * 0.7, for retransmission_avg in [4000, 8000) cache_size * 0.8, for retransmission_avg in [8000, 18000) cache_size * 0.9 for retransmission_avg in [18,000, infinity)

The high watermark is corrected after each write to disk.

[0144] Definitions. At any time, the network path may have "available bandwidth" or "no bandwidth available". A path has available bandwidth when the sum of the bandwidth used by all flows at the moment traversing the path is less than the path bottleneck bandwidth and some bandwidth remains unused. Conversely, the path has no bandwidth available when the sum of the network bandwidth requested by all flows is greater than the path bottleneck bandwidth. In this case, the bandwidth demand exceeds the supply and the bandwidth must be shared by different link flows. "Justice for bandwidth sharing" refers to the relative bandwidth used by individual flows.

[0145] Various embodiments of the present invention provide stable efficient data rate and full utilization of unused bandwidth on shared links in the presence of other IP data flows when bandwidth is available. In networks without available bandwidth, such embodiments automatically regulate their transmission speed for fair sharing of bandwidth with TCP.

[0146] Such embodiments include an adaptive speed control mode using network queuing delay as a network congestion signal (or vice versa, bandwidth availability). In networks with available bandwidth, signaled by low queuing delay, these embodiments determine the insertion speed as a function of the measured queuing delay. It has been shown in the prior art that queuing delay is an accurate indicator of congestion for adapting the TCP transfer rate to the dynamically variable available bandwidth and using equation-based speed control in the form of a queuing delay function to maintain stable high TCP throughput in some networks with bandwidth high speeds. Stable high bit rate in the prior art only applies to situations where there is negligibly low packet loss (but still reducing bitrate when packet losses occur) and only in broadband networks (does not use bandwidth in low speed networks), at the expense of justice sharing bandwidth for other TCP flows. In networks with available bandwidth, the proposed embodiments do not reduce the rate of random loss events (using the delay-based adaptation of reliable UDP transport not sensitive to packet loss) and thus maintain high bit rate even in media with high levels of packet loss, such as a wireless network . The proposed embodiments also use modified scaling parameters to ensure that the system approaches full bandwidth utilization in all practical networks (from several kilobits per second to gigabits per second), not just high speed networks and low packet losses). In addition, the proposed embodiments automatically use TCP-friendly speeds in networks without available bandwidth.

[0147] In networks where there is currently no bandwidth available, the proposed embodiments are capable of providing a bit rate fair distribution of other bandwidths by matching the calculated input speed with a proportional number of TCP flows taking place in the same network conditions. It has been shown in the prior art that equation-based speed control can be used to match the UDP transport input speed with the equivalent TCP speed under similar operating conditions and obtain a fair distribution of bandwidth relative to TCP, but at the expense of stability, bit rate and bandwidth utilization. The proposed system accurately determines when there is no bandwidth available and obtains a fair distribution

TCP while maintaining stable, high bit rate and full bandwidth utilization when bandwidth is available.

[0148] It has been pointed out in the prior art that the congestion / upload speed x (t) window of all TCP implementations develops according to the equation:

x (t + 1) - x (t) = ki (t) (1 - pi (t) / ui (t)) (Equation 1)

Where ki (t): = ki (xi (t), Ti (t)) and ui (t): = ui (xi (t), Ti (t)) [0149] ki (xi, Ti) is a function of gain which determines dynamic properties such as stability and speed response but does not affect the balance properties.

[0150] ui (xi, Ti) is a marginal benefit function that sets equilibrium properties such as equilibrium velocity allocation and equity.

[0151] pi (t) is a measure of congestion or loss probability or queuing delay. [0152] Ti (t) is sometimes a return trip.

[0153] In order to adapt the upload speed for networks with available bandwidth, some embodiments employ a delay-based TCP approach known in the art for reliable UDP transport. The delay-based approach has an marginal benefit function:

ui = ai (t) / xi (t) where ai (t) is the protocol parameter and xi (t) is the current speed, and the suggested gain function:

ki = γ * ai (t) and overload measure, pi, the difference between the base round trip time, brtti, and the current round trip time, srtti.

pi = brtti -srtti [0154] In some embodiments, brtti is the smallest round trip time measured during transfer. For srtti, these examples measure the latency of a round trip network instead of the round trip path delay, as explained below, and calculate the smoothed round-trip time using the same recursive predictive estimation function that is used to calculate RTO for retransmission.

[0155] To obtain a stable equilibrium speed for reliable UDP transport, as shown in the prior art for TCP, this approach seeks to bring changes in send speed over time to (0). This is achieved by adjusting the size of speed and direction in such a way that the ratio of the overload measure to the benefit function (pi / ui) coincides to 1, which means that x (t + 1) -x (t) in Equation1 converges to 0. [0156] By expressing Equation 1 byui and ki with γ = 1/2 and simplifying the components, the general equation for speed update is:

Rate i + 1 = 1/2 * (Rate i * BaseAvg and +1+ Rate i + a) (Equation 2)

Where:

α = 2 * 10^-5 * TargetRate * block_size [in bits]

BaseAvgi + 1 = 1, when brtti + 1 &lt;5 and srtti + 1 &lt;

= brtti + 1 / srtti + 1, otherwise

BaseAvg is set to 1 when brtt and srtt are small to handle cases where brtt is so small that it is of the same order as the accuracy of RTT measurements.

[0157] In some embodiments, α is the coefficient of adaptation of the linear function of the target speed for convergence over a wide range of bandwidths and regulated aggressiveness. As shown in the prior art, the α coefficient is the expression of the number of packets that must be buffered in the queues of the transfer path for the sending source at xi to achieve equilibrium and represents the "aggressiveness" of the speed control algorithm. Flows with the same α value will share bandwidth fairly, while flows with a higher α value will proportionally cover a larger proportion of bandwidth.

[0158] In these embodiments, unlike the prior art, α is regulated as a linear function of the target speed so as to allow convergence to the stable target speed for all practical bandwidths (extending from 100 Kbps to 1 Gbps).

[0159] Just as accurate estimation of the round trip path time is useful in determining efficient retransmission wait (RTO), accurate round trip time measurements are useful for accurately calculating queuing delay. Queuing delay only applies to the network portion of the transfer path, so various embodiments measure the second round trip time value, RTT, which does not include processing time at end hosts. Using the same recursive estimator function as used to calculate RTO for retransmission, these embodiments calculate the smoothed weighted RTT of the network and use this value to calculate the ratio of the current RTT of the network to the base RTT used in the speed update function (Equation 2).

[0160] For measuring network latency, some embodiments use a method similar to measuring the round trip path time for RTO. In these embodiments, the recipient generates an accurate "clock stamp" using the best clock mechanism offered by each operating system. The recipient generates this tag just before sending the retransmission request and embeds it in the retransmission request. If you don't need to send retransmission requests (e.g. when there are no losses in the "forward" channel) the recipient generates "empty" retransmission, which is sent on the minimum frequency, e.g. once every 10 ms. The clock marker is built into the retransmission request PDU and travels the network from recipient to sender. The sender performs an accurate calculation of the time used to process the retransmission request and adds this time to the clock stamp received in the retransmission request PDU, effectively subtracting the processing time. Then it embeds a clock stamp into the data PDU and sends it to the recipient. After receiving the PDU containing the clock stamp, the recipient determines the network delay by subtracting the received clock stamp from the current clock indication. This method is accurate because it uses the highest accuracy clock mechanism available in the operating system and takes into account the time the request was processed by the sender.

[0161] Some embodiments also offer the ability to fairly share bandwidth, or share with proportional aggressiveness, with any TCP implementation in a network congestion situation, in networks without available bandwidth. These embodiments share the bandwidth evenly or in a configurable proportion with any TCP compatible implementation (i.e. with any protocol developing according to Equation 1 outlined previously) by calculating the steady state speed of one TCP flow in measured network conditions (e.g., as a function of network delay and / or packet loss). These embodiments use queuing delay as a signal that there is no bandwidth available, and do not sacrifice full use of bandwidth on links that have available bandwidth, while providing configurable justice on links that currently do not have available bandwidth.

[0162] As indicated in the prior art, using the fact that the upload speed of all TCP implementations develops according to equation (1) (described above):

x (t + 1) - x (t) = ki (t) (1 - pi (t) / ui (t)) (Equation 1)

The expression for equilibrium velocity for any TCP implementation based on losses or delay can be found by setting pi (t) / ui (t) = 1.

[0163] The proposed embodiment uses a delay based overload algorithm, shown in equation 3.

Xi = α ri / srtti - brtti (Equation 3) [0164] As indicated in the prior art, the equilibrium velocity for the most commonly used implementation of TCP (TCP Reno) is expressed by equation 4:

Xi = α ri / rtti * ρ^Λ0.5 (Equation 4)

Where α ri is a TCP Reno parameter, depending on the MTU constant and size.

[0165] In some embodiments, the two equilibrium velocities are compared for deriving the adaptation parameter α depending on the queuing delay and the equilibrium velocity function for a given TCP in equation 5.

ai = (srtti -brtti) * α ri / (rtti * pb0.5) (Equation 5)

The obtained α parameter is then used to calculate the speed of fair bandwidth sharing (using Equation 3), i.e. the speed equal to the TCP speed for currently measured network parameters (e.g. round trip time and / or packet loss). The method of accurate measurement of the return trip time has already been described. Packet speed can be measured in a number of ways, for example using an estimated weighted moving average.

[0166] It should be noted that the same method can be used to match the equilibrium speed of TCP protocols to different response functions as these TCP protocols are introduced, e.g., High-speed TCP or scalable TCP ( Scalable TCP).

High Speed ​​TCP: Xi = α hi / Ti * pb0.84 Scalable TCP: Xi = α si / Ti * pi [0167] The speed control functionality according to some embodiments of the present invention comprises two main components. Finding the α factor to obtain a speed equal to TCP speed in terms of queuing delay, packet loss and round-trip time for fair bandwidth sharing when congestion occurs (no bandwidth available); and accurately determine when overload occurs, using queuing delay, to signal entry into a TCP-friendly state. These embodiments set the overload conditions at which TCP-friendly speed should be used, using a two-state machine that operates in the adaptive x mode of the present invention (to use unused bandwidth when bandwidth is available) and in TCP mode (to ensure fairness , if there is no available bandwidth).

[0168] These embodiments enter TCP friendly mode only when overload occurs, and do not leave TCP friendly mode until it becomes known that the overload is over. These embodiments use a hysteresis model to determine the moment of switching modes. If the round-trip time increases and the queuing delay is sufficiently greater than the base rtt, TCP mode is started. After entering TCP mode, the system remains in it until it decreases rapidly and the queuing delay becomes close enough to the base rtt to indicate that queuing has actually dropped.

[0169] The specific parameters used in some embodiments have been determined experimentally:

Letdrtt = srtt -brtt -10.

- The initial state is x (mode x-mode)

- In x mode: jeslisrtt has increased since the last sampleidrtt> 0.2 * brtt, switches to TCP mode, otherwise stay in x mode

- In TCP mode: if srtt has decreased since the last sample and drtt &lt;0.5 * brtt, switch to x mode, otherwise stay in TCP mode [0170] This method gives very good results for concurrent flows in different network conditions.

[0171] The parameters of the speed control model provide applications with adjustable "control knobs". By presenting α as a configurable parameter, you can change the target speed or aggressiveness during the transfer.

[0172] An application may set its aggressiveness to the number (and type) of TCP flows as a speed corresponding to one standard TCP flow, or two other standard TCP flows. The application can also choose a specific mode, such as "leaky" mode, when the flow decreases to the minimum threshold when sharing with TCP, but increases while appropriating the entire bandwidth when it operates alone.

[0173] Some embodiments introduce a highly efficient leaky transfer by allowing the transfer to use the entire bandwidth as long as there is no other activity in the network and withdrawing to a very low speed when network activity is detected. By making transfers in the speed mode adapted and setting a very low aggressiveness coefficient, the flow will use all available bandwidth acting in the no overload mode and will withdraw completely when entering the overload mode. During congestion, the transfer application can set a minimum speed threshold for this transfer to guarantee delivery time. Users of a liquid application can change the minimum threshold "on the fly" by sacrificing the network bandwidth for the transfer time.

[0174] Some embodiments include optional encryption elements for encrypting and decrypting data blocks on the fly.

[0175] At the beginning of the transfer, a secure TCP channel is established with a remote endpoint using existing methods such as SSH or SSL / TLS. The receiver in such an embodiment generates a random symmetrical encryption key for a given user-configurable cipher (encryption algorithm ) and exchanges it with the sender using a secure channel. In some embodiments, the endpoints may decide to periodically change the encryption key and exchange new keys through the secure channel. The sender encrypts each block of data to ensure data confidentiality and adds a Message Authentication Code to confirm the authenticity of the data. Such a method is provided as an option in various embodiments, such as application-level data transfer applications. This provides applications with means to securely transfer data over public, unsecured networks, such as the Internet.

[0176] Some embodiments also provide the ability to control and monitor file transfers. Such embodiments provide a TCP socket management interface so that the management application can run on the same or different computer than the managed transfer endpoint. The interface allows control and monitoring operations, such as the ability to start and stop transfer, change the transmission speed on the fly, suspend and restore transmission and change the transmission parameters on the fly, such as enabling or disabling the adapted speed mode, changing the aggressiveness. Control and monitoring operations also include operations such as the ability to read basic transfer statistics, read transfer statistics necessary for progressive download, and read specific FASP statistics, such as retransmission data structure parameters, disk write statistics, and adaptive speed parameters. This interface is a mechanism that integrates various elements of the transport embodiment into applications. The interface also allows applications to implement transfer policies such as prioritizing and managing bandwidth usage.

[0177] Some embodiments offer a reliable UDP protocol at the application level to allow applications to change the transfer speed in an ongoing transfer. The transfer manager uses one of the two endpoints involved in data transfer using the management interface. Both the sender and the recipient, when controlled by the same transfer manager, have a dedicated processing thread enabling the exchange of monitoring and control messages with the transfer manager, independent of the data processing thread (s). After receiving control commands, such as a change in flight speed, the dedicated management thread of the sender and recipient stores new values, and the main data processing thread periodically checks and retrieves new values.

[0178] If the manager controls the recipient, passes the desired minimum or maximum speed thresholds to him. The recipient uses the new values ​​in the calculation of the target speed required for the adapted speed mode, or sets the target speed to the maximum speed threshold if operating in constant speed mode. The recipient sends the target speed, calculated or set, to the sender in periodic statistical messages, and the sender complies.

[0179] If the manager controls the sender, pass the desired minimum or maximum speed thresholds to him. In constant speed mode, the sender will use the new speed as his constant target speed, ignoring the target speed requested by the recipient in the statistics messages. In the adapted speed mode, the sender stores the minimum and maximum thresholds set by the manager and compares them with the target speed requested by the recipient. If the target speed requested by the recipient is greater than the maximum speed threshold, the sender sets his target speed to the maximum threshold speed. If the target speed requested by the recipient is less than the minimum speed threshold, the sender sets his target speed to the minimum threshold. In other cases, the sender sets his target speed to the target speed requested by the recipient.

[0180] With the predictable nature of data transfer according to various embodiments of the present invention, the user may choose to set the transfer time instead of the transfer speed. An application example may allow the user to set the transfer time or estimated arrival time and calculate the target speed required to achieve this goal. The transfer management can then set the target flight speed as described above.

[0181] Suspending an ongoing transfer is a special case of setting the target speed on the fly (setting the speed to 0), but this requires that both the sender and the receiver completely stop sending or attempting to receive data. This property is useful for unplanned bandwidth prioritization. To suspend, the transfer manager sets the target speed or maximum speed threshold in adaptive mode to 0. The sender learns about the new speed from the transfer manager or from the recipient via statistical messages. When the sender detects this special case, it stops sending data and waits for the transfer manager to set the target speed to a value greater than 0. If the recipient is controlled by the manager, he learns with a new speed setting of 0 directly. If the sender is controlled by the transfer manager, he sends a control message to the recipient to inform him of his suspended status. In some embodiments, it is important for the recipient to be aware of the state of suspension in order to avoid causing data reception to be delayed.

[0182] The transfer manager, via the management interface, forwards the speed control mode settings (constant speed, adapted speed and bandwidth aggressiveness) to the sender or recipient. If the recipient is controlled by the transfer manager, he stores and enters a new speed control mode. The recipient sends a fixed or calculated target speed to the sender via a statistical or control message. If the sender is controlled by the transfer manager, he stores the new settings and sends them to the recipient via a control message. In one embodiment, the bandwidth aggressiveness can be expressed as a multiple of standard TCP aggressiveness. Some embodiments include the ability to support a continuous sliding scale of aggressiveness relative to standard TCP and other TCP. This can be disclosed to the end user through the management interface as flow type compatibility with a continuous index for matching the aggressiveness value against this flow typotype.

[0183] Within the scope of the OSI protocol stack, some embodiments of the present invention provide an integrated data transport layer, session layer, and service layer protocol.

[0184] Various embodiments include the integration in the operating system of a file transfer structure such as SSH and SCP. SSH provides user authentication and a key exchange structure.

Some embodiments use this structure to set encryption keys if they operate in optional secure mode. SCP actually provides a user interface for remote file copying. Some embodiments include integration with SCP as an alternative high-performance TCP data path.

[0185] Some embodiments store transfer metadata providing the ability to restore data transfer without loss of data or with minimal loss of data that has already been transferred. The transfer can be restored from any sender storing the same file, not necessarily the same sender. This enables redundant systems to be used where applications can take action after the sender's system fails or connects to it when a backup sender or connection route is available. The restore service offers several layers of integrity checking, balancing integrity guarantees and verification time (verification time counts, again, total speed from end to end).

[0186] The nature of the block transfer system based on the target speed, the methods and software described herein, together with the equation based speed control, provide the embodiments with transfer rules. Some of these transfer policies apply to bandwidth allocation, prioritization, and manual file transfer control.

a) Rules for bandwidth allocation. Having an organization with multiple locations and network link capacities between those locations, an administrator or bandwidth management application can determine the allocation of network resources between different file transfer applications. The maximum transfer speeds for each flow can be transferred to a file transfer application and applied. Transfer speed limits may be set before or during file transfers. When allocating time-dependent bandwidth, when file transfers must meet certain transfer time limits, one embodiment allows the setting of a minimum transfer speed. The flow will "behave fairly" in the event of congestion, but will not fall below the minimum transfer speed. This guarantees a minimum delivery time, but at the cost of unfairly forcing all remaining traffic to compete for the remaining bandwidth.

b) Rules for prioritizing. Some embodiments may associate the priority level with the speed control aggressiveness coefficients. As a result, high priority traffic will automatically receive more bandwidth. In the extreme case of prioritizing low priority traffic may be stopped when it competes with high priority traffic. It also allows leaking transfers that do not affect other traffic by setting the leaking traffic priority to the lowest level, which causes it to stop in the presence of any other movement.

c) Manual control of file transfer policies. The management interface provided with some embodiments allows users or administrators to change transfer parameters on the fly. This includes suspending transfers to allow the acceleration of other transfers as well as slowing or accelerating current transfers.

[0187] Some embodiments provide applications with the following parameters: target speed (or maximum speed for adaptive speed control), minimum speed (for adaptive speed control), and adaptation factor. These parameters can be set before the transfer starts or changed during the transfer. Based on these parameters, applications can intelligently control file transfers to obtain:

· Transfer files at a constant speed by selecting a fixed speed control and providing the target speed.

· File transfer with link capacity, but adapting fairly in the presence of competing traffic by selecting adaptive speed control and providing maximum speed higher than or equal to link capacity.

· File transfer at a given speed, but decreasing to the minimum speed in the presence of overload by selecting the adapted speed control and providing the minimum speed. The flow will be adapted to share the link with competing traffic, but its speed will not be less than the specified minimum, guaranteeing delivery time.

· File transfer with link capacity, but not affecting any competing traffic by selecting Adapted Speed ​​Control and providing the minimum value for the Adaptation Factor. The flow will run at link capacity, but will stop in the presence of competing traffic, thus not affecting the normal operation of the network at all. This can be used for efficient leaking transfers.

By allowing applications to change the transfer speed during its duration, applications can suspend and restore transfers by setting the target speed to zero and then back to a non-zero value.

[0188] Some embodiments also provide an intelligent block cache service. This allows these embodiments to determine if file segments have already been transferred to the recipient's system and to use the cached data again instead of sending it over the network. This cache service offers several layers of integrity checking. The type of integrity checking can be determined by the application or automatically determined by FASP using the optimization function. The optimization function balances the time to verify cache integrity against network performance and transfer queue. If the transfer is faster than local integrity verification, the cache service chooses to transfer data again. Otherwise, the locally cached data is used again. [0189] Some further embodiments provide all necessary control service supervision to enable applications to perform two-way pipelined data transfer, also known as progressive download. Two-way data transfer includes one transfer endpoint that receives data and at the same time sends the same data to a third transfer endpoint or uses data. Examples of two-way stream transfer include an application that downloads a media file and at the same time delivers that file to the media player. or a cache application that downloads a file on behalf of the user and stores the cache files when it is delivered to the user at the same time.

[0190] One such embodiment includes a method of obtaining a pipelined data transfer by starting data transfer from A to B at the desired speed and disclosing in B the effective reception speed, loss level, and the amount of continuous data received, starting from the beginning of the file. The method further includes determining the transfer time from B to C based on the data disclosed in B and the desired transfer speed from B to C. The method also includes disclosing in B the effective speed xfer = up | down auth = yes | no enc = yes | no | any maxrate = <val>

defrate = <val>

adapt = yes | no port = <val>

sign = <val>

transfer from B to C and the amount of data sent. Based on this information, the method can decide if the pipe function is working properly. In the event that the transfer from B to C precedes the transfer from A to B, the method may slow down or even suspend the transfer from B to C. In addition, the method includes disclosing the lowest block number that can be requested by the transfer from B to C. This way the method can remove in B data to this place, which is useful in case of limited memory.

[0191] Some embodiments involve identifying and transferring files using references. The transmission endpoint can download or send files or catalogs based on specific references. References, in various embodiments, include identification of the remote file or directory, transport parameters such as transfer speed, adaptive speed control and encryption. [0192] One of the examples of reference formats is as follows:

fasp: // <severname> / <path> [? <option> & <option> ...] <server-name> is the remote machine name (FQDN) or IP address.

<path> may point to a directory or file.

[0193] At least one of the following options is available for reference in various embodiments:

"Up" represents sending, in which case the path represents the destination directory

If set to "yes", the transfer requires user authentication. If set to "yes", encrypted download is forced, and if set to "no", unencrypted download is forced. If set to "none" or not present, the user can choose to encrypt or not encrypt.

Sets the maximum allowed speed to <val> Kbps. The user can choose the transfer speed to this value.

Sets the default speed to <val> Kbps. The user can choose a different speed up to the maximum allowed value.

If set to "yes", adaptive speed control is used.

Sets the UDP port to <val>.

Reference chain reference as a security measure to ensure integrity.

[0194] By making the transfer available through the reference service, FASP can be easily integrated in applications such as download or upload on the web, replacing e-mail attachments with FASP download references, login and logout in the asset management system. Some embodiments of the present invention use UDP to carry the content of data blocks. However, the same goals can be achieved using virtually any transport mechanism or network layer. Alternative transports and network layers, for example, may include a user-defined poIP transport protocol or a modified TCP stack at endpoints. The TCP stacks at the endpoints can be modified to act as UDP, e.g. without flow control or retransmission. Blocks can be sent as TCP packets over the network, which may offer the ability to set a "firewall", thus avoiding specific firewall settings and intrusion detection for UDP. Alternative transports include non-IP networks that offer services similar to IP network services: best effortless routing and packet delivery between at least two endpoints involved in the data transfer operation. Such networks include satellite networks, packet radio, wireless broadcast or point-to-point networks, and ATM networks.

[0195] The architecture of some embodiments is two-tier. The first layer in such embodiments provides a protocol by offering applications a block file transport service. Layer two is a minimal file transfer application built on top of the protocol.

[0196] However, variants of this architecture may include the implementation of systems and methods as part of the operating system, as the driver, kernel module or simply part of the monolithic operating system. Other variants include the implementation of the present invention in a monolithic data transfer application (i.e. a single-layer approach).

[0197] Other embodiments include the use of a capturing proxy server, transparent or not, to capture existing TCP traffic and forward it over the network according to various items described herein to a remote proxy server that forwards data to the remote end of TCP applications using TCP. An interceptor proxy server can be a portion of a program running on endpoint machines, a portion of a program running on file or application server machines, or a hardware device attached to the network. Still other embodiments include the present invention within a network file system. Another embodiment includes a specialized application gateway for wireless or satellite networks for efficient mass transport.

[0198] Some embodiments include methods and algorithms for obtaining reliability, performance, security and management by adjusting certain protocol parameters. The values ​​of these parameters are set according to the network environment or operating system. Reliability, performance, security and management are obtained by manipulating these parameters or the way they are calculated. These parameters include:

• Block size • Retransmission expiration γ and η parameters • File cache size • File cache lower and upper level indicator • File cache retransmission average • Parameter α speed control mode compatibility with FASP and TCP • Step parameters of the average base speed control function • Speed ​​Control Parameter C • Speed ​​control parameter coefficients for switching states between FASP and

TCP.