Efficient screen image transfer

CA Q2S07Ï32 2Û15-0S-2J

WO 2014/139947

PCT/EP2014/054575 connection, and a frame transfer engine. The algorithm database comprises a plurality of image compression algorithms. The frame transfer engine executes on a computer processor and is configured to receive a plurality of updates made to the source display, store at least some of the updates in a queue, and select, based on a bandwidth of the network connection, a size of the update, and sizes and times of updates currently present in the queue, an image compression algorithm in the algorithm database for current transfer over the network connection,

1005( In general, in one aspect, the invention relates to a system comprising a client computer system comprising a target display and a server computer system. The server computer system is connected via a network connection to the client computer system. The server computer system comprises a source display and a memory configured to store an algorithm database comprising a plurality of algorithms, a transmission queue, and a frame transfer engine. The frame transfer engine executes on a computer processor and is configured to determine that the transmission queue comprises an untransmitted queue item, wherein the untransmitted queue item comprises a display frame and a timestamp indicating a time at which the frame transfer engine received the display frame; obtain a first image compression algorithm from the algorithm database, process the display frame using the first image compression algorithm to obtain a processed display frame, and transmit the processed display frame to the client computer system via the network connection, |0Q6) In general, in one aspect, the invention relates to a method for updating a remote display. The method comprises receiving a first display frame of a source display, generating a first queue item comprising the first display frame, generating a first query using the first queue item, and querying an algorithm database with a first query. The method further comprises receiving, in response to the first query, a first algorithm identifier identifying a first image compression algorithm, processing the fust display frame using the first image compression algorithm to obtain a processed first display frame, and transmitting the processed first display frame to a target computer system via a network connection.

[006aj In accordance with one embodiment of the present invention, there is provided a system comprising: a source display, externally updatable; an algorithm database comprising a plurality of image compression algorithms; a network connection; and a frame transfer engine executing on a computer processor and configured to: receive a plurality of updates made to the source display, wherein the plurality of updates comprises at least a first update and a second update; store at ieast the first update and the seeond update in a queue; select, based on a first bandwidth of the network connection, a size of the first update, and sizes and times of updates currently present in the queue, a first image compression algorithm in the algorithm database for current transfer over the network connection; process the first update using the first image compression algorithm to obtain a processed first update; merge the first update with the second update to obtain a merged update; select a second image compression algorithm in the algorithm database for current transfer over the network connection based on a second bandwidth of the network connection, a size of the merged update, and sizes and times of updates currently present in the queue; process the merged update using the second image compression algorithm to obtain a processed merged update; and transmit the processed merged update to the target computer system via the network connection.

|096h| In one preferred embodiment, the frame transfer engine is further configured to before the transmission of the processed first update is complete, receive a third update from the source display; generate a second queue item comprising the third update and a second timestamp indicating a time at which the frame transfer engine received the third update; abort the transmission of the processed first update based on a determination that transmission of the third update

CA 2907992 2017-09-27 foliowing the transmission of the first update will exceed the Tguar; process a fourth update using a third compression algorithm to obtain a processed fourth update, wherein the processed fourth update is a size capable of transmission to the target computer system via the network connection with a third bandwidth within the Tguar; and transmit the processed fourth update to the target computer system via the network connection.

[006cj In one embodiment the third compression algorithm is used in response to a determination that the third bandwidth is larger than the first bandwidth, and wherein the processed fourth update is a higher quality than the processed first update.

[006d] In another embodiment the third compression algorithm is used in response to a determination that a rate of updates to the source display has increased, and wherein the processed fourth update is a lower quality than the processed first update.

1006c] In one embodiment, the frame transfer engine is further configured to:

determine that the transmission of the processed first update has completed; determine that the first image compression algorithm is a lossy compression algorithm; generate a flag in the first queue item indicating that the first update was transmitted using the lossy compression algorithm; and merge the first queue item into the queue.

[006f] In another embodiment, the frame transfer engine is further configured to:

determine that the transmission of the processed first update has completed;

3a

CA 2907992 2017-09-27 determine that the first image compression algorithm is a lossless compression algorithm; process a fifth update using the lossless compression algorithm to obtain a processed fifth update; and transmit the processed fifth update to the target computer system via the network connection.

j006g) Another embodiment of the present invention provides a method for updating a remote display comprising: receiving a first display frame and a second display frame of a source display; generating, using a computer processor, a first queue item comprising the first display frame, and a second queue item comprising the second display frame; generating a first query using the first queue item; querying an algorithm database with the first query; receiving, in response to the first query, a first algorithm identifier identifying a first image compression algorithm; processing, using the computer processor, the first display frame using the first image compression algorithm to obtain a processed first display frame; merging the first display frame with the second display frame to obtain a merged display frame; generating, using a computer processor, a third queue item comprising the merged display frame; generating a second query using the third queue item; querying an algorithm database with the second query; receiving, in response to the second query, a second algorithm identifier identifying a second image compression algorithm; processing, using the computer processor, the merged display frame using the second image compression algorithm to obtain a processed merged display frame; and transmitting the processed merged display frame to a target computer system via a network connection, |006h] In one embodiment, generating the first query using the first queue item comprises: obtaining a first display frame size from the first queue item; determining a first bandwidth of the network connection; and generating the first query using the first display frame size and the first bandwidth.

3b

CA 2907992 2017-09-27

[Û06ÎJ In another embodiment, the method further comprises; before the transmission of the processed first display frame is complete, receiving a third display frame from the source display; generating a second queue item comprising the third display frame and a second timestamp indicating a time at which the third display frame was received; and aborting the transmission of the processed first display frame based on a determination that transmission of the third display frame following the transmission of the first display frame will exceed the transfer guarantee time.

[0(16j] In another embodiment, the method further comprises; determining that the transmission of the processed first display frame has completed; determining that the first image compression algorithm is a lossless compression algorithm; processing a fourth display frame using the lossless compression algorithm to obtain a processed fourth display frame; and transmitting the processed fourth display frame to the target computer system via the network connection.

|0(16k| A still further embodiment of the present invention provides a method for updating a remote display comprising: receiving a plurality of display frames of a source display; storing, using a computer processor, at least some of the display frames in a queue; selecting a plurality of image compression algorithms from an algorithm database; and performing, using the plurality of image compression algorithms selected, a single update of the remote display for current transfer over a network connection based on a bandwidth of the network connection, a size of the display frame, and sizes and times of display frames currently present in the queue.

3c

CA 2907992 2017-09-27

Other aspects of the invention will be apparent from the following description anil the appended claims.

BRIEF DESCRIPTION OF DRAWINGS (008J FIG. 1 shows a system in accordance with one or more embodiments of the invention.

[009] FiG. 2 shows a frame transfer engine in accordance with one or more embodiments of the invention.

foot 9] FiG. 3 Shows a transfer queue in accordance with one or mo it embodiments of the invention.

[0011] FIG. 4 shows a flow chart in accordance with one ortnore embodiments of the invention.

(4012) FIG. 5 shows a flew chart in accordance with one or more embodiments of the invention.

(00131 FiG. 6 shows a i!ow chart in accordance with one or more embodiments of the invention.

[0014) FiG. 7A shows a flow diagram in accordance with one or more embodiments of the invention.

(0015] FIG. 7B show an example timeline in accordance with one or more embodiments of the invention.

|00t 61 FIG. fi shows a computer system iu accordance will¹, one or more embodiments of the invention.

3d

CA 2907992 2017-09-27

CA Q2907ÏS2 2015-09-23

WO 2014/139947

PCT/EP2014/O54575

DETAILED DESCRIPTION

[0017) Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

(0018) In the following detailed description of embodiments of the invention, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0019} In general, embodiments of the invention provide a method and system for transmitting video between computer systems and across a network. In one aspect, embodiments of the invention may be used to send a mirror image of the computer display from one computer to another in such a way as the displays are perceived as being updated simultaneously. Specifically, embodiments of the invention provide a method and system for transmitting frames of a source display over a network when the source display is updated at inconsistent intervals. Embodiments of the invention transmit updates by varying the compression quality while maintaining a minimum frame rate.

[0020) For example, in one embodiment of the invention, one user may interact with a software program executing on that user’s computer, and a second user can observe the interactions on a second computer, while both the second user and the second computer are in a physically remote location, Each interaction by the first user causes an update to the user’s local display, and embodiments of the invention may be used to transmit that update to the second user’s computer for display to the second user.

[0021) FIG. 1 shows a diagram of a system in accordance with one or more embodiments of the invention. As shown in FIG. 1, the system includes a server computer system (100) and a client computer system (102). Tlie server

CA O29O7Ï32 2015-09-23

WO 2014/139947

PCT/EP2014/054575 computer system ( 100} includes an operating system (104) executing one or more applications (application A (106A), application N ( 106N)) in accordance with one or more embodiments of the invention. The server computer system (100) also includes a source display (108) used by the operating system to output a visual interface to a display device (not shown) in accordance with one or more embodiments of the invention. The server computer system (100) further includes a frame transfer engine (110). In one or more embodiments of the invention, a client computer system (102) includes a frame receiver engine (112) operatively connected to a target display (114). The frame receiver engine (112) is connected to the frame transfer engine (110) via a network (116) (e.g., a network connection, a communication channel, a network channel, a connection channel, communication link, etc). Each of these elements is described below¹.

|O022) In one or more embodiments of the invention, the server computer system (100) is any computing device with a computer processor configured to execute the frame transfer engine (110). The server computer system (100) may be a desktop computer, a remote server, or a virtual server executing within a virtual machine, Similarly, in one or more embodiments of the invention, the server computer system (100) may be a portable computing system such as a laptop computer, smartphone, tablet computer, or personal digital assistant. In one embodiment of the invention, the server computer system (100) may be a special purpose device, such as a still or video camera, enabled to communicate with the client computer system (102) over the network ( 116).

10023) In one or more embodiments of the invention, the server computer system (100) communicates with a client computer system (102) over a network (116), such as the Internet, The client computer system (102) may be a desktop computer or a portable computing system such as a laptop computer, smartphone, tablet computer, or personal digital assistant. The client computer system (102) may also be implemented as a special purpose device, such as a television or other type of display device with hardware capable of executing

CA Q£?O7»92 2015-9^-23

WO 2014/139947

PCT/EP2014/054575 the frame receiver engine (112). Alternatively, in one embodiment of the invention, the client computer system ( 102) may be a set-top box attached to a display device.

|0024| In one or more embodiments of the invention, an operating system (104) is a program or set of programs configured to provide access to resources on the server computer system (100) to applications (application A (106A), application N (106N)). Such resources include hardware resources such as a processor, random access memory, persistent memory, network interface devices, display devices, and input devices such as a keyboard, mouse, or touchscreen.

|0025| Continuing with FIG. 1, the operating system (104) provides applications (application A (106A), application N (106N)) the ability to display text or graphics on a display device via the source display (108) in accordance with one or more embodiments of the invention. In one embodiment of the invention, the source display (108) is a combination of hardware and software used to transmit graphical output elements to a display device (not shown). Specifically, the source display (308) may be a set of locations in memory used by the operating system (104) to transmit graphical data to a display device. The source display (108) may include processing logic, such as a graphics processing unit, used to render a display frame for presentation on the display device. In one embodiment of the invention, tlie source display (108) is integrated into the operating system (104). In one embodiment of the invention, the source display (108) includes the functionality to provide display frames to the frame transfer engine (110). In some embodiments of the invention, the source display (1 OR) may be a “virtual display” without an associated physical device connected to server computer system (100) (e.g., a server without a video card, a smartphone having a low-resolution display and a virtual source display of a much higher resolution).

1(1026) In one or more embodiments of the invention, tlie frame transfer engine (110) is a program configured to obtain a series of display frames from the source

CA Q2S07ÏS2 2015-0^-23

WO 2014/139947

PCT/EP2014/054575 display (108), and initiate a transfer of the display frames to the frame receiver engine (112). In one or more embodiments of the invention, the frame transfer engine (110) is executed by the operating system (104) in the same or similar manner as other applications (application A (106A), application N (106N)). Alternatively, in one embodiment of the invention, the frame transfer engine (110) operates as part of the operating system (104) (for example, as a series of kernellevel processes), in one embodiment of the invention, frame transfer engine (110) may be implemented in hardware (for example, as an application-specific integrated circuit). Further detail regarding the frame transfer engine (110) is shown in relation to FIG. 2 and described below.

(0927) In one or more embodiments of the invention, each display frame captured by the frame transmission engine corresponds to the entire visual display presented to a user of the server computer system. Alternatively, in one embodiment of the invention, each display frame corresponds to a portion of the visual display at which an update was made (e.g., location on a screen where text was added to an electronic document), in one embodiment of the invention, the frame transmission engine captures only the visual display corresponding to a single application (e.g., a single window displayed on an operating system instead of all windows displayed).

10928) In one embodiment of the invention, the display frames are transmitted from the frame transfer engine (110) to the frame receiver engine (112) in a queue items. Further, although FIG. 1 depicts the frame transfer engine (110) to the frame receiver engine (112) as communicating directly via the network (116), in one embodiment of the invention, the queue items (or elements extracted from the queue items) are transmitted using a network stack (not shown) within the operating system <104), and received by a corresponding network stack (not shown) on the client computing system (102). In one embodiment of the invention, the queue items are divided into data packets, and sent to the client computing system (102) using one or more network protocols,

CA O29O7Ï32 2015-09-2J

WO 2014/139947

PCT/EP2014/O54575

[0029J in one or more embodiments of the invention, the network (116) includes any link (such as a communication channel, network connection, network channel, communication link, etc.) between the server computer system (100) and the client computer system (102). In one embodiment of the invention, the network (116) may host different network protocols, and different network protocols may be implemented at different network layers. For example, at the transport layer, such network protocols may include, but are not limited to, transmission control protocol (TCP) and/or user datagram protocol (UDP). In addition, network (116) may be implemented as a link directly between devices, such as a Universal Serial Bus (USB) link, a Bluetooth link (Bluetooth* is a registered trademark of Bluetooth SIG, Inc.), or any other type of device communications protocol.

10030) In one or more embodiments of the invention, the frame receiver engine (112) is a combination of hardware and software executing on the client computer system (102) used to receive and decode the display frames. In one or more embodiments of the invention, the display frames may be received encoded in common graphic formats (such as joint photographic experts group (JPEG) and/or portable network graphics (PNG)), which are subsequently decoded by the frame receiver engine ( 112).

10031) In one or more embodiments of the invention, the target display ( 114) is a combination of hardware and software used to present the display frames decoded by the frame receiver engine (112). The target display (114) may include processing logic, such as a graphics processing unit, used to render a display frame for presentation on a display device operatively connected to the client computer system (102). in one embodiment of the invention, the target display (114) is integrated into a client operating system (not shown). In some embodiments of the invention, the target display ¢114) may be a “virtual display” without an associated physical device connected to client computer system (102) (e.g.. a client without a video card, etc.).

CA Q2S07ÏS2 2O15-09-2J

WO 2014/139947

PCT/EP2014/054575

[0032) FÏG. 2 shows a diagram of a frame transfer engine (such as the frame transfer engine (110) in FIG. 1) in accordance with one or more embodiments of the invention. As shown in FIG. 2, the frame transfer engine (200) includes a queue item generator (202), a transfer queue (204), an image compression engine (206), a transmission engine (208), an update engine (210), and an algorithm database (212), The algorithm database (212) stores a range of compression algorithms that includes a lowest quality (LQ) algorithm (214) and a highest quality (HQ) algorithm ¢216). Each of these elements is described below.

|0033| In one or more embodiments of the invention, the queue item generator (202) includes functionality to receive notice that the source display has been updated, and obtain a display frame corresponding to the updated portion of the source display. In one or more embodiments of the invention, the queue item generator (202) generates a queue item that may include the received display frame, a location of the source display to which the display frame corresponds, a timestamp corresponding to the time tlie queue item generator (202) received the display frame, and a lower quality transfer flag (LQTF) initially set to false. Once generated, the queue items may be merged with the remaining queue items in the transfer queue. More detail regarding the queue items and transfer queue is provided in regard to FIG. 3. In one embodiment of the invention, a queue item is referred to as an update.

(00341 Continuing with FIG. 2, the image compression engine (206) is configured to obtain a queue item from the transfer queue (204) and apply a compression algorithm to the display frame to generate a processed queue item in preparation for transfer by the transmission engine (208) in accordance with one or more embodiments of the invention. The image compression engine (206) is notified by the update engine (210) regarding which compression algorithm to apply,

[0035) In one embodiment of the invention, the transmission engine (208) is configured to obtain the processed queue items from the image compression

CA 02907Ï32 2015-09-2J

WO 2014/139947

PCT/EP2914/054575 engine (206) and initiate a transfer of the processed queue item to a client computer system, in one embodiment of the invention, the transmission engine (208) uses a network interface provided by the operating system to initiate the transfer of the processed queue item. In one embodiment of the invention, tlie processed queue item is divided into datagrams or packets and transferred across a network link before being reassembled on the client computer system. In one or more embodiments of the invention, the transmission engine (208) provides a recent measurement of the bandwidth (BW) between the server computer system and the client computer system. In one or more embodiments of the invention, the transmission engine (208) is configured to provide an indication to the update engine (210) regarding whether a transfer of a queue item is currently underway.

(0036} In one or more embodiments of the invention, the update engine (210) is configured to gather information from the transfer queue (204), image compression engine (206), and the transmission engine (208) to generate a query for the algorithm database (212). In one embodiment of the invention, the update engine (210) obtains a display frame size (the size of the corresponding display frame in square pixels) and a display frame target size (desired size in bytes of the display frame in the processed queue item). The update engine (210) submits the display frame size and the display frame target size in a query to the algorithm database (212) to obtain an identifier of a compression algorithm which may be applied to the display frame in order to produce a processed display frame that is a substantially similar size to the submitted display frame size target.

(0037} Continuing with FIG. 2, the algorithm database (212) is a collection of searchable algorithm identifiers each associated with an algorithm, in accordance with one or more embodiments of the invention. Each algorithm is associated with a typical compression ratio which allows the image compression engine (206) to calculate the display frame target size in bytes based on the display frame size in square pixels. The algorithm identifier

CA O29O7Ï32 2O15-09-2J

WO 2014/139947

PCT/EP2014/O54575 corresponding to the algorithm with the lowest compression ratio is the highest quality (HQ) algorithm (216). In one embodiment of the invention, the HQ algorithm uses lossless compression (such as PNG). In one or more embodiments of the invention, HQ algorithm (216) can be without compression. The algorithm identifier corresponding to the algorithm with the highest compression ratio is ihe lowest quality (LQ) algorithm (214). Upon evaluating a query that would require a compression ratio higher than that of the LQ algorithm (214), the algorithm database may be configured to respond with an algorithm identifier indicating “no algorithm” or “null.” It should be noted that in some embodiments of the invention, the algorithm database (212) may try using actual algorithms with actual pixel data to calculate the optimal algorithm, instead of using typical compression ratios. It should ftirther be noted that algorithm database (212) can be implemented in many ways including, but not limited to, a list of algorithms embedded into the source code of a program or a collection of files.

$038} In one or more embodiments of the invention, each compression algorithm may correspond to a conversion process for transforming the display frame from its initial format (which may be similar to a plain uncompressed red-green-blue (RGB) bitmap format) into a different or altered format. In one or more embodiments of the invention, display frames are processed into lossless formats (i.e., highest quality) or lossy formats (i.e., lower quality). Further, different types of lossy compression processes result in varying levels of resulting quality. A high quality lossy compression may result in a graphic image nearly visually indistinguishable from the same image processed using a lossless compression process. In addition, graphics processed using lossless compression are generally larger than the same graphics processed using a lossy compression. Similarly, graphics processed using a higher quality lossy compression are generally larger than the same graphics processed using a lower quality lossy compression. Examples of different lossless and lossy compression processing algorithms include, but are not limited to, discrete

CA Q2S07Ï82 2015-0^-23

WO 2014/139947

PCT/EP2014/O54575 cosine transform (DCT), Fourier transform, and various algorithms based on Lempel-Ziv (LZ), such as LZ77, Huffman and'or arithmetic encodings, or any combination of these algorithms. Resulting image types include, but are not limited to, bitmap images, PNG images, and JPEG images.

10039) in one or more embodiments of the invention, the display frame size target may be calculated using the most recent measurement of the BW and the maximum amount of time allotted for the transfer. For example, a query targeting the algorithm database (212) may include a display frame size of X square pixels, and a display frame size target of Y kilobytes (kBytes). Alternatively, tlie query may instead include a display frame size of X square pixels, and a display frame target size equal to tlie current BW times the time available to transfer. As another example, a query targeting the algorithm database (212) may include a display frame size of 20,000 square pixels (which corresponds to 100x200 pixels), and a display frame size target equal to the number of bytes that may be transferred over a connection with a BW of 1 kilobytes/second during a target time period of 5 seconds, which would be 5 kBytes.

|0040) FIG. 3 shows a diagram of a transfer queue (such as transfer queue (204) in FIG, 2) in accordance with one or more embodiments of the invention. As shown in FIG. 3, a transfer queue (300) includes a number of queue items (queue item A (302A), queue item N (302 N)). Each queue item (queue item A (302A), queue item N (302N)) includes a display frame (display frame A (304A), display frame N (304N)), a location (location A (306A), location N (306N)), a timestamp (timestamp A (308A), timestamp N (308N)), and a lower quality transfer flag (LQTF) (LQTF A (310A), LQTF N (310N)). Each of these elements is described below.

[00411 In one or more embodiments of the invention, the display frame (display frame A (304A), display frame N (304N)) corresponds to an image received from the display source by the queue item generator. In one or more embodiments of the invention, the display frame (display frame A (304A), display frame N (304N)) may be a portion of the entire source frame displayed

CA O2SO7ÏS2 2Û15-QS-2J

WO 2014/139947

PCT/EP2014/054575 at a given moment in time via the display source. For example, a display frame (display frame A (304A), display frame N (304Nj) may be an image 100 pixels wide and 200 pixels tall corresponding to the lower left comer of the entire source frame. In one or more embodiments of the invention, the display frame (display frame A (304A), display frame N (304N)) corresponds to the portion of the entire source frame where a change in the source frame has occurred. For example, the display frame (display frame A (304A), display frame N (304N)) obtained by the queue item generator may correspond to the portion of the screen that displays the time of day, and may have been obtained in response to a detection that the portion of the screen has changed (e.g., because the minute value displayed has just been updated).

100421 In one or more embodiments of the invention, the location (location A (306A), location N (3O6Nj) corresponds to the position of the display frame (display frame A (304A), display frame N (304N)) relative to the entire source display. Referring again to an example display frame that is an image 100 pixels wide and 200 pixels tall corresponding to the lower left comer of the entire source frame, the location (location A (306A), location N (306N)) of that display frame may be the pixel coordinates of the upper-left comer of the entire source display frame. In one or more embodiments of the invention, the location (location A (306A), location N (306N)) may additionally describe the dimensions of the display frame. Further, in one or more embodiments of the invention, tbe location (location A (306A), location N (306N)) may describe display frame shapes with a complexity greater than that of a regular polygon (e.g., combinations of areas).

(0043) In one embodiment of the invention, the timestamp (timestamp A (308A), timestamp N (308N)) corresponds to the time or clock value at the point the display frame was received. In one or more embodiments of the invention, the age of the display frame may be calculated by subtracting the current clock value from the timestamp (timestamp A (308A), timestamp N (308N)).

CA Q2S07Ï32 2Û15-QS-2J

WO 2014/139947

PCT/EP2014/054575 (0044) In one embodiment of the invention, the LQTF (LQTF A (310A), LQTF N (3ION)) is a Boolean indicator that the display frame associated with the LQTF has previously been transmitted to the client computer system using an algorithm with an output quality less than that of the HQ algorithm. Stated differently, a LQTF equal to true indicates that the associated display frame has been transferred at less-than-full quality. In one or more embodiments of the invention, once a queue item is transferred using the HQ algorithm, then the queue item is removed from the transfer queue (300), Alternatively, once a queue item is transferred using an algorithm that is less than the HQ algorithm, the LQTF (LQTF A (310A), LQTF N (3ION)) is changed from false to true, and the queue item witl either be eventually merged with other queue items, or subsequently be transferred using the HQ algorithm and removed from the transfer queue (300).

|0045) In one embodiment of the invention, new queue items are added to the transfer queue (300) using a merging process. In one or more embodiments of the invention, queue items are merged with existing queue items in the transfer queue (300) in a variety of ways. For example, depending upon the circumstances (discussed below) a current queue item is merged with (i) only the queue items that have a LQTF equal to false, (ii) only the queue items that have a LQTF equal to true, or (iii) all other queue items in the transfer queue that have a LQTF equal to false, with a timestamp within a set period of the timestamp of the current queue item, |0046| Further, in one or more embodiments of the invention, the merging process results in the elimination or trimming of overlapping areas of the display frame. Depending upon the circumstances (also discussed below) merging a current queue item with existing queue items results in the trimming of the overlapping areas of the existing queue items. Alternatively, merging a current queue item with existing queue items results in the trimming of the overlapping areas of the current queue item. In some circumstances, merging display frames that completely overlap each other

CA Q2S07Ï32 2015-0^-23

WO 2014/139947

PCT/EP2014/054575 results in the elimination of the queue item targeted for trimming. For the puiposes of this application, a merge operation in which a current queue item is merged with all existing queue items with a LQTF equal to false, where the current queue item is targeted for trimming, is referred to as merge[LQTF=false/trim current). A merge operation in which a crurent queue item is merged with all existing queue items regardless of LQTF value, where the the existing queue items are targeted for trimming, is referred to as merge[LQTF=any/trim existing],

100471 FÏG. 4 shows a flowchart for initializing a frame transfer engine in accordance wdth one or more embodiments of the invention. While the various steps in these flowcharts are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel,

100481 In Step 410, an empty transfer queue is generated, in Step 412, the algorithm database is populated with algorithm identifiers corresponding to the available compression algorithms, in one or more embodiments of the invention, if algorithm database is hardcoded then the algorithm database need not be populated. In Step 414, the queue item generator receives an initial display frame from the source display.

(0049) FiG. 5 shows a flowchart receiving a new display frame in accordance wdth one or more embodiments of the invention. While the various steps in these flowcharts are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel.

(0050) In Step 510, the queue item generator receives a notification that the source display has been updated. Next, the queue item generator receives a display frame and generates a new queue item using the received display frame

CA 02S07Ï32 2015-0^-23

WO 2014/139947

PCT/EP2014/054575 (Step 512). In Step 514, the update manager determines whether there is currently a transfer in progress. In one or more embodiments of the invention, the update manager determines whether a transfer is currently in progress by querying the transmission engine.

|0051| If in Step 514, a determination is made (typically by the update engine) that a current transfer underway, then, in Step 516, a determination is made (typically by the update engine) whether to abort the current transfer. In one or more embodiments of the invention, the determination regarding whether to abort the current transfer requires a determination as to whether the queue items in the transfer queue may be transmitted at a rate no less than a set minimum frequency. Stated differently, the update engine determines whether at least one update will be received within a set period of time from when the update occurred. The set period of time is referred to as the guarantee time (Tguar), A transfer rate at which at least one update will be received within the Tguar is referred to as satisfying the “soft guarantee’’ of the Tguar.

(0052} In one or more embodiments of the invention, whether Tguar may be maintained at the current transfer rate is determined by calculating an abort algorithm and a continue algorithm. The continue algorithm is obtained from a query' generated for transferring the update display frame and all other frames in the transfer queue in the time remaining after the current transfer has completed. In other words, the update engine determines whether Tguar can be maintained for the update display frame if the transfer of the update display frame does not begin until the current transfer finishes.

[0053} The abort algorithm is calculated as the lower quality algorithm between two query responses. The first abort algorithm is the algorithm for transferring the update display frame and all other frames in the transfer queue within Tguar from the current time, if the transfer were to begin immediately. In other words, the update engine determines whether Tguar can be maintained for ail the existing updates if the current transfer is abandoned, and the update display frame is merged with the transfer queue and the transfer is restarted.

CA Q2S07ÏS2 2025-0^-23

WO 2014/139947

PCT/EP2014/054575

The second abort algorithm is the algorithm for transferring the update display frame within the time left from original Tguar for the current transfer. In other words, the update engine determines whether Tguar can be maintained for the current transfer if the current transfer is abandoned and restarted. If both of the abort algorithms result in a higher quality compressed image than the continue algorithm, then the update engine aborts the current transfer.

|0954| if the current transfer is not aborted, then in Step 518, the new queue item is added to the transfer queue using merge[LQTF=any/trim existing]. In Step 520, as a result of the merge operation, older queue items that have been previously transmitted (i.e., LQTF is true) and have been overwritten by more recent updates are reduced in size or eliminated completely.

|9955| If, in Step 516, a determination is made that transfer should be aborted, then in Step 522, the current transfer is aborted. The transfer may be aborted, for example, if large portions of the source display are being updated.

|0956| If there is no transfer in progress in Step 514, or if the transfer has been aborted in Step 522, then the update engine queries the algorithm database using the current BW, and a time period of Tguar (Step 524). At Step 526, the new queue item is merged with the transfer queue using merge[LQTF=any/trim existing]. As a result of the merge operation, older queue items that have been previously transmitted and have been overwritten by more recent updates are reduced in size or eliminated completely (Step 528). In Step 530, the transfer of the new queue item is initiated using the new algorithm,

19957) FiG, 6 shows a flowchart for determining the next queue item to be sent in accordance with one Or more embodiments of the invention. While the various steps in these flowcharts are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel,

CA &2907932 2015-09-23

WO 2014/139947

PCT/EP2014/054575

10058) In one or more embodiments, notification is received (typically by the update engine) that a transfer of a queue item has completed (Step 610). In Step 612, a determination is made (typically by the update engine) whether the completed transfer was performed using the HQ algorithm. If in Step 612, a determination is made that the HQ algorithm was used, then the current queue item that has just completed transmission to the client computer system is removed from the transmission queue. The queue item is removed because the target screen has already been updated with the highest quality display frame, and any further updates of that portion of the target display that do not represent updates to the source display (but rather only improvements to the clarity or sharpness of the image) are unnecessary.

[0059) If, at Step 612, a determination is made (typically by the update engine) that the HQ algorithm was not used, then in Step 616, the LQTF of the current queue item is set to true. In one or more embodiments of the invention, setting the LQTF to true indicates that a lower quality version of the display frame has already been sent to the client computing device, and the display frame may be resent at a higher quality if resources allow (and thus clarifying or sharpening the corresponding portion of the target display).

10000) At Step 618, tire current item is merged into the transfer queue using merge[LQTF=faIse/trim current]. As a result, any area of the display frame in the current queue item that overlaps with areas of any display frames in queue items that have not been transferred at all, are removed from the current queue item. In one or more embodiments of the invention, the current queue item may be removed completely if the entire display frame is overlapped by portions of display frames in other queue items yet to be transferred.

|006l) At Step 620, the update engine determines whether the queue is empty. If the queue is empty, then no further action is necessary until the source display is updated . If the queue is not empty, then in Step 622, a determination is made whether at least one queue item has a LQTF equal to false has not been transmitted to client computer system at all), if no

CA 02907932 2015-09-23

WO 2014/139947

PCT/EP2014/O54575 queue item exists with a LQTF equal to false, then at Step 624, the oldest queue item is transferred across a network connection or communication channel using the HQ algorithm, in one or more embodiments of the invention, when the transfer queue contains only queue items with a LQTF equal to true, then all updates to the target display have been transmitted, even if they were transmitted at a low image quality. The remaining queue items are then transferred in order to clarify or sharpen the image shown on the target display.

[0062} If in Step 622, a determination is made (typically by the update engine) that at least one queue item has a LQTF equal to false, then the update engine queries the algorithm database to obtain a first algorithm (Step 626). In one or more embodiments of the invention, the fust algorithm is used to determine whether the Tguar can be maintained for both the next queue item and all pending queue items in the transfer queue. In one or more embodiments of the invention, the first algorithm is determined by obtaining (i) the algorithm for all non-transfened items in the queue using a display frame target size of the current BW multiplied by Tguar (this ensures that, unless new updates come in, all items together will satisfy Tguar “soft guarantee”) and (ii) the algorithm for the first non-transfereed item using a display frame target size of the current BW multiplied by the Tguar remaining for the first lion-transferred item (this ensures that first item in queue will satisfy Tguar “soft guarantee”). If either of the queries is returned from the algorithm database as no algorithm (i.e., the algorithm to satisfy the query would have a lower quality then LQ algorithm), then the update engine merges all non-transfened items in the queue. Subsequently, the update engine then queries the algorithm database using the resulting merged queue item, the current BW, and the Tguar. The merged queue item would then be transferred across the network connection or communication channel using the resulting algorithm,

10063} If, in Step 628, a determination is made (typically by the update engine) that there exists an algorithm in the algorithm database such that Tguar can be

CA Q2907Ï32 2Û15-0S-2J

WO 2014/139947

PCT/EP2014/054575 maintained for both the next queue item and all pending queue items in the transfer queue, then the next queue item is transferred across a network connection or communication channel using the lower quality of tire two determined algorithms (Step 630).

10064] If a determination is made (typically by the update engine) that either the next queue item or the remaining queue items require an algorithm with a lower resulting quality than the resulting quality of LQ algorithm (Step 628), then the current bandwidth is not sufficient to satisfy Tguar “soft guarantee” at the current rate of updates. In this case, in some embodiments, ail queue items in the transfer queue with a LQTF equal to false are merged (Step 632). In Step 634, one or more queries of the algorithm database are made (typically by the update engine) using a display frame size of the next queue item, and a display frame target size as the current BW times the Tguar. In Step 636, the next queue item is transferred across a network connection or communication channel using the algorithm identified from the query result, (0065) FIG. 7A shows an example in accordance with one or more embodiments of the invention. The example shown in FIG. 7A is for explanatory purposes only and not intended to limit the scope of the invention. In the example depicted in FIG. 7A, the system includes a server computer (700) and a tablet computer (702), The server computer (700) includes an operating system ¢704) executing application (706), The server computer (700) also includes a source display (708) used by the operating system to output a visual interface to a display device (not shown). The server computer (700) further includes frame transfer engine (710), Tablet (702) includes a frame receiver engine (712) operatively connected to a tablet display (714), which outputs the received video display to a tablet screen (720). The frame receiver engine (712) is connected to the frame transfer engine (710) via a network connection (716), such as a wired or wireless communication link.

(0066) FIG. 7B shows an example timeline in accordance with one or more embodiments of the invention. The example timeline in FÏG. 7B depicts an

CA 02907ÏS2 2015-09-23

WO 2014/139947

PCT/EP2014/054575 interaction between the elements depicted in FIG. 7 A, The example timeline shown in FIG. 7B is for explanatory purposes only and not intended to limit the scope of the invention.

|0067| In Step 750, the queue item generator (722) of the frame transmission engine (710) receives display frame UÛ and prepares a queue item for UO, In Step 752, the update engine (730) is notified of the update, and determines whether a transfer is currently in progress. For the purposes of this example, there is no current transfer in progress. The update engine (730) then determines a new transfer algorithm by querying the algorithm database. The update engine 730 obtains the current BW from the transmission engine, and generates a query using the size of UO and display frame target size calculated as the guarantee time (Tguar) multiplied by the current BW. For the purposes of this example, assume the size of U0 is 20,000 square pixels, the Tguar is 25 milliseconds (ms), and the current BW is 700 kilobits per second (or .8 bits per ms). The display frame size target, therefore, is calculated to be 20 kilobits. The algorithm database responds to the queiy with an algorithm identifier of a JPEG algorithm to convert UO. The update engine (730) configures the image compression engine to use the JPEG algorithm, and initiates the transfer of U0 to the tablet computer (714). A compressed version of U0 is generated resulting in a JPEG with a size of approximately 20 kilobits.

|0068| in Step 754, the transmission of the compressed version of U0 across a community channel is initiated. At Step 756, the frame receiving engine (712) on the tablet computer (702) receives the compressed version of U0, In Step 758, the compressed version of U0 is decoded and rendered as F0 by the tablet display (714). In Step 760, the tablet computer (702) sends an acknowledgement across the network connection or communication channel that the transfer has completed.

(0069) In Step 762, the queue item generator (722) of the frame transmission engine (710) receives display frame Ul and prepares a queue item for Ul. In Step 764, the update engine (730) is notified of the update, and determines whether a transfer is currently in progress. Because the server computer (700)

CA Q2S07Ï32 2Û15-0S-2J

WO 2014/139947

PCT/EP2014/054575 has not yet received an acknowledgement of completion from the tablet computer (702), the update engine (730) determines that a transfer is still in progress.

[0070) In Step 764, the update engine ¢730) determines whether to abort the current transfer. For the purposes of the example, assume that frame transmission engine (710) determines whether to abort the current transfer using a comparison of a continue algorithm and an abort algorithm. The continue algorithm is obtained from a query generated for transferring the update display frame Ul and all other frames in the transfer queue (except U0) in the time remaining after the current transfer has completed. In other words, the update engine determines whether Tguar can be maintained for Ul if the transfer of Ul does not begin until the current transfer finishes.

|0071) The abort algorithm is calculated as the lower quality algorithm between two query responses. The first abort algorithm is the algorithm for transferring the update display frame Ul and all other frames in the transfer queue (including UO) within the Tguar from the current point in time, if the transfer were to begin immediately. In other words, the update engine determines whether Tguar can be maintained for all the existing updates if the current transfer is abandoned, and U0 is merged with Ul and the transfer is restarted. The second abort algorithm is the algorithm for transferring update display frame U0 within the time left from original Tguar guarantee for U0. In other words, the update engine determines whether Tguar can be maintained for U0 if the current transfer is abandoned, and the transfer is restarted.

|0β72| If both of the abort algorithms result in a higher quality compressed image than the continue algorithm, then the update engine (730) aborts the current transfer. For the purposes of this example, assume that the continue algorithm results in a higher quality algorithm than at least one of the abort algorithms. The updated engine ¢730), therefore, merges Ul into the transfer queue. Although the transfer queue is empty in the present example, any

CA O29O7Ï32 2O25-Q9-2J

WO 2014/139947

PCT/EP2014/054575 current queue items in the transfer queue that have already been transferred (i.e., LQTF is true) that overlap with Ut would be trimmed or eliminated.

|O073| In Step 766, the update engine (730) is notified that the transfer of UO has completed. The update engine (730) must now determine which queue item to transfer next. The update engine (730) determines whether UO was transferred using the highest quality algorithm, which it was not. The update engine (730) then sets the LQTF of UO to true, and merges UO into the transfer queue. The update engine (730) then determines that the queue is not empty (it includes both U0 and Ul). The update engine (730) determines that at least one queue item has a LQTF equal to false, because Ul has not be transferred yet. The update engine then obtains the first algorithm by obtaining (i) the algorithm for all non-transfened items in the queue using a display frame target size of the current BW multiplied by Tguar (this ensures that all current items will satisfy Tguar “soft guarantee,” unless new updates are received) and (ii) the algorithm for the first non-transferred item using a display frame target size of the current BW multiplied by the Tguar remaining for the first nontransfened item (this ensures that first item in queue will satisfy Tguar “soft guarantee”). If either of the queries is returned from the algorithm database as no algorithm (i.e., the algorithm to satisfy the query wOirid have a lower quality then LQ algorithm), then the update engine (730) would merge all nontransferred items in the queue, and would then query the algorithm database using the resulting merged queue item, the current BW, and Tguar. In that case, the merged queue item would then be transferred across the network connection or communication channel using the resulting algorithm.

(0074) For the purposes of this example, assume that neither of the first algorithms is returned from the algorithm database as no algorithm. In Step 768, the update engine (730) initiates the transfer of Ul using the lower-quality first algorithm. In Step 770, the transmission of the compressed version of Ul across the network connection or communication channel is initiated. At Step 772, the frame receiving engine (712) on the tablet computer (702) receives the

CA Q2S07ÏS2 2015-09-2J

WO 2014/139947

PCT/EP2014/O54575 compressed version of UI. In Step 774, the compressed version of UI is decoded and rendered as FI by the tablet display (714), |O075| Embodiments of the invention may be implemented on virtually any type of computer regardless of the platform being used. For example, as shown in Fig, 8, a computer system ¢800) includes one or more processors) (802) such as a central processing unit (CPU) or other hardware processor(s), associated memory (804) (e.g., random access memory (RAM), cache memory, flash memory, etc.), a storage device (806) (e.g., a hard disk, an optical drive such as a compact disk drive or digital video disk (DVD) drive, a flash memory stick, e/c.), and numerous other elements and functionalities typical of today’s computers (not shown). In one or more embodiments of the invention, the processor (802) is hardware. For example, the processor may be an integrated circuit. The computer system (800) may also include input means, such as a keyboard (808), a mouse (810), or a microphone (not shown). Further, the computer system (800) may include output means, such as a monitor (812) (e.g., a liquid crystal display (LCD), a plasma display, or cathode ray tube (CRT) monitor). The computer system (800) may be connected to a network (814) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, or any other type of network) via a network interface connection (not shown). Those skilled in the art will appreciate that many different types of computer systems exist, and the aforementioned input and output means may take other forms. Generally speaking, the computer system (800) includes at least the minimal processing, input, and/or output means necessary to practice embodiments of the invention.

(0076) Further, those skilled in the art will appreciate that one or more elements of the aforementioned computer system (800) may be located at a remote location and connected to the other elements over a network. Further, embodiments of the invention may be implemented on a distributed system having a plurality of nodes, where each portion of the invention may be located on a different node within the distributed system. In one embodiment of the

CA Q2907Ï32 2O15-09-2J

WO 2014/139947

PCT/EP2014/054575 invention, the node corresponds to a computer system. Alternatively, the node may correspond to a processor with associated physical memory. The node may alternatively correspond to a processor or micro-core of a processor with shared memory and/or resources. Further, software instructions in the form of computer readable program code to perform embodiments of the invention may be stored, temporarily or permanently, on a non-transitory computer readable storage medium, such as a compact disc (CD), a diskette, a tape, memory, or any other computer readable storage device,

[00771 While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.

[0078} Wide the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, wili appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.

The embodiments ofthe present invention for which an exclusive property or privilege is claimed are defined as follows:

1. A system comprising:

a source display, externally updatable;

an algorithm database comprising a plurality of image compression algorithms; a network connection; and a frame transfer engine executing on a computer processor and configured to: receive a plurality of updates made to the source display, wherein the plurality of updates comprises at least a first update and a second update;

store at least the first update and the second update in a queue;

select, based on a first bandwidth of the network connection, a size of the first update, and sizes and times of updates currently present in the queue, a first image compression algorithm in the algorithm database for current transfer over the network connection;

process the first update using the first image compression algorithm to obtain a processed first update;

merge the first update with the second update to obtain a merged update; select a second image compression algoritlim in the algorithm database for current transfer over the network connection based on a second bandwidth of the network connection, a size of the merged update, and sizes and times of updates currently present in the queue;

process the merged update using the second image compression algorithm to obtain a processed merged update; and transmit the processed merged update to the target computer system via the network connection.

2. The system of claim 1, wherein the frame transfer engine uses transfer guarantee time (Tguar) to determine which image compression algorithm shall be used for the current transfer of die update in the queue, wherein Tguar comprises a maximum time which any update is allowed to stay within the frame transfer engine before it is completely transferred.

CA 2907992 2017-09-27