Efficient encoding of alternative graphic sets
22 claims: 4 independent, 18 dependent
- 1分散コンピューティングシステムにおいて、使用されるグラフィック言語の種類にかかわらず、符号化メカニズムを決定することによって、ローカルデバイス上で実行される1つまたは複数のアプリケーションに関する、リモートデバイス上で表示させるグラフィック・セットを符号化するコンピュータ実行方法であって、 ローカルデバイス上で1つまたは複数のアプリケーションを実行し、グラフィック・セットを生成するステップであって、前記グラフィック・セットは、前記1つまたは複数のアプリケーションに関するグラフィック表示の少なくとも一部をレンダリングする際に使用される、1つもしくは複数のリソース、1つもしくは複数の表示命令、または前記リソースおよび前記表示命令の両方を含む、ステップと、 前記アプリケーションが、1つまたは複数のデータ圧縮モジュールが前記グラフィック・セットを圧縮することを助けるために使用される、前記グラフィック・セット内の1つまたは複数のフィールドの内容を記述するメタデータを抽出するステップと、 前記アプリケーションが、前記グラフィック・セットに対して圧縮の種類を選択する、前記リモートデバイスによってサポートされる1つまたは複数のデータ圧縮メカニズムを記述する符号化種類データを、前記リモートデバイスから受信するステップと、 前記アプリケーションが、前記抽出されたメタデータおよび前記受信された符号化種類データに基づいて、前記グラフィック・セットの1つまたは複数の部分に対して符号化メカニズムを決定するステップと、 前記アプリケーションが、前記決定された符号化メカニズムに基づいて、前記グラフィック・セットの1つまたは複数の部分を符号化するステップと を含むことを特徴とするコンピュータ実行方法。
- 2前記1つまたは複数の表示命令は、ディスプレイ上の位置付け、サイズ、色、または形状のうちの1つまたは複数の観点でリソースを記述する1つまたは複数のプロパティを含むことを特徴とする請求項1に記載のコンピュータ実行方法。
- 3前記メタデータに基づいて、前記フィールドのうちの1つまたは複数 を、 マシンフォーマットからネットワークフォーマットに変換し、前記1つまたは複数のフィールドのビット長を削減することを特徴とする請求項1に記載のコンピュータ実行方法。
- 4前記メタデータに基づいて、前記フィールドのうちの1つまたは複数 を 、可変長の形態で符号化することを特徴とする請求項1に記載のコンピュータ実行方法。
- 5前記メタデータに基づいて、差分符号化を使用し、リソースの変化だけを前記リモートデバイスに送信するように、前記グラフィック・セット全体よりも少ないバイト数を使用して符号化することを特徴とする請求項1に記載のコンピュータ実行方法。
- 6前記メタデータに基づいて、インターオーダ圧縮メカニズムを使用し、特定の種類の前記フィールドのうちの1つまたは複数を、 以 前に符号化された 同じ種類の 構造に基づいて符号化することを特徴とする請求項1に記載のコンピュータ実行方法。
- 7前記リモートデバイスによってサポートされる前記1つまたは複数のデータ圧縮メカニズムを、接続時にローカルデバイスとリモートデバイスとの間でネゴシエーションし、選択される前記符号化メカニズムは、 前記データ 圧縮メカニズム のそれぞれ の 、 所与のグラフィック・セットに対する圧縮比にさらに基づくことを特徴とする請求項1に記載のコンピュータ実行方法。
- 8前記1つまたは複数のデータ圧縮モジュールは、RLE、MPEGベース、JPEG、GIF、ZIP、LZベース、JBIG、DejaVuに基づく圧縮メカニズムのうちの1つまたは複数を含むことを特徴とする請求項7に記載のコンピュータ実行方法。
- 9分散コンピューティングシステムにおいて、1つまたは複数のアプリケーションに関するどのリソースをリモートデバイスに送信するべきかを判定することによって、ローカルデバイス上で実行中の前記アプリケーションに関するグラフィック表示を 前記 リモートデバイス のディスプレイ 上にレンダリングさせるために 前記 リモートデバイスに送信するコンピュータ実行方法であって、 前記 ローカルデバイス上で 、前記 アプリケーションを実行して、1つもしくは複数のリソース、1つもしくは複数の表示命令、または前記リソースおよび前記表示命令の両方を含むグラフィック・セットを生成するステップであって、前記グラフィック・セットは、前記アプリケーションに関するグラフィック表示の少なくとも一部をレンダリングする際に使用される、 生成する ステップと、 前記アプリケーションが、(1)記憶されるリソースの種類にかかわらず、前記グラフィック・セットに対応する1つもしくは複数のリソースが前記リモートデバイスに送信され、再利用の目的で集中キャッシュに記憶されたかどうかを判定する配信状態情報、(2)前記ローカルデバイスから少なくとも1つのリソースを転送することなしに前記アプリケーションに関する前記少なくとも1つのリソースを前記リモートデバイス上に表示するために前記リモートデバイス上で現在利用可能な専用のリソースを判定する、サポートされるアプリケーションの情報、(3)帯域幅もしくはシステム制約を節約するために、画質を落としたリソースを最初に送信し、前記画質を落としたリソースを改善するアップデートをその後送信するように、リソースの一部を前記リモートデバイスに漸進的に送信するべきであるかどうかを判定するシステム制約データ、または(4)前記グラフィック・セットに対応する1つまたは複数のリソースがユーザによって現在見られることができるか否かを記述する可視性情報、のうちの1つまたは複数に関する情報を含む、前記ローカルデバイス上にあらかじめ記憶されたリソース情報に基づいて、前記リモートデバイスに送信するべきリソースを判定し、前記グラフィック・セットの一部を選択し、符号化するステップと、 前記アプリケーションが、前記符号化されたグラフィック・セットを、前記リモートデバイスに送信するステップと を含むことを特徴とするコンピュータ実行方法。
- 10前記ローカルデバイス、前記リモートデバイス、または前記ローカルデバイスおよび前記リモートデバイスの両方は、前記集中キャッシュ内にサイズの異なるリソースをキャッシュすることによって引き起こされるメモリのフラグメンテーションを制限するメカニズムを使用することを特徴とする請求項9に記載のコンピュータ実行方法。
- 11前記専用のリソースは、前記アプリケーションに関する境界、タイトルバー、ツールバー、またはウィンドウ操作アイコンの1つまたは複数を含むことを特徴とする請求項9に記載のコンピュータ実行方法。
- 12前記可視性情報は、前記1つまたは複数のリソースのZオーダ、透明度、または最小化/最大化状態についての情報を含むことを特徴とする請求項9に記載のコンピュータ実行方法。
- 13前記 画質を落としたリソースを改善するアップデート に関するリソースは、ビットマップ、曲線、または格子のうちの1つまたは複数を含み、前記画質を落としたリソースの画質の低下は、色、詳細、サンプルの点の数のうちの1つまたは複数によるものであることを特徴とする請求項9に記載のコンピュータ実行方法。
- 14前記 画質を落とした リソースは、フォームコントロールであり、前記ユーザは、前記画質を落としたリソースを改善するアップデートを受信することなしに前記フォームコントロールを操作することができることを特徴とする請求項13に記載のコンピュータ実行方法。
- 15分散コンピューティングシステムにおいて、ローカルデバイス上で実行される1つまたは複数のアプリケーションに関してリモートデバイス上でグラフィック表示を生成する際に使用される 1つまたは複数の レンダリングデータ構造を同期するコンピュータ実行方法であって、 前記 ローカルデバイス上で 前記 アプリケーションを実行し、特定のグラフィック言語に対応し、前記レンダリングデータ構造を修正する際に使用される1つもしくは複数のリソース、1つもしくは複数の表示命令、または前記リソースおよび前記表示命令の両方を含むグラフィック・セットを生成するステップであって、前記アプリケーションのそれぞれは 前記 特定のグラフィック言語の 前記 レンダリングデータ構造を生成し、前記レンダリングデータ構造は、 前記 リソースに関する状態を保有し、 かつ 前記リモートデバイス上で前記アプリケーションに関する 前記 グラフィック表示を構成するために使用される保持モードデータ構造である、 生成する ステップと、 前記アプリケーションが、1つまたは複数のデータ圧縮モジュールが認識し、符号化することができる1つまたは複数の異なるグラフィック言語に関する複数の 前記 グラフィック・セットの間で共通の1つまたは複数のフィールドに対する構造の種類を前記データ圧縮モジュールが特定することを助けるために、前記フィールドの記述を含むメタデータを抽出するステップと、 前記アプリケーションが、前記抽出したメタデータに基づいて、前記フィールドを符号化するステップと、 前記アプリケーションが、前記符号化された フィールドを含む グラフィック・セットを前記リモートデバイスに送信するステップと、 前記アプリケーションが、前記レンダリングデータ構造を、前記ローカルデバイスと前記リモートデバイスの間で同期するステップと を含むことを特徴とするコンピュータ実行方法。
- 16前記表示命令は、ディスプレイ上の位置付け、サイズ、色、または形状のうちの1つまたは複数の観点でリソースを記述する1つまたは複数のプロパティを含むことを特徴とする請求項15に記載のコンピュータ実行方法。
- 17前記メタデータに基づいて、前記フィールドのうちの1つまたは複数 を、 マシンフォーマットからネットワークフォーマットに変換し、前記フィールドのビット長を削減することを特徴とする請求項15に記載のコンピュータ実行方法。
- 18前記メタデータに基づいて、前記フィールドのうちの1つまたは複数 を 、可変長の形態で符号化することを特徴とする請求項15に記載のコンピュータ実行方法。
- 19前記メタデータに基づいて、差分符号化を使用し、リソースの変化だけを前記リモートデバイスに送信するように、前記グラフィック・セット全体よりも少ないバイト数を使用して符号化することを特徴とする請求項15に記載のコンピュータ実行方法。
- 20前記メタデータに基づいて、インターオーダ圧縮メカニズムを使用し、特定の種類の前記フィールドのうちの1つまたは複数を、 以 前に符号化 し た 同じ種類の 構造に基づいて符号化することを特徴とする請求項15に記載のコンピュータ実行方法。
- 21前記符号化されたグラフィック・セットを送信するステップと 前記アプリケーションが、前記符号化されたグラフィック・セットのZオーダ、透明度、および最小化/最大化状態の少なくとも1つに基づき、前記符号化されたグラフィック・セットの可視性を判定するステップと、 前記アプリケーションが、前記ローカルデバイスと前記リモートデバイス間の帯域幅判定するステップと、 前記アプリケーションが、前記判定された可視性および帯域幅に基づき、前記符号化されたグラフィック・セットの可視部分を先に、前記リモートデバイスに送信するステップと、 前記アプリケーションが、前記判定された可視性および帯域幅に基づき、前記符号化されたグラフィック・セットの不可視部分を後から、前記リモートデバイスに送信するステップと を含むことを特徴とする請求項1に記載のコンピュータ実行方法。
- 22前記符号化されたグラフィック・セットを送信するステップと 前記アプリケーションが、前記符号化されたグラフィック・セットのZオーダ、透明度、および最小化/最大化状態の少なくとも1つに基づき、前記符号化されたグラフィック・セットの可視性を判定するステップと、 前記アプリケーションが、前記ローカルデバイスと前記リモートデバイス間の帯域幅判定するステップと、 前記アプリケーションが、前記判定された可視性および帯域幅に基づき、前記符号化されたグラフィック・セットの可視部分を先に、前記リモートデバイスに送信するステップと、 前記アプリケーションが、前記判定された可視性および帯域幅に基づき、前記符号化されたグラフィック・セットの不可視部分を後から、前記リモートデバイスに送信するステップと を含むことを特徴とする請求項9に記載のコンピュータ実行方法。
Independent claims22
74 paragraphs, as filed
The present invention relates to an efficient method of coding an alternative graphic set.
As computerized systems became more commonplace, there was an increasing need to distribute computer system files and processing resources in both large and small networks. In general, computer systems and related devices convey information over networks for a number of reasons, such as, for example, exchanging personal electronic messages, selling goods, providing account information, and so on. However, one will recognize that as computer systems and their associated applications have become more sophisticated, so have the difficulties associated with sharing data and resources over networks.
Some current practices for distributing resources within an organization's network may include centralized servers (or local devices) that share resources with one or more clients (or remote devices). The client does not install such resources locally. Such systems typically come with remote clients using specialized protocols such as Remote Desktop Protocol (RDP) and Independent Computing Architecture (ICA). Share the application. Using such a protocol, the client computer system accesses a centralized network server that hosts network resources of interest and interacts with those resources as if they were installed locally. For example, you can send mouse and keyboard events).
The network server then processes those interactions, creates the corresponding rendering information for the data, and returns both the processed data and the rendered rendering information to the client. The client computer system then receives the data and rendering information and uses a client-side video driver to render and display the received data locally. Ideally, this interaction between the client computer system and the network server actually processed the data using the installed resources of the client computer system itself. It is done seamlessly like. Unfortunately, such systems can be constrained by network throughput, and network throughput constraints can be applied to local client computers in terms of interaction and processing under load. It can create a "lag" between what the system sees.
Another type of system, which is similar in most respect to the centralized sharing model described above, is broadcasting (or broadcasting) configured to send window data information to other receiving client computer systems on the network. "Send") Includes client computer system. This feature is sometimes referred to as "desktop sharing." In this example, the broadcasting computer (eg, "instructor" in the learning environment) and the receiving computer system (eg, "student") are common to allow sharing of the instructor's computer desktop screen and locally installed applications. Connect using the installed application program. Similar to the centralized computing system scenario, the client computer system may be able to interact with the window displayed on the instructor's computer as if it were the window of the student's computer itself. is there.
Most of today's (such as the systems mentioned above) rather than transmitting the entire bitmap, because bitmaps are costly in terms of bandwidth consumption when transmitted over a network connection (eg the Internet). The system sends graphic primitives and other operations that tell client-side subroutines what to draw and how to draw something. For example, the client may be instructed to draw a rectangle, along with information about where the rectangle should be drawn, how big it is, what color it is, and so on. For example, a rectangle can be used to draw a button for the user interface, a boundary around the document, or any other purpose for which the shape of the rectangle may be useful. Of course, it is more sophisticated and is used as a primitive that may require more processing to be done to transfer the operation to the remote client and perform the operation on the remote client. There are many other shapes and operations that can be made.
<p> The use of primitives described above has made networking systems more seamless, but as applications continue to acquire more sophisticated graphical interfaces and other displays, the use of primitives described above becomes more processing-intensive. In addition, the information sent locally to the remote device to render the graphic on the client's display is generally an immediate presentation in which the tiled window results in loss of graphic information. Used in mode). For example, with the instant display mode, only the information needed to draw the visible part of the window is available. In other words, there is no graphic information retained for the other window-covered background window portion, that is, the graphic information is retained only for the top-level window. Therefore, when a window is moved to the foreground, new information is needed to draw the window. Due to the progress of graphic sophistication mentioned above, frequent transmission of this information is overwhelming the system when frequent updates are needed, for example when windows are shuffled, rotated and rearranged. May put a load. This also poses various difficulties when it relates to more advanced animations.</p>
<p> The drawbacks and drawbacks of current networks identified above are overcome through exemplary embodiments of the invention. For example, the embodiments described herein provide a mechanism used to efficiently encode using resources for an application that runs on a local device and is displayed on a remote device. It should be noted that this summary is provided to introduce in a simplified form the selection of concepts further described below in "Best Modes for Carrying Out the Invention". This summary is not intended to identify the important or essential features of the claims stated in the claims and is used to help determine the scope of the claims stated in the claims. Not intended to be.</p><p> One exemplary embodiment presents a graphic object for display on a remote device with respect to an application running on the local device by determining the appropriate coding mechanism, regardless of the type of graphic language used. It provides a mechanism for efficient coding. The mechanism provides running applications on local devices, each of which produces a graphic display based on a particular graphic language for display on a remote device. In addition, a graphics set for a particular graphics language is received, and that graphics set contains resources and / or instructions used to render at least a portion of the graphics display for the application. Also received is coded data containing information about: That is, (1) describe the contents of the fields in the graphic set that are used by the data compression module to help compress the graphic set more efficiently than when the graphic set is in its normal form. Metadata and / or coded type data that describes the data compression mechanism supported by the remote device to select an efficient compression type for the graphics set. Based on the coded data received, the appropriate coding mechanism is determined for the various parts of the graphic set.</p><p> Another exemplary embodiment is a graphic object on a remote display device for an application running on the local device by determining which resources (if any) for the application should be sent to the remote device. Provides efficient rendering of. In this embodiment, applications are also run on the local device, and each of those applications produces a graphic display for transmission to the remote device. A graphic set containing resources and / or display instructions is then received and the graphic set can be used to render at least a portion of the graphic display for the application. Also received is resource data containing information about: That is, (1) distribution status information for determining whether a resource corresponding to a graphic set has been sent to a remote device and stored in a centralized cache for reuse, regardless of the type of resource stored. (2) Supported application information to determine the dedicated resources currently available on the remote device to display the resources on the remote device with respect to the application without transferring the resources from the local device, ( 3) To save bandwidth or other system constraints, the resource portion is sent first with a completely degraded version, followed by updates that improve the degraded version. Whether system constraint data to determine if it should be progressively sent to the remote device, and / or (4) the resource corresponding to the graphic set, is currently available to the user on the remote device. It is the visibility information that describes. Based on the resource information received, a portion of the graphic set is selected for coding.</p><p> Another exemplary embodiment provides efficient synchronization of the rendered data structure used in generating a graphic display on a remote device for an application running on the local device. In this embodiment, as in other embodiments, the applications run on a local device, each of the applications produces a rendered data structure for a particular graphic language, and the rendered data structure is a state relating to the resource. Retain mode data that is used to configure a graphic display about an application on a remote device. structure). A graphics set containing resources and / or display instructions for a particular graphics language is then received and used to modify the rendered data structure. In addition, to help the data compression module identify the type of structure for a common field between graphic sets for different graphic languages that the data compression module can more easily recognize and code properly. Metadata is received that contains a description of the fields for the graphic set. Based on the metadata received, the fields in the graphic set send to the remote device and synchronize the rendering data structure used to configure the graphic display on the remote device for the application between the local device and the remote device. Is encoded to do so.</p><p> Further features and advantages of the invention are described in the description that follows, which may reveal to some extent or may be learned by practicing the invention. The features and advantages of the present invention can be realized and acquired using the devices and combinations specifically indicated in the appended claims. These and other features of the invention become more fully apparent from the following description and the appended claims and can be learned by the practice of the invention described below.</p><p> A more specific description of the invention, briefly described above, is illustrated in the accompanying drawings to illustrate how the above and other advantageous features of the invention can be obtained. It is done by reference to a specific embodiment of the invention. These drawings show only typical embodiments of the invention and therefore, with the understanding that they should not be considered a limitation of the scope of the invention, the invention is more specific and detailed through the use of the accompanying drawings. Shown and explained.</p>
The present invention extends to methods, systems, and computer program products for efficiently remotely transmitting graphic sets used in rendering the display of local applications on remote devices. Embodiments of the present invention can include dedicated or general purpose computers that include various computer hardware or modules, as discussed in more detail below.
As a preface, it is recognized and understood that the examples and description herein refer to the Patent Applicant's terminology for convenience in various implementations. However, references to such specific terms should not be construed as limiting the embodiments herein to a particular operating system or other type of system. Rather, the basic functions described herein can be performed in any computing environment or operating system where the functions described herein are desired.
As mentioned above, the previously identified difficulties and disadvantages of remotely transmitting the graphic representation of the current network are overcome through the exemplary embodiments provided herein. For example, one embodiment is field coding, which is a mechanism used to identify a graphic set field for a particular graphic language so that the commonality of various fields across different graphic languages is identified. encoding) is provided. Once identified, the redundancy or identified commonality associated across the various data types within the fields of the graphics set can be efficiently encoded. For example, redundancy or commonality between fields can be removed or efficiently compressed by applying one or more of the following techniques: That is, (1) field conversion between machine and network formats based on the metadata or other information provided for the field, (2) variable length field coding (eg 2/3 / 4-byte coding), (3) An array of coordinates, where points can be coded as differences to previous points in the array, and the differences can be represented by fewer bytes than absolute coordinates. Differential coding used to encode, (4) inter-order compression used to encode a particular type of structure based on the same type of previously encoded structure. compression). The above mechanism can now be applied in other protocols (eg, Remote Desktop Protocol (RDP)), but the embodiments provided herein use field coding. Note that it extends to other graphic languages other than Graphic Design Interface (GDI), such as Windows Presentation Foundation (WPF) information.
In another embodiment, dissimilar resources are provided with a cache of resources so that they are treated in a similar manner when storing them. Current mechanisms (such as RDP) allow later operations or instructions to reclaim resources to save bandwidth and store resources on the client, but the present invention (eg, eg). Extend the caching mechanism for use in other graphics languages (other than GDI like WPF). For example, current mechanisms store resource types (eg, glyphs, bitmaps, sprites, etc.) in separate caches, thereby providing an non-extensible approach to reusing resources. Therefore, the embodiment provides a more comprehensive and extensible mechanism that provides a centralized cache for all resources regardless of their resource type. Therefore, the resource can be used multiple times within the rendered data structure or even across data structures for different applications, so the resource is sent once to the remote device and substructures of various configurations ( For example, it only needs to be used across subtrees).
In yet another embodiment, the type of encoding or compression for remotely transmitting a resource or other item in the graphics set may be determined based on the type of compression mechanism supported by the remote device. it can. In such cases, the available compression mechanism can be negotiated between the client (ie remote device) and the server (ie local device) at connection time, but is used by the server to compress the resources. The exact compression mechanism is determined by the local device at the time of compression. Therefore, the local device chooses one of the negotiated formats based on how well each format compresses a given data. For example, one type of compression that can be selected from various compression techniques can be used for a single resource. The resulting compression mechanism that most efficiently reduces the amount of data can then be used to send resources (or other data) to remote devices. The local server can choose either a lossless or lossy compression format based on the various resources or other data it compresses.
Yet another exemplary embodiment provides improved responsiveness by rendering with partially transmitted resources. Therefore, based on various system constraints such as bandwidth or display device constraints, some of the resources can be sent with a full render instruction to render something meaningful to the application. For example, bitmaps or other resources can include images compressed using such gradual techniques. In such cases, the color image may be initially inaccurate because not all the data needed to decompress the complete image has arrived at the client device. However, the remote device can use a blurry image or other poor quality image for the first rendering.
As an example, a button on an application can have very inaccurate colors, but is good enough to allow the user to interact with the button without having to wait for the final version of the button. Buttons can still be displayed in the same way. As more data from the image arrives from the network, the remote device can update the image and re-render part of the data structure that contains the image. Thus, in most cases, the user can simply use the application without updating all of the image data, dramatically improving the perceived responsiveness of the user. It should be noted that although the use of color degradation mechanisms has been used, any kind of gradual coding or interlacing technique can also be used. Moreover, such gradual or interlaced mechanisms can be applied to bitmaps or other similar resources, but also work well for images with various arrays such as curves or grids. Please note that.
Yet another exemplary embodiment provides a mechanism for determining which part of a graphic set (if any) should be sent to a remote device and in what order. The graphic. For example, a graphic display or some of the resources may often be invisible to the user. Therefore, Z-order, transparency, minimized / maximized states, etc. play an effective role in determining whether an application or application resource produces user-visible output. Unless you can see the application or the application's resources, it may not be necessary to send the application's content or resources remotely until later. Therefore, updates can be delayed until bandwidth allows. In addition, the local server can prioritize which parts of the rendered data structure or display are sent to the remote device based on such visibility information.
Yet another embodiment provides the use of a dedicated resource on the remote device to eliminate the transfer of the resource between the local device and the remote device when rendering the dedicated resource on the remote device. .. For example, in most cases where graphic data is sent remotely, resources such as boundaries, title bars, and / or other icons reside on both the server and the remote computer. For example, if both the remote and local servers install the same (or similar) application, the icon for the local application is probably present in the binary resource portion for the remote application. In such cases, the local device may be able to instruct the remote device to use these various resources without the server or local device having to send bytes of resources for the remote device. is there.
Although more specific references to advantageous features are given below with respect to the figures, embodiments within the scope of the invention provide computer-executable instructions or data structures stored on a computer-readable medium. Also includes computer-readable media for transport or possession. Such a computer-readable medium can be any available medium that can be accessed by a general purpose or dedicated computer. By way of example, but not by limitation, such computer-readable media are RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or computer-executable instructions or data. Any other medium that can be used to transport or store the desired program code means in the form of a structure and can be accessed by a general purpose or dedicated computer can be included. When information is transferred or provided to a computer via a network or another communication connection (either wired, wireless, or a combination of wired or wireless), the computer naturally sees the connection as a computer-readable medium. Therefore, of course, any such connection is called a computer-readable medium. The above media combinations should also be included in the scope of computer readable media.
Computer-executable instructions include, for example, instructions and data that cause a multipurpose computer, a dedicated computer, or a dedicated processing device to perform a particular function or set of functions. Although the subject matter of the invention has been described in a language specific to the behavior of structural features and / or methods, the subject matter of the invention defined in the appended claims is not necessarily limited to the particular features or behaviors described above. I want to be understood. Rather, the particular features and behaviors described above are disclosed as exemplary embodiments that implement the claims.
As used herein, the term "module" or "component" can refer to a software object or routine that runs on a computing system. The various components, modules, engines, and services described herein can be implemented as objects or processes (eg, as separate threads) running on a computing system. The systems and methods described herein are preferably implemented in software, but hardware or a combination of software and hardware is also possible and conceivable. In this description, a "computing entity" may be any computing system already defined herein, or any module or combination of modules running on a computing system.
Figure 1A shows a distributed system used to remotely display graphic information about an application on a local device. As shown, application 115 can run on local device 105, and the display of that application is intended for remote device 110. Note that application 115 can be any of a number of applications, such as text editors, spreadsheet applications, or any other well-known application. In addition, the environment for the remote device 110 and the local device 105 is such that the remote device 110 runs the application 115 on the local device 105 and also sees and controls the application from the remote device 110 outside the network of the local device 105. It can be the presentation type environment you want (eg, desktop sharing as described above) or a networking system. Thus, communication between the local device 105 and the remote device 110 can span any well-known both local and distributed networks, such as LANs, the Internet, and so on.
Regardless of the type of application or network used to establish the communication channel between the local device 105 and the remote device 110, the application 115 follows the user input received from the remote device 110 to make application program interface (API) calls. You can do 120. Such API call 120 affects the graphic display of various applications 115. For example, API call 120 may minimize / maximize the display of application 115, move icons or other resources in the display, or interact with the display of graphics for one or more applications 115. It could be any number of well-known ways to change the display of. Needless to say, it is not essential that the input arrives in the form of API call 120, as the graphic display of various applications 115 will change, as will be recognized. Accordingly, the use of API Call 120 to influence the graphic display of various applications 115 is used herein for illustrative purposes only and limits or so limits the scope of embodiments herein. Otherwise it is not intended to be narrowed.
However, when using such a call 120, the API call 120 can call a synthesis engine 130 that produces a graphics set 155 that can contain display instructions 160 and / or various resources 165. The display instruction 160 is the type of resource 165, its position in the display (ie, xy coordinates), the size and / or shape of the resource 165, or any other well-known used to display various resources 165. It can contain information such as properties or operations. Resource 165 can also represent any number of well-known icon, text, glyph, sprite, bitmap, and other image types. Therefore, the display instructions 160 and resource 165 described herein are any number of different operations performed on the resource and any number of image data used to display the graphics of application 115. Should be broadly interpreted to include.
Generally, graphic set 155 is used to generate rendered data structures 178 for various applications 115. And in turn, the rendered data structure 178 can be a tree-like structure used to describe the display of application 115, where each node represents a description of resources, properties, or their relationships. For example, one node in the data tree can represent a button, while another node describes the color, size, shape, or other properties of the button, while other nodes are in the display of application 114. Represents relationships and interactions with other components or resources of. Therefore, as will be recognized, graphic set 155 may be used to modify, generate, or otherwise update the rendered data structure 178, at which time the rendered data structure 178. Can be used to configure one or more displays corresponding to application 115. Immediate display mode for other graphic languages mentioned above Note that, unlike presentation mode), the use of rendered data structures 178,185 allows for a retained mode in which state with respect to resources is retained. As described in more detail below, this allows many advantageous features, along with other embodiments described herein.
Note that in one embodiment, it is desirable to synchronize the rendered data structure 178 on the local device 105 with a similar rendered data structure 185 on the remote device 110 side. In such cases, the exemplary embodiments provided herein can efficiently encode various graphic sets 155 for updating or modifying the rendering data structure 185 on the remote device 110. .. However, some mechanisms provided herein are used for graphic languages that support rendered data structures 178,185, while other embodiments include such rendered data structures 178,185. Note that it is equally applicable to unsupported graphic languages. In fact, many of the embodiments described herein include, but are not limited to, GDI, WPF, and other known (and potentially unknown) graphic languages. Used for various graphic languages. Therefore, the following discussion of the various embodiments should be broadly interpreted to be applicable across a wide range of graphic languages.
Regardless of the graphics language used to generate the graphics set 155, the encoding decision module 150 generally has a rendering data structure 178 (ie, retained mode composition). Note that it works directly on data) or tree). The coding decision module 150 may send the rendered data 178 immediately when the application 115 calls the synthesis engine 130 120, or it may send the data 178 later when the network allows it. For example, application 115 could call engine 130 to generate a circle 120. The instruction 120 may be translated into a direct call to the coding decision module 150, the latter immediately encoding the circular instruction and transmitting it over the line. An alternative model could be informed that the coding decision module 150 has added a circle to the rendered data structure 178 (eg, a composite tree) and encodes when the data 178 should be sent. It is possible to let the decision module 150 decide. The first model can be considered a "push" model, while the second model can be a "pull" model. The difference between the two is simply when the coding decision module 150 gets an update from the rendered data structure 178. Both models work, but push models can be more limited than pull models. Therefore, the most efficient update mechanism can be a composite between the two (ie, data 178 is pushed to the network as fast as the bandwidth allows, but in the presence of some network congestion, the system Adopts a pull model driven by network availability events).
Regardless of the model used to transfer the data 178, in one embodiment the coding decision module 150 uses graphic set metadata to identify various types of fields within graphic set 155. A coded table 135 with 140 (also referred to herein simply as "metadata") can be used. For example, metadata 140 can describe various types of fields within graphic set 160, and coding decision module 150 can efficiently transfer display instructions 160 and resource 165 to remote device 110. Can be used to properly determine how to best encode various fields. More specifically, metadata 140 can be used to identify different types of data structures for fields that are common between multiple graphic sets 155 for different graphics languages. Such information can then be used to help the data compression module 175 more easily recognize and properly encode the different fields of the graphics set 155.
For example, display instruction 160 has a string or other binary representation that traditionally contains the type, position (eg, xy coordinates), color, or other information of the resource stored or serialized in machine format. there is a possibility. Therefore, metadata 140 can be used to recognize these fields as strings stored in machine format, which can then be converted to network format. For example, coordinates (or other strings or binary fields) are typically machine size words in graphic set 155 on the local device 105 used for rendering. Although held as word), machine words are most often two to four times larger than the actual byte size required to store such coordinates. In such cases, the machine word should be converted to a smaller size and placed in the network packet, thereby making these fields more efficient before the data compression module 175 is transmitted to the remote device 110. Allows to be encoded in.
Other embodiments use graphic set metadata 140 to describe other fields or resources within graphic set 155 to identify the most efficient mechanism for compression. For example, the coding determination module 150 can use metadata 140 to determine the type of resource 165 to select the data compression module 175 that can most effectively compress such information. For example, some bitmaps are best encoded using a run-length coding (RLE) mechanism. Therefore, metadata 140 can be used to identify resource 165 as a bitmap, and the appropriate RLE can be selected to properly encode such resource.
Of course, there are many other types of resources 165 and fields for display instructions 160 that can be identified and properly encoded based on the provided graphic set metadata 140. For example, one field may be identified as being best encoded in a variable length format, while another field may be better suited for differential coding (eg, one point before in the array). Coordinates that can be encoded as differences to points and the differences can be represented by fewer bytes than absolute coordinates). Other examples may include interorder compression, which is typically used to encode a particular type of structure based on the same type of previously encoded structure. Of course, as will be recognized, the metadata 140 displays any number of different displays so that a large number of compression modules 175 can be appropriately selected by the coding decision module 150 as needed. It can be used to describe instruction 160 and resource 165. Therefore, the above use of a particular type of data for coding, or any particular kind of coding described herein, is for purposes of illustration only and limits the embodiments herein. Or otherwise not intended to be narrowed.
Metadata 140 is shown in a coding table 135 that is separate from graphic set 155, but other embodiments also consider metadata 140 as part of graphic set 155. In addition, the various module arrangements described herein can be combined with other configurations and / or schematic layouts, and / or can be divided into other configurations and / or schematic layouts. Please note. Accordingly, the schematic layouts shown in the various figures and the modules or components described herein are used for purposes of illustration only and, unless expressly stated in the claims, the scope of the various embodiments. Is not intended to be limited or otherwise narrowed.
Also note that the metadata 165 can be run-time information or compile-time information. For example, an extensible markup language ((XML) or other suitable binary format) document can contain metadata 165 that is exchanged at run time. The difference between the two is the fact that, in general, run-time information is pushed by the synthesis engine 130 through specific function calls that allow metadata files (eg XML documents) to be passed to the encoding decision module 150. Is. Normally, run-time metadata 165 is pushed only once at initialization. From that point on, the compositing or rendering engine 130 does not have to show the entire graphic set 155 to the coding decision module 150 each time the coding decision module 150 calls the encoder. Instead, the synthesis engine 130 prefixes each order (ie, display instruction and / or resource) in the stream with the well-defined type defined in the first exchange of metadata 165 (ie). Or you just need to identify). The following example may help.
For example, assuming that the metadata 165 is exchanged at run time, the metadata 165 could look like the following XML file:
<tables num="1"><img file="JP5080554B2_D0001.tif" /></tables>
In the GFXMETADATA structure above, two order types are declared. The first order type is an order type that contains two floating point numbers. The second order type is an array containing one signed integer, one unsigned integer, and one array of coordinates. This metadata should be transmitted to the coding decision module 150 at the time of initialization. After initialization, a second structure (ORDER) can be used to send the order to the encoder 150. Note how the second structure does not describe the type, only the value. Even in the XML representation, it is understood that by separating the metadata from the second data structure (ORDER), less information is needed to convey the content about the order to the coding decision module 150. .. Metadata 140 is very static, so it makes sense to be sent only once. Order data, on the other hand, is dynamic data and, in general, should not have added any static information to the order data. Such an advantage becomes more apparent when the binary format (discussed below) is used for both data and metadata instead of XML. The encoder or coding determination module 150 may only look at the type of order and will know how to code the value based on the description of the metadata.
For example, other files (eg, interface definition or description language (IDL) files) represent a compile-time contract between encoder 175 or coding decision module 150 and synthesis engine 130. If the contract is changing, the two components will need to be recompiled using a common (eg IDL) file. Using a common file ensures that the device works for the same binary data structure format. However, it should be noted that writing the data at compile time will probably result in faster operation of the coding decision module 150 (described above), as the coding code itself is generated directly from the description of the graphic data. However, the compile-time description is highly dependent on the language used. For example, in C and C ++, a set of macros can be used to define graphic primitives. By using macros, one can define both the machine format structure and the coding or the network code itself.
Once the proper data compression 175 encodes one or more fields in the graphics set 155, these fields (or graphics set 155) are sent to the remote device 110 where they (or) Graphic set 155) is decompressed using module 180 and regenerated as shown in graphic set 190. Thus, the synthesis engine 195 can use them to generate a rendered data structure 185 that is in sync with the rendered data structure 178 on the local device 105. Further, the rendering data structure 185 can then be configured so that the display driver 104 can generate a suitable display 102 representing the display of the application 115 from the local device 105. Thus, when a user on the remote device 110 interacts with the display 102, a different graphic set 155 can be generated and the rendered data structures 178, 185 are encoded as described above for synchronization.
In another exemplary embodiment, the code type selected or the data compression module 175 can be based on their supported code type on the remote device 110. For example, during connection initialization (or at some other time thereafter), the type of data compression can be negotiated between the local device and the remote device 110. Therefore, as illustrated, the remote device 110 may transmit a supported coding type 184, which in turn may be included in the list of available coding mechanisms 145. it can. This list 145 of the available coding mechanisms is then used by the coding decision module 150 to select the appropriate data compression module 175 to compress the fields in the graphic set 155 as described above. Can be done.
The available coding mechanism 145 is generally negotiated between the remote device 110 and the local device 105 at connection time, but the strict compression mechanism used by the local device 105 to compress part of the graphics set 155. Note that is determined on-the-fly when compressed. Therefore, the remote device 110 has no prior knowledge of the exact type of data compression 175 that will be used. However, since the supported coding type 184 is determined in advance, the local device 105 ensures that the use of any such supported coding type 184 can be handled by the remote device 110. Can be done. It should also be noted that the list 145 of available coding mechanisms can also be based on the coding mechanisms available on the local device 105. In other words, the data compression module 175 and the data decompression module 180 need to have a common type among them.
Also, as discussed herein, any well-known type of coding mechanism can be used (eg, RLE, MPEG-based, JPEG, GIF, ZIP, LZ-based, JBIG, Note that DejaVu, or other well-known pattern or statistics-based compression mechanism that is either a lossless or lossy compression format). Also, one or more of the fields in the graphics set 155 may be of various coding or data compression modules 175 for the most efficient compression of data for transmission over the network to remote device 110. Note that it can be coded using a hierarchical relationship.
FIG. 1B shows a similar distributed system as described above with both local 105 and remote device 110. Note that Figure 1B is a simplified version of the system described above and is therefore shown with components or modules omitted to simplify the schematic. However, in the embodiments shown in the figure, rather than determining which coding type should be used to encode the graphic set 155, various mechanisms are applied to the remote device 110. Used to select resources (if any) to send. In one such embodiment, a cache of application resources is provided that is treated in a similar manner when dissimilar resources store those resources.
For example, the first application (eg, application 1) 115 can make a request for resource "A" 106. When a resource is sent over the network to the remote device 110, the delivery status of such resource can be stored in resource table 112 (in this case, the delivery status of resource "A" 114 has been transferred. Shows). On the remote device 110, resource "A" 106 is stored in centralized cache 128 as shown in Listing 118 of the resources in centralized cache 128. Therefore, the next time application 115 wants to use resource "A" 106, instead of sending the resource to remote device 110, resource manager 108 identifies that the resource has already been sent and simply simply Provide display instruction 160 with resource ID 161 for transmission to resource device 110. In other words, the resource (in this case, resource "A" 106) only needs to be sent once during the connection between the local 105 device and the remote 110 device, and various different applications (eg, similarly). It can be used repeatedly across applications 2) that call resource "A" 106.
Some caching mechanisms used by certain protocols (eg RDP) allow for caching resources, but in general such caching are the types of resources used (eg glyphs, bitmaps, sprites). Please note that it is divided based on (etc.). Such mechanisms may be as efficient as the more comprehensive centralized caching mechanisms provided herein, but these approaches are not extensible. By providing a centralized cache 128 for storing all resources 180, the present invention is such that across different rendering data structures 178, 185 that can be used across various applications 115. Extend the ability to use resource 106. For example, the comprehensive caching mechanism provided herein (eg using RDP) has the additional advantage that the resource is transmitted only once and used by multiple nodes in the composite structure or tree. For example, if an icon is used by multiple applications 115, it is present in each of the composite subtrees (or rendered data structures 178, 185) that correspond to those applications 115.
In the absence of such a caching mechanism, the resource would have to be sent to the remote device 110 for each node that uses the resource. However, when using the centralized storage mechanism provided herein, resource 165 is transmitted only once when resource 165 is first added to a node and is used by multiple nodes in the composite data structure. Also note that local device 105 and remote device 110 may or may not use a mechanism that limits memory fragmentation caused by caching resources of significantly different sizes. Moreover, non-extensible approaches can still provide an efficient mechanism for accessing resources, so such approaches are used in conjunction with the comprehensive cache provided herein. Note that you can.
Yet another exemplary embodiment provides a mechanism for improving responsiveness by rendering with partially transmitted resources. In such cases, system constraints such as bandwidth, or display constraints on the remote device 110, can be identified by the coding decision module 150, which includes the resource manager 108. Based on such system constraint 116, resource manager 108 can determine that only partial resource 163 should be sent with display instruction 160 for graphic set 155 as described above. More specifically, in general, there are two pieces of information needed to render graphic set 155. The first part, referred to herein as display instruction 160, describes the actual rendering operation for resource 165. In general, they include instructions on how resource 165 should be used for rendering and which resource 165 should be used for rendering. The second part represents the resource 168 used for rendering.
As can be understood, in general, all rendering or display instructions 160 need to be sent to the client or remote device 110 before rendering can begin. However, this is not the case for resource 165. Resource 165 does not have to be 100% available whenever rendering starts for something that makes sense to be rendered by application 115. For example, the first bitmap resource 165 can contain images compressed using gradual techniques. For example, image colors can be initially inaccurate due to various bandwidth constraints, or other considerations. However, the remote device can use this blurry, strangely colored, or otherwise degraded image for rendering on the remote device.
For example, if resource 165 could be a button, the color could be very inaccurate, or some other quality degradation could be seen, but the button display. Can be sufficient to allow the user to interact with the button without having to wait for the final version of the button. However, as more data from the image arrives from the network, the remote device 110 can update the image or rendered data structure 185. Therefore, if most users can use the application without the presence of all of the resources 165, the perceived responsiveness of the user is dramatically improved. Note that the example uses a technique of degrading color quality, but any kind of gradual coding or interlacing mechanism can be used as well.
Also note that this partial rendering mechanism can work well for systems in retained mode, but may not work very well for other models (eg GDI). The reason is that in possession mode, when the rendered data structure 185 (eg a composite tree) is updated with the improved resource 165, all the information is on the remote 110 to cause a redraw. is there. However, in other non-holding models (eg GDI models), there may not be a mechanism for reconstructing on the remote 110 side, so generally when drawing is done with inaccurate resources. There is no mechanism to refresh the drawing until the local 105 or the server side uses that resource 165 again in the drawing operation. However, even in non-retention models, there may be cases where it is preferable to render using the inaccurate resource 165 rather than waiting for the entire resource 165 before rendering anything.
FIG. 1D shows the above embodiment using an array of coordinates displayed on curve 142. As shown, the complete resource set 144 contains a number of points along curve 142. However, embodiments may use system constraint 116 to determine that only partial resource set 148 should be transmitted. For example, in this example, the curve 142 is not accurately represented, but only some of the points along the curve 146 that still allow a reasonable drawing of the entire image are transmitted. In other words, the exact curve 142 is approximated by a number of points where each point represents a digital sample for that curve, and the more samples there are, the more accurate the curve looks. However, depending on the curve 142, a close enough representation 146 can be rendered using only a portion of the number of samples. The rest of the points can be updated later. As the update arrives from the local device 105, the remote device 110 updates the set of points for that curve 142 and re-renders the data structure 185 that uses that curve 142. With each update, the rendering curve 146 gradually approaches the final intended shape 142.
Note that the above example uses graphical curves or grids, and simple buttons, but more advanced resources 165 can also use this feature. For example, an image whose final form is a 3D object can first be shown in an intermediate form of a 2D object. As bandwidth becomes available, more and more information can be sent to update an image to its final 3D form. Needless to say, as you will recognize, there are many different resources 165 and mechanisms that can take advantage of this feature (ie, how to send some of the resources, and what parts of the resources to send. Do you?) Exists. Therefore, the above examples are used herein for illustrative purposes only and are not intended to limit the scope of these and other embodiments.
In yet another exemplary embodiment, the resource manager 108 can determine or identify dedicated resources on both the local device 105 and the remote device 110. Therefore, information 122 for supported applications can be transferred to the local device 105 once the connection is initialized (or at any time thereafter). Generally, this information includes application resources 124 that are supported on the remote device 110 and stored in the resource store 126. Information 122 of this supported application can then be used by the resource manager 108 to determine what type of resource should be sent to the remote device 110.
For example, both the remote 110 and the local device 105 may install certain applications (eg, certain types of text editors). Therefore, the resource 124 for the icon or other related application is also placed on both devices 105, 110. In these cases, the local device 105 simply uses display instruction 160 and resource ID 161 to use the appropriate icon for the remote device 110, without the local device 105 actually having to send the resource 124 for the appropriate icon. Can be ordered. However, it should be noted that such a model assumes that resource 124 is stored in serialized form. Those resources 124 can be converted as needed, but in general the conversion is done at the hardware layer. Thus, for a protocol or graphic language that does not translate resources, the local device 105 can propagate the well-known resource ID 161 to the client instead of sending data about the resource 124 over the line. Such use of the dedicated application resource 124 on the remote device 110 is particularly advantageous when the size of the resource 124 is large.
Also note that the exact resource 124 does not have to be present on both the local 105 and remote 110 devices. For example, some applications 115 have resources 124 that are common among some applications 115. For example, most applications 115 have boundaries, a title bar, and window instructions for minimize / maximize, full / partial screens, and / or close icons. Therefore, in general, the application resource 124 used on the remote device 110 corresponds to the exact application 115 on the local machine 105, but the resource 124 for those applications is not necessarily the exact application 115 on the local machine 105. There is no need to support application 115.
Figure 1C shows the use of various resources that can be used by both the local device 105 and the remote device 110, or are dedicated. In this example, the boundary 132 resource can be used to border the window of the application screen. In addition, the title bar 135 and window manipulation icon 136 may also be application resources 124 that can be accessed and used on the remote device 110. Similarly, a toolbar 138 with various components or icons is also available. However, information that is manipulated or otherwise modified on the local machine 105 needs to be transmitted over the line according to the embodiments described above.
In yet other exemplary embodiments, the coding and other resource selection mechanisms described above can be implemented as part of a single component. The interface (or contract) between this component and the application or synthesis engine can be defined by an interface definition language such as IDL or XML. However, the contract should provide the coding decision module 150 with both the resource 165 data and the metadata 140 required to efficiently encode the resource 165. In this way, changing the layout or organization of resource 165 does not require an update to the coding decision module 150. Only an update to the metadata 140 is required and this update can be accomplished by modifying the interface or contract.
The present invention can also be described in terms of methods involving functional steps and / or non-functional behaviors. The following is a description of the steps and / or actions that can be performed in carrying out the present invention. Functional steps usually describe the invention in terms of the results achieved, while non-functional actions describe more specific actions to achieve a particular result. Although functional steps and / or non-functional actions may be described or claimed in a particular order, the present invention is in any particular order or combination of steps and / or actions. Not necessarily limited to. In addition, the use of steps and / or actions in the claims detailing (only) is used to indicate the desired specific use of such terms.
As mentioned above, FIGS. 2-4 show flow diagrams for various exemplary embodiments of the present invention. The following description of FIGS. 1 to 4 may refer to the corresponding elements from FIGS. 1A to 1D. References may be made to specific elements from these figures, but such references are used for illustrative purposes only and, unless expressly stated in the claims, are the scope of the embodiments described. Is not intended to be limited or otherwise narrowed.
Figure 2 efficiently encodes a graphic object for display on a remote device for an application running on the local device by determining the appropriate coding mechanism, regardless of the type of graphic language used. The flow chart about the method 200 is shown. Method 200 includes operation 205 running the application (s) on the local device. For example, application 115 can run on 105 on the local device, and application 115 produces a graphic display based on a particular graphic language for display on the remote device 110. Such graphics languages can include GDI, WPF, or other types of currently known or future graphics languages.
Method 200 also includes operation 210 to receive a graphic set for a particular graphic language. For example, the coding determination module 150 can receive a graphics set 155 for a particular graphics language, which may include display instructions 160 (s) and / or (s). ) Includes resource 165. Such a graphic set 155 is used to render at least a portion of the graphic display for application 115 on the remote device 110. These display instructions can include properties that describe the resource in terms of position, size, color, shape, etc. on the display. Further note that the resource can be any of the well-known resources such as glyphs, icons, sprites, bitmaps, or any other image.
In addition, method 200 includes operation 220 to receive encoded data for (1) metadata describing the contents of the graphic set and (2) coding type data describing the data compression mechanism supported by the remote device. .. For example, the coding determination module 150 is used to help the data compression module 175 compress the graphics set 155 more efficiently than when the graphics set 155 is in its normal form. You can receive graphic set metadata 140 that describes the contents of the fields in 155. Note that this normal form may be a serialized or non-serialized form. Alternatively, or in conjunction with it, the coding determination module 150 can receive a list 145 of the available coding mechanisms supported by the remote device 110. This coded type data describes the data compression mechanism supported by the remote device 110 to select an efficient compression type for graphic set 155 as described above.
Based on the coded data received, Method 200 also includes operation 225 to determine the appropriate coding mechanism for a portion of the graphic set. For example, based on the metadata 140 and the list 145 of available coding mechanisms, the coding decision module 150 determines which data compression module 175 most efficiently encodes various parts of the graphic set 155. Can be determined.
Figure 3 efficiently renders graphic objects on a remote display device for an application running on the local device by determining which resources (if any) for the application should be sent to the remote device. Method 300 is shown. Method 300 includes operation 305 running the application (s) on the local device. For example, application 115 can run on local device 105, and each of those applications produces a graphic display for transmission to remote device 110. Method 300 also includes operation 310 to receive a graphic set containing (s) display instructions and / or (s) resources. For example, the coding decision module 150 should be used in conjunction with the resource manager 108 to render at least part of the graphic display for application 115 on the remote device 110 resources 165 and / or display instructions 160. Can receive graphic set 155 including.
Method 300 also includes operation 315 to receive resource data on (1) delivery status information, (2) supported application information, (3) system constraint data, and / or (4) visibility information. More specifically, the resource manager 108 can receive resource data for determining the distribution status of the resource 165. For example, the resource status table 112 has the corresponding resource (eg resource A) regardless of the type of resource stored. Used by Resource Manager 108 to determine the delivery status of resources corresponding to graphic set 155 to determine if 114) was sent to remote device 110 and stored in centralized cache 128 for reuse. Can be done. In other words, different types of resources are stored in the centralized cache 128 so that different types of resources are treated in the same way when storing them. Note that local device 105 and remote device 110 may or may not use a mechanism that limits memory fragmentation caused by caching resources of significantly different sizes.
In addition, the coding decision module 150 or the resource manager 108 determines the dedicated resource 124 currently available on the remote device 110 to display such resources without transferring the resources from the local device 105. In addition, you can receive information 122 for supported applications. In other words, if both the remote device 110 and the local device 105 install similar or same applications, the resources or icons reserved on the remote device 110 can be identified and displayed, and therefore of them. Resources do not need to be sent from the local device 105 to the remote device. Such dedicated resources can include boundaries, title bars, toolbars, or any other form of icon or resource that is standard across both applications.
In addition, the coding decision module 150 or resource manager 108 will be sent first with a completely degraded version to save bandwidth or other system constraints, with updates to improve the degraded version. System constraint information 116 for determining whether some of the resources should be progressively transmitted to the remote device 110 can also be received for subsequent transmission. For example, as shown in Figure 1D, a complete resource set 144 showing curve 142 at high sampling is first sent as a partial resource set 148 containing only part of the total sampling set as shown from curve 146. Can be done. Note that the partial resources that are progressively transmitted to the remote device may include bitmaps, curves, grids, or other image forms. The degraded version also includes inaccuracies in color, detail, number of sample points, or other degraded image quality, while the degraded version is for users of remote device 110. Note that it should contain enough information to make it possible to recognize the resource. Further note that a resource can be a button, checkbox, or other interactive item, and the user should still be able to interact with the item without receiving the full resource. ..
In addition, the coding decision module 150 or resource manager 108 has visibility from the resource state table 112 that describes whether one or more resources 165 corresponding to graphic set 155 are currently visible to the user. Information 113 can be received. Such visibility information can include information about the resource's Z-order, transparency, minimized / maximized state, and so on. Thus, resources that may not be visible can be delayed in sending to the remote device 110 until bandwidth allows or is needed to see.
Based on the resource information received, the method 300 includes an action 320 of selecting a part of the graphic set for encoding. In other words, based on the above delivery status, supported application information, system constraint data, and / or visibility information, part of graphic set 155 (ie, part of a display instruction field, or part of a resource). ) Is selected for encoding using the data compression module 175.
Figure 4 shows how to efficiently synchronize the rendered data structures used to generate a graphic display on a remote device for an application running on the local device. Method 400 includes operation 405 to run the application (s) on the local device. For example, as mentioned above, application 115 can run on local device 105, each of those applications produces a rendering data structure 178 for a particular graphic language, and those rendering data structures 178 are resources. A retained mode data structure that holds the state for 165 and is used to configure a graphic display for application 115 on the remote device 110.
Method 400 also includes operation 410 receiving a graphic set containing (s) resources and / or (s) display instructions. For example, the coding decision module 150 can receive a graphics set 155 containing display instructions 160 and / or resources 165. The display instruction 160 can include properties that describe the resource 165 in terms of position, size, color, shape, etc. on the display.
Method 400 also includes operation 114 to receive metadata containing a description of the graphic set. For example, the coding decision module 150 compresses the type of structure for a common field among multiple graphic sets 155 for different graphic languages that the data compression module 175 can more easily recognize and encode appropriately. To help module 175 identify, graphic set metadata 140 may be received that contains a description of the fields in graphic set 155. For example, based on metadata 140, a field can be converted from machine format to network format, which reduces the bit length of the field for a better compression ratio than when in machine form. .. In addition, the metadata 140 can be used for coding in a variable length form, or for differential coding where only changes in the resource are used to be sent to the remote device. Changes can be encoded using fewer bytes than transmitting the entire graphic set 155. In addition, metadata 140 can be used for the interorder compression mechanism used to encode certain types of fields based on the same type of previously encoded structure.
Note that the metadata 140 may or may not be attached to the graphics set 155. For example, in general, exchanging metadata 155 is a one-time event that indicates the type of graphics set 155 used (eg, order / instruction or resource). Each graphic set 155 can then be prefixed with a type (or identifier). The coding determination module 150 can then see this identifier and can select the appropriate data compression module 175, the appropriate coding order, etc. based on the first exchanged metadata 155.
Based on the metadata received, Method 400 includes operation 420 to encode the fields of the graphic set. For example, based on the graphic set metadata 140 that describes the fields in the graphic set 155, the coding decision module 150 encodes the fields for transmission to the remote device 105, with the local 105 and the remote 110 device. Such data can be used to synchronize the rendered data structures 178, 185 between, and the rendered data structures 178, 185 are used to configure the graphic display on the remote device 110 with respect to the application 115. ..
The invention can be practiced in other particular embodiments without departing from the spirit or essential features of the invention. The embodiments described should be considered only exemplary in all respects and not limited. Therefore, the scope of the present invention is shown not by the above description but by the appended claims. All changes that have the same meaning and scope as the claims should be included in the claims.
<figref num="1A">FIG. 5 illustrates a distributed system that uses various information to efficiently encode a graphics set used when rendering a display on a remote device, according to an exemplary embodiment.</figref><figref num="1B">FIG. 6 illustrates a distributed system that utilizes information about various resources to determine which part of a resource (if any) should be encoded for transmission to a remote device, according to an exemplary embodiment. is there.</figref><figref num="1C">FIG. 5 is a diagram illustrating a portion of a dedicated resource or icon that is available on a remote device and can be used so that there is no transfer of resources from the local device, according to an exemplary embodiment.</figref><figref num="1D">FIG. 6 illustrates a mechanism for improving responsiveness by rendering with partially transmitted resources, according to an exemplary embodiment.</figref><figref num="2">FIG. 5 is a flow diagram relating to a method for efficiently encoding a graphic object for display on a remote device according to an exemplary embodiment.</figref><figref num="3">FIG. 5 is a flow diagram of a method of efficiently rendering a graphic object on a remote display device according to an exemplary embodiment.</figref><figref num="4">FIG. 5 is a flow diagram of a method of efficiently synchronizing the rendered data structures used in generating a graphic display on a remote device, according to an exemplary embodiment.</figref>
8 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP2004318913A | Cites | Japan |
| JP07336676A | Cites | Japan |
| JP2004511852A | Cites | Japan |
| JP2003509785A | Cites | Japan |
| JP2000511364A | Cites | Japan |
| US20050228890A1 | Cites | United States of America |
| US20040189677A1 | Cites | United States of America |
| US05083262A | Cites | United States of America |
| WO2004104759A1 | Cites | World Intellectual Property Organization (WIPO) |
30 members in 14 offices
Priority claims9
| Document | Office | Kind | Date |
|---|---|---|---|
| 11375961 | United States of America | – | |
| 37596106 | United States of America | A | |
| 37596106 | United States of America | A | |
| 2007001101 | United States of America | W | |
| 2007001101 | United States of America | W | |
| 2006375961 | – | – | – |
| 2007001101 | – | – | – |
| US20060375961 | – | – | – |
| WO2007US01101 | – | – | – |
Members30
| Document | Office | Kind | |
|---|---|---|---|
| AU2007225421A1 | Australia | A1 | |
| CA2642529A1 | Canada | A1 | |
| US2007220168A1 | United States of America | A1 | |
| WO2007106211A1 | World Intellectual Property Organization (WIPO) | A1 | |
| TW200805141A | Taiwan Province of China | A | |
| MX2008011658A | Mexico | A | |
| KR20080111450A | Republic of Korea | A | |
| EP2005310A1 | European Patent Office (EPO) | A1 | |
| CN101401087A | China | A | |
| IL193515A0 | Israel | A0 | |
| JP2009530706A | Japan | A | |
| RU2008136867A | Russian Federation | A | |
| US2010278442A1 | United States of America | A1 | |
| CN101401087B | China | B | |
| BRPI0708763A2 | Brazil | A2 | |
| AU2007225421B2 | Australia | B2 | |
| EP2005310A4 | European Patent Office (EPO) | A4 | |
| RU2439675C2 | Russian Federation | C2 | |
| AU2012200858A1 | Australia | A1 | |
| KR101159396B1 | Republic of Korea | B1 | |
| US8244051B2 | United States of America | B2 | |
| JP2012198882A | Japan | A | |
| JP5080554B2This record | Japan | B2 | |
| US8351716B2 | United States of America | B2 | |
| IL193515A | Israel | A | |
| MY149001A | Malaysia | A | |
| AU2012200858B2 | Australia | B2 | |
| JP5373135B2 | Japan | B2 | |
| TWI437486B | Taiwan Province of China | B | |
| CA2642529C | Canada | C |
23 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Cancellation because of no payment of annual feesLAPS | LAPS | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Written notification of registration of transferJAPANESE INTERMEDIATE CODE: R350R350 | R350 | |
| Request for change of ownership or part of ownershipJAPANESE INTERMEDIATE CODE: R313113S111 | S111 | |
| Renewal fee payment (event date is renewal date of database)FPAY | FPAY | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A821A521 | A521 | |
| Notification of appointment of power of sub attorneyJAPANESE INTERMEDIATE CODE: A7433RD13 | RD13 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 5080554
- Publication, DOCDB
- 5080554
- Publication, EPODOC
- JP5080554B
- Application
- 2009500354
- Application, DOCDB
- 2009500354
- Application, EPODOC
- JP20090500354
Titles2
- Japanese
- 代替的グラフィック・セットの効率的な符号化
- English
- Efficient coding of alternative graphics sets
Classification
- CPC, 6
- H03M7/30
- G06F15/16
- G06F9/542
- G06F2209/545
- G06F3/0481
- G06F9/54
- IPC, 1
- G06T1 00
