Mirroring graphics content to external display
Abstract
Problem to be solved.To mirror graphic contents to an external display. A data processing system generates image data by compositing graphic contents generated by an application program executed in the system. The data processing system stores the image data in the first frame buffer, and displays the image generated from the image data in the first frame buffer on the internal display device of the data processing system. The scaler performs a scaling operation on the image data in the first framebuffer, stores the scaled image data in the second framebuffer, and the scaled image in the second framebuffer. The image generated from the data is displayed on an external display device. The scaler performs scaling operations asynchronously on a composite of graphic content. [Selection diagram] Fig. 5A

Term
5.3 yearsto projected expiry
Projected expiry 11 January 2032, counted from filing; an application has no term until it is granted.
- Priority
- Filed
- Published
- Today
- Projected expiry
20 claims: 8 independent, 12 dependent
- 1データ処理システムによってグラフィックコンテンツをコンポジットして画像データを発生する段階と、 前記画像データを第1のフレームバッファに記憶する段階と、 前記第1のフレームバッファ内の画像データから発生された画像を前記データ処理システムの内部ディスプレイに表示する段階と、 前記第1のフレームバッファ内の画像データに対してスケーラーによってスケーリング動作を遂行する段階であって、前記スケーラーは、前記グラフィックコンテンツのコンポジットに対して非同期でスケーリング動作を遂行する段階と、 前記スケーリングされた画像データを第2のフレームバッファに記憶する段階と、 前記第2のフレームバッファ内の前記スケーリングされた画像データから発生された画像を表示するためのデータを、前記データ処理システムに結合された外部ディスプレイへ送信する段階と、を備えた方法。
- 2前記スケーリングされた画像データは、前記第1の画像フレームに関連したデータを含む、請求項1に記載の方法。
- 3前記第1の画像フレームに関連した画像データに対してスケーリング動作が完了する前に、グラフィックコンテンツをコンポジットして、後続画像フレームに関連した画像データを発生する段階を更に備えた、請求項2に記載の方法。
- 4手前のフレームに関連したスケーリングされた画像データから発生された画像が外部ディスプレイ上での表示を完了したかどうか決定する段階を更に備えた、請求項2に記載の方法。
- 5手前のフレームに関連したスケーリングされた画像データから発生された画像が外部ディスプレイ上での表示を完了していない場合には前記第1のフレームに関連したスケーリングされた画像データをドロップする段階を更に備えた、請求項4に記載の方法。
- 6前記スケーリング動作は、前記画像データのサイズ、解像度、方向及びカラーのうちの少なくとも1つを調整することを含む、請求項1に記載の方法。
- 7グラフィックコンテンツに対するコンポジットコマンドのリストを発生するように構成された処理装置と、 前記処理装置に結合され、前記コンポジットコマンドのリストを実行して、グラフィックコンテンツに基づいて画像データを発生するように構成されたグラフィック処理ユニットと、 前記グラフィック処理ユニットに結合され、前記画像データから発生された画像を表示するように構成された内部ディスプレイと、 前記グラフィック処理ユニットに結合され、前記画像データに対してスケーリング動作を遂行して、外部ディスプレイに表示されるべき画像のためのスケーリングされた画像データを発生するように構成されたスケーラーユニットであって、前記処理装置及びグラフィック処理ユニットの動作に対して非同期で前記スケーリング動作を遂行するスケーラーユニットと、を備えた装置。
- 8前記スケーリングされた画像データは、第1の画像フレームに関連したデータを含む、請求項7に記載の装置。
- 9前記グラフィック処理ユニットは、更に、前記スケーラーユニットが前記第1の画像フレームに関連した画像データに対するスケーリング動作を完了する前に、コンポジットコマンドを実行して、後続画像フレームに関連した画像データを発生するように構成された、請求項8に記載の装置。
- 10前記処理装置は、更に、手前のフレームに関連したスケーリングされた画像データから発生された画像が外部ディスプレイ上での表示を完了したかどうか決定するように構成される、請求項8に記載の装置。
- 11前記処理装置は、更に、手前のフレームに関連したスケーリングされた画像データから発生された画像が外部ディスプレイ上での表示を完了していない場合には前記第1のフレームに関連したスケーリングされた画像データをドロップするように構成された、請求項10に記載の装置。
- 12前記スケーリング動作は、前記画像データのサイズ、解像度、方向及びカラーのうちの少なくとも1つを調整することを含む、請求項7に記載の装置。
- 13データ処理システムの内部ディスプレイ装置上に表示するために第1画像のための第1のグラフィックコンテンツをデータ処理システムにより受信する段階であって、前記第1のグラフィックコンテンツは、データ処理システムで実行されているアプリケーションプログラムにより発生されたものである段階と、 前記データ処理システムに結合された外部ディスプレイ装置に前記第1画像を自動的にミラーリングする段階と、 前記外部ディスプレイ装置に表示する第2画像のための第2のグラフィックコンテンツをアプリケーションプログラムが発行するかどうか決定する段階と、 そのアプリケーションプログラムが第2のグラフィックコンテンツを発行する場合は、前記自動ミラーリングをディスエイブルする段階と、を備えた方法。
- 14前記アプリケーションプログラムが第2のグラフィックコンテンツを発行する場合は、前記第2の画像のための画像データを前記外部ディスプレイ装置へ送信する段階を更に備えた、請求項13に記載の方法。
- 15外部ディスプレイ装置に第1画像を自動的にミラーリングする前記段階は、 前記第1画像のための画像データに対するスケーリング動作をスケーラーにより遂行して、スケーリングされた画像データを発生することを含み、前記スケーラーは、前記第1のグラフィックコンテンツのコンポジットに対して非同期でスケーリング動作を遂行し、そして前記スケーリングされた画像データは、第1の画像フレームに関連したデータを含む、請求項13に記載の方法。
- 16前記第1の画像フレームに関連した画像データに対してスケーリング動作が完了する前に、グラフィックコンテンツをコンポジットして、後続画像フレームに関連した画像データを発生する段階を更に備えた、請求項15に記載の方法。
- 17データ処理システムの内部ディスプレイ装置上に表示するために第1画像のための第1のグラフィックコンテンツをデータ処理システムにより受信する段階であって、前記第1のグラフィックコンテンツは、データ処理システムで実行されているアプリケーションプログラムにより発生されたものである段階と、 前記データ処理システムに結合された外部ディスプレイ装置に表示する第2画像のための第2のグラフィックコンテンツをアプリケーションプログラムが発行するかどうか決定する段階と、 そのアプリケーションプログラムが第2のグラフィックコンテンツを発行しない場合には、前記外部ディスプレイに前記第1画像を自動的にミラーリングする段階と、 そのアプリケーションプログラムが第2のグラフィックコンテンツを発行する場合には、前記自動ミラーリングをディスエイブルして、前記第2画像のための画像データを前記外部ディスプレイ装置へ送信する段階と、を備えた方法。
- 18外部ディスプレイ装置に第1画像を自動的にミラーリングする前記段階は、 前記第1の画像の画像データに対してスケーラーによってスケーリング動作を遂行することを含み、前記スケーラーは、前記第1のグラフィックコンテンツのコンポジットに対して非同期でスケーリング動作を遂行し、そして前記スケーリングされた画像データは、第1の画像フレームに関連したデータを含む、請求項17に記載の方法。
- 19前記第1の画像フレームに関連した画像データに対してスケーリング動作が完了する前に、グラフィックコンテンツをコンポジットして、後続画像フレームに関連した画像データを発生する段階を更に備えた、請求項18に記載の方法。
- 20実行されたときにデータ処理システムが請求項1から6又は13から19のいずれかに記載の方法を遂行するようにさせるインストラクションを記憶した非一時的マシン読み取り可能な記憶媒体。
Independent claims20
50 paragraphs, as filed
The present invention relates to the field of display-driven architecture, and more particularly to mirroring graphic content to an external display.
<u style="single">Related application</u>: This application claims the benefit of US Provisional Patent Application No. 61 / 431,776 filed on January 11, 2011, which is hereby incorporated by reference.
Electronic devices such as computer systems, wireless cellular phones, mobile computing devices, tablet computing devices or other data processing systems often have internal visual display devices built-in. This internal display device is used to display visual content generated by or stored in an electronic device. The internal display device is, for example, an internal liquid crystal display (LCD) device.
<p> Some electronic devices also include connectors such as the Extended Video Graphic Array (XVGA) connector or other connectors known in this technology for connecting to external visual display devices. The external display device is, for example, a stand-alone LCD or a cathode ray tube (CRT) display. In one example, the electronic device can be configured so that both the internal display device and the external display device display the same visual content at the same time. This is referred to as mirror mode. Alternatively, one electronic device may be configured so that two display devices display different images, such as a portion of an extended desktop, allowing the user to move windows between the two displays to separate them from one desktop. Allows you to move windows to your desktop.</p>
<p> In some embodiments described herein, the data processing system generates image data by compositing graphic content generated by an application program running in the data processing system. The data processing system stores the image data in the first frame buffer, and displays the image generated from the image data in the first frame buffer on the internal display device of the data processing system. The scaler of the data processing system performs a scaling operation on the image data in the first framebuffer, stores the scaled image data in the second framebuffer, and in the second framebuffer. The image generated from the scaled image data is displayed on an external display device coupled to the data processing system. The scaling operation involves adjusting at least one of the size, resolution, orientation and color of the image data.</p><p> The scaler performs scaling operations asynchronously with respect to compositing graphic content. In one embodiment, the scaled image data is associated with a first image frame. The scaler is asynchronous in that it operates separately from other components such as the CPU or GPU. The data processing system starts compositing the graphic content before the scaling operation is completed for the image data related to the first image frame, and generates the image data related to the subsequent image frame.</p><p> In one embodiment, the data processing system determines whether the image generated from the scaled image data associated with the foreground frame has completed its display on the external display. If the image generated from the scaled image data associated with the foreground frame does not complete its display on the external display, the data processing system drops the scaled image data associated with the first frame. ..</p><p> In another embodiment, the data processing system automatically mirrors the image to the external display device unless the application program issues additional graphic content for display on the external display device. The data processing system determines whether the application program publishes the second graphic content of the second image for display on the external display device, and if the application program publishes the second graphic content, it is external. The display device disables automatic mirroring. In another embodiment, the data processing system first determines whether the application program publishes additional graphic content, and then enables or disables mirroring accordingly. Mirroring includes the asynchronous scaling behavior described above.</p><p> The present invention will be described in detail below as an example with reference to the accompanying drawings, but the present invention is not limited thereto.</p>
<figref num="1">It is a block diagram which shows the display drive architecture which mirrors graphic content to an external display by one Embodiment.</figref><figref num="2">It is a block diagram which shows the data processing system and the external display for mirroring graphic content by one Embodiment.</figref><figref num="3">FIG. 6 is a block diagram showing a more general embodiment of a display driven architecture for mirroring graphic content to an external display according to one embodiment.</figref><figref num="4">It is a block diagram which shows the software stack for mirroring a graphic content to an external display by one Embodiment.</figref><figref num="5A">It is a flowchart which shows the clone mirroring method by one Embodiment.</figref><figref num="5B">It is a flowchart which shows the clone mirroring method by one Embodiment.</figref><figref num="6A">It is a flowchart which shows the automatic clone mirroring method by one Embodiment.</figref><figref num="6B">It is a flowchart which shows the automatic clone mirroring method by one Embodiment.</figref><figref num="7">It is a block diagram which shows the time line for the clone mirroring operation by one Embodiment.</figref><figref num="8A">It is a figure which shows one Example of the scaling operation at the time of mirroring a graphic content to an external display by one Embodiment.</figref><figref num="8B">It is a figure which shows one Example of the scaling operation at the time of mirroring a graphic content to an external display by one Embodiment.</figref><figref num="9">It is a block diagram which shows the normative API architecture by one Embodiment.</figref><figref num="10">An embodiment of a software stack using one or more APIs according to an embodiment is shown.</figref>
In the following detailed description of embodiments of the present invention, reference is made to the accompanying drawings in which similar elements are indicated by the same reference numbers and exemplifying specific embodiments embodying the present invention. These embodiments are described in sufficient detail to allow those skilled in the art to practice the invention, and other embodiments are available and are logically, mechanically, electrically, without departing from the scope of the invention. It should be understood that functional and other changes will be made. Therefore, the following detailed description does not imply any limitation and the scope of the invention is defined solely by the claims.
FIG. 1 is a block diagram showing a display drive architecture that mirrors graphic content to an external display according to an embodiment of the present invention. In one embodiment, the architecture comprises an internal display device 140 and any external display device, such as an external display device 150. The data processing system 101 shown in FIG. 1 is a portable general purpose computer system or a portable special purpose computer system or other type of data processing system such as a cellular telephone, a smartphone, a personal digital assistant, an embedded electronic device, or a consumer. Included in electronic devices for people. The data processing system 101 includes a processing device 103 such as a central processing unit (CPU) that includes one or more microprocessors. This CPU 103 is coupled to the graphics processing unit (GPU) 105 via one or more buses, as is known in this technology.
The data processing system 101 further comprises a hardware video decoder, a software video decoder, or a video decoder embodied in both hardware and software for decoding compressed and optionally encrypted media data. It also has 107. The video decoder 107 is used to decode video from a camera or from a storage device that stores video content, such as a movie. The data processing system 101 includes one or more memory devices such as RAM, flash, etc. that can store the data used to generate the displayable content and the displayable content itself. The memories 109A, 109B and 109C are, as is known in the art, individual memories such as a single integrated circuit or multiple integrated circuits embodying a single memory space or the like, or the same memory. The CPU 103 and GPU 105 generate images or other data to be displayed and store those images in a buffer of memory. As shown in the embodiment of FIG. 1, the GPU 105 generates an image to be displayed and stores the image in the memory 109B and the memory 109C, while the CPU 103 stores the image to be displayed it generated. Remember to 109B. Alternatively, the CPU 103 can write to both memories 109B and 109C, while the GPU can write to one or both memories. Memory 109A is used to store compressed or encrypted movie-like decoded video data. The output from the video decoder 107 is video data that can be temporarily stored in the memory 111. The memory 109B, the memory 109C, and the video data memory 111 are connected to the compositor 115.
The compositor 115 receives output from memory 109B, 109C and video data memory 111. In addition, the compositor 115 receives metadata (not shown) and uses known techniques to composite the images in memory 109B and 109C with the decoded video content from video data memory 111 using the metadata. .. The metadata identifies the location of the source rectangle, the location of the video data, and the location of the image in memory 109B and 109C to form a composite image. For example, the movie is displayed in a window with the position and size specified by the metadata, and this window is provided by other images such as desktops, or images of the user interface, such as memories 109B and 109C. It is displayed above the image. In one embodiment, the output from the compositor 115 is a 24-bit RGB output.
The output of compositor 115 is stored in the framebuffer. The framebuffer drives a video display (eg, internal display device 140 or external display device 150) from one or more memory buffers (eg, buffers 131-134) that contain a complete frame of data (ie, graphic content). It is equipped with a video output device. The information in the memory buffer contains a color value for each pixel on the screen of the display device. An additional alpha channel can be used to hold information about pixel transparency. In one embodiment, the output of compositor 115 is stored in buffer 131 (and optionally buffer 132). The contents of buffer 131 are used to drive the internal display device 140.
In one embodiment, the processing device 101 also includes a scaler 120. The scaler 120 performs a scaling operation on the data stored in the buffer 131. This allows a mode of operation in which the graphic content is cloned (mirrored) to a secondary display such as the external display device 150. The external display device 150 has different properties and / or capabilities than the internal display device 140, and the scaling operation allows the graphic content to be displayed correctly on the external display device 150. The scaling operation performed by the scaler 120 includes adjusting the size, resolution, orientation, color or other characteristics of the image. The scaler 120 performs scaling operations based on input commands received from CPU 103, GPU 105 or other sources. The output of the scaler 120 is stored in buffer 133 (and optionally buffer 134) as needed. The contents of buffer 133 are used to drive the external display device 150. The display devices 140 and 150 may be any one of various types of display devices, for example, LCD (liquid crystal display), LED (light emitting diode) display, organic LED display, CRT (cathode ray tube) display, and the like. it is obvious.
In one embodiment, the internal display device 140 and the external display device 150 support different frame rates. For example, the internal display device 140 supports 60 frames per second (fps), while the external display device 150 supports only 24 fps. Through the techniques described herein, the data processing system 101 attempts to mirror each frame displayed on the internal display device 140 to the external display device 150. However, the difference in refresh rate prevents each frame from being fully mirrored. The scaler 120 is configured to determine when each frame is rendered to the internal display device 140 and when the corresponding frame is rendered to the external display device 150. If the data processor 101 does not finish rendering the previous frame to the external display device 150 when attempting to render the next frame to the internal display device 140, the scaler 120 drops the next frame. It is configured to complete the rendering of the previous frame, and when the previous frame is complete, continue rendering the subsequent frame. This process helps to synchronize the graphic content displayed on both the internal display device 140 and the external display device 150.
FIG. 2 is a block diagram showing a data processing system and an external display for mirroring graphic content according to an embodiment of the present invention. In one embodiment, the data processing system 201 is a general purpose or special purpose computer with a tablet form factor as shown in FIG. A large portion of one side of the device comprises an internal display 203 on which a multi-touch sensitive touch screen input device is integrally mounted and aligned. The internal display 203 is one representation of the internal display device 140 shown in FIG. The data processing system 201 includes one or more buttons or switches, eg, button 205, that allow the user to interact with the device. The data processing system 201 also includes, among other things, a connector 207 that allows an external display device such as the external display device 221 to be connected. It is clear that the data processing system 201 includes other connectors, such as a connector for a charger or power supply, and a connector for synchronization, such as a USB (Universal Serial Bus) connector for a USB cable, and the like. There will be. The data processing system 201 also includes a wireless transceiver, such as a wireless transceiver for WiFi, or a wireless transceiver for cellular telephones or other communications.
In one embodiment, the data processing system 201 is connected to the external display device 221 through a cable 210. In other embodiments, the data processing system 201 and the external display device may be connected in another form, such as wireless communication, as described above. A large portion of device 221 on one side comprises an external display 223. The external display device 221 is one representation of the external display device 150 shown in FIG. 1, and includes, for example, a computer monitor, television, projector, or other display device. The external display device 221 comprises one or more buttons or switches, such as the button 225, to allow the user to interact with the device. The external display device 221 also includes, among other things, a connector 227 that allows the connection of the data processing system 201. It will be clear that the external display device 221 may also include other connectors, as described for the data processing system 201. In one embodiment, according to the techniques described herein, the graphic content generated by the data processing system 201 and displayed on the internal display 203 is cloned or mirrored to the external display device 221 and displayed on the external display 223. ..
FIG. 3 is a block diagram showing a more general embodiment of a display driven architecture for mirroring graphic content to an external display according to one embodiment. This architecture 300 includes components similar to the data processing system shown in FIG. 1, with some components such as compositor 115 and scaler 120 embodied within ASIC311 shown in FIG. This ASIC is an application-specific integrated circuit configured to provide the functionality of related components. The memories 109A, 109B and 109C of FIG. 1 are embodied as random access memory (RAM) 307 coupled to the other components shown in FIG. 3 through bus 309. Non-volatile storage 313 stores software such as the operating system components described herein, as well as user applications such as web browsers, e-mail applications, word process applications, document viewing applications, and other known user applications. To do. The CPU 103 and GPU 105 in FIG. 1 are the processing devices 303 and GPU 305 in FIG. The internal display 340 shown in FIG. 3 is the same as the internal display 140 shown in FIG. In another embodiment, it is clear that ASIC311 embodies processing device 303 and GPU 305, and memory 307 and non-volatile storage device 313 may be replaced with flash memory coupled to ASIC311 via bus 309. Let's go.
FIG. 4 is a block diagram showing a software stack for mirroring graphic content to an external display according to an embodiment of the present invention. The software stack 401 may be embodied in the data processing system 101 shown in FIG. 1 or may be embodied in the data processing system 300 shown in FIG. The various software components of software stack 401 may be stored in memory such as RAM 307 or non-volatile storage device 313 or a combination thereof during execution of the software components. These components may be stored in a non-volatile storage device such as a hard drive or flash memory when it is not executed.
The software stack comprises multiple display drivers, such as the internal display driver 403 and the external display driver 405, each of which can be configured to communicate with other software components, such as the framebuffer library 411, or other components. .. The internal display driver 403 and the external display driver 405 have conventional operations with respect to the control of displays known in the art (eg, internal display device 140 and external display device 150, respectively), in addition to aspects relating to embodiments described herein. Can be carried out. The software component of software stack 401 uses the customary call and return process in which the calling application calls to another software process and waits for the return of the value that is the answer to that call. In addition, software components can make these calls using the application programming interfaces (APIs) described herein.
The framebuffer library 411 of software stack 401 implements software routines that manage framebuffers, such as framebuffers 131-134, to drive one or more displays in a data processing system. The window server software component 413 implements a known software process for managing windows for an application. In addition, window server 413 makes API calls to manage the operation of compositor 115 and scaler 120. For example, the window server 413 instructs the compositor 115 to composite the media data received from memories 109B and 109C into a single image frame stored in buffer 131. In one embodiment, the frame represents an image that is displayed for 1/60 seconds. Further, the window server 413 instructs the scaler 120 to perform a scaling operation on the data in the buffer 131 in order to make it suitable for display on the external display device 150.
The application launcher 417 is a software process that allows a user to launch a plurality of applications at a time or only one application at a time based on an embodiment. In one embodiment, the application launcher is a software program known by Apple Inc. as a springboard, an application launcher provided on the iPhone. User application 407 is one of a plurality of user applications such as web browsers, document viewers, picture viewers, movie players, word process or text editing applications, email applications, or other known applications. The user application 407 provides a software framework or one or more software libraries for creating and drawing user graphic content or user interface objects, such as buttons, windows, and other user interface elements and components known in the art. Can be used.
FIG. 5A is a flowchart showing a clone mirroring method according to an embodiment of the present invention. The method 500 includes hardware (eg, circuits, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions performed on a processor to perform hardware simulation), or a combination thereof. It is carried out by the processing logic including. This processing logic is configured to mirror the graphic content to an external display device. In one embodiment, method 500 is performed by the data processing system 101 shown in FIG. 1 or the data processing system 300 shown in FIG.
Referring to FIG. 5A, at block 510, the method 500 receives graphic content from the application. The application includes, for example, a user application 407 that draws multiple pieces of graphic content intended to be displayed as a single image. The graphic content is temporarily stored in a memory such as memory 109B or 109C. At block 520, method 500 composites the graphic content into an image (ie, a frame) and stores the result in a framebuffer. The compositor 115 composites the content from the memory 109B and / or 109C into the image data at the instruction of the GPU 105, and stores the result in the buffer 131. At block 530, method 500 displays the composited image data from buffer 131 on an internal display such as internal display device 140.
At block 540, method 500 performs a scaling operation on the composited image data so that the image is properly displayed on an external display. The external display has different characteristics than the internal display, so the scaling operation involves adjusting the size, resolution, orientation, color or other characteristics of the image. In one embodiment, the scaling operation is performed by the scaler 120. The scaled image data output by the scaler 120 is stored in another framebuffer such as buffer 133. At block 550, method 500 displays the scaled image data from buffer 133 on an external display such as external display device 150.
FIG. 5B is a flowchart showing a clone mirroring method according to an embodiment of the present invention. This method 560 is accomplished by processing logic configured to mirror the graphic content to an external display device. In one embodiment, method 560 is performed by the data processing system 101 shown in FIG. 1 or the data processing system 300 shown in FIG.
Referring to FIG. 5B, in blocks 565-580, method 560 receives graphic content from the application, composites the graphic content, and puts the result in a framebuffer, as described above for blocks 510-540 in FIG. 5A. The steps of storing, displaying the frame of the composite image on the internal display, and performing a scaling operation on the frame of the composite image are performed. At block 585, method 560 determines if the front frame has completed display on an external display device where the graphic content is mirrored. For example, the window server software component 413 receives instructions from the external display driver 405 when the entire contents of a frame buffer, such as frame buffer 133, are displayed on the external display device 150. Since the operation of the scaler 120 is asynchronous, processing for subsequent frames of the image may have already started. At block 585, method 560 determines that the previous frame is completely displayed, and at block 590, displays the current frame of the scaled image data on an external display. However, if Method 560 determines that the previous frame is not completely visible, instead of displaying the current frame, it drops the current frame of the scaled image in block 595 and returns to block 565. And process subsequent frames. Dropping the current frame in this state helps prevent image delays and ensures that the internal and external displays are in sync.
FIG. 6A is a flowchart showing an automatic clone mirroring method according to an embodiment of the present invention. This method 600 is performed by processing logic configured to automatically mirror the graphic content to the external display device if the user application has not yet published the graphic content for display on the external display device. .. In one embodiment, the method 600 is performed by the data processing system 101 shown in FIG. 1 or the data processing system 300 shown in FIG.
Referring to FIG. 6A, at block 610, method 600 detects the presence of an external display device connected to the data processing system. In one embodiment, the external display driver 405 sends a signal to other software components when a cable (eg, cable 210) is connected to connector 207 of the data processing system 201. At block 620, method 600 mirrors the graphic content drawn by the user application to an external display device. This mirroring is performed according to method 500 described above with reference to FIG. 5A. In one embodiment, the data processing system is configured to constantly mirror the graphic content regardless of whether an external display is connected. However, in other embodiments, mirroring is enabled when an external display is detected.
At block 630, method 600 determines whether a user application running in a data processing system draws an individual image specifically intended to be displayed on an external display device. Some applications display separate content on internal and external displays. For example, a media player application (eg, a movie player) displays the actual media content on an external display device while displaying controls (eg, play, pause, stop, fast forward, rewind, etc.) on the internal display device. .. The application specifies specific content for either an internal or external display in metadata with images or through other identifiers. At block 630, method 600 searches for specifically specified graphic content for an external display. At block 630, method 600 continues to mirror the graphic content to the external display at block 620 if the application determines that it is not giving a separate image for the external display device. At block 630, method 600 disables mirroring to the external display device and displays the individual image on the external display when the application determines that it is giving a separate image for the external display device. To do.
FIG. 6B is a flowchart showing an automatic clone mirroring method according to an embodiment of the present invention. In one embodiment, the method 650 is performed by the data processing system 101 shown in FIG. 1 or the data processing system 300 shown in FIG.
Referring to FIG. 6B, at block 660, method 650 detects the presence of an external display device connected to the data processing system, as in FIG. 6A. In other embodiments, the processing system is preconfigured with default settings (eg, mirroring enable or disable) that are used without detecting the presence of an external display device. At block 670, method 650 disables mirroring to an external display device. In one embodiment, this is the default setting and therefore no changes are required. At block 680, method 650 determines whether a user application running in a data processing system draws an individual image specifically intended to be displayed on an external display device. At block 680, the method 650 keeps the mirroring in a disable state and displays the individual image on the external display when the application determines that it has given the individual image for the external display device. At block 680, when method 650 determines that the application has given a separate image for an external display device, at block 690, mirroring is enabled and the graphic content is mirrored to the external display. In other embodiments, the data processing system first determines whether the application has drawn a separate image for display on an external display device, and then determines whether mirroring should be enabled or disabled accordingly. To do.
FIG. 7 is a block diagram showing a timeline for the clone mirroring operation according to the embodiment. The timeline 700 shows the relative timing of different behaviors associated with mirroring graphic content to an external display. These operations are performed by a data processing system such as the data processing system 101 shown in FIG. 1 or the data processing system 300 shown in FIG. In this embodiment, the timeline 700 includes three separate timelines showing a subset of the operations performed by the CPU (eg, CPU 103), GPU (eg, GPU 105) and scaler (eg, scaler 120). Timeline 700 also shows, in one embodiment, how the CPU, GPU and scaler schedule operations asynchronously to each other. Asynchronous scheduling allows operations to be performed on a single frame until all operations have been completed for the previous frame. This will use system resources more efficiently.
In one embodiment, the CPU serves to generate a list of composite commands for a series of frames A, B, and C. Generating a list of composite commands involves identifying which fragments of graphic data (and their position in memory 109B, 109C) should be composited into each frame. Generating a list of composite commands for frame A takes some length of time (eg, time t0 to time t1). When the CPU finishes generating the list of composite commands for frame A at t0, the GPU begins executing composite commands for frame A to generate content for output buffer A (eg, framebuffer 131). Executing a composite command for frame A takes a certain length of time (eg, time t1 to time t3). If the CPU has no other instructions to process (eg, from another application), the CPU will start generating a list of composite commands for subsequent frame B. This occurs, for example, at time t2, before the processing of frame A by the entire system is complete.
When the GPU finishes executing the composite command for frame A at time t3, the scaler begins executing the scaling command for the contents of output buffer A. Scaling commands include the commands described above for the scaler 120, or other commands. If the GPU has no other action to perform, at time t4, the GPU begins executing a composite command on frame B to generate content for output buffer B. When the GPU finishes executing the composite command for frame B at time t5, the scaler begins executing the scaling command for the contents of output buffer B. Similarly, when the GPU starts executing a composite command for frame B at time t4, the CPU starts generating a list of composite commands for another subsequent frame C. At time t6, the GPU starts executing composite commands for frame C to generate content for output buffer C, and at time t7, the scaler starts executing scaling commands for the content in output buffer C. To do. Execution of operations by the CPU, GPU and scaler can continue in a similar asynchronous manner for all subsequent frames to be processed by the data processing system.
8A and 8B are diagrams showing an embodiment of a scaling operation when the graphic content is mirrored to an external display according to the embodiment. The scaling operation shown in FIGS. 8A and 8B involves rotating the image. In certain embodiments, the data processing system 801 is configured to automatically rotate the image so that the image always appears in the same direction to the user when the data processing system 801 is turned or rotated. .. For example, the house shown on the display of the data processing system 801 is correctly oriented regardless of how the data processing system 801 is turned in FIGS. 8A and 8B. This orientation can affect the mirrored image on the external display.
In FIG. 8A, the data processing system 801 and the external display device 821 are similarly oriented (ie, so that the long edges are horizontal). When the graphic content is mirrored by the data processing system 801 (eg, via method 500) to the external display device 821, the scaler does not need to rotate the image because the image is still displayed correctly. However, in FIG. 8B, the data processing system 801 is rotated and oriented at different angles with respect to the external display device 821. When the graphic content is mirrored directly to the external display device 821, the image appears to be turned on its side. The scaler (eg, scaler 120) receives information about the orientation of the data processing system 801 from other system components or via user input and rotates the image accordingly during the scaling process. As a result, the image is correctly oriented when displayed on the external display device 221. By rotating the image, the aspect ratio can change between the internal display and the external display. In certain embodiments, black bars may be added to the displayed image to maintain the correct ratio, as shown in FIG. 8B.
The scaling operations shown in FIGS. 8A and 8B also include adjusting the aspect ratio of the displayed image. For example, in one embodiment shown in FIG. 8A, the internal display of the data processing system 801 has a first aspect ratio, eg 4: 3. The external display device 821 has an aspect ratio different from that of the internal display, for example 16: 9. The scaler (eg, scaler 120) may receive information about the aspect ratio of the external display device 821 from the external display device 821 itself or via user input and adjust the aspect ratio of the image accordingly during the scaling process. it can.
FIG. 9 is a block diagram showing a normative API architecture used in certain embodiments of the present invention. As shown in FIG. 9, API architecture 1100 includes API implementation components 1110 (eg, operating system, library, device driver, API, application program, or other module) that implement API 1120. API1120 identifies one or more functions, methods, classes, objects, data structures, formats and / or other features of API enforcement components used by API call component 1130. API1120 can identify at least one call convention that specifies how a function of an API enforcement component receives a parameter from an API call component and how that function returns a result to the API call component. .. API call component 1130 (eg, operating system, library, device driver, API, application program, or other module) uses API1120 to access and use the features of API enforcement component 1110 identified by API1120. Make an API call through. The API implementation component 1110 responds to the API call and returns a value to the API call component 1130 via API 1120.
It is clear that API Enforcement Component 1110 contains additional features, methods, classes, data structures, and / or other features that are not specified via API 1120 and cannot be used with API Call Component 1130. It should be understood that the API call component 1130 may be on the same system as the API enforcement component 1110, or it may be located remotely and access the API enforcement component 1110 using the API 1120 over the network. Figure 9 shows a single API call component 1130 that interacts with API 1120, but other API call components written in a different language (or the same language) than API call component 1130 use API 1120. Please also understand that it is good.
The API Enforcement Component 1110, API 1120, and API Call Component 1130 are stored on a machine-readable medium that includes a mechanism for storing information in a form readable by a machine (eg, a computer or other data processing system). .. For example, machine-readable media include magnetic disks, optical disks, random access memory, read-only memory, flash memory devices, and the like.
In the normative embodiment of Figure 10 (software stack), an application can make a call to service A or B using the service API and to the operating system (OS) using the OS API. Services A and B can make calls to the OS using the OS API.
In the above description, the present invention has been described with reference to specific normative embodiments. It will be apparent that various modifications can be made without departing from the broad spirit and scope of the invention as defined in the claims. Therefore, the specification and drawings are merely examples and do not imply any limitation.
In order to better understand the many embodiments of the present invention, examples of specific systems, components, methods, etc. have been described in detail. However, those skilled in the art will appreciate that at least some embodiments of the invention can be embodied without these particular details. In other respects, in order to avoid unnecessarily obscuring the present invention, well-known components or methods are not described in detail and are expressed in simple block diagram form. Therefore, the specific details mentioned above are merely examples. Specific embodiment may vary from these normative details and may be considered separately within the scope of the present invention.
Embodiments of the present invention include the various operations described above. These operations are performed by hardware components, software components, or a combination thereof. The term "join" as used herein means being directly joined or indirectly joined via one or more intervening components. Also, any signal given via the various buses described herein is time-multiplexed with other signals and given via one or more common buses. In addition, the interconnection between circuit components or blocks is shown as a bus or as a single signal line. Alternatively, each of the buses may be one or more single signal lines, or each of the single signal lines may be a bus.
Certain embodiments may be embodied as computer program products that include instructions stored on machine-readable media. These instructions are used to program general purpose or special purpose processors to perform the operations described herein. Machine-readable media include mechanisms for storing or transmitting information in a form readable by a machine (eg, a computer) (eg, software, processing application). Machine-readable media include magnetic storage media (eg floppy disks), optical storage media (eg CD-ROM), magnetic / optical storage media, read-only memory (ROM), random access memory (RAM), and erase. It includes, but is not limited to, possible programmable memory (eg, EPROM and EEPROM), flash memory, or another type of medium suitable for storing electronic instructions.
Further, certain embodiments are embodied in a decentralized computing environment in which machine-readable media are stored and / or executed by two or more computer systems. In addition, the information transferred between the computer systems is pulled or pushed across the communication medium connecting the computer systems.
The digital processing equipment described herein includes one or more general-purpose processing equipment such as a microprocessor or a central processing unit, a controller, and the like. Alternatively, the digital processing device may be one or more special purpose processing devices, such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like. The target processing device may be included. In another embodiment, for example, the digital processor may be a network processor having a plurality of processors including a core unit and a plurality of microengines. Further, the digital processing device may include a general-purpose processing device and a special purpose processing device in combination.
The operations of the methods described herein have been illustrated and described in a particular order, but the order of the operations of each method may be changed to perform one operation in the reverse order, or at least partially perform one operation with another. It may be carried out at the same time. In another embodiment, the sub-operations of the instructions or individual operations may be intermittent and / or alternate.
In the above description, embodiments have been described with respect to purpose or in a purpose-oriented environment. It should be understood that the present invention is not limited to embodiments in a goal-oriented environment, and that other embodiments may be embodied in other programming environments having characteristics similar to the goal-oriented concept.
In the above description, the present invention has been described with reference to specific normative embodiments. However, it is clear that various changes and modifications can be made without departing from the broad scope of the invention as defined in the claims. Therefore, the specification and the accompanying drawings are merely examples and do not imply any limitation.
101: Data processing system 103: Processing unit (CPU) 105: Graphic processing unit (GPU) 107: Video decoder 109A, 109B, 109C: Memory 115: Compositor 120: Scaler 131-134: Buffer 140: Internal display device 150: External display device 201: Data processing system 203: Internal display 205: Button 207: Connector 210: Cable 221: External display device 223: External display 225: Button 227: Connector 300: Architecture 303: Processing device 305: GPU 307: Memory 309: Bus 311: ASIC 313: Non-volatile storage device 340: Internal display 401: Software stack
14 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| JP2017092711A | Cited by | Japan | Search report |
| JP2014068315A | Cited by | Japan | Search report |
| JP2015095814A | Cited by | Japan | Search report |
| JP2017092711A | Cited by | Japan | Search report |
| JP2021179972A | Cited by | Japan | Search report |
| JP2014068315A | Cited by | Japan | Search report |
| JP2021179972A | Cited by | Japan | Search report |
| JP2005208684A | Cites | Japan | Search report |
| JP2006337859A | Cites | Japan | Search report |
| JPH1091135A | Cites | Japan | Search report |
19 members in 7 offices
Priority claims10
| Document | Office | Kind | Date |
|---|---|---|---|
| 201161431776 | United States of America | P | |
| 201161431776 | United States of America | P | |
| 61431776 | United States of America | – | |
| 13154268 | United States of America | – | |
| 201113154268 | United States of America | A | |
| 201113154268 | United States of America | A | |
| 2011154268 | – | – | – |
| 2011431776 | – | – | – |
| US201113154268 | – | – | – |
| US201161431776P | – | – | – |
Members19
| Document | Office | Kind | |
|---|---|---|---|
| EP2474904A2 | European Patent Office (EPO) | A2 | |
| US2012176396A1 | United States of America | A1 | |
| KR20120081574A | Republic of Korea | A | |
| WO2012096967A1 | World Intellectual Property Organization (WIPO) | A1 | |
| EP2474904A3 | European Patent Office (EPO) | A3 | |
| JP2012177902AThis record | Japan | A | |
| CN102681810A | China | A | |
| TW201243822A | Taiwan Province of China | A | |
| KR20140053934A | Republic of Korea | A | |
| JP5492232B2 | Japan | B2 | |
| KR101401216B1 | Republic of Korea | B1 | |
| JP2014160469A | Japan | A | |
| US8963799B2 | United States of America | B2 | |
| TWI480851B | Taiwan Province of China | B | |
| US2015130842A1 | United States of America | A1 | |
| CN102681810B | China | B | |
| US9411550B2 | United States of America | B2 | |
| US2017017452A1 | United States of America | A1 | |
| US9864560B2 | United States of America | B2 |
21 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written permission of extension of timeJAPANESE INTERMEDIATE CODE: A602A602 | A602 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 |
Numbers
- Publication
- 2012177902
- Publication, DOCDB
- 2012177902
- Publication, EPODOC
- JP2012177902
- Application
- 15890
- Application, DOCDB
- 2012015890
- Application, EPODOC
- JP20120015890
Titles2
- Japanese
- グラフィックコンテンツの外部ディスプレイへのミラーリング
- English
- Mirroring graphic content to an external display
Classification
- CPC, 11
- G06F3/1431
- G09G5/395
- G06F2200/1614
- G09G5/12
- G09G2340/04
- G09G2340/0407
- G09G2340/0435
- G09G2340/0485
- G09G2360/04
- G09G5/373
- G09G5/377
- IPC, 5
- G09G5 00
- G06F3 048
- G06F3 0484
- G09G5 36
- H04N5 445