Dynamic adjustment of cloud game data streams to output device and network quality

Dynamic adjustment of cloud game data stream for output device and network quality

Dynamic adjustment of cloud game data streams to output device and network quality

This disclosure is about the dynamic adjustment of data streamed through the network. In detail, the aspect of the present disclosure relates to a system and method for determining the display capability of an output device and then formatting two or more data streams assembled and displayed on the output device.

With the increasing popularity of digital streaming services and various cloud-based computing solutions, the ability to quickly and accurately transfer large amounts of data between remote devices is a critical task. Currently, the data streaming service does not consider the specifications of the device that intends to display data on it. Therefore, scaling and pixel changes between devices can produce undesirable display situations, for example, text is unreadable on a smaller screen or the aspect ratio of the display may not be suitable for a particular media segment.

In addition, when using applications that are sensitive to latency, such as video games, digital streaming services and cloud-based computing solutions can experience limitations in terms of network quality and bandwidth, which are remote Created or used during data transfer between devices. These limitations can cause delays in data transmission and can therefore cause delays, which often produce inconsistencies during the use of the application. Although the client device will try to achieve the lowest latency through a variety of methods, it is inevitable that due to the difference in factors such as the decoding speed of the transmitted data, the rendering rate, the input polling or even the client network connection, each user The end device will experience different amounts of delay. In some forms of media, the latent display of certain media, such as text or user interface (UI) data, can be more detrimental to the users enjoyment, resulting in user frustration and perhaps even abandoning the title or completely. platform.

In addition, on a variety of media playback devices available to consumers, the orientation of the display can change rapidly depending on the needs of the user. Certain device orientations may make it almost impossible to observe the media title, which would otherwise be normally visible in the alternative orientation. Therefore, in this technology, it is necessary to find alternative measures for adjusting and displaying the media streamed through the network, which takes into account network delay, display orientation, and specific display capabilities of various devices.

According to certain aspects of the present disclosure, a non-transitory computer-readable medium may contain computer-readable instructions embodied therein. These instructions can be combined to implement a method when executed. The method may include determining the display capability of the output device. It is possible to create and format two or more data streams that are assembled and displayed on the output device. The data stream can include Separate streams of data and user interface (UI) data. The data stream can then be sent to the output device.

According to some implementations of the present disclosure, a method for streaming data to an output device may include determining the display capability of the output device. It is possible to create and format two or more data streams that are assembled and displayed on the output device. The data stream may contain separate information for video data and user interface (UI) data. The data stream can then be sent to the output device.

According to some implementation schemes of the present disclosure, a computing system may include at least one processor unit and at least one memory unit coupled to the at least one processor unit. At least one processor unit and at least one memory unit can be configured to perform a method. The method may include determining the display capability of the output device. Create and format two or more data streams that are assembled and displayed on the output device. The data stream may contain separate information for video data and user interface (UI) data. The data stream can then be sent to the output device.

<p>10, 100Internet</p><p>12, 103, 104Output device</p><p>14Video data</p><p>16Text</p><p>18Map information</p><p>102Server</p><p>120,120'Streaming data</p><p>140, 140', 190Stream</p><p>150,150'Prioritized stream</p><p>182Request</p><p>183Device Information</p><p>184,184'Display ability information</p><p>186Ability</p><p>188Data flow</p><p>200Block Diagram</p><p>206, 206', 206"User Interface</p><p>231, 231', 231" processor</p><p>232, 232', 232"Memory unit</p><p>233, 233', 233" program</p><p>234, 234', 234"mass storage device</p><p>236, 236', 236"Data</p><p>237, 237', 237"display unit</p><p>239, 239', 239"Network interface</p><p>240, 240', 240"Support circuit</p><p>241, 241', 241"input/output (I/O) circuit</p><p>242, 242', 242"Power Supply (P/S)</p><p>243, 243', 243"clock</p><p>244, 244', 244"Cache</p><p>245', 245"controller</p><p>250, 250', 250"Bus</p><p>260Telecom Network</p><p>370Server-side command set</p><p>372, 374, 376, 378Server command</p><p>480A collection of output device commands</p><p>482, 484, 486, 488output device command</p>

By considering the following detailed description in conjunction with the accompanying drawings, the teachings of this disclosure can be easily understood, in the drawings: picture<b>1</b>For an example of a screenshot of a part of a media title displayed on the output device, the media title is derived from data streamed through the network, and FIG. 1 is intended to illustrate various components for displaying the title.

picture<b>2</b>The flow of an exemplary system based on certain aspects of the present disclosure picture.

picture<b>3</b>It is a block diagram of an exemplary system according to certain aspects of the present disclosure.

picture<b>4</b>It is a flowchart of an exemplary asset management technique from the server side according to certain aspects of the present disclosure.

picture<b>5</b>It is a flowchart of an exemplary asset management technique from the output device side according to certain aspects of the present disclosure.

Although the following detailed description contains many specific details for illustrative purposes, anyone who is generally familiar with the art will understand that many changes and modifications to the following details are within the scope of the present invention. Therefore, the exemplary implementation scheme of the present disclosure described below is explained without any general loss to the requested invention and without imposing restrictions on the requested invention.

<b>introduce</b>

The aspect of this disclosure relates to a system and method for dynamically adjusting UI and video streaming data in response to different device types, orientations, and network quality.

According to some aspects, a client device that is configured to operate on the network can provide a user with a list of one or more digital assets that can be borrowed from the providing user. You can then request the use of assets that can be borrowed from the user who makes the offer. The user can then use it by himself The person receives certain rights, such as access to applications or application features. An alternative embodiment provides a method in which the user making the provision can grant asset rights to another user without first receiving the request.

<b>Implementation plan details</b>

In the picture<b>1</b>, Provides an example of a screenshot of the media title displayed on the output device. The media title is<b>10</b>The streaming data is derived. This example is intended to illustrate various components of a streaming title. For example, in the output device<b>12</b>Above, depending on the orientation of the device or the quality of the network connection, various parts of the display can vary in quality and clarity. For example, such as<b>14</b>The video data can be clear when displayed independently of the device, where the video data is displayed on the device<b>14</b>. However, such as text<b>16</b>And map information<b>18</b>The UI data may be difficult to properly display on a smaller screen, or may appear unreadable or unusable when formatted to fit a smaller display.

Now go to figure<b>2</b>, To provide information on how to access the Internet<b>100</b>Illustrative examples of streaming data and displaying media titles on various devices. In detail, the figure<b>2</b>Describes an exemplary method flow for providing several data streams for different output devices according to certain implementation schemes of the present disclosure. It should be noted that the figure<b>2</b>The exemplary method of providing application assets in the<b>4</b>and<b>5</b>One or more aspects common to the methods indicated in. It is also important to note that the figure<b>2</b>The examples are only simplified examples, and the purpose is to illustrate only certain aspects of how the data stream can be provided according to the present disclosure.

Initially, it is important to note that, in this example, the data stream is provided to an output device that requests the stream from a server containing media information, but the data can be requested from an alternative source and the data can be delivered by the alternative source. In this example, as shown in<b>2</b>As shown, using the output device<b>103</b>User to server<b>102</b>Request a stream of media titles. The server receives this request<b>182</b>And determine the output device from the list of compatible devices<b>103</b>Ability<b>186</b>. Related capabilities can include parameters such as: screen size in pixels, physical screen size, screen resolution, color format (e.g., 8-bit, 15-bit, 16-bit, 24-bit, 30-bit, 36 Bit or 48-bit color) and similar parameters.

Additional examples of parameters that can be used include but are not limited to the following examples: pixel aspect ratio (usually assumed that the pixels are square and have the same width and height. Especially for old SD TV signals, this may not be the case); color space information (RGB, YUV, etc.), gamma correction information, screen orientation, display backlight level ("brightness level"), display refresh rate. Regarding the refresh rate, it may be useful to know the maximum refresh rate (60 Hz), but the device can also support lower or even variable refresh rates.

Some of these parameters can change over time. For example, for handheld devices such as tablets and smart phones, the orientation can be changed. When streaming to a device connected to a different display device at the local end, the display capability may also change. For example, a user who is streaming video to a device such as a tablet or phone can suddenly connect the device to a 4K display. This can change some of these capabilities.

There may also be some other related graphics processing units (GPU) parameter. For example, some GPUs (especially GPUs on embedded platforms) support video overlays. Specifically, the frame may have multiple layers, and these layers are combined ("composited") in a certain manner before the frame is output to the display. This can allow independent presentation of video and UI. For example, the video stream can be presented on a different layer from the UI. The UI and the video stream can have different update frequencies and neither need to "wait" for each other. For example, digital TVs use overlays to layer the system menu system on top of the "TV image".

In an alternative embodiment, the server can request device information from the output device when the display capability is unknown<b>183</b>, And the output device can then send the information to the server<b>184</b>. For example, the output device may include one or more sensors, such as an inertial sensor and/or a user-facing camera, which allows the output device to sense the orientation of the display screen relative to the user. The display screen can be rectangular but has a non-square aspect ratio (ratio of width to height). server<b>102</b>The output device can be requested to indicate whether the output device is in a horizontal orientation or a vertical orientation relative to the user, so that the data to be streamed can be optimally formatted for presentation in this orientation.

Information about device capabilities (such as whether the device supports 480p, 720p...) gives server information, which is technically possible. It would also be useful for the server to know certain current settings of the device such as orientation and output resolution. Other information such as color depth can be secondary, for example, because it can be adjusted at the client device, or the server will specify this information for compression efficiency reasons or to reduce the required bandwidth.

server<b>102</b>You can also request information about the relationship between the server and the output Device<b>103</b>Network between<b>100</b>The quality of information. Such information may include parameters such as network delay, available network bandwidth, packet loss, or network protocol. This information can be used to format the data stream<b>188</b>. For example, some network protocols such as the Unified Datagram Protocol (UDP) are highly unreliable. If the server<b>102</b>Know the internet<b>100</b>Use UDP, then stream<b>188</b>Can be assembled to include output device<b>103</b>It can be used to retrieve the lost data packets and forward error correction (FEC) packets.

Streaming transmission technology often uses FEC technology. Before starting streaming to the client device, the server sets a bandwidth budget for streaming, such as 8Mbps. To set the bandwidth budget, it is useful for the server to pick some client device settings such as video resolution, frame rate, and compression settings.

During the streaming transmission, the server can monitor the bandwidth and adjust the video resolution, frame rate and compression settings accordingly based on network parameters such as bandwidth and packet loss. Similar settings can be adjusted for audio streaming, and this adjustment can become more important. Knowing that the audio settings allow the server to optimize the audio stream for, for example, six-channel surround sound settings (for example, 5.1 surround sound) or stereo settings.

Certain streaming transmission techniques slightly reduce the frame rate or resolution slightly. For example, some video game streaming systems capture video from HDMI. Therefore, if the game generates 60fps and the streaming rate needs to be reduced to 30fps, the streaming server directly discards the frame. In addition, if the resolution needs to be reduced, the server can only scale the video frame received from the game. By way of example, the game can generate video at 720P, and if server<b>102</b>If you need to stream at 540P, only zoom the frame in the software.

Servers used for some applications such as online video games often run application software on a hardware different from the hardware used to handle the streaming of the video to the client device. The streaming server and application hardware can be located in the data center and connected via a local network. According to the aspect of the present disclosure, an application program that generates a video frame for the video stream can be used to complete the formatting of the video data. By way of instance rather than restriction, the streaming server can notify the hardware running the application that the hardware needs to reduce the frame rate or display resolution. The application can then adjust the frame rate or resolution or otherwise adjust the video formatting for the output device<b>103</b>. Applications that generate video frames can often perform better in adjusting the frame rate, display resolution, or other video formatting parameters. In addition, formatting the video in this way can even save a little power (a lower resolution or frame rate is less powerful for the server and therefore uses less power).

Once the device display capability information is determined, the server can create separate streams for video and UI information<b>188</b>. Perform this step so that the output device<b>103</b>The UI and video data information are separately formatted on the upper side, allowing separate scaling of the UI and video data to ensure that all aspects of the streaming title are clear and usable. These formatted streams are delivered<b>190</b>To output device<b>103</b>, In the output device<b>103</b>Receive and display these streams in a manner consistent with the display capability of the output device<b>140</b>. In an alternative embodiment, the output device can prioritize the display of the streams<b>150</b>. By reality By way of example rather than limitation, if each aspect of UI data is contained in a separate stream, the output device can prioritize the display of these streams in the order of (text data, video data, map data, etc.). Some aspects of the display remain clear during the time of the network delay, while other aspects become unclear.

As an example, the "game" stream used for online video games may be a compressed video stream. The UI stream used for the game may not necessarily be a video stream, but may be text, bit-mapped data, vectorized graphics, and other types of data as described above. Compatible with output device<b>103</b>Share these types of data. In some implementations, the output device<b>103</b>May already have some of the data locally, for example as memory<b>232’</b>Or cache<b>244’</b>Data in<b>236’</b>The part. For example, the font used to render text may already be available on the client. In some implementations, the output device<b>103</b>It may be possible to configure UP based on user settings. In this implementation, the user can, for example, use the output device<b>103</b>Any data above can override bitmaps or fonts to create a product of the game "mod" that is often used in massively multiplayer online games (MMO).

picture<b>2</b>The example in the example also provides an example of a second user requesting to stream a media title from the server, and is provided to show that the data can be streamed to various devices that each have their own corresponding data stream, so that a large amount of output can be achieved. Receive and display data in a consistent way with the display capabilities of the device<b>140’</b>。

It should be emphasized that the figures are provided for illustrative purposes only.<b>2</b>The examples are depicted in order to highlight certain aspects of this disclosure. In fact, this report The implementation plan of the case can be shown in the figure<b>2</b>Additional or alternative considerations that are not depicted in the examples are included, and are comparable to the figure<b>2</b>The simplified scheme depicted in is more complicated.

Some implementation schemes of the aspect of the present disclosure include a system configured to dynamically adjust the content of streaming transmission. By way of example rather than by way of restriction, the figure<b>3</b>The depiction includes three devices<b>102</b>、<b>103</b>and<b>104</b>The distributed computing system, and the computing system<b>102</b>、<b>103</b>and<b>104</b>It is configured to transmit data via the network according to certain aspects of the present disclosure. In some implementations, the device<b>102</b>Can be configured to perform with reference diagrams<b>2</b>And/or figure<b>4</b>A command of one or more patterns shared by the described patterns. In some implementations, the device<b>103</b>and<b>104</b>Can be configured to perform with the reference diagram above<b>2</b>Or the following reference picture<b>5</b>A command of one or more patterns shared by one or more of the described patterns. Device<b>102</b>、<b>103</b>and<b>104</b>Each of them can be configured with suitable software and/or hardware to implement various aspects of the methods described herein. Device<b>102</b>、<b>103</b>and<b>104</b>Each of these can be servers, embedded systems, mobile phones, personal computers, laptops, tablets, portable game devices, workstations, game consoles, portable devices such as smart watches, "server keys" (dongle)" devices and the like. As used herein, the term "server key device" refers to a device that is plugged into other equipment such as a computer or TV to add functions to the equipment. Such functions may include copy protection, audio services, video services, game services, data services, or other services that are only available when the server key is attached. Examples of server key devices include, but are not limited to, Internet content streaming that can be used on "smart" mobile phones Transmission device and infrared remote control adapter. Output device<b>103</b>、<b>104</b>With server<b>102</b>Some components in common. In Figure 3, the server<b>102</b>The components are expressed as reference numbers without apostrophe, output device<b>103</b>The corresponding components are expressed as the same reference numbers with an apostrophe ('), and the output device<b>104</b>The corresponding components are expressed as the same reference numbers with double prime (").

According to some implementation schemes, the installation<b>102</b>It can be a server that can be configured to format and provide media streaming data, and the device<b>103</b>and<b>104</b>It can be an output device used by users who want to display media titles. server<b>102</b>Can be assembled to create and format media<b>188</b>, And, for example, use an Internet connection or a local area network connection to transfer data through the network<b>260</b>delivery<b>190</b>To output device<b>103</b>or<b>104</b>. It should be noted that the network<b>260</b>No need to connect to the Internet. In some implementations, streaming can occur from a device located in a given location to another device located in that location via a local area network. In addition, even if the Internet<b>260</b>It is a wide area network, which can also be realized by a technology different from the Internet, such as a cable network.

Even if the requested media title is the same, it can be targeted to the corresponding device<b>103</b>or<b>104</b>Format the data stream separately<b>188</b>, Because the device capability of the corresponding output device<b>186</b>Can vary. By way of example rather than restriction, data flow<b>188</b>It can include streams used for video data, UI data, text, inventory data, map data, or audio data. In some implementations, certain types of data streams can be compressed to reduce the number of bits of data that needs to be transmitted in the stream. In detail, video data and audio data are usually compressed before transmission.

Device<b>102</b>、<b>103</b>and<b>104</b>Each of them may include one or more processor units<b>231</b>、<b>231’</b>、<b>231”</b>, The one or more processor units<b>231</b>、<b>231’</b>、<b>231”</b>It can be configured according to well-known architectures, such as single core, dual core, quad core, multi-core, processor-coprocessor, unit processor, and the like. Device<b>102</b>、<b>103</b>and<b>104</b>Each of them may also include one or more memory cells<b>232</b>、<b>232’</b>、<b>232”</b>(For example, RAM, DRAM, ROM, and the like). Processor unit<b>231</b>、<b>231’</b>、<b>231”</b>Executable can be stored in memory<b>232</b>One or more programs in<b>233</b>、<b>233’</b>、<b>233”</b>, And the processor<b>231</b>、<b>231’</b>、<b>231”</b>Can be, for example, through the data bus<b>250</b>、<b>250’</b>、<b>250”</b>Access memory and operatively couple to memory<b>232</b>、<b>232’</b>、<b>232”</b>. Memory unit<b>232</b>、<b>231’</b>、<b>231”</b>Can include information<b>236</b>、<b>236’</b>、<b>236”</b>, And the processor unit<b>231</b>、<b>231’</b>、<b>231”</b>Available information<b>236</b>、<b>236’</b>、<b>236”</b>To implement the program<b>233</b>、<b>233’</b>、<b>233”</b>. According to various aspects of the present disclosure, used in devices<b>102</b>、<b>103</b>and<b>104</b>Data of each<b>236</b>、<b>236’</b>、<b>236”</b>Can include, for example, self-output devices<b>103</b>or<b>104</b>Transfer to server<b>102</b>Request for streaming data<b>120</b>, And for self-server<b>102</b>delivery<b>140</b>To output device<b>103</b>or<b>104</b>Specially formatted data stream<b>188</b>Set. The collection. Program<b>233</b>、<b>233’</b>、<b>233”</b>Optionally include instructions that, when executed by the processor, perform one or more operations associated with the following: request streaming of data for the media title<b>120</b>, To determine the capability of the output device from which the request was received<b>186</b>, Create and format two or more video streams containing video and UI data respectively<b>188</b>So that it can be displayed on the output device<b>103</b>or<b>104</b>Up, or receive and display<b>140</b>Formatted to best Use output device<b>103</b>or<b>104</b>Data flow that has the ability to display; for example, a kind of<b>2</b>,picture<b>4</b>And/or figure<b>5</b>A method with one or more characteristics shared by the method. For example, server<b>102</b>The program<b>233</b>、<b>233’</b>、<b>233”</b>Can include instructions that are executed by the processor<b>231</b>、<b>231’</b>、<b>231”</b>When executed, the server causes the server according to the figure<b>2</b>The method described in and/or the aspect of the server side of the sending of the streaming data is used to format and send the streaming data to at least one receiving device<b>103</b>or<b>104</b>. In an alternative embodiment, the server<b>102</b>The program<b>233</b>、<b>233’</b>、<b>233”</b>Can include instructions that are executed by the processor<b>231</b>、<b>231’</b>、<b>231”</b>When executed, the server causes the server according to the figure<b>2</b>The method described in and/or the aspect of the server side of the sending of streaming data to at least one receiving device<b>103</b>or<b>104</b>Request to display capability information. Output device<b>103</b>or<b>104</b>The program<b>233</b>、<b>233’</b>、<b>233”</b>Can include instructions that are executed by the processor<b>231</b>、<b>231’</b>、<b>231”</b>When executed, the output device is caused according to the figure<b>2</b>The method described in and/or the aspect of the output device side of the sending of the streaming data to request the streaming data<b>120</b>, Which can then be used by the server<b>102</b>Provide the streaming data. In an alternative embodiment, the output device<b>103</b>or<b>104</b>The program<b>233</b>、<b>233’</b>、<b>233”</b>Can include instructions that are executed by the processor<b>231</b>、<b>231’</b>、<b>231”</b>When executed, the output device is caused according to the figure<b>2</b>The method described in and/or the aspect of the output device side of the transmission of the streaming data is used to transmit the display capability information<b>184</b>To server<b>102</b>。

Device<b>102</b>、<b>103</b>and<b>104</b>Each of them can also include well-known support circuits<b>240</b>、<b>240’</b>、<b>240”</b>, Such as, for example, via the bus<b>250</b>、<b>250’</b>、<b>250”</b>Input/output (I/O) circuits that communicate with other components of the system<b>241</b>、<b>241’</b>、<b>241”</b>(In the output device<b>103</b>、<b>104</b>Can be Coupling to the controller<b>245’</b>、<b>245”</b>); power supply (P/S)<b>242</b>、<b>242’</b>、<b>242”</b>; Clock (CLK)<b>243</b>、<b>243’</b>、<b>243”</b>; And cache<b>244</b>、<b>244’</b>、<b>244”</b>. Device<b>102</b>、<b>103</b>and<b>104</b>Each of them can include mass storage devices as needed<b>234</b>、<b>234’</b>、<b>234”</b>, Such as disk drives, CD-ROM drives, tape drives, flash memory or similar, and mass storage devices<b>234</b>、<b>234’</b>、<b>234”</b>Programs and/or data can be stored. Device<b>102</b>、<b>103</b>and<b>104</b>Each of them may also include a display unit if necessary<b>237</b>、<b>237’</b>、<b>237”</b>. Display unit<b>237</b>、<b>237’</b>、<b>237”</b>It can take the following forms: cathode ray tube (CRT), flat screen, touch screen, or other devices that display text, numbers, graphic symbols, or other visual objects. Device<b>102</b>、<b>103</b>and<b>104</b>Each of them can also include a user interface<b>206</b>、<b>206’</b>、<b>206”</b>To promote the device<b>102</b>/<b>103</b>or<b>104</b>Interaction with users. user interface<b>206</b>、<b>206’</b>、<b>206”</b>This may include a keyboard, mouse, light pen, game control pad, touch interface or other devices. The user interface may also include audio I/O devices such as speakers and/or microphones.

User interface<b>206</b>、<b>206’</b>、<b>206”</b>Interact with any of the computer systems. By way of example, the server<b>102</b>Can be a cloud game server, and output device<b>103</b>or<b>104</b>Can be a cloud game client, and video game users can use the user interface<b>206</b>、<b>206’</b>、<b>206”</b>Interact with a video game, which is run by the server<b>102</b>Execute and stream to output device<b>104</b>. user interface<b>206</b>、<b>206’</b>、<b>206”</b>The part can be displayed on the display unit<b>237</b>、<b>237’</b>、<b>237”</b>The graphical user interface (GUI) on the<b>102</b>/<b>103 or 104</b>User interaction. system<b>102</b>/<b>103 or 104</b>Can include network interface<b>239</b>、<b>239’</b>、<b>239”</b>, The web interface<b>239</b>、<b>239’</b>、<b>239”</b>It is configured to allow the use of Wi-Fi, Ethernet ports, or other communication methods. Network interface<b>239</b>、<b>239’</b>、<b>239”</b>Suitable hardware, software, firmware, or some combination thereof can be incorporated to facilitate the passage through the telecommunications network<b>260</b>According to certain aspects of this disclosure, data transmission using unreliable protocols can be supported. Network interface<b>239</b>、<b>239’</b>、<b>239”</b>It can be configured to realize wired communication or wireless communication via a local area network and a wide area network such as the Internet.

As shown<b>4</b>As shown, the set of server-side commands 370 can be implemented by the server 102, for example. Server command<b>370</b>Can be formed in memory such as<b>232</b>、<b>232’</b>、<b>232”</b>Or mass storage device<b>234</b>、<b>234’</b>、<b>234”</b>On non-transitory computer-readable media. Server side command 370 can also be a process control program<b>233</b>、<b>233’</b>、<b>233”</b>The part. As indicated at 372, the server command<b>370</b>Can include information for requesting output device<b>183</b>Command, the output device information<b>183</b>If necessary, including going through the network<b>260</b>From one or more client devices<b>103</b>or<b>104</b>The display capability of the received output device. instruction<b>370</b>Can include methods for formatting two or more content streams based on the capabilities of the output device<b>188</b>Instruction<b>374</b>. Afterwards, in<b>376</b>Where the instructions can include<b>190</b>To output device<b>103</b>or<b>104</b>Instruction, and subsequently, in<b>378</b>The location may include instructions for formatting the data stream or delivering priority information to the corresponding output device.

As shown<b>5</b>As shown, for example, through the output device<b>103</b>or<b>104</b>Implement output device instructions<b>480</b>Set. The collection. Output device instructions<b>480</b>Can be formed in memory such as<b>232</b>、<b>232’</b>、<b>232”</b>Or mass storage device<b>234</b>、<b>234’</b>、<b>234”</b>On non-transitory computer-readable media. Output device instructions<b>480</b>It can also be a process control program<b>233</b>、<b>233’</b>、<b>233”</b>The part. As in<b>482</b>Location indication<b>480</b>It can include instructions for delivering output device information. The output device information includes<b>260</b>By one or more client devices<b>103</b>or<b>104</b>Transmission output device display ability<b>183</b>. instruction<b>470</b>Can include for receiving based on the output device's capabilities<b>140</b>Two or more content streams<b>188</b>Instruction<b>474</b>. Afterwards, in<b>476</b>Where the instructions can include<b>150</b>Data stream on output device<b>103</b>or<b>104</b>The instructions shown above, and subsequently, in<b>478</b>The place can include formatting text or other user interface (UI) to make it compatible with the output device<b>103</b>or<b>104</b>The display ability is consistent with the instructions to display the content of the streaming transmission.

The above components can be implemented by hardware, software, firmware or some combination thereof.

Although the above-mentioned example uses a game stream and a UI stream that are combined on the client side, it can also include additional streams, such as a video chat stream. Such additional streams can come from other users' client devices point-to-point and can be combined by these client devices. In some implementations, the server<b>102</b>On the program<b>233</b>It can be seen that the output device<b>103</b>and<b>104</b>There is a video chat stream between and resources can be reserved for handling this stream. Due to the delay, the server<b>102</b>It can coordinate the point-to-point (p2p) work phase to make the client device<b>103</b>、<b>104</b>Combine video chat streams with other streams. There may be multiple types of local content that can be compounded by the client device.

The aspect of this disclosure allows formatting and streaming of data in different formats, and the data in these different formats can be displayed together to optimally Display on different devices.

Although the above is a complete description of the preferred embodiments of the present invention, it is possible to use various alternatives, modifications, and equivalents. Therefore, the scope of the present invention should not be judged with reference to the above description, but should be judged with reference to the scope of the appended patent application and the complete scope of the equivalent of the scope of the patent application. Any feature described herein (whether preferred or not) can be combined with any other feature described herein (whether preferred or not). In the scope of the following patent applications, the indefinite article "One" refers to the number of one or more of the items following the article, unless expressly stated otherwise. The scope of the attached patent application shall not be interpreted as including component plus functional limitation, unless the term "component for..." is used in a given claim to explicitly state such limitation.