System for collaborative conferencing using streaming interactive video

TECHNICAL FIELD OF THE INVENTION

The present disclosure of the subject invention generally relates to the field of data processing systems that improve the ability of users to manipulate video and audio, and connect to them.

BACKGROUND

Recorded and phonorecords kinonositeli are a part of public life since the days of Thomas Edison. At the beginning of the 20th century were widespread written phonorecords (rolls and records) and kinonositeli (cinematography and movies), but both technologies are, however, in its early stage of development. At the end of the 1920s, movies were combined with sound market-based consumer goods, after that there were colored movies with sound. Broadcast gradually took the form of means of distributing audio broadcast mass market, advertising-supported. When in the mid-1940s, has set the standard television (TV) broadcasting and television joined the radio broadcast as a form of means of disseminating the mass market, and pre-recorded movies, or moving picture on the air came into the house.

By the mid-20th century, most American homes were phonographs to play recorded audio media, radio reception for direct audio transmission and a TV for playback of audio / video (A / V) direct transfer of media. Very often, these 3 "media player" (player, radio and television) have been merged into one body with the joint use of conventional speakers, which became a "media center" for the home. Although the media choices were limited to the consumer, "ecosystem" media was quite stable. Most consumers know how to use "media players", and to take full advantage of their functionality. However, media publishers (largely film studios and television studios and music companies) could supply its media and in theaters and for home use and do not suffer from widespread piracy or "secondary sales", ie resale of used storage medium. Generally, publishers do not receive income from "secondary sales", and as such, it reduces the revenue that publishers might otherwise receive from the purchaser used the information carrier as a result of new sales. Despite the fact that in the middle of the 20th century, of course, use the sold records, such sales do not greatly influence the publishers of records, because, unlike a movie or video - that adult usually looks once or only a few time - a musical soundtrack can be heard hundreds or even thousands of times. Therefore the support of musical information is much less "transient" (ie, it has a lasting value for the adult consumer) than the movie / video carriers. After buying the records if the user liked the music, it will probably be to store it for a long time.

Since the mid-20th century to the present media ecosystem has undergone a number of radical changes, both to the benefit of consumers and publishers, and to their detriment. At the widespread introduction of tape, especially cassette tapes with high quality stereo sound, of course, consumers have become much more convenient. But it also marked the beginning of what is now a widespread practice in relation to consumer media: piracy. Of course, many consumers have used cassette tapes to record on it their own records simply for convenience, but an increasing number of consumers (for example, students in the dormitory with free access to the collection plates to each other) could make pirated copies. In addition, consumers can write to tape the music on the radio, instead of buying the records or tape from the publisher.

The emergence of consumer VCR (video cassette recorder) has further convenience of consumers, since now a VCR can be set to record TV programs, which could see at a later time, and it also led to the creation of video, where access to films and TV programs can be provided "on demand". The rapid development of home storage mass market since the mid-1980s led to an unprecedented level of choice and convenience for consumers, but also led to the rapid expansion of multimedia publishing market.

At present, consumers are faced with a wide range of multimedia, as well as with a variety of storage devices, many of which are tied to specific types of media, or the publisher. An avid consumer of media can have a stack of devices connected to the TV and computers are in different areas of the house, resulting in a web of cables leading to one or more televisions and / or a personal computer (PC), as well as to a group of remote controls . (As used herein, the term "personal computer" or "PC" refers to any kind of computer suitable for home or office, including a desktop computer, or other Macintosh® computers that do not require the use of operating system Windows, Windows-compatible devices, varieties of UNIX, laptops, etc.) These devices may include a video game console, VCR, DVD-player, audio processor / amplifier surround sound, satellite set-top box, a television set-top box cable TV, etc. And for passionate consumers may be many devices with similar functions because of compatibility issues. For example, a user can belong to both HD-DVD, and Blu-ray DVD player, or as a Microsoft Xbox®, and Sony Playstation®'s video system. In fact, because of the incompatibility of some of the games for all versions of the game console user can belong to XBox, and later, for example Xbox 360®. Often consumers confusing that a video input and a remote control to use. Even after a disc is inserted into the appropriate player (e.g., DVD, HD-DVD, Blu-ray, Xbox or Playstation), the video input and audio input for this device and found appropriate remote control, the consumer, however, is faced with technical problems. For example, in the case of widescreen DVD, the user may need to first identify and then set the appropriate aspect ratio on your monitor or TV screen (eg 4: 3, Full, Zoom, Wide Zoom, Cinema Wide etc.). Similarly, the user may need to identify and then set the appropriate audio format from your surround sound system (for example, AC3, Dolby Digital, DTS, etc.). Most consumers are not aware that it may not use multimedia content in full functionality of your television or audio system (for example, when watching a movie in a compressed form in the wrong aspect ratio or when listening to audio with stereo sound, not surround sound) .

More and more data storage devices, Internet-based technologies are added to the device stack. Sound devices like system Sonos® Digital Music, streamed audio directly from the Internet. Similarly, devices like set-top box Slingbox ™, record video and transfer it to flow through the home network or via the Internet, where it can be viewed remotely on a PC. And IP-TV services (IPTV) offer services similar to cable TV, through digital subscriber line (DSL) or other Internet connection at home. Recently, also attempts to integrate multiple multimediafunktsy in a single device, such as Moxi® Media Center and PC operating system Windows XP Media Center Edition. While each of these devices offers an element of the mechanism for the functions that it performs, each offline ubiquitous and simple access to most media. Moreover, the production of such devices frequently cost hundreds of dollars, often because of the need for expensive processing and / or local storage. Additionally, these modern consumer electronic devices typically consume a lot of power, even in standby mode, which means that they are costly in time and energy is used inefficiently. For example, the device can continue to operate if the user forgot to turn it off or switched to another video input. And since none of the devices is not a complete solution, it must be integrated with another stack of devices in the home, that still leaves the user with a web of wires and a large number of remote controls.

Furthermore, when many of the latest devices, Internet-based technologies do work properly, they typically offer media in a more generic form than it might otherwise be available. For example, devices that transmit video streams via Internet, often streamed video only, not online "extras" that often accompany the DVD, such as "opinion of" video games or movie director commentaries. This is due to the fact that frequently the interactive material is displayed in a particular format intended for a particular device that handles interactivity locally. For example, each disk DVD, HD-DVD and Blu-ray has its own particular interactive format. Any home storage device or local computer that could be developed to support all popular formats may require a certain level of sophistication and flexibility that probably would make their operation unacceptably expensive and complicated for the consumer.

In addition to this problem, if the later is a new format, the local device may not exist hardware to support this new format, which will mean that the consumer will have to buy to enhance the local storage device. For example, if later introduced high-resolution video or stereoscopic video (e.g., one video stream for each eye), the local device may not have the computational power to decode the video, or it may not have the hardware to output the video in the new format (e.g. assume that a stereoscopic perception is achieved by a video at 120 frames / s, synchronized with the shutter glasses, the frames are supplied to each eye at 60 frame / s, if the video equipment the consumer can only support video at 60 frames / sec, this option It will not be available without the purchase of upgraded hardware).

The issue of complexity and obsolescence of storage devices is a serious problem when it comes to sophisticated interactive multimedia, especially video games.

Modern video game application is mainly divided into four major machine-dependent hardware platforms: Sony PlayStation® 1, 2 and 3 (PS1, PS2 and PS3), Microsoft Xbox® and Xbox 360® and Nintendo Gamecube® and Wii ™, and PC-based games . Each of these platforms is different from the others so that games written to run on one platform usually do not execute on a different platform. There may also be problems of compatibility devices from generation to generation. Despite the fact that most of the developers of gaming software create game programs, independent of a specific platform for the execution of a particular game on a particular platform requires its own software layer (often referred to as "the mechanism of the development of game programs") to adapt the game for use on a particular platform. Each platform is sold to the consumer as a "console" (ie, self-contained unit for connection to the TV or monitor / speaker), or she is a PC. Typically, the video games are sold on optical recording medium, for example Blu-ray DVD, DVD-ROM or CD-ROM, which contains the video game embodied as a complicated real-time application. Since the speed of home broadband increased, video games are becoming more and more available for download.

The requirements to the specific platform in order to achieve compatibility with software video games are extremely high because of the nature of real-time and high computational demands of advanced video games. For example, one would expect full compatibility of video games from generation to generation (for example, XBox to XBox 360 or the Playstation 2 ("PS2") for the Playstation 3 ("PS3")), the same way as there is a general compatibility of business applications ( for example, Microsoft Word) from one PC to another with a faster processor or core. However, with video games is not the case. Since manufacturers of video games with the release of the video game generation, as a rule, try to achieve the best possible performance for this price point, it is often significantly alter the architecture of the system so that many games written for the system of the previous generation is not executed on the system of a later generation. For example, XBox is based on processor families x86, while the XBox 360 is based on a family of PowerPC.

Methods may be used prior emulation architecture, but given that video games are real-time applications, it is often impossible to achieve an identical behavior emulation. This harms consumers, producers video game consoles and video games software publisher. For the consumer, this means the need for video game consoles, both old and new generation connected to the TV to be able to conduct all the games. For the console manufacturer it means cost associated with emulation and slower adoption of new consoles. And for the publisher, this means the need to release multiple versions of new games to reach all potential customers - not only the release of version for each brand video games (for example, XBox, Playstation), but often a version for each version of the brand (eg, PS2 and PS3). For example, a separate version of "Madden NFL 08" Electronic Arts has been developed for the XBox, XBox 360, PS2, PS3, Gamecube, Wii and PC, along with other platforms.

Portable devices, such as cellular ("cell") phones and portable media players, are also a problem for game developers. An increasing number of devices connected to the Wi-Fi data networks, and can upload a video game. But in the market there is a wide range of mobile phones and storage devices with a wide range of different display resolution and computing power. Furthermore, since these devices usually have restrictions on power consumption, cost and weight, they are generally not improved hardware acceleration performance of graphics operations, such graphics processor ("GPU"), such as devices manufactured Corporation NVIDIA, Santa Clara, CA (Santa Clara, CA). Consequently, game software developers typically develop a given computer game simultaneously for many different types of portable devices. The user may find that the computer game is not available specifically for his cell phone or portable media player.

In the case of the home gaming console manufacturers hardware platforms usually prescribed license fee for developers of game software for possible publication on its game platform. Cellular operators also usually prescribed license fee for the publisher of games to download games to a cell phone. In the case of computer games there is no license fee paid for the publication of games, but game developers tend to face higher costs due to higher levels of indirect costs of customer service to support a wide range of configurations of the PC and because of questions installations that may occur. In addition, PC, as a rule, are less obstacles to the piracy of game software as technologically proficient user can easily reprogram them, and the game can easily republish unlicensed and easier to disseminate (eg through the Internet). Accordingly, the developer of gaming software, there are costs and unfavorable conditions for publication on game consoles, mobile phones and PC.

For publishers gaming console and computer software costs are not limited to. To distribute games through retail channels, publishers appoint wholesale price below the selling price for retailers to get net income. The publisher also typically has to pay the cost of manufacturing and distributing the physical media containing the game. The retailer also often assigns to the publisher's expense "payment price protection" to cover any unforeseen expenses, such as when the game is not sold out, or if the value of the game is reduced, or if the retailer has to reimburse part or all of the wholesale price, and / or take back the game from buyer. In addition, retailers also tend to confer on the publisher the cost of paying sales support games in the leaflet. In addition, retailers increasingly are buying games for users who no longer play them, and then sell them as second-hand games, as a rule, without sharing the income from their sale with games publisher. Supplement to the burden imposed on the game publishers, is the fact that the games are often unlicensed republish and distribute over the Internet for users to download and manufacture of free copies.

Since the speed of broadband Internet access increases and broadband connections have become more wide-spread in the United States and around the world, in particular to the house and to the Internet-cafe, where rented PC, connected to the Internet, the game is increasingly spread through the boot in a PC or console. Additionally, broadband connections are increasingly being used to conduct multiplayer online games and massively multiplayer online games (both are in the present disclosure, designated the abbreviation "MMOG"). These changes reduce some of the costs and issues associated with the spread of retail. Download online game aimed at overcoming some of the adverse conditions for the publishers of games due to the fact that the distribution costs are generally reduced, and the cost of unsold information carrier are small or non-existent. But the game downloaded, still are subject to piracy, and because of their size (often many gigabytes in size), they can be loaded for a long time. In addition, a variety of games can fill the small drives (capacity) as those sold with portable computers or video game consoles. However, for large game or MMOG need to be able to lead this game by online connection, the problem of piracy is reduced, as is normally required that the user have a valid user account. Unlike linear media (such as video and music), which can be copied to the camera when shooting video display screen or a microphone for recording audio from the speakers, each experience with video games is unique and can not be copied using a simple video / audio recording. Accordingly, even in regions where it is not ensured strict compliance with copyright laws and piracy is rampant, MMOG can be protected from piracy, and hence can be supported entrepreneurship. For example, MMOG "World of Warcraft" media conglomerate Vivendi SA successfully introduced around the world, and does not suffer from piracy. Many online games or play MMOG, such as MMOG "Second Life" ("Second Life") company Linden Lab, form income for operators of games through economic models, built-in games, where you can buy, sell and even create resources using online tools means. Accordingly, to pay for the use of online gaming mechanisms may be used in addition to conventional programs or buy game subscription service.

While piracy in many cases may be reduced due to the nature of online (games), or MMOG, operator interactive games, however, it faces other problems. Many of the games to execute them properly requires considerable local (ie home) for online processing resources (games) or MMOG. If the performance of the local machine the user is low (for example, a computer without a GPU, the notebook entry-level), then it may not be able to play the game. Furthermore, as game consoles become obsolete, they will also lag behind the current level of development, and perhaps can not handle advanced games. Even if we assume that the local user's PC can satisfy the computing requirements of the game, there is often the difficulty of installation. There may be incompatible driver (for example, if a new game is loaded, it can install the new version of the graphics driver, which renders a previously installed the game, depending on the old version of the graphics driver, which will not function). The console can be run out of space on your hard drive is loaded a lot of games. Complex games tend to take patches are downloaded from the developer of games over time, as detected and corrected the error, or if modifications are made in the game (for example, if a game developer believes that the level of play is too complex or too simple). These patches need to download again. But sometimes, not all users are downloading all the patches. In other cases, the downloaded patches introduced other compatibility issues or consumption of disk space.

Also, during game play may need to download large amounts of data for the graphics or behavioral information to the local PC or console. For example, if a user enters the room in a MMOG and faced with a scene or a character composed of image data, or lines of conduct that are not on the local machine the user, then the data of the character or scene to be loaded. This can result in a significant delay during game play if the Internet connection is not fast enough. And if a character or a scene, faced require much memory or processing power, exceeds the capabilities of a PC or a local console, you can create a situation where the user can not continue and must continue to play with graphics of inferior quality. Accordingly, online game or MMOG games often limit their claims to the computational complexity, and / or memory. Additionally, they often limit the amount of data transfers during the game. And online games or games can also narrow the MMOG market of users who can lead these games.

In addition, experienced users technically increasingly decompile local copies of games and these games are modified so that they could cheat. Hoaxes can be as simple as to make the re-pressing faster than it can a person (for example, to shoot a firearm very quickly). In games that support in-game transaction resources, deception can reach the level of falsification, which results in a fraudulent transaction, which include the actual economic value of the resources. When the economic model of online (games) or MMOG games based on resources such transactions, this can result in consequences that cause significant harm to the operators of games.

The cost of developing a new game grew as PC and console became able to display all the more complicated the game (for example, with more realistic graphics, such as ray tracing in real time, and more realistic courses of action, such as physics simulation in real time). At the dawn of the video game industry to develop video games it has been a process very similar to the development of application software, that is, most of the development costs amounted to software development, in contrast to the development of such graphics, sound and behavior of elements or "resources" as those that can be designed for the movie spatial special effects. Currently, many of the program of work on the development of sophisticated video games are more like to develop films with special effects than software development. For example, many video games provide simulate 3D (three-dimensional) worlds and form a more photo-realistic (ie, computer graphics, which seem so realistic, as a frame with a picture of the game actors, photographically) characters, props and surroundings. One of the most promising aspects of the development of games is to create photorealistic computer-generated human face that can not be distinguished from a human face in the game actors. Capture technology person, such as Contour ™ Reality Capture, designed Mova, San Francisco, CA (San Francisco), capture and track the exact geometry of the actor's face with high resolution when it is in motion. This technology enables the visualization of 3D faces on a PC or a game console, which is virtually indistinguishable from the captured face in the game actors. Accurately capture and visualization "fotorealnogo" human face are useful in several respects. Firstly, often used in video games very easily recognizable celebrities or athletes (often well-paid), and deficiencies may be apparent to the user that distracting during play or makes viewing uncomfortable. Often, a high degree of detail is required to achieve a high degree of photorealism, demanding a large number of visualization polygons and high-resolution textures, possibly with the polygons and / or textures changing on a frame by frame basis during facial movements.

When the scene, there are a large number of polygons, with detailed textures are quickly replaced, PC or game console, the game supports, may not have sufficient RAM memory for storing data sufficient number of textures and polygons to the required number of frames of animation generated in the segment of the game. In addition, the PC or game console is usually available one optical disk drive or a magnetic disk, which is usually much slower than the RAM, and typically can not keep up with the maximum data rate that the GPU can receive when rendering polygons and textures. Current games typically charged majority of polygons and textures into RAM, which means that the scene is largely limited in complexity and duration of the volume of RAM. In the case of facial animation, for example, this may limit a PC or a game console, or to individuals with low resolution, which is not fotorealnym or to fotorealnogo entity that can be animated for only a limited number of frames, before the game pauses, and loads polygons and textures (and other data) for more frames.

Watching the slow movement of the progress bar on the screen when displaying a PC or console messages like "Loading ..." ("Loading ...") is accepted as an integral part of today's users lack of sophisticated video games. Delay after the next scene from the disc ("CD" in this description, unless otherwise specified, refers to nonvolatile optical or magnetic media, as well as nediskovym storage media, such as semiconductor flash memory) can take several seconds or even several minutes. It is a waste of time and can be quite frustrating player. As discussed earlier, most or all of the delay may be due to the load time of the polygon, textures or other data from a disk, but it may also happen that the part load time is spent on preparing the processor and / or GPU in the PC or console data for the scene . For example, the video game of football may allow players to select from a large number of players, teams, stadiums, and weather conditions. So, depending on what particular combination is chosen, for a scene may require different polygons, textures and other data (collectively "objects") (e.g., different teams have different colors and patterns on their form). It may be possible to transfer many or all of the various changes and pre-compute many or all of the objects in advance and save these objects on the disc used to store games. But if the amount of change is large, the amount of memory required for all objects that may be too large and does not fit the disc (or too unrealistic to download). Accordingly, existing PC and console systems are typically limited both in complexity and duration data reproducing scenes and suffer from long load times for complex scenes.

Another significant limitation systems, application software and video game systems in the prior art is that they are increasingly using large databases, e.g., of 3D objects such as polygons and textures, that need to be loaded into the PC or game console for processing. As discussed above, the load such databases can take a long time if stored on a local disk. Download Time, however, is usually much higher if the database is stored remotely and is accessed via the Internet. In such a case can take minutes, hours or even days to download a large database. In addition, the creation of such databases is often associated with high costs (for example, 3D model detailed sailing vessel, equipped with a high mast, for use in the game, a movie or a historical documentary), and they are intended for sale to the local end-user. However, the database is at risk of unauthorized use after loading a local user. In many cases, the user needs to download a database for evaluation to check whether it satisfies his needs (for example, whether the suit for 3D character playing a satisfactory appearance when the user performs a particular movement). Long load times can be a deterrent to the user, evaluating 3D database before deciding to purchase.

Similar issues occur in the MMOG, in particular, such as games that allow users to utilize increasingly customized characters independently. For a PC or games console output to the screen character, they must have access to a database of 3D geometry (polygons, textures, etc.), as well as lines of conduct (such as whether the character is a shield, this durable enough shield to reject a spear or not) for this character. Typically, when a user first leads MMOG, a large number of databases for the character already comes with the original copy of the game, which is available locally on optical disk or downloaded games on the disc. But as the game progresses, if the user is faced with a character or an object database that is not available locally (for example, if another user has created a character, self-tuning), then before the character or object can be displayed on the screen, their The database must be loaded. This can result in a significant delay of the game.

Given the level of sophistication and complexity of the problem for the other video game developers and publishers of video games, video game consoles associated with the prior art it is that the development of video games often takes two to three years at a cost of tens of millions of dollars. Given the fact that the new platform video game console introduced at a rate of about one every five years, game developers need to start developing these games years before the release of a new game console to the video game went on sale simultaneously with the release of the new platform. Sometimes produce multiple consoles from competing manufacturers around the same time (for example, at intervals of one or two years), but still unknown to the popularity of each console, eg which console will make the biggest sales of video game software. For example, in the recent cycle of consoles it has been planned to introduce Microsoft XBox 360, Sony Playstation 3 and Nintendo Wii about a common time period. But in the years before the mentioned administrations game developers had to essentially "do their bets" on what platform console will be more successful than others, and to give their development resources, respectively. Movie companies must also apportion its limited production resources on the basis of what they estimate to be the probability of success of the film, long before the release of this film. Given the growth in the level of investment required for the video games production games becoming more and more like film production, and companies producing games usually give their inputs on the basis of its assessment of the future success of a particular video game. But unlike the movie companies, this rate is based not only on the success of the production itself, rather it is based on the success of a game console on which the game should run. Release many games consoles at once may reduce the risk, but because of this extra work program increases cost, and frequently there is a delay of the actual release of the game.

Terms and conditions of the user application software development on the PC all require a large amount of computation, become dynamic and interactive, not only to make them more visually appealing to users, but also to make them more useful and intuitive. For example, the new operating system Windows Vista ™, and the next version of the operating system Macintosh® include visual animated effects. Improved graphical tools such as Maya ™ from Autodesk, Inc., provides the ability to reproduce moving images and the most sophisticated 3D visualizations that push the boundaries of modern CPU and GPU. However, the computational demands of these new tools cause some practical problems for users and software developers of such products.

Since the visual display of an operating system (OS) must work on a wide range of classes of computers - including computers prior generation that are no longer sold, but which, nevertheless, can be replaced by operating system (OS) to a new and - Graphic OS requirements are limited to a considerable extent "lowest common denominator" of computers for which it is intended OS, which typically includes computers that do not include a GPU. This is a very limited functionality of graphical tools OS. Moreover, portable computers c battery (e.g., laptops) limit the visual display capability since high computational activity level of the CPU or GPU, usually resulting in a higher level of power consumption and lower the battery life. Portable computers typically include software that automatically lowers processor activity to reduce power consumption when the processor is not used. In some models, the computer user can manually lower the processor activity. For example, a laptop VGN-SZ280P Sony Corporation contains a switch labeled "Stamina", with one side (for low performance, more battery life) and "Speed" on the other (for high performance level, less battery life). OS, running on a laptop computer to be usable even if the computer is working with a capacity equal to a fraction of their maximum capacity. Accordingly, the graphics performance OS is often much lower than available in the art computing power.

Applications with lots of features that require a large amount of computation, like Maya, often sold with the expectation that they will be used on high-end PC. This is usually determined by the requirement of "lowest common denominator" more expensive and less portable with a much better performance. As a consequence, these applications much more limited target audience than the universal OS (or universal desktop applications like Microsoft Office), and, as a rule, they are sold in much smaller quantities than the software universal OS, or the universal application software. The potential audience is also limited, because potential users are often difficult to test such applications that require a large amount of computations in advance. For example, assume that the students are required to learn how to use Maya, or potential buyers who are already well known, such applications require experience Maya before to invest in the purchase (which may also include the purchase of cars older models that can perform Maya). Despite the fact that a student or a prospective buyer can download the trial version of Maya or get a physical copy of the data carrier with the demo version of Maya, if they do not have a computer that can execute Maya with all its potential (for example, c handle complex 3D scene), then they can not fully evaluate the product. This significantly limits the audience for such applications with broad functionality. It also helps to increase the sales price, as the cost of development is usually paid off far fewer purchases than in the case of universal application.

Expensive applications also encourage people and companies to use "pirate" copies of the application software. As a result, a high performance application software suffers from rampant piracy, despite significant efforts publishers of such software piracy is reduced in various ways. However, even when using a "pirate" of applications with broad functionality, users can not avoid having to invest in costly modern PC to execute pirated copies. Consequently, despite the fact that users of pirated software may use the application rates equal to a fraction of its actual retail price, they, nevertheless, have to buy or buy expensive PC to fully utilize this application.

This applies to users of pirated video games with high performance. Despite the fact that the pirates can get games for a price equal to a fraction of their actual prices, they nevertheless have to buy expensive computer equipment (for example, PC with an improved GPU and a high-performance video game console, like XBox 360), necessary for the conduct of the game properly. Given that video games are typically a pastime for consumers, the additional cost for a high performance video game systems can be an obstacle. This situation is getting worse in countries (such as China), where the average annual income of workers is currently quite low relative to the average annual income of workers in the United States. As a result, a much smaller percentage of people owns a high-performance gaming video system or a high-performance PC. In such countries are quite common Internet-cafe, in which users pay to use a computer connected to the Internet. Often, these Internet-cafes have an older model or low-productivity PC without high signs, such as GPU, which would otherwise allow players to conduct video-intensive computing. It is crucial to the success of games that run on low-end PC, such as "World of Warcraft" media conglomerate Vivendi, which is very successful in China, where it is usually played in Internet cafes. On the contrary, the game requires a lot of computation, like "Second Life", much less likely to be played on the PC, established in the Chinese Internet-cafe. Such games are actually available to users who only have access to low-productivity PC in the Internet-cafe.

Barriers also exist for people who are considering the purchase of video game and would like to experience the first demo version of the game downloading it over the Internet to your home computer. The demo version is fully functional video games often version of the game with some features unavailable or limitations imposed on the number of the game. It may involve a lengthy process (possibly hours), download gigabytes of data before the game will be installed and run on a PC or console. The PC can also mean finding out what special drivers are needed (eg, OpenGL drivers or DirectX) for this game, download the appropriate version, install them, and then determine whether to conduct on the PC this game. This last step may include determining whether there is sufficient capacity at the PC for processing (CPU and GPU), sufficient RAM and a compatible OS (for example, some games are executed on Windows XP, but does not execute on Vista). Accordingly, after a long process when you try to start the demo video, the user can find that the demo version of the video may not be reproduced, taking into account the configuration of the user's PC. Worse, if after downloading new drivers the user to test the demo version of these drivers may not be compatible with other games or applications that users are constantly using on PC, respectively, install the demo version may cause previously working games or applications in the idle state. These obstacles not only disappoint the user, but they create obstacles video game software publishers and developers of video games for the marketing of their games.

Another problem that results in economic inefficiency has to do with the fact that these PC or game console is usually designed to provide a certain level of performance requirement for applications and / or games. For example, some PC have greater or lesser RAM, slower or faster CPU and slower or faster GPU, if they generally have a GPU. Some games or applications take advantage of all the computing power of a PC or console, while many games and applications do not use them. If the game or application that the user selects required performance less than the maximum capacity of the local PC and the console, the user can spend money wasted on unused signs of a PC or console. In the case of the console console manufacturer can pay more than necessary to finance the cost of the console.

Another problem that exists in the marketing and the use of video games, includes a record of what the user is viewing other games before he makes a purchase of the game. In the prior art there are several approaches to the recording of video games for replay later. For example, in US patent application number 5558339 reported recording game state information, including game controller operation during "game play" (game process) in the video game client computer (owned by identical or different user). This state information can be used later to replay some or all of the operations of the game on the video game client computer (e.g., PC or console). A major shortcoming of this approach is that in order for the user to view the recorded game, he must have a client computer video game that can play the game, and should have a video game application running on the computer so that the gameplay is identical with repeated playback status recorded games. Additionally, the video game application has to be written in such a way that there is no possible execution difference between the recorded game and playing the game repeatedly.

For example, a graphical representation of the game is usually calculated on a per-frame basis. For many games the game logic sometimes may take less or more time than one frame period, for calculating the graphical representation displayed on a screen for the next frame, depending on whether the scene is particularly complex, or if there are other delays that slow down execution ( for example, on a PC can be executed by another process, which takes CPU cycles from gaming applications). In a game like this may eventually meet the "threshold" frame, which is calculated for some less time than one frame period (say, a few CPU cycles less). Once again calculated identical scene with identical information about the status of the game, it can easily take a few CPU cycles longer than one frame period (for example, if the internal CPU bus a little out of phase with the external bus DRAM, and it introduces a number of time units CPU cycle latency, even if there is a big delay due to another process, takes milliseconds of CPU time at the handling of the game). Therefore, when the game is played again, the frame is calculated in two frame periods, but not during the one frame period. Some of conduct based on how often in the game, a new frame is calculated (for example, when the game fetches input from game controllers). When the game is played, this discrepancy in relation to the time for the different lines of behavior does not affect game play, but it can result in the fact that when you play the game again will indicate a different result. For example, if ballistics basketball calculated a steady speed of 60 frames / sec, but the sample input of the game controller is based on computing speed frames, the speed of the calculated frame could be 53 frames / s, when the game was recorded, but 52 frame / s, when the game re-play, which can lead to a mismatch between what you entered in a basketball basket or not, that will result in a different outcome. Accordingly, the use of state of the game to record the video game requires a very careful design of the game program to ensure that replay with identical information about the status of the game will bring an identical outcome.

Another approach of the prior art for recording video is a simple recording of video output PC and video games (eg, VCR, recorder to a DVD or video capture card on the PC). Video then can be rewound and played again, or alternatively, the recorded video can be discharged to the Internet, typically after being compressed. The disadvantage of this approach is that when a 3D game sequence frames repeatedly reproduced, the user is limited to viewing the sequence of frames with only a viewpoint from which the sequence of images was recorded. In other words, the user can change the viewpoint of the scene.

Further, when compressed video frames recorded game sequence played on a home PC or game console is made available to other users through the Internet, even if the video is compressed in realtime, it may be impossible to upload this compressed video in real-time to the Internet. The reason that this occurs is that many homes in the world that are connected to the Internet have highly asymmetric broadband connections (e.g., DSL and cable modem typically have much higher bandwidth of the downstream data than strip upstream data transmission). Compressed video sequences with high resolution often have larger bandwidth than upstream bandwidth data network, making them impossible to upload in realtime. Accordingly, there may be a significant delay after the game play sequence of frames (perhaps minutes or even hours) before another user on the Internet can view the game. Despite the fact that this delay is acceptable in certain situations (for example, to view, the player that occurred earlier), it makes it impossible to view the games live (for example, competition in basketball with the participation of the winners) or with the ability to "snooze" when the game played live.

Another approach of the prior art provides the viewer with a television, video games viewed live, but only under the control of the television production team. Some TV channels in the United States and in other countries, provide channels for the viewing of video games, for which the television audience may watch the video of specific users (for example, the player with the highest rating, participating in competitions). This is done in the presence of video output video game systems (PC and / or consoles) fed into the equipment for processing and distributing the video for the television channel. It differs little from the TV channel broadcast on the basketball game live, in which several cameras provide a transmission line in the air at different angles around the basketball court. After that, the TV channel can use his equipment for processing special effects and video / audio to manipulate the output of various video game systems. For example, a TV channel may impose the text over the video from the video game, which indicates the status of the different players (the same way as you can superimpose text during basketball games live), and a television channel may impose additional audio from the commentator, who may discuss actions occurring during games. In addition, the output of a video game can be combined with cameras, recording actual video game players (such as showing their emotional reactions to the game).

One problem with this approach is that such a transmission line live video should be available for the TV channel hardware for processing and distribution of real-time video to the excitement he had a direct transfer. However, as discussed earlier, this is often impossible to carry out, when playing the video system is controlled from the home, especially if part of the broadcast includes live video from a camera that captures real video player. In addition, during the event there is a concern that the player who is at home can modify the game and cheat, as described above. For these reasons, such a broadcast video game television channels are often so arranged that the players and game video concentrated in public places (e.g., a television studio or in the arena) where the television production equipment can accept line video transmission from a plurality of video game systems, and possibly cameras for direct transfer.

Despite the fact that such Videogame TV channels of the prior art can provide a very exciting show for the television audience, which is a hands-on experience, close to the actual sporting event, such as videoigrokami presented as "athletes," and in terms of their action in the world of video games, and in terms of their actions in the real world, these video game is often limited positions where players are in close physical proximity to each other. And as TV channels broadcast, each broadcast channel can display only one video stream, which is selected by the production team of TV channel. Because of these limitations and the high cost of broadcast time, production equipment and production groups such television channels typically only show players rated involved in important competitions.

In addition, the television channel broadcasting full-screen video image of the entire TV audience, shows only one video game at a time. This is a very limited possibility to choose the viewer. For example, the viewer may not be interested in the game (s) show (bubbled) at this time. Another viewer may be interested only in game viewing a particular player, which is not shown on TV channel at a given time. In other cases, the viewer may be interested only in watching how skilled the player controls a specific level in the game. However, other viewers may want to control the viewpoint, which look a video game that is different from the view point, the selected industrial group, etc. In other words, the viewer may have multiple preferences when watching video games that are not provided by a particular broadcast television network, even if there are several different TV channels. For all the above reasons video game television channels of the prior art have significant limitations in presenting video games to viewers.

Another drawback of the prior video game equipment and application software systems is that they are complex, and commonly suffer from errors, crashes and / or unintended and undesired behaviors lines (collectively "error"). Despite the fact that games and applications tend to go through the process of configuring and debugging (often referred to as "quality assurance software" or SQA) before the release, almost invariably, after the release of the game or application to a wider audience, in operation suddenly errors . Unfortunately, the software developer it is difficult to identify and find many of the bugs after release. Software developers can be difficult to find errors. Even when they find an error, they may be available only limited information to identify the causes of this error. For example, the user can call the customer service game developer and leave a message that the management of the game screen began to flicker, then become dark blue and PC hung. This gives the group SQA very little useful information for error detection. Some games or applications that establish a relationship online can sometimes give more information in some cases. For example, it can sometimes be used "watchdog" process for the implementation of the current control games or applications for "crash." Watchdog process may collect statistics about the status of applications or games (for example, on the status of stack memory usage, how far advanced the game or application, etc.), when there is a crash, and then pass that information to the group through the SQA Internet. But in a difficult game or application deciphering such information may take a very long time to accurately determine what the user was doing at the time of crash. Even then, it may be impossible to determine the sequence of events that led to the crash.

Another problem with the PC and game consoles, is that they are subject to maintenance, which causes great inconvenience to the consumer. Questions service also affect the manufacturer's PC and game console because they are usually required to send a special box for the safe transport of broken a PC or console, and then bear the costs of repair, if the PC or console under warranty. On the publisher's software application or game can also be affected by the loss of sales (or use the online service) due to the fact that the PC and / or console in a state of repair.

1 shows a game video system of the prior art, such as Sony Playstation® 3, Microsoft Xbox 360®, Nintendo Wii ™, PC-based Windows or Macintosh corporation Apple. Each of these systems includes a central processing unit (CPU) for execution of the control program, as a rule, a graphics processor (GPU) for performing advanced graphical operations, and various types of input / output (I / O) for exchanging data with external devices and users. For simplicity, these components are shown grouped together as a single block 100. Also shown that play video system of the prior art of Figure 1 includes 104 storage on optical media (for example, high-capacity storage discs), 103 storage on your hard drive data storage and control program video games, network connection 105 for conducting games with several participants, to download games, patches, demos, or other media, random access memory (RAM) 101 for storing the control program, is currently being executed CPU / GPU 100, a game controller 106 for receiving input commands from the user during gameplay and display 102 (e.g., SDTV / HDTV or a computer monitor).

The prior art system shown in Figure 1 has several limitations. Firstly, 104 drives the optical disk drives 103 and hard drives tend to have much slower access speeds as compared with the access speed RAM 101. When working directly through RAM 101 CPU / GPU 100 can, in practice, process far more polygons per second than is possible when the control program and the data are read directly from the drive 103 or a hard disk drive 104 of the optical disk due to the fact that RAM 101 generally has much higher bandwidth and does not suffer from delays of disc mechanisms relatively long search. But in these prior art systems provided only limited amount of RAM (e.g., 256-512 MB). Therefore, often requires a sequence of frames, "Loading ...", during which the RAM 101 is filled with data periodically for the next scene video.

Some systems attempt to combine the loading control program along with gameplay, but it can be done only when we know the sequence of events (for example, if the vehicle is traveling on the road, it can be loaded geometry for upcoming buildings located on the edge of the road, while car rides). For complex and / or rapid scene changes, this type of overlapping usually does not work. For example, in the case where the user is in the middle of the battle and the RAM 101 is full of data of objects in the field of view at this point, if the user to quickly move the image to the left to view objects that are not currently loaded into the RAM 101, the there will result a discontinuity in action, as it will not have enough time to download new items from the storage device 103 on hard drives or optical media data 104 into RAM 101.

Another problem with the system of Figure 1 arises due to limitations of memory storage capacity of hard drives 103 and optical media 104. Although disk storage devices can be manufactured with a relatively large storage capacity (e.g., 50 gigabytes or more), they nevertheless do not provide sufficient storage capacity for certain scenarios encountered in current video games. For example, as previously mentioned, a soccer video game to the user may be provided the opportunity to select from dozens of teams, players and stadiums throughout the world. For each team, each player and each stadium needs a large number of cards displaying textures and environment maps to describe the 3D surfaces in the world (for example, each team has a unique T-shirt, and each requires a unique texture map display).

One technique used to address this latter problem is the fact that in the game to pre-compute texture mapping maps and environment maps once they are selected by the user. This can include multiple processes that require large amount of computation, including recovery of compressed images, 3D mapping, shading, organizing data structures, etc. As a result, the user may be a delay while the video game executed in these calculations. One way to reduce this delay, in principle, is to perform all of these calculations - including every change of command of the team and the stadium - when the game was originally developed. Release version of the game can then include all of these pre-processed data stored on optical media 104, or to a one or more servers on the Internet with just switched preprocessed data to the command data, the first team, the choice of the stadium, downloaded via the Internet to a drive 103 The hard disk when the user makes a selection. As a practical matter, however, are pre-loaded data for each change possible in the conduct of the game can easily be terabytes of data, far exceeding the capacity of today's optical storage devices. Furthermore, the data for the command data, command structure, stadium selection could easily be hundreds of megabytes of data or more. With respect to the home network connection of, say, 10 Mb / s, loading the data via a network connection 105 may take a longer time than calculating said data locally.

Accordingly, the architecture of the game of the prior art shown in Figure 1, the user undergoing significant delays between major scene transitions of complex games.

Another problem associated with such prior art approaches, as the one depicted in Figure 1 is that over time, video games are becoming more advanced and require more processing power CPU / GPU. Accordingly, even under the assumption of an unlimited amount of RAM hardware requirements exceed the maximum level of video processing power available in these systems. As a result, users must upgrade gaming equipment every few years to keep up (or run newer games at lower quality levels). One consequence of the trend of further improvements in video games is that video game home computers, are typically economically inefficient because their cost is usually determined in accordance with the requirements of the highest performance game they can support. For example, XBox 360 can be used to play a game similar to "Gears of War", which requires a high-performance CPU, GPU and hundreds of megabytes of RAM, or XBox 360 can be used for playing Pac Man, games of 1970, which requires only kilobytes of RAM and CPU performance is very low. In fact, XBox 360 has enough processing power to accommodate the many parallel games Pac Man at the same time.

Video game computers typically off for more time per week. According to research by Nielsen Entertainment in July 2006 of active players aged 13 years and older, on average, active players spend fourteen hours a week playing video games console, or only 12% of the time in a week. This means that the average video game console is used 88% of the time, which is an inefficient use of an expensive resource. This is especially important given the fact that the video game console are often financed by the manufacturer to reduce the purchase price (with the expectation that the subsidy will be returned as a result of royalties from future purchases of software video games).

Video game consoles are also costs associated with almost any consumer electronic device. For example, the electronics and mechanisms of systems to be mounted in the housing. The manufacturer shall provide a guarantee on service. The retailer, which sells the system should profit from any sale of the system and / or from the sale of video game software. All of these factors add to the cost of the video game console, which must be funded or the manufacturer, with the transmission to the consumer, or both.

Furthermore, the main problem of the video game industry is piracy. The security mechanisms used by virtually every main game video system, "hack" with time, which results in the unauthorized copying of video games. For example, the security system the Xbox 360 has been hacked in July 2006, and users can now download illegal copies online. The games that can be downloaded (for example, games for the PC or Mac), are particularly vulnerable to piracy. In certain regions of the world where piracy is poorly controlled, in essence, there is no viable market for stand-alone video game software because users can buy pirated as easily as legitimate copies at a price equal to a very small fraction of the price. Moreover, in many parts of the world the price of the game console is such a high percentage of income that even if piracy is controlled, few people could afford a modern gaming system.

In addition, the market for used games reduces the income of the video game industry. When a user loses interest in the game, he could sell the game retailer that sells this game to other users. It is unauthorized, but common practice significantly reduce the revenues of publishers of games. Similarly, the decline in sales usually occurs approximately 50% when switching to another platform every few years. It is because people stopped buying games for older platforms, when they learn that the soon to be released new version of the platform (for example, when going to release the Playstation 3 users stop buying games for the Playstation 2). Taken together, the decrease in sales and an increase in development costs related to new platforms, can have a very significant adverse impact on the profitability of game developers.

The new game consoles are also very expensive. The Xbox 360, Nintendo Wii and Sony Playstation 3 - all sold at retail for hundreds of dollars. Powerful PC gaming systems can cost up to $ 8,000. It provides users with a significant investment of money, especially given the fact that the hardware becomes obsolete after a few years, and the fact that many systems are buying for children.

One approach to solving the above problems is an online game in which the program that controls the game, and the data available on the server and client machines are delivered on-demand as a compressed video and audio that are streamed over a broadband digital network. Some companies, such as G-Cluster in Finland (now a subsidiary of Japanese corporation SOFTBANK Broadmedia), are currently providing these services online. Similar games service became available in local area networks, such as LANs hotels and offers DSL providers and cable television. The main drawback of these systems is the problem of latency, i.e. the time it takes for the signal to pass to and from the game server, which is typically located in the "Central Station" of the operator. Dynamic video-fighters (also known as video "twitch" ("twitch")) require very low latency between the time when the user operates the game controller, and the time when the display screen is updated and displays the result of the operation of the user. Low latency is required for the user to have a feeling that the game responds "immediately." Users may be satisfied with different latency intervals depending on the type of game and the skill level of the user. For example, the wait time of 100 ms may be acceptable for slow haphazard game (such bakgammon) slow or role-playing game, but in a dynamic-action film waiting time exceeding 70 or 80 ms, may degrade the user game and thus is unacceptable. For example, in a game that requires fast reaction time, accuracy deteriorates with increasing the waiting time of 50 to 100 ms.

When a game or application server installed in a nearby managed network environment, or where the network path to the user is predictable and / or it can withstand peak bandwidth is much easier to manage wait times, and in terms of the maximum waiting time, and in terms of constancy latency (e.g., so the user observes steady motion from digital video streamed over a network). This level of control can be achieved between the distributor cable television network to the home subscriber cable television, or DSL central office to DSL subscriber's home, or in a commercial private branch network (LAN) from a server or a user. You can also get a specially separated private point-to-point between the two companies, which have a guaranteed bandwidth and latency. But in a game or application system that hosts games in a server center connected to the general Internet, and then transmits a stream of compressed video to the user through a broadband connection, many factors affect the waiting time, which results in serious limitations in the use of the prior art .

In a typical home, connected to a broadband connection, the user can have a DSL or cable modem for broadband services. For such broadband transmission time back and forth between home and the total Internet user is typically 25 ms (and sometimes more). In addition, there are quantities timeout signal back and forth, which are a consequence of the routing data through the Internet to a server center. Timeout transmission over Internet varies according to the route that the data is given and the delays that occur due to routing it. In addition to routing delays, transmission time to and fro is also due to the velocity of the light passing through the light conductor, which connects most of the Internet. For example, for each 1000 miles (1600 km), the transmission time to and fro is approximately equal to 22 ms due to the speed of light through a light guide and other losses.

Additional latency can occur due to the data rate of transmission streams via the Internet. For example, if the user is provided with service DSL, which are sold as "DSL service six megabits per second," in practice the user will probably get less than 5Mbps bandwidth downlink data stream at best, and will likely be observed that the compound occasionally deteriorates due to various factors such as congestion during peak periods a digital subscriber line access multiplexer (DSLAM). The same question may be a decrease in the rate of data transfer cable modem used to connect sold as "service cable modem 6 megabits per second," to speed much less than that, if there is an overload in the local general coaxial cable, forming link through the neighborhood, or elsewhere in the cable modem system network. If data packets at a steady rate of 4 Mbit / s will be streamed in the same direction in the format of a User Datagram Protocol (UDP) from a server center through such connections, if everything is working correctly, then the data packets are transmitted without additional latency, but case of congestion (or other impediments) and if the streaming data is available to the user only 3.5 Mbit / s, in the typical situation or packets are discarded, which will ultimately lead to loss of data, or packets will queue up at the point of congestion, until they will not be sent, therefore, the introduction of additional waiting time. At various points there are different overload capacity of the queue for storing delay the package, respectively, in some cases, packages that can not pass through the point of overload, reset immediately. In other cases, several megabits queue cost data and eventually sent. However, in almost all cases, queues at points of congestion limited in capacity, and after exceeding these limits queue overflow and packets are discarded. Accordingly, to eliminate the additional waiting time (or, even worse, the loss of packets) must be the elimination of excess capacity data transfer speed of the game or application server to the user.

The waiting time is also a consequence of the time required to compress video in the server and restore compressed video in the client device. The waiting time also occurs when a video game running on the server calculates the next frame to be displayed. Currently available video compression algorithms available or suffer from high rates of data transmission or because of the large latency. For example, the compression algorithm of moving MJPEG images is only intra-lossy compression algorithm, which is characterized by low latency. Each frame of video is compressed independently of each other frame of video. When a client device receives the video frame-compressed moving picture compression MJPEG, it can immediately restore the compressed frame and display it with the resulting as a result of very low latency. But due to the fact that each frame is compressed separately, said algorithm can not use the similarities between successive frames, and as a result intraframe-only video compression algorithms suffer from very high data rates. For example, video 640 × 480, 60 frames / sec (fps), compressed by compression algorithm MJPEG moving picture, may require 40 Mb / s (megabits per second) or more (the transmission rate) data. Such high data rates for video windows with low resolution demanding excessively costly in many broadband applications (and of course for most consumer applications based on Internet-technologies). Furthermore, due to the fact that each frame is compressed independently, artifacts in the frames that may be caused by lossy compression are likely to appear in different places in successive frames. This may result in what appears to the viewer as a moving visual artifacts when the restored compressed video.

Other compression algorithms, such as MPEG2, H.264 or VC9 from corporation "Microsoft", as used in prior art configurations, can achieve high compression ratios, but at the expense of a large latency. These algorithms use interframe as well as intraframe compression. Periodically, these algorithms perform only intra-frame compression. This frame is known as a key frame (commonly referred to as I-frame). Then, these algorithms typically compare the I-frame with the previous frame and the subsequent frames. Rather than compress the preceding frame and the subsequent frames independently, said algorithm determines what has changed in the image of the preceding and subsequent frames relative to I-frame, and then stores those changes as the so-called B-frames for the changes preceding the I-frame and P-frames for change, following the I-frame. This results in much lower data rates than intraframe-only compression. But this is usually achieved at the expense of greater latency. I-frame is typically much larger than a B-frame or P-frame (often 10 times larger), and as a result it takes proportionately longer transmission time for a given data rate.

For example, consider a situation where the I-frames in a 10-fold more B-frames and P-frames, and there are 29 B-frames + 30 P-frames = 59 intermediate frames for each I-i-Frame, or a total of 60 frames for each " GOP "(GOP). Accordingly, at 60 frames / sec per second there is a 1 to 60-HR GOP. Assume that the maximum data transfer rate on the transmission channel is equal to 2 Mbit / s. For video of higher quality in this channel compression algorithm may output a data stream of 2 Mbit / s, and in view of the above factors as a result obtained 2 megabits (Mbits) / (59 + 10) = 30 394 bits each-Frame and 303 are 935 bits Each I-frame. When the compressed data recovery algorithm takes the compressed video stream for sustainable video playback is required to restore a compressed frame and display them on the screen at regular intervals (for example, 60 frames / sec). To obtain this result, if any frame transmission waiting time exists, then all frames must be delayed at least for the duration of the waiting time, respectively, the largest waiting time for waiting for the frame for each video frame. I-frames are administered largest transmission waiting time values, since they are largest, and the entire I-frame must be received before it can be restored compressed I-picture and displayed on the screen (or any intermediate block, depending on I- frame). With considering that the data rate on the channel is equal to 2 Mbit / s, the transmission of I-frames takes 303 MB 935/2 = 145 ms.

The interframe video compression system as described above using a large percentage of the bandwidth of the transmission channel, there are large quantities waiting time due to the large size of an I-frame relative to the average frame size. Or, in other words, despite the fact that the algorithms interframe compression prior art achieved a lower average-frame data rate than when only-intraframe compression algorithms (e.g. 2 Mbit / s compared to 40 Mbit / s) are, in order however, suffer from a high peak-frame data rate (e.g., 303 × 60 935 = 18.2 Mbit / s) because of the large I-frames. However, it should be appreciated that the above analysis assumes that all P-frames and B-frames is much less than I-frames. While this is generally true, it is not true for frames with high image complexity uncorrelated with the prior frame, with a large displacement or a scene change occurs. In such cases, P-frames or B-frames can become as large as I-frames (if a P-frame or B-frame becomes larger than I-frame, a sophisticated compression algorithm usually "provides" I-frame, and replaces P-frame or a B-picture is an I-frame). Accordingly, in the digital video stream at any time there may be a data rate peaks having a size of I-frame. Accordingly, in the case of compressed video, when the average video data rate approaches the throughput data rate transmission channels (as is often the case, given the requirements of high data rates for video) the high peak data rates due I- frames or large P-frames or B-frames result in a longer time-out frame.

Undoubtedly, in the foregoing discussion describes only the waiting time compression algorithm, creating more B-frames, P-frames or I-frames in the GOP. If B-frames are used, the waiting time will be even greater. The reason that this occurs is that before the B-frame can be displayed, all of B-frames following the B-frame and I-frame must be received. Accordingly, in the sequence group of pictures (GOP), e.g., BBBBBIPPPPPBBBBBIPPPPP, where there are 5 B-frames before each I-frame, the first B-frame can not be displayed on the display device restoration of the compressed video as long as not to be taken subsequent B-frames and I-frame. Accordingly, if the video stream is transmitted at 60 frames / sec (i.e., 16.67 ms / frame), then before can be restored compressed first B-frame need to 16.67 × 6 = 100ms to receive Five B-frames and I-frames, independently of the speed of the channel bandwidth and is only 5 B-frames. Compressed video sequence with 30 B-frames are fairly common. And with a small channel bandwidth is 2 Mbit / s, the impact of latency caused by the size of the I-frame is a great addition to the effect of the waiting time for waiting for the B-frames. Accordingly, the channel of 2 Mbit / s with a large number of B-frames is quite simple to exceed 500ms latency or more using a video compression technology of the prior art. If B-frames are not used (due to lower compression ratio for given quality level), the B-frame is not the waiting time, but continue waiting time cause peak frame sizes, described above.

The problem is compounded by the very nature of many video games. Video compression algorithms utilizing GOP structure, as described above, largely optimized for use with live video or motion picture material intended for passive viewing. Typically, the camera (or a real camera and the virtual camera in the event of a computer-generated animation) and the scene is relatively stable, simply because if the camera or the scene moves too abruptly, the material of the film or video (a) are generally not nice to look at, and (b) if it looks, it is usually the viewer is not closely following the action of the camera suddenly turns sharply (for example, when the camera hits when shooting, when a child blows out the candles on a birthday cake, and suddenly turns sharply away from the cake and returned, the viewers tend to focus on the child and the cake and not pay attention to a brief interruption when the camera is suddenly moved). In the case of a video interview, or video teleconferencing camera may be held in a fixed position and not move at all, resulting in a very very small amount of data peaks. But 3D video with high activity differs by a constant movement (for example, consider the 3D race where the entire frame is in fast motion during a race, or a look at the game, first-person shooters, where the virtual camera is constantly moving jerkily). These video games can result in a sequence of frames with large and frequent peaks, where the user may need to see clearly what is going on during these sudden movements. As such, compression artifacts are far less acceptable in a 3D video games with high activity. Accordingly, the video output of many video games, because of their nature, outputs compressed video stream with very high and frequent peaks.

Given the fact that users of dynamic video games, fighters are not tolerant of high latency, and taking into account all of the above reasons for the waiting time up to now there were restrictions placed on the server of video games that aired the video stream on the Internet. Further, users of applications that require a high degree of interactivity suffer from similar limitations if the applications are hosted in a general Internet and transmitting the video stream. Such services require a network configuration in which the hosting servers are installed directly in the switchboard (in the case of cable broadband) or the central office (in the case of Digital Subscriber Lines (DSL)), or within a LAN (or on a specially separated private connection) in a commercial setting, so that the route and distance from the client device to the server is controlled to minimize latency and peaks can be adapted so that they will not cause the latency. Network LAN (typically having a velocity in the range of 100 Mbit / s, 1 Gbit / s) and leased lines with corresponding bandwidth typically can support requirements for peak bandwidth (e.g., peak bandwidth of 18 Mbits / s equal to insignificant share of LAN bandwidth of 100 Mbit / s).

Requirements for peak bandwidth may also be provided with a broadband network infrastructure related to residential buildings, if the special adaptation. For example, in a cable television digital video traffic can be given dedicated bandwidth which can handle peaks, such as large I-frames. And the system can be provided DSL DSL modem at a higher speed, taking into account the large peaks, or may be provided specially divided compound which can process data at higher transmission rates. But conventional cable modem infrastructure and DSL, connected to the general Internet, have a much lower tolerance for the requirements of peak bandwidth for compressed video. Accordingly, the online services that host video or application server center located at a distance from the client devices, and then transmit a stream of compressed video output on the Internet through a common broadband connections related to residential buildings, suffer from significant waiting times and restrictions on peak bandwidth transmission - particularly in relation to games and applications that require very low latency (eg, games, first-person shooters and other multi-user, interactive games, Adventure or applications that require low response times).

SUMMARY OF THE INVENTION

The present invention addresses the above many problems and disadvantages of the prior art. The technical result achieved when exercising this invention is in particular to provide low latency for remote operation and / or use by users games and applications that run on servers hosting service by transmitting streaming compressed interactive video client user devices via a network such as the Internet, even with limited bandwidth. For this hosting service is provided comprising a plurality of geographically distributed server farms hosting services, each of which includes servers that the performance of the game or application, form a stream of compressed low-latency interactive streaming video and / or audio. Also provided is means for multicasting said flow into a plurality of client devices adapting to different video, audio and / or network characteristics of individual client devices, and means for enabling user input from one or more client devices in a hosting service for managing interactive streaming video.

BRIEF DESCRIPTION OF DRAWINGS

The present disclosure of the subject invention will become fully apparent from the following detailed description and the accompanying drawings, which however, should not be taken as limiting the scope of the invention disclosed specific embodiments illustrated, but should be understood as intended only for explanation and understanding.

1 shows the architecture of video game of the prior art.

2a-2b shows a high level system architecture according to one embodiment.

3 illustrates actual, estimated, and the required data rate for transmission of information between client and server.

4a illustrates a hosting service and a client employed according to one embodiment.

4b shows the illustrative value of the waiting time related to the transfer of information between the client and the hosting service.

4C illustrates a client device according to one embodiment.

4d illustrates a client device according to another embodiment.

In 4E illustrates an exemplary block diagram of a client device according 4c.

On 4f illustrates an exemplary block diagram of the client device for 4d.

5 shows an exemplary form of video compression which may be employed according to one embodiment.

6a illustrates an exemplary form of video compression which may be employed in another embodiment.

6b shows peak data rate associated with transmission of video sequences with low activity, low complexity.

6C shows the peaks of the data rate associated with transmission of video sequences with a high activity, a high level of complexity.

7a-7b shows illustrative video compression techniques employed in one embodiment.

Figure 8 depicts additional exemplary methods of video compression used in one embodiment.

9A-9C depict exemplary methods, used in one embodiment to reduce the peak data rate.

10A-10b illustrates one embodiment which efficiently packs fragments within packets.

11A-11D depict embodiments that employ forward error correction techniques.

Figure 12 illustrates one embodiment which uses multi-core processors for compression.

13A-13B shows the geographic location and connection between the hosting service according to various embodiments.

Figure 14 shows the illustrative value of the waiting time related to the transfer of information between the client and the hosting service.

Figure 15 illustrates an exemplary architecture of the server hosting the service center.

Figure 16 shows an exemplary screen shot of one embodiment of a user interface that includes a plurality of video windows in real time.

Figure 17 shows a user interface of 16 after selecting a specific video window.

Figure 18 shows a user interface of 17 after plowing specifically mentioned the video window to full screen.

Figure 19 illustrates an exemplary user video joint, combined on-screen games with several participants.

Figure 20 illustrates an exemplary user page for the player in the hosting service.

Figure 21 illustrates an exemplary 3D interactive advertising.

Figure 22 depicts an exemplary sequence of steps for fotorealnogo output image having a textured surface, based on the capture surface of a live performance.

Figure 23 depicts an exemplary user interface page that allows selection of linear media content.

Figure 24 - graph which illustrates the time that will pass before the web-page becomes active as compared with the speed of the connection.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

In the following description specific details are set forth, such as device types, system configurations, ways of exchanging information, etc. to provide a thorough understanding of the subject disclosure. However, those skilled in the art to which this invention pertains will appreciate that these specific details are not required for use of the described embodiments.

2a-2b, provided high-level architecture of the two embodiments in which video games and applications are hosted 210 hosting service and they get access client devices 205, 211 in the territory of the user (note that the "user area" means any place where the user is located, including outdoors if using a mobile device) over the Internet 206 (or other public or private network) according to the subscriber service. Client devices 205 may be universal computers, such as PC Windows Microsoft, and PC Linux, or Macintosh computers corporation Apple, Inc. wired or wireless connection to the Internet, with internal or external display device 222, or they may be dedicated client devices such as set-top box (with a wired or wireless connection to the Internet), which outputs video and audio to a monitor or TV set 222, or they It can be mobile devices, presumably with a wireless connection to the Internet.

Any of these devices may have their own user input devices (e.g., keyboards, buttons, touch screens, light pen, or trackpad, video capture cameras and / or camera motion tracking, etc.), or they may use external devices 221 input (e.g., keyboard, mouse, game controllers, light pen, video capture cameras and / or camera motion tracking, etc.) connected via wires or wirelessly. As described in more detail below, the hosting service 210 includes servers of various levels of performance, including servers with powerful processing CPU / GPU. At the time of the game or the use of applications in service 210 hosting a home or office client device 205 receives input of the controller and / or keyboard from the user, and then it sends this input controller over the Internet 206 service 210 hosting, which is in response executes the program , control the game and generates successive frames of video output (video sequence) for the game or application software (for example, if a user clicks on a button that can move the character on the screen to the right, then the game program creates a sequence of video images, which depict a character, move to the right ). This sequence is then compressed video using the video compression device with low latency, and then hosting service 210 sends a video stream with low latency through the Internet 206. home or office client device then decodes the compressed video stream and renders the recovered video images on a monitor or TV . Consequently, the requirements for computer graphic technique and client device 205 are significantly reduced. It is required that the client 205 has the processing power only to send the input keyboard / controller to the Internet 206 and decoding and restoring the compressed video stream received from the Internet 206, which is currently virtually any personal computer can execute the software on its central processing unit (eg , dual-core processor Corporation Intel (Intel Corporation Core Duo CPU), operating with a frequency of about 2 GHz, can recover compressed HDTV 1280 × 720, encoded using compression devices such as H.264 and Windows Media VC9). And, in the case of any client devices, specialized integrated circuits can perform video decompression for such standards in real-time with a much lower cost and with far less power consumption than a universal CPU, for example, which may be required for a modern PC. Namely, to perform the function of delivery of input data controller and restore compressed video, home client devices 205 do not require any specialized graphics processing units (GPU), optical drive or hard drives, such as a game video system of the prior art shown in FIG .1.

As the games and application software becomes more complex and more photorealistic, they require a GPU, CPU better performance, more RAM and a faster disk drives with higher capacity and processing power service 210 hosting can be continuously upgraded, but the final the user does not need to update your home or office client platform 205, since the requirements for its processing will remain constant for the display resolution and frame rate for a given algorithm restore compressed video. Accordingly, the hardware limitations and compatibility issues, currently available, do not exist in the system illustrated in Figures 2a-2b.

Furthermore, because the game and application software executed only in servers in the service 210 host, there is no copy of the game or application software (either in the form of optical media, or as downloaded software) in the office ("office", as used In this specification, unless stated otherwise, it includes any non-residential environment without permanent accommodation including classrooms, for example) or in user's home. This greatly reduces the likelihood that a game or application software to be illegally copied (illegal use) and also reduces the likelihood that valuable database, which can be (used) a game or application software is illegally used. In fact, if the gaming requires specialized servers (for example, demanding a very expensive, big or noisy equipment) or software, which in practice can not be used at home or in the office, even if it is obtained a pirated copy of a game or an application software, it can not be used at home or in the office.

In one embodiment, the service 210 hosting provides the development tools software developers 220 software application or game (which usually refers to the companies of software development, gaming or television studios or publishers of application software or games) that project video games, in order to enable them to design a game that you can perform in the service 210 hosting. Such tools allow developers to use signs hosting service that would not normally available in standalone PC or game console (e.g., fast access to very large databases of complex geometry ("geometry", unless otherwise stated, is used herein to refer to polygons, texture, equipment, lighting, line of conduct, and other components and parameters that define the database 3D)).

Different business models are possible for this architecture. According to one model, the hosting service 210 receives a monthly fee from the end user pays a royalty developers 220, as shown in FIG. 2a. In an alternative implementation, shown in Figure 2b, developers get 220 monthly fee directly to the user and the service paid 210 for hosting content hosting applications or games. These underlying principles are not limited to any particular business model for hosting applications or online gaming.

Characteristics of compressed video

As discussed earlier, a significant problem associated with the provision of services for video games or services in applied software online is the issue of waiting time. The waiting time of 70-80 ms (the time when the input device is actuated by the user to the point where a response is displayed on the display screen) is at the upper limit for games and applications requiring a small response time. However, to achieve such delay is very difficult in the context of the architecture shown in Figures 2a and 2b due to a number of practical and physical constraints.

As indicated in Figure 3, when the user subscribes to the Internet-service connection is usually estimated nominal maximum data rate 301 to the user's home or office. Depending on the policy providers and equipment characteristics for routing may be more or less of a strict application of the maximum data rate, but typically the actual available data rate is lower for one or more different reasons. For example, in the central office DSL or cable line in the local dial-up connection may be too much network traffic, or a cable network may contain noise caused by dumping packets, or the provider can set the maximum number of bits per month for each user. Currently, the maximum transmission speed of the downstream data services for cable and DSL connection, usually varies from a few hundred kilobits per second (kbit / s) and 30 Mbit / s. Cellular services are typically limited to hundreds of Kbit / s downstream data. However, the speed of broadband services and the number of users who subscribe to broadband, increase significantly with time. Currently, some analysts estimate that 33% of US broadband subscribers have a downstream transmission speed of 2Mbps / s or more. For example, some analysts predict that by 2010 over 85% of US broadband subscribers will have a data rate of 2 Mbit / s or more.

As indicated in Figure 3, the actual available max data rate 302 may fluctuate over time. Accordingly, in the context of software applications or online gaming with low latency it is sometimes difficult to predict the actual available data rate for a particular video. If the rate of 303 data required to support a given level of quality at given number of frames per second (frame / s) with a given resolution (e.g., 640 × 480, 60 frames / sec) for a certain amount of scene complexity and motion rises above the actual available maximum data rate 302 (as indicated by the peak in Figure 3), then several problems may occur. For example, some Internet-services will simply be dropped packets, which will result in loss of data or distorted / lost images on the video screen of the user. In other services, additional packages will be temporarily placed in a buffer (ie, queued) and these packages will provide customers with the available data rate that will result in an increase in waiting time - an unacceptable result for many video games and applications. Finally, some providers of Internet-services will consider increasing the data rate as a malicious attack, such as an attack denial of service (known technique used by hackers to block network connections), and disrupt the user's connection to the Internet for a certain period of time. Accordingly, the embodiments described herein, steps are taken to ensure that the data rate required for a video game does not exceed the maximum available data rate.

Architecture hosting service

4a shows the architecture of the hosting service 210 according to one embodiment. Hour 210 host or may be located in a single server center, or can be distributed across a plurality of server centers (to provide connection to a user with less waiting time, which are routes with less waiting time to a certain server centers than others, to provide alignment load among users and provide redundancy in case of failure of one or more server farms). 210 hosting service could eventually include hundreds of thousands or even millions of servers 402 serving a very large number of users. 401 System Management Services Hosting provides centralized management for service hosting and manages 210 routers, servers, video compression, accounting and billing, etc. In one embodiment, the control system 401 hosting service is implemented in a distributed data processing system based on the Linux operating system, connected to a RAID disk array type, used to store the databases for user information, server information, and system statistics. The foregoing description of various operations, implemented 210 hosting service if they are not assigned to other special systems are initiated and controlled by the control system 401 hosting service.

Hosting service 210 includes a number of servers 402 such as those servers that are currently commercially available from Intel, IBM, Hewlett Packard, and others. Alternatively, the servers 402 can be assembled in a custom configuration of components, or, finally, they can be integrated so that an entire server is implemented as a single chip integrated circuit. While this scheme is shown for example a small number of servers 402, in actual use can be only one server 402, or millions of servers 402 or more. All of the servers 402 may be configured in the same way (as an example of some of the configuration parameters, with the same capacity and the type of CPU, with or without a GPU, and if with a GPU, then with the same capacity and the type of GPU, with the same number of the CPU and the GPU, with the same the amount and type / speed of RAM, and with the same configuration RAM), or various subsets of the servers 402 may have the same configuration (e.g., 25% of the servers can be performed in a certain way, 50% a different way, and 25% yet another way), or every server 402 It may be different.

In one embodiment, the servers 402 without the disk, i.e. rather than having its own local mass storage device (regardless of whether it is optical or magnetic storage device or memory device for semiconductors, such as a flash memory or other mass memory, performs a similar function), each server accesses to the total mass memory via a high-speed backbone network connection or accelerated. In one embodiment, this compound is a high-speed storage network 403 (SAN), connected to a set of matrices 405 independent disks redundancy (RAID) with connections between devices implemented using Gigabit Ethernet technology. As is known to those skilled in the art, SAN 403 may be used to combine multiple storage arrays 405 RAID type together, resulting in extremely high bandwidth - approaching or possibly exceeding the bandwidth available from the RAM, used in current gaming consoles and PC. And, while a RAID disk arrays based on rotating media, such as magnetic media, frequently have significant latency to access during the search, a RAID disk arrays based on semiconductor storage can be implemented with much lower access latency. In another configuration, some or all of the servers 402 provide some or all of their own mass storage device locally. For example, the server 402 can store information frequently accessed, such as its operating system, and a copy of a video game or application to a local storage based on flash memory with low latency, but it can use the SAN for access to the disk arrays 405 type RAID, based on rotating media with a long search to access large databases of geometry or game state information less frequently.

Furthermore, in one embodiment, the hosting service 210 uses the video compression logic 404 with low latency as described in detail below. Video compression logic 404 may be implemented in software, hardware, or any combination thereof (certain embodiments of which are described below). Video compression logic 404 includes logic for compressing audio-visual material.

When the operation while reference video game or using an application at the site 211 of the user through the keyboard, mouse, game controller or other device 421 input logic 413 of the control signal on the client 415 transmits control signals 406a-406b (generally in the form of packets UDP), representing pressing (and other types of user inputs) actuated by the user to the hosting service 210. The control signals from a given user are routed to the appropriate server 402 (or servers, if user input device responsive plurality of servers). As shown in Figure 4a, control signals 406a may be routed to the servers 402 via the SAN. Alternatively or in addition, control signals 406b may be routed directly to the servers 402 hosting service network (e.g., Ethernet local area network). Regardless of how they are transmitted, the server or servers execute the game or application software in response to the control signals 406a-406b. Although not shown 4a various networking components such as a firewall (s) and / or gateway (s) may process incoming and outgoing traffic at the border of the hosting service 210 (e.g., between the hosting service 210 and the Internet 410) and / or at the border of the territory of 211 people between the Internet 410 and the home office or the client 415. The graphical output and audio output executable game or application software - that is, new sequences of video images - are provided to logic 404 is the video compression with low latency, which compresses the sequences of video images according to the methods of video compression, low-latency, for example, according to methods described herein and transmits a compressed video stream, typically with compressed or uncompressed audio, back to the client 415 via the Internet 410 (or, as described below, through an optimized high-speed network service that betrays bypassing the general Internet). Thereafter, the logic 412 restore compressed video with low latency at the client 415 restores the compressed video and audio streams and renders the recovered video stream, and typically plays the recovered audio to the display 422. Alternatively, the audio can be reproduced from the speakers separated from 422 display, or not play at all. We note that, despite the fact that input device 421 and display 422 are shown in Figures 2a and 2b as a standalone device, they may be integrated within client devices such as portable computers or mobile devices.

Home or office client 415 (previously described in Figure 2a and 2b as a home or office client 205) can be a very economical device with low power consumption, with very limited computing performance and graphics performance, and may have a very limited local mass storage or it may be entirely absent. On the contrary, each server 402 connected to the SAN 403 and a plurality of RAID 405, can be extremely high-performance computing systems, and, of course, if shared by multiple servers in the configuration of parallel processing, then there is almost no restrictions on computer system performance and graphics performance, which can be used. And because video compression 404 low-latency and video compression 412 low latency, the user gets the impression that the said computing power of servers 402 is provided to the user. When the user presses a key on the device input 421, the display 422 is updated in response to pressing this key without significant delay in terms of perception as in the case of the local execution of the game, or application software. Accordingly, with the home or office client 415, which is a computer with very low productivity, or only cost integrated circuit that implements the recovery of compressed video with low latency and logic 413 control signal, the user is provided with virtually arbitrarily chosen computational power from a remote location, and it seems that they are locally. This gives users the ability to run more advanced, CPU-intensive (usually new) video game and the most high-performance applications.

4C shows a very basic and cost-effective home or office client device 465. This device is an embodiment of the client's home or office 415 at 4a and 4b. Its length is approximately equal to 5 centimeters. It has 462 slot Ethernet, which provides an interface to an Ethernet cable with Power over Ethernet (PoE), through which it is powered and receives the connection to the Internet. It can perform network address translation (NAT) within a network that supports NAT. In the office environment, many new Ethernet switches have both PoE and PoE is brought directly to the Ethernet jack in the office. In this case, all that is required - is the Ethernet cable from the wall jack to the client 465. If the existing Ethernet connection does not transfer energy (for example, in a house with a DSL or cable modem, but without PoE), the commercially available fuel-efficient wall "modules "(ie, power supplies) that take an Ethernet cable without the power output and the Ethernet PoE.

The client 465 contains control signal logic 413 (in Figure 4a) which is connected to Bluetooth wireless interface, which interfaces with Bluetooth input devices 479, such as a keyboard, mouse, game controller and / or microphone and / or headset. In addition, one embodiment of client 465 can display video at 120 frames / s, when combined with the display 468 can support video 120 frames / s, and transmit signals (typically through infrared) a shutter glasses 466 to alternately closing the one eye, then the other with each successive frame. The result, perceived by the user is a stereoscopic 3D image that "pops" out of the display screen. One such display 468, which supports such an operation is the Samsung HL-T5076S. Since the video stream for each eye is separate, in one embodiment two independent video streams are compressed hosting service 210, the frames interleaved in time, and compressed frames are restored as two independent decompression processes within client data 465.

The client 465 also includes logic 412 restore compressed video with low latency, which restores compressed incoming video and audio output through HDMI (High Definition Multimedia Interface) connection cable 463, which is connected to the SDTV (TV standard definition) or HDTV (High Definition) 468, if the TV with video and audio, or to a monitor 468 that supports HDMI. If your monitor does not support 468 by HDMI, it can be used HDMI-DVI (digital video interface), but the audio will be lost. According to the standard HDMI, characteristics 464 of the display (for example, supported resolutions, frame rate) are transferred from the display 468, and this information is then returned via the connection 462 to the Internet back into service 210 hosting, so it can transmit a stream of compressed video in a format suitable for display.

4d shows a home or office client device 475 that is identical to the home or office client device 465 shown in FIGURE 4c, except that it has more external interfaces. In addition, the client 475 can accept either PoE for power, or it can run from an external power supply adapter (not shown) which is inserted into the socket on the wall. Using client 475 USB input, video camera 477 provides compressed video to client 475, which is discharged by the client 475 in the hosting service 210 for use described below. Despreader low latency, using the compression techniques described below is embedded in the chamber 477.

In addition to the Ethernet connector to connect to the Internet client 475 also has 802.11g wireless interface to the Internet. Both interfaces can use NAT in a network that supports NAT.

Furthermore, in addition to the HDMI to output video and audio, the client 475 also has a connector Dual Link DVI-I (dual channel DVI-I), which includes an analog output (with adapter cable provides a reference output VGA). It also has analog outputs for composite video and S-video.

For audio, the client 475 has left / right RCA jacks analog stereo and digital audio output, he has an output TOSLINK (optical output).

In addition to a Bluetooth wireless interface to input devices 479, it also has USB jacks to interface to input devices.

In 4E illustrates one embodiment of the internal architecture of client 465. Either all or some of the devices shown in the diagram can be implemented in a field programmable gate array, a custom ASIC circuit or a plurality of individual devices, or custom, or commercially available ready- form.

The Ethernet PoE 497 is connected to the interface 481 Ethernet. Power 499 is supplied from the Ethernet PoE 497 and is connected to other devices in the client 465. The bus 480 is a common bus for exchanging information between devices.

Manages the CPU 483 (almost any small CPU, for example, CPU Series R4000 with a speed of one million instructions per second (MIPS) and a clock frequency of 100 MHz with built-in RAM meets), performing small client management application from the flash memory 476, ensures that the protocol stack network (ie, the interface Ethernet), and also communicates with the service hosting 210, and configures all the devices in the client 465. It also manages the interfaces with the input device 469, and sends the packets back into service 210 hosting user data controller protected by Forward Error Correction, if necessary. In addition, the control CPU 483 monitors the packet traffic (for example, if packets are lost or delayed, and makes a note of the time of receipt). This information is sent back to the host service 210 so that it can constantly be monitoring the network connection and correct that it sends, respectively. In the flash memory 476 during manufacture of the original control program is loaded to the control CPU 483, and the serial number that is unique to a particular client 465 pieces of equipment. This serial number allows the host service 210 to uniquely identify the client 465 pieces of equipment.

Bluetooth interface 484 communicates with the input device 469 in a wireless manner through its antenna located inside the client 465.

The apparatus 486 is a compressed video restoration unit restoring compressed video with low latency that is configured to implement the video decompression described herein. There are a large number of video decompression devices, or commercially available off the shelf, or as Intellectual Property (IP) design that can be integrated into an FPGA or a custom ASIC circuit. One company offering IP to the decoder H.264, it is Ocean Logic, Manly, NSW Australia. The advantage of using IP is that the compression techniques used herein do not conform to compression standards. Some standard data decompression device are sufficiently flexible to provide a method of compression described in this document, but some can not provide them. But, by IP, there is complete flexibility in redesigning the device for restoring the compressed data as required.

The output of the recovery of the compressed video subsystem 487 is connected to the video output that connects Video video output interface 490 HDMI.

Subsystem 488 restore the compressed audio is realized or a standard recovery unit of compressed audio, which is commercially available, or it can be implemented as IP, or the restoration of compressed audio can be implemented within the control processor 483, which may, for example, to implement reduction apparatus compressed audio Vorbis.

The device, which carries out the restoration of the compressed audio subsystem 489 is connected to the audio output that connects to the audio output audio interface 490 HDMI.

On 4F illustrates one embodiment of the internal architecture of the client 475. As can be seen, the architecture is identical to the architecture of the client 465 with the exception of additional interfaces and an optional external DC power from the power adapter that is inserted into a socket on the wall, and, if so is used, it replaces the power that can come from 497 PoE Ethernet. Features that are present at the client 465 will not be repeated below, but the additional functionality is described as follows.

CPU 483 communicates with additional devices and configures them.

WiFi subsystem 482 provides wireless access to the Internet, as an alternative to Ethernet 497 through its antenna. In commercially available WiFi subsystem from a wide range of manufacturers, including Atheros Communications, Santa Clara, CA (Santa Clara, CA).

Subsystem 485 USB provides an alternative option in relation to the Bluetooth connectivity for wired devices 479 input USB. USB subsystem are fairly standard and generally available for the FPGA circuits and ASIC, and they are often built into the existing ready-made devices performing other functions, such as the restoration of the compressed video.

Subsystem 487 video output displays a wider range of video outputs than within the client 465. In addition to providing 490 HDMI video output, it provides a DVI-I 491, S-video and composite video 492 493. In addition, when used for digital video interface DVI- I 491, 464 of the display characteristics of the display are transmitted back to the control CPU 483 so that he could inform the host service 210 on the characteristics of the display 478. All interfaces provided by subsystem 487 video output interfaces are fairly standard and generally available in many kinds.

O subsystem 489 audio outputs audio in a digital format via a digital interface 494 (S / PDIF and / or TOSLINK) and audio in analog form through the stereo analog interface 495.

Analysis of the transmission time to and fro

Undoubtedly, the implementation of the advantages outlined in the previous section, the transmission time to and fro between a user's operation using the input unit 421 and the appearance of the result of this action, the display 420 should not exceed 70-80 ms. This latency must take into account all of the factors in the path from input device 421 to the user area 211 to the hosting service 210 and back to the user area 211 to the display 422. Figure 4b illustrates the various components and networks over which signals must pass and higher these components and networks is the time scale on which the withdrawn in an orderly manner illustrative value of the waiting time that can be expected in the implementation. Note that 4B is simplified and shows only critical path routing. Other routing of data used for other features of the system is described below. Bidirectional arrows (e.g., arrow 453) indicate the transmission time to and fro, and unidirectional arrows (e.g., arrow 457) indicate the time of signal transmission in one direction, and "~" denote an approximate measure. It is pointed out that there will be real-life situations in which it will be impossible to obtain the listed values ​​latency, but in most cases in the US, using compounds by cable modem and DSL with an area 211 of the user, these quantities waiting time can be obtained in the cases described in next section. Also note that, despite the fact that the mobile wireless communication with the Internet will certainly function imaging system, in most current US cellular data systems (such as EVDO) there are very large values ​​of the waiting time and can not be obtained values ​​timeout depicted 4b. However, these underlying principles may be implemented in future cellular technologies that may be implemented by the latency level.

Since the device 421 input on site 211 the user after the user operates the apparatus 421 input control signal the user is sent to client 415 (which may be a self contained device such as a set-top box, or it may be software or hardware software operating in another device such as a PC or mobile device) and is divided into packets (in UDP format in one embodiment) and the packet is given a destination address for transmission to the hosting service 210. The package will also contain information to indicate from which user the control signals. After that the package (s) of the control signal is sent (are) through the device 443 firewall / router / NAT (network address translation) to the interface 442 WAN. WAN interface 442 is an interface device that provides for the area 211 by an ISP user (provider of Internet-services). Interface 442 WAN can be a cable or modem, DSL, transceiver WiMax, transceiver for fiber-optic communication line interface cellular data, data interface over IP via the mains (Internet Protocol-over-powerline) or any other of a variety of interfaces Internet. Furthermore, the device 443 Firewall / Router / NAT (and possibly WAN interface 442) may be integrated into the client 415. An example of this would be a mobile phone, which includes software to implement the functionality of home or office client 415, and and means for routing and connection to the Internet wirelessly through some standard (eg, 802.11g).

Thereafter, interface 442 WAN routes the control signals to what is called in this description "entry point to Internet" 441 ISP Internet-service provider (ISP) user, which is a tool that provides an interface between data communications, WAN, connected with the territory 211 Users and general Internet or private network. Characteristics of the entry point to the Internet vary depending on the nature of Internet-services are provided. For DSL, as a rule, is a telephone company central office, where the DSLAM. For cable modems, as a rule, it is the central station multisystem operator (MSO) cable line. For cellular systems, it typically will be a control room associated with cellular antenna mast. But whatever the nature of the entry point to the Internet, she further routes the packet (s) of the control signal to the general Internet 410. Package (s) of the control signal further routed to the WAN interface with the service 441 210 hosting, through what will most likely be an interface transceiver for fiber-optic communication lines. WAN 441 further routes packets control signal logic 409 routing (which may be implemented in many different ways, include Ethernet switches and routing servers), which defines the user's address and routes the control (s) signal (s) to the corresponding server 402 for this User.

The server 402 receives the control signals as input signals for the game or application software that are executed on the server 402, and uses them for processing the next frame of the game or application. After the formation of the next frame of video and audio output from server 402 to video compression apparatus 404. Video and audio may be output from server 402 to device 404 by various compression means. Firstly, the compression device 404 may be built into server 402, respectively, the compression may be implemented locally within server 402. Or, the video and / or audio may be output in the form of packets through a network connection such as Ethernet connection, a network that is either a private network between the server 402 and the device 404 video compression, or be shared by a network such as SAN 403. Or video can be output from the video output connector of the server 402, such as a VGA or DVI, and then captured by the video compression unit 404. In addition, audio can be output from the server 402 or digital audio (for example, through the S / PDIF connector or TOSLINK), or analog audio is digitized and encoded audio compression logic in the device 404 video compression.

After capturing the video compression apparatus 404 is a video and audio generated during that frame period, from the server 402, the video compression apparatus compresses video and audio using techniques described below. After the video and audio compression, the package is formed with an address to send back to the user client 415, and routed to WAN interface 441, which then routes the video and audio packets through the general Internet 410, which then routes the video and audio packets to the point in the entrance 441 ISP Internet user, which routes the video and audio packets to the WAN Interface 442 user area, which routes the video and audio packets to the device 443 Firewall / Router / NAT, which further routes the video and audio packets to the client 415.

Client 415 restores the compressed video and audio and then outputs the video on the display screen 422 (or the built-in customer display), and sends the audio to the display 422, or in a separate amplifier / speakers or amplifier / speakers, built-in client.

In order for the user to interpret the whole process just described, without delay, two-way delay should be less than 70 or 80 ms. Some of the delay time-out in the described way back and forth are controlled by the hosting service 210 and / or the user, and others are not controlled. However, the following measurements are based on the analysis and testing of a large number of real-world scenarios are approximate.

The transfer time in one direction to send the control signal 451 is typically less than 1 ms, routing back and forth through 452 is typically performed by using the public switch the consumer level firewall / router / NAT on the Ethernet network for approximately 1 ms. I (compounds) of the user ISP differ significantly in their delayed signal 453 back and forth, but providers of cable modem and DSL, as a rule, there is a delay of between 10 and 25 ms. The transmission time to and fro on the general Internet 410 can be very different, depending on how the routed traffic, and whether there are any failures on the route (and these issues are discussed below), but typically the general Int ernet provides fairly optimal routes, and the waiting time is largely determined by speed of light through the light guide with the distance to the destination. As further discussed below, the inventors hosting service 210 at most approximately 1000 miles (1600 km) on which it is expected to be located from the user area 211. The actual transmission time over the Internet to a distance of 1,000 miles (1,600 km) (2,000 miles (3,200 km) round-trip) is approximately 22 ms. 441 WAN interface with the service 210 is generally host interface fiber-optic communication line with high speed commercial grade with little latency. Accordingly, the total standby time Internet 454 is typically between 1 and 10ms. Can be achieved by routing 455 latency in one direction through the hosting service 210 is less than 1 ms. The server 402 typically calculates a new frame for a game or an application in less time than one frame period (that is at 60 frames / sec is 16.7 ms), respectively, of 16 ms is acceptable to use the maximum time of 456 transmitting a signal in one side. In an optimized hardware implementation of compression algorithms compress audio and video, as described herein, the compression 457 can be performed for 1 msec. In less optimized versions, the compression may take 6ms (certainly even less optimized versions could take longer, but such implementations may affect the overall latency transmission back and forth, and may require that other quantities latency was smaller (e.g. It can be reduced allowable distance through the general Internet) to maintain a target latency of 70-80 ms). Values ​​of the transmission time to and fro for the Internet 454 (connection) 453 User ISP, and routing 452 for area user already been considered, so it remains to consider timeout recovery 458 of compressed video, which, depending on whether implemented whether restoration 458 compressed video in dedicated hardware, or whether it is implemented in software on a client device 415 (e.g., PC or mobile device) may vary depending on the display size and performance of the CPU for restoration of the compressed data. Typically, the restoration of the compressed data 458 takes between 1 and 8ms.

Accordingly, by adding the values ​​of all the waiting time in the worst case, observed in practice, it is possible to determine the time of signal transmission to and fro in the worst case, that is, as you might expect, the user will experience the system shown in Figure 4a. They are: 1 + 1 + 25 + 22 + 1 + 16 + 6 + 8 = 80ms. And, of course, in practice (with reservations, discussed below), it is approximately equal to the time signal back and forth, observed when using a pilot version of the model shown in Figure 4A using available off the shelf PC Windows as the client devices and home Connections cable modem and DSL within the United States. Sure, the script that best worst case, can result in a much smaller amount of time waiting, but they can not be based in the development of commercial services, which are widely used.

For timeout values ​​derived in an orderly manner 4b, signal transmission through general Internet, requires that the device 404 and the video compression apparatus 412 restoring the compressed video by 4a in the client packet flow 415 formed with very specific characteristics, for example, a sequence of packets, formed all the way from the service 210 hosting to display 422, has not been delay or excessive packet loss and, in particular, constantly satisfy the constraints of bandwidth available to the user to connect to the Internet user through the interface 442 WAN and firewall / router / NAT 443. In addition, the device is a video compression should create a stream of packets, which is stable enough, so that he could avoid the inevitable packet loss and packet reordering, which occurs in the conventional Internet and network broadcasts.

Video compression with low latency

To achieve the above objectives, in one embodiment, a new approach to video compression, whereby a reduced latency and peak bandwidth requirements for transmitting video transmission. Before describing these embodiments will be provided by the analysis of modern video compression methods according to Figures 5 and 6a-6b. Clearly, these methods may be employed according to underlying principles if the user is provided with a bandwidth sufficient for the data at the transfer rate required by these techniques. Note that in this specification is not considered audio compression, except for the observation that it is implemented simultaneously and in synchronism with the video compression. Audio compression techniques exist in the art, which meet the requirements for this system.

5 illustrates one particular prior art method for compressing video in which each individual video frame 501-503 is compressed by compression logic 520 using a particular compression algorithm to generate a sequence of compressed frames 511-513. One embodiment of this method is the "moving image compression algorithm MJPEG", in which each frame is compressed according to the still image compression algorithm developed by the Joint Expert Group machined photographic images (JPEG) based on the discrete cosine transform (DCT). However, there may be used various other types of compression algorithms, although still complying with these underlying principles (e.g., compression algorithms based on wavelet, e.g., JPEG-2000).

One problem with this type of compression is that it reduces the data rate of each frame, but it does not take the similarities between successive frames to reduce the data rates of all video streams. For example, as shown in Figure 5, assume that the frame rate is 640 × 480 × 24 bits / pixel = 640 × 480 × 24/8/1024 = 900 Kilobytes / frame (KB / frame), for a given quality of image, by an algorithm MJPEG moving picture compression can compress the stream by a factor of 10, resulting in a data stream of 90 KB / frame. At 60 frames / sec would require a channel bandwidth of 90 Kbytes × 8 bits × 60 frames / sec = 42.2 Mbit / s, which would be too broad bandwidth for almost all home connections to the Internet in the United States is currently too wide Bandwidth for many office connections to the Internet. Certainly, given that it will require a constant stream of data with a high bandwidth, and it will be the only one user, even in an office environment LAN, it will consume a lot of bandwidth LAN Ethernet 100 Mb / s, and strong load switches Ethernet, supporting LAN. Accordingly, the compression for motion video is inefficient when compared with other methods of compression (e.g., compression methods, described below). In addition, compression algorithms of one frame like JPEG and JPEG-2000 that use lossy compression algorithms, output the compression artifacts that may not be noticeable in still images (e.g., an artifact within dense foliage in the scene may not appear artifact as the eye does not know exactly how dense foliage should look like). But if the scene is moving, the artifact may be noticeable, because the eye notices that artifact (varies) from frame to frame, despite the fact that the artifact is in an area of ​​the scene where it might not be noticeable in a still image. This results in the perception of "background noise" in the sequence of frames, similar in appearance noisy "snow" visible during boundary analog TV reception. Undoubtedly, this type of compression may, however, be used in certain embodiments described herein, but generally speaking, for a given perceptual quality requires the removal of background noise in the scene, a high data rate (i.e., a low compression ratio ).

Other types of compression, such as H.264, or Windows Media VC9, MPEG2 and MPEG4, are all more efficient at compressing a video stream because they use the similarities between successive frames. All these methods are based on the same general methods of video compression. Accordingly, although the H.264 standard will be described, however, identical general principles apply to various other compression algorithms. A large number of devices, H.264 compression and decompression device data are available, including software library with open source × 264 H.264 compression and software library with open source FFmpeg to restore compressed data H.264.

6a and 6b illustrates an exemplary method of compression of the prior art in which a sequence of uncompressed video frames 501-503, 559-561 are compressed by compression logic 620 into a series of "I-frames" 611, 671, "P-frames" 612 -613 and "B-frames" 670. The vertical axis in Figure 6a generally denotes the resulting size of each of the encoded frames (although the frames are not drawn to scale). As described above, video coding using I-frames, B-frames and P-frames understood by those skilled in the art. In a few words, I-frame 611 is a compression based on DCT full uncompressed frame 501 (the same image compression JPEG, as described above). P-frames 612-613 generally much smaller than the I-frames 611 because they use data of the previous I-frame or P-frame, that is, they contain data indicating the changes between the previous I-frame or P-frame. B-frames 670 are similar to P-frames except that B-frames use the frame in the following reference frame as well as, possibly, the frame in the preceding reference frame.

In the following discussion it will be assumed that the required frame rate is 60 frames / second, that each I-frame is approximately 160 Kbps, the average P-frame and B-picture is 16 Kbps, and the new I-frame is generated every second. With this set of parameters the average data rate is 160 kbps 16 kbps + = 1.1 × 59 Mbit / s. This data rate is well within the maximum data rate for many modern high speed Internet connections to homes and offices. By this method also can solve the problem of background noise, when only the intra coding, because the P- and B-frames track differences between the frames, so compression artifacts do not appear and disappear from frame to frame, thus decreasing the background noise problem described above .

One problem with the foregoing types of compression is that although the average data rate is relatively low (e.g., 1.1 Mbit / s), the transfer of one I-frame may take several frame periods. For example, using prior art methods, a network connection 2.2 Mbit / s (e.g., DSL or cable modem with a peak of 2.2 Mbit / s the maximum available data rate 302 3a) is typically enough to stream video from rate of 1.1 Mbit / s with an I-frame 160 kbit / s every 60 frames. This can be accomplished if the compressed data restoration apparatus will queue up 1 second of video compressed video restoration. 1 second 1.1 megabits of data is transmitted, which can be readily achieved at the maximum available data rate of 2.2 Mbit / s, even assuming that the available data rate may be periodically reduced by 50%. Unfortunately, this prior art approach results in latency of 1 second for the video because of the 1-second video buffer at the receiver. This delay corresponds to the requirements of many applications of the prior art (eg, linear video playback), but is too long waiting time for dynamic video game-fighters, which is not permitted while waiting for more than 70-80 ms.

If we try to eliminate the 1-second video buffer, it still does not lead to a reduction in the waiting time, sufficient for dynamic video game-fighters. For example, the use of B-frames as described above, necessitates the reception of all B-frames, preceding I-frame and I-frame. Assuming that 59 frames are not-I-frames between roughly divided P- and B-frames, it is accepted by at least 29 B-frames and I-frame before any B-frame can be displayed. Accordingly, regardless of available bandwidth, this inevitably entails a delay 29 + 1 = 30 frames duration of 1/60 second each, or 500ms latency. Obviously, this is too long.

Accordingly, another approach must be excluded and B-pictures use only the I- and P-frames. (One consequence of this is that the data rate is increased for a given quality level, but for consistency in this example, we will continue to assume that the size of each I-frame is equal to 160 kbps and the average P-frame is 16 Kbps, and accordingly, the speed Data is still equal to 1.1 Mbit / s). This approach eliminates the inevitable latency introduced B-frames, because decoding of each P-frame depends only on the previous received frame. A problem that remains with this approach is that an I-picture is so larger than the average P-frame, which is a channel with a low bandwidth, as usually happens in most homes and in many offices, the transmission of I-frame adds substantial latency . This is illustrated in Figure 6b. Speed ​​624 video data rate below the available maximum data 621 except for the I-frames, where the peak data rate required for the I-frames 623 far exceeds the available maximum data rate 622 (and even the rated maximum data rate 621). The data rate required by the P-frames, less available maximum data rate. Even if the peaks of 2.2 Mbit / s available maximum data rate constantly remain at its peak rate of 2.2 Mbit / s, the transmission of I-frames takes 160 Kbit / 2.2 Mbps = 71 ms, and if the available maximum speed 622 data reduced by 50% (1.1 Mbit / s), the transmission of I-frames will take 142 ms. Accordingly, the waiting time in the transmission I-frame is approximately between 71-142 ms. This waiting time is in addition to the timeout values ​​identified in Figure 4b, which in the worst case add up to 70 ms, so the resulting total transmission time to and fro, from the moment when the user actuates the input device 421 to how the image appears on the display 422 is equal to 141-222 milliseconds, which is too large. And if the available maximum data transfer rate becomes less than 2.2 Mbit / s, the waiting time is increased further.

We also note that, in general, there are serious consequences, "the creation of plug" ISP through a peak speed of data transmission 623, which is much higher than the available data rate of 622. The equipment at different ISP behaves differently, but the next line of action are quite common among the ISP cable modem and DSL when receiving packets with much higher data rate than the rate available 622 Data: (a) packet delay means putting them in all (the introduction of latency), (b) discharge of some or all of the packages, (c) termination of the connection for a certain period of time (most likely because the ISP is concerned that this is a malicious attack, such an attack "denial of service" ). Accordingly, the transmission packet stream at full data rate with characteristics such as the characteristics shown in Figure 6b is not a viable option. The peaks 623 may be queued in the host service 210 and sent at a data rate below the available maximum data rate, which introduces an unacceptable latency described in the preceding paragraph.

Furthermore, the sequence 624 of video stream data rate is illustrated in Figure 6b, is a "normal" sequence of the data rate of the video stream, and may be the kind of data rate sequence that can be expected, is obtained by compressing the video from a video sequence that It does not change very much and shows very little movement (for example, as usually happens during video teleconferencing, where the cameras are in a fixed position, and few of them moved, and the objects in the scene, for example, people are sitting talking, little exercise).

The sequence of 634 data rate of the video stream shown in Figure 6C, is a sequence which would normally be expected from a video monitor with a much larger number of actions, such as that can be formed in a film or video game, or in some application software. Note that along with peaks of 633 I-frame, there are peaks of P-frame, such as 635 and 636 that are quite large and exceed the available maximum data rate on many occasions. Despite the fact that these peaks P-frame is not quite as large as the peaks of the I-frame, they are, nevertheless, are too large for transmission over the channel at full data rate, and as is the case with peaks I-frame They, P-frame peaks must be transmitted slowly (hence latency from increasing).

The channel-bandwidth (e.g., LAN 100 Mbit / s or a private connection with a wide bandwidth of 100 Mbit / s) in the network may be allowed, large peaks, such as peaks 633 I-frame or peaks 636 P-frame, and in principle It can be kept low latency. However, such networks are often shared by many users (e.g., in an office environment), and such data is "with peaks" affect performance LAN, particularly if the network traffic is routed to the particular shared connection (e.g., from a remote processing center and storage Data in the office). First of all, we should consider that in this example the video stream with relatively low resolution of 640 × 480 pixels, 60 frames / sec. Streams HDTV 1920 × 1080 60 frame / s is easily treated with modern computers and displays, and more commercially available displays with a resolution of 2560 × 1440 with 60 frames / sec (for example, a display of 30 "of the corporation Apple, Inc). Video sequences with high activity with 1920 × 1080 60 frames / sec may require 4.5 Mbit / s using H.264 compression for a reasonable quality level. If it is assumed that I-frames reach their peak at the nominal data rate 10X, the result obtained by peaks 45 Mbit / s, as well as smaller, but nevertheless significant, peak P-frames. If several users receive the video streams on an identical network 100 Mbit / s (e.g., a private network connection between an office and the center of data processing and storage) it is easy to see how the peaks of the video stream for multiple users can be combined with the overflow of network bandwidth and overflow bandwidth backbone switches that support users in the network. Even in the case of network technology Gigabit Ethernet, if a sufficient number of peaks sufficient number of users together simultaneously, it could overwhelm the network or the network switches. And when a video with a resolution of 2560 × 1440 becomes more commonplace, the average video stream data rate may be equal to 9.5 Mbit / s, as a result, possibly leading to the peak data rate of 95 Mbit / s. There is no doubt that the connection is 100 Mbit / s between the central data processing and storage and the office (which is currently the only high-speed connection) will be completely overflowing with peak traffic individual user. Accordingly, even though the LAN and private network connections can allow streaming video with peaks streaming video with high peaks it is not desirable and might require special planning and adaptation IT department office.

Sure, standard linear video applications, these issues are not a problem, because the data rate is "smoothed out" at the time of transmission, and data for each frame is lower than the maximum available speed data 622, and the buffer is stored in the client sequence of I-, P - and B-frames before they are recovered. Accordingly, the data rate over the network remains close to the average data rate of the video stream. Unfortunately, this introduces latency, even if B-frames are not used, which is unacceptable for applications with low latency, such as video games and applications require low response times.

One prior art solution to reduce the video streams that have high peaks is to use the method commonly referred coding "Constant bit rate" (constant bit rate, CBR). Although it appears that the term CBR imply that all frames are compressed so that they had the same bit rate (i.e., size), it usually refers to a compression scheme, wherein the maximum allowed bit rate for a certain number of frames (in our case, 1 frame). For example, in the case of Figure 6C, if the restriction applies CBR to encode that limits the bit rate, for example up to 70% of the rated maximum data rate 621, then the compression algorithm limits the compression of each of the frames so that any frame that is typically compressed with using more than 70% of the rated maximum speed of 621 data is compressed by a smaller number of bits. The result of this is that frames that would normally require more bits to maintain a given quality level, "lack" bits and the image quality of those frames worse than the image quality of other frames that do not require more bits than the 70% ( speed) with a maximum data transmission speed of 621. This approach can lead to acceptable results for certain types of compressed video, where (a) expected little movement or scene changes and (b) users can be considered acceptable periodic deterioration. A good example of an application suitable for CBR, is holding a video teleconferencing, as it contains very little peaks, and if the quality deteriorates for a short period (e.g., if the camera pans, which results in significant scene motion and large peaks, during the panning may not be enough bits for high-quality image compression, which could result in degraded image quality), it is acceptable for most users. Unfortunately, CBR poorly suited for many other applications in which there are scenes of high complexity or a great motion, and / or require a fairly constant level of quality.

Compression logic 404 with low latency employed in one embodiment uses several different techniques to address the range of problems associated with the transmission of a stream of compressed video with low latency, while maintaining a high quality level. Firstly, the compression logic 404 with low latency generates only I-frames and P-frames, thus reducing the need for waiting for several frame periods to decode each B-frame. Furthermore, as shown in Figure 7a, in one embodiment, the compression logic 404 with low latency divides each uncompressed frame 701-760 into a series of "fragments" ("tile", "tile") and individually encodes each tile or I-frame or P-frame. The group of compressed I-frames and P-frames is called in this description "R-blocks" 711-770. In the specific example shown in Figure 7a, each uncompressed frame is divided into 4 × 4 matrix of 16 tiles. However, the underlying principles are not limited to any specific circuit units.

In one embodiment, the compression logic 404 with low latency divides the video frame into multiple fragments, and encodes (i.e., compresses) one fragment from each frame as an I-frame (i.e., the fragment is compressed as if it were a separate 1/16 video frame size of the full image, and the compression used for this "mini"-frame is an I-frame compression), and the remaining parts of the image as the P-frames (ie, the compression used for each "mini" 1 / 16 (portion) of the frame is a P-frame compression). Fragments compressed as the I-frames and as P-frames should be called "I-fragment" and "P-moieties", respectively. With each successive video frame, the segment to be coded as an I-fragment varies. Accordingly, the frame period, only one fragment from the fragments in a video frame is an I-fragment and the remaining fragments are P-fragments. For example, in Figure 7a, tile 0 of uncompressed frame 701 is encoded as I-I0 fragment, while the remaining 1-15 fragments encoded as a P-pieces P1 to P15 to output the R-frame 711. In the next uncompressed video frame 702, fragment 1 uncompressed Frame 701 is encoded as I-I1 fragment, and other fragments 0 and 2 through 15 are encoded as P-fragments, P0 to P2 and P15 at, for outputting R-frame 712. Accordingly, I-slices and P-slices for fragments gradually alternate time for consecutive frames. The process continues until until formed R- (frame) 770 to the last fragment in the matrix encoded as an I-fragment (i.e., I15). Thereafter, the process is repeated to form another R-frame, such as frame 711 (i.e., I-fragment coding for tile 0) etc. Although not shown in Figure 7a, in one embodiment, the first R-frame of the video sequence of R-frame only contains I-fragments (i.e. subsequent (R)-frames contain the reference image data, which is calculated on the basis of the movement ). Alternatively, in one embodiment, the sequence at the initial time uses a I-fragment identical to normal, but does not include P-fragments for those portions that have not encoded by the I-fragment. In other words, certain fragments are not encoded with any data until the first is not received I-fragment thus eliminates the initial peak in the data rate 934 on the video 9a, which is explained in more detail below. Furthermore, as described below, various other dimensions and shapes can be used to fragment, although still complying with these underlying principles.

Logic 412 restore compressed video, executable in the client 415, restores the compressed each fragment, as if it was a separate video sequence of small I- and P-frames, and then transmits each piece of the frame buffer, control the display 422. For example, I0 and P0 of R-frames 711-770 are used to reconstruct tile 0 compressed video and imaging. Similarly, I1 and P1 from R-frames 711-770 are used to reconstruct tile 1, and so on. As mentioned above, the restoration of the compressed I-frames and P-frames known in the art, and the recovery of compressed I-slices and P-fragments can be done by having a plurality of video decompression devices operating in the client 415. Although it seems that increase the number of processes increases the computational burden on the client 415, it actually is because the fragment itself proportionally smaller relative to the number of additional processes, so the number of pixels displayed on the screen is the same as if there were one process and using normal I- and P -frames full size.

This R-frame method greatly reduces the bandwidth peaks typically associated with I-frames illustrated in Figure 6b and 6c because any given frame is usually composed of P-frames, which are typically less than I-frames. For example, again assume that a conventional I-frame is 160 Kbps, then I-fragments of each of the frames illustrated in Figure 7a, are approximately equal to 1/16 of this value, or 10 kbps. Similarly, suppose that a normal P-frame is 16 Kb, then the P-frames for each of the fragments shown in Figure 7a may be roughly equal to 1 kbps. The end result is an R frame of approximately 10-kbps + 15 kbps × 1 = 25 kbps. Accordingly, each sequence of 60 frames is equal to 25 kbps × 60 = 1.5 MBit / s. Accordingly, at 60 frames / second required to support a channel bandwidth of 1.5 Mbit / s, but with much smaller peaks appearing due to the I-fragments, distributed over an interval of 60 frames.

Note that in previous examples with the same assumption that the data rates for I-frames and P-frames average data rate was 1.1 Mbit / s. The reason for this is that in the previous examples, a new I-frame is administered only once every 60 frame periods, whereas in this example, the 16 tiles that make up an I-picture cyclically repeated at 16-frame period, and substantially equivalent to I -frames is introduced every 16 frame periods, resulting in a slightly higher average data rate. In practice, though, introducing more frequent I-frames does not increase the data rate linearly. This is due to the fact that a P-picture (or P-Slice) initially encodes the subsequent frame difference from the previous. Accordingly, if the prior frame is quite similar to the next frame, the P-frame is very small, if the prior frame is quite different from the next frame, the P-picture is very large. But since the P-frame is largely derived from the previous frame, rather than from the actual frame, the resulting encoded frame may contain more errors (e.g., visual artifacts) than an I-frame with an appropriate number of bits. And, when one P-frame follows another P-frame, then the accumulation of errors can occur, which degrades in the presence of a long sequence of P-frames. Further, complications despreader video detects that the image quality deteriorates after a sequence of P-frames and, if necessary, it will allocate more bits subsequent P-frames to improve the quality or, if it is the most efficient course of action, it replaces a P-frame I -frames. Accordingly, when used long sequence of P-frames (e.g., 59 P-frames, as in the previous examples above) particularly when the scene is very difficult and / or there is a lot of motion typically requires more bits for the P-frames as they removal from I-frames.

In other words, if you look at P-frames from the opposite point of view, P-frames that immediately follow the I-frame, usually require fewer bits than P-frames, which are more distant from the I-frame. Thus, in the example shown in Figure 7a, all P-frames are removed from them prior I-frame is not more than 15 frames, while, for example, in the above example, P-frame can be within 59 frames of I- frame. Accordingly, at a higher frequency of I-frames is less than P-frames. Clearly, the exact relative sizes vary depending on the nature of the video stream, but in the example of Figure 7a, if the I-fragment is 10 Kbps, the size of P-fragments averaging can only be equal to 0.75 kbps, which results in 10 kbps + 15 kbps × 0.75 = 21.25 Kbps, or (speed) of 60 frames per second data rate is 21.25 kbps × 60 = 1.3 MBit / s, or about 16% greater than the speed data stream with an I-frame followed by 59 P-frames of 1.1 Mbit / s. Again, the relative performance of these two approaches to video compression vary depending on the video sequence, but typically, the experience shows that using R-frames require about 20% more bits for a given level of quality than using sequences I / P- frames. But of course, R-frames substantially reduce the peaks that much reduces latency by using said video sequences, as compared with the waiting time for a sequence of I / P-frames.

R-frames may be configured in many different ways, depending on the nature of the video sequence, the reliability of the channel, and the available data rate. In an alternative embodiment, a track number different from 16 in the configuration 4 × 4. For example, it may be used 2 fragment in configuration 2 × 1 or 1 × 2, can be used 4 fragments in a configuration of 2 × 2, 4 × 1 or 1 × 4 may be used six fragments in configuration 3 × 2, 2 × 3, 6 × 1 × 1 or 6 or 8 fragments can be used in the configuration 4 × 2 (as shown in Figure 7b), 2 × 4, 8 × 1 or 1 × 8. Note that the fragments do not necessarily need to be square, and the video frame does not have to be square or rectangular. Fragments may take any form which is best suited for your application, or video.

In another embodiment, the cycle of the I- and P-fragments not fixed number of fragments. For example, in the configuration 4 × 8 2 fragments can, however, be used with a periodic repetition sequence through the elements 16, as shown in Figure 7b. Sequential uncompressed frames 721, 722, 723 are each divided into 8 fragments 0-7 and each tile is compressed individually. The R-frame 731 only tile 0 is compressed as an I-fragment and the remaining fragments are compressed as P-fragments. In the next R-frame 732 all 8 pieces are compressed as P-fragments and further into the next R-frame 733 fragment 1 is compressed as an I-fragment and the remaining fragments are all compressed as P-fragments. And so sequencing continues for 16 frames, with the I-fragment is formed only through the frame, respectively, the last I-fragment will be generated for tile 7 during the period of the 15th frame (not shown in Figure 7b) and during the period 16th frame compressed frame 780-R using all P-fragments. Further, the sequence begins again with tile 0 compressed as an I-fragment, while other portions are compressed as P-fragments. As in the prior embodiment, the very first frame of the entire video sequence usually consists of all I-fragments to provide support elements (reference) for the P-fragments from this point. Cyclic repetition of I-slices and P-fragments should not even be an even multiple of the number of fragments. For example, with 8 fragments, each frame with an I-fragment can be followed by 2 frames with all P-fragments will be used before another I-fragment. In another embodiment, in specific fragments may be established sequence I-fragments are more likely than other fragments if, for example, it is known that in certain areas of the screen more traffic that requires frequent I-fragments, while others are more static (for example, featuring a counting game) requiring less frequent I-fragments. Furthermore, despite the fact that each frame 7a-7b illustrates one I-fragment in the same frame can be encoded in a plurality of I-fragments (depending on the bandwidth of the transmission channel). Conversely, certain frames or frame sequences may be transmitted with no I-fragments (i.e., only P-fragment).

The reason the approaches described in the preceding paragraph works well is that, unless spread I-fragments in each individual frame, it seems, it will result in large peaks, the behavior of the system is not so simple . Since each tile is compressed separately from the other fragments, that, as the fragments are smaller, coding each tile can become less efficient, because the compression unit of the fragment can not use similar image features and similar motion from the other fragments. Accordingly, the division of the screen into 16 fragments generally will result in a less efficient encoding than dividing the screen into 8 fragments. But, if the screen is divided into 8 pieces, and this causes the input complete I-frame every 8 frames instead of every 16 frames, it results in a much higher data rate overall. Accordingly, by introducing a full I-frame every 16 frames instead of entering his every 8 frames, the overall data rate is reduced. Furthermore, by using 8 larger fragments, instead of 16 smaller fragments reduces the overall data rate, which also reduces to some degree the data peaks caused by the larger fragments.

In another embodiment, the video compression logic 404 with low latency by 7a and 7b controls the allocation of bits to different fragments of frames in the R-configuration or by preliminary setting, based on the known characteristics of the video sequence that must be compressed, or automatically, based on a continuous analysis of the image quality in each tile. For example, some video games, races, the front part of the car player (which is relatively stationary in the scene) takes up most of the bottom half of the screen, while the upper half of the screen is full looming highway, buildings and landscapes that are almost always in motion. If the compression logic 404 allocates an equal number of bits of each fragment, the fragments in the lower half of the screen (fragments 4-7) in uncompressed frame 721 on Figure 7b, will generally be compressed with higher quality than the pieces in the upper half of the screen (fragments 0-3) in an uncompressed frame 721 to 7b. If you know that this particular game, or this particular stage of the game have such characteristics, the operators of 210 hosting service can configure compression logic 404 to allocate more bits fragments located in the upper part of the screen than the fragments found at the bottom of the screen. Alternatively, logic 404 compression can evaluate the quality of the compression of fragments after the compressed frames (by using one or more of the plurality of quality parameters of compression, for example, a peak signal / noise ratio (PSNR)), and if it determines that over a certain window of time quality of certain fragments constantly improving, then it gradually allocates more bits fragments whose quality deteriorates until, until the level of quality of the various fragments will not be about the same. In an alternative embodiment, the compression logic 404 allocates bits device for obtaining higher quality in a particular moiety or group of moieties. For example, with higher quality in the center of the screen than at the edges, it may provide a better overall perception.

In one embodiment, to improve resolution of certain regions of the video stream, the logic 404 of video compression use fewer fragments to encode areas of the video stream with relatively more scene complexity and / or high motion than areas of the video stream with relatively less scene complexity and / or lower movement. For example, as shown in Figure 8, smaller fragments are applied around a moving character 805 in one area of ​​one R-frame 811 (for which there may be a sequence of R-frames with identical dimensions fragments (not shown)). Further, when the character 805 moves to a new area of ​​the image, smaller fragments are used around this new area within another R-frame 812, as shown. As mentioned above, various other sizes and shapes can be used as "fragments", although still complying with these underlying principles.

While the cyclic I / P-fragments described above substantially reduce the peaks in the data rate of the video stream, they do not eliminate the peaks entirely, particularly in the case of rapidly-changing or highly complex video images that are found in movies, video games, and some application software. For example, during a sudden fader for complicated scenes may be followed by another complex frame, which is totally different from him. Despite the fact that several I-fragments may be just before the scene transition within a few frame periods, they do not help in this situation because the new frame material has no relation to the previous I-fragments. In such a situation (and in other situations where even though not everything changes, much of the image changes), the device 404 video compression determine that many, if not all, P-fragments are more efficiently coded as I-fragments in result is a very high peak data rate for the frame.

As discussed previously, it is simply the case that with most consumer Internet connections to a level (and many office connections) simply is not feasible for data creating "plug" that exceeds the available maximum data rate 622 to the position shown 6c, together with the rated maximum data rate 621. Note that the rated maximum data rate 621 (e.g., "DSL 6 Mbit / s") is substantially digit outlets for users considering the purchase of connecting to the Internet, but generally it does not guarantee a level of performance. For the purposes of this application, this is not relevant, since the only issue is the available maximum speed of 622 data during the transmission of video streams over the connection. Consequently, 9A and 9C, as we describe the solution to the problem of the peaks, the nominal maximum data rate is not shown on the chart and shows only the available maximum speed of 922 data. The data rate of the video stream does not exceed the available maximum data rate 922.

To solve this problem, first, making the device 404 of video compression is to determine the peak data rate 941, which is a data rate the channel is able to handle steadily. This rate can be determined in several ways. One such method is to gradually sending a test stream with all the higher data rate of service 210 by placing the information on a server in the client 415 by 4a and 4b, and providing customer feedback regarding hosting service level of packet loss and latency . When the packet loss and / or latency begins to increase sharply, which indicates the approach to the available maximum data rate 922. After this, 210 Hosting service can gradually reduce the data rate of the test flow as long as the client 415 does not tell you that for a sufficient period of time, the flow test was passed with an acceptable level of packet loss and latency is almost minimal. This establishes a peak maximum data rate 941, which is then used as a peak data rate for streaming video. Over time, the peak rate of 941 data will vary (for example, if another user who lives in this house, will begin to actively use the connection to the Internet), and the client 415 will be required to carry out the current control of the increase in latency or packet loss with an indication of that the available maximum data rate 922 falls below the previously established peak data rate 941, and if so, the peak data rate 941. Similarly, if over time the client 415 finds that the packet loss and latency remain at optimum levels, it may send a request to the device video compression slowly increase the speed of data to verify that the increase is available to the maximum data rate (eg, discontinued if another user who lives in the house intensively use a connection to Internet), and then waits until until packet loss and / or higher latency will not indicate that the available maximum speed of 922 data transmission has been exceeded, and It can be found again a lower level for the peak data rate 941, but one that may be higher than the level before testing was to increase the data rate. Accordingly, using this technique (and other techniques like it) can be found by the peak data rate 941, and adjusted periodically as needed. The peak rate of 941 data sets the maximum data rate that can be used by device 404 video compression to stream video to the user. The logic for determining the peak data rate may be implemented at the user site 211 and / or 210 in the service hosting. At site 211 user client device 415 performs the calculations to determine the peak data rate and transmits this information back to the service 210 hosting a service 210 hosting server 402 performs the calculations to determine the peak data rate based on statistics received from the client 415 (e.g., packet loss, latency, maximum data transfer rate, etc.).

9A shows an exemplary data transmission rate of the video stream 934, which contains a substantial scene complexity and / or significant movement formed compressing method using the cyclic I / P-fragment described above and illustrated in Figures 7a, 7b and Fig. 8. Video compression apparatus 404 has been configured to output compressed video at an average data rate that is below the peak data rate 941, and note that almost all the time, the video stream data rate remains below the peak data rate 941. When comparing velocity data 934 from the data transmission rate of the video stream 634 of Figure 6c created using I / P / B- or I / P-frames, it is seen that the output of compression cyclic I / P-fragment obtained much more smooth the transmission rate. However, the peak 952 2 × frame (which is close to the peak speed of 942 data 2 ×) and the peak 954 4 × frame (which is close to the peak speed of 944 data 4 ×) data rate exceeds the peak speed of 941 data, which is unacceptable . In practice, even with the video with high activity in the videogame with a rapid change of the peaks that exceed the peak rate 941 data, there is less than 2% of frames, peaks in excess of peak rate 942 data 2 ×, are rare, and peaks in excess of peak rate 943 Data 3 ×, do not occur rarely. But when they do occur (for example, during the Transition), they require a data rate is necessary for the video output of good quality.

One way to solve this problem is simply to configure the video compression apparatus 404 so that the output of its maximum data rate is a peak data rate 941. Unfortunately, the resulting quality of the video output during peak frame is bad, because the compression algorithm "lacks" in bits. The result is compression artifacts when carried sudden transitions or rapid movement, and over time the user realizes that the artifacts are always crop up when there are sudden changes or fast motion, and it can it is very annoying.

Despite the fact that the human visual system is quite sensitive to the visual artifacts that appear during sudden changes or fast motion, it is not very sensitive to notice a decrease in frame rate in such situations.

In fact, when there are sudden changes, it turns out that the human visual system is busy tracking changes, and she does not notice, if the frame rate for a short time drops from 60 frames / sec to 30 frames / sec and then immediately returns to 60 frames / sec. And in the case of very significant transition, such a sudden change in the scene, the human visual system does not notice, if the frame rate drops to 20 frames / s or 15 frames / sec and then immediately returns to 60 frames / sec. Until frame rate reduction occurs infrequently, to a human observer, it appears that the video is continuously transmitted at 60 frames / sec.

This property of the human visual system is used in the method shown in Figure 9b. Server 402 (in Figures 4a and 4b) outputs the uncompressed video stream output at a steady frame rate (60 frames / s in one embodiment). The timeline shown that each frame 961-970 is displayed every 1 / 60th of a second. Each uncompressed video frame, since the frame 961, is displayed in the video compression device 404 with low latency, which compresses the frame in a time shorter than the period of the frame, with the conclusion of the first frame compressed frame 981 1. The data output to the compressed frame 981 1 may be larger or smaller, depending upon many factors, as previously described. If the data size is so small that they can be transmitted to the client 415 during a frame period (1 / 60th of a second) or less at the peak rate 941 data, they are transmitted over a period of 991 transmit (xmit time) (length of the arrow It indicates the duration of the transmission). During the next frame period, server 402 outputs uncompressed frame 2 962, it is compressed to compressed frame 2 982, and it is transmitted to client 415 during transmit time 992, which is smaller than the frame period, with peak data rate 941.

Further, during the next frame period, server 402 outputs uncompressed frame 963 3. When it is compressed by video compression apparatus 404, the resulting compressed frame 3 983 is more data than can be transmitted at a peak data rate 941 in one frame period. Accordingly, it is transmitted during transmit time 993 (peak 2 ×), which occupies the entire frame period and part of the next frame period. Further, during the next frame period, server 402 outputs uncompressed frame 964 different 4 and outputs it to the video compression apparatus 404, but the data is ignored, and they are represented by reference numeral 974. This is because video compression apparatus 404 is configured to ignore the next uncompressed video frame, that come when it is still transmitting the previous compressed frame. Undoubtedly, the device recovery compressed video client 415 accepts the frame 4, but it's just going to continue to display the display frame 422 for 3 in two periods of the frame (ie, a short time will reduce the frame rate to 60 frames / sec to 30 frames / from).

For the next frame 5 server 402 outputs uncompressed frame 5 965, which is compressed to compressed frame 5 985 and transmitted within 1 frame during transmit time 995. The device recovery compressed video client 415 restores the compressed frame 5, and displays it on the display screen 422. The server 402 outputs an uncompressed frame 6 966, the device 404 video compression compresses it into a compressed frame 986 6, but this time the size of the resulting data is a big. The compressed frame is transmitted during transmit time 996 (4 × peak) 941 at the peak rate of data transmission, but the transmission frame takes almost 4 frame period. Over the next three frame periods of the video compression apparatus 404 ignores 3 frames from server 402, and the compressed data restoration device client 415 continuously outputs the frame 6 on the display 422 during 4 frame periods (i.e., briefly reduces the frame rate from 60 fps / sec to 15 frames / sec). Finally, the server 402 outputs the block 970 10, the device 404 video compression compresses it into compressed frame is 987, 10 and it is transmitted during period 997 of transmission and the recovery of the compressed data of the client 415 restores the compressed frame 10 and displays it on the display screen 422, and again resumed output video at 60 frames / sec.

Note that despite the fact that the device 404 video compression resets the video frames of the video stream generated server 402, it does not drop audio data, regardless of what form enters audio, and it continues to compress the audio data when video frames are reset, and transmits them to the client 415 which continues to restore compressed audio and provide the audio device that the user uses for audio playback, whatever it may be. Accordingly, the audio continues to play completely during periods when the frames are discarded. Audio compression using a relatively small percentage of bandwidth, compared to compressed video, and as a result does not have a significant impact on the overall data rate. Although it is not shown in any of the data rate diagrams, there is always a throughput data rate reserved for the compressed audio stream within the peak data rate 941.

The example just described for Figure 9b was chosen to illustrate how the frame rate drops during data rate peaks, but what it does not illustrate how to use methods with the cyclic I / P-fragment described above, such rate peaks Data dropped and the subsequent frames are rare even during sequences with high activity / high complexity of the scene, for example, sequences that are found in video games, movies and some application software. Consequently, the reduced frame rate are infrequent and short-lived, and the human visual system does not notice them.

If just describe sledge frame rate reduction mechanism applied to the video stream data rate illustrated in Figure 9A, then the resulting video stream data rate is illustrated in 9C. In this example, peak 2 × 952 was reduced to flattened peak 2 × 953 and 4 × 955 peak has been reduced to flattened peak 4 × 955, and the entire data transmission rate of the video stream 934 remains at the peak data rate 941 or below.

Accordingly, using the methods described above, the video with high activity can be transmitted with low latency through the general Internet and through a connection to the Internet consumer level. Further, in an office environment in a LAN (e.g., Ethernet 100Mbps or 802.11g wireless) or on a private network (e.g., a compound of 100 Mbit / s between the central data processing and storage and offices) video stream with high activity can be transmitted without peaks so that a plurality of users (e.g., transmitting 1920 × 1080 60 frame / s 4.5 Mbit / s) can use the LAN or shared private data connection without overlapping peaks crowding the network backbone or network switch.

Adjusting the data rate

In one embodiment, the hosting service 210 initially assesses the available maximum data rate 622 and latency of the channel to determine an appropriate data rate for the video stream and then dynamically adjusts the data rate in response. To adjust the data rate of the hosting service 210 may, for example, modify the image resolution and / or the number of frames / second of the video stream to be sent to the client 415. Also, the hosting service can adjust the quality level of the compressed video. When changing the resolution of the video stream, for example with the resolution of 1280 × 720 640 × 360, logic 412 to restore compressed video client 415 may cause the image scale to maintain identical image size on the display screen.

In one embodiment, in a situation where the channel signal is lost, the hosting service 210 pauses the game. In the case of games with several participants hosting service informs other users that said user out of the game and / or suspend the game with other users.

Dropped or delayed packets

In one embodiment, if data is lost due to packet loss between the video compression apparatus 404 and the client 415 at 4a or 4b, or due to the packet received out of order that arrives too late to restore the compressed frame, and meet latency reconstructed frame recovery logic 412, the compressed video can reduce visual artifacts. In implementing the streaming I / P-frame if there is a lost / detainee package, this affects the entire screen, possibly causing a complete freeze of the screen for a certain period of time, or display other visual artifacts all over the screen. For example, if a lost / delayed packet causes the loss of an I-frame, the compressed data recovery apparatus will be no support member for all subsequent P-frames as long as there is no accepted new I-frame. If the P-frame is lost, it will affect the P-frames for the entire screen, which followed. Depending on how long it will be before an I-frame appears, this will have a longer or shorter visual impact. Using interleaved I / P-fragments, as depicted in Figures 7a and 7b, much less likely that a lost / delayed packet impact the entire screen since it will affect only the fragments contained in the affected packet. If each piece of data sent in a single packet, then if a packet is lost, it will only affect one fragment. Clearly, the duration of the visual artifact will depend on whether the packet is lost I-fragment, and if the P-fragment is lost, then, after a number of frames displayed I-fragment. But, given that different pieces of the screen updated by I-frames very frequently (perhaps every frame), even if one piece is affected on the screen, the other portions may not be affected. Further, if some event cause a loss of several packets at once (e.g., spike in power supply voltage near the line DSL, which briefly interrupts the data flow), then some of the fragments will be affected more than others, but because of the fact that some fragments will rapidly updated with the new I-fragment, then it will affect them only for a short time. Furthermore, with the implementation of streaming I / P-picture I-frames are not only the most critical frame, but the I-frames are extremely large, so if there is an event that triggers a reset / delayed packet, there is a strong likelihood that there will be affected I-frame (i.e., if any part of an I-frame is lost, it is unlikely that the I-frame can be restored at all) than a much smaller I-fragment. Because of all these reasons, the use of I / P-fragments results in a much smaller visual artifacts when packets are dropped / delayed than I / P-frames.

One embodiment is directed to reducing the impact of lost packets by a rational packing compressed packet fragments in TCP (transmission control protocol) packets or UDP (User Datagram Protocol). For example, in one embodiment, the fragments are aligned at packet boundaries whenever possible. 10A shows how fragments can be packaged within the series of packets 1001-1005 without implementing this feature. Namely, the cross pieces 10a on the packet boundary and packed inefficiently so that the loss of a single packet results in the loss of multiple frames. For example, if the package lost in 1003 or 1004, then lost three fragments that results in visual artifacts.

In contrast to Figure 10b shows a fragment of the packing logic 1010 for a rational packing of fragments within packets to reduce the effect of packet loss. First, the logic 1010 package fragment aligns fragments on the Limits of the package. Accordingly, fragments T1, T3, T4, T7, and T2 are aligned along the boundary of packets 1001-1005, respectively. Logic package fragment is also aimed at placing the fragments inside the packages most efficient way possible without crossing the boundaries of the package. Based on the size of each fragment fragments T1 and T6 are combined in one packet 1001, T3 and T5 are combined in one packet 1002 fragments T4 and T8 are combined in one packet 1003 fragment T7 added to packet 1004 and T2 fragment added to packet 1005. Thus According to this scheme, a single packet loss will result in the loss of no more than 2 fragments (rather than three fragments as shown in Figure 10A).

One additional advantage in respect to the embodiment shown in Figure 10b is that the fragments are transmitted in an order different from the order in which they are displayed within the image. Accordingly, if adjacent packets are lost due to identical events preventing transmission it will affect areas which are not near each other on the screen, creating a less noticeable artifacts in the display.

In one embodiment, the methods apply forward error correction (FEC) to protect certain portions of the video stream from channel errors. As is known in the art, FEC techniques, such as Reed-Solomon and Viterbi form and information data added for error correction to data to be transmitted over the communication channel. If an error occurs in the master data (for example, I-frame), you can use the FEC to correct this error.

FEC codes increase the data rate of the transmission, preferably so they are only used where they are most needed. If data is sent, which can not result in a very noticeable visual artifact, it may be preferable to not use FEC codes to protect the data. For example, a lost-P moiety which is directly in front of I-fragment create a visual artifact on the screen (i.e., on a fragment on the screen will not be updated) in only 1 / 60th of a second. The human eye does not notice a visual artifact. The further P-fragments are separated from the next I-fragment, the more noticeable is the loss of P-fragment. For example, if the cyclic pattern moiety is I-fragment, followed by 15 P-fragments before the I-section will be available again, then if the lost P-moiety immediately following the I-fragment, this will result in that that this fragment will show an incorrect image for 15 frame periods (at 60 frames / s is equal to 250 ms). The human eye is easy to notice the gap in the flow duration of 250 ms. Accordingly, the farther the P-fragment from the next new I-fragment (i.e., the closer P-fragments located in previous I-fragment), the more noticeable the artifact becomes. As previously discussed, though, in general, the closer P-moiety located to the previous I-fragment, the smaller the data for that P-fragment. Accordingly, P-fragments that follow the I-fragments are not only more important to protect against loss, but they are smaller in size. And, in general, the less data you want to protect, the smaller the FEC code is necessary for their protection.

Accordingly, as shown at 11a, in one embodiment, because of the importance of I-fragments in the video stream, only I-fragments provided FEC codes. Accordingly, FEC 1101 contains error correction code for I-fragment 1100 and FEC 1104 contains error correction code for I-fragment 1103. In this embodiment, no FEC is generated for the P-fragments.

In one embodiment, shown in 11b, FEC codes are also generated for P-fragments, which are most likely to cause visual artifacts if lost. In this embodiment, FEC codes 1 105 provide error correction codes for the first 3 P-fragments, but not for the subsequent P-fragments. In another embodiment, FEC codes are generated for P-fragment data size which is the smallest (which will tend to self-select P fragments before all occurring after I-fragment, whose protection is the most important).

In another embodiment, instead of sending FEC code with a fragment, the fragment is transmitted twice, each time in a different packet. If you lost / detained one package, it uses a different package.

In one embodiment, shown in 11C, codes for FEC 1113 and 1111 are generated for audio packets, 1110 and 1112, respectively, transmitted from the hosting service concurrently with the video. It is particularly important to maintain the integrity of the audio in the video stream, because the distorted audio (for example, clicking or hissing sounds) results in a particularly unpleasant practical experience of the user. FEC codes help to ensure representation of the audio content to the client computer 415 without distortion.

In another embodiment, instead of sending FEC code with audio data, the audio data is transmitted twice, each time in a different packet. If you lost / detained one package, it uses a different package.

Furthermore, in one embodiment, shown in 11D, codes 1121 and 1123 are used for FEC 1120 and 1122 commands the user input, respectively (e.g., pressing) transmitted upstream from the client 415 to hosting service 210. This is important because the loss of keystrokes or mouse movements in a video game or application may result in unpleasant practical experience of the user.

In another embodiment, instead of sending FEC code with user input command data, the user input command data is transmitted twice, each time in a different packet. If you lost / detained one package, it uses a different package.

In one embodiment, the hosting service 210 assesses the quality of the communication channel with the client 415 to determine whether to use FEC and, if so, what portions of the video, audio and user commands with which FEC should be applied. Qualification "quality" of the channel may include functions such as estimation of the packet loss, latency, etc, as described above. If the channel is particularly unreliable, then the hosting service 210 may apply FEC to all of I-fragments, fragments P-, audio and user commands. In contrast, if the channel is reliable, then the hosting service 210 may apply FEC only to audio and user commands, or may not apply FEC to audio or audio to the video, or may not use FEC. They can be used various other permutations of application FEC, although, as before, with the implementation of these underlying principles. In one embodiment, the hosting service 210 continually monitors the conditions of the channel and hence changes the FEC policy.

In another embodiment according to figures 4a and 4b, when a packet is lost / delayed, resulting in the loss of data fragment, or if, perhaps because of a particularly large packet loss, FEC can not correct the missing data fragment, then 415 client evaluates how many shots left before the new I-fragment will be accepted, and compares this number with the time signal back and forth from the client 415 to a host service 210. If the transmission time to and fro is smaller than the number of frames remaining until the expected receipt of the new I-fragment, then the client 415 sends a message to the service 210 by placing the information on the server to request the new I-fragment. This message is routed to the video compression apparatus 404, and instead of forming the P-fragment to fragment whose data had been lost, it generates I-fragment. Given that the system shown in Figures 4a and 4b, intended to ensure the transmission time to and fro, which is usually less than 80 ms, to obtain that the fragment corrected within 80ms (at 60 fps / s frame duration is 16.67 ms, respectively, the total period of the frames waiting time of 80 ms results in a correction of the fragment within the 83.33 ms, which is equal to 5 periods of the frame - a noticeable gap, but far less noticeable than, e.g. gap of 250 ms to 15 frames). When the device 404 is compression produces a I-fragment out of its usual cyclic order, if the I-fragment is the fact that the bandwidth of that frame exceeds the available bandwidth, then the device 404 compression delay cycles other fragments so that other portions taking P-fragments During this time frame (even though one moiety must typically take-I fragment within the frame), and then starting with the next frame continues normal cyclic repetition, and a fragment which normally I-fragment in a previous frame, takes I- fragment. Although this action briefly delays the phase of the cyclical repetition of R-frame, it is usually not visually noticeable.

Implementation despreader / recovery unit of the compressed video and audio

Figure 12 illustrates one particular embodiment which uses multi-core and / or multi-processor 1200 for compressing eight fragments in parallel. In one embodiment, a computer system with a quad-CPU Xeon, dual processor, which operates with a clock speed of 2.66 GHz or higher, with each core device implements compression H.264 × 264 open source as an independent process. However, there could be various other configurations of hardware / software, although still with the performance of the underlying principles. For example, each of the CPU cores can be replaced with H.264 compression device, realized in the FPGA. In the example shown in Figure 12, cores 1201-1208 are used for simultaneous processing of I-slices and P-fragments as the eight independent threads. As is well known in the art, modern multi-core and multi-processor computer systems in nature can perform multi-threading when integrated with multi-threaded operating systems, for example, Microsoft Windows XP Professional Edition (or a 64-bit or 32-bit) and Linux.

In the embodiment shown in Figure 12, since each of the 8 cores is responsible for just one piece, it operates largely independently from the other cores, each separate instance takes × 264. Map seizure DVI-based PCI Express × 1, such as Sendero Video Imaging IP Development Board on Microtronix, Oosterhout, the Netherlands, is used to capture uncompressed video with a resolution of 640 × 480, 800 × 600 or 1280 × 720, and FPGA on the map uses direct memory access (DMA) to transfer the captured video over DVI to system RAM. Fragments are arranged in the layout 1205 4 × 2 (although they are illustrated as square fragments, in this embodiment, they have a resolution of 160 × 240). Each instance × 264 configured to compress one of the 8 pieces of 160 × 240, and they are synchronized such that, after initial compression of I-fragment, each core enters into a cycle, each frame is out of phase with the other, to compress one I-fragment followed by seven P-fragments, illustrated in Figure 12.

Each frame period, the resulting fragments are combined in the compressed packet stream, using the techniques described above, and then compressed fragments are transmitted to the target client 415.

Although not shown in Figure 12, if the data rate of the combined 8 fragments exceeds a predetermined peak data rate 941, then all 8 x264 processes are deposited onto a number of frame periods, which is required before until all the data of said united 8 fragments can not be transmitted.

In one embodiment, the client 415 is implemented as software on the PC, on which are executed 8 copies FFmpeg. The receiving process takes 8 fragments, and each fragment is routed to an instance of FFmpeg, which restores compressed fragment and renders it in the appropriate location of the fragment displayed 422.

The client 415 receives an input keyboard, mouse or game controller device drivers from entering the PC and sends it to the server 402. The server 402 further applies the received data input and applies them to a game or an application running on the server 402, which is a PC, running under Windows using a dual-core (Core Duo) CPU Intel 2,16 GHz. Server 402 further displays the new frame and displays it through its DVI output, or from an on-board graphics system, or through a DVI output card PCI 8800GTX NVIDIA.

At the same time the server 402 outputs the audio outputted game or applications through its digital audio output (eg, S / PDIF), which is connected to a digital input on the twin quad PC-based Xeon, which sells video compression. Audio compression apparatus with Vorbis open source is used to compress the audio simultaneously with the video using any core that is available for the processing flow. In one embodiment, the core, which compresses its fragment, first executes the audio compression. Compressed audio is then transmitted along with the compressed video and restored to the client 415 using a device restore compressed audio Vorbis.

Distribution Center server hosting service

Light through glass, such as the light conductor extends at a rate equal to a fraction of the speed of light in vacuum, and therefore, can be determined the exact speed of light in a light guide. But in practice, taking into account the time delays due to routing, transmission inefficiencies and other losses, the authors observed that the optimum value of the waiting time on the Internet reflect transmission speeds closer to 50% the speed of light. Accordingly, an optimum transmission time to and fro over a distance of 1000 miles (1600 km) is about 22 ms and an optimum transmission time to and fro over a distance of 3000 miles (4800 km) is approximately 64 ms. Accordingly, one server on one coast of the United States, is too far away for the customer on the other coast (which may be at a distance of 3000 miles (4800 km)) with the required standby time. However, as shown in 13A, if the center server 1300 hosting service 210 is located in the center of the United States (for example, in Kansas, Nebraska, etc.) so that the distance to any point in the continental United States is about 1,500 miles (2400 km) or less, the transmission time to and fro on the Internet can only equal 32 ms. 4B note that, despite the fact that the values ​​of the waiting time in the worst case, admissible for (compound) User ISP 453, equal to 25 ms, in general, we observed a time-out value is closer to 10-15 ms of cable systems dial-up and DSL. Furthermore, according to Figure 4b it is assumed that the maximum distance from the user area 211 to the hosting center 210 is 1000 miles (1600km). Accordingly, in a typical used time of signal transmission to and fro (in a compound) User ISP 15 ms and a maximum distance via the Internet 1500 miles (2400 km) for the transmission time to and fro 32 ms total transmission time to and fro from when the user actuates input device 421 and observes the display 422 the answer is equal 1 + 1 + 15 + 32 + 1 + 16 + 6 + 8 = 80ms. Accordingly, transmission over the Internet at distance 1500 miles (2400 km), and can usually be obtained response time of 80 ms. It can provide any territory by short enough time, 453 standby (for connection) by ISP in the continental US for access to one server center, which is located in the center.

In another embodiment, shown in Figure 13b, server centers, HS1-HS6, service hosting 210 strategically located throughout the United States (or any other geographical region), and some large server hosting service centers are located close to the centers of high population density (eg , HS2 and HS5). In one embodiment, the server centers HS1-HS6 exchange information via a network 1301 which may be the Internet or a private network or a combination of both. With respect to a plurality of server center services can be provided with less waiting time for users who have time, 453 standby ISP user is large.

Despite the fact that the distance across the Internet is certainly a factor that increases the signal back and forth through the Internet, sometimes other factors play a role, which generally are not associated with latency. Sometimes a packet stream is routed through the Internet to a remote location and back again, resulting in latency due to the long link. Sometimes the equipment to route located on the way (the signal) is not operating properly, resulting in transmission delay. Sometimes the way overloaded with traffic, which introduces a delay. And sometimes there is a failure that does not allow the ISP to route the user to the destination. Accordingly, despite the fact that the general Internet usually provides connections from one point to another with a fairly reliable and optimal route and latency that is largely determined by distance (especially in the compounds of telecommunication, which result in routing outside of the local area Users) such reliability and latency in any case is not guaranteed and often can not be obtained from the area of ​​the user at the destination in the general Internet.

In one embodiment, when client 415 the user first connects to the service 210 hosted playing video games, or use an application, the client communicates with each of the server centers HS1-HS6 hosting service available after startup (e.g., using the methods described above). If the waiting time is small enough for a particular connection, then that connection is used. In one embodiment, the client is communicating with all server center hosting services, or a subset, and the selected server center with the compound with the lowest latency. The client may select the service center with the compound with the lowest latency and service centers may identify the service center with the compound with the lowest latency and provide this information (e.g., in the form Internet address) to the client.

If a specific server center hosting service is overloaded and / or user application or game while waiting for connecting to other less loaded server center hosting service is valid, the client 415 can be redirected to another server center hosting services. In this situation, a game or an application, executed by the user, shall be suspended at the server 402 in the congested heart of the server and user data on the status of applications or games are transmitted to the server 402, located in the center of the other server hosting service. After that game or application is resumed. In one embodiment, the hosting service 210 waits until until the game or application has reached the point of suspension of natural (e.g., between levels in a game, or after the user initiates operation "preservation" in Appendix) for said transmission. In another embodiment, the hosting service 210 waits until until user activity ceases for a predetermined period of time (e.g., 1 minute) and then initiates said transfer at this time.

As described above, in one embodiment, the hosting service 210 subscribes to bypass service 440 of Figure 14 to the Internet to provide their customers guaranteed latency. Internet bypass services, as used herein, are services that provide private network routes through from point to point (points) on the Internet with guaranteed characteristics (e.g., latency, data rate, etc.). For example, if the service 210 hosting receives a lot of traffic from people using the service DSL AT & T, proposed to San Francisco, instead of routing are in San Francisco central office AT & T service 210 hosting can rent a private data connection with a large-capacity provider Service (perhaps at the AT & T company, either directly or from another provider), located between the San Francisco central office and one or more server farms to service 210 hosting. Then, if the routes of all server centers HS1-HS6 hosting services through a common Internet to the user, located in San Francisco, using DSL AT & T as a result lead to too much waiting time, instead of them can be used in a private data connection. Although private data connections are generally are more expensive than the routes through the general Internet, so long as they are a small percentage of compounds service 210 hosting service to users, the overall impact on the cost will be low, and users will experience more consistent service experience.

Server centers often have two levels of backup power in the event of a failure in the power supply system. The first level is usually a battery backup (or an alternative that is in readiness for the immediate use of an energy source such as a flywheel, which is maintained in working order and connected to the generator), which immediately provides power when a failure occurs in the supply line, and maintains a server center. If the fault in the power supply is short-lived and a rapid return of the supply line (for example, one minute), the battery - it is all that is needed to maintain a central server running. However, if a failure occurs in the power supply system for a longer period of time, usually trigger generators (e.g., diesel), which take over the functions of the batteries and can run as long as they have fuel. Such generators are extremely expensive because they must be able to derive such power, which usually gets the server center of the supply line.

In one embodiment, each of the services HS1-HS5 hosting jointly uses user data so that if one server center failure occurs in the power system, it can pause the games and applications that are in process (execution), and then transmits the data of the game or application state from each server 402 to servers 402 located at other server centers, and then will notify the client 415 of each user, and instructs it to send information on the new server 402. Given that such situations occur infrequently, it may be acceptable transfer user server center hosting services, which can not provide the optimum waiting time (ie, the user simply has to put up with higher latency for a failure in the power supply), which provides much more opportunities for the transfer of users. For example, taking into account the different time zones in the United States on the East Coast users can go to bed at 23:30, while the people on the West Coast at 20:30 starting to make the most of video games. If at this time a failure occurs in the power supply system in the server center hosting services, located on the West Coast, you may not have a sufficient number of servers 402 located on the West Coast in the other server hosting service centers to manage all users. In this situation, some users may be transmitted to the server hosting service centers, located on the East Coast, where there are available servers 402, and the only consequence for users is increasing waiting times. After the transfer of users from the server center, which left without electricity, the center server can start in due course off their servers and equipment so that all equipment has been turned off to discharge the battery (or other backup power plug immediately). Consequently, the server center can avoid the cost of the generator.

In one embodiment, during periods of heavy load hosting service 210 (or as a result of peak user load, or due to failure of one or more server farms) is transmitted to other users of the server centers based on the requirements to the waiting time of the game or application they are using. Accordingly, users who use games or applications that require low latency, preferred (the provision) of available connections to servers with low latency with limited power.

Signs hosting service

Figure 15 illustrates an embodiment of components of a server center for hosting service 210 that is used in the following description features. As is the case with the hosting service 210 illustrated in Figure 2a, the components of this server center controls and coordinates the operation control system 401 hosting service 210, unless otherwise specified.

Incoming traffic on the Internet in 1501 from 415 customers user is directed to the incoming routing 1502. As a rule, the incoming traffic on the Internet in 1501 is included in the server center via the high-speed fiber optic connection to the Internet, but it will be enough any means network connection to meet the requirements of bandwidth, reliability, and low standby time. Inbound routing is a system 1502 network (a network may be implemented as a network of Ethernet, a network of fiber-optic channels, or through any other means of transport) routing switches and servers that support the switch that receives incoming packets and routes each packet to the appropriate server application 1521-1525 / games ("app / game"). In one embodiment, the package which is delivered to a particular server application / game is a subset of data received from the client and / or may be translated / changed by other network components (e.g., including network components such as gateways and routers) in the processing center and storage. In some cases, packets are routed to more than one server 1521-1525 at a time, such as if the game or application is executed in parallel simultaneously on multiple servers. 1511-1512 Disk Array RAID type is connected to the network in 1502 with inbound routing so that the servers 1521-1525 applications / games may be read from disk arrays 1511-1512 RAID type and write to them. Moreover, disc type RAID array 1515 (which may be implemented as a plurality of types of RAID disk arrays) is also connected to the inbound routing 1502 and data from the disk array of RAID type 1515 can be read from the server 1521-1525 application / game. Inbound routing 1502 may be implemented in a wide range of network architectures of the prior art, including a tree structure of switches, with the inbound traffic in 1501 on the Internet at its root, a honeycomb structure, connecting all the various devices, or as a set of interconnected subnetworks with traffic focused internal communication between devices separated from traffic, focused, among other devices. One type of network configuration is a SAN, which, although it is generally used for memory devices can also be used for normal high-speed data transfer between devices. In addition, servers 1521-1525 application / game everyone can have multiple network connections to the input routing of 1502. For example, the server 1521-1525 may have a network connection to the subnet connected to the disk arrays 1511-1512 type of RAID, and other network connection to the subnet connected to other devices.

Servers 1521-1525 application / game may all be configured the same, some - in different ways, or all - differently, as previously described in relation to servers 402 in the embodiment shown in Figure 4a. In one embodiment, each user, when using the hosting service is typically (in use), at least one server 1521-1525 application / game. For ease of explanation, assume that the user uses a 1521 application / games, but multiple servers can be used by a single user and multiple users can share a single server 1521-1525 application / game. User input control signal sent from client 415 as previously described is received as an incoming traffic on the Internet 1501, and is routed through inbound routing 1502 to the server 1521 the application / game. Server 1521 applications / games using user input control signal as an input control signal to the game or an application running on the server and calculates the next frame of video and audio associated with it. Server 1521 application / game then outputs uncompressed video / audio in 1529 shared video compression 1530. Server application / game may output the uncompressed video via any means, including one or more compounds of the technology Gigabit Ethernet, but in one embodiment the video is output via connection DVI, and the audio and other compressed data and information about the state of the communication channel output from compound a universal serial bus (USB).

Shared video compression 1530 compresses uncompressed video and audio server 1521-525 apps / games. Compression may be implemented entirely in hardware, or in hardware executing software. For each server, 1521-1525 application / game may (be) allocated compression device, or if the compression device is fast enough, the device can be used for compressing video / audio compression of the 1521-1525 multi-server applications / games. For example, at 60 frames / sec video frame time is equal to 16.67 ms. If the compression apparatus may compress a frame in 1 ms, then it is possible to use a compression device for compressing the video / audio server 16 1521-1525 application / game by receiving input from one server after another, wherein the device maintains the state of each compression process of a video compression / audio and context switches to the extent that it cyclically switches between the streams of video / audio from the server. This results in significant savings in hardware costs for compression. Since different servers will be completing frames at different times, in one embodiment, the device resources compression are in a shared pool 1530 with shared storage means (e.g., RAM, Flash) for storing the state of each compression process, and when the frame server 1521 -1525 finished and ready for compression, the control means determines which compression resource is available at that time, provides the compression resource state of the server and the compression frame of uncompressed video / audio to compress.

Note that part of the state of compression of each server includes information about the compression, such as the recovered data frame buffer of the previous frame, which can be used as a supporting element for P-fragments, the resolution of the video output, compression quality, the mosaic structure, the allocation of bits for each fragments, compression quality, audio format (eg stereo, surround sound, AC Dolby® 3). But the compression process state also includes information about the state of the communication channel with respect to the peak data rate 941 and the fact whether the output at the moment the previous frame (as illustrated in Figure 9b) (and as a result the current frame should be ignored), and possibly whether there are channel characteristics which should be considered in the compression, such as excessive packet loss, which affect decisions regarding the compression (e.g., in terms of the frequency of I-fragment, etc). As the peak rate of 941 data or other channel characteristics change over time, as determined by the server 1521-1525 application / game that supports data monitoring each user sent from the client 415, the server 1521-1525 application / game sends the relevant information shared hardware compression 1530.

Shared hardware compression in 1530 and broken up into packets compressed video / audio using means such as the means described above, and if required, by application of codes FEC, duplication of certain data, or other measures for the appropriate permit the adoption of the client 415 video stream / audio data and restoring the compressed data with the highest possible quality and reliability.

Some applications, such as described below, require that the video / audio of the server 1521-1525 application / game is accessible to many resolution (or a variety of other formats) at the same time. If the server 1521-1525 application / game so notifies the shared resource 1530 hardware compression, the uncompressed video / audio server 1529 of 1521-1525 application / games will be compressed at the same time in different formats with different resolution and / or with different structures fixes / packages. In some cases, some resources compression can be shared by a plurality of compression processes, compressive identical video / audio (e.g., in many compression algorithms, there is a step in which the image is scaled to a plurality of sizes before applying compression. If desired output images of different sizes, this step can be used for the maintenance of several compression processes simultaneously. In other cases, the required format for each individual resource compression. In any case, video / audio compression in 1539 with all required different resolutions and all the different formats required for the server 1521-1525 applications / games ( be it one or many) displayed simultaneously in the outbound routing 1540. In one embodiment, the output of the compressed video / audio 1539 is carried in UDP format, respectively, it is a unidirectional stream of packets.

Network 1540 Outbound Routing includes a set of switches, and routing servers, which direct each stream of compressed video / audio to the user (s) who (th) it is intended, or to other destinations via the interface outgoing traffic in 1599 of Internet (which is usually connected a fiber optic interface with the Internet), and / or back to the delay buffer 1515, and / or back to the inbound routing 1502, and / or through a private network (not shown) for video distribution. Note that (as described below) outgoing routing 1540 can output the video / audio stream to multiple destinations simultaneously. In one embodiment, this is implemented using a multicast protocol Internet (IP), which is transferred to the flow of UDP, for transmitting a stream to a plurality of destinations simultaneously, and this is repeated broadcast routing switchers and servers in 1540. Outbound Routing multiple destinations Broadcast can be 415 clients multiple users (available) through the Internet, many servers 1521-1525 applications / games (available) through the routing of incoming 1502 and / or one or more buffers 1515 delays. Accordingly, the output of the 1521-1522 server is compressed in one or a plurality of formats, and each compressed stream is sent to one or multiple destinations.

Furthermore, in another embodiment, if multiple servers 1521-1525 application / game are used simultaneously by one user (e.g., in a parallel processing configuration to create the 3D output of a complex scene) and each server outputs a part of the resulting image, the video output of multiple servers 1521 -1525 can be combined shared hardware compression in 1530 in a combined frame, and from that point on, it is treated as described above, as if it was coming out of one server 1521-1525 application / game.

Note that in one embodiment, a copy (in at least the resolution of the video being viewed by the user, or more) of all video generated servers 1521-1525 application / game, stored in the buffer 1515 delays of at least some number of minutes (15 minutes in one embodiment). This allows each user to "rewind" the video of each session to view the previous act or acts (in the case of the game). Accordingly, in one embodiment, for each output 1539 stream of the compressed video / audio routed to a user client 415 is also implemented in the multicast buffer 1515 delays. When the video / audio stored in the buffer 1515 delays directory delay buffer 1515 provides a cross reference between the network address of the server 1521-1525 application / games, which is the source of the detainee video / audio, and place in the delay buffer 1515, which delayed the video / audio may be found.

Game in real time, immediately outputted to the screen, with the possibility of immediate gaming

Servers 1521-1525 apps / games are not only used for the execution of the application or video game to the user, but they can also be used to create user interface applications for 210 hosting service that supports navigation in 210 hosting service and other features. The screen shot of this application user interface is shown in Figure 16, the screen "Game Finder" ("finder of the game"). This particular user interface screen allows a user to watch 15 games, which are carried out in real time (or delayed) by other users. Each of the "miniature" video windows, such as 1600 is a window with video in real time in a motion showing one the video from one user to the game. The form shown in miniature, can be view identical to that which the user sees, or it may be delayed a view (for example, if the user is a fighting game, the user may require that other users can not see where he is, and he can decide to hold any kind of gameplay it for some period of time, say 10 minutes). Type can also be a field of view of the camera of the game that is different from any kind of user. Through menu selections (not shown in this example), the user can select a sample for simultaneous viewing games based on various criteria. As a small sampling of exemplary selections the user may select a random selection of games (such as shown in Figure 16), all of one kind of games (all of which are different players), only the high-flying game players, players at a given level in the game, or lower Players listed (for example, if the player is exploring base), players who are "friends" (or rivals), the game in which the highest number of viewers, etc.

Note that, in general, each user decides whether other watch his game or application, and if so, which of them and when can watch it if you can watch it only with a delay.

Server 1521-1525 application / game that generates the user interface screen shown in Figure 16, 15 requests transmission line video / audio by sending a message to the server 1521-1525 application / game for every user, from whom he asks for the game. The message is sent via the inbound routing 1502 or another network. The message includes the size and format of the requested video / audio and identifies the user viewing the user interface screen. This user may decide to select "privacy" ("confidentiality") and does not allow any other users to view video / audio of his game (or with a view point or from another point of view), or as described in the previous paragraph, the user may decide to provide the ability to view video / audio of his game, but the delay to watch video / audio. Server 1521-1525 application / game user receives the request and agrees to provide the ability to view its video / audio, sends a confirmation to the requesting server, and it also notifies the shared hardware compression in 1530 the need to create an additional stream of compressed video in the requested format or requested screen size (assuming the format and screen size is different from those already formed), and it also specifies the destination for the compressed video (i.e., the requesting server). If the requested video / audio only delayed, the requesting server 1521-1525 application / games notify and he asks delayed video / audio buffer 1515 delays by searching the location of the video / audio directory is located in the buffer 1515 delays, and network 1521-1525 addresses server applications / games, which is the source of the detainee video / audio. After the formation and processing of these requests up to 15 streams of video thumbnail size in real-time routed from the outgoing to the incoming routing 1540 routing 1502 Server 1521-1525 application / game that forms the screen of the user interface, and restored and displayed by the server. Delayed video / audio streams may be too large a screen size, and if so, the server 1521-1525 application / game restore compressed stream and reduce the size of the video streams to thumbnail. In one embodiment, requests for audio / video are sent (and it controls them) in a central service "control", such hosting service control system according to Figure 4a (not shown in Figure 15), which further forwards these requests to the appropriate server 1521 -1525 apps / games. Furthermore, in one embodiment, the request may not be required because the thumbnails "pushed" to the clients of those users who have given permission for it.

Audio of the 15 games, all mixed at the same time can create a cacophony of sounds. The user may choose to mix all of the sounds together in this way (perhaps just to get the feeling of "noise" created by all the action viewed), or the user may decide to listen to only the audio from one game at a given time. Selecting one game is performed by moving the yellow frame 1601 select the track in this game (move the yellow frame can be performed by using the arrow keys on the keyboard, moving the mouse, you move the joystick or by pressing the directional buttons on another device such as a mobile phone) . After you select a game from the game play audio only. In addition, the 1602 displays information about the game. In the case of this game, for example, the logo of the publisher ("EA") and the logo of the game "Need for Speed ​​Carbon" ("Need for speed"), and orange horizontal bar indicates relative terms the number of people leading this game, or watch her in this a particular moment (in this case a lot, so the game is a "hot"). In addition, to provide a "statistical data" ("Stats") indicating that there are 145 players who are actively leading 80 different copies of Need for Speed ​​Game (ie it can be carried out either as a single player game or a game with several participants ) and there are 680 viewers (of which this user is). Note that these statistics (and other statistics) are going to the control system 401 hosting service and stored in disk arrays, RAID type for the 1511-1512 record books of the service 210 hosting and for proper billing of customers and payments to publishers that provide content. Some statistics are recorded due to the actions of management services 401, and some are reported in 401 separate server management services 1521-1525 application / game. For example, Server 1521-1525 application / game, executing an application Game Finder (search tool game) sends messages to the control system 401 hosting service where games are viewed (and when they stopped to view) so that it can update the statistics on how many games displayed on the screen). Some statistics are available for the front-end user applications, for example, for this application Game Finder (search tool of the game).

If the user clicks on the activation button on their input device, he sees that the thumbnail video in the yellow frame increases, while it remains live to full screen size. This effect is illustrated in Figure 17. Note that video window 1700 has increased in size. To accomplish this, the impact of the application server 1521-1525 / 1521-1525 game asks the server application / games, executing the selected game, to obtain a copy of the video stream for the size of the entire screen (with a resolution of 422 by the display) game, routed him. Server 1521-1525 application / games, executing the game notifies shared hardware compression device 1530 that a copy of the miniature size of the game is no longer required (if another server 1521-1525 application / game does not require a thumbnail), and then he prescribes to He sent a copy of the video to full screen size in the 1521-1525 application server / video games with plowing. Username leading game may or may not have a display with a resolution of 422 identical to the resolution of the display by enhancing the image of the game. Further, other viewers of the game may or may not have a display 422 with a resolution identical to the resolution of the user's display, magnifies the image of the game (and may have different audio playback means, e.g. stereo or surround sound). Accordingly, a shared hardware device 1530 compression determines what forms you have the appropriate stream of compressed video / audio that meets the requirements of the user requesting the said video / audio stream, and if it already exists, it shall notify the outbound routing of 1540 so that it is routed up stream server 1521-1525 application / game with zooming video, and if (it) does not compress another copy of the video, which is suitable for the user (the) issues a command to send outbound routing said flow back into the incoming routing server 1502 1521-1525 application / games, zooming video. This server is now receiving a full screen version of the selected video, and gradually restores it scales it to full size.

Figure 18 illustrates how the screen looks after the final game zoom up to full screen and the game is shown with the full resolution of the display 422 of the user as indicated by the image pointed to by arrow 1800. The server 1521-1525 application / game. executable application search engine game, sends messages to other servers in the 1521-1525 application / game that provided a thumbnail, that they are no longer needed, and messages to the server 401 Management service hosting that other games are no longer visible. At this moment the only image that it generates, is the imposition in 1801 of another graphic on the top of the screen that provides information and controls for the user menu. Note that as you move this game, the audience rose to 2 503 spectators. With so many spectators there must be a lot of spectators with displays 422 resolutions that are identical or similar (1521-1525 each server applications / games can scale the video to adjust compliance).

Since the image play a game with several participants, the user may decide to accede to the game at a time. 210 Hosting Service may provide, or may not allow the user to join the game for various reasons. For example, you may have to pay to play the game, and he decided not to pay, the user may not have sufficient ranking to join this particular game (for example, it can not compete with other players), or connect to the Internet user can does not have enough low latency to enable the user to play the game (for example, there is no limit on the waiting time for game viewing, accordingly, a game that is carried away (of course, on another continent), you can watch without worrying about time-out, but to play the game, the wait time should be small enough so that the user (a) enjoyed the game and (b) on an equal footing with other players who may have connections with less waiting time). If a user is provided with the opportunity to play the game, the server 1521-1525 application / game that provides a user interface by Game Finder (search engine of the game) sends a request to a server 401 hosting service management initiated (ie, a position and launched) server 1521-1525 application / game that is configured according to the specific conduct of the game, to download games from the disk array type 1511-1512 RAID, and then the control server 401 hosting service issues a command to the input routing 1502 for transmitting control signals from the user to the game server application / games, which currently provides hosting games, and it instructs the shared hardware compression in 1530 to switch from the video / audio compression from an application server / game which performs application hosting Game Finder (search engine of the game) to a video compression / audio from the server applications / games, which currently provides hosting games. Staff clock pulse service applications / games Game Finder (search tool game) and a new server application / game that provides hosting games, not synchronized, and as a result, probably, there is a time difference between the two clock pulses. Since the shared hardware 1530 video compression video compression starts after server 1521-1525 application / game completes the video frame, the first frame of a new server can be completed before the (pass) full frame period of the old server, which can occur before It is completed transmission of a previous compressed frame (e.g., consider a period 992 for transmission 9b: if uncompressed frame 3 963 were completed floor frame period before, it would have faced transmission period 992). In such a situation shared hardware 1530 video compression ignores the first frame of the new server (for example, just as ignored 974 frame 964 4) and client 415 holds the last frame of the old server to the additional frame period, and shared hardware 1530 video compression begins to compress video frame period of the new server applications / games, which carries hosting games. Visually, the user move from one server applications / games on the other will be uninterrupted. 401 server management hosting service then notifies the game server 1521-1525 application / game that carried hosting Game Finder (search tool of the game), to switch to the standby state, as long as he does not need again.

Thereafter, the user can play the game. Exceptional is that it seems that the game can be carried out immediately (as it is downloaded to the game server 1521-1525 applications / games from the disk array RAID type 1511-1512 at a rate of gigabits / second), and the game is loaded on the server, fully fit for the game, together with the operating system fully configured to play a perfect driver, the configuration of the registry (for Windows), and the other applications that may interfere with the functioning of the game, the server is not executed.

Also, as we move the user in the game, each of the segments of the game is loaded into the server at gigabit speed / second (i.e., 1 gigabyte loads in 8 seconds) from the disk type RAID array 1511-1512, and because of the vast storage capacity of disk 1511-1512 array type RAID (since it is a shared resource among many users, it can be very large and still be cost-effective), setting or adjusting the geometry of another segment of the game can be pre-computed and stored on disk array 1511-1512 RAID type and loaded very quickly. Furthermore, due to the fact that the hardware configuration and computational capacity of each server 1521-1525 application / game known, may be previously performed calculations (using) vertex and pixel shaders.

Accordingly, the game starts almost immediately, it is executed in an ideal environment, and subsequent segments are downloaded almost immediately.

But in addition to these advantages, the user can watch others lead the game (via Game Finder (search tool of the game), as previously described, and other means), as well as to decide on whether the game is interesting, and if so, examine the recommendations of when observing the other. And the user can test the demo version of the game immediately, and there is no need to wait for a long download and / or install, and the user can play the game immediately possible to test for a small fee, or on a long term basis. And the user can play the game on the PC Windows, Macintosh, on television, at home, while traveling or even on a mobile phone through a wireless connection with a sufficiently low latency. And that's all you can do and still never have a physical copy of the game.

As previously mentioned, the user can decide not to provide others (users) the ability to view its gameplay, to provide the ability to view his game after a delay, to provide the ability to view his game to selected users, or enable viewing of its games to all users. No matter what the video / audio will be stored in one embodiment, for 15 minutes in a delay buffer 1515, and the user can "rewind" and view his prior game, and pause, play it back slowly, fast forward, rewind, etc. as he can do when watching TV with a digital video recorder (DVR). Although in this example, the user plays a game, means identical "DVR" is available if the user is using an application. This may be useful when viewing prior work and in other applications as detailed below. Further, if the game was designed with winding means based on the information about the state of the game so that the camera view can be changed, etc., then it means "3D DVR" will also be supported, but it will require the game to be designed with its support. Means "DVR" using a buffer 1515 delays works with any game or application, of course, limitations to the video that is generated during the use of the game or application, but in the case of games with the means DVR 3D user can control the "through-span" (fly through ) in the 3D segment previously held and to record the resulting video buffer 1515 delays, and to record the state of the game for the segment of the game. Accordingly, the concrete "through rapid movement" is recorded as compressed video, but since the state of the game is also recorded, the other through the rapid movement for an identical segment of the game then maybe.

As described below, each user has the service of hosting 210 page user (User Page), where they can post information about themselves and other data. Among other things, users are allowed to video segments of gameplay (gameplay), which they retained. For example, if the user has overcome a particularly difficult a tough test in the game, the user can "rewind" to the point directly in front of his great achievement in the game, and then issue a command to the service 210 hosting to save the video segment of a length (eg, 30 seconds) on the user's (User Page) of the user to view other users. To achieve this, it is - just a matter of server 1521-1525 apps / games that the user is using, take a video stored in the buffer 1515 delays, 1511-1512 disk array RAID type, and then make a reference to this segment of the video on the page the user (User Page) of said user.

If the game has means 3D DVR, as described above, then the game state information required for the 3D DVR can also be recorded by the user and made available on the user's (User Page) of said user.

In case the game is designed with the possibility of having "visitors" (ie users who can navigate through the world of 3D and watch the action without it), along with active players, the application Game Finder (search tool game) enables users join games as visitors, as well as players. In terms of implementation, the system 210 hosting there is no difference whether the user is a visitor or an active player. The game is loaded on the server 1521-1525 application / game, and the user controls the game (for example, by controlling a virtual camera that looks to the world). The only difference is the user experience with the game.

The joint work of multiple users

Another feature of the service 210 hosting is the ability to work together multiple users when they view the video in real time, even when using a significantly different devices for viewing. It is useful in the conduct of the game and using the application.

Many PC and mobile phones are equipped with video cameras, and there are means for performing video compression in real time, in particular when the image is small. In addition, commercially available small cameras which can be connected to the television, and not difficult to implement real-time compression, or in software or using one of many hardware compression devices to compress the video. Furthermore, in many PC and all mobile phones exist microphones and earphones commercially available with microphones.

Such cameras and / or microphones, combined with local means of video / audio compression (in particular, using the methods of video compression, low-latency, described herein) allows a user to transmit video and / or audio from the area 211 the user service 210 hosting with control data input device. When such methods are used, it is possible to obtain the means illustrated in Figure 19: The user can output your video and audio to the screen in 1900 in the application or game for another user. This example is a multi-player game, where teammates jointly participate in a car race. Video / audio user can selectively view / listen only to his teammates. And since virtually no waiting time by using the methods described above, players can talk to each other or show a gesture to each other in real-time without perceptible delay.

The integration of video / audio is performed by the receipt of the compressed video and / or audio from the camera / microphone user 1501 how incoming traffic on the Internet. Then in 1502 a member of the routing routes the video and / or audio to game servers 1521-1525 application / game that will provide the ability to view / listen to video and / or audio. Further, users of the gaming servers 1521-1525 application / game that decided to use video and / or audio, restore it and integrate as required to display within a game or application, for example, as shown in the position in 1900.

In the example in Figure 19 shows how such co-operation is used in the game, but such collaboration can be a powerful tool for applications. Consider a situation where a large building designed for New York architects in Chicago for the developer, who is in New York, but the decision involves a financial investor who travels and is located at the airport in Miami, and the decision must be made about certain elements of the building project on the basis of how it is consistent with the buildings located around it, so that investors and developers have been satisfied. Assume that the architectural firm has a high resolution monitor with a camera connected to a PC in Chicago, the developer has a laptop with a camera in New York, and the investor has a mobile phone with a camera in Miami. Architectural firm can use the service 210 hosting for a powerful application for architectural design with the possibility of a highly realistic 3D visualization and the ability to use a large database of buildings in New York, as well as a database designed buildings. The application of architectural design is performed on one or if you need more processing power, several of the servers 1521-1525 application / game. Each of these three users located in different places, connected with the service 210 hosting, and each has a simultaneous view of the output application of architectural design, but the size of it will be amended accordingly shared hardware compression in 1530 for the device and the characteristics of the network connection that is every user (e.g., the architectural firm can see the images of 2560 × 1440, 60 frame / s, through a commercial Internet connection of 20 Mbit / s, the developer in New York may see an image of 1280 × 720, 60 frames / s, the DSL connection 6 Mbit / s on his laptop, and the investor can see the image of 320 × 180, 60 frames / sec, on a cellular data connection of 250 kbit / s on your mobile phone. Every participant hears the voice of the other participants (holding conference calls handled by any of the a plurality of widely available software packages for conference calls in the server (s) 1521-1525 application / game) and by activating the button on the device user input the user can output video of himself on the screen with your local chamber. In the course of the meeting of architects will be able to show how the structure looks like when they spin it and then quickly moved along it to another building in the area with an extremely photorealistic visualization of 3D, and all participants see an identical video with a resolution of the display of each participant. It does not matter that none of the local device, used by any party can not handle 3D animation with realism, not to mention the loading or storing huge database required for imaging the surrounding buildings in New York. From the point of view of each of the users, despite the fact that they are located at a distance, and in spite of various local devices they simply will have a seamless experience with an incredible degree of realism. And when one participant is required to be a person to be seen with a better transfer of his emotional state, he can do it. In addition, if either the developer or investor is required to obtain control of the architectural program and use their own input device (regardless of whether it is a keyboard, mouse, keypad or touch screen), they can do it, and she will answer any perceptible time waiting (assuming that his network connection is acceptable latency). For example, in the case of a mobile phone if the mobile phone is connected to a WiFi network at the airport, it will have very low latency. But if he uses a cellular data network, currently available in the US, it is likely to experience a significant delay. However, for most purposes of the meeting, when the investor watches the architects manage rapid movement along the building, or for video teleconferencing, even while waiting for the mobile to be acceptable.

Finally, at the end of the joint conference developers and investors to make comments and to disconnect from the service host, in the architectural firm will be able to "rewind" the video conference, which was recorded in a buffer 1515 delays and view comments, facial expressions and / or actions relating to the 3D model of the building, made during the meeting. If there are specific segments that they want to save, then these segments video / audio can be transferred from the buffer 1515 delays in 1511-1512 disk array RAID type for archiving and playback later.

Furthermore, in terms of cost, if the architects need only use the computing power and the large database of New York City for a 15 minute conference call, they need only pay for the time during which the resources are used, rather than possessing powerful working stations and the purchase of an expensive copy of a large database.

Extensive public video services

210 hosting service provides an unprecedented opportunity for the installation of extensive public video services on the Internet. Figure 20 illustrates an exemplary user page (User Page) for the player in the service 210 hosting. As is the case with the application Game Finder (search tool games), User Page (User) is an application that runs on one of the servers 1521-1525 application / game. All miniatures and video window on this page is constantly moving video is displayed (if the segments are short, they are cycled).

Using a video camera or video through the discharge member (whose user name is "KILLHAZARD") can put a video of himself in 2000 that can be viewed by other users. Video is stored in the disk array type 1511-1512 RAID. Also, when other users switch to page the user (User Page) KILLHAZARD, if KILLHAZARD is using the service 210 hosting at this time, is displayed video 2001 in real time what he is doing (assuming he permits users to browse his user page (User Page) to monitor them). This is done by the server 1521-1525 application / game that provides application hosting User Page (user page), with a request from the system management services 401, whether KILLHAZARD active, and if so, the server 1521-1525 application / game that he It uses. After that, using identical techniques used by Game Finder (search tool of the game), a compressed video stream with appropriate resolution and format will go to the server 1521-1525 application / game executable applications User Page (user page) and it will be displayed . If a user selects the window gameplay realtime KILLHAZARD and then appropriately clicks on their input device, the window is increased (again using identical methods as in applications Game Finder (search engine game)) and video in real-time fills screen display with a resolution of 422 by observing corresponding characteristics of connecting to the Internet by watching.

The main advantage of this approach over the approaches of the prior art is that a user viewing the User Page (page) User can see a game playing on a real time, which is not owned by the user and it can not have a local computer or game console with the possibility of play games. What the user sees the user's User Page (User) "in action", a leading game gives him a great opportunity, and it is an opportunity to learn about the game, which is viewed by the user may need to test or improve their performance in it.

Recorded camera or uploaded videos from friends 2002 KILLHAZARD as shown in the User Page (on page user), and under each video clip there is a text that indicates whether this fellow online and leads the game (for example, six_shot is playing "Eragon" and MrSnuggles99 is offline, etc.). When you click on the menu (not shown) videoclips buddy switch to display recorded video, or unloaded in the video in real time what the shortstop, who are currently in service in 210 games hosting do at this point in their games. Accordingly, it becomes Game Finder (search engine of the game), the grouping of friends. If you choose to play shortstop, and the user clicks on it, it will increase up to full screen, and the user will be able to watch the game played back on the full screen in real time.

Again, the user is viewing the game buddy, does not belong to a copy of this (game), and do not belong to local computing resources / resources to conduct the game console of the game. Viewing the game is in fact urgent.

As previously described above, when a user plays a game in the hosting service 210, the user can "rewind" the game and find a video segment, which he want to save, and then saves the video segment to his User's Profile (User Page). They are called "Brag Clip" ("brag clips"). All segments in 2003 are Brag Clip videos 2003 KILLHAZARD saved from the previous games, which he led. In the position shown in 2004, how many times Brag Clip has been viewed, and when viewed Brag Clip, users have the opportunity to evaluate it, and 2005 the number of orange icons in the form of a keyhole indicates how high a rating. Brag Clip 2003 cycled constantly when a user views a page the user (User Page), along with the rest of the video on the page. If the user selects and clicks on one of the Brag Clip 2003 it increased to represent Brag Clip 2003 and DVR control elements provide the ability to play the video, pause it, rewind, fast forward, passing by stages, etc.

Play Brag Clip Server 2003 is implemented 1521-1525 application / game, loaded segments compressed video stored in the disk array type 1511-1512 RAID, when the user writes Brag Clip, and reducing it and reproducing it.

Brag Clip 2003 may also be a segment of the video "DVR 3D" (ie, a sequence of states of the game from the game, which can be reproduced and re-enables the user to change the point of view of the camera) of the games that support such means. In this case, the information about the state of the game is saved along with the record of a particular compressed video "through-flight", which made the user when the recorded segment of the game. When viewed User Page (User) and all the thumbnails and video windows are constantly cycled, Brag Clip 2003 DVR 3D cycle repeats constantly Brag Clip 2003 which was recorded as compressed video when the user writes "through flight" segment of the game. But when the user selects a Brag Clip DVR 3D 2003 and clicks on it, along with controls DVR, providing the ability to play compressed video Brag Clip, the user can click on a button, which gives him the means DVR 3D segment of the game. He will be able to manage "through-flight" camera for a segment of the game on their own, and if he would need (and the user that owns this page the user provides this possibility), it will be able to burn alternative "through flight" Brag Clip in the form of compressed video, (which) will then be available to other viewers of the page the user (either immediately or after the owner of the page the user will be able to view the Brag Clip).

This means Brag Clip DVR 3D 2003 is ensured through activation of the game, which is ready to start the replay of the recorded information on the state of play on another server, 1521-1525 application / game. Since the game can be activated almost instantaneously (as previously described) it is not difficult to activate it, the reproduction of it is limited to the game state recorded segment Brag Clip, and then allow the user to perform "through passage", and the camera records a compressed video buffer 1515 delays. After completing the user perform "flown through" the game is turned off.

From the point of view of the user activation "through passage" through 2003 Brag Clip DVR 3D requires no more effort than the control elements DVR control line Brag Clip 2003. He may not know anything about the game, or even how to play the game. He is the only operator of the virtual camera looking into the world of 3D games for the segment recorded another.

Users can also apply their own audio to Brag Clip, which is either recorded from microphones or unloaded. Therefore, Brag Clip can be used to create a custom animation with the characters and the action of the games. This animation method is commonly known as "machinima".

As we move users in the games they reach differing skill levels. Reproduced Game report these achievements in the service management system 401, and the skill levels are displayed on the User Page (user page).

Interactive animated advertisements

Makes a transition online ads from the text to still images, to video, and now to interactive segments, typically implemented using animation thin clients like Adobe Flash. The reason why the use of thin clients animation, is that users are experiencing impatience with the delay (of pre-emptive right to receive) the product or service referred to them. In addition, thin clients are executed on a very low-end PC, and, in fact, the person giving the ads can be quite sure that online ad will run properly. Unfortunately, thin clients animations, such as Adobe Flash, are limited in the degree of interactivity and duration of experience with (to reduce download time).

Figure 21 shows an interactive advertising, in which the user must choose the exterior and interior color of the car during the rotation of the car in the showroom when the means of ray tracing in real time shows how the car looks. Next, the user selects the "avatar" to control the car, and then the user can take the car for a ride or track for the race, or at an exotic location such as Monaco. The user can select the motor with a large swept volume and the best tires and then can see how configuration changes affect the ability of the car to accelerate, or to keep the road.

Sure, said advertising is actually a complicated video game 3D. But to play such an advertisement on a PC or video game console may need to download 100 megabytes, and in the case of PC can require the installation of special drivers, and may even be impossible to her performance, if no PC meeting the requirements of computing the GPU or CPU. Accordingly, such advertisements are practically feasible in the prior art configurations.

The service 210 hosting these ads run almost immediately and perfectly executed regardless of the capabilities of the client 415 of the user. Accordingly, it runs more quickly than the interactive advertisements thin client, with considerably more extensive use and experience a high degree of reliability.

Streaming geometry during real-time animation

1511-1512 Disk Array RAID type and routing part 1502 can provide data rates that are so big and timeout values ​​so small that it is possible to design video games and applications that are based on disk arrays 1511-1512 RAID type and routing incoming 1502 for reliable delivery of geometry on-the-fly in the middle of gameplay or an application during real-time animation (e.g., a "through passage" with a complex database.

With prior art systems, for example, a game video system shown in Figure 1, the mass storage devices available, particularly in practical home used devices, are far too slow to stream geometry in during game play except for situations where the required geometry It is somewhat predictable. For example, in a driving game, where there is a specified highway geometry for the buildings that appear in sight, it may be acceptable predictable, and mass storage devices can search in advance to the point where the geometry is approaching.

But in a difficult stage with unpredictable changes (eg, in a scene of the battle with complex characters overall) if RAM to a PC or a game video system is completely filled with the geometry of objects, now it is in sight, and then the user suddenly turns your character to see what is behind it, if the geometry has not been pre-loaded into RAM, it may be a delay before displaying it on the screen.

In the service of hosting 210 disk arrays 1511-1512 RAID type flow can transmit data at speeds exceeding the speed of Gigabit Ethernet technology, and in the case of the SAN can achieve a speed of 10 gigabits / second technology of 10 Gigabit Ethernet or other network technologies. With 10 Gigabits / second of gigabytes of data downloaded in less than a second. During frame 60 frame / sec (16.67 ms) can be loaded with approximately 170 megabits (21 megabytes) of data. Rotating media, of course, even in a RAID configuration still cause the value of the waiting time, exceeding the period of the frame, but the memory RAID device based on flash memory will eventually be as large as disk arrays RAID type of rotating media, and will not cause such high latency. In one embodiment, a write-through cache with a massive RAM for access with very low latency.

Accordingly, with sufficiently high network speed and sufficient mass memory with sufficiently low latency flow geometry can be transmitted in game servers 1521-1525 application / game with the speed at which the CPU and / or GPU may process the 3D data. Accordingly, in the example above, where the user suddenly turns your character and looks back, the geometry of all the characters are back, can be loaded before the character completes the rotation, and therefore, the user will feel as if he or she photorealistic are in a world that is so real, like acting.

As discussed previously, one of the last frontiers in a photo-realistic computer animation is the human face, and because of the sensitivity of the human eye to the imperfections of the slightest error in fotorealnom face can result in a negative reaction from the viewer. Figure 22 depicts how the live performance, captured with the use of technology Contour ™ Reality Capture (the subject of applications in the process of joint examination: "Apparatus and method for capturing the motion of a performer" Ser. No. 10/942609, filed September 15, 2004, "Apparatus and method for capturing the expression of a performer" Ser. No. 10/942413, filed September 15, 2004, "Apparatus and method for improving marker identification within a motion capture system" Ser. No . 11/066954, filed February 25, 2005, "Apparatus and method for performing motion capture using shutter synchronization" Ser. No. 11/077628, filed March 10, 2005, "Apparatus and method for performing motion capture using a random pattern on capture surfaces, "Ser. No. 11/255854, filed October 20, 2005," System and method for performing motion capture using phosphor application techniques, "Ser. No. 11/449131, filed June 7, 2006, "System and method for performing motion capture by strobing a fluorescent lamp," Ser. No. 11/449043, filed June 7, 2006, "System and method for three dimensional capture of stop-motion animated characters," Ser. No. 11/449127, filed June 7, 2006, the rights to which are owned by each of the present application CIP), resulting in a very smooth surface occupied, then to a mesh surface, there are a large number of polygons (ie, the movement of the polygon exactly follows the movement of the person). Finally, when the video is displayed on the live performance mesh surface for outputting textured surface fotorealny output result.

Although by current technology GPU can render the number of polygons in the mesh surface and texture and light the surface in realtime, if the polygons and textures are changed every frame period (that leads to the most fotorealnym results), then all the available RAM modern PC or video game console drained quickly.

Using the streaming geometry techniques described above, it becomes possible to apply in practice, continuous feed geometry game servers 1521-1525 application / game so that they can animate continuously fotorealnye face with allow the creation of video games with faces that are almost indistinguishable from those in the game actors.

Integration of linear content with interactive features

Movies, television programs and audio material (collectively, "Linear Content") are widely available for home and office users in many kinds. Linear content can be purchased on the physical storage media such as recording media CD, DVD, HD-DVD and Blu-ray. It also can be recorded by a DVR from a broadcast transmission, cable television and satellite broadcast transmission. And it is available as a content pay per view (PPV) via satellite and cable television, and as video on demand (VOD) on cable television.

More and more linear content is available via the Internet and how to download and streaming content. Currently, in fact, there is no one place where you can experience all the symptoms associated with linear media. For example, DVD, and other optical media, video, tend to have interactive features that are not available anywhere else, like filmmaker commentaries, short films, "the view," etc. Online music sites cover art and song information is generally not available on CD, but not all CD available online. And on the Web-sites associated with TV programs, there are often additional features, blogs and comments sometimes actors or creative personnel.

In addition, for many movies, or sporting events often have video games that are released (for movies) often with linear media, or (in the case of sporting events) that can be closely related to the real events (for example, trade players).

210 hosting service is well suited for the delivery of content in a linear arrangement of disparate forms of interrelated content. Of course, delivery of movies requires no more effort than delivering a highly interactive video games, and hosting service 210 can deliver a linear content in a wide range of devices in homes or offices, or mobile devices. Figure 23 depicts an exemplary user interface page for hosting service 210, which depicts a sample linear content.

But, unlike most linear content delivery system hosting service 210 may also deliver related interactive components (e.g., the menus and features on the DVD, the interactive graphic superimposing one upon another at HD-DVD and animation Adobe Flash, as explained below) on Web- sites. Accordingly, the limitations of the client device 415 is no longer to impose restrictions on what features are available.

In addition, host system 210 may dynamically and in real time to assemble linear content with video game content. For example, if the user is watching Quidditch match in Harry Potter and decides that she would like to try to play Quidditch, she can simply click on the button and a movie will be paused, and it will immediately move to the segment of the video game Harry Potter Quidditch. After the match Quidditch - one click of a button and a movie will be immediately continued.

With fotorealnoy graphics and production technology, in which the photographically captured video can not be distinguished from the characters to the performance of the actors, when the user makes a transition from playing Quidditch in the movie with the performance of the actors to play Quidditch in a video game on the hosting service, as described in this document, these two scenes virtually impossible to distinguish. This provides a completely new creative options available to directors and linear content and interactive content (eg, video games), as the boundaries between these two worlds become indistinguishable.

Using a hosting service architecture shown in Figure 14, the viewer may be granted control of the virtual camera in the movie 3D. For example, in a scene that takes place inside the train car, you can allow the viewer to control the virtual camera and inspect the car in the course of the story. It is assumed that all objects 3D ("resources") are available in the car, as well as meeting the requirements of the level of computing capacity with the ability to visualize scenes in real time, as well as the original movie.

And even for entertainment, formed not on the computer can be provided very exciting interactive signs. For example, in the movie "Pride and Prejudice", in 2005, there are many scenes in ornate old English mansion. For certain scenes in the mansion, the user can pause the video, and then control the camera to take a trip through the mansion, or maybe around the neighborhood. To accomplish this, the camera can be carried through the mansion with a lens of the "fish eyes", while it keeps a record of its position, much the same way as was done QuickTime VR Corporation prior art Apple, Inc. Then different frames can be converted so that the image will not be distorted and then stored in the disk array RAID type 1511-1512 together with the film and played back when the user decides to go on a virtual tour.

With regard to sporting events, sporting events, happening in real time, for example, the game of basketball can be transmitted through 210 hosting service to users watched it as they can watch it on a conventional TV. Once users have looked a specific game, video of the game (in the end, with basketball players who look as fotorealno how real players) may appear on the screen, and players start with an identical position, and users (perhaps everyone thus gets control one player) can restore the game and see if they can do better than these players.

Service 210 Hosting described in this document, it is very well suited to support the world of the future, because it can involve computing power and resources of the mass memory that it is impractical to install at home or in most office environments, and also its computing resources are always modern, with the latest computing hardware available, while at home there will always be at home with video games and PC previous generations. In hosting service 210, all of this computing complexity is hidden from the user, so even though they may use a very sophisticated systems, from the user perspective, this is a simple as changing channels on a television. In addition, users can access all the computing power and the experiences that might be used to provide the computing power from any client 415.

Games with several participants

In cases where a game is a game with several participants, then it can communicate with the game servers 1521-1525 application / game through a network of inbound routing through the network in 1502 and the bridge to the Internet (not shown) with the servers or gaming machines that do not 210 operate in the service of hosting. When playing the game with a number of parties by means of computers in general, Internet game servers 1521-1525 application / games will have the advantage of extremely fast access to the Internet (compared to if the game is running on the server the house), but they are limited by the capabilities of other computers, who lead the game on slower connections, and possibly also limited by the fact that the game servers on the Internet are designed to ensure the lowest common denominator, which can be home computers at a relatively slow consumer connections to the Internet.

But when playing with several participants being completely inside the server hosting the service center 210, then you can achieve great distinction. Each game server 1521-1525 application / game that provides hosting games for the user to be connected to other game servers 1521-1525 application / games, as well as any servers that hosting the central control of the game with several participants at extremely high speeds with a compound with extremely low latency and huge, very fast storage arrays. For example, if you are using Gigabit Ethernet technology for network routing incoming 1502, the game servers 1521-1525 apps / games are in communication with each other and communicate with any servers that hosting the central control of the game with several participants at the speed of gigabit / second with may only latency of 1 ms or less. In addition, disk arrays 1511-1512 RAID type can respond very quickly and then transfer data at speeds of gigabits / second. As an example, if a user customizes own character in terms of look and equipment so that the character has a large amount of geometry and behavior patterns, which are unique to the character, from the prior art systems, which are limited to game client packages running at home while a PC or a game console, if the character to appear in the sight of another user, that user must wait until after a long, slow loading, all data and the geometry of conduct booted his computer. Within 210 hosting service can be identical loading of Gigabit Ethernet with the service of a disk array RAID type 1511-1512 at a rate of gigabits / second. Even if the home user has a connection to the Internet 8 Mbit / s (which is extremely fast by modern standards), Gigabit Ethernet that is 100 times faster. Accordingly, what takes one minute for quick connection to the Internet, will take less than one second on Gigabit Ethernet.

Groups best players and competitions

210 hosting service is very good for the competition. Since the game is not running on the local client for users there is no way to cheat. Furthermore, due to the fact that the output routing 1540 may implement multicasting UDP streams, the hosting service 210 can transmit basic event thousands of people in the audience at once.

In fact, when there are certain video streams that are so popular that thousands of people are taking an identical flow (eg, display types of the main competition), it may be more efficient to send the video stream content delivery network (CDN), such as Akamai or Limelight, for the mass distribution of many client devices 415.

Similar levels of efficiency can be obtained when the CDN is used to display the pages of Game Finder (search tool game) teams best players.

For major events, you can use real sports commentator celebrities for comments during certain matches. Although a large number of users will face major competition and a relatively small number of users will play in the competition. Audio from a sports commentator celebrities can be routed to the game servers 1521-1525 application / game that host users playing in the competition, and hosting of any copies of the game modes visitor during the competition, and this audio can be superimposed on top of audioigry. Video sports commentator celebrities may also be imposed on the game, perhaps only for display on the screen of the visitor.

Faster loading web-pages

The source of the web transport protocol, Hypertext Transfer Protocol (HTTP), was conceived and defined in a period when the company had only a high-speed connection to the Internet, and users who have been online, using dial-up modem or ISDN. At this time, the "gold standard" for high-speed connections has a line T1, which enables data transfer speeds of 1.5 Mbit / s symmetric (ie equal to the speed of data transmission in both directions).

Currently, the situation is completely different. Average speed home connections via cable modem connections, or DSL in most developed countries have a much higher data rate downlink data stream, than a T1 line. In fact, in some parts of the world through input technology in building optical cable (fiber-to-the-curb) serves to house a data rate of 50 to 100 Mbit / s.

Unfortunately, HTTP was not designed (it also has not been implemented) for the actual use of the significant increase in speed. Web-site is a collection of files on a remote server. In very simple words, HTTP requests first file, waits for downloading and then requests a second file, it expects the downloads, etc. In fact, HTTP allows several "open connections", ie request multiple files at a time, but because of the established standards (and the desire to prevent the overloading of web-servers) it is possible to only a very small number of open connections. In addition, because of the way creating Web-browser pages are often not aware that many pages can be simultaneously available for immediate download (that is, only after the analysis (parsing) pages it becomes clear that there should be a new file is loaded, for example image ). Accordingly, the files on the web-site is essentially loaded one by one. And because of a challenge-and-response, used by HTTP, the latency associated with each downloadable file is about (getting access to the typical web-servers in the United States), 100 ms.

With respect to compounds with relatively low speed is not a big problem because the boot files themselves dominates the waiting time for web pages. But, since the speed increasing compounds, especially for complex web pages, problems begin to arise.

In the example shown in Figure 24, is a typical commercial web-site (this particular web-site is the site of a leading brand sports shoes). This web-site file 54 exists. The files include HTML, CSS, JPEG, PHP, JavaScript and Flash files, and include video content. A total of 1.5 must be downloaded megabyte to the page has become active (i.e., the user can click on it and begin to use it). There are several reasons for the large number of files. First, this web-page is complex and is up to date, and, secondly, it is a web-page, which is assembled dynamically based on user information, access to the page (for example, from which country user, what language, whether the user has made purchases before, etc.), and depending on all of these factors, different files are downloaded. However, it is very typical commercial web page.

24 shows the amount of time that elapses before the web page is active, as the connection speed grows. At speed connections in 2401 to 1.5 Mbit / s using a conventional web-server via a conventional web-browser is required 13.5 seconds to a web page becomes active. At speed 2402 Connection 12 Mbit / s load time is reduced to 6.5 seconds, or about twice as fast. But in 2403 the connection speed of 96 Mbit / s download time is reduced to only about 5.5 seconds. The reason this happens is that at such a high speed of loading the boot files themselves is minimal, but the latency per file, roughly 100ms, still remains, resulting in a 54 file × 100 ms = 5.4 seconds waiting time. Accordingly, regardless of the speed of connection to the home time to the web site to become active is always equal to at least 5.4 seconds. Another factor is the organization of the queue on the server side. Each HTTP request is added to the end of the line, respectively, on a busy server, it has a significant impact, because to get from the web-server every little element, HTTP requests must wait their turn.

One way to solve this problem is to abandon the HTTP or override it. Or perhaps it is better that the owner of a web-site joined files into a single file (for example, in the format of Adobe Flash). But in practice, the company, as well as many other (company) have invested a lot into the architecture of your website. Further, while some homes there are compounds 12-100 Mbit / s, the majority of homes still exist slower speeds, and HTTP work well at slow speed.

One alternative is hosting web-browser on the server 1521-1525 application / game and file hosting for web-servers such as disk arrays 1511-1512 RAID (or perhaps, in RAM or on a local storage on servers 1521-1525 application / games offering hosting web-browser). Due to the very fast interconnection inbound routing through 1502 (or local storage) instead of 100 ms latency when using HTTP for each file, there will be a slight latency when using HTTP for each file. Then, instead of a user in his home accessed the web page through HTTP, the user can access the web page through client 415. Then, even with compound 1.5 Mbit / s (because this web page does not require much bandwidth for its video) web-page is active is less than 1 second for each communication line 2400. Essentially, there is no time to wait before the display of the active page web-browser running on a server, 1521-1525 application / game, and there is no noticeable latency to display the video output of 415 client web-browser. As the user enters a mouse on a web page and / or typing on it, the information is sent to the user's input in the web-browser, running on the server 1521-1525 application / games, and web-browser responds accordingly.

One disadvantage of this approach is that, if the compression device is constantly transmitting video data, then bandwidth is used, even if the web page becomes unmovable. This can be corrected by performing compression device to transmit data only when (and if) the web page changes, and then transmit the data only in the portion of the page that change. While there are some web pages with Flash banners, etc. that are constantly changing, such web pages tend to irritate, and usually web pages are fixed, if there is no reason for any movement (eg, video). For these web pages is probably the case when using the service 210 hosting will be sent less data than conventional web-server, because it will only be broadcast virtually on-screen image, no executable code thin client and there are no large objects that can never be viewed, such as switching pictures.

Accordingly, using the 210 service hosting for hosting legacy Web pages load time of Web pages can be reduced so that the opening of a Web page is similar to switching TV channels: the web page is actually active immediately.

Providing debugging games and applications

As mentioned earlier, video games and graphics applications with real-time applications are very complex and, as a pr Awilo when they are released in actual use, they contain errors. While software developers get feedback from users (the message), error, and they may have some means of reverse transfer of machine state after a crash, it is very difficult to identify precisely what caused the crash or improper performance of games or real-time applications.

When a game or application is executed in the service 210 hosting, video / audio games or applications permanently stored in the buffer 1515 delays. In addition, the watchdog process is executed by each server 1521-1525 application / game that is regularly reported in 401 management hosting services that the server 1521-1525 application / game is executed smoothly. If the watchdog process does not send a message, the system management server 401 attempts to contact the server 1521-1525 application / game and if successful, gathers all available machine state. All available information together with the video / audio recorded buffer 1515 delays will be sent to the software developer.

Accordingly, when the software application developer or a game receives notification of a crash of the hosting service 210, (it) is time-lapse recording that led to this crash. This information can be very valuable in detecting errors and correcting them.

Also note that when there is a server crash 1521-1525 application / game, the server is restarted with the last point of the restart and the user transmits a message of apology for the technical problem.

Sharing resources and reduce costs

The system shown in Figures 4a and 4b, provides many advantages for both end users and for developers of applications and games. For example, typically, home and office client systems (e.g., PC or game consoles) are only used for a short amount of time per week. According to a press release issued Oct. 5, 2006, Nielsen Entertainment "Active Gamer Benchmark Study" active players spend an average of 14 hours a week to play on video game consoles and about 17 hours a week on handheld computers. This report also states that the whole game activity (including management games for the PC, Pocket PC, console) active players on average 13 hours a week. Given a higher duration of the video game console, 24 × 7 = 168 hours a week, which means that the home video game console, the active player is only 17/168 = 10% of the time in a week. Or 90% of the video game console is in use. Given the high cost of video game consoles and the fact that producers finance such devices, it is a very inefficient use of expensive resources. PC, companies are also typically used only a fraction of the time per week, especially stationary desktop PC, is often required for applications with greater functionality, e.g., Autodesk Maya. Despite the fact that several companies are working all the time and on holidays, and some PC (e.g., laptops brought home for work in the evening) are used all the time and holidays, most business activities tend to concentrate between about 9 hours to 17 hours in the time zone of the companies, Monday through Friday, excluding holidays and breaks (eg lunch), and since most PC's used when the user is actively working with a PC, it follows that the desktop PC, are typically used at this time . If we assume that the PC always uses 9 hours to 17 hours, 5 days a week, it means that the PC used 40/168 = 24% of the time per week. High-performance desktop PC are very expensive investment for the company, and this is reflected in the very low level of use. Schools that teach in desktop computers can use the computer for an even smaller part of the week, and even though it varies depending on the time of study, much of the learning takes place during the day from Monday to Friday. Accordingly, in general, PC and video game consoles are used only a small part of the time in a week.

That is because many people work in companies or in school during the day, Monday through Friday, except holidays, these people usually do not play video games at the time, and so, in fact, they play video games as Typically, at other times, such as in the evenings, on weekends and holidays.

Given the configuration of hosting service depicted in Figures 4a, usage pattern, as described in the two paragraphs above, result in a very efficient use of resources. Clearly, there is a limit on the number of users that can be served by 210 hosting service at this time, particularly when users require rapid response in real time for complex applications, similar to complicated video games 3D. But unlike the video game console at home, or PC, used by the company, which, as a rule, do not use most of the time, 402 servers can be reused by different users at different times. For example, high-performance server with high-performance dual 402 CPU and dual GPU and more RAM can be used by companies and schools from 9 am to 5 pm on weekdays, but used by players, leading sophisticated video games in the evenings, on weekends and holidays. Similarly, applications with low performance can be used by companies and schools in low-server 402 CPU Celeron, without a GPU (or very low-productivity GPU) with limited RAM to during working hours, and play with low performance can use low-productivity server 402 for a non-working time.

In addition, with the arrangement of hosting services described in this document, the resources are shared by virtually thousands, if not millions, of users. In general, only a small percentage of the total number of users of online services using the service at the moment. If we consider the use of video games Nielsen statistics listed above, it is easy to understand why. If the active players play console games only 17 hours per week, and assuming that the peak usage time of the game usually occurs after hours, after hours in the evening (from 17 hours to 24 hours, 7 × 5 days = 35 hours / week) and on weekends (from 8 hours to 24 hours, 16 × 2 = 32 hours / week), ie 35 + 32 = 65 peak hours per week for 17 hours of gameplay. The exact peak user load on the system is difficult to estimate for many reasons: some users will play during off-peak times, there may be periods of the day when there are peaks of the partition to user groups, and these peak times can be affected by the type of ongoing games (for example, children's games are is likely to play in the evening in an earlier time), etc. But, given that the average number of hours that the player holds the game, is much smaller than the number of hours of the day when a gamer is likely to play a game, only a fraction of the number of users of the hosting service 210 will be using it at a given moment. For this analysis, we assume that the peak load is 12.5%. Accordingly, only 12.5% ​​of resources of bandwidth compression and calculations are used at present, resulting in only 12.5% ​​of the hardware cost to support a given user to maintain a given level of performance game due to reuse of resources.

Moreover, given that some games and applications require more computing power than others, resources may be allocated dynamically based on the ongoing game or applications executed by users. Accordingly, the user selects an application or a game with a low-performance, low-productivity is highlighted (less expensive) server 402, and the user selects an application or a game with a high-performance, high-capacity will be allocated (more expensive) server 402. Of course, this game or application can have sections with higher performance and lower characteristics of the game or application, and the user can switch from one server 402 to another server 402 between the sections of the game or application to support the work of the user on the server 402 with the lowest cost that meets the needs of the application or game. Note that disk arrays such as 405 RAID, which are much faster than one drive available, even low-productivity servers 402, which will have the advantage of faster transfer speeds of disk. Accordingly, the average cost of each server 402 in all major games or used applications is much less than the cost of the most expensive server 402, which executed the application or game with the highest performance, but even low-productivity servers 402 extract the benefits of disk capacity of the disk arrays 405 RAID type .

In addition, the server 402 in 210 hosting service can be nothing more than a motherboard without a PC peripheral interfaces or disk storage devices except the network interface and may eventually be integrated into a single-chip integrated circuit only with a fast network interface with the SAN 403. In addition , 405 types of disk arrays RAID, are likely to be shared among a much larger number of users than the existing drive, so the cost of disk storage for each active user will be much smaller than one magnetic disk. All of this equipment is likely to be in front in the situation room for installing servers with controlled environmental characteristics. If there is a failure of the server 402, it can be quickly fixed or replaced in service 210 hosting. In contrast, PC or game console in the home or office must be rigid autonomous apparatus which must withstand acceptable wear due to bumps or falls, requires housing has at least one magnetic disk drive, has to withstand adverse environmental conditions (e.g. being wedged in the superheated audio videokorpus with other equipment), requires warranty service, it must be packed and shipped and sold by the retailer, who probably gets the retail mark-up. Furthermore, PC or game console must be performed in accordance with the peak characteristics of expected applications or games that require the most computationally intensive to be used at some point in the future, even though the application or game with a low-performance (or sections games or applications) can run most of the time. And if there is a failure of a PC or console, they repair is expensive, time-consuming (with an adverse impact on producers, users and software developers).

Accordingly, given that the system shown in Figure 4a provides a user experience with comparable experience with using a local computing resource, for a user in the home, office or school to experience a given level of computing power, much less provide these computational capabilities through the architecture shown in Figure 4a.

Eliminating the need to modernize

In addition, users no longer have to worry about upgrading a PC and / or consoles to play new games and manage new applications with higher performance. Any game or application hosting service 210 no matter what type of server 402 is required for this game, or applications available to users, all games and applications are executed almost immediately (ie, fast-loading of the disk arrays 405 RAID type or local storage servers 402) and properly with the latest updates and bug fixes (ie, software developers can choose the ideal configuration of the server for the server (s) 402, which (s) perform (s) the given game or application, and then configure the server (s) 402 with the best drivers, and then over time, developers can provide updates, bug fixes, etc., for all copies of the game or application hosting service 210 simultaneously). Of course, once the user starts to use the service 210 hosting, the user is likely to discover that games and applications continue to provide the best experience of use (for example, through the update and / or correct errors, and it may happen that the user will find a year later that new game or application is now available in the service 210, which uses calculations (for example, a high-performance GPU), which did not even exist a year earlier, respectively, the year before that it was impossible, that the user has bought the said technology that could perform game or application a year later. Since the computing resource, which is a game, or who performs the application is invisible to the user (ie the user's perspective, the user simply selects the game or application that start running almost immediately - in much the same as if the user switches the TV channels), hardware user will be "upgraded" without the user even if he was aware of the modernization.

Eliminating the need for backups

Another important issue for users in companies, schools and homes are backups. The information stored in the local PC and video game consoles (for example, in the case of the console, rankings and achievements in the game the user) may be lost in the event of a drive failure or accidental erasure. There are many applications available that provide manual or automatic backup for PC, and the state of the game console can be discharged to the online server backup, but local backups are usually copied to another local hard drive (or other storage device), which should be stored in a suit a safe place and be arranged, and redundancy in the online services are often limited because of low speed upstream data available through typical low-cost connection to the Internet. By hosting service 210 for data 4a stored in the disk type RAID arrays 405, may be configured using methods such RAID configuration of the prior art, known to those skilled in the art, a way that, if there is a disk failure, the data They will not be lost and the technician in the server center, which is a damaged disk will be notified and then (it) to replace the disk, which will then be automatically updated so that the disk array is a RAID was again fault tolerance. In addition, since all magnetic disk are next to each other with the (software) fast LANs between them through the SAN 403, the server center it is easy to organize, to regularly create backup copies of all disk systems on the secondary storage device, which can stored in the server center or moved beyond it. From the perspective of service users 210 hosting their data just keep protected and they will never have to think about backups.

Access to demo

Users often need to experience a game or application prior to purchase. As described earlier, there are tools of the prior art for testing the demo version (the verb form "demo" means testing a demo version, which is also called "demo", but as a noun) games and applications, but in each of them there are limitations and / or deficiencies . Using a hosting service 210 users to easily and conveniently test the demo version. In fact, everything that the user does - it selects a demo version through the user interface (eg, interface, described below) and test it. The demo version is loaded on the server 402, suitable for the demo version, almost immediately, and it is executed like any other game or application. Regardless of whether you want to demo a very high-performance server 402 or server 402 low-productivity, and no matter what type of home or office client 415 uses the user, from a user perspective, the demo version just works. Publisher software demo or game, or application can regulate it, what kind of a demo version to give the user to test and at what time, and, of course, this demo version can include user interface elements that allow the user to access the full version demonstrated the game or application.

Since demos are likely to be offered at a lower price or for free, some users may try to re-use the demos (especially game demos, repeated game that may be fun). 210 hosting service can use different methods to limit the use of the demo version for the user. The most direct approach is to set a user ID for each user, and limit the number of test demo version provided for the user ID. The user, however, may set up multiple user ID, especially if they are free. One way to deal with this problem is to limit the number of test demo version provided for the client 415. If the client is a standalone device, then this device there is a serial number, and the service host 210 may limit the number of times a customer can access demo with this serial number. If the client 415 is running as software on a PC or other device, then a serial number can be assigned to service 210 Host and stored on the PC and used to limit use of the demo, but considering the fact that the PC can be reprogrammed by users, and the serial number can be erased or changed, another possible option for hosting service is 210 accounting protocol address Media Access Control (MAC) network adapter PC (and / or other machine-specific identifiers such as the serial number of hard drives, etc.) and limiting the use of the demo version for him. Given that the MAC-addresses of network adapters can be changed, but this is not a reliable way. Another approach is to limit the number of test demo for this IP-address. Despite the fact that IP-addresses can be periodically reassigned providers of DSL and cable modem connection, in practice this does not happen very often, and if you can define (for example, by contacting the ISP) that this IP is in the block of IP-address to access DSL or cable modem connection, related to residential buildings, then for the home, as a rule, can be found a small number of uses of the demo version. Furthermore, there may be multiple devices in the home for the NAT router, sharing an identical IP-address, but typically in a residential environment there are a limited number of such devices. If the IP-address is in the unit, service company, then the company can be found to a greater number of demos. But in the end, the combination of all the above mentioned approach is the best way to limit the number of demos for the PC. Despite the fact that there can not be a reliable method by which the full commitment and skillful user can be limited in the number of repeat performances, demos, with the creation of a large number of barriers can create a sufficient deterrent, that the violation of the operating system demo will be impractical for most PC users and soon, they will use demos as implied testing of new games and applications.

Benefits for schools, companies and other institutions

Significant advantages, in particular, get businesses, schools and other institutions that use the system shown in Figure 4a. As companies and schools there are significant costs associated with the installation, maintenance of the PC and modernization, especially when it comes to PC performance for applications with high performance, for example, Maya. As previously stated, PC, are typically used only a fraction of the time a week, and as in the home, the cost of PC with a given level of performance is much higher than in the environment of the school or office environment than in a server center environment.

In the case of large companies or schools (such as large universities), it may be appropriate to IT departments of organizations have established support centers and server computers that are accessed remotely through a connection-level LAN. There are several solutions for remote access of computers via LAN or via private broadband connection between the offices. For example, through Terminal Server Microsoft Windows Terminal Server or through the application of remote access to the desktop PC (virtual network computing), such as VNC, by RealVNC, Ltd., or by means of thin client from Sun Microsystems, users can remotely access a PC or server with a range of quality of response time graphic devices, and practical experience of the user. Further, such self-managed server centers are typically assigned to the same company or school and as such, can not take advantage of the combination use, which is possible when a variety of applications (e.g., entertainment applications and business applications) use identical computing resources at different times weeks. Accordingly, many companies and schools is no scale, resources and expertise to install their own server center, where there is a network connection with a speed LAN to each user. Indeed, a large percentage of schools and businesses have identical connection to the Internet (e.g., DSL, cable modems) as well as in homes.

However, in such organizations may, however, a need for a very high-computing or continuously or intermittently. For example, in a small architectural firm may be only a small number of architects with a relatively modest computing needs when performing design works, but it may occasionally require very high-performance computing 3D (for example, when creating a 3D "through the passage of a" new architectural project for a client). The system shown in Figure 4A, is very well suited for such organizations. These organizations are not required other than the network connection type, are identical to those available for homes (eg, DSL, cable modem), and, as a rule, are very economical. They can either use the cost PC, the client 415, or completely do without a PC and use the cost-specialized device that simply implements the logic of the control signal 413 and the restoration of 412 compressed video with low latency. These features, in particular, are attractive to schools that may be a problem with the theft of a PC or damage components that require careful handling in PC.

This arrangement solves several problems for these companies (and many of these benefits are also shared by home users that perform universal computation. For example, the operating costs (which, ultimately, should be returned in some form for users to have a viable business) may be much less, because (a) that computing resources are shared with other applications, which have different peak periods of use for a week, (b) the organization can access (and bear the cost of) high-performance computing resources only when required ( c) organizations should not provide resources for backup, or in any other way support the high-performance computing resources.

Excluding piracy

In addition, games, applications, interactive movies, etc. It can no longer be used illegally as the present time. Since the game is performed in the center of the service, users do not provide access to the underlying control program, respectively, there is nothing that can be illegally used. Even if the user can copy the source code, it can not execute it on a standard game console or home computer. This opens up the markets of the world in such places as China, where the standard video games are not available. Resale of used games as possible.

For game developers there are fewer market discontinuities as is the case at present. 210 hosting service can be gradually updated over time as requirements change to the games, unlike the current situation where a whole new generation of technology users and developers forced to modernize and developer of games depends on the timely delivery of the hardware platform.

Interactive video streaming

The above description provides a wide range of applications, which provide a new opportunity to the underlying idea of ​​an interactive video streaming, based on common Internet-technologies with low latency (which also implicitly includes the audio along with the video, as used in this specification). Prior art systems that provide video streaming through the Internet, allow only applications that can be realized with the interaction of high latency. For example, the basic playback controls for the video line (for example, pause, rewind, fast forward) work, respectively, with high latency, and you can choose between linear video transmission lines. And, as stated earlier, the nature of certain video games allows them to conduct high latency. But the big latency (or low compression ratio) approaches of the prior art video streaming severely limits the possible applications of streaming video or restricts their use to specialized network environments, and even in these environments, the prior art methods introduced significant load on the network. The technology described herein opens the door for the wide range of applications possible with an interactive streaming video with low latency through the Internet, particularly for those that are provided through Internet connection to the consumer level.

In fact, with such small client devices, the client 465 at 4c, which is enough to provide extended user experience with virtually any number of computing power, an arbitrary amount of fast memory and an extremely fast web experience among high-end servers, it provides the possibility of a new era of computing. Furthermore, since the bandwidth requirements do not increase, the growth of computing power of the system (i.e., because the bandwidth requirements are attached only to the display resolution, quality and frame rate), after the ubiquitous high-speed connection to the Internet (for example, through universal coverage of wireless communication with low latency), reliability and a high enough capacity to meet the needs of the display 422 of all users, the question arises whether the necessary thick clients (for example, PC, or mobile phones running Windows , Linux, OSX, etc.) or even thin clients (e.g., Adobe Flash or Java) for typical consumer and business applications.

The emergence of interactive streaming video results in a revision of the assumptions about the architecture of a computer system. An example of this is an embodiment of a server hosting a service center 210 shown in Figure 15. Way video buffer delay and / or group video 1550 is a feedback loop in which the transmitted user group online streaming video output of servers 1521-1525 application / game is fed back into the servers 1521-1525 apps / games or in real time on the road in 1552, or according to the selected delay path 1551. This enables a wide range of practical applications (e.g. such as those depicted in Figure 16, Figure 17 and Figure 20), which would be impossible or impractical, or by the local computer architectures prior or server art. But, as a more general architectural feature, what provides the feedback loop 1550 is recursion at the streaming interactive video, since video can be returned back indefinitely as the application requires it. It provides a wide range of the application, which previously were not available.

Another key architectural feature is that the video stream is unidirectional UDP. This enables effectively an arbitrary level multicast streaming interactive video (for comparison, two-way flow, such as flow TCP / IP, would create more traffic congestion in the network due to the transmission of information back and forth as the number of users). Multicast is an important means in the server center because it allows the system to quickly respond to the increasing demands of users Internet (and, in fact, the world's population) in carrying out communications based on the "one to many" or "many to many" . Again, the examples discussed in this specification, for example, in Figure 16, which depicts the use of interactive and streaming video recursion and multicasting are just the tip of a huge iceberg of possibilities.

In one embodiment, the various functional modules illustrated in this specification and associated steps may be performed by specific hardware components that contain "sewn" logic to perform these steps, for example, an application specific integrated circuit ("ASIC"), or any combination programmed computer components and custom hardware components.

In one embodiment, said modules may be implemented on a programmable digital signal processor ("DSP"), such TMS320x architecture company Texas Instruments (e.g., TMS320C6000, TMS320C5000, etc ...). You can use various other DSP, though, as before, with the implementation of these underlying principles.

Embodiments may include various steps as set forth above. The steps may be embodied in machine-executable commands that cause the execution of certain phases of the universal or specialized processor. The various elements that do not belong to these underlying principles, such as computer memory, hard drive, input devices, have not been included in the drawings in order to avoid obscuring the pertinent aspects.

Elements of the disclosed subject matter may also be provided as a computer readable storage medium for storing the machine-executable instructions. The computer readable medium may include, for example, flash memory, optical disks, CD-ROM, ROM DVD, RAM, EPROM, EEPROM, magnetic or optical cards, propagation medium, or computer-readable media of another type suitable for storing computer instructions. For example, the present invention may be downloaded as a computer program which may be transferred from a remote computer (e.g., server) to a requesting computer (e.g., a client) through data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem or network connection).

It should also be understood that elements of the disclosed subject matter may also be provided as a computer program product that may include a machine readable medium having stored instructions that may be used to program a computer (e.g., a processor or other electronic device) to perform a sequence operations. Alternatively, the operations may be performed a combination of hardware and software. The computer readable medium can include, for example, floppy diskettes, optical disks, CD-ROM and magneto-optical disks, ROM, RAM, EPROM, EEPROM, magnetic or optical cards, propagation medium, or media / machine-readable medium of another type suitable for storage computer instructions. For example, elements of the disclosed subject matter may be downloaded as a computer program product, wherein the program may be transferred from a remote computer or electronic device to a requesting process by data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem or network connection) .

Furthermore, although the disclosed subject matter has been described in conjunction with specific embodiments, there is a fairly numerous modifications and variations within the scope of the subject disclosure. Accordingly, the specification and drawings should be considered as illustrative and not limiting.