Add-on Management Methods

CROSS-REFERENCE TO RELATED APPLICATIONS

p-0002This application claims benefit of and priority to U.S. Provisional Patent Application Ser. No. 61/382,470 entitled “Add-on Management” and filed Sep. 13, 2010. This application is related to U.S. patent application Ser. No. 12/826,130 filed Jun. 29, 2010 and entitled “Video Game Overlay,” and U.S. patent application Ser. No. 12/826,489 filed Jun. 29, 2010 and entitled “Voice Overlay.” The disclosures of all the above commonly owned patent applications are hereby incorporated herein by reference.

BACKGROUND

p-00031. Field of the Invention

p-0004The invention is in the field of computer systems and specifically in the field of server side execution of computer programs.

p-00052. Related Art

p-0006Add-ons are sets of computing instructions that are used to extend the functionality of or otherwise augment a computer program. Add-ons are used in a wide variety of applications. For example, an Adobe Reader® add-on may be used to enable Microsoft word to print to a .pdf file, or to enable a browser to view .pdf files. Add-ons are popular in on-line video games. For example, the popular multi-player on-line game World of Warcraft® has available hundreds of add-ons that can be used to enhance game play. These add-ons are typically provided by third parties. To add an add-on to a computer program a user downloads the add-on from a website and installs the add-on in an appropriate directory on the user's computer. As used herein, the term “add-on” is used to refer to optional computing instructions configured to extend the functionality of a computer program, where the add-on is developed specifically for the computer program. An add-on may be developed by a third-party or by the developer of the computer program. Some computer programs are configured to look for add-ons in specific directories. Add-ons are distinguished from other software utilities, such as printer drivers, in that they are configured to augment a specific program. Add-ons are sometimes referred to as “plug-ins.”

p-0007Add-on data is data included with, generated or organized by an add-on. For example, in a video game add-on data may include the locations of objects within a game, inventory information, records of purchases and sales, auction house data, characteristics of non-player characters, maps, historically information, audio data, event timing information, and/or the like.

p-0008Computer programs often include computing instructions or an application programming interface (API) configured for communication between the computer program and an add-on. For example, it is common for a computer program to be configured to look in a specific directory for add-ons. This directory can be a subdirectory of a directory in which the computer program is stored.

SUMMARY

p-0009Some embodiments of the invention include systems and methods of using add-ons in systems where a computer program is executed on a server and streaming video generated as a result of the program's execution is provided to a remote client. The add-ons can be installed on the server or the client. If an add-on is installed on the client and the computer program on a server, then an application programming interface (API) is provided such that the computer program can operate as if the add-on was on the server local to the computer program. The API on the server is configured to pass communications between the add-on and the computer program over a communication network.

p-0010If the add-on is installed on the server, the computer program may be installed on either the client or a location geographically remote from the client. Various embodiments of the invention can include, for example, utilities configured for remotely installing add-ons, computing instructions configured to provide a virtual environment for execution of the add-on; storage of add-on data, turning add-ons on and off, executing add-ons securely, maintaining an approved add-on list; and/or the like.

p-0011Various embodiments of the invention include game server system comprising a video source configured to use game logic to generate video frames and to encode the video frames to a video stream, the video stream being based on a game environment of a video game, the game logic being augmented by an add-on; an I/O device configured to communicate the video stream to a geographically remote client and to receive game commands from the client; add-on storage configured to store the add-on in a location accessible to the video game, and to store add-on data of the add-on in a location accessible to the add-on; an add-on manager configured to create an add-on environment configured for use of the add-on by the video game, associate the add-on with a user account, and associate the add-on data of the add-on with the user account; and a processor configured to execute the game logic or the add-on manager.

p-0012Various embodiments of the invention include a system comprising a video source configured to use game logic to generate video frames and to encode the video frames to a video stream, the video stream being based on a state of a video game and game commands received from a geographically remote client, the game logic being augmented by an add-on; an I/O device configured to communicate the video stream to the geographically remote client and to receive the game commands from the client; an add-on manager configured to automatically update the add-on in response to receiving a new version of the game logic; and a processor configured to execute the game logic or the add-on manager.

p-0013Various embodiments of the invention include a system comprising game logic installed on a first computing device and configured to generate video frames and to present the video frames on a display; an first API filter installed on the first computing device and configured to redirect commands between the game logic and an add-on, the add-on being installed on a second computing device geographically remote from the first computing device and being configured to augment the functionality of the game logic; an I/O device configured to communicate the commands between the first computing device and the second computing device; and a processor configured to execute the game logic.

p-0014Various embodiments of the invention include method of executing an add-on, the method comprising installing the add-on on a game server system configured to provided a video stream to a client, the video stream being a result of execution of game logic whose functionality the add-on is configured to extend, the client being geographically remote from the game server system; executing the game logic; receiving a game command from the client at the game logic; passing the received game command from the game logic to the add-on; generating add-on data using the add-on; and storing the add-on data in a location geographically remote from the client, the add-on data being associated with an account of a user of the client.

p-0015Various embodiments of the invention include method of installing an add-on, the method comprising identifying the add-on to be installed; establish an add-on environment configured for the add-on to be accessed by a computer program, the computer program being executed using a client and being located on a server geographically remote from the client; placing the add-on in the add-on environment; associating the add-on with an account of a user of the client; identifying storage for add-on data generated by the add-on; and associating the storage or the add-on data with the account.

p-0016Various embodiments of the invention include method of automatically updating an add-on, the method comprising receiving an updated version of a game logic at a video source, the video source being configured to generate a video stream based on game commands received from a geographically remote client and to provide the video stream to the remote client; determining that an add-on is used to augment functionality of the game logic; sending a request for an updated version of the add-on to a third party, the request including an identity of the add-on and an address to which the update should be provided; receiving the updated version of the add-on via a computing network, the updated version of the add-on being configured to augment the updated version of the game logic; and automatically installing the add-on in an add-on environment configured for the add-on to be used by the game logic.

BRIEF DESCRIPTIONS OF THE DRAWINGS

p-0017<figref idrefs="DRAWINGS">FIG. 1</figref> illustrates a system configured to execute a computer program augmented by one or more add-ons, according to various embodiments of the invention.

p-0018<figref idrefs="DRAWINGS">FIG. 2</figref> is a block diagram of embodiments of a video source configured to serve multiple video games augmented by add-ons, according to various embodiments of the invention.

p-0019<figref idrefs="DRAWINGS">FIG. 3</figref> illustrates further details of an add-on manager, according to various embodiments of the invention

p-0020<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates further details of a client, according to various embodiments of the invention.

p-0021<figref idrefs="DRAWINGS">FIG. 5</figref> illustrates methods of installing and/or using an add-on, according to various embodiments of the invention.

p-0022<figref idrefs="DRAWINGS">FIG. 6</figref> illustrates methods of automatically updating an add-on, according to various embodiments of the invention.

DETAILED DESCRIPTION

p-0023Systems in which a computer program is executed on a server and a video stream output of the computer program is provided to a geographically remote client are disclosed in the commonly owned patent applications cited above. Various embodiments of the invention provide an ability to use add-ons in these systems. The add-ons are installed, executed, supported and otherwise managed by an add-on manager that can be located at the server, the client or some other location. For example, the add-on manager may be used to install an add-on within an appropriate environment of the server such that it can be accessed through an API of the computer program. Add-on data generated by the add-on and the add-on itself are typically associated with a specific, client, user account, avatar, and/or the like. In other embodiments of the invention the computer program but not the add-on is installed on the client.

p-0024<figref idrefs="DRAWINGS">FIG. 1</figref> illustrates a Game System <b>100</b> configured to execute a computer program augmented by one or more add-ons, according to various embodiments of the invention. Game System <b>100</b> is configured to provide a video stream to one or more Clients <b>110</b> via a Network <b>115</b>. Game System <b>100</b> typically includes a Video Server System <b>120</b> and an optional Game Server <b>125</b>. Video Server System <b>120</b> is configured to provide the video stream to the one or more Clients <b>110</b> with a minimal quality of service. For example, Video Server System <b>120</b> may receive a game command that changes the state of, or a point of view within, a video game, and provide Clients <b>110</b> with an updated video stream reflecting this change in state with minimal latency. The Video Server System <b>120</b> may be configured to provide the video stream in a wide variety of alternative video formats, including formats yet to be defined. Further, the video stream may include video frames configured for presentation to a user at a wide variety of frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second, although higher or lower frame rates are included in alternative embodiments of the invention. In some embodiments, Video Server System <b>120</b> is configured to provide three dimensional video data. E.g. matched video streams rendered from points of view separated by the distance between a person's eyes.

p-0025Clients <b>110</b>, referred to herein individually as <b>110</b>A, <b>110</b>B, etc., may include terminals, personal computers, game consoles, tablet computers, telephones, televisions, set top boxes, kiosks, wireless devices, digital pads, stand-alone devices, handheld game playing devices, and/or the like. Typically, Clients <b>110</b> are configured to receive encoded video streams, decode the video streams, and present the resulting video to a user, e.g., a player of a game. The processes of receiving encoded video streams and/or decoding the video streams optionally includes storing individual video frames in a receive buffer of the client. The video streams may be presented to the user on a display integral to Client <b>110</b> or on a separate device such as a monitor or television. Clients <b>110</b> are optionally configured to support more than one game player. For example, a game console may be configured to support two, three, four or more simultaneous players. Each of these players may receive a separate video stream, or a single video stream may include regions of a frame generated specifically for each player, e.g., generated based on each player's point of view. Clients <b>110</b> are optionally geographically dispersed. The number of clients included in Game System <b>100</b> may vary widely from one or two to thousands, tens of thousands, or more. As used herein, the term “game player” is used to refer to a person that plays a game and the term “game playing device” is used to refer to a device used to play a game.

p-0026Clients <b>110</b> are configured to receive video streams via Network <b>115</b>. Network <b>115</b> may be any type of communication network between computing devices including, a telephone network, the Internet, wireless networks, powerline networks, local area networks, wide area networks, private networks, and/or the like. Network <b>115</b> explicitly does not include communication channels completely within a computing device, such as a motherboard bus. In typical embodiments, the video streams are communicated via standard protocols, such as TCP/IP or UDP/IP. Alternatively, the video streams are communicated via proprietary standards.

p-0027A typical example of Clients <b>110</b> is a personal computer comprising a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. The decoding logic may include hardware, firmware, and/or software stored on a computer readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the particular encoding scheme used.

p-0028Clients <b>110</b> may, but are not required to, further include systems configured for modifying received video. For example, a client may be configured to perform further rendering, to overlay one video image on another video image, to crop a video image, and/or the like. Clients <b>110</b> may be configured to receive various types of video frames, such as I-frames, P-frames and B-frames, and to process these frames into images for display to a user. In some embodiments, a member of Clients <b>110</b> is configured to perform further rendering, shading, conversion to 3-D, or like operations on the video stream. A member of Clients <b>110</b> is optionally configured to receive more than one audio or video stream. Input devices of Clients <b>110</b> may include, for example, a keyboard, a joystick, a pointing device, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.

p-0029The video stream (and optionally audio stream) received by Clients <b>110</b> is generated and provided by Video Server System <b>120</b>. As is described further elsewhere herein, this video stream includes video frames (and the audio stream includes audio frames). The video frames are configured (e.g., they include pixel information in an appropriate data structure) to contribute meaningfully to the images displayed to the user. A meaningful contribution is a contribution that is readily observable by a game player. As used herein, the term “video frames” is used to refer to frames including predominantly information that is configured to contribute to, e.g. to effect, the images shown to the user. Most of the teachings herein with regard to “video frames” can also be applied to “audio frames.” Clients <b>110</b> may be configured to receive more than one video stream at the same time. For example, Client <b>110</b>B may be configured to receive a matched pair of video streams configured to form a three dimensional image when one of the streams is presented to one eye and the other stream is presented to the other eye.

p-0030Clients <b>110</b> are typically configured to receive inputs from a user. These inputs may include game commands configured to change the state of the video game or otherwise affect game play. The game commands can be received using input devices and/or may be automatically generated by computing instructions executing on Clients <b>110</b>. The received game commands are communicated from Clients <b>110</b> via Network <b>115</b> to Video Server System <b>120</b> and/or Game Server <b>125</b>. For example, in some embodiments, the game commands are communicated to Game Server <b>125</b> via Video Server System <b>120</b>. In some embodiments, separate copies of the game commands are communicated from Clients <b>110</b> to Game Server <b>125</b> and Video Server System <b>120</b>. The communication of game commands is optionally dependent on the identity of the command. Game commands are optionally communicated from Client <b>110</b>A through a different route or communication channel that that used to provide audio or video streams to Client <b>110</b>A.

p-0031In various embodiments, Clients <b>110</b> are configured to communicate add-on data between each of Client <b>100</b> and Video Server System <b>120</b>, or between different members of Clients <b>110</b>. For example, an add-on that enables voice communication directly between clients can include the communication of audio data between clients. Add-on data that augments the functionality of a server side computer program may be stored on one of Clients <b>110</b> for use by a server side add-on or for use by a client side add-on. Add-on data may also be stored on Video Server System <b>120</b> or on Add-on Server <b>165</b>, and be used by add-ons located on one o Clients <b>110</b>, Video Server System <b>120</b> or Add-on Server <b>165</b>.

p-0032Game Server <b>125</b> is optionally operated by a different entity than Video Server System <b>120</b>. For example, Game Server <b>125</b> may be operated by the publisher of a multiplayer game. In this example, Video Server System <b>120</b> is optionally viewed as a client by Game Server <b>125</b> and optionally configured to appear from the point of view of Game Server <b>125</b> to be a prior art client executing a prior art game engine. Communication between Video Server System <b>120</b> and Game Server <b>125</b> can occur via Network <b>115</b>. As such, Game Server <b>125</b> can be a prior art multiplayer game server that sends game state information to multiple clients, one of which is Video Server System <b>120</b>. Video Server System <b>120</b> may be configured to communicate with multiple instances of Game Server <b>125</b> at the same time. For example, Video Server System <b>120</b> can be configured to provide a plurality of different video games to different users. Each of these different video games may be supported by a different Game Server <b>125</b> and/or published by different entities. In some embodiments, several geographically distributed instances of Video Server System <b>120</b> are configured to provide game video to a plurality of different users. Each of these instances of Video Server System <b>120</b> may be in communication with the same instance of Game Server <b>125</b>. Communication between Video Server System <b>120</b> and one or more Game Server <b>125</b> optionally occurs via a dedicated communication channel. For example, Video Server System <b>120</b> may be connected to Game Server <b>125</b> via a high bandwidth channel that is dedicated to communication between these two systems.

p-0033Video Server System <b>120</b> comprises at least a Video Source <b>130</b>, an I/O Device <b>145</b>, a Processor <b>150</b>, and non-transitory Storage <b>155</b>. Video Server System <b>120</b> may consist of one computing device or be distributed among a plurality of computing devices. These computing devices are optionally connected via a communications system such as a local area network.

p-0034Video Source <b>130</b> is configured to provide a video stream, e.g., streaming video or a series of video frames that form a moving picture. In some embodiments Video Source <b>130</b> is also configured to provide an audio stream. In some embodiments, Video Source <b>130</b> includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and to maintain a copy of the state of the video game based on the received commands. This game state includes the position of objects in a game environment, as well as typically a point of view. The game state may also include properties, images, colors and/or textures of objects. The game state is typically maintained based on game rules, as well as game commands such as move, turn, attack, set focus to, interact, use, and/or the like. Part of the game engine is optionally disposed within Game Server <b>125</b>. Game Server <b>125</b> may maintain a copy of the state of the game based on game commands received from multiple players using geographically disperse clients. In these cases, the game state is provided by Game Server <b>125</b> to Video Source <b>130</b>, wherein a copy of the game state is stored and rendering is performed. Game Server <b>125</b> may receive game commands directly from Clients <b>110</b> via Network <b>115</b>, and/or may receive game commands via Video Server System <b>120</b>.

p-0035Video Source <b>130</b> typically includes rendering logic, e.g., hardware, firmware, and/or software stored on a computer readable medium such as Storage <b>155</b>. This rendering logic is configured to create video frames of the video stream based on the game state. All or part of the rendering logic is optionally disposed within a graphics processing unit (GPU). Rendering logic typically includes processing stages configured for determining the three-dimensional spatial relationships between objects and/or for applying appropriate textures, etc., based on the game state and viewpoint. The rendering logic produces raw video that is then usually encoded prior to communication to Clients <b>110</b>. For example, the raw video may be encoded according to an Adobe Flash® standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, or the like. The encoding process produces a video stream that is optionally packaged for delivery to a decoder on a remote device. The video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800×600, 1280×720 (e.g., 720p), 1024×768, although any other frame sizes may be used. The frame rate is the number of video frames per second. A video stream may include different types of video frames. For example, the H.264 standard includes a “P” frame and an “I” frame. I-frames include information to refresh all macro blocks/pixels on a display device, while P-frames include information to refresh a subset thereof. P-frames are typically smaller in data size than are I-frames. As used herein the term “frame size” is meant to refer to a number of pixels within a frame. The term “frame data size” is used to refer to a number of bytes required to store the frame.

p-0036In alternative embodiments Video Source <b>130</b> includes a video recording device such as a camera. This camera may be used to generate delayed or live video that can be included in the video stream of a computer game. The resulting video stream, optionally includes both rendered images and images recorded using a still or video camera. Video Source <b>130</b> may also include storage devices configured to store previously recorded video to be included in a video stream. Video Source <b>130</b> may also include motion or positioning sensing devices configured to detect motion or position of an object, e.g., person, and logic configured to determine a game state or produce video-based on the detected motion and/or position.

p-0037Video Source <b>130</b> is optionally configured to provide overlays configured to be placed on other video. For example, these overlays may include a command interface, login instructions, video frames rendered from another game player's point of view, messages to a game player, images of other game players, video feeds of other game players (e.g., webcam video). In embodiments of Client <b>110</b>A that include a touch screen interface, the overlay may include a virtual keyboard, joystick, touch pad, and/or the like. In one example of an overlay a player's voice is overlaid on an audio stream. Video Source <b>130</b> optionally further includes one or more audio sources.

p-0038In embodiments wherein Video Server System <b>120</b> is configured to maintain the game state based on input from more than one player, each player may have a different point of view comprising a position and direction of view. Video Source <b>130</b> is optionally configured to provide a separate video stream for each player based on their point of view. Further, Video Source <b>130</b> may be configured to provide a different frame size, frame data size, and/or encoding to each of Client <b>110</b>. Video Source <b>130</b> is optionally configured to provide 3-D video.

p-0039I/O Device <b>145</b> is configured for Video Server System <b>120</b> to send and/or receive information such as video, commands, add-ons, add-on data, requests for information, a game state, client identities, player identities, game commands, security information, audio data, and/or the like. I/O Device <b>145</b> typically includes communication hardware such as a network card or modem. I/O Device <b>145</b> is configured to communicate with Game Server <b>125</b>, Network <b>115</b>, and/or Clients <b>110</b>. I/O Device <b>145</b> is configured to receive the information from more than one of Clients <b>110</b>. I/O Device <b>145</b> is optionally configured to receive the information as packets using a standard such as TCP or UDP.

p-0040Processor <b>150</b> is configured to execute logic, e.g. software, included within the various components of Video Server System <b>120</b> discussed herein. For example, Processor <b>150</b> may be programmed with software instructions in order to perform the functions of Video Source <b>130</b>, Game Server <b>125</b>, and/or an Add-on Manager <b>160</b>. Video Server System <b>120</b> optionally includes more than one instance of Processor <b>150</b>. Processor <b>150</b> may also be programmed with software instructions in order to execute commands received by Video Server System <b>120</b>, or to coordinate the operation of the various elements of Game System <b>100</b> discussed herein. Processor <b>150</b> may include one or more hardware device. Processor <b>150</b> is an electronic processor.

p-0041Storage <b>155</b> includes non-transitory analog and/or digital storage devices. For example, Storage <b>155</b> may include an analog storage device configured to store video frames. Storage <b>155</b> may include a computer readable digital storage, e.g. a hard drive, an optical drive, or solid state storage. Storage <b>115</b> is configured (e.g. by way of an appropriate data structure or file system) to store video frames, artificial frames, a video stream including both video frames and artificial frames, audio frame, an audio stream, add-ons, add-on data, and/or the like. Storage <b>155</b> is optionally distributed among a plurality of devices. In some embodiments, Storage <b>155</b> is configured to store the software components of Video Source <b>130</b> discussed elsewhere herein. These components may be stored in a format ready to be provisioned when needed.

p-0042Video Server System <b>120</b> optionally further comprises Add-on Manager <b>160</b>. Add-on Manager <b>160</b> is configured to facilitate the use of add-ons to augment computer programs on Video Server System <b>120</b> and/or Clients <b>110</b>. For example, in some embodiments, Add-on Manager <b>160</b> is configured to augment a game program within Video Source <b>130</b>. In some embodiments, Add-on Manager <b>160</b> is configured to augment a game program within Clients <b>110</b>.

p-0043Tasks that may be performed by Add-on Manager <b>160</b> include, but are not limited to, installing an add-on, updating an add-on, executing an add-on, storing add-on data, and communicating add-on data. In some embodiments, Add-on Manager <b>160</b> includes computing instructions configured for redirecting communications between an add-on and the associated computer program. Add-on Manager <b>160</b> may redirect messages from the computer program, originally directed toward a specific location relative to the computer program, to an alternative location. For example, if the computer program included within Video Source <b>130</b> is configured to access add-ons and/or add-on data in a specific directory relative to the computer program, Add-on Manager <b>160</b> may be configured to redirect that access to a different location within Video Server System <b>120</b>, to one of Clients <b>110</b>, or to a separate Add-on Server <b>165</b>. The specific directory in which the computer program is configured to access add-ons may be a subdirectory of a directory in which the computer program is executed. The location to which access is redirected is optionally a virtual environment. Alternatively, if the computer program is within Client <b>110</b>B and is configured to access add-ons within a specific directory of Client <b>110</b>B, then Add-on manager may be configured to redirect this access to a location within Video Server System <b>120</b> and/or Add-on Server <b>165</b>. As such, an add-on on Client <b>110</b>B can be used to augment a computer program on Video Server System <b>120</b>, or vice versa. Likewise, an add-on on Add-on Server <b>165</b> can be used to augment a computer program on either Clients <b>110</b> or Video Server System <b>120</b>. A virtual environment is an environment in which some location or physical device, such as a storage location or path, is simulated using software.

p-0044Add-on Manager <b>160</b> includes hardware, firmware, and/or software stored on a computer readable medium. All or part of Add-on Manager <b>160</b> is optionally disposed on a computing device separate from one or more other elements of Video Server System <b>120</b>. For example, all or part of Add-on Manager <b>160</b> is optionally disposed on Clients <b>110</b>, Add-on Server <b>165</b>, or at some other location within Game System <b>100</b>. Further details of Add-on Manager <b>160</b> are discussed elsewhere herein.

p-0045Optional Add-on Server <b>165</b> is a system configured for storage and/or execution of one or more add-ons. Add-on Server <b>165</b> may also be used to store add-on data. Communications between Add-on Server <b>165</b> and Video Server System <b>120</b> and/or Clients <b>110</b> can occur directly or through Network <b>115</b>. Add-on Server <b>165</b> is optionally managed by the publisher or developer of an add-on. In some embodiments, communications from a computer program are redirected to a location of Add-on Server <b>165</b>, from a location proximate to the computer program. For example, access to add-on data or an add-on executable may be redirected from Client <b>110</b>B or Video Server System <b>120</b> to Add-on Server <b>165</b>. Game System <b>100</b> may include more than one Video Server System <b>120</b> and/or more than one Add-on Server <b>165</b>.

p-0046Add-on Server <b>165</b> includes hardware, firmware, and/or software stored on a computer readable medium. For example, in some embodiments Add-on Server <b>165</b> includes a computing device, digital storage, an electronic processor and I/O devices. The digital storage can include add-on executable code, add-on data, images, all or part of Add-on Manager <b>160</b>, computing instructions configured to manage add-on's communication with other elements of Game System <b>100</b>, and/or the like.

p-0047<figref idrefs="DRAWINGS">FIG. 2</figref> is a block diagram of embodiments of Video Source <b>130</b> configured to serve multiple video games augmented by add-ons, according to various embodiments of the invention. The multiple video games can include multiple instances of the same video game and/or instances of different video games. The video games are optionally multiplayer games. For example, a game state of one or more of the video games may be maintained by one or more instances of Game Server <b>125</b> each based on inputs received from multiple clients. The elements of Video Source <b>130</b> illustrated in <figref idrefs="DRAWINGS">FIG. 2</figref> are optionally executed using Processor <b>150</b>.

p-0048The video games are executed using different instances of Game Logic <b>210</b>, individually labeled <b>210</b>A, <b>210</b>B, <b>210</b>C, etc. Game Logic <b>210</b> is configured to use a game state to determine a game environment that can be rendered to a video stream configured to be presented to a game player. The game environment is a two or three dimensional virtual environment including in-game objects, avatars, locations of objects, their shapes, textures, and spatial relationships there between, and the like. A game environment can include vertex data, transformation data and texture data, and/or the like.

p-0049The rendering of the game environment is typically based on one or more points of view associated with a specific game player. These points of view can be drastically different, such as an overhead view and a forward view. Or, the points of view can be from positions differing by approximately the distance between a person's pupils, and configured to create a 3D image. Video Source <b>130</b> may include more than 1, 2, 3, 5, 7, 15, or 31 instances of Game Logic <b>210</b>. Game Logic <b>210</b> is optionally a client of Game Server <b>125</b> and may be configured to communicate with Game Server <b>125</b> via Network <b>115</b>.

p-0050Game Logic <b>210</b> is configured to receive game commands from one or more of Clients <b>110</b> and to process the received commands according to a set of game rules. These rules cover, for example, how avatars interact with other avatars or in game objects, avatar movement, game instance management, and/or the like. Game Logic <b>210</b> is optionally also configured to generate audio data based on events within the game. This audio data may represent a gunshot, a splash, an engine, voice, flying, rain, music, or any other sound that could occur in a game. For example, an event such as one object hitting another may result in audio data representing a related sound. Game Logic <b>210</b> includes hardware, firmware, and/or software stored on a computer readable medium.

p-0051Each instance of Game Logic <b>210</b> can be disposed on a separate computing device or several instances of Game Logic <b>210</b> could be disposed on the same computing device or a single instance of Game Logic <b>210</b> could be disposed across multiple computing devices. Instances of Game Logic <b>210</b> can dynamically change the number and/or identify of computing devices used depending on the requirements of the game environment the user is currently experiencing. Instances of Game Logic <b>210</b> disposed on the same computing device are optionally executed within separate virtual machines or virtual I/O shells. In some embodiments, different instances of Game Logic <b>210</b> are configured to communicate game commands and/or game state information directly to each other, e.g., without necessarily communication through Game Server <b>125</b>.

p-0052Game Logic <b>210</b> typically executes on top of an Operating System (OS) <b>215</b>. Operating System <b>215</b> may include Windows™, Linux, Unix, Mac OS™, Solaris™, and/or the like. A virtual machine platform may operate between Operating System <b>215</b> and Game Logics <b>210</b>. The virtual machine platform can include commercial systems such as ESX, Hyper-V, and/or the like. In these embodiments, one or more of Game Logic <b>210</b> can be executed within a virtual machine. Multiple instances of Game Logic <b>210</b> may execute on the same instance of Operating System <b>215</b>. For example, <figref idrefs="DRAWINGS">FIG. 2</figref> illustrates Game Logic <b>210</b>A and Game Logic <b>210</b>B both executing on the same Operating System <b>215</b>. Instances of Game Logic <b>210</b> executing on the same Operation System <b>215</b> may, or may not, be configured for playing the same video game. For example, Game Logic <b>210</b>A, <b>210</b>B and <b>210</b>C may all be World of Warcraft® clients, or may be clients of World of Warcraft®, Eve® and Call to Arms® respectively.

p-0053In some embodiments, the game environments determined by Game Logic <b>210</b> are passed to an optional Virtual 3D Video Driver <b>220</b>. Virtual 3D Video Driver <b>220</b> is configured to appear, from the point of view of Game Logic <b>210</b>, as a non-virtual 3D video driver controlling a graphics processing unit. Each instance of Game Logic <b>210</b> may be associated with its own instance of Virtual 3D Video Driver <b>220</b>, or Virtual 3D Video Driver <b>220</b> may be shared by two or more instances of Game Logic <b>210</b>. In some embodiments, game audio determined by each Game Logic <b>210</b> is passed to an optional Virtual Audio Driver <b>285</b>. Game audio includes sound data configured to be presented to a game player as part of an audio stream. The game audio can result from rules of the game, e.g., shooting a gun should result in a popping sound, or may be received from other sources such as music tracks, members of Clients <b>110</b>, a microphone, and/or the like.

p-0054In alternative embodiments, Game Logic <b>210</b> is configured to include the functionality, as discussed herein, of Virtual 3D Video Driver <b>220</b> and/or Virtual Audio Driver <b>285</b>. In these embodiments, Virtual 3D Video Driver <b>220</b> and/or Virtual Audio Driver <b>285</b> are not required.

p-0055Virtual 3D Video Driver <b>220</b> is further configured to pass the received game environments to a (non-virtual) 3D Driver <b>230</b>. Optionally the delivery of game environments to 3D Driver <b>230</b> is coordinated by the various instances of Virtual 3D Video Driver <b>220</b>. For example, delivery can be coordinated such that 3D Driver <b>230</b> receives game environments from only one or a minimum number of Virtual 3D Video Driver <b>220</b> at a time. In typical embodiments, each of Virtual 3D Video Driver <b>220</b> is configured such that they appear to be a separate process and a separate source of video data to 3D Driver <b>230</b>. As such, 3D Driver <b>230</b> is configured to keep track of which video data results in which video frames after rendering.

p-0056The video data received by 3D Driver <b>230</b> are passed to Graphics Processing Unit (GPU) <b>235</b> for rendering into raw video frames. Graphics Processing Unit <b>235</b> is optionally used to render more than one video stream in parallel. For example, Graphics Processing Unit <b>235</b> may generate a first video stream based on video data received from Game Logic <b>210</b>A, generate a second video stream based on video data received from Game Logic <b>210</b>B, generate a third video stream based on video data received from Game Logic <b>210</b>C, etc. in parallel. The parallel production of video streams includes the generation of these streams at the same time. However, parallel production of video streams may, but does not necessarily, include the processing of individual frames at the same time within Graphics Processing Unit <b>235</b>. For example, in some embodiments 3D Driver <b>230</b> alternatively passes the video data generated by the various members Game Logic <b>210</b> to Graphics Processing Unit <b>235</b>. Data generated by Game Logic <b>210</b>A is used to make a video frame, and subsequently data generated by Game Logic <b>210</b>B is used to make a video frame, etc. In this case, the video streams are produced in parallel while individual frames are produced in series. Alternatively, more than one video frame may be generated within Graphics Processing Unit <b>235</b> at the same time. In this case a first part of Graphics Processing Unit <b>235</b> is used to generate one frame while a second part of Graphics Processing Unit <b>235</b> is used to generate a second frame, the first and second frames arising from video data produced by different Game Logic <b>210</b>. In some embodiments, one set of graphical computation cores is used to generate the first frame while a second set of graphical computation cores is used to generate the second frame, at the same time. The resulting video frames are passed back to the control of 3D Driver <b>230</b>.

p-0057Virtual 3D Video Drivers <b>220</b> are optionally configured to manage the transfer of raw rendered video frames from 3D Driver <b>230</b>. For example, Virtual 3D Drivers <b>220</b> may be used to coordinate the transfer of video frames from the control of 3D Driver <b>230</b> to a Shared Memory <b>240</b>. (Following rendering, the video frames are in a memory of Graphics Processing Unit <b>235</b> or a memory managed by 3D Driver <b>230</b>. In either case they are under the control of 3D Driver <b>230</b>.) As Virtual 3D Video Drivers <b>220</b> manages the communication of video data and frames to and from 3D Driver <b>230</b>, in some embodiments, they are capable of placing the video frames within parts of Shared Memory <b>240</b> associated with specific Video Dynamic Encoding Pipelines (DEPs) <b>245</b>. The Video DEPs <b>245</b> are individually identified as Video DEP <b>245</b>A, Video DEP <b>245</b>B, Video DEP <b>245</b>C, etc. In these embodiments, each Video DEP <b>245</b> is assigned specific memory locations within Shared Memory <b>240</b>, and is configured to retrieve video data from those locations.

p-0058In other embodiments, Virtual 3D Drivers <b>220</b> are configured to manage the transfer of video data to Shared Memory <b>240</b> based on timing. In these embodiments, the transfers managed by each of Virtual 3D Drivers <b>220</b> are synchronized and the Virtual 3D Drivers <b>220</b> notify each Video DEP <b>245</b> when data for them is in Shared Memory <b>240</b>. Once this data is retrieved by the notified member of Video DEPs <b>245</b>, data for another member of DEPs is transferred from 3D Driver <b>230</b> to Shared Memory <b>240</b> under the control of Virtual 3D Video Drivers <b>220</b>. In these cases pixels read back from the local GPU memory to Shared Memory <b>240</b> can be based on a schedule. As used herein the terms “in sync” and “synchronized” are meant to mean that the two events are related in time by a schedule, by a timing signal, by a time delay, or that one event does not start until the other event has satisfied a condition, e.g., has completed. For example, Virtual 3D Drivers <b>220</b> may operate in sync such that a game environment is provided to 3D Driver <b>230</b> from a first of Virtual 3D Drivers <b>220</b> once Graphics Processing Unit <b>235</b> has completed rendering of a game environment from a second of Virtual 3D Drivers <b>220</b>. The timing of this synchronization is optionally selected to make optimum use of Graphics Processing Unit <b>235</b>.

p-0059As a result of the management by Virtual 3D Video Drivers <b>220</b>, the multiple video streams can be stored in Shared Memory <b>240</b> without confusing which frames belong to which video stream. Shared Memory <b>240</b> is optionally configured to store audio as well as video data. This audio data may be stored and/or managed in ways similar to that discussed with respect to video data.

p-0060Virtual Audio Drivers <b>285</b> are optionally part of a virtual I/O shell between Game Logic <b>210</b> and OS <b>215</b>. They are configured to appear from the point of view of Game Logic <b>210</b> as an audio driver and to pass any audio data they receive from Game Logic <b>210</b> to one of Audio DEP <b>270</b>. For example, in some embodiments, the Virtual Audio Driver <b>285</b> associated with Game Logic <b>210</b>A is configured to send audio data to Audio DEP <b>270</b>A, and the Virtual Audio Driver <b>285</b> associated with Game Logic <b>210</b>B is configured to send audio data to Audio DEP <b>270</b>B, etc.

p-0061One or more of Game Logic <b>210</b> are optionally each associated with an Add-on Environment <b>287</b>. Add-on Environments <b>287</b> include hardware, firmware and/or software stored on a computer readable medium and are configured for the associated Game Logic <b>210</b> to access add-ons and/or add-on data. For example, in some embodiments Add-on Environment <b>287</b> includes a directory structure configured to store add-ons and/or add-on data. The specific directory structure can depend on the identity of Game Logic <b>210</b>A or <b>210</b>B. For example, if the Game Logic <b>210</b>A is configured to look for add-ons in a subdirectory called “add-ons” and to look for add-on data in subdirectories of the add-ons directory, then the Add-on Environment <b>287</b> can include these subdirectories or virtual representations of these subdirectories. Typically, Add-on Environments <b>287</b> are configured such that Game Logic <b>210</b>A can operate as if the add-on were installed locally to Game Logic <b>210</b>A, e.g., within the same directory structure or in the same local file system as Game Logic <b>210</b>A. This allows the use of instances of Game Logic <b>210</b>A and add-ons that are configured for installation together on the same computing device, even when Game Logic <b>210</b> and the associated add-ons are installed on separate computing devices coupled by an embodiment of Network <b>115</b> such as the internet.

p-0062In some embodiments Add-on Environments <b>287</b> include computing instructions configured to redirect communications between Game Logic <b>210</b> and add-ons (and add-on data) stored at one of Clients <b>110</b>, Add-on Server <b>165</b>, and/or other locations within Video Server System <b>120</b>. For example, Add-on Environments <b>287</b> can include computing instructions to send and receive messages via an API of Game Logic <b>210</b> and to redirect these messages. Add-on Environments <b>287</b> optionally include a filter configured to identify which messages from the API should be redirected and to redirect particular messages to different locations as needed. A similar filter can be included local to the Add-on. This filter can also be considered part of Add-on Environments <b>287</b> and is configured to receive messages (e.g., data or commands) from the add-on and direct these messages to the associated member of Game Logic <b>210</b>. For example, when an add-on responds to a request or command from Game Logic <b>210</b>A the response to the request or command is redirected back to Game Logic <b>210</b>A. The redirection that occurs in either direction can include communication of data packets via network <b>115</b>. Add-on Environments <b>287</b> are typically generated by or include parts of Add-on Manager <b>160</b>.

p-0063Shared Memory <b>240</b> includes random access memory (RAM) or a similar memory configured for efficient reading and writing of video data. Shared Memory <b>240</b> is configured to store video data for a plurality of different Video DEP <b>245</b>. Video data for different Video DEPs <b>245</b> is optionally stored at the same time in Shared Memory <b>240</b>. Shared Memory <b>240</b> may consist of a single hardware device or may include multiple devices.

p-0064Video DEPs <b>245</b> are dynamically allocated encoding pipelines that are each configured to encode video data rendered using Graphics Processing Unit <b>235</b>. Each member of Video DEPs <b>245</b> is configured to encode to video formats specified at the time the Video DEP <b>245</b> is provisioned. This format specification is typically based on the needs of one of Clients <b>110</b> and/or the capabilities of the communication path between Video Server System <b>120</b> and the Client <b>110</b>. Video DEPs <b>245</b> are optionally provisioned dynamically in response from a request from one of Clients <b>110</b>. For example, when Client <b>110</b>A connects to Video Server System <b>120</b> and sends a request for a video stream, Video DEP <b>245</b> can be provisioned to include elements, such as an encoder, selected specifically to meet needs of Client <b>110</b>A. As is discussed elsewhere herein, a member of Video DEPs <b>245</b> is optionally configured to use more than one encoding scheme.

p-0065Video DEPs <b>245</b> each include a Grabber <b>247</b> configured to grab video data from Shared Memory <b>240</b> and transfer this video data to a memory of the Video DEP <b>245</b>. The Grabber <b>247</b> is optionally under the control of a member of Virtual 3D Video Drivers <b>220</b>. For example, in various embodiments, Grabber <b>247</b> is controlled by a member of Virtual 3D Drivers <b>220</b> to grab the video data from a specific location within Shared Memory <b>240</b> and/or at a specific time in synchronization with the transfer of video data to other members of Video DEPs <b>245</b>.

p-0066Video DEP <b>245</b> each optionally include an Overlay Logic <b>290</b> configured to overlay one or more video images on the video data, e.g., frames, generated by Graphics Processing Unit <b>235</b>. An overlay is a first image to be placed on a second image, or a sound added to another sound. Overlays may be applied with varying levels of transparency. For example, the first image can be opaque and, thus, completely hide covered parts of the second image, or the first image can be partially transparent in which case the second image can still be somewhat visible under the first image. An overlay may cover all or part(s) of the underlying image. For example, an overlay may be configured to cover certain pixels within a larger area. In this case the overlay is mapped to pixels on a display of one of Clients <b>110</b> or mapped to pixels within an image generated by Graphic Processing Unit <b>235</b>. The mapping can be accomplished in various ways. For example, a lower left pixel of the overlay may be assigned to a specific pixel on a video frame generated by Graphics Processing Unit <b>235</b>. In some embodiments, Overlay Logic <b>290</b> is configured to resize an overlay prior to applying the overlay. Overlay Logic <b>290</b> is optionally located in other positions within Video DEP <b>245</b>A. For example, after Preprocessor <b>250</b>.

p-0067The overlays can include a wide variety of pixel/video images. For example, an overlay can include a real-time or static image of a game player received via the internet, a real-time or static image of a sporting (or other) event, an image of a input control (e.g., “a,” “b,” “TAB,” “Return,” “Space Bar,” “Function Key,” “Arrow,” and/or other keys or input devices), a map, text, and/or the like. Overlays can vary widely in size and shape. In some cases the overlay is the result of rendering a 3D game environment. More than one overlay can be placed on an image. These overlays can be overlapping or separate. Overlays are optionally received in an encoded format and decoded prior to being applied.

p-0068The presence and content of overlays are optionally responsive to received game commands, identity of one or more game players, an account type, identity of a game being played, video captured from a prior or real-time sporting event or real-world game, game rules, and/or the content of video generated by Game Logic <b>210</b>. For example, an overlay may include a menu resulting from receipt of a game command. An overlay may include content responsive to a pixel pattern, e.g., image, included within video generated by Game Logic <b>210</b>. Overlay Logic <b>290</b> is optionally configured to apply multiple overlays to a single video frame and these overlays may be from different sources.

p-0069Overlay Logic <b>290</b> is optionally configured to apply an overlay after a video frame has been preprocessed using Preprocessor <b>250</b> and/or after encoding using one of Encoders <b>255</b>. In some embodiments the overlay includes an image received at an input of Video Source <b>130</b> (e.g., I/O device <b>145</b>) via the internet and Overlay Logic <b>290</b> includes video processing logic configured to generate the overlay from the image. The received image is optionally received from an image source such as a camera or a file storage.

p-0070Video DEPs <b>245</b> each optionally also include a Preprocessor (PRE) <b>250</b>. Preprocessor <b>250</b> is configured to perform a color space conversion such as RGB to YUV and/or a scaling operation to increase or decrease the resolution of the video frame. Preprocessor <b>250</b> is optional in embodiments wherein the output of Graphics Processing Unit <b>235</b> is in the YUV color space or some other desired color space. Multiple Preprocessors <b>250</b> may be included in a Video DEP <b>245</b> configured to produce multiple video streams having video frames of different sizes.

p-0071Video DEPs <b>245</b> each include at least one Encoder (ENC) <b>255</b>. Encoders <b>255</b> are individually identified as Encoder <b>255</b>A, Encoder <b>255</b>B, etc. Each of Encoders <b>225</b> is configured to encode the video data according to a specific codec, and optionally a specific color depth and/or frame size. For example, Encoders <b>225</b> may be configured to encode video data to Adobe Flash® standard, .flv, .wav, .avi, .mpg, H.264, H.263, On2, VP6, VC-1, WMA, and/or other codecs discussed herein.

p-0072A member of Video DEPs <b>245</b> may include one, two or more Encoders <b>255</b>. These encoders may be configured to encode to different codecs and/or the different formats of the same codec. For example. Encoder <b>255</b>A may be configured to encode according to a Flash standard at a first frame size and color depth while Encoder <b>255</b> is configured to encode to the same Flash standard at a second frame size and color depth. The identity of Encoders <b>255</b> within each member of Video DEPs <b>245</b> is typically determined at the time the Video DEP <b>245</b> is provisioned. For example, a single command or command set may be used to create (provision) Video DEP <b>245</b>A and specify which components the created Video DEP <b>245</b>A should include. The creation of Video DEP <b>245</b>A is discussed further elsewhere herein. A member of Video DEPs <b>245</b> including two or more Encoders <b>255</b> is alternatively viewed as two or more separate Video DEPs <b>245</b> having some but not all components in common. For example, Video DEP <b>245</b>A and Video DEP <b>245</b>B may have the same Preprocessor <b>250</b> but different Encoders <b>255</b>.

p-0073In one example, Encoder <b>255</b>A is configured to use a codec for H.264 while Encoder <b>255</b>B is configured to use a codec for H.263. Having two or more different encoders available enables Video DEP <b>245</b>A to change encoding during delivery of a video stream. The change in encoding can be from one type of encoding to another, or merely a change in characteristics of a specific type of coding. For example, the characteristics may change in terms of color depth, number of frames per second, encoding options, number of pixels, and/or the like. In some embodiments, Video DEP <b>245</b>A is optionally configured to switch between Encoder <b>255</b>A and <b>255</b>B in response to a change in the characteristics of Client <b>110</b>A or the communication channel between Client <b>110</b>A and Video Source <b>130</b>.

p-0074In practice, when a different codec other than that being used is required a new Video DEP <b>245</b> is spawned and executed in parallel with the current Video DEP <b>245</b> for a short period of time. The new Video DEP <b>245</b> is optionally a branch of the original Video DEP <b>245</b>. For example, some components of the original Video DEP <b>245</b> may be used in the new Video DEP <b>245</b>. These two Video DEP <b>245</b> may have components logically arranged in a tree structure.

p-0075In some embodiments, Video DEP <b>245</b> are configured to use two or more different encoders to generate two or more different video streams at the same time. These video streams are based on the same game environment rendered by Graphics Processing Unit <b>255</b> and, thus, include essentially the same materials (with the possible exception of overlays) and can be sent to different places. For example, one of the video streams can be sent to Client <b>110</b>A while the other is sent to Client <b>110</b>B. Alternatively, one of the video streams can be sent to Client <b>110</b>A and the other to a website where third parties can watch the video. This website is optionally part of a social networking site or a game player's site. The two different video streams may be different in the frame rate, encoding type, frame size, color depth, etc. For example a video stream delivered to a social networking website can be of much lower quality than the video stream delivered to a game player who is playing a game using Client <b>110</b>A. The second video stream may be directed to a game player who is playing the game or to people who are merely observing the game play. A video stream is optionally directed to more than one place.

p-0076Video DEP <b>245</b>A optionally includes one or more Post Processors (Post) <b>260</b>. Individual examples of Post Processors <b>260</b> are labeled <b>260</b>A and <b>260</b>B. Post Processors <b>260</b> are configured to package an encoded video frame in a container in a format appropriate for communication over a network according to a public or proprietary network protocol. For example, some protocols such as Adobe RTMP require post processing while other video standards such as H.264 Annex B do not require post processing. Each of Post Processors <b>260</b> may be associated with a specific member of Encoders <b>255</b>, or several Post Processors <b>260</b> may be configured to receive encoded video frames from one member of Encoders <b>255</b>.

p-0077The output of Post Processors <b>260</b> is directed to I/O Device <b>145</b> for delivery to one or more of Clients <b>110</b>. The Elements of Video DEPs <b>245</b> discussed herein include hardware, firmware and/or software stored on a computer readable medium. For example, each of Video DEPs <b>245</b> may represent a set of software loaded into memory and executing using an electronic processor.

p-0078Audio DEPs <b>270</b> are configured to receive audio data generated by members of Game Logic <b>210</b> and encode this audio data to audio packets. The encoded data packets are then sent to Clients <b>110</b> using I/O Device <b>145</b> and Network <b>115</b>. Audio DEPs <b>270</b> are also optionally configured to place an audio overlay on the audio data received from Game Logic <b>210</b>. Different Audio DEPs <b>270</b> are individually labeled <b>270</b>A, <b>270</b>B, <b>270</b>C, etc.

p-0079As with Video DEPs <b>245</b>, Audio DEPs <b>270</b> are typically allocated as needed and assigned to process audio data from a specific member of Game Logic <b>210</b>. This audio data may be based on events within a video game. For example, actions taken by one or more players of the video game may result in audio data according to game rules (e.g., falling in water makes a splashing sound). Allocation of Audio DEP <b>270</b>A is typically responsive to the same factors and performed in the same manner as allocation of Video DEP <b>245</b>A. For example, the elements included in Audio DEP <b>270</b>A may be responsive to needs of one of Clients <b>110</b>. Audio DEP <b>270</b> may receive audio data from Shared Memory <b>240</b> and/or directly from one of Virtual Audio Drivers <b>285</b>.

p-0080Audio DEP <b>270</b>A optionally includes a Grabber <b>275</b> configured to receive audio data from one of Virtual Audio Drivers <b>285</b>, to receive audio data from one or more sources external to Video Server System <b>120</b>, and/or to get audio data from Shared Memory <b>240</b>. For example, in some embodiments Grabber <b>275</b> is configured to retrieve data received from sources external to Video Source <b>130</b>, such as one or more of Clients <b>110</b>, over the internet. The audio received, from any of these sources, can include voice audio, music, sound effects, sound tracks, alerts, and/or the like. For example, the voice of a game player using Client <b>110</b>A may be received from Client <b>110</b>A, or the game player may provide music to be heard by a game player using another of Clients <b>110</b>. Audio data may be received, as audio streams, from more than one game player in parallel.

p-0081Audio DEP <b>270</b>A optionally further includes Overlay Logic <b>295</b>. Overlay Logic <b>295</b> is configured to place one or more audio overlays on audio data generated by a member of Game Logic <b>210</b>. The audio overlays can include audio, music, sound effects, sound tracks, alerts, and/or the like. For example, in some embodiments, an audio overlay includes voice data received as data packets from one or more game players over the internet. These data packets are optionally decoded by Overlay Logic <b>295</b> prior to being included in the overlay. Placement of overlays on the audio data generated by a member of Game Logic <b>210</b> optionally includes addition of this data to the generated audio data. All or part of an audio overlay can be received from a source external to Audio Source <b>130</b>. For example, in some embodiments the overlay includes audio data received at I/O Device <b>145</b> via the internet and Overlay Logic <b>290</b> includes audio processing logic configured to generate an audio overlay from this audio data. When referring to audio overlays, the identities of which data is the overlay and which is the data on which the overlay is being applied may be arbitrary as, in some embodiments, audio data does not have a sense of layering order that can be possible in video data.

p-0082Audio DEP <b>270</b>A further includes an Audio Encoder <b>280</b>A. Audio Encoder <b>280</b>A is configured to encode the audio data generated by a member of Game Logic <b>210</b> and the audio overlay into an audio packet according to an audio codec. The encoding is typically performed after the audio overlay(s) have been applied to the audio data.

p-0083Grabber <b>275</b>, Audio Encoder <b>280</b>A and Overlay Logic <b>295</b> include hardware, firmware, and/or software stored on a computer readable medium. Audio DEP <b>270</b>B and Audio DEP <b>270</b>C are alternative embodiments of Audio DEP <b>270</b>A.

p-0084The operation of Video Source <b>130</b> is typically managed by a Controller <b>265</b>. Controller <b>265</b> includes hardware, firmware and/or software stored on a computer readable medium. For example Controller <b>265</b> may include software stored in memory and executed using a microprocessor.

p-0085In some embodiments, Controller <b>265</b> is configured to provision instances of Game Logic <b>210</b>A, Virtual Audio Driver <b>285</b> and Virtual 3D Video Driver <b>220</b> in response to a request to play a game. For example, if a request for a specific video game is received from Client <b>110</b>A, Controller <b>265</b> may retrieve Game Logic <b>210</b>A, Virtual Audio Driver <b>285</b> and Virtual 3D Video Driver <b>220</b> from Storage <b>155</b> and place these in working memory. Game Logic <b>210</b>A may be placed in communication with both Client <b>110</b>A and Game Server <b>125</b>. Game Logic <b>210</b>A is then executed using Processor <b>150</b> in order to play the video game.

p-0086In some embodiments, Controller <b>265</b> is configured to allocate instances of Video DEP <b>245</b> and/or Virtual Audio Driver <b>285</b> in response to a request to play a game. For example, Controller <b>265</b> may be configured to first determine or receive the requirements for Video DEP <b>245</b>A and Audio DEP <b>285</b>A and then provision computing instructions that satisfy those requirements. Video DEP <b>245</b>A and Audio DEP <b>285</b> are then associated with one of Game Logic <b>210</b> and one of Virtual 3D Driver <b>220</b>. The provisioned Video DEP <b>245</b>A is optionally configured to retrieve raw video data from a specific area within Shared Memory <b>240</b>.

p-0087In some embodiments the requirements for Video DEP <b>245</b>A and/or Audio DEP <b>285</b>A are determined by querying a user of Client <b>110</b>A. For example, Controller <b>265</b> may be configured to cause text messages to appear on Client <b>110</b>A, the messages requesting that a user of Client <b>110</b>A enter characteristics of a video stream desired by a user. These characteristics can include a connection type, a frame size, an encoding scheme, a frame rate, a color depth, and/or the like. The entered characteristics are conveyed back to Controller <b>265</b>, which then used these characteristics to select elements to include in Video DEP <b>245</b>A and/or Audio DEP <b>285</b>A when they are provisioned. For example, if the user requests a specific encoding scheme, then an Encoder <b>255</b> configured to use the requested encoding scheme is selected and included in Video DEP <b>245</b>A.

p-0088In some embodiments the requirements for Video DEP <b>245</b>A and/or Audio DEP <b>285</b>A are determined automatically. For example, information gathered using a client qualifier can be used to determine the requirements for Video DEP <b>245</b>A. In this example, if Client <b>110</b>A includes only a Microsoft Silverlight® decoder and an Adobe Flash® decoder, then at least one of the corresponding encoder configurations or variants thereof is required in Video DEP <b>245</b>A. Characteristics of the video stream to be generated using Video DEP <b>245</b>A, such as frame size, color depth and frame rate, can also be determined by Controller <b>265</b> in this way.

p-0089The determination of requirements for Video DEP <b>245</b>A and/or Audio DEP <b>270</b> may include both the use of characteristics provided by a user and characteristics automatically determined. In some embodiments, the possible characteristics are limited by a user's account type, payment and/or subscription. For example, a user may pay to receive a higher quality video stream.

p-0090Controller <b>265</b> may also manage sharing of Shared Memory <b>240</b>. For example, Controller <b>265</b> may configure Video DEP <b>245</b>A and/or Audio DEP <b>285</b>A to use specific memory locations within Shared Memory <b>240</b> and/or Controller <b>265</b> may be configured to configured Video DEP <b>245</b>A and/or Audio DEP <b>285</b>A to access Shared Memory <b>240</b> in response to certain timing signals. The timing signals being timed for synchronization of access to Shared Memory <b>240</b> by different Video DEPs <b>245</b>.

p-0091<figref idrefs="DRAWINGS">FIG. 3</figref> illustrates further details of Add-on Manager <b>160</b>, according to various embodiments of the invention. In general Add-on Manager <b>160</b> is configured to enable remote selection and installation of add-ons, to facilitate execution of add-ons using add-on environments, to manage add-on data, to associate add-ons and add-on data with specific user (game player) accounts or avatars, and/or the like. Add-on Manager <b>160</b> typically includes software stored on a computer readable medium, but may also include firmware or hardware. Add-on Manager <b>160</b> can be disposed on a single computing device or distributed among more than one computing device. For example, in various embodiments, components of Add-on Manager <b>160</b> are disposed on Video Server System <b>120</b>, Add-on Server <b>165</b> and/or Client <b>110</b>B.

p-0092Add-on Manager <b>160</b> includes an Add-on Installer <b>310</b> configured for installing add-ons in association with a computer program. As used herein, the association of an add-on and computer program is meant to indicate that the add-on is configured to augment or otherwise extend the functionality of the computer program. Typically, the add-on is configured for association with a specific computer program. The installation is optionally performed remotely, e.g., an installation at Video Server System <b>120</b> can be performed in response to commands received from Client <b>110</b>B via Network <b>115</b>. These commands are received and processed by Add-on Manager <b>160</b>.

p-0093The installation performed by Add-on Installer <b>310</b> can include creation of an Add-on Environment <b>287</b> having characteristics compatible with execution of the add-on. As discussed elsewhere herein, an Add-on Environment <b>287</b> can include a directory structure, computing instructions configured to redirect communications, etc. For example, the installation can include the creation of directories, subdirectories, or other elements of a directory structure, relative to Game Logic <b>210</b>. In some embodiments, the installation includes the addition of computing instructions configured to redirect messages to other devices. The installation of these computing instructions can be at locations of Game Logic <b>210</b> and/or of the add-on.

p-0094Installation can be commenced and controlled via an Add-on Interface <b>320</b>. Add-on Interface <b>320</b> includes a user interface configured for presentation to users of Clients <b>110</b>, computing instructions configured to generate and communicate this user interface via Network <b>115</b>, and/or computing instructions configured to receive commands entered by the user via the interface. In various embodiments, the user interface is configured for a user to select add-ons to be installed, pay for add-on or add-on data storage space, select which add-ons should be activated for particular avatars, turn on and off add-ons, update add-ons, to managed an approved add-on list, and/or de-install add-ons. Selections and the entry of commands is typically accomplished by clicking or entering text, etc., in that part of Add-on Interface <b>320</b> that is presented by a user. The results of these actions are communicated to other parts of Add-on Interface <b>320</b> for processing. Part of Add-on Interface <b>320</b> is optionally included in a video stream communicated to Clients <b>110</b> as an overlay.

p-0095Using Add-on Interface <b>320</b> a user can install one or more add-ons remotely on Video Server System <b>120</b> or Add-on Server <b>165</b>, and/or install appropriate computing instructions for redirection at locations of add-ons and Game Logic <b>210</b> remote from Clients <b>110</b>. For example, a user may select one or more add-ons from a list of approved add-ons. This list of approved add-ons is optionally stored in Storage <b>155</b> or Add-on Server <b>165</b>. Approved add-ons are add-ons that have been qualified for use with Game Logic <b>210</b> by a game publisher, a manager of Video Server System <b>120</b>, or other authority. Using Add-on Interface <b>320</b> the user can optionally also select which account, avatar and/or game the add-ons should be associated with and where the add-on should be stored, e.g., at one of Clients <b>110</b>, at Video Server System <b>120</b>, and/or at Add-on Server <b>165</b>. For example, the user may specify that an add-on should only be used with one or more specific avatar but not with another avatar.

p-0096In some embodiments of the invention a user is limited in the number of add-ons and/or amount of add-on data storage the user has available. This limit is managed by Add-on Manager <b>160</b> and may be increased by purchasing additional space. For example, in one embodiment, a user receives three virtual “slots” in which add-ons can be placed and can purchase additional slots by paying money or performing some action such as bringing in additional players.

p-0097Add-on Manager <b>160</b> is optionally configured for a user to turn on and off installed add-ons. For example, a user may wish to use an add-on only when engaging in certain in-game activities such as player versus player encounters. Turning off an add-on does not necessarily mean that it is de-installed. In some embodiments, add-ons that are off do not require an add-on slot. In these embodiments, a user can have more add-ons installed than available slots. Alternatively, each installed add-on, whether or not active, may require a slot.

p-0098In some embodiments Add-on Manager <b>160</b> is configured to update add-ons. These updates can be manual and/or automatic. Manual updates include, for example, a user providing Add-on Manager <b>160</b> with a web address of an add-on publisher or with add-on code. The add-on can be retrieved by the Add-on Manager <b>160</b> from the publisher or from one of Clients <b>110</b>. Automatic add-on update includes automatic retrieval of an add-on from an add-on publisher or other source. For example, when an add-on is detected as being out of date, Add-on Manager <b>160</b> is optionally configured to automatically request an update from a publisher or other source of that add-on and automatically install the update if available. This process can be transparent to the game player. In some instances, Add-on Manager <b>160</b> will wait until an add-on is available from the source and then perform the update.

p-0099With either manual or automatic add-on updates, Add-on Manager <b>160</b> is optionally automatically configured to detect when an add-on is out of date and notify the user. Add-on Manager <b>160</b> may also be configured to receive a notice from an add-on publisher that a new version of an add-on is available.

p-0100Add-on Manager <b>160</b> is optionally configured for a user of Clients <b>110</b> to manage add-ons via Add-on Interface <b>320</b>. This management performed via Add-on Interface <b>320</b> can include, for example, updating add-ons, purchasing add-ons, configuring add-ons, moving add-ons, viewing installed add-ons, activating and inactivating add-ons, installing add-ons, de-installing add-ons, and/or the like. In some embodiments, Add-on Interface <b>320</b> is configured to perform any add-on management function that could be performed on a client side add-on. The de-installation optionally includes the removal of computing instructions configured for the redirection of messages between an add-on and Game Logic <b>210</b>, and/or the removal of an Add-on Environment <b>287</b>. As discussed elsewhere herein, in some embodiments Add-on Manager <b>160</b> is configured to automatically update add-ons. The update mode is optionally managed using Add-on Interface <b>320</b>.

p-0101Add-on Manager <b>160</b> optionally further includes an Account Manager <b>325</b> configured to manage the association of add-ons and/or add-on data with a user of Client <b>110</b>B or other Clients <b>110</b>. When an add-on is installed by a user (game player), Account Manager <b>325</b> associates the add-on with a specific video game, user, account of the user, and/or avatar. Typically this association is stored as part of a list of add-ons installed by each user. When a user starts a video game, Account Manager <b>325</b> reads this list and identifies add-ons associated with the user. These add-ons and the appropriate add-on environments are then provisioned along with the provisioning of Game Logic <b>210</b>, Video DEP <b>245</b>A and/or Audio DEP <b>270</b>A. Add-ons are optionally further provisioned or de-provisioned as the user switches between avatars or activates and de-actives add-ons.

p-0102Account Manager <b>325</b> manages storage of add-on data in association with a user when the user is not playing a video game. The add-on data may be stored at Storage <b>155</b>, Add-on Server <b>165</b> and/or Client <b>110</b>. When game play is started this data is typically copied to the appropriate Add-on Environment <b>287</b> for use by one or more provisioned add-on. When game play is completed the copy of the add-on data in the Add-on Environment <b>287</b> is used to update the stored copy. The add-on data is stored in a database, file structure or the like and includes identifying information, such as a record index or a file name, configured to identify the add-on data as belonging to a specific user (i.e., game player). In some embodiments Account Manager <b>325</b> maintains an index configured to indicate which files of add-on data belong to a specific user and/or are associated with a specific account or avatar.

p-0103In various embodiments, Account Manager <b>325</b> is configured to manage the number of add-on slots and/or storage accessible to each user, to charge the user for extra slots and/or storage, and/or to offer the user extra slots or storage for purchase. For example, if a user attempts to activate more add-ons than there are slots available, Account Manager may use Add-on Interface <b>320</b> to offer the user an opportunity to purchase more slots.

p-0104Add-on Manager <b>160</b> optionally further includes an Add-on Updater <b>330</b>, configured to update add-ons. Updates can occur when a new version of an add-on is available or in response to new versions of Game Logic <b>210</b>. For example, when a new version of Game Logic <b>210</b> is released by a game publisher third party add-ons for that game will typically need to be tested and/or updated. Add-on Updater <b>330</b> is configured to monitor for such events, e.g., release of new versions of add-ons and Game Logic <b>210</b>, and to perform updates in response. In some embodiments, the process includes receiving information that indicates that a new version of Game Logic <b>210</b> is available, notifying add-on publishers that an updated version of an add-on is required, receiving the updated add-on, and replacing the out of date add-on with the updated add-on. Any or all of these steps can be performed automatically or can involve input from a user of Client <b>110</b>B. In some embodiments, Add-on Updater <b>330</b> is configured to automatically retrieve add-ons from a third party location, such as an ftp server or website.

p-0105Add-on Manager <b>160</b> further includes an Add-on Storage <b>335</b>. Add-on Storage <b>335</b> includes at least one electronic storage device configured to store data, images, and/or executable computing instructions. For example, Add-on Storage <b>335</b> can include volatile or non-volatile memory, a hard drive, and optional drive, static or dynamic random access memory, and/or the like. Add-on Storage <b>335</b> is optionally distributed among a plurality of devices including Video Server System <b>120</b>, Add-on Server <b>165</b> and Clients <b>110</b>.

p-0106Add-on Storage <b>335</b> optionally includes Add-ons <b>340</b>. Add-ons <b>340</b> comprise data, images, executable code, scripts, or other game content configured to be used to extend the functionality of Game Logic <b>210</b>. Add-ons <b>340</b> may be installed as add-ons on Video Server System <b>120</b>, Add-on Server <b>165</b> Clients <b>110</b>, or elsewhere. Add-ons <b>340</b> are optional in embodiments where add-ons are retrieved for execution directly from third party sources. Add-on Storage <b>355</b> optionally further includes an Approval List <b>345</b> of approved Add-ons <b>340</b>. The computing instructions within Add-ons <b>340</b> are optionally shared by more than one game player. For example, a single add-on, when provisioned, can be associated with both Game Logic <b>210</b>A and Game Logic <b>210</b>B, different computing threads independently accessing the computing instructions.

p-0107Add-on Storage <b>355</b> optionally includes API Filters <b>350</b>. API Filters <b>350</b> comprise the computing instructions configured to redirect communications between one of Add-ons <b>340</b> and Game Logic <b>210</b>, as discussed elsewhere herein. Such computing instructions may be configured such that an add-on on Add-on Server <b>165</b> or one of Clients <b>110</b> can be used to extend functionality of Game Logic <b>210</b> on Video Server System <b>120</b>. Alternatively, using API Filter <b>350</b> an add-on on Video Server System <b>120</b> or on Add-on Server <b>165</b> may be used to extend the functionality of Game Logic <b>210</b> installed on one of Clients <b>110</b>. When API Filters <b>350</b> are installed in proper locations, an add-on can be used to extend functionality of a computer program even when the add-on and computer program a separated by Network <b>115</b>.

p-0108Add-on Storage <b>355</b> optionally includes Add-on Data <b>355</b> included with, generated or organized by an add-on. Add-on Data <b>355</b> may include data generated by more than one add-on. Typically, this data is copied to the appropriate Add-on Environment <b>287</b>, when game play is initiated or an Add-on <b>340</b> is activated. When game play is completed the add-on data in the Add-on Environment <b>287</b> is used to update the copy of the add-on data stored in Add-on Storage <b>355</b>.

p-0109<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates further details of a client, according to various embodiments of the invention. I/O <b>405</b> includes an input/output device configured for Client <b>110</b> to send and/or receive information such as video, commands, add-ons, add-on data, requests for information, a game state, client identities, player identities, game commands, security information, audio data, and/or the like. In some embodiments, I/O includes a modem or Ethernet port.

p-0110Client-side Storage <b>410</b> includes non-transitory analog and/or digital storage devices. For example, Client-side Storage <b>410</b> may include an analog storage device configured to store video frames. Client-side Storage <b>410</b> may include a computer readable digital storage, e.g. a hard drive, an optical drive, or solid state storage. Client-side Storage <b>410</b> is configured (e.g. by way of an appropriate data structure or file system) to store video frames, artificial frames, a video stream including both video frames and artificial frames, audio frame, an audio stream, add-ons, add-on data, and/or the like. Some of these add-ons can be associated with the instance of Game Logic <b>210</b> on Client <b>110</b>B, while the same Game Logic <b>210</b> is also associated with add-ons installed on a device geographically remote form Client <b>110</b>B.

p-0111API Filter <b>350</b> is part of an Add-on Environment <b>287</b> configured to redirect communications as discussed elsewhere herein. Processor <b>420</b> includes both a central processing unit (CPU) and at least one graphics processing unit (GPU). Display <b>430</b> is configured to display a video stream generated local to Client <b>110</b>B and/or generated by Video Server System <b>120</b>.

p-0112User Input <b>440</b> includes, for example, a keyboard, a joystick, a pointing device, a telephone, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.

p-0113<figref idrefs="DRAWINGS">FIG. 5</figref> illustrates methods of installing and/or using an add-on, according to various embodiments of the invention. These methods may be applied in embodiments in which the add-on is installed on Add-on Server <b>165</b>, Video Server System <b>120</b>, or one of Clients <b>110</b>, and embodiments in which Game Logic <b>210</b> is installed on one of Clients <b>110</b> or Video Server System <b>120</b>. Common to all these embodiments is that at least one of an add-on and the associated Game Logic <b>210</b> are installed and used at a location other than one of Clients <b>110</b> and separated from the one of Clients <b>110</b> by Network <b>115</b>. For example, if Game Logic <b>210</b> is installed on Client <b>110</b>B then at least one associated add-on is installed and used on a different member of Clients <b>110</b>, Add-on Server <b>165</b>, Video Server System <b>120</b>, or a third party location. If Game Logic <b>210</b> is installed in a location other than Client <b>110</b>B then add-ons can be installed on Client <b>110</b>B, or any of the other locations discussed herein. Those steps of <figref idrefs="DRAWINGS">FIG. 5</figref> involving installation of add-ons are optionally performed separately from those steps involving using the installed add-ons. Both installation and execution of an add-on may be initiated by a game player using one of Clients <b>110</b>, or may be specified as part of a default configuration for Game Logic <b>210</b>. Some embodiments include installing more than one add-on.

p-0114Considering first those embodiments in which Game Logic <b>210</b> is installed on Video Server System <b>120</b>, the installation process can begin with a Select Add-on Step <b>510</b>. In this step the add-on to be installed is selected, typically by a game player using one of Clients <b>110</b>. The selection can be from a list of add-ons presented to the game player using Add-on Interface <b>320</b>, or may include the game player selecting an add-on from a storage location or a third party source.

p-0115In some embodiments only approved add-ons can be selected. In these embodiments a Confirm Add-on Step <b>515</b> is optionally used to confirm that the selected Add-on <b>340</b> is included in an approved list of add-ons. This check can include looking for a name (or other identifier) of the add-ons in the approved list or comparing a checksum, URL, or other signature of an approved add-on with a corresponding characteristic of the selected add-on.

p-0116In an Establish Environment Step <b>520</b> Add-on Manager <b>160</b> is used to establish an instance of Add-on Environment <b>287</b> for executing the selected add-on. In various embodiments, the Add-on Environment <b>287</b> is established on Video Server System <b>120</b>, Add-on Server <b>165</b>, or one of Clients <b>110</b>. For example, on Video Source <b>130</b> establishing the Add-on Environment <b>287</b> may include creating an appropriate directory structure such that the add-on can be installed in a location expected by the associated Game Logic <b>210</b>. In a more specific example, if Game Logic <b>210</b> is configured to look in a subdirectory called “add-ons,” relative to a directory of Game Logic <b>210</b>, then Establish Environment Step <b>520</b> may include creating this subdirectory in the proper location.

p-0117Establishing Add-on Environment <b>287</b> on Add-on Server <b>165</b> or one of Clients <b>110</b> typically includes placing computing instructions configured to redirect communications between Game Logic <b>210</b> and the add-on, at appropriate locations. For example, a first API Filter <b>350</b> may be installed at an API of Game Logic <b>210</b> and a second API Filter <b>350</b> may be installed at a location at which the add-on will be executed. These API Filters <b>350</b> are configured to redirect messages to each other such that the add-on and the associated Game Logic <b>210</b>, which are configured to operate together on the same computing device, can instead operate by communicating with each other over Network <b>115</b>. Establishing Add-on Environment <b>287</b> optionally includes configuring each of the API Filters <b>350</b> such that can communicate with each other, regardless of where they are installed. For example, each of API Filters <b>350</b> may be configured with an IP address or universal resource locator of the corresponding API Filter <b>350</b>.

p-0118In a Receive Add-on Step <b>525</b> the add-on to be installed is received from Storage <b>155</b>, Add-on Storage <b>335</b>, one of Clients <b>110</b> or a third party location. The add-on is optionally received via Network <b>115</b>. The add-on is received at the location it is to be executed, e.g. the Add-on Environment <b>287</b> established in Establish Environment Step <b>520</b>. Some embodiments of Receive Add-on Step <b>525</b> include communicating the add-on from, Add-on Storage <b>335</b>, Client <b>110</b>B or a third party location to Video Server System <b>120</b>. Some embodiments of Receive Add-on Step <b>525</b> include communicating the add-on from Add-on Storage <b>335</b>, Storage <b>155</b>, or a third party location to Client <b>110</b>B.

p-0119In a Place Step <b>530</b> the received add-on is installed/placed in the Add-on Environment <b>287</b> established in Establish Environment Step <b>520</b>. This installation can occur within a directory structure of Video Source <b>130</b>, or at the locations of API Filters <b>350</b> installed on Add-on Server <b>165</b>, video Source <b>130</b>, and/or Clients <b>110</b>. Installation can include provisioning of computing instructions of the add-on and optionally provisioning of default add-on data. Establish Environment Step <b>520</b> is optionally performed after or at the same time as Place Step <b>530</b>. For example, API Filters <b>350</b> can be installed after the add-on is installed. Place Step <b>520</b> optionally includes provisioning of add-on data along with the add-on. The add-on data may or may not be from the same location as the add-on. For example, the add-on may be received from Add-on Server <b>165</b> while the add-on data is received from Client <b>110</b>B. The received add-on data can be default add-on data or add-on data that has been modified by actions of a game player.

p-0120In an optional Associate Step <b>535</b> the add-on installed in Place Step <b>530</b>, and optionally the provisioned add-on data, are associated with a one of Clients <b>110</b> or a user of one of Clients <b>110</b>, i.e. a game player. This association can involve inclusion of an identifier of the add-on in a list of add-ons assigned to the user or copying of the add-on into a directory assigned to the user. The association with the user can be specific to an account of the user, a user, a particular video game, one or more particular avatars, and/or the like. Associate Step <b>530</b> may be optional when the add-on is installed on one of Clients <b>110</b> or in a location that is already associated with an account, a user, etc. In such cases the association with a location, device, address, etc. may serve in place of or in addition to an association with a particular user.

p-0121Once an add-on is associated with an account, a user, or location, etc. it can be provisioned and/or used every time the user requests execution of Game Logic <b>210</b>. For example, one or more add-ons associated with a user can be provisioned at approximately the same time as Game Logic <b>210</b>A and Video DEP <b>245</b>A. The provisioning process can include variation of Steps <b>510</b> through <b>530</b> as described herein. For example, when an add-on is provisioned after previously being installed Select Add-on Step <b>510</b> can merely include parsing a list of add-ons previously associated with a user or location, etc. Receive Add-on Step <b>525</b> can merely include retrieving the add-on and associated add-on data from one or more storage locations such as Storage <b>155</b>, Add-on Storage <b>335</b>, or Client Side Storage <b>410</b>.

p-0122Once an add-on and any associated add-on data are provisioned, use of the add-on can begin with an Execute Game Step <b>540</b>. In this step Game Logic <b>210</b> is executed, typically in response to a request received from one of Clients <b>110</b>. The execution optionally includes communications between Video Server System <b>120</b> and Game Server <b>125</b> such that a state of a multiplayer video game is maintained at Video Server System <b>120</b>.

p-0123In a Receive Command Step <b>545</b> a game command is received by one of Game Logic <b>210</b> from one of Clients <b>110</b>. This typically occurs via Network <b>115</b>. The game command can be a command configured for control of or use by an add-on. For example, the game command can be configured to cause the add-on to display certain information, to perform certain functions or can include data to be processed by the add-on.

p-0124In a Forward Command Step <b>550</b> the game command received in Receive Command Step <b>550</b> is identified as a command for the add-on and forwarded to the add-on. In embodiments in which the add-on is installed on the same computing device as Game Logic <b>210</b> that received the command, the command can be passed directly from the API of this Game Logic <b>210</b> to an API of the add-on. For example, the game command can be passed by the Game Logic <b>210</b>A API to a memory location of the Add-on API.

p-0125In embodiments in which the add-on is installed on a different computing device than the executed Game Logic <b>210</b>, API Filters <b>350</b> are used to communicate the received command between the Game Logic <b>210</b> and the add-on. For example, the command may be first communicated from Game Logic <b>210</b>A to a first API Filter <b>350</b> on Video Source <b>130</b>. This first API Filter <b>350</b> is configured to appear as the Add-on API from the point of view of the Game Logic <b>210</b>A API. Thus the communication from Game Logic <b>210</b>A to the first API Filter <b>350</b> can be accomplished by providing the command, or a pointer to the command, to a memory location of the first API Filter <b>350</b>. The first API filter <b>350</b> is configured to communicate the command to a second API Filter <b>350</b> at the location of the add-on, e.g., at one of Clients <b>1108</b> or Add-on Server <b>165</b>. This communication can occur over Network <b>115</b> using TCP/IP, UDP/IP, or some other communication standard. Upon receipt of the command, the second API Filter <b>350</b> is configured to forward the command to the add-on. The second API Filter <b>350</b> is configured to appear as the Game Logic API from the point of view of the add-on. Thus, communication from the second API Filter <b>350</b> to the add-on can be accomplished by providing the command, or a pointer to the command, to a memory location of the add-on.

p-0126In a Process Command Step <b>555</b> the command is processed using the add-on. Processing can include the generation of add-on data. For example, the command may include data provided by a user or by Game Logic <b>210</b>A. The command may cause the add-on to record or detect events in the game and generate add-on data based on these events. The command may cause the add-on to provide audio or image data for inclusion in an audio or video stream generated by Video Server System <b>120</b> to one of Clients <b>210</b>. The command may cause an image to be converted between formats. For example, be converted to a pdf file. This audio or image data, or any other data resulting from Process Command Step <b>555</b>, is optionally communicated to Video Server System <b>120</b> or one of Clients <b>110</b>, for example, via Network <b>115</b>. The command may cause any other result known in currently existing add-on technology.

p-0127In an Optional Store Step <b>560</b> add-on data generated in Process Command Step <b>550</b> is stored. This storage may be proximate to the add-on, or alternatively at a location in communication with the add-on via Network <b>115</b>. For example, regardless of where the add-on is located the data may be stored on Video Server System <b>120</b>, Add-on Server <b>165</b>, and/or one of Clients <b>110</b>. Storage may occur in real-time or after a delay. For example, in some embodiments a working copy of the add-on data local to the add-on is used during a specific execution of Game Logic <b>210</b>A, and when this execution is completed the add-on data is stored elsewhere.

p-0128In an optional Generate Step <b>565</b> video data is generated as a result of the add-on processing the game command. For example, the game command may represent a request to have add-on data, locations, maps, text, notices, items, and/or the like displayed to a game player (or a sound presented to the game player). The video data may include two dimensional or three dimensional images. For example, to indicate a location of a resource a two dimensional image/symbol may be read from storage and converted to video data by being applied as an overlay. The use of overlays is discussed further in U.S. patent application Ser. No. 12/826,130 filed Jun. 29, 2010 and entitled “Video Game Overlay.” The disclosure of this patent application is hereby incorporated herein by reference. Alternatively, the video data may be the result of the add-on adding a two or three dimensional object to a game environment which is then rendered to video data using GPU <b>235</b>.

p-0129In an optional Provide Step <b>570</b>, the video data generated in Generate Step <b>560</b> is provided to one or more of Clients <b>110</b> as part of a video stream. Typically, these Clients <b>110</b> include the member of Clients <b>110</b> from which the game command was received in Receive Command Step <b>545</b>, and the video stream is provided via Network <b>115</b>. For example, if the game command is received from Client <b>110</b>B the video may be provided to Client <b>110</b>A and a third destination, as well as Client <b>110</b>B. The third destination can be a storage place, a website, a social networking site, a television, a phone, and/or the like.

p-0130Now considering those embodiments in which Game Logic <b>210</b> is installed on Client <b>110</b>B and at least one add-on is external to Client <b>110</b>B during use. In these embodiments Video Server System <b>120</b> is optional as the game environment can be generated based on a game state, the game environment can be rendered to video, and the video can be displayed, all on Client <b>110</b>B. The game state can be based on state information received from Game Server <b>125</b>, received from another member of Clients <b>110</b>, and/or based solely on information within Client <b>110</b>B. The steps illustrated in <figref idrefs="DRAWINGS">FIG. 5</figref> may be varied as follows:

p-0131In Establish Environment Step <b>520</b> the Game Logic <b>210</b> and the add-on are necessarily on different computing devices. This is because, in the embodiments being discussed, Game Logic <b>210</b> is on Client <b>110</b>B and at least one of Game Logic <b>210</b> and an add-on are on different computing devices separated by Network <b>115</b>. Establish Environment step <b>520</b>, therefore includes provisioning API Filters <b>350</b> at both the location of Game Logic <b>210</b> and the location of the add-on. In some embodiments, one Game Logic <b>210</b> and one add-on are both on Client <b>110</b>B while at least one other add-on is on a different computing device, e.g. Add-on Server <b>165</b> and/or Video Server System <b>120</b>. In these embodiments, Establish Environment step <b>520</b> is directed toward creating an Add-on Environment <b>287</b> for the add-on(s) on the different computing device.

p-0132In Receive Add-on Step <b>525</b> the add-on is received at Add-on Server <b>165</b>, Video Server system <b>120</b>, a different Client <b>110</b>, or a third party location, such as a web server, file server, or the like controlled by a developer of the add-on. In Place Step <b>530</b> the add-on is installed at a location other than the Client <b>110</b> on which the instance of Game Logic <b>210</b> to be augmented by the add-on is installed. For example, if Game Logic <b>210</b> is disposed on Client <b>210</b>B, the add-on can be installed on Client <b>210</b>C, Add-on Sever <b>165</b>, Video Server System <b>120</b>, or a third party location. Place Step <b>530</b> optionally includes communicating add-on data from Client <b>210</b>B, Add-on Server <b>165</b>, or Video Server System <b>120</b> to the location at which the add-on is installed.

p-0133In Associate Add-on Step <b>535</b> the list of add-ons used to save add-on associations is optionally stored on Client <b>210</b>B. Execute Game Step <b>540</b> and Receive Command Step <b>545</b> are optionally confined to Client <b>110</b>B. For example, Receive Command Step <b>545</b> may include communicating a command from User Input <b>440</b> to Processor <b>420</b>, rather than communicating the command via Network <b>115</b>. In Forward Command Step <b>550</b> the command received in Receive Command Step <b>544</b> is forwarded from Client <b>110</b>B to the location of the add-on. Any data resulting from Process Step <b>555</b> is optionally communicated back to Client <b>210</b>B.

p-0134In optional Store Step <b>560</b> data generated in Process Command Step <b>550</b> is stored on Client <b>110</b>B, the location of the add-on, and/or elsewhere. Provide Step <b>570</b> can include merely providing the video data generated in Generate Step <b>560</b> from Processor <b>420</b> to Display <b>430</b>. Although, is some embodiments, this video data can be also provided to other members of Clients <b>110</b> or to third party locations discussed herein.

p-0135Returning now to the more general case in which Game Logic <b>210</b> can be disposed on either Clients <b>110</b> or Video Server System <b>120</b>. <figref idrefs="DRAWINGS">FIG. 6</figref> illustrates methods of automatically updating an add-on, according to various embodiments of the invention. The add-on is typically updated in response to receipt of an updated version of Game Logic <b>210</b>. Any combination of the steps illustrated in <figref idrefs="DRAWINGS">FIG. 6</figref> can be performed automatically. It is possible that a request for an updated add-on is made before a request is received to execute the updated version of Game Logic <b>210</b>. The methods illustrated in <figref idrefs="DRAWINGS">FIG. 6</figref> are optionally used to update more than one add-on in response to receiving one updated Game Logic <b>210</b>.

p-0136In a Receive Game Logic Update Step <b>610</b> a new version of Game Logic <b>210</b> is received at Video Source <b>130</b> or Client <b>210</b>B. This new version is typically received from a publisher of the associated video game or an intermediary thereof. The new version is optionally received via Network <b>115</b>. The new version of Game Logic <b>210</b> is configured to replace a previous version. Receive Game Logic Update Step <b>610</b> optionally includes disabling any out of date add-ons.

p-0137In a Determine Add-on Usage Step <b>615</b> it is determined whether an add-on has been used to augment the functionality of the previous version of Game Logic <b>210</b>. Determine Add-on Usage Step <b>615</b> can include, for example, examining a list of add-ons associated with specific Game Logic <b>210</b>, video game, users, user accounts, Clients <b>110</b>, etc. This list may be maintained by Account Manager <b>325</b>. Alternatively, Determine Add-on Usage Step <b>615</b> can include examining storage locations configured to store add-ons and looking for add-ons configured to augment Game Logic <b>210</b>.

p-0138In a Send Request Step <b>620</b> a request for an updated add-on made. This request is typically made to a third party, such as a publisher of the add-on or intermediary thereof. The request includes an address to which the update should be provided, and optionally an identity of the add-on and new version of Game Logic <b>210</b> with which the add-on should be compatible. For example, the request may indicate that a new version of “Deadly Boss Mods” for the “World of Warcraft version 6.542” Game Logic <b>210</b> be sent to an address of Video Server <b>120</b>, Add-on Server <b>165</b> or Client <b>110</b>B. The request is optionally sent via Network <b>115</b>. Send Request Step <b>620</b> can be performed between a time the updated version of Game Logic <b>210</b> is received in Receive Game Logic Update Step <b>610</b> and a first time a request to execute the updated version of Game Logic <b>210</b> is received. Send Request Step <b>620</b> can be performed automatically in response to receiving the new version of Game Logic <b>210</b> in Receive Game Logic Update Step <b>610</b>. Send Request Step <b>620</b> is optionally repeated until an updated add-on is received. For example, the request can initially be sent every 10 minutes and later sent every hour.

p-0139In a Receive Add-on Update Step <b>625</b> the updated add-on is received at the address provided in Send Request Step <b>620</b>. The updated add-on is configured to augment the new version of Game Logic <b>210</b> received in Receive Game Logic Update Step <b>610</b> and can be received via Network <b>115</b>. Receive Add-on Update Step <b>625</b> is optionally performed between the time the updated version of Game Logic <b>210</b> is received in Receive Game Logic Update Step <b>610</b> and a time a first request to execute the updated version of Game Logic <b>210</b> is received.

p-0140In an Install Step <b>630</b> the updated add-on received in Receive Add-on Update Step <b>625</b> is installed. This installation may be in an add-on on storage such as Add-on Storage <b>335</b> or Client-side Storage <b>410</b>. Alternatively, the installation may be in an Add-on Environment <b>287</b> where it can be used without further provisioning. The installation process is optionally similar to Place Step <b>530</b> as discussed with respect to <figref idrefs="DRAWINGS">FIG. 5</figref>. Install Step <b>630</b> is optionally performed automatically, e.g., without need for a game player to order the installation.

p-0141In a Receive Execution Request Step <b>635</b> a request to execute the new version of Game Logic <b>210</b>, received in Receive Game Logic Updates Step <b>610</b>, is received. This request is received from a game player via one of Clients <b>110</b>, and is optionally received at Video Source <b>130</b> via Network <b>115</b>. Receive Execution Request Step <b>635</b> may occur before or after any of Send Request Step <b>620</b>, Receive Add-on Update Step <b>625</b> and Install Step <b>630</b>.

p-0142In an optional Request Donation Step <b>640</b> a request to donate to a publisher of the add-on is presented to a game player. Typically this is the game player that made the request for execution in Receive Execution Request Step <b>635</b>. The request to donate is optionally presented as an overlay on the video stream generated using Video Server System <b>120</b>.

p-0143In an optional Notify User Step <b>645</b> the game player that made the request for execution in Receive Execution Request Step <b>635</b> is notified that an new version of the add-on has been received. The notice is optionally presented as an overlay on the video stream generated using Video Server System <b>120</b>.

p-0144Several embodiments are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations are covered by the above teachings and within the scope of the appended claims without departing from the spirit and intended scope thereof. For example, while the example presented herein are directed at computer games, in other embodiments the systems and methods taught are applied to other types of computer programs. These other types of computer programs can include graphics programs, spreadsheets, word processors, database programs, accounting programs, project management programs, video editing programs, image editing programs, website development programs, inventory programs, e-mail programs, file management programs, document management/viewing programs, drawing/computer aided design programs, presentation programs, and/or the like. Further, while many of the examples discussed herein refer to a single add-on. The systems and methods disclosed can be applied to groups of add-ons. These groups can be handled as an add-on package.

p-0145The embodiments discussed herein are illustrative of the present invention. As these embodiments of the present invention are described with reference to illustrations, various modifications or adaptations of the methods and or specific structures described may become apparent to those skilled in the art. All such modifications, adaptations, or variations that rely upon the teachings of the present invention, and through which these teachings have advanced the art, are considered to be within the spirit and scope of the present invention. Hence, these descriptions and drawings should not be considered in a limiting sense, as it is understood that the present invention is in no way limited to only the embodiments illustrated.

p-0146Computing systems referred to herein can comprise an integrated circuit, a microprocessor, a personal computer, a server, a distributed computing system, a communication device, a network device, or the like, and various combinations of the same. A computing system may also comprise volatile and/or non-volatile memory such as random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), magnetic media, optical media, nano-media, a hard drive, a compact disk, a digital versatile disc (DVD), and/or other devices configured for storing analog or digital information, such as in a database. The various examples of logic noted above can comprise hardware, firmware, or software stored on a computer-readable medium, or combinations thereof. A computer-readable medium, as used herein, expressly excludes paper. Computer-implemented steps of the methods noted herein can comprise a set of instructions stored on a computer-readable medium that when executed cause the computing system to perform the steps. A computing system programmed to perform particular functions pursuant to instructions from program software is a special purpose computing system for performing those particular functions. Data that is manipulated by a special purpose computing system while performing those particular functions is at least electronically saved in buffers of the computing system, physically changing the special purpose computing system from one state to the next with each change to the stored data.