3D object camera customization system

3D Object Camera Customization System

[0001] This application is a divisional application of the patent application with application number 202080047024.6 filed on June 19, 2020 and invention name "3D object camera customization system".

PRIORITY CLAIM

[0003] This application claims priority to U.S. patent application serial number 16/457,461 filed on June 28, 2019, which is incorporated herein by reference in its entirety.

Technical Field

[0004] The present disclosure generally relates to visual presentation of virtual content, and more particularly, to rendering three-dimensional objects within a real-world environment captured in a camera feed of a computing device.

Background technique

[0005] Virtual rendering systems can be used to create an engaging and entertaining augmented reality experience in which three-dimensional (3D) virtual object graphical content appears to exist in the real world. Such systems allow a user to select from a predefined list of 3D objects and display the selected 3D object in a view fed by a camera.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the accompanying drawings that are not necessarily drawn to scale, similar reference numerals may describe similar components in different views. In order to easily identify the discussion of any particular element or behavior, one or more of the most significant digits in the figure number refers to the figure number in which the element is first introduced. In the figures of the accompanying drawings, some embodiments are shown by way of example and not limitation, wherein:

[0007] FIG. 1 is a block diagram illustrating a messaging system for exchanging data (e.g., messages and associated content) over a network according to an example embodiment;

[0008] FIG. 2 is a block diagram illustrating further details regarding a messaging system according to an example embodiment;

[0009] FIG. 3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system according to an example embodiment;

[0010] FIG. 4 is a schematic diagram illustrating the structure of a message generated by a messaging client application for communication according to an example embodiment;

[0011] FIG. 5 is a block diagram illustrating various components of a surface environment representation object generation system that may be provided as part of a message passing system according to an example embodiment;

[0012] FIGS. 6 and 7 are flowcharts illustrating example operations of a surface environment representation object generation system in performing a method for generating a message including a surface environment representation object according to an example embodiment;

[0013] FIGS. 8-12A and 12B are interface diagrams illustrating various interfaces provided by a messaging system according to some example embodiments;

[0014] FIG. 13 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein according to an example embodiment; and

[0015] FIG. 14 is a diagram illustrating a method capable of being read from a machine-readable medium (e.g., a machine-readable storage medium) according to an example embodiment.

A block diagram of components of a machine that reads instructions and performs any one or more of the methodologies discussed herein.

Detailed ways

The following description includes systems, methods, techniques, instruction sequences, and computer program products that implement the illustrative embodiments of the present disclosure. In the following description, for the purpose of explanation, many specific details are set forth to provide an understanding of the various embodiments of the subject matter of the present invention. However, it will be apparent to those skilled in the art that embodiments of the subject matter of the present invention may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0017] Users are always looking for new ways to interact with content as if it existed in the real world. Typical augmented reality systems enable users to insert virtual objects into video frames and control their position in the real-world environment depicted in the video frames. However, simply inserting virtual objects into video frames without considering other objects (e.g., surfaces) in the video frames can make the resulting video frames with virtual objects look unreal, especially because the depth of the objects cannot be utilized. Some systems allow users to select from a predefined list of virtual objects to insert into the real environment depicted in the camera feed (e.g., real-time video received from a camera). Although such systems generally work well when presenting such objects in a real-world environment, the lack of the ability to customize and manipulate objects reduces the system's appeal and interest to users.

[0018] Among other things, embodiments of the present disclosure improve the functionality of electronic messaging and imaging software and systems by providing functionality that allows users to modify the surface texture of virtual objects, such as three-dimensional (3D) subtitles or two-dimensional (2D) subtitles, user-generated content, pre-generated content, curated content, geographic filters, pictures, lenses, graphic objects, stickers, emoticons, and GIFs, to represent portions of an image depicting a first real-world environment. Specifically, portions of the virtual objects inserted by the user can be transparent or partially transparent to reveal and represent the underlying image of the real-world environment, and/or modified to include other images of the real-world environment, such as using additional camera feeds. Virtual objects with modified surface textures representing portions of the image depicting the first real-world environment are then rendered as if they existed in other real-world environments and/or other viewpoints of the real-world environment. In this way, the disclosed embodiments create a new and more attractive way for users to communicate with each other. That is, instead of exchanging messages using predetermined objects that cannot be manipulated or personalized by users, the disclosed embodiments allow users to create virtual objects with surface environment representations with personalized content (e.g., messages).

[0019] The disclosed embodiments allow users to modify virtual objects so that the graphical characteristics (e.g., visual attributes, such as color and texture) of the created virtual objects represent the real-world environment of the user who is creating the message. This enables users to convey information to other users using messaging with rich content. According to the disclosed embodiments, the graphical user interface functionality enables easy generation of environment-based virtual objects from within the messaging application and does not require the user's complex photo editing skills.

[0020] In some embodiments, an image and/or video of a first real-world environment content item, such as a beach, ocean, and sky, generated by an image capture device (e.g., a digital camera) is displayed, and 2D/3D text may be displayed over a portion of the first real-world environment content, such as displayed on a beach. The surface of the text (e.g., the color of the text characters) is modified using the color and style of the portion of the first real-world environment in which the text is displayed, so that it appears as if the text is at least partially transparent, thereby revealing the underlying real-world environment. Specifically, the surface texture of one character of text displayed on the beach is modified to be at least partially transparent in a manner that represents the color and style of sand (e.g., sand white), and the surface texture of another character of text displayed on the ocean is similarly modified to represent the color of the ocean (e.g., wave white).

The text with the modified surface texture is stored and sent to another user to be presented on top of a second real-world environment depicted in the image (e.g., an environment depicting an office scene). In this way, text with the sandy white beach and ocean color characters mixed with the first real-world environment is presented and can be moved around in an image depicting an office scene in a second real-world environment. According to the disclosed embodiments, a first user located at a first real-world environment (e.g., a beach) can convey a text message to a second user, who may be in an office (another real-world environment) that is enriched with the visual attributes of the first user's surroundings (e.g., a beach). This significantly enhances the meaning of the message that the first user is trying to convey to the second user.

1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes a plurality of client devices 102, each of which hosts a plurality of applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0022] Thus, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and the messaging server system 108 via the network 106. The data exchanged between the messaging client applications 104 and between the messaging client applications 104 and the messaging server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0023] In some embodiments, the messaging client application 104 presents a graphical user interface to the user for selecting or creating a virtual surface environment representation object, such as a 3D surface environment representation object and/or a 2D surface environment representation object. The user can activate the camera of the messaging client application 104 to view an image of the user's real-world surroundings/environment (e.g., camera feed) in real time. The user can instruct the messaging client application 104 to add a virtual object to the real-world image captured by the camera at a location selected by the user. The location can be selected in 3D and/or 2D space. Specifically, the user can manipulate the virtual object to reposition the virtual object relative to the real-world object.

[0024] In some embodiments, the messaging client application 104 receives user input to modify the added virtual object to a virtual surface environment representation object. That is, after or before the user places the virtual object in the real-world environment depicted in the real-time image, the user can click on the screen or make another suitable gesture to instruct the messaging client application 104 to modify the surface texture of the virtual object using the underlying visual attributes of the real-world environment on which the virtual object is positioned. In one embodiment, the messaging client application 104 changes the transparency of the surface pixels of the virtual object so that the pixels within the boundary of the virtual object represent the underlying portion of the image of the real-world environment on which the virtual object is positioned. The user can then instruct the messaging client application 104 to change the position of the virtual object in the 2D/3D space. As the virtual object with the modified surface texture is repositioned, the surface texture can be continuously updated to continue to represent the underlying portion of the image of the real world on which the object is positioned in real time. In this way, when the user moves the object around, all pixels within the object boundary continue to show the pixels of the real-world environment in which the pixels within the object boundary are located in a partially transparent manner.

[0025] After the object is placed in the 2D/3D space on the portion of the image depicting the real-world environment, the user can instruct the messaging client application 104 to capture and store the surface texture of the virtual object in its current state. That is, the pixels of the surface texture of the virtual object are maintained at the current value, which depicts the portion of the real-world environment on which they are positioned in a partially transparent manner. After the messaging client application 104 captures and stores the current state of the surface texture, the virtual object is locked in a manner that allows the user to continue to manipulate the position in the 2D/3D space within the real-world environment depicted in the camera feed. But now, as the locked virtual object moves around, the surface texture is no longer updated and modified to represent the underlying image of the real-world environment.

[0026] For example, the real world environment depicted in the camera feed includes an ocean, a beach, trees, and a sky. A user can position a 2D/3D virtual object, such as a subtitle with text, in the real world environment. Specifically, a first group of letters of the virtual object can be placed above the ocean, and a second group of letters of the virtual object can be placed above the beach. The messaging client application 104 can be instructed to modify the surface texture of the virtual object to generate a surface environment representation object. In response, the first group of letters can become partially transparent to show the ocean on which they are positioned (e.g., the pixel values on which the first group of letters are positioned are copied to the pixel values of the pixels within the boundaries of the first group of letters). Similarly, the second group of letters can become partially transparent to show the ocean on which they are positioned (e.g., the pixel values on which the second group of letters are positioned are copied to the pixel values of the pixels within the boundaries of the second group of letters). The user can manipulate the position of the subtitles, and the pixel values within the letter boundaries continue to be updated to represent (e.g., show, display, have an appearance based on) the underlying image portion on which they are positioned in a partially transparent manner.

[0027] In some embodiments, the surface texture/appearance of the virtual object can be generated to simulate a reflective (e.g., mirror-like) surface located in the first environment (e.g., rather than being based on partial transparency). For example, the surface texture/appearance of the virtual object can represent content items in the environment that are in front of the virtual object (rather than behind the virtual object) relative to the position of the virtual object in 3D space.

The user can instruct the messaging client application 104 to lock the current state of the virtual object. In response, when the user moves the virtual object around in the real world environment, the surface texture and pixel value are no longer updated. Like this, the first group of letters with the surface texture of the ocean can be moved to be positioned above the beach in the real world environment, but continue to show the surface texture of the ocean. And, the second group of letters with the surface texture of the beach can be moved to be positioned above the sky in the real world environment, but continue to show the surface texture of the beach. Specifically, the first group and the second group of letters are no longer in a partially transparent state, and can be moved around to other parts of the real world environment in the following manner: the pixel value of the letter when the letter was last positioned on a part of the environment when the letter was in the partially transparent state is used to cover the part where the letter is positioned.

[0029] In some embodiments, a user can capture, store and share a virtual surface environment representation object with another user of another messaging client application 104. For example, a second user in a real-world office environment can activate a camera feed. The second user can add the received virtual surface environment representation object to a camera feed that depicts a real-time image of the second user's surroundings (e.g., an office environment). In some cases, when the second user receives the object, the received virtual surface environment representation object is automatically placed in the camera feed. That is, when the second user receives the virtual surface environment representation object from another user, the second user's camera feed is automatically activated and displays the received virtual surface environment representation object. In some cases, the second user receives the virtual surface environment representation object by means of a message (which may or may not include an indication that the message includes the virtual surface environment representation object). In response to the second user selecting the message (e.g., the user opening the message to view its contents), the virtual surface environment representation object is retrieved and automatically displayed in the second user's camera feed. In this way, the second user can view text with the surface texture of the beach environment generated by the first user in the office environment while the first user is in the beach environment. The second user can move the surface environment representation object around within the office environment depicted in the real-time image. This causes the second user to view a character that partially shows an image of the ocean and beach from the first user's real-world environment in the second user's image of the environment.

[0030] In some embodiments, the user may instruct the messaging client application 104 to add a virtual object, such as a hat, to the user's real-world environment depicted in a real-time image captured by an initial camera feed of the user's device. For example, the real-world environment depicted in the image includes a person's head. The user may instruct the messaging client application 104 to add a virtual object, such as a hat, to the user's real-world environment depicted in a real-time image captured by an initial camera feed of the user's device.

A virtual object, such as a hat, is added to a real-world environment, and the user can position the virtual object on top of the head. Specifically, the messaging client application 104 can place a virtual hat on top of a person's head in the real-world environment. The user can activate the surface environment representation lens feature of the messaging client application 104, and can select a portion of the virtual object to be modified. For example, the user can draw a shape to represent the user's real-world environment, and/or the messaging client application 104 can modify a predetermined portion of the virtual object, such as the front of the hat, to represent the real-world environment. In one embodiment, the messaging client application 104 can activate another camera, such as another front camera or a rear camera. The messaging client application 104 can receive a real-time image of the activated camera and insert the received image into the portion of the virtual object selected to be modified to represent the real-world environment.

[0031] After placing the virtual object in the 2D/3D space on the portion of the image depicting the real-world environment, and after the user is satisfied with the real-time image received from the additional camera and presented on the virtual object, the user can instruct the messaging client application 104 to capture and store the surface texture of the virtual object in its current state. That is, the image currently received from the additional camera is stored and maintained at the position on the virtual object. The user continues to manipulate the position in the 2D/3D space in the real-world environment depicted in the initial camera feed. But now, as the locked virtual object moves around, the surface texture is no longer updated and modified to represent the image of the additional camera. For example, the front of the virtual hat that depicts the last image received from the additional camera feed can be moved around to other parts of the real-world environment depicted in the initial camera feed. The user can also send this virtual object to another user for presentation and manipulation in the second user's environment or in a different camera feed.

The messaging server system 108 provides server-side functionality to specific messaging client applications 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client application 104 or by the messaging server system 108, it should be understood that the location of certain functions within the messaging client application 104 or the messaging server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client application 104 if the client device 102 has sufficient processing power.

[0033] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to, receiving data from, and processing data generated by the messaging client applications 104. As examples, the data may include message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 100 is invoked and controlled through functions available via the user interface (UI) of the messaging client applications 104.

[0034] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to an application server 112 and provides a programming interface to the application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0035] Specifically, the API server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the API server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to invoke the functions of the application server 112. The API server 110 exposes various functions supported by the application server 112, including: account registration; login function; message sending from a particular messaging client application 104 to another messaging client application 104 via the application server 112; media files (e.g., images or videos) sent from the messaging client application 104 to the messaging server application 114 and for possible access by another messaging client application 104; collection of media data (e.g., stories)

settings; retrieval of such collections; retrieval of a friend list of a user of client device 102; retrieval of messages and content; addition and removal of social graph friends; positioning of friends within a social graph; and open application events (e.g., associated with messaging client application 104).

[0036] The application server 112 hosts a number of applications and subsystems, including a messaging server application 114, an image processing system 116, and a social networking system 122. The messaging server application 114 implements a number of message processing techniques and functions, particularly with respect to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., called stories or galleries). These collections are then provided by the messaging server application 114 to the messaging client application 104. Given the hardware requirements of other processor- and memory-intensive data processing, such processing can also be performed on the server side by the messaging server application 114.

[0037] The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically on images or videos received within the payload of a message at the messaging server application 114.

[0038] The social networking system 122 supports various social networking functions and services and makes these functions and services available to the messaging server application 114. To do so, the social networking system 122 maintains and accesses an entity graph within the database 120. Examples of functions and services supported by the social networking system 122 include identifying other users of the messaging system 100 who have a relationship with a particular user or who a particular user is "following," and identifying other entities and interests of a particular user.

The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with messages processed by the messaging server application 112 is stored.

2 is a block diagram showing further details about the messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 112, which in turn contain a number of subsystems, namely a short timer system 202, a collection management system 204, and an annotation system 206.

[0041] The short-term timer system 202 is responsible for enforcing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the short-term timer system 202 incorporates multiple timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on duration and display parameters associated with a message or collection of messages (e.g., a story).

[0042] The collection management system 204 is responsible for managing media collections (e.g., collections of text, images, video, and audio data). In some examples, content collections (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be available for a specified period of time, such as the duration of an event related to the content. For example, content related to a concert can be provided as a "story" during the concert. The collection management system 204 can also be responsible for publishing an icon that provides a notification of the existence of a particular collection to the user interface of the messaging client application 104.

[0043] The collection management system 204 further includes a curation interface 208 that allows collection managers to manage and curate specific content collections. For example, the curation interface 208 enables event organizers to curate content collections related to specific events (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, compensation can be paid to users to include user-generated content in a collection. In this case, the curation interface 208 operates to automatically pay these users for the use of their content.

[0044] Annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, annotation system 206 provides a media overlay of a message processed by messaging system 100.

Generate and publish related functions. The annotation system 206 is operable to provide media overlays (e.g., filters or lenses) to the messaging client application 104. In another example, the annotation system 206 is operable to provide media overlays to the messaging client application 104 based on other information, such as social network information of the user of the client device 102. The media overlay can include audio and visual content and visual effects. Examples of audio and video content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays.

[0045] Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay includes text that can be overlaid on top of an image or video generated by the client device 102. In another example, the media overlay includes an identification of a location overlay (e.g., Venice Beach), the name of a live event, or the name of a business overlay (e.g., Beach Cafe).

[0046] The annotation system 206 includes a surface environment representation object generation system 210 that provides functionality to receive a virtual object (e.g., a virtual object with a transparent type of surface environment representation or a virtual object with an area in which an additional camera feed is presented) and generate a virtual surface environment representation object from the received virtual object and display and track the virtual surface environment representation object at a location relative to the client device 102 in a 2D/3D space captured in a camera feed (also referred to as a "camera stream", "video stream" or "video feed" by those of ordinary skill in the art) of the client device 102. The virtual surface environment representation objects generated, displayed and tracked by the surface environment representation object generation system 210 include: a virtual surface environment representation object that is generated and modified to represent a portion of a real-world environment, and the object is positioned on the portion of the real-world environment in an image depicting the real-world environment. The virtual surface environment representation object represents any user-generated content, curated content, image, text, video, subtitle, animation, or other visual item selected by a user or automatically identified, which has a surface texture that depicts a portion of an image captured by one or more cameras of the user device.

[0047] The surface environment representation object generation system 210 provides functionality to enable users to create, edit, and preview virtual surface environment representation objects by providing a virtual object and positioning the object on a desired portion of an image depicting a real-world environment. The real-world environment may be a pre-recorded video, a pre-captured image, a real-time video, or a real-time image. To this end, the surface environment representation object generation system 210 includes an editing interface 212 and a preview interface 214. The editing interface 212 allows users to create and select virtual objects (e.g., the user may select a 2D/3D image, a 2D/3D graphic element or avatar, or a video from a list, or may manually type in a set of characters corresponding to text or subtitles).

[0048] The editing interface 212 enables the user to edit the virtual object using keyboard input and other types of input including touch screen-based gestures. For example, the user can change the scale, color scheme, size or any other visual attribute of the virtual object. After the user is satisfied with the edited virtual object, the user can instruct the surface environment representation object generation system 210 to create a virtual surface environment representation object based on the virtual object input by the user. The preview interface 214 allows the user to preview and view the generated virtual surface environment representation object before generating a message including the virtual surface environment representation object. The preview interface 214 can also enable the user to edit the presentation of the virtual surface environment representation object (for example, by changing the scale, orientation or placement in 2D/3D space of the virtual surface environment representation object on the display screen). The surface environment representation object generation system 210 creates a virtual surface environment representation object from the virtual object selected by the user according to the process described in conjunction with Figures 6 and 7.

[0049] The surface environment representation object generation system 210 can cause a virtual surface environment representation object to be displayed at a location in a 3D space captured within a camera feed (e.g., on a display of the client device 102) based on a reference surface (e.g., the ground) detected in the 3D space. As will be discussed in further detail below, the surface environment representation object generation system 210 includes a redundant tracking system that includes a set of tracking subsystems configured to track the virtual surface environment representation object at a location in 2D/3D space based on a set of tracking markers and conversions between the tracking subsystems. The surface environment representation object generation system 210 can further track the virtual surface environment representation object in six degrees of freedom (6DoF) based on the availability of the tracking markers.

to switch between full-width tracking and three-degree-of-freedom (3DoF) tracking.

3 is a schematic diagram 300 illustrating data that may be stored in a database 120 of a messaging server system 108, according to certain example embodiments. Although the contents of the database 120 are illustrated as including a plurality of tables, it should be understood that the data may be stored in other types of data structures (e.g., as an object-oriented database).

[0051] The database 120 includes message data stored in a message table 314. The entity table 302 stores entity data, including an entity map 304. The entities whose records are maintained in the entity table 302 may include individuals, corporate entities, organizations, 2D and/or 3D objects, 3D object templates, 3D object textures, places, events, virtual objects configured to have at least a predetermined portion in a transparent, partially transparent, or locked state, virtual objects configured with selectable or predetermined portions to represent additional camera feeds, and the like. Regardless of the type, any entity about which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0052] Entity graph 304 further stores information about relationships and associations between entities. By way of example only, such relationships may be social, professional (e.g., working in a common company or organization), interest-based, or activity-based.

[0053] The database 120 also stores annotation data in the form of examples of filters and lenses in the annotation table 312. The filters and lenses whose data are stored in the annotation table 312 are associated with videos (whose data are stored in the video table 310) and/or images (whose data are stored in the image table 308) and are applied to these videos and/or images. Filters are overlays that are displayed as overlays on images or videos during presentation to the receiving user. On the other hand, lenses include real-time special effects and/or sounds that can be added to the images of the camera feed when the user is composing a message. In contrast, filters are applied to images or videos after the images or videos are captured at the client device 102, while lenses are applied to the camera feed of the client device 102, so that when the images or videos are captured at the client device 102, the lenses are applied and the applied lenses are incorporated as part of the generated images or videos. Filters and lenses can be of various types, including user-selected filters and lenses from a filter gallery or a lens gallery presented to the sending user by the messaging client application 104 when the sending user is composing a message. Any features described with respect to filters also apply to lenses, and vice versa.

[0054] As described above, the video table 310 stores video data, which in one embodiment is associated with messages whose records are maintained within the message table 314. Similarly, the image table 308 stores image data associated with messages whose message data is stored in the entity table 302. The entity table 302 may associate various annotations from the annotation table 312 with the various images and videos stored in the image table 308 and the video table 310.

[0055] The story table 306 stores data about a collection of messages and related image, video, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content that the user has created and sent/broadcast. To this end, the UI of the messaging client application 104 can include a user-selectable icon to enable the sending user to add specific content to his or her personal story.

[0056] A collection may also constitute a "live story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curation stream of content submitted by users from different locations and events. Users whose client devices have location services enabled and are at a public location event at a particular time may be presented with an option to contribute content to a particular live story, for example, via a user interface of a messaging client application 104. Based on the user's location, the messaging client application 104 may identify a live story to the user. The end result is a "live story" told from a community perspective.

Another type of content collection is called a "location story" which allows the client device 102 to be located in a specific location.

In some embodiments, contributions to location stories may require a second level of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

4 is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 for communication with another messaging client application 104 or a messaging server application 114 according to some embodiments. The content of a particular message 400 is used to populate a message table 314 accessible to a messaging server application 114 stored in a database 120. Similarly, the content of the message 400 is stored in memory as "in transit" or "in flight" data of a client device 102 or an application server 112. The displayed message 400 includes the following components:

[0059] Message identifier 402: A unique identifier that identifies message 400.

[0060] Message text payload 404: text that will be generated by the user via the user interface of the client device 102 and included in the message 400.

Message Image Payload 406: Captured by the camera component of the client device 102 or from the client device

Image data retrieved from the memory of 102 and included in the message 400 .

[0062] Message video payload 408: Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400.

[0063] Message audio payload 410: audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400.

[0064] Message annotation 412: Annotation data representing an annotation to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400 (e.g., a filter, sticker, or other enhancement).

[0065] Message duration parameter 414: A parameter value indicating, in seconds, the amount of time that the contents of the message (eg, message image payload 406, message video payload 408, message audio payload 410) will be presented or made accessible to the user via the messaging client application 104.

[0066] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values may be included in the payload, with each of these parameter values being associated with a content item included in the content (e.g., a specific image within the message image payload 406, or a specific video within the message video payload 408).

[0067] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "stories") with which a particular content item in the message image payload 406 of the message 400 is associated. For example, multiple images within the message image payload 406 may each be associated with multiple content collections using an identifier value.

[0068] Message tags 420: Each message 400 may be tagged with a plurality of tags, each tag indicating the subject of the content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included within the message tags 420. The tag values may be manually generated based on user input, or may be automatically generated using, for example, an image identifier.

[0069] Message sender identifier 422: indicates an identifier (eg, a messaging system identifier, an email address, or a device identifier) of the user of the client device 102 that generated the message 400 and sent the message 400.

[0070] Message recipient identifier 424: An identifier (e.g., a messaging system identifier, an email address, or a device identifier) that indicates the user of client device 102 to which message 400 is addressed. The content (e.g., value) of each component of message 400 may be a pointer to a location in a table where the content data value is stored. For example, an image value in message image payload 406 may be a pointer to a location (or address) within image table 308. Similarly, a message video value may be a pointer to a location (or address) within image table 308.

The values within payload 408 may point to data stored in video table 310, the values stored in message annotations 412 may point to data stored in annotations table 312, the values stored in message story identifier 418 may point to data stored in story table 306, and the values stored in message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 302.

FIG. 5 is a block diagram illustrating the functional components of the surface environment representation object generation system 210, which configures the surface environment representation object generation system 210 to render the surface environment representation object from the virtual object in the 2D/3D space depicted in the real-time camera feed. The surface environment representation object generation system 210 is shown to include a rendering module 502, a tracking module 504, and an interruption detection module 506. The various components and modules of the surface environment representation object generation system 210 can be configured to communicate with each other (e.g., via a bus, a shared memory, or a switch). Although not illustrated in FIG. 5, in some embodiments, the surface environment representation object generation system 210 may include or communicate with one or more cameras (e.g., one or more front cameras and/or one or more rear cameras), which are configured to generate a camera feed including image data, and the image data includes an image sequence (e.g., a video). In an embodiment, the surface environment representation object generation system 210 can activate multiple cameras on the user device according to the type of virtual object selected by the user. The first camera may be activated to display a live image of the user's real-world environment in full screen, and the second camera may be activated to display a live image of the user's real-world environment on a dedicated or user-selected portion of the virtual object located in the feed from the first camera.

[0072] Any one or more of the components and modules described may be implemented using hardware alone (e.g., one or more processors 1404 of a machine) or a combination of hardware and software. For example, any component and module of the surface environment representation object generation system 210 may physically include an arrangement of one or more of the processors 1404 (e.g., a subset of one or more processors of a machine or a subset thereof), which is configured to perform the operations described herein for the component. As another example, any component and module of the surface environment representation object generation system 210 may include software, hardware, or both, which configures an arrangement of one or more processors 1404 (e.g., among one or more processors of a machine) to perform the operations described herein for the component. Therefore, different components and modules of the surface environment representation object generation system 210 may include and configure different arrangements of such processors 1404 or a single arrangement of such processors 1404 at different points in time.

[0073] In addition, any two or more components and modules of the surface environment representation object generation system 210 may be combined into a single component or module, and the functionality described herein for a single component or module may be subdivided between multiple components and modules. In addition, according to various example embodiments, the components and modules described herein as being implemented in a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0074] The tracking system 504 may include a first tracking subsystem 504A, a second tracking subsystem 504B, and a third tracking subsystem 504C. Each tracking subsystem tracks the position of the surface environment representation object in the 2D/3D space based on a set of tracking markers.

[0075] Tracking systems often experience tracking failures due to environmental conditions, user actions, unexpected visual interruptions between the camera and the tracked object/scene, etc. Traditionally, such tracking failures result in interruptions in the presentation of virtual objects in 3D space. For example, virtual objects may disappear or otherwise behave irregularly, thereby interrupting the illusion that the virtual objects are presented in 3D space. This undermines the perceived quality of the overall 3D experience.

[0076] Traditional tracking systems rely on a single method (natural feature tracking (NFT), simultaneous localization and mapping (SLAM), gyroscopes, etc.), each of which has breaking points in practical use due to inaccurate sensor data, movement, loss or occlusion of visual markers, or disruption of the dynamics of the scene. In addition, each method may have its own limitations in terms of capabilities. For example, a gyroscope tracking system can only track items with 3 apertures. In addition, due to the inherent limitations of each individual system, using a single tracking

The system may provide inaccurate or unstable position estimates. For example, an NFT system may not provide adequate pitch, yaw, or roll estimates due to inaccurate visual tracking alone, while a gyro tracking system provides inaccurate translation (up, down, left, right).

[0077] In order to solve the above-mentioned problems of the conventional tracking system, the surface environment representation object generation system 210 includes a plurality of redundant tracking subsystems 504A-C, which enable seamless switching between the tracking subsystems. The plurality of redundant tracking subsystems 504A-C solve the problems of the conventional tracking system by merging a plurality of tracking methods into a single tracking system 504. The tracking system 504 is capable of combining 6DoF and 3DoF tracking techniques by combining and switching between a plurality of tracking systems based on the availability of tracking markers (e.g., roll, pitch, yaw, natural features, etc.) tracked by the tracking system. Therefore, when the markers tracked by any one tracking system become unavailable, the surface environment representation object generation system 210 seamlessly switches between 6DoF tracking and 3DoF tracking, thereby providing an uninterrupted experience for the user. For example, in the case of a visual tracking system (e.g., NFT, SLAM), the tracking markers that are usually analyzed to determine the orientation can be replaced by gyro tracking markers from a gyro tracking system. Therefore, this will be able to switch between 6DoF tracking and 3DoF tracking based on the availability of the tracking markers.

[0078] In some example embodiments, in order to switch between 6DoF tracking and 3DoF tracking, the surface environment representation object generation system 210 collects and stores tracking markers within a tracking matrix, which includes translation markers (e.g., up, down, left, right) and rotation markers (e.g., pitch, yaw, roll). Thus, the translation markers collected by the NFT system can be extracted from the tracking matrix and used when future translation markers collected by the NFT system become inaccurate or unavailable. At the same time, the gyroscope continues to provide rotation markers. In this way, when the mobile device loses the tracking marker, the tracked object presented in the 3D space will not be suddenly changed at the frame when the tracking marker is lost. Subsequently, when the target tracked object reappears on the screen and obtains a new translation T1, the translation portion of the view matrix will use the new translation T1 and use T1 to work as the translation of the view matrix.

[0079] The rendering module 502 of the surface environment representation object generation system 210 is configured to obtain a virtual object and generate and render a surface environment representation object from the obtained virtual object in a 3D space captured within a real-time camera feed generated by a camera. When rendering the surface environment representation object, the surface environment representation object generation system 210 assigns the surface environment representation object to a position in the 2D/3D space based on a real-world reference surface detected in the 2D/3D space.

[0080] The surface environment representation object generation system 210 can then track the position of the surface environment representation object relative to the user device in 3D space through one or more tracking systems in 6DoF. For example, one or more tracking systems of the surface environment representation object generation system 210 can collect and analyze a set of tracking markers (e.g., roll, pitch, yaw, natural features, etc.) to track the position of the surface environment representation object relative to the user device in 2D/3D space with 6DoF. In such an embodiment, the surface environment representation object generation system 210 can switch between tracking systems based on the availability of tracked markers to maintain consistent tracking in 6DoF.

[0081] The interruption detection module 506 monitors the tracking markers to detect interruptions. When the interruption detection module 506 detects an interruption of one or more markers, making 6DoF tracking unreliable or impossible, the surface environment representation object generation system 210 switches to tracking the surface environment representation object in 2D/3D space with 3DoF to prevent display interruption. For example, the surface environment representation object generation system 210 can switch from a first tracking subsystem 504A (or a first group of tracking subsystems in a tracking subsystem group) that tracks an object with 6DoF to a second tracking subsystem 504B (or a second group of tracking subsystems) in a tracking subsystem group, wherein the second tracking system is capable of tracking the surface environment representation object in 3D space with 3DoF based on the available tracking markers.

[0082] In some example embodiments, a group of tracking systems of the surface environment representation object generation system 210 includes a gyro tracking system, an NFT system, and a SLAM tracking system. Each tracking system in the group of tracking systems can analyze the tracking markers to track the position of the virtual object in the 3D space. For example, in order to track the virtual object with 6DoF, the surface environment representation object generation system

The system 210 may require at least six tracking markers to be available. When tracking markers become obstructed or unavailable for various reasons, the surface environment representation object generation system 210 may switch between available tracking systems in the tracking system group to maintain 6DoF, or switch to 3DoF when necessary.

[0083] It will be readily appreciated that the surface environment representation object generation system 210 provides surface environment representation objects (e.g., 3D subtitles, avatars, or 3D animations, videos, or images) that are consistently rendered in real-world 3D space in a variety of environments and situations. In many applications, it may be desirable to provide a strong consistency in the position of these virtual objects as one or more users, cameras, or other tracking items move around in the environment. This may involve identification and use of specific fixed reference points (e.g., fixed surfaces) in the real-world environment. Failure to use fixed reference points or items may result in floating or other undesirable inconsistencies in the rendering and presentation of virtual objects.

[0084] In order to ensure a strong consistency in the position of the virtual object, annotation data in the example form of a rendering lens can be used, which is specifically used for the surface environment representation object tracking and rendering described herein. In particular, a surface perception lens is a rendering lens that identifies and references the real-world surface (e.g., the ground) used to consistently render and present the surface environment representation object in 3D space. The surface perception lens can be a specific part or submodule within the rendering module 502. This surface perception lens of the rendering module 502 can be configured to identify a reference surface based on visual camera content, and can also use other device inputs (e.g., gyroscopes, accelerometers, compasses) to determine what is the appropriate surface in the 3D space depicted in the live camera feed. Once the reference surface is determined, the virtual surface environment representation object can be positioned relative to the reference surface. In one example, the reference surface in the 3D space is the ground. In this example, the surface environment representation object generation system 210 renders the surface environment representation object at a position in the 3D space so that the surface environment representation object appears to be on or slightly above the 3D space. The surface environment representation object rendered by the surface environment representation object generation system 210 includes a surface texture of the first real-world environment over which the surface environment representation object is positioned in the image. In some embodiments, the surface environment representation object rendered by the surface environment representation object generation system 210 includes a surface texture of another camera feed depicting other portions of the real-world environment in which the surface environment representation object is located in the image. Such an object is stored and used by the surface environment representation object generation system 210 to track and render a surface environment representation object of a first real-world environment in a second real-world environment depicted in another image on the same or a different user device.

[0085] Figures 6 and 7 are flowcharts illustrating example operations of a surface environment representation object generation system in performing methods 600 and 700 for generating a message including a surface environment representation object according to an example embodiment. Methods 600 and 700 may be implemented by temporary or non-temporary computer-readable instructions, which are executed by one or more processors so that the operations of methods 600 and 700 may be performed in part or in whole by functional components of the surface environment representation object generation system 210; therefore, methods 600 and 700 are described below with reference to them by way of example. However, it should be understood that at least some operations of methods 600 and 700 may be deployed on various other hardware configurations, and methods 600 and 700 are not intended to be limited to the surface environment representation object generation system 210. Any of the operations described in conjunction with methods 600 and 700 may be performed in an order different from that shown and described or omitted entirely.

[0086] At operation 602, the annotation system 206 receives a first input to activate a surface representation object lens. The surface representation object lens can be selected from a set of lenses. For example, a list of multiple lenses is presented in a live feed from a camera, and a user can click on a given lens in the list to immediately add the lens to the live feed. As an example, at operation 604, the user can click on the surface representation object lens to view a list of available surface representation object lens types or objects from which the user can select.

At operation 604, the surface environment representation object generation system 210 causes the edited image to be displayed on the client device 102.

Interface 212. The editing interface 212 enables the user to input the selection of the virtual object and the modification or editing of the virtual object, which provides the basis for generating the surface environment representation object from the virtual object. In some embodiments, the virtual object selected by the user is a 2D or 3D object, and the generated surface environment representation object is a 2D/3D object. To this end, the editing interface 212 may include a keyboard or other input mechanism to enable the user to input the selection of 2D/3D content (e.g., one or more text characters, image selections, video selections, animation selections, etc.). The 2D/3D content input by the user is displayed as a 2D/3D overlay on the camera feed generated by the camera of the client device 102. The camera feed presents the first real world environment of the client device 102.

[0088] At operation 606, the surface environment representation object generation system 210 receives input by a user of the client device selecting a first virtual object using the editing interface 212. The input can adjust the position of the 2D/3D object in the first real-world environment so that the 2D/3D object covers different parts of the first real-world environment depicted in the camera feed. For example, the user can select a virtual object including a portion configured to be in a partially transparent state or a locked state. Specifically, the virtual object can be a subtitle area that adds one character at a time as the user types the characters of the subtitle. Each character can include a border, and the pixels within the border are configured to be at least partially transparent to show and illustrate the portion of the real-world environment on which the character is positioned. When additional characters are entered and/or when the subtitles move around in the 2D/3D space in the real-time camera feed, the pixels within the character borders are updated so that the surface texture shows the portion of the real-world environment on which they are positioned in a partially transparent manner.

[0089] The user can input instructions to lock the state of virtual objects so that the current pixel values representing the portion of the real-world environment on which they are positioned in a partially transparent manner are locked. This allows the user to manipulate the position and orientation of the virtual object in 2D/3D space without having to update the pixel values and without having to continue to update the surface texture to represent the underlying portion of the real-world environment in a partially transparent manner. In this way, when the virtual object is positioned on a different portion of the real-world environment depicted in the real-time camera feed, the virtual object can use its surface texture to represent the first portion of the real-world environment on which the object was positioned when the pixels were in the partially transparent state.

[0090] As another example, a user may select a virtual object that includes a portion dedicated to or selectable to represent an image from another real-time camera feed. Specifically, the virtual object may be positioned in a 2D/3D space in a real-time camera feed received from a first camera (e.g., a front camera), and a portion of the surface texture of the virtual object may be modified with content received from a real-time camera feed received from a second camera (e.g., a rear or front camera). The user may dynamically change whether to use a front camera or a rear camera to provide content for the surface texture of the virtual object. The user may input an instruction to lock the state of the virtual object so that the current pixel value represented in the content of the currently received second real-time camera feed is locked. This allows the user to manipulate the position and orientation of the virtual object in 2D/3D space without updating the pixel value or continuing to update the surface texture to represent the content received from the second real-time camera feed. In this way, when the virtual object is positioned on a different portion of the real-world environment depicted in the first real-time camera feed, the virtual object may use its surface texture to represent the first portion of the real-world environment on which the object is positioned when the pixel is updated by the second camera feed.

[0091] At operation 610, the surface environment representation object generation system 212 detects a third input, and in response to detecting the third input, at operation 612 the surface environment representation object generation system 212 causes display of the preview interface 214. For example, the third input may include a motion-based input, such as a change in the orientation of the client device 102. For example, if the user is pointing the camera of the client device 102 in an upward orientation, a two-dimensional representation of the 2D content is presented in the editing interface 212. If the user changes the orientation of the camera to face downward, the surface environment representation object generation system 212 may switch from displaying the editing interface 212 to displaying the preview interface 214. The preview interface 214 includes a presentation of the surface environment representation object. The surface environment

The representation object may be rendered at a location in the 2D/3D space captured in the camera feed, the location being based on a reference surface detected in the 2D/3D space (eg, a ground or floor surface) and based on the input received at operation 606 .

[0092] At operation 614, the messaging system 100 generates a message including one or more images to which the surface environment representation object is applied. When generating the message, the messaging system 100 may record a video of a user-specified or predetermined length (e.g., 3 seconds) including one or more images from a camera feed to which the surface environment representation object is applied. The messaging system 100 may further apply one or more user-specified filters to the recorded one or more images in generating the message. The messaging system 100 may also send the surface environment representation object to a second user so that the second user can place the surface environment representation object in a different real-time camera feed of another environment. In this way, the surface environment representation object can be generated using content items of the first real-world environment and rendered for display in a second real-world environment by another user.

[0093] As shown in method 700 of Figure 7, in some embodiments, method 600 may include operations 702, 704, 706, 708, and 710. Consistent with these embodiments, operations 702, 704, 706, 708, and 710 may be performed as part of operations 606 and/or 610 (e.g., as a subroutine or sub-operation), wherein the surface environment representation object generation system 210 generates a surface environment representation object from an input virtual object and causes display of a preview interface including a rendering of the virtual surface environment representation object within a first or second real-world environment captured within a live or recorded first or second camera feed.

[0094] At operation 702, the rendering component 502 captures a first image of a first environment of the user device through a camera of the user device. For example, the rendering component 502 activates the rear camera of the user device and presents a real-time image depicting the real-world environment in the full screen or a partial screen.

[0095] At operation 704, the rendering component 502 overlays the first virtual object on a portion of the first image depicting the first environment. For example, the rendering component 502 receives a user selection of a virtual object configured for surface environment representation. The virtual object can be a virtual object configured to present a content item, which comes from the first environment on which the first virtual object is positioned in a partially or completely transparent manner. In another embodiment, the virtual object can be a virtual object configured to display an image received from a second camera feed (e.g., a front or rear camera), which captures the content item from the first environment. In this case, the rendering component 502 can identify an area of the virtual object dedicated to displaying the image from the second camera feed. The rendering component 502 can receive user input, which manipulates the position and orientation of the virtual object in the 2D/3D space relative to the real-time image of the camera feed on which the object is positioned.

[0096] At operation 706, the rendering component 502 modifies the surface of the first virtual object using the content captured by the user device. For example, in the case where the selected object is configured to present the content fed from the real-time camera in a transparent or partially transparent manner, the rendering component 502 can copy the pixel values of the real-time image portion on which the virtual object is positioned to the pixel values within the boundary of the virtual object. The boundary can be opaque or completely transparent. The pixel values can be copied pixel by pixel. Specifically, the rendering component 502 can determine the 2D/3D position of the virtual object in the real-time image. The rendering component 502 can identify the first pixel position of the virtual object in the real-time image and retrieve the pixel value of the real-time image at the same first pixel position. The rendering component 502 can copy the pixel value of the real-time image at the first pixel position to the first pixel position of the virtual object. The rendering component 502 can continue to process each pixel of the virtual object in the same manner until all pixel values are updated to the corresponding pixel values of the real-time image on which they are positioned.

[0097] In the case where the virtual object is configured to present content from another camera feed, the rendering component 502 activates the second camera feed and receives and displays the image from the second camera feed on a specified portion of the virtual object. Once the rendering component 502 receives a user request to lock the image displayed in the specified portion, the rendering component 502 locks the object and allows the user to manipulate the size, shape, and position of the object, which now includes the image last viewed from before the lock was requested.

The second camera feed receives the image. Manipulation can be performed while the virtual object is rendered over the live camera feed initially received.

[0098] At operation 708, the rendering component 502 stores a second virtual object including the first virtual object with the modified surface. For example, in response to receiving a user request to lock the virtual object, the rendering component 502 maintains an image of the portion of the real-world image on which the object was positioned just before receiving the request to lock the virtual object, depicted in a transparent manner in the virtual object. As another example, the rendering component 502 locks the object including the last received image from the second camera feed before the lock is requested. The locked virtual object is stored as the second virtual object (e.g., the surface environment representation object).

[0099] At operation 710, the rendering component 502 generates a second virtual object for display on a portion of the second image depicting the second environment. For example, the rendering component 502 allows the user to send the second virtual object to the second user for display in a real-time camera feed of the second user's user device. As another example, the rendering component 502 allows the user to reposition and manipulate the object in the camera feed showing other portions of the first environment.

[0100] Figures 8-12 are interface diagrams illustrating various interfaces provided by a messaging system according to some example embodiments. Figure 8 specifically illustrates an example in which a virtual surface environment representation object is generated within a given environment of a first user device and is moved after being locked to be positioned in a second portion of the given environment of the first user device.

[0101] As shown in Figure 8, the user activates the lens feature of the messaging application, thereby allowing the user to add a virtual object to the real-world environment depicted in the real-time image. For example, the virtual object selected by the user is a 2D subtitle. In response, the rendering component 502 allows the user to enter the characters of the 2D subtitle, and when the characters are entered, the rendering component 502 presents the characters 810 on the real-world image received from the real-time camera feed. The rendering component 502 updates the surface texture of the character so that the part of the character 810 within the subtitle boundary is at least partially or completely transparent to show the underlying real-world image on which the character is positioned. That is, the rendering component 502 copies the pixel value from the position of the real-world image portion on which the character 810 is positioned to the pixel within the character boundary.

As shown in the figure, when character 810 is positioned above the oak color table, the pixel of character 810 in the border is updated to the same pixel value of the oak color table.When moving camera (for example, right translation) to show another part of real world environment 830 (for example, people working at desk), the pixel of character 810 is updated to show the part of people working at desk that these characters are positioned thereon in a transparent manner.Character 820 shows the pixel value of update, thereby shows that character 820 is transparent, at least partly in the border of subtitle, to show bottom real world image part.Rendering component 502 receives user input to lock virtual object and create 3D surface environment representation object, for example, locks the pixel value of subtitle to prevent the pixel in character border from being updated when virtual object moves around or when camera feed changes.Specifically, character 840 shows the pixel value of locking and is positioned in another part of real world environment in 3D space (for example, when camera is translated to the left). For example, when character 840 is positioned in another portion of the real-world environment (e.g., then panned to the left), real-world environment 830 is no longer visible in the other portion of the real-world environment, and character 840 shows pixel values of real-world environment 830 (e.g., a person working at a desk when the camera pans to the right).

[0103] Figure 9 specifically shows an example in which a virtual surface environment representation object is generated in a first environment of a first user device and sent to a second user device to be positioned and displayed in a second environment of the second user device after being locked. As an example, the virtual object selected by the user is a 2D subtitle 920. In response, the rendering component 502 allows the user to enter characters of the 2D subtitle 920, and when the characters are entered, the rendering component 502 will present the characters 920 on top of the real-world image 910 received from the real-time camera feed of the first user device. The rendering component 502 updates the surface texture of the character so that the portion of the character 920 within the subtitle boundary is at least partially or completely transparent to reveal the underlying real-world image 910 on which the character is positioned. That is, the rendering component 502 updates the pixel value from the position of the portion 910 of the real-world image on which the character 920 is positioned.

Copied to pixels within character 920 boundaries.

[0104] The rendering component 502 receives user input to lock the virtual object and create a 2D surface environment representation object, such as locking the pixel values of the 2D subtitle to prevent the pixels within the character 920 boundary from being updated when the virtual object is moved around, sent to the second user, or when the camera feed changes. Specifically, the character 940 shows the locked pixel values and is positioned in the 3D space in the second real-world environment 930 after being sent to the second user. For example, when the character 940 is positioned in the second real-world environment 930 of the second user device, the real-world environment 910 is no longer visible in the second real-world environment 930, and the character 940 shows the pixel values of the real-world environment 910.

[0105] Figure 10 specifically shows an example in which a virtual surface environment represents that an object is generated in a given environment of a first user device and is redirected in a 3D space after being locked in the same given environment of the first user device. As shown in Figure 10, a user activates a lens feature of a messaging application, thereby allowing a user to add a virtual object to the real-world environment depicted in a real-time image. For example, the virtual object selected by the user is a 2D subtitle. In response, a rendering component 502 allows the user to input characters of a 2D subtitle 1020, and when the characters are input, the rendering component 502 presents the characters 1020 on the real-world image 1010 received from the real-time camera feed. The rendering component 502 updates the surface texture of the characters 1020 so that the part of the characters 1020 in the subtitle boundary is at least partially or completely transparent to show the underlying real-world image 1010 on which the characters are positioned. That is, the rendering component 502 copies pixel values from the 2D and 3D positions of the real-world image 1010 parts on which the characters 1020 are positioned to the pixels in the character boundary.

[0106] The rendering component 502 receives user input to lock the virtual object and create a 3D surface environment representation object, such as locking the pixel value of the subtitle to prevent the pixels within the character boundary from being updated when the virtual object is moved around or the camera feed changes. In response, the rendering component 502 creates a 3D surface environment representation object 1030, which maintains the current value of the first surface 1032 (e.g., the front surface) of the 3D surface environment representation object (to represent the underlying real world image portion 1010), and the second surface 1034 (e.g., the surface perpendicular to the first surface in 3D space) is replaced with a single pixel value. This makes the 3D surface environment representation object 1030 appear to come out of the screen and be positioned and oriented in 3D space. The 3D surface environment representation object 1030 can be reoriented in 3D space to appear to lie on the road, as shown in the 3D surface environment representation object 1040 reoriented in the 3D space of the same real world image 1010. The first surface 1032 of the 3D surface environment representation object 1040 has pixel values of the portion of the real-world image 1010 on which the object was positioned before being locked and re-oriented on a different portion of the real-world image 1010. As shown, a portion 1042 of the real-world image 1010 (on which a portion of the virtual object was positioned before being locked) is shown above a portion 1044 of the 3D surface environment representation object 1040. Even though the 3D surface environment indicates that object 1040 is no longer positioned on top of portion 1042 (after being locked and re-oriented).

[0107] FIG. 11 specifically illustrates an example 1100 in which a virtual surface environment representation object is generated within a given environment of a first user device and is redirected in a 3D space within the same given environment of the first user device after being locked. As shown in FIG. 11, a user activates a lens feature of a messaging application, thereby allowing a user to add a virtual object 1120 to a real-world environment depicted in a real-time image 1110. For example, the virtual object selected by the user is a virtual hat 1120. In response, the rendering component 502 allows the user to position the virtual hat in 2D/3D space on the real-world image 1110 (e.g., a person's head) received from a first real-time camera feed. The rendering component 502 identifies a portion 1130 of the virtual object 1120 that is configured to present content from a second real-time camera feed. The rendering component 502 activates a second camera (e.g., selected by the user as a front or rear camera), receives real-time images from the second camera, and presents those images from the second camera on the portion 1130 of the virtual object 1120. In this way, multiple camera feeds are shown in example 1100 such that a virtual object 1120 is positioned over an image from a first camera feed and over an image from a second camera feed.

The image is placed on a portion of the virtual object. In some embodiments, the rendering component 502 allows the user to select an image, video, or animation to be inserted into the portion 1130 instead of activating the second camera feed. In this case, one camera feed is used so that the virtual object 1120 is positioned over the image from the camera feed and the user-selected image, video, or animation is placed on a dedicated or user-selectable portion 1130 of the virtual object 1120.

[0108] As shown in Figure 12A, the rendering component 502 receives user input to lock the virtual object 1120, such as locking the pixel values of the virtual object 1120 to prevent the pixels from being updated by the second camera feed or the selected image, video or animation when the virtual object 1120 is moved around or when the first camera feed changes. In response, the rendering component 502 creates a surface environment representation object 1210 that maintains the current value of a portion 1230 (e.g., the front surface of the virtual hat) when the surface environment representation object 1210 is moved around or when the camera feed changes. Specifically, portion 1230 shows the locked pixel values of the surface environment representation object 1210 and does not change when the surface environment representation object 1210 is positioned in another portion of the real world environment 1220 in 3D space (e.g., when the camera pans to the left).

[0109] In some embodiments, a virtual object 1120 displayed on a first user device is sent to a second user device in a message. FIG. 12B shows a virtual object 1120 created by a first user device and received by a second user device. The second user device can be operated by a second user and can present a real-time image from a camera feed of the second user device. In FIG. 12B, the real-time camera feed presents the face 1284 of the second user. As shown in FIG. 12B, a virtual object 1120 (which can be a virtual hat) can be placed on the top of the second user's head as a virtual object 1282. Any image, video or animation inserted into a portion of the virtual object by the first user device is also presented in a portion 1280 of the virtual object 1282. In this way, the first user can create a virtual object that is presented on the first user device, as shown in FIG. 11. The first user device can present the virtual object as being worn by the first user. The first user can send the created virtual object to the second user. As shown in FIG. 12B, the second user device of the second user can present the received virtual object to the second user and can position the virtual object so that it looks as if the virtual object is also being worn by the second user.

FIG. 13 is a block diagram illustrating an example software architecture 1306, which can be used in conjunction with various hardware architectures described herein. FIG. 13 is a non-limiting example of a software architecture, and it should be understood that many other architectures can be implemented to facilitate the functions described herein. The software architecture 1306 can be executed on hardware such as the machine 1300 of FIG. 13, which includes a processor 1304, a memory 1314, and an input/output (I/O) component 1319, etc. A representative hardware layer 1352 is shown and can represent, for example, the machine 1300 of FIG. 13. The representative hardware layer 1352 includes a processing unit 1354 with associated executable instructions 1304. The executable instructions 1304 represent executable instructions of the software architecture 1306, including implementations of the methods, components, etc. described herein. The hardware layer 1352 also includes a memory and/or storage module memory/storage 1356, which also has executable instructions 1304. The hardware layer 1352 can also include other hardware 1358.

[0111] In the example architecture of Figure 13, software architecture 1306 can be conceptualized as a layer stack, wherein each layer provides specific functionality. For example, software architecture 1306 may include layers such as operating system 1302, library 1320, application 1316, framework/middleware 1318, and presentation layer 1314. In operation, application 1316 and/or other components within the layer may call API call 1308 via the software stack and receive response 1312 in response to API call 1308. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework/middleware 1318, while other operating systems may provide such layers. Other software architectures may include additional or different layers.

[0112] The operating system 1302 can manage hardware resources and provide public services. The operating system 1302 may include, for example, a kernel 1322, a service 1324, and a driver 1326. The kernel 1322 may act as an abstraction layer between the hardware and other software layers. For example, the kernel 1322 may be responsible for memory management, processor management (e.g., scheduling), component management, network, security settings, etc. The service 1324 may provide other public services for other software layers. The driver 1326 is responsible for controlling or interfacing with the underlying hardware.

For example, depending on the hardware configuration, the driver 1326 includes a display driver, a camera driver, a Bluetooth® driver, a flash driver, a serial communication driver (such as a Universal Serial Bus (USB) driver), a Wi-Fi® driver, an audio driver, a power management driver, and the like.

Library 1320 provides the public infrastructure used by application 1316 and/or other components and/or layers.The function that library 1320 provides allows other software components to perform tasks in a more easy way than directly interfacing with underlying operating system 1302 functions (e.g., kernel 1322, service 1324 and/or driver 1326).Library 1320 may include system library 1344 (e.g., C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc.In addition, library 1320 may include API library 1346, such as media library (e.g., supporting the presentation and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics library (e.g., OpenGL framework that may be used to present two-dimensional and 3D in the graphics content on a display), database library (e.g., SQLite that may provide various relational database functions), webpage library (e.g., WebKit that may provide web browsing functions), etc. The library 1320 may also include a variety of other libraries 1348 to provide numerous other APIs to the applications 1316 and other software components/modules.

[0114] Framework/middleware 1318 (sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by applications 1316 and/or other software components/modules. For example, framework/middleware 1318 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework/middleware 1318 can provide a wide range of other APIs that can be used by applications 1316 and/or other software components/modules, some of which can be specific to a particular operating system 1302 or platform.

[0115] Applications 1316 include built-in applications 1338 and/or third-party applications 1340. Examples of representative built-in applications 1338 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and/or game applications. Third-party applications 1340 may include applications developed using ANDROIDTM or IOSTM software development kits (SDKs) by entities other than the supplier of a particular platform, and may be mobile software running on a mobile operating system such as IOSTM, ANDROIDTM, WINDOWS®Phone, or other mobile operating systems. Third-party applications 1340 may call API calls 1308 provided by a mobile operating system (such as operating system 1302) to facilitate the functionality described herein.

[0116] Applications 1316 can create user interfaces to interact with system users using built-in operating system functions (e.g., kernel 1322, services 1324, and/or drivers 1326), libraries 1320, and framework/middleware 1318. Alternatively or additionally, in some systems, interaction with the user can occur through a presentation layer (e.g., presentation layer 1314). In these systems, the application/component "logic" can be separated from the various aspects of the application/component that interact with the user.

[0117] FIG. 14 is a block diagram illustrating the components of a machine 1400 that can read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more methods discussed herein according to some example embodiments. Specifically, FIG. 14 shows a diagrammatic representation of a machine 1400 in the example form of a computer system, in which instructions 1410 (e.g., software, programs, applications, small applications, apps, or other executable codes) for causing the machine 1400 to perform any one or more methods discussed herein can be executed. Therefore, instructions 1410 can be used to implement the modules or components described herein. Instructions 1410 convert a general non-programmed machine 1400 into a specific machine 1400 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 1400 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1400 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer (or distributed) network environment. The machine 1400 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, or a server computer.

A laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1410 in sequence or otherwise, which instructions specify actions to be taken by the machine 1400. In addition, while only a single machine 1400 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute instructions 1410 to perform any one or more of the methodologies discussed herein.

[0118] The machine 1400 may include a processor 1404, memory/storage 1406, and I/O components 1418, which may be configured to communicate with each other, such as via a bus 1402. In an example embodiment, the processor 1404 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1408 and a processor 1412 that may execute instructions 1410. The term "processor" is intended to include a multi-core processor 1404, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although FIG. 14 shows multiple processors, the machine 1400 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0119] The memory/storage device 1406 may include a memory 1414 and a storage unit 1416, such as a main memory or other storage device, both of which are accessible by the processor 1404, such as via the bus 1402. The storage unit 1416 and the memory 1414 store instructions 1410 that implement any one or more of the methods or functions described herein. The instructions 1410 may also reside, in whole or in part, in the memory 1414, in the storage unit 1416, in at least one of the processors 1404 (e.g., in a cache of the processor), or in any suitable combination thereof during execution thereof by the machine 1400. Therefore, the memory 1414, the storage unit 1416, and the memory of the processor 1404 are examples of machine-readable media.

[0120] The I/O components 1418 may include a variety of components to receive input, provide output, generate output, transmit information, exchange information, capture measurements, and the like. The specific I/O components 1418 included in a particular machine 1400 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that the I/O components 1418 may include many other components not shown in Figure 14. The grouping of the I/O components 1418 according to function is merely to simplify the following discussion and the grouping is by no means restrictive. In various example embodiments, the I/O components 1418 may include output components 1426 and input components 1428. The output components 1426 may include visual components, for example, displays such as plasma display panels (PDPs), light emitting diodes (LEDs), and the like. (LED) display, liquid crystal display (LCD), projector or cathode ray tube (CRT), acoustic component (e.g., speaker), tactile component (e.g., vibration motor, resistance mechanism), other signal generator, etc. Input component 1428 may include alphanumeric input component (e.g., keyboard, touch screen configured to receive alphanumeric input, optical keyboard or other alphanumeric input component), point-based input component (e.g., mouse, touch pad, trackball, joystick, motion sensor or other pointing instrument), tactile input component (e.g., physical button, touch screen providing position and/or touch force or touch gesture, or other tactile input component), audio input component (e.g., microphone), etc.

[0121] In further example embodiments, the I/O component 1418 may include a biometric component 1430, a motion component 1434, an environmental component 1436, or a position component 1438, as well as a number of other components. For example, the biometric component 1430 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or EEG-based identification), etc. The motion component 1434 may include an acceleration sensor component

The environment component 1436 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases to ensure safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The location component 1438 may include a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or a barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0122] Communication can be achieved using a variety of technologies. I/O components 1418 may include communication components 1440, which are operable to couple machine 1400 to network 1432 or device 1420 via coupling 1424 and coupling 1422, respectively. For example, communication components 1440 may include a network interface component or other suitable device to interface with network 1432. In further examples, communication components 1440 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., low-power Bluetooth®), Wi-Fi® components, and other communication components to provide communication via other means. Device 1420 may be another machine or any of a variety of peripheral devices (e.g., peripheral devices coupled via USB).

[0123] In addition, the communication component 1440 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 1440 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a quick response (QR) code, an Aztec code, a DataMatrix, a Dataglyph, a MaxiCode, a PDF417, an UltraCode, a UCCRSS-2D barcode and other optical codes), or an acoustic detection component (e.g., a microphone for an audio signal of an identification tag). In addition, various information can be derived via the communication component 1440, such as positioning via Internet Protocol (IP) geolocation, positioning via Wi-Fi® signal triangulation, positioning via detection of an NFC beacon signal that can indicate a specific location, and the like.

Glossary:

[0125] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding or carrying instructions executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be transmitted or received over a network using a transmission medium via a network interface device and using any of a variety of well-known transmission protocols.

[0126] In this context, "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a PDA, a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communication device that a user may use to access a network.

[0127] "Communication network" in this context refers to one or more portions of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or a portion of a network may include a wireless or cellular network,

And the coupling can be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide microwave interoperability access (WiMAX), long term evolution (LTE) standards, other technologies defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0128] "Short message" in this context refers to a message that is accessible within a limited duration. A short message can be text, image, video, etc. The access time of a short message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is short-term.

[0129] "Machine-readable medium" in this context refers to a component, device, or other tangible medium that is capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache, other types of storage (e.g., erasable programmable read-only memory (EEPROM), and/or any suitable combination thereof. The term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) that can store instructions. The term "machine-readable medium" should also be taken to include any medium or combination of multiple media that can store instructions (e.g., code) executed by a machine, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device, as well as a cloud-based storage system or storage network that includes multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0130] The term "component" in this context refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, and/or other techniques for partitioning or modularization that provide specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processes. A component can be a packaged functional hardware unit designed for use with other components, and part of a program that generally performs specific functions of related functions. A component can constitute a software component (e.g., code contained on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that is capable of performing certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware component that runs to perform certain operations as described herein. A hardware component can also be mechanically, Electronically or any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include a programmable logic or circuit that is temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is tailored specifically for performing the configured function and is no longer a general-purpose processor. It should be understood that the decision to implement a hardware component mechanically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate or perform certain operations described herein in a certain manner. Considering an embodiment in which a hardware component is temporarily configured (e.g., programmed), each hardware component does not need to be configured or instantiated at any one time instance. For example, in hardware components including general-purpose processors that are configured by software to become special-purpose processors

In the case of a processor, the general processor can be configured as different special-purpose processors (for example, including different hardware components) at different times. The software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one time instance and to constitute different hardware components at different time instances. Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components existing at the same time, communication can be achieved by signal transmission between two or more hardware components (for example, through appropriate circuits and buses). In an embodiment where multiple hardware components are configured or instantiated at different times, communication between these hardware components can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a memory device that is communicatively coupled to it. Then, another hardware component can access the storage device later to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device, and can operate on a resource (for example, a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (for example, by software) or permanently configured to perform the relevant operations. Whether it is a temporary configuration or a permanent configuration, Such processors can all constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refers to hardware components implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by a processor, where one or more specific processors are examples of hardware. For example, at least some operations of a method can be performed by one or more processors or processor-implemented components. In addition, one or more processors can also operate to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including a processor), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The performance of certain operations may be distributed between processors, not only residing in a single machine, but also deployed on multiple machines. In some example embodiments, a processor or a processor-implemented component may be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, a processor or a processor-implemented component may be distributed in multiple geographic locations.

[0131] In this context, a "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals for operating a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may further be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0132] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when a particular event occurred, such as giving a date and time, sometimes accurate to a fraction of a second.