Device location based on machine learning classifications

DEVICE LOCATION BASED ON MACHINE LEARNING CLASSIFICATIONS

preference

This application claims the benefit of priority to U.S. Patent Application Serial No. 15/692, 990, filed on August 31, 2017, and is a continuation of U.S. Patent Application Serial Number No., filed April 30, 2018 15/967, 201, the benefit of priority of each of these is hereby claimed and incorporated herein by reference in its entirety.

Technical field

Embodiments of the present disclosure relate generally to determining device locations, and more particularly, to, but not limited to, accurate computer device location determinations based on machine learning classifications.

A computer (eg, a smartphone) may approximate its location using geolocation services. For example, a computer may determine its location using global positioning system (GPS) data generated by an onboard GPS sensor. However, often such information cannot pinpoint which venue (eg, restaurant, gas station) the computer is currently located in when multiple venues exist near the computer.

To facilitate the identification of a discussion of any particular element or act, the most significant number or numbers in a reference number refer to the figure (“FIG.”) number in which the element or act first introduced. 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network. 2 is a block diagram illustrating further details regarding a messaging system with an integrated virtual object machine learning system, in accordance with example embodiments. 3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system, in accordance with certain example embodiments. 4 is a schematic diagram illustrating the structure of a message generated by a messaging client application for communication, in accordance with some embodiments; 5 shows that access to content (eg, ephemeral messages, and associated multimedia payloads of data) or content collections (eg, ephemeral message stories) will be time-limited (eg, short-lived);) is a schematic diagram illustrating an example access-restriction process in terms of 6A illustrates example components of a client venue system in accordance with some example embodiments. 6B illustrates example components of a server venue system in accordance with some example embodiments. 7 depicts a flow diagram of a method for implementing improved venue selection using machine learning classifications in accordance with some example embodiments. 8 depicts a flow diagram of a method for generating classifications for images, in accordance with some demonstrative embodiments. 9 depicts a flow diagram of a method for selecting one or more venues from a venue data set, in accordance with some demonstrative embodiments. 10 illustrates a data structure that may be used to filter or otherwise select venues and display content, in accordance with some demonstrative embodiments. 11 shows an example of metadata tags having a portion of a data structure, in accordance with some demonstrative embodiments. 12 shows an example presentation of an image of a wine bar with overlay content, in accordance with some example embodiments. 13 depicts an example presentation of a casino with overlay content, in accordance with some example embodiments. 14 shows an example presentation of a swimming pool area with overlay content, in accordance with some demonstrative embodiments. 15 is a block diagram illustrating a representative software architecture that may be used with the various hardware architectures described herein. 16 is capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, in accordance with some demonstrative embodiments. A block diagram illustrating the components of a machine.

The following description includes systems, methods, techniques, instruction sequences, and computing machine program products that implement exemplary embodiments of the present disclosure. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the subject matter. It will be apparent, however, to one skilled in the art that embodiments of the subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques have not been necessarily shown in detail.

A client device, such as a smartphone, may approximate its location using geolocation services. For example, the smartphone may use the onboard GPS sensor to determine the smartphone latitude and longitude. However, GPS accuracy may be limited by environmental factors such as interfering buildings and other objects interfering with GPS signals. For example, due to physical obstacles (eg, physical walls), the client device's GPS sensor may actually show the device as one restaurant when the client device is located hundreds of feet away from another restaurant. Also, client device GPS accuracy may be limited by the protocol. For example, military-grade GPS can be very accurate, but consumer-grade GPS often has distance resolution limitations so that consumer-grade GPS devices are not used for malicious purposes. For these reasons, geolocation data generated by the client device may not accurately describe in which venue (eg, cafe, restaurant, gas station) the client device is currently located.

To attempt to improve location determinations, the client device may send GPS data to a server for further analysis. However, it may be inaccurate if the client device GPS signal is poor or if there are many venues near the data sent to the server, in which case subsequent predictions generated by the server will also be inaccurate.

To this end, a venue system running on the client device may leverage the visual cues of the environment surrounding the client device to select a given venue from a set of possible venues. Visual cues may be identified using a machine learning scheme, such as one or more neural networks (eg, convolutional neural networks) that have been trained for image processing. Initially, the client device may gather its geolocation data and other information potentially helpful in location identification, eg, a list of Internet Protocol (IP) networks visible to the client device. The client device may send the collected information to a location server for further analysis.

The server receives the request, accesses the venue database, and returns the number of venues closest to the client device (eg, 5 venues). The venue database accessed by the server may include location data (eg, latitude/longitude data, address data, visible IP network data) for venues somewhere in the world. The server may use geolocation data and other data (eg, visible IP networks) to determine which venues in the database are closest to the client device.

According to some demonstrative embodiments, in response to receiving the set of venues from the server, the client device then attempts to filter out the venues from the set of venues using visual cues from images of the surrounding environment. In particular, for example, the venue system implements one or more trained neural networks to analyze image data generated by the client device. For example, the venue system may use trained neural networks to analyze images (eg, images, video, live video feeds) captured by a user using their smartphone. This approach leverages the fact that the server accesses a potentially very large database of venues that are not storable on the client device and further leverages the client device's access to visual cues of the surrounding environment.

In some demonstrative embodiments, the venue system on the client device uses up to three machine learning schemes to determine visual information for venue selection. In these example embodiments, the first learning scheme is configured to determine whether the client device is in the external environment or the internal environment, the second machine learning scheme is configured to directly determine the type of venue, and the third machine The learning scheme is configured to recognize objects (eg, coffee mugs, pizza slices, etc.) in a given venue. Further details of engines are discussed below with reference to FIG. 6A.

According to some demonstrative embodiments, an example can help illustrate how a machine learning scheme and accompanying logic are implemented. Assume that the client device is at a coffee shop, and the user uses a smartphone to image the coffee shop, for example by taking a picture or video feed of the coffee shop. In response to the coffee shop's shooting, the venue system automatically sends a location request to the server and receives four nearby venues as potential locations of the client device. Four nearby venues are assumed to include: an outdoor rooftop restaurant, the headquarters of a social media technology company, an Italian restaurant and a coffee shop.

The Venue system then applies machine learning schemes to the image to classify attributes about the image. In particular, for example, a first machine learning scheme determines that the image is of an internal environment, a second machine learning scheme determines that the image is most likely an Italian restaurant or a coffee shop, and a third machine learning scheme determines that the image is of coffee. Identify the mug. Based on these determinations, the venue system excludes an outdoor restaurant because it is outside and the imaged environment has been determined by the first machine learning scheme to be inside. The second machine learning scheme determined that the image had an equal likelihood of being an Italian restaurant or coffee shop, and that the image was less likely to be an office setting (eg, headquarters of a technology company), so that the headquarters were Filtered or otherwise unselected. Finally, based on the third machine learning system identifying the coffee cup in the image, the venue system selects the coffee shop as the most likely venue for the current location of the client device.

According to some demonstrative embodiments, each venue may be categorized into a data structure including categories and subcategories. Categories contain venue types at a higher level of granularity, and subcategories subcategorize corresponding venue types. For example, the food category may have subcategories including a Mexican restaurant, a fast food restaurant, an Italian restaurant, a coffee shop, and the like. Other categories and subcategories may be similarly organized. Each specific venue can be generalized to a broader level that can be more easily tagged. For example, "Tony's Family Italian Restaurant - World's Best!" is abstracted into an Italian restaurant venue type that can more easily be assigned tags and stored on the limited memory of a client device (eg, a smartphone). can be

Each category and subcategory may have metadata tags that describe the category or subcategory. Metadata tags may include objects associated with a given type of venue, as well as environmental conditions (eg, “outside”, “inside”) (eg, an Italian restaurant subcategory may include “pizza”, “ red and white checkered pattern", and "spaghetti"; while coffee shop subcategories include "coffee mug", "coffee bean", and "Starbucks logo" (Starbucks logo)".

The venue system may select matching venues using classifications generated by machine learning schemes. Referring to the example above, the venue system has filtered out venues with “outdoor” tags because it has been determined that the image is of the internal environment. Further, the second machine learning scheme determined that the image was unlikely to be of an office, and returned the same numerical probabilities for Italian restaurants and coffee shops; Accordingly, the venue system may filter out or otherwise not select the technology company headquarters venue. The third machine learning scheme has successfully identified the coffee mug in the image and the coffee mug is a metadata tag of the coffee shop category, so the coffee shop is selected as the current venue of the client device. Other examples are discussed below with reference to the drawings.

Once the venue is selected, the venue system or other applications on the client device may use the selected venue to create location-based user interface (UI) content. Continuing with this example, when a coffee shop is determined, a cartoon avatar (eg, bitmoji) of a user operating the client device may be overlaid on the image of the coffee shop. An image with overlaid content may be posted to a social media network, for example, as a short-term message, as described in more detail below. Other examples of UI content include banners, captions (eg, captions outside the bounds of an image indicating where the image was taken), or augmented reality content, and the like. UI content can be categorized using the same data structure of categories and subcategories used to structure the venue types. In this way, when a venue of a given category (or subcategory) is selected, UI content for overlay can be easily retrieved.

1 is a block diagram illustrating an example messaging system 100 for exchanging data (eg, messages and associated content) over a network. Messaging system 100 includes a number of client devices 102, each of which hosts a number of applications, including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and the messaging server system 108 via a network 106 (eg, the Internet).

Accordingly, each messaging client application 104 may communicate and exchange data with other messaging client applications 104 and with the messaging server system 108 over the network 106. The data exchanged between the messaging client devices 102 and between the messaging client application 104 and the messaging server system 108 includes functions (eg, commands that invoke functions), as well as a payload. data (eg, text, audio, video or other multimedia data).

Messaging server system 108 provides server-side functionality to specific messaging client applications 104 over network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client application 104 or by the messaging server system 108, within the messaging client application 104 or messaging server system 108. It will be appreciated that the location of the specific functionality of ' is a design choice. For example, it may be technical to initially deploy certain technologies and functionality within the messaging server system 108, and later transfer those technologies and functionality to the messaging client application 104 when the client device 102 has sufficient processing capacity. may be preferable.

Messaging server system 108 supports various services and operations provided to messaging client application 104. Such operations include sending data to, receiving data from, and processing the data generated by the messaging client application 104. This data may include, as examples, message content, client device information, location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. Data exchanges within the messaging system 100 are initiated and controlled through functions available through user interfaces (UIs) of the messaging client application 104.

Referring now specifically to messaging server system 108, an application programming interface (API) server 110 is coupled to application server 112 to provide a programmatic interface. Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120 in which data associated with messages processed by application server 112 is stored.

The API server 110 receives and transmits message data (eg, commands and message payloads) between the client devices 102 and the application server 112. Specifically, the API server 110 is a set of interfaces (eg, routines and protocols) that can be called or queried by the messaging client application 104 to invoke the functionality of the application server 112. provides API server 110 exposes various functions supported by application server 112, which include: account registration; login function; sending messages, via the application server 112, from one messaging client application 104 to another messaging client application 104; sending media files (eg, images or video) from the messaging client application 104 to the messaging server application 114 for possible access by another messaging client application 104; setting up a collection of media data (eg, a story); retrieval of these collections; retrieving a list of friends of the user of the client device 102; retrieval of messages and content; addition and deletion of friends to and from the social graph; the location of friends within the social graph; and opening application events (eg, related to the messaging client application 104).

Application server 112 hosts a number of applications and subsystems, including messaging server application 114, image processing system 116, social networking system 122, and server venue system 150. The messaging server application 114 may process multiple messages, particularly related to the aggregation and other processing of content (eg, text and multimedia content) included in messages received from multiple instances of the messaging client application 104. Implement technologies and functions. As will be described in greater detail, text and media content from multiple sources may be aggregated into collections of content (eg, called stories or galleries). These collections are then made available to the messaging client application 104 by the messaging server application 114. Other processor, and memory-intensive processing of data may also be performed server-side by messaging server application 114, taking into account the hardware requirements for such processing.

The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically with respect to the image or video received within the payload of the message at the messaging server application 114.

The social networking system 122 supports various social networking functions and services and makes these functions and services available to the messaging server application 114. To this end, social-networking system 122 maintains and accesses an entity graph in database 120 (eg, entity graph 304 of FIG. 3). Examples of functions and services supported by social-networking system 122 include identification of other users of messaging system 100 with which a particular user has a relationship or "followed" by a particular user, and identification of other entities and Contains the interests of a specific user.

The server venue system 150 is configured to receive, from the client device 102, a request for venues near the location of the client device 102. Additional details of the server venue system 150 are discussed below with reference to FIG. 6B.

Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120 in which data associated with messages processed by messaging server application 114 is stored.

2 is a block diagram illustrating further details regarding the messaging system 100, in accordance with example embodiments. Specifically, the messaging system 100 is shown including a messaging client application 104 and an application server 112, which in turn include a number of subsystems: a short-lived timer system 202, a collection management system 204., the annotation system 206 and the client venue system 210.

Short-lived timer system 202 is responsible for enforcing temporary access to content granted by messaging client application 104 and messaging server application 114. To this end, the short-lived timer system 202 selectively selects messages and associated content via the messaging client application 104 based on display parameters and duration associated with the message or collection of messages (eg, a story). It contains a number of timers that display and enable access to it. Additional details regarding the operation of the short-term timer system 202 are provided below.

The collection management system 204 is responsible for managing collections of media (eg, collections of text, images, video, and audio data). In some examples, a collection of content (eg, messages comprising images, video, text, and audio) may be organized into an “event gallery” or “event story”. Such a collection may be made available for a specified period of time, such as the duration of an event to which the content relates. For example, content related to a music concert may be made available as a “story” for the duration of the music concert. The collection management system 204 may also be responsible for posting an icon providing notification of the existence of a particular collection in the user interface of the messaging client application 104.

Collection management system 204 further includes a curation interface 208 that allows collection managers to manage and curate specific collections of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (eg, delete inappropriate content or duplicate messages). Additionally, the collection management system 204 automatically curates the content collection using machine vision (or image recognition technology) and content rules. In certain embodiments, a reward may be paid to a user for including user-generated content in a collection. In such cases, the curation interface 208 operates to automatically pay such users for using their content.

Annotation system 206 provides various functions that enable a user to annotate, otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to the creation and publication of media overlays for messages processed by the messaging system 100. The annotation system 206 effectively supplies a media overlay (eg, a geofilter or filter) to the messaging client application 104 based on the geolocation of the client device 102. In another example, the annotation system 206 may provide a media overlay to the messaging client application 104 based on social network information of the user of the client device 102, or other information, such as a venue selected by the client venue system 210. supply effectively. A media overlay may include audio and visual content and visual effects. Examples of audio and visual content include pictures, texts, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to a media content item (eg, a photo) residing on the client device 102. For example, the media overlay includes text that may be overlaid on top of a photo created by the client device 102. In another example, the media overlay includes an identification of a location (eg, Venice Beach), a name of a live event, or a name of a seller (eg, Beach Coffee House). In another example, the annotation system 206 uses the geolocation of the client device 102 to identify a media overlay that includes the name of the merchant at the geolocation of the client device 102. The media overlay may include other indicia associated with the merchant. Media overlays are stored in database 120 and can be accessed through database server 118.

In one exemplary embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geolocation on a map and upload content associated with the selected geolocation. The user can also specify situations in which certain content should be provided to other users. The annotation system 206 creates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select a particular media overlay associated with a geolocation through a bidding process. For example, the annotation system 206 associates the media overlay of the highest-bid seller with a corresponding geolocation for a predefined amount of time.

3 is a schematic diagram illustrating data 300 that may be stored in database 120 of messaging server system 108, in accordance with certain example embodiments. Although the content of database 120 is illustrated as including a number of tables, it will be appreciated that data may be stored in other types of data structures (eg, as an object-oriented database).

Database 120 includes message data stored in message table 314. Entity table 302 stores entity data including entity graph 304. Entities for which records are maintained in entity table 302 may include individuals, corporate entities, organizations, objects, places, events, and the like. Regardless of the type, any entity for which messaging server system 108 stores data may be a recognized entity. Each entity has a unique identifier as well as an entity type identifier (not shown).

Entity graph 304 further stores information regarding relationships and associations between entities. These relationships may be, for example, social, professional (eg, working in a general legal entity or organization), interest-based, or activity-based.

Database 120 also stores annotation data in annotation table 312 in the form of exemplary filters. Filters for which data is stored in annotation table 312 are associated with and applied to videos (where data is stored in video table 310) and/or images (where data is stored in image table 308). do. In one example, the filters are overlays that are displayed while being overlaid on an image or video during presentation to the receiving user. Filters can be of various types, including user-selected filters from a gallery of filters presented to the sending user by the messaging client application 104 when the sending user is composing a message. can Other types of filters include geolocation filters (also known as geofilters) that can be presented to a sending user based on geographic location. For example, geolocation filters specific to a neighborhood or a particular location may be presented in the user interface by the messaging client application 104 based on geolocation information determined by a global positioning system (GPS) unit of the client device 102. have. Another type of filter is a data filter that may be selectively presented to a sending user by the messaging client application 104 based on information or other inputs gathered by the client device 102 during the message generation process. Examples of data filters include the current temperature at a particular location, the current speed at which the transmitting user is moving, the battery life for the client device 102, or the current time.

Another annotation data that may be stored in the image table 308 is so-called “lens” data. "Lens" can be real-time special effects and sounds that can be added to an image or video.

As noted above, video table 310 stores video data associated with messages for which records are maintained in message table 314, in one embodiment. Similarly, image table 308 stores image data associated with messages for which message data is stored in message table 314. Entity table 302 may associate various annotations from annotation table 312 with various images and videos stored in image table 308 and video table 310.

The story table 306 stores data about collections of messages and associated image, video, or audio data, compiled into a collection (eg, a story or gallery). Creation of a particular collection may be initiated by a particular user (eg, each user whose record is maintained in entity table 302). A user may create a “personal story” in the form of a collection of content created and transmitted/broadcast by that user. To this end, the user interface of the messaging client application 104 may include user selectable icons to enable the sending user to add specific content to their personal story.

Collections may also constitute “live stories,” which are collections of content from multiple users created manually, automatically, or using a combination of manual and automated techniques. For example, a “live story” may constitute a curated stream of user-submitted content from various locations and events. Its client devices 102 enable location services and contribute content to a particular live story, eg, via the user interface of the messaging client application 104, to users at a common location or event at a particular time. options may be presented. The live story may be identified to the user by the messaging client application 104 based on its location. The end result is a "live story" told from a community perspective.

An additional type of content collection is a “location story” that enables a user whose client device 102 is located within a particular geographic location (eg, a college or university campus) to contribute to a particular collection. is known as In some embodiments, contributing to a location story may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (eg, is a student on a university campus).

4 is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 for communication to an additional messaging client application 104 or messaging server application 114, in accordance with some embodiments. to be. The content of a particular message 400 is used to populate a message table 314 stored in the database 120, accessible by the messaging server application 114. Similarly, the content of message 400 is stored in memory as “in transit” or “in-flight” data of client device 102 or application server 112. Message 400 is shown to include the following components:

· Message Identifier 402 : A unique identifier that identifies the message 400.

· Message Text Payload 404 : Text generated by a user through the user interface of the client device 102 and included in the message 400.

· Message Image Payload 406 : Image data captured by the camera component of the client device 102 or retrieved from the memory of the client device 102 and included in the message 400.

· Message Video Payload 408 : Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400.

· Message Audio Payload 410 : Audio data captured by the microphone or retrieved from the memory component of the client device 102 and included in the message 400.

· Message Annotations 412 : Annotation data (eg filters; stickers, or other reinforcements).

· Message Duration Parameter 414 : The content of message 400 (eg, message image payload 406, message video payload 408, and message audio payload 410) is determined by messaging client application 104.), a parameter value indicating the amount of time, in seconds, presented to the user or made accessible.

· Message geolocation parameter 416: geolocation data (eg, latitude and longitude coordinates) associated with the content payload of message 400. A plurality of message geolocation parameter 416 values may be included in the payload, each of these parameter values being specific to the content (eg, a specific image in the message image payload 406, or a specific image in the message video payload 408). video) is associated with the respective content items included in it.

· Message Story Identifier 418: Identifies values that identify one or more collections of content (eg, a “story”) with which a particular content item in the message image payload 406 of the message 400 is associated. For example, each of a plurality of images in the message image payload 406 may be associated with a plurality of content collections using identifier values.

· Message tag 420: One or more tags, each of which represents a subject of content included in the message payload. For example, if a particular image included in the message image payload 406 depicts an animal (eg, a lion), a tag value representing the related animal may be included in the message tag 420. The tag values may be generated manually based on user input, or automatically generated using, for example, image recognition.

· Message sender identifier 422: an identifier (eg, a messaging system identifier, email address, or device identifier) representing the user of the client device 102 from which the message 400 was generated and from which the message 400 was sent.

· Message Recipient Identifier 424: An identifier representing the user of the client device 102 to which the message 400 is addressed (eg, a messaging system identifier, email address, or device identifier).

The content (eg, values) of the various components of message 400 may be pointers to locations in tables where content data values are stored. For example, the image value in the message image payload 406 may be a pointer (or its address) to a location in the image table 308. Similarly, values in message video payload 408 may point to data stored in video table 310, values stored in message annotation 412 may point to data stored in annotation table 312, and message story Values stored in identifier 418 may point to data stored in story table 306, and values stored in message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 302. have.

5 illustrates that access to content (eg, short-term message 502, and an associated multimedia payload of data) or content collection (eg, short-term message story 504) will be time-limited (eg, It is a schematic diagram illustrating the access-restriction process 500 in terms of being short term).

The short-term message 502 is shown associated with a message duration parameter 506, the value of which is to be displayed by the messaging client application 104 to the receiving user of the short-term message 502. determine the amount of time In one embodiment, if the messaging client application 104 is an application client, a short-term message (502) is available for viewing.

A message duration parameter 506 and a message recipient identifier 424 are shown as inputs to a message timer 512, which indicates that a short-lived message 502 has received a particular receipt identified by the message recipient identifier 424. Responsible for determining the amount of time shown to the side user. In particular, the short-lived message 502 will only be visible to the relevant recipient user for a period of time determined by the value of the message duration parameter 506. Message timer 512 is shown providing an output to a more generalized short-term timer system 202 that is responsible for the overall timing of display of content (eg, short-term message 502) to a receiving user.

Short-term messages 502 are shown in FIG. 5 as being included within short-term message stories 504 (eg, personal stories or event stories). The short message story 504 has an associated story duration parameter 508, the value of which determines the duration of time during which the short message story 504 is presented and accessible to users of the messaging system 100. For example, the story duration parameter 508 may be the duration of a music concert, where the short message story 504 is a collection of content related to that concert. Alternatively, a user (owning user or curator user) may specify a value for story duration parameter 508 when performing setup and creation of short-term message story 504.

Additionally, each short-lived message 502 in the short-term message story 504 has an associated story engagement parameter 510 whose value is the amount of time the short-term message 502 will be accessible within the context of the short-term message story 504. determine the duration. Thus, a particular short-term message 502 may “expire” and become inaccessible within the context of the short-term message story 504 before that short-term message story 504 itself expires with respect to the story duration parameter 508. have. The story duration parameter 508, the story engagement parameter 510, and the message recipient identifier 424 each provide inputs to the story timer 514, which, in operation, the specific short-term message of the short-term message story 504. Determines whether 502 will be displayed to a particular recipient user and, if so, for how long. Note that the short-lived message story 504 also recognizes the identity of a particular recipient user as a result of the message recipient identifier 424.

Thus, the story timer 514 controls the overall lifetime of the individual short-term messages 502 included in the short-term message stories 504 as well as the associated short-term message stories 504 during operation. In one embodiment, each and every short-term message 502 in the short-term message story 504 remains viewable and accessible for a period of time specified by the story duration parameter 508. In a further embodiment, a particular short-term message 502 may expire within the context of the short-term message story 504, based on the story engagement parameter 510. Note that the message duration parameter 506 can still determine the duration of time for which a particular short-term message 502 is displayed to the receiving user, even within the context of the short-term message story 504. Thus, the message duration parameter 506 determines that a particular short-term message 502 is sent to the receiving user regardless of whether the receiving user is viewing the short-term message 502 within or outside the context of the short-term message story 504. Determines the duration of the displayed time.

Short-lived timer system 202 may further remove certain short-lived messages 502 from short-lived message stories 504 based on a determination that, during operation, it has exceeded an associated story engagement parameter 510. For example, when the sending user sets the 24 hour story engagement parameter 510 from posting, the short-term timer system 202 removes the relevant short-term message 502 from the short-term message story 504 after a specified 24 hours. something to do. The short-lived timer system 202 is also configured when the story engagement parameter 510 for each and every short-term message 502 in the short-term message story 504 has expired, or the short-term message story 504 itself is a story duration parameter. Acts to remove the short-term message story 504 when it has expired with respect to 508.

In certain use cases, the creator of a particular short-lived message story 504 may specify an indefinite story duration parameter 508. In this case, the expiration of the story participation parameter 510 for the last remaining short-term message 502 in the short-term message story 504 will determine when the short-term message story 504 itself expires. In this case, the new short-term message 502 added to the short-term message story 504 uses the new story participation parameter 510 to equalize the lifetime of the short-term message story 504 to the value of the story participation parameter 510. effectively extend the

In response to short-term timer system 202 determining that short-term message story 504 has expired (eg, is no longer accessible), short-term timer system 202 sends messaging system 100 (eg, For example, specifically communicate with the messaging client application 104) such that an indication (eg, an icon) associated with the associated short-lived message story 504 is no longer displayed within the user interface of the messaging client application 104.. Similarly, when the short-term timer system 202 determines that the message duration parameter 506 for a particular short-term message 502 has expired, the short-term timer system 202 causes the messaging client application 104 to 502) no longer display the associated indication (eg, icon or text identification).

6A illustrates example internal functional components of a client venue system 210 in accordance with some demonstrative embodiments. As shown, the client venue system 210 includes a location engine 600, an image engine 605, an environment classification engine 610, a venue classification engine 615, an object classification engine 620, and a selection engine 625. and a display engine 630. The location engine 600 is configured to send a location request to a location server in accordance with some demonstrative embodiments. For example, the request may include latitude and longitude information generated from an onboard GPS sensor of the client device, particularly the client device 102. The location server (see FIG. 6b) may use the latitude and longitude information to determine which venues are closest to the client device and send the closest venues (eg, up to the closest 5 venues) to the client device.. In some demonstrative embodiments, the request further includes IP networks visible to the client device. In these example embodiments, the location server may further identify which venues are closest to the client device using the name or type of visible IP networks.

The image engine 605 manages identifying one or more images using the image sensor of the client device 102. In some demonstrative embodiments, the image engine 605 identifies images that have already been created by the messaging client application 104 (and stored in the image table 308) and the client venue system 210 for further processing. call it to In some demonstrative embodiments, the image engine 605 interfaces with an image sensor of the client device 102 to generate one or more images. Further, in any of the example embodiments, the images identified by the image engine 605 may be part of a live video feed that is being dynamically displayed on the screen of the client device 102 as the video is recorded.

The environment classification engine 610 is configured to receive the images identified by the image engine 605 and classify each image as being an image of an outdoor environment or an image of an indoor environment. The classification generated by the environmental classification engine 610 may be stored as metadata of each classified image (eg, as metadata of an image file, or as separate data referencing the image file). In some demonstrative embodiments, the environment classification engine 610 is implemented as a machine learning scheme, such as a convolutional neural network. The convolutional neural network in environment classification engine 610 may be trained on a training set of images of outdoor and indoor environments. Examples of outdoor environments imaged in the training set include parks, streets, airplane interiors, amphitheatres, and the like. Examples of indoor environments imaged in the training set include bowling alleys, bedrooms, kitchens, classrooms, ballrooms, offices, theaters, and the like. In some demonstrative embodiments, the classifications output by the environment classification engine 610 are output as a numerical likelihood that a given image will be of a particular environment, as will be appreciated by one of ordinary skill in the art (For example, an image of a park may yield the following classification probabilities: outer=0.93, inner=0.20, where decimal numbers correspond to percentages).

The venue classification engine 615 is configured to receive the images identified by the image engine 605 and classify each image as a type of venue. The classification generated by the venue classification engine 615 may be stored as metadata of each classified image file or as separate data referencing the image file. In some demonstrative embodiments, the venue classification engine 615 is implemented as a machine learning scheme, such as a convolutional neural network. The convolutional neural network in the venue classification engine 615 may be trained on a training set of images of different types of venues or places. In contrast to the environment classification engine 610 (which outputs the probabilities of indoor and outdoor environments), the venue classification engine 615 is used for specific places where a given image is known (eg, a bar, Italian restaurant, park, beach).)). The convolutional neural network in the venue classification engine 615 may be trained on a training set of images including images of different places, eg, images of bars, images of beaches, images of casinos, images of mountain ranges.. In some exemplary embodiments, as will be appreciated by one of ordinary skill in the art, the classifications output by the venue classification engine 615 are output as a numerical likelihood that a given image will be of a particular venue. (For example, an image of a beach may yield the following classification probabilities: beach=0.83, yard=0.75, bar=0.34, Italian restaurant=0.30).

The object classification engine 620 is configured to classify the physical objects depicted in the images. The classifications generated by the object classification engine 620 may be stored as metadata of each classified image file or as separate data referencing a corresponding image file. In some demonstrative embodiments, object classification engine 620 is implemented as a machine learning scheme, such as a convolutional neural network. The convolutional neural network in the object classification engine 620 will be trained on a training set of images of different types of physical objects, such as coffee mugs, wine glasses, pizza, Christmas trees, swimming pools, vehicles, animals, etc. can In some demonstrative embodiments, the classifications output by the object classification engine 620 are numerically capable of that a given image feature in the image is of a particular object, as would be appreciated by one of ordinary skill in the art. Output as a road (eg, a wine glass image feature within a bounded region of interest (ROI) within a given image may have the following classifiers: wine glass=0.93, frog=0.05, basketball=0.30, coffee mug=0.60).

The selection engine 625 selects a venue from a venue data set (eg, a venue data set returned from an application server) using classification data generated by other engines, in accordance with some demonstrative embodiments. manage things Also, the selection engine 625 may be further configured to select one or more UI display elements based on the type of the selected venue. For example, if the selection engine 625 determines that the client device 102 is within a bar (eg, a bar subcategory), the selection engine 625 may include optional lens data for overlays on one or more images and Automatic selection of optional cartoon avatars.

The display engine 630 is configured to generate a presentation of one or more images (eg, images, live video feeds) having the selected display elements. Continuing the example above, display engine 630 may display a cartoon avatar of a person with a martini via the bar's live image feed. According to some demonstrative embodiments, a portion of a live image feed (eg, image, video sequence) with overlaid display elements is a short-term message on a social network (eg, via annotation system 206). may be stored or otherwise posted as 502.

6B illustrates example functional components of a server venue system 150 in accordance with some example embodiments. As illustrated, the server venue system 150 includes an interface engine 633 and a venue engine 685. The interface engine 633 is configured to receive requests, such as a venue request, from the client device 102. The venue engine 635 is configured to use the information sent in the venue request to determine venues closest to the client device 102. The venue engine 635 is further configured to send the set of closest venues to the client device 102 in response to the location request. In some demonstrative embodiments, the set of venues returned to the client device 102 is the 5 closest to the client device 102.

Also, in some demonstrative embodiments, one or more of the engines of the client venue system 210 are integrated into the server venue system 150 as fallbacks. For example, instances of environment classification engine 610, venue classification engine 615, object classification engine 620, and selection engine 625 may cause client device 102 to perform classifications and/or selections. If it does not have sufficient computational power (eg, processing speed, memory space), it may be executed from the application server 112. In these embodiments, the client device 102 sends the image data and location data to the server venue system 150 for processing. The server venue system performs classifications (eg, the operations of FIG. 7 and below) and sends the data back to the client device for further operations (eg, display of overlay data based on the identified venue). send.

7 depicts a flow diagram of a method 700 for implementing improved venue selection using machine learning classifications, in accordance with some demonstrative embodiments. At operation 705, location engine 600 sends a communication to a server (eg, application server 112) requesting the set of venues closest to the client device. The request may include GPS data generated by the client device and other information used to identify nearby venues (eg, visible IP networks).

At operation 710, the location engine 600 receives the venue data set from the server. For example, the location engine may receive the five venues closest to the client device 102. In operation 715, the image engine 605 generates one or more images using the client device 102. For example, the image engine 605 generates an image or video (eg, sequentially multiple images) using an image sensor on the client device 102. In some demonstrative embodiments, the client device 102 displays a live video feed on a screen of the client device 102, and one or more images of operation 715 are sampled from the live video stream.

In some demonstrative embodiments, operation 715 occurs prior to operation 705; That is, the image is created before the venue request is sent. For example, the user may create an image, and in response to the image being generated, the client venue system 210 transmits a request of operation 705 so that the client device 102 approximates the image from which it was taken. Decide where you are in time.

In operation 720, the client venue system 210 classifies one or more images based on what the images depict. For example, if the image generated in operation 715 is of a coffee mug in a coffee shop, then in operation 720 the client venue system 210 uses a machine learning scheme to determine whether the image contains a coffee mug. It can be identified as belonging to the environment. In some demonstrative embodiments, the client venue system 210 implements the machine learning scheme as one or more neural networks. Further details of the classification of operations 720 are as discussed in detail below with reference to FIG. 8.

In operation 725, the selection engine 625 selects a venue from the venue data set based on the classification data generated in operation 720. For example, selection engine 625 may determine, based on the classification data (eg, indoor environment, coffee mugs depicted), that the coffee shop is the most likely venue type and thus selects the coffee shop venue type..

At operation 730, the selection engine 625 selects one or more display objects based on the selected venue. The selected content may be pre-associated with venue categories and subcategories in the data structure. At operation 735, the display engine 630 creates a presentation of the image with the selected display objects, according to some demonstrative embodiments.

In some demonstrative embodiments, the selection engine 625 passes information about which display objects were selected to the annotation system 206, which then uses the information to generate a short-lived message 502. can do. At operation 740, the annotation system 206 posts the presentation to the social-networking platform as a short-term message 502, in accordance with some demonstrative embodiments.

8 shows a flow diagram of a method 800 for generating classifications for images, in accordance with some demonstrative embodiments. Operations 805 - 815 of method 800 may be performed as a subroutine of operation 720 in FIG. 7, in accordance with some demonstrative embodiments. In operation 805, the environment classification engine 610 receives an image as input and generates a numerical likelihood that the image is of an external environment and a numerical likelihood that the image is of an internal environment. In operation 810, the venue classification engine 615 receives an image as input and generates a numerical likelihood that the image will be of different possible venues. The numerical likelihood output may depend on how many venues the venue classification engine 615 has been trained. For example, if the convolutional neural network is trained on images of 10 different venues, in operation 810, the venue classification engine 615 generates 10 different numerical likelihoods for the given image.

In operation 815, the object classification engine 620 generates numerical probabilities that the object depicted in the given image is an object of a known type. For example, the object classification engine 620 may first apply an image feature detection scheme (eg, Sobel edge detection, blob detection, etc.) to find different regions of interest (ROIs) within a given image. Each ROI is a polygon surrounding a given image feature. The image data of each ROI may then be input to an object classification engine 620 for classification. The object classification engine 620 may generate, for each ROI, a plurality of likelihoods that the ROI will contain an object of a given type. For example, if object classification engine 620 is trained on images of five different objects, in operation 815, object classification engine 620 calculates five different numerical likelihoods for each ROI. create

9 depicts a flow diagram of a method 900 for selecting one or more venues from a venue data set, in accordance with some demonstrative embodiments. At operation 905, the selection engine 625 identifies a set of potential venues (eg, the venue data set returned from the application server 112). In operation 910, the selection engine 625 filters out venues in the set of potential venues that do not match the environmental classification data generated in operation 805 of FIG. 8. For example, if the environment classification data indicates that the given image is of an external environment, potential venues that are indoor environments are filtered in operation 910. Venues with both outdoor and indoor metadata tags (eg, a Las Vegas casino modeled after ancient Greece having a famous indoor casino area and also a famous outdoor swimming pool area) may not be filtered out in operation 910., the reason is that the user may be in the outer area or inner area of the venue. At operation 910, the selection engine 625 determines whether the filtering processes of operation 910 have narrowed the number of venues sufficiently. For example, if there are 5 venues and 4 are filtered out in operation 910, the remaining venues are output as those selected by the selection engine 625 in operation 910. In some embodiments, if a preset number of venues remain in operation 915, the remaining venues are output in operation 920 as selected ones. The remaining venues (and associated overlay content) may be presented to the user as options, and the user may select the correct venue. In some demonstrative embodiments, when selection engine 625 presents different venue options to users, selection engine 625 records which venue is most frequently selected by users. Then, when the user is presented with multiple venue options, the most frequently user selected venue is placed at the top of the option list.

Assuming there are still venues (eg, more than one venue in the set), the method 900 continues to operation 905. In operation 905, the selection engine 625 determines whether any of the remaining venues with the venue classifications generated in operation 810 exceed a predefined threshold. If one or more of the remaining venue classifications exceed a predetermined threshold, the one or more venues are output as selected ones in operation 930. For example, if the threshold is set to.90 and the Italian restaurant receives a numerical likelihood of.45 and the coffee shop receives a numerical likelihood of.95, then in operation 930 the coffee shop selects do.

Also, in some demonstrative embodiments, objects recognized by other engines may push the numerical likelihood of the venue above a threshold. Each recognized entity may have a set weight or value that affects the numerical likelihood of the corresponding venue. For example, suppose the threshold is set to 0.90, the Italian restaurant receives a numerical likelihood of.70, and the coffee shop receives a numerical likelihood of.80, so both are below the threshold. and is not selected. However, it is later assumed that the coffee cup is detected by the object classification engine 620. Recognition of a coffee cup may add 0.2 to the numerical likelihood of a coffee shop thereby pushing the coffee shop above a threshold and causing the coffee shop to be selected by the selection engine 625.

At operation 925, assuming that none of the venue classifications have strong signals (eg, signals exceeding a threshold), the method 900 continues to operation 935. In operation 935, the selection engine 625 determines whether any of the detected objects identified in operation 815 match tags of the remaining venues. If the objects match the tag, the venues are output as selected in operation 940. For example, if a coffee mug is detected and the coffee mug is a tag for a food category and a coffee shop subcategory, any venues in the set that match the food or coffee shop are selected in operation 940. Meanwhile, when no recognizable objects are detected in the image, or when the detected objects do not match the remaining venues, the remaining venues are output as selected ones in operation 945. For example, if three venues remain, the three venues may be displayed to the user as options and the user may select the correct one.

Also, as indicated by dashed arrows extending from operations 935 - 925, the objects analyzed in operation 935 may trigger a readjustment of the venue numerical likelihood (eg, a recognized object push above the threshold).

In some demonstrative embodiments, operation 925 is skipped. For example, after operation 915, the process continues directly to operation 935 for object-based selection. In such embodiments, the client venue system is filtering out non-matching environments and then using the objects to determine the correct venue.

Also, in some exemplary embodiments, selected venues may be further modified to indicate whether they are of an exterior portion of the venue or of an interior portion of the venue. For example, instead of selecting "casino" as the venue, "outside the casino" is selected. The UI content displayed along with the image of the venue may depend on the venue, and may also determine whether the user is inside or outside the venue.

10 illustrates a data structure 1005 that may be used to filter or otherwise select venues and display content, in accordance with some demonstrative embodiments. In some demonstrative embodiments, data is stored in relational databases, graph databases, and other forms of structured data. An example data structure 1005 includes categories including a food category 1010, an arts and entertainment category 1015, a store category 1020, an outdoor category 1025, a travel category 1030, and an education category 1035. include those Each of the categories may be associated with or otherwise linked to subcategories that further refine the given category. For example, food category 1010 is sub-categorized into subcategories including Mexican restaurant food subcategory 1010A, wine bar subcategory 1010B, and coffee shop subcategory 1010C. Additionally, the arts and entertainment category 1015 is sub-categorized into subcategories including a stadium subcategory 1015A, a theater subcategory 1015B, and a museum subcategory 1015C. In addition, the store category is further subdivided into subcategories including a gas station subcategory 1020A, a mall category 1020B, and a high end category 1020C. Further, the outdoor category 1025 is sub-categorized into subcategories including a beach subcategory 1025A, a park subcategory 1025B, and a lake category 1025C. Further, the travel category 1030 is further subdivided into an airport category 1030A, a train station category 1030B, and a hotel category 1030C. Further, education category 1035 is sub-categorized into subcategories including college subcategory 1035A, community college subcategory 1035B, and high school subcategory 1035C. Although a limited number of categories and subcategories are shown in the example of FIG. 10, by those of ordinary skill in the art, the number of categories and corresponding subcategories is expanded to include additional categories and subcategories as needed. It can be recognized that it can be

Each of the venues returned as part of the venue data set is pre-categorized into at least one of categories or subcategories of the data structure. Each of the categories and subcategories may be associated with metadata tags that may be used to filter or select categories based on machine learning generated classifications. The data structure and tags may be stored in a local memory of a client device, eg, client device 102.

11 shows an example of metadata tags having a portion of a data structure, in accordance with some demonstrative embodiments. In particular, food category 1010 is displayed with its associated subcategories including Mexican restaurant food subcategory 1010A, wine bar subcategory 1010B, and coffee shop subcategory 1010C. Each of the categories and subcategories has metadata tags that describe the corresponding data object. Tags may be used to filter and select venues as discussed in FIG. 9. As illustrated, the food category has both indoor tags and outdoor tags, since food venues can be outdoor, indoor, or both (indoor restaurants with outdoor patios). Likewise, subcategories also have environmental tags indicating whether the venues are indoors, outdoors, or both. Subcategories have additional tags that further describe properties or objects that are likely to be detected (eg, by object classification engine 620); For example, wine bar 1010B has a wine bottle tag and a wine glass tag. When the detected object corresponds to a given tag, a venue may be selected as discussed in FIG. 9.

Additionally, each of the categories and subcategories may have associated or referenced UI content indicating the category or subcategory venue type. For example, as illustrated, wine bar subcategory 1010B is linked to UI content 1100 (eg, avatars, captions) corresponding to the associated category or type of subcategory. Examples of UI content are shown in FIGS. 12-14.

12 shows an example presentation of an image 1200 of a wine bar with overlay content 1210A and 1210B, in accordance with some example embodiments. In the example of FIG. 12, the client venue system 210 indicates that the user (eg, a person user taking a picture of a wine bar with the client device 102) is in an indoor environment and displays the wine glasses 1205 in image 1200. was determined to be additionally detected. The client venue system then selects a wine bar subcategory using the methods 700-900 and data structure 1000 above. In response to the wine bar selection, the associated UI content 1100 is retrieved and overlaid on the image 1200. In particular, as illustrated, the user's female avatar 1210B is overlaid on the wine bar image 1200 along with the caption “Wine Time!” 1210A. In some demonstrative embodiments, the UI content incorporated into the image is three-dimensional (eg, a dancing cartoon hot dog that appears to be dancing on top of a wine bar counter).

13 shows an example presentation of an image 1300 of a casino with overlay content 1305, in accordance with some example embodiments. In the example shown in FIG. 13, the user has captured an image of the casino and the client venue system 210 implements the approaches described above, such that the image is of an indoor environment, and further that image 1300 is of casino chips and/or greens. It was decided to depict a felt table (eg, table 1310). In response to the decisions, the client venue system selects an indoor casino environment and incorporates the overlay content 1305 (a cartoon avatar of the user gambling) into the image 1300.

14 shows an example image 1400 of a swimming pool area with overlay content 1410 and 1415, in accordance with some demonstrative embodiments. In the example of FIG. 14, the client venue system 210 may determine that the user is at the casino but outside the building (according to environmental data). The object classification engine 620 also identified a pool 1405. In response to the determinations, the client venue system 210 selects outdoor content for the casino venue and overlays it on the image 1400. In particular, the caption “Vegas Baby!” 1415 and the user's expectations are displayed on the image 1400, the user avatar 1410.

The presentations displayed in FIGS. 12-14 may then be posted to the social network as a short-term message 502 as described above. In this way, the venue system disclosed herein prevents the client device from quickly and more accurately determining the current venue and posting a venue-customized short-term message while the user is in or near the venue. Implement visual cues, logic, and specific data structures that allow.

15 is a block diagram illustrating an example of a software architecture that may be installed on a machine, in accordance with some demonstrative embodiments. It will be appreciated that FIG. 15 is only a non-limiting example of a software architecture, and that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1502 may run on hardware, such as machine 1600 of FIG. 16, including processors 1610, memory 1630, and I/O components 1650, among others. A representative hardware layer 1504 is illustrated and may represent, for example, the machine 1600 of FIG. 16. Representative hardware layer 1504 includes one or more processing units 1506 having associated executable instructions 1508. Executable instructions 1508 represent executable instructions of software architecture 1502 that includes implementations of the methods, modules, etc. of FIGS. X-X. The hardware layer 1504 also includes memory or storage modules 1510 that also have executable instructions 1508. Hardware layer 1504 may also include other hardware 1512 representing any other hardware of hardware layer 1504, such as other hardware illustrated as part of machine 1500.

In the example architecture of FIG. 15, software architecture 1502 may be conceptualized as a stack of layers, each layer providing a specific functionality. For example, the software architecture 1502 may include layers such as an operating system 1514, libraries 1516, frameworks/middleware 1518, applications 1520, and a presentation layer 1544. have. During operation, applications 1520 and/or other components in the layers invoke API calls 1524 through the software stack and respond to API calls 1524 (messages (1526)), returned values, and the like. The illustrated layers are representative in nature and not all software architectures have all layers. For example, some mobile or special purpose operating systems may not provide a frameworks/middleware layer 1518, while others may provide such a layer. Other software architectures may include additional or different layers.

The operating system 1514 may manage hardware resources and provide common services. Operating system 1514 may include, for example, a kernel 1528, services 1530, and drivers 1532. The kernel 1528 may serve as an abstraction layer between the hardware and other software layers. For example, the kernel 1528 may be responsible for memory management, processor management (eg, scheduling), component management, networking, security settings, and the like. Services 1530 may provide other common services for different software layers. Drivers 1532 may be responsible for controlling or interfacing with underlying hardware. For example, the drivers 1532 may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (eg, Universal Serial Bus (USB) driver), depending on the hardware configuration.), Wi-Fi® drivers, audio drivers, power management drivers, and the like.

Libraries 1516 may provide a common infrastructure that may be used by applications 1520 and/or other components and/or layers. Libraries 1516 are typically more complex than by other software modules directly interfacing with underlying operating system 1514 functionality (eg, kernel 1528, services 1530, or drivers 1532). Provides functionality that allows performing tasks in an easy-to-follow manner. Libraries 1516 may include system libraries 1534 (eg, the C standard library) that may provide functions such as memory allocation functions, string manipulation functions, math functions, and the like. Libraries 1516 also include media libraries (eg, libraries that support the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphic libraries, etc. (e.g., an OpenGL framework that can be used to render 2D and 3D graphical content on a display), database libraries (e.g., SQLite, which can provide various relational database functions), web libraries (e.g. For example, it may include API libraries 1536 such as WebKit) capable of providing a web browsing function. Libraries 1516 may also include various other libraries 1538 that provide many other APIs to applications 1520 and other software components/modules.

Frameworks 1518 (sometimes referred to as middleware) may provide a high-level common infrastructure that may be utilized by application 1520 or other software components/modules. For example, frameworks 1518 may provide various graphic user interface (GUI) functions, high-level resource management, high-level location services, and the like. Frameworks 1518 may provide a broad spectrum of other APIs that may be used by applications 1520 and/or other software components/modules, some of which are specific to a particular operating system or platform. can do.

Applications 1520 include built-in applications 1540 and/or third party applications 1542. Examples of representative built-in applications 1540 may include, but are not limited to, a home application, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, or a gaming application.

Third party applications 1542 may include any of the built-in applications 1540, as well as a broad collection of other applications. In a particular example, an Android application by an entity other than a third-party application 1542 (eg, a vendor of a particular platform)^TMor IOS^TMApplications developed using a software development kit (SDK)) are^TM, Android^TM, Windows® Phone, or mobile software running on a mobile operating system such as other mobile operating systems. In this example, third party applications 1542 may call API calls 1524 provided by a mobile operating system, such as operating system 1514, to facilitate the functionality described herein.

Applications 1520 include built-in operating system functions (eg, kernel 1528, service 1530 or drivers 1532), libraries (eg, system libraries 1534), APIs 1536 and other libraries 1538), or frameworks/middleware 1518 that create user interfaces that interact with users of the system. Alternatively or additionally, in some systems, interactions with the user may occur through a presentation layer, such as presentation layer 1544. In these systems, the application/module “logic” may be separated from the aspects of the application/module interacting with the user.

Some software architectures use virtual machines. In the example of FIG. 15, this is illustrated by virtual machine 1548. The virtual machine creates a software environment in which applications/modules can run as if they were running on a hardware machine (eg, machine 1600 of FIG. 16). Virtual machine 1548 is hosted by a host operating system (eg, operating system 1514), and typically, but not always, interfaces with a host operating system (eg, operating system 1514) as well as interfaces with host operating system (eg, operating system 1514). Rather, it has a virtual machine monitor 1546 that manages the operation of the virtual machine 1548. The software architecture runs within a virtual machine 1548 such as an operating system 1550, libraries 1552, frameworks/middleware 1554, applications 1556, or presentation layer 1558. These software architecture layers executing within virtual machine 1548 may be the same as or different from the corresponding layers previously described.

16 illustrates a machine 1600 in the form of a computer system within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed, in accordance with an exemplary embodiment. A schematic representation is illustrated. Specifically, FIG. 16 illustrates instructions 1616 (eg, software, program, application, applet, It shows a schematic representation of a machine 1600 in an exemplary form of a computer system in which an app, or other executable code, may be executed. For example, instructions 1616 may cause machine 1600 to execute method XYZ of FIG. 16. Additionally or alternatively, instructions 1616 may implement figures X-X, and the like. Instructions 1616 transform a generic, unprogrammed machine 1600 into a specific machine 1600 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, machine 1600 may operate as a standalone device or be coupled (eg, networked) to other machines. In a networked deployment, machine 1600 may operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1600 may include, but is not limited to, a server computer, client computer, personal computer (PC), tablet computer, laptop computer, netbook, set-top box (STB), PDA, entertainment media systems, cellular telephones, smart phones, mobile devices, wearable devices (eg smart watches), smart home devices (eg smart appliances), other smart devices, web appliances, network routers, network switches, It may include a network bridge, or any machine capable of executing instructions 1616 sequentially or otherwise specifying operations to be performed by machine 1600. Also, although only a single machine 1600 is shown, the term “machine” refers to machines 1600 individually or jointly executing instructions 1616 to perform any one or more of the methodologies discussed herein.) should also be taken to include a collection of

Machine 1600 may include a processor 1610, memory 1630, and I/O components 1650 that may be configured to communicate with each other, such as via a bus 1602. In an exemplary embodiment, processors 1610 (eg, central processing unit (CPU), Reduced Instruction Set Computing (RISC) processor, Complex Instruction Set Computing (CISC)) processor, graphics processing unit (GPU), Digital Signal Processor (DSP), ASIC, Radio-Frequency Integrated Circuit (RFIC), other processor, or any suitable combination thereof) may include, for example, a processor 1612 and a processor 1614 that may execute instructions 1616. The term “processor” is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as “cores”) capable of executing instructions concurrently. 16 shows multiple processors 1610, machine 1600 may include a single processor with a single core, a single processor with multiple cores (eg, a multi-core processor), multiple processors with a single core, multiple cores multiple processors, or any combination thereof.

Memory 1630 may include main memory 1632, static memory 1634, and storage unit 1636, both of which are accessible to processors 1610, such as via bus 1602. Main memory 1630, static memory 1634, and storage unit 1636 store instructions 1616 that implement any one or more of the methodologies or functions described herein. Instructions 1616 may also, during their execution by machine 1600, fully or partially, in main memory 1632, in static memory 1634, in storage unit 1636, in one of processors 1610, in at least one (eg, in a cache memory of a processor), or any suitable combination thereof.

I/O components 1650 may include a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, and the like. The specific I/O components 1650 included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may possibly include a touch input device or other such input mechanism, whereas a headless server machine may not include such a touch input device. will be. It will be appreciated that I/O components 1650 may include many other components not shown in FIG. 16. I/O components 1650 are grouped according to function only to simplify the following discussion and the grouping is by no means limited. In various demonstrative embodiments, I/O components 1650 can include output components 1652 and input components 1654. The output components 1652 are visual components (eg, a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a display such as a cathode ray tube (CRT))., auditory components (eg, speakers), haptic components (eg, vibration motors, resistance mechanisms), other signal generators, and the like. Input components 1654 include alphanumeric input components (eg, a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (eg, For example, a mouse, touchpad, trackball, joystick, motion sensor or other pointing device), tactile input components (for example, a physical button, touch screen or touch screen that provides the location and/or force of touches or touch gestures) other tactile input components), audio input components (eg, a microphone), and the like.

In further example embodiments, I/O components 1650 include biometric components 1656, motion components 1658, environment components 1660, among a wide variety of other components.), or position components 1662. For example, biometric components 1656 can represent expressions (eg, hand expressions, facial expressions, vocal expressions, body gestures)., or eye tracking), which measures biosignals (eg blood pressure, heart rate, body temperature, perspiration, or brain waves) that identifies a person (eg, voice identification, retina identification, face identification, fingerprint identification, or electroencephalogram-based identification), and the like. Motion components 1658 may include acceleration sensor components (eg, an accelerometer), gravity sensor components, rotation sensor components (eg, a gyroscope), and the like. Environment components 1660 include, for example, lighting sensor components (eg, photometer), temperature sensor components (eg, one or more thermometers detecting ambient temperature), humidity sensor components, pressure sensor component (eg, barometer), acoustic sensor components (eg, one or more microphones to detect background noise), proximity sensor components (eg, infrared sensors to detect nearby objects), gas sensor (e.g., gas detection sensors that detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or that may provide indications, measurements or signals corresponding to the surrounding physical environment. It may contain other components. Position components 1662 include location sensor components (eg, a GPS receiver component), an altimeter that detects altitude sensor components (eg, barometric pressure from which altitude can be derived). or barometers), orientation sensor components (eg, magnetometers), and the like.

Communication may be implemented using a wide variety of technologies. I/O components 1650 may include communication components 1664 operable to couple machine 1600 to network 1680 or devices 1670 via coupling 1682 and coupling 1672, respectively. can For example, communication components 1664 may include a network interface component or other device suitable for interfacing with network 1680. In further examples, communication components 1664 include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (eg, Bluetooth® ® Low Energy), Wi-Fi® components, and other communication components that provide communication via other modalities. Devices 1670 may be other machines or any of a variety of peripheral devices (eg, peripheral devices coupled via USB).

Moreover, communication components 1664 may detect identifiers or may include components operable to detect identifiers. For example, the communication components 1664 may include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (eg, a one-dimensional barcode such as a Universal Product Code (UPC) barcode). an optical sensor that detects barcodes, Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, multidimensional barcodes such as UCC RSS-2D barcodes, and other optical codes), or acoustic detection component (eg, microphones that identify tagged audio signals). In addition, various information such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that may indicate a particular location, and etc. It may be derived via communication components 1664.

executable instructions and machine-readable medium

Various memories (i.e., memory of 1630, 1632, 1634, and/or processor(s) 1610) and/or storage unit 1636 may implement any one or more of the methodologies or functions described herein. may store one or more sets of instructions and data structures (eg, software) implemented or used by it. These instructions (eg, instructions 1616), when executed by the processor(s) 1610, cause various operations to implement the disclosed embodiments.

As used herein, the terms "machine-storage medium", "device-storage medium", and "computer-storage medium" mean the same thing and may be used interchangeably in this disclosure. These terms refer to single or multiple storage devices and/or media (eg, a centralized or distributed database, and/or associated caches and servers) that store executable instructions and/or data do. Accordingly, the terms should be taken to include, but are not limited to, solid state memories, and optical and magnetic media including memory internal to or external to processors. Specific examples of machine storage media, computer storage media, and/or device storage media include non-volatile memory, which includes, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM).), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as removable disks and internal hard disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine-storage media”, “computer-storage media”, and “device-storage media” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which is covered under the term “signal medium” discussed below.

Transmission medium

In various demonstrative embodiments, one or more portions of network 1680 are ad hoc networks, intranets, extranets, VPNs, LANs, WLANs, WANs, WWANs, MANs, the Internet, portions of the Internet, portions of the PSTN, plain old telephone service) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more of these networks. For example, network 1680 or a portion of network 1680 may include a wireless or cellular network, and combination 1682 may include a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or other types of cellular or wireless coupling. In this example, combining 1682 is 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3G, 4G (fourth generation) wireless networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Third Generation Partnership Project (3GPP), including Long Term Evolution (LTE) standards; Any of various types of data transfer technology may be implemented, such as other standards defined by various standard-setting organizations, other long range protocols, or other data transfer technologies.

Instructions 1616 can be implemented using a transmission medium over a network interface device (eg, a network interface component included in communication components 1664) and through a number of well-known transport protocols (eg, hypertext (HTTP) transfer protocol))). Similarly, instructions 1616 may be transmitted or received using a transmission medium via coupling 1672 (eg, peer-to-peer coupling) to devices 1670. The terms “transmission medium” and “signal medium” mean the same thing and may be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” should be taken to include any intangible medium capable of storing, encoding, or carrying instructions 1616 for execution by machine 1600, and or analog communication signals or other intangible medium that facilitates communication of such software. Accordingly, the terms "transmission medium" and "signal medium" should be taken to include any form of modulated data signal, carrier wave, or the like. The term "modulated data signal" means a signal having one or more of its properties set or changed in such matters as for encoding information in the signal.

Glossary

“CARRIER SIGNAL” in this context refers to any intangible medium capable of storing, encoding, or carrying instructions 1616 for execution by machine 1600, and such instructions 1616) digital or analog communication signals or other intangible media for facilitating the communication of The instructions 1616 may be transmitted or received over the network 1680 using a transmission medium via a network interface device and using any one of a number of well-known transport protocols.

A “CLIENT DEVICE” in this context refers to any machine 1600 that interfaces with the communications network 1680 to obtain resources from one or more server systems or other client devices 102. Client device 102 may be a mobile phone, desktop computer, laptop, PDA, smartphone, tablet, ultrabook, netbook, multi-processor system, microprocessor-based or programmable consumer electronic device system, game console, set-top box, or user can be, but is not limited to, any other communication device that can be used to access network 1680.

In this context, "COMMUNICATIONS NETWORK" means ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), WWAN (wireless WAN), metropolitan area network (MAN), Internet, part of the Internet, part of the Public Switched Telephone Network (PSTN), plain old telephone service (POTS) networks, cellular telephone networks, wireless networks, Wi-Fi® networks, other type of network, or a combination of two or more such networks. For example, a network or portion of network 1680 may include a wireless or cellular network, and the combination may include a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or other type of cellular or It may include wireless coupling. In this example, combining is 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3G, fourth generation (4G) technology) wireless networks, Universal Mobile Telecommunications System (UMTS), High-Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Third Generation Partnership Project (3GPP), including Long-Term Evolution (LTE) standards, various other standards defined by standard-setting organizations; Any of various types of data transfer technology may be implemented, such as other long-range protocols, or other data transfer technologies.

An “EMPHEMERAL MESSAGE” in this context refers to a message 400 accessible for a time-limited duration. The short-term message 502 may be text, an image, a video, or the like. The access time for the short-lived message 502 may be set by the message sender. Alternatively, the access time may be a default setting or a setting specified by the recipient. Regardless of the setting technique, message 400 is temporary.

“MACHINE-READABLE MEDIUM” in this context refers to, but is not limited to, a component, device, or other tangible medium capable of temporarily or permanently storing instructions 1616 and data, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (eg, erasable programmable read-only memory (EPROM)) and/or any suitable combination thereof. The term “machine-readable medium” refers to a single medium or a plurality of media that may store instructions 1616 (eg, a centralized or distributed database, and/or an associated cache). servers and servers). The term “machine-readable medium” also means that the instructions 1616, when executed by one or more processors 1610 of the machine 1600, cause the machine 1600 to use any of the methodologies described herein. It should be taken to include any medium or combination of multiple media that can store instructions 1616 (eg, code) for execution by machine 1600 to cause one or more of them to be performed. Thus, “machine-readable medium” refers to “cloud-based” storage systems or storage networks that include a single storage device or device, as well as multiple storage devices or devices. The term "machine-readable medium" by itself excludes signals.

"COMPONENT" in this context means a device, a physical device, with boundaries defined by function or subroutine calls, branch points, APIs or other technologies that provide for the partitioning or modularization of particular processing or control functions. Refers to an entity, or logic. Components may be combined with other components through their interfaces to perform machine processes. A component may be a packaged functional hardware unit designed for use with a portion of a program that performs a particular function among other components and generally related functions. Components may constitute either software components (eg, code embodied on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing specific operations, and may be configured or arranged in a specific physical manner. In various demonstrative embodiments, one or more computer systems (eg, stand-alone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (eg, processor 1612 or Group 1610) may be organized by software (eg, an application or application portion) as a hardware component that operates to perform specific operations as described herein. A hardware component may also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform particular operations. The hardware component may be a special purpose processor such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform specific operations. For example, a hardware component may include software executed by a general purpose processor or other programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of machine 1600) uniquely tailored to perform the configured functions and are no longer general purpose processors 1610. That the decision to implement a hardware component mechanically, on dedicated and permanently configured circuitry, or on temporarily configured circuitry (eg, configured by software) may be driven by cost and time considerations. you will know well Thus, the phrase “hardware component” (or “hardware-implemented component”) is a tangible entity, ie, physically configured to operate in a particular manner or perform the particular operations described herein. It should be understood to encompass entities that are permanently configured (eg, hardwired) or temporarily configured (eg, programmed).

Considering embodiments in which hardware components are temporarily configured (eg, programmed), each of the hardware components need not be configured or instantiated at any one instance in time. For example, where the hardware components include a general-purpose processor 1612 configured by software to be a special-purpose processor, the general-purpose processor 1612 may be configured at a different time (eg, including different hardware components) at each different special purpose processor. It can be configured as purpose processors. The software configures the particular processor 1612 or processors 1610 accordingly, for example, to configure a particular hardware component in one time instance and a different hardware component in a different time instance.

A hardware component may provide information to and receive information from other hardware components. Accordingly, the described hardware components may be considered as being communicatively coupled. Where multiple hardware components exist concurrently, communication may be achieved via signal transmission between or between (eg, via appropriate circuits and buses) two or more of the hardware components. In embodiments where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be accomplished, for example, through storage and retrieval of information in memory structures to which the multiple hardware components may access. can For example, one hardware component may perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Additional hardware components can then, at a later time, access the memory device to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices, and may process a resource (eg, a collection of information).

The various operations of the example methods described herein may be performed, at least in part, by one or more processors 1610 that are temporarily configured (eg, by software) or permanently configured to perform the related operations. have. Whether temporarily or permanently configured, such processors 1610 may constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented component” refers to a hardware component implemented using one or more processors 1610. Similarly, the methods described herein may be implemented at least in part by a processor, with the particular processor 1612 or processors 1610 being an example of hardware. For example, at least some of the operations of the method may be performed by one or more processors 1610 or components implemented by processors. Moreover, the one or more processors 1610 may also operate to support performance of related operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as an example of a machine 1600 that includes processors 1610), and these operations are performed by a network 1680 (eg, the Internet) and through one or more appropriate interfaces (eg, APIs). The performance of certain operations may not only reside within a single machine 1600, but may be distributed among processors 1610 deployed across multiple machines 1600. In some demonstrative embodiments, the processors 1610 or components implemented by the processor may be located in a single geographic location (eg, within a home environment, office environment, or server farm).. In other example embodiments, the processors 1610 or components implemented by the processor may be distributed across multiple geographic locations.

A “PROCESSOR” in this context refers to manipulating data values according to control signals (eg, “commands”, “operation codes”, “machine code”, etc.) and operating the machine 1600. refers to any circuit or virtual circuit (physical circuit emulated by logic executing in real processor 1612) that produces corresponding output signals that are applied to The processor may be, for example, a central processing unit (CPU), reduced instruction set computing (RISC) processor, complex instruction set computing (CISC) processor, graphics processing unit (GPU), digital signal processor (DSP), ASIC, radio frequency integrated circuit (RFIC), or any combination thereof. Processor 1610 further comprises a multi-core processor 1610 having two or more independent processors 1612, 1614 (sometimes referred to as “cores”) capable of concurrently executing instructions 1616. can be

"TIMESTAMP" in this context refers to a sequence of characters or encoded information that identifies when a particular event occurred, e.g., provides a date and time, sometimes accurate to the nearest fraction of a second..

The following are illustrative examples:

One. A method comprising: sending, using one or more processors of a client device, a request for venue data to one or more servers, the request comprising geolocation data generated by the client device; receiving, from one or more servers, a venue data set comprising a plurality of venues, each venue of the plurality of venues associated with tags describing the venue; identifying an image generated by the client device; classifying the object depicted in the image using a machine learning scheme; selecting a venue from the venue data set based at least in part on a classified object matching a tag associated with the venue; selecting one or more display elements previously associated with the selected venue; and displaying, on the client device, a presentation comprising one or more display elements and an image.

2. The method of Example 1, further comprising determining environmental data for the image using an additional machine learning scheme, wherein the environmental data describes whether the image is from an external environment or an internal environment.

3. The method of Examples 1-2, wherein the selection of the venue is further based, at least in part, on environmental data of the image matching a tag associated with the venue.

4. 4. The method of any one of claims 1 to 3, wherein the image is part of an active video feed generated by the client device; and the presentation displays one or more display elements overlaid on the active video feed.

5. The method of Examples 1-4, wherein the plurality of venues are categorized into categories and subcategories on the client device, and the tags are metadata of the categories or subcategories.

6. The one or more display elements of Examples 1-5, wherein the one or more display elements are categorized into categories and subcategories, wherein the one or more display elements are selected in that the one or more display elements and the selected elements share at least one of the same category or subcategory. How to pre-associate with the venue.

7. The method of Examples 1-6, wherein the one or more display elements include an avatar of a user of the client device.

8. The method of Examples 1-7, wherein the geolocation data comprises global positioning system (GPS) data generated by a GPS sensor on the client device.

9. The method of Examples 1-8, wherein the venue data set is a subset of the larger venue data set that includes venues that are not proximate to a location indicated by the client device's geolocation data.

10. The method of Examples 1-9, wherein the machine learning scheme comprises one or more neural networks trained to classify physical objects using the set of images, the set of images comprising images of the plurality of physical images.

11. The additional machine learning scheme of Examples 1-10, wherein the additional machine learning scheme includes one or more neural networks trained to classify environments as internal or external, wherein the one or more neural networks include images of a plurality of images of external and internal environments. how it is trained on a set of images that

12. The method of Examples 1-11, further comprising receiving, on the client device, an instruction to save the presentation.

13. The method of Examples 1-12, further comprising posting the presentation as a short-term message to a social networking site.

14. A system comprising: one or more processors in a machine; and a memory storing instructions that, when executed by the one or more processors, cause the machine to perform any of the methods of Examples 1-13.

15. A machine-readable storage medium embodying instructions that, when executed by a machine, cause the machine to perform any of the methods of Examples 1-13.