Image data for enhanced user interactions
21 claims: 13 independent, 8 dependent
- 1方法であって、 ディスプレイと1つ以上の画像センサとを備える電子デバイスにおいて、 アプリケーション内のコンテンツを前記ディスプレイ上に表示することであって、前記コンテンツは前記アプリケーションが第1の構成にある間に表示される、ことと、 前記コンテンツを表示しながら、前記電子デバイスの前記1つ以上の画像センサから画像データをキャプチャすることと、 前記画像データをキャプチャした後、前記コンテンツから離れるナビゲーション要求を受信することと、 前記コンテンツから離れるナビゲーション要求を受信したことに応じて、 第1のコンテンツロック基準のセットが満たされているという判定に従っ て、前 記コンテンツから離れるナビゲーションを防止 しながら、前記コンテンツから離れる前記ナビゲーション要求の視覚的表現を前記ディスプレイ上に表示することであって、 前記第1のコンテンツロック基準のセットは、不正ユーザが前記デバイスを使用していることを前記キャプチャされた画像データが示すときに満たされる第1の基準を含む、ことと、 前記第1のコンテンツロック基準のセットが満たされていないという判定に従って、前記要求に従い前記コンテンツから離れるナビゲーションを行うことと、を含む、方法。
- 2前記第1のコンテンツロック基準のセットがもはや満たされないという判定に従って、前記コンテンツから離れるナビゲーションを可能にすることを更に含む、請求項1に記載の方法。
- 3前記第1の コンテンツ ロック基準のセットは、前記電子デバイスの正規ユーザが前記デバイスを使用していないことを前記キャプチャされた画像データが示すときに満たされる第2の基準を含む、請求項1又は2に記載の方法。
- 4前記第1の コンテンツ ロック基準のセットは、前記キャプチャされた画像データに前記不正ユーザが存在し、前記キャプチャされた画像データに正規ユーザが存在しないことを、前記キャプチャされた画像データが示すときに満たされる第3の基準を含む、請求項1から3のいずれか一項に記載の方法。
- 5前記第1の コンテンツ ロック基準のセットは、前記キャプチャされた画像データに正規ユーザが存在するか否かにかかわらず、前記キャプチャされた画像データに前記不正ユーザが存在することを前記キャプチャされた画像データが示すときに満たされる、請求項1から3のいずれか一項に記載の方法。
- 6第2のコンテンツロック基準のセットが満たされているという判定に従って、前記電子デバイスの少なくとも1つの機能を無効化することを更に含む、請求項1から5のいずれか一項に記載の方法。
- 7前記第1の コンテンツ ロック基準のセットと前記第2の コンテンツ ロック基準のセットとは異なる、請求項6に記載の方法。
- 8第3のコンテンツロック基準のセットが満たされているという判定に従って、前記アプリケーションを、前記第1の構成と比較して前記アプリケーションの動作を制限する第2の構成に切り替えることを更に含む、請求項1から7のいずれか一項に記載の方法。
- 9第4のコンテンツロック基準のセットが満たされているという判定に従って、前記アプリケーション内に前記コンテンツを表示し続けながら、前記電子デバイスの他の機能をロックすることを更に含む、請求項1から8のいずれか一項に記載の方法。
- 10第5のコンテンツロック基準のセットが満たされているという判定に従って、前記電子デバイスにおいて受信された通信に関連する通知の前記表示を防止することを更に含む、請求項1から9のいずれか一項に記載の方法。
- 11前記第5の コンテンツ ロック基準のセットは、不正ユーザが前記電子デバイスを使用していることを前記キャプチャされた画像データが示すときに満たされる第4の基準を含み、前記第5の コンテンツ ロック基準のセットは、前記第4の基準が満たされる場合に満たされ、 前記第1の コンテンツ ロック基準のセットは、前記キャプチャされた画像データにおける正規ユーザの不在を前記キャプチャされた画像データが示すときに満たされる第5の基準を含む、請求項10に記載の方法。
- 12前記第4の基準が満たされていることに従って、前記電子デバイス上のアプリケーション間のナビゲーションを防止することと、 前記第5の基準が満たされていることに従って、前記アプリケーション内のナビゲーションを防止することと、を更に含む、請求項11に記載の方法。
- 13前記キャプチャされた画像データにおける前記電子デバイスの不正ユーザの存在を前記キャプチャされた画像データが示すかどうかを判定することを更に含む、請求項1から12のいずれか一項に記載の方法。
- 14前記画像データは光学的データ及び深度データを含み、前記第1のコンテンツロック基準のセットが満たされているかどうかを判定することは前記光学的データ及び前記深度データに基づいている、請求項1から13のいずれか一項に記載の方法。
- 15前記コンテンツから離れるナビゲーションは、現在表示されているコンテンツをスクロールすることによって並進させることを含む、請求項1から14のいずれか一項に記載の方法。
- 16前記コンテンツから離れるナビゲーションは、アプリケーション内のコンテンツアイテム間を切り替えることを含む、請求項1から15のいずれか一項に記載の方法。
- 17前記コンテンツから離れるナビゲーションは、アプリケーションを切り替えること、又はホーム画面を表示するために前記アプリケーションを閉じることを含む、請求項1から16のいずれか一項に記載の方法。
- 18前記電子デバイスの正規ユーザに関連付けられたロック解除情報を受信することと、 前記ロック解除情報が真正であるかどうかを判定することと、 前記ロック解除情報が真正であるという判定に従って、前記コンテンツから離れるナビゲーションを可能化することと、を更に含む、請求項1から17のいずれか一項に記載の方法。
- 19請求項1から18に記載の方法のいずれかをコンピュータに実行させるコンピュータプログラム。
- 20電子デバイスであって、 請求項19に記載のコンピュータプログラムを記憶するメモリと、 前記メモリに記憶された前記コンピュータプログラムを実行することができる1つ以上のプロセッサと、を備える、電子デバイス。
- 21電子デバイスであって、 請求項1から18に記載の方法のいずれかを実行する手段、を備える、電子デバイス。
Independent claims21
323 paragraphs, as filed
(Cross-reference to related applications) This application claims priority to US Patent Provisional Application No. 62 / 399,226 entitled "IMAGE DATA FOR ENHANCED USER INTERACTIONS" filed on September 23, 2016. For all purposes, the entire content is incorporated herein by reference. This application claims priority to Danish Patent Application No. PA2017 70418, entitled "IMAGE DATA FOR ENHANCED USER INTERACTIONS", filed May 31, 2017, for all purposes by reference. The entire content is incorporated herein.
(Technical Fields) The present disclosure relates generally to computer user interfaces, and more specifically to techniques for using avatars and / or image data for extended user interaction.
Avatar is used to represent a user of an electronic device. The avatar can represent the user's appearance, or can represent the user's idealized or completely fictitious representation. The avatar can then be associated with the user such that the appearance of the avatar to other users displays a trigger for the association or link with the user.
Many electronic devices today include sensors such as image sensors. For example, some smartphones may include image sensors on both the front and back of the phone, and may even include multiple image sensors on the same side. These image sensors are typically used to capture still images and videos, which can be shared and viewed later.
However, some techniques for extending user interaction with electronic devices using avatars and / or image data are generally cumbersome and inefficient. For example, some existing technologies use complex and time consuming user interfaces that may involve multiple key presses or keystrokes. Existing technologies require more time than necessary, wasting user time and device power. The latter problem is particularly serious in battery-powered devices.
Accordingly, the technology provides electronic devices with faster, more efficient methods and interfaces for extending user interaction using avatars and / or image data. Such methods and interfaces optionally complement or replace other methods for extending user interaction with avatars and / or image data. Such methods and interfaces reduce the cognitive burden on the user and create a more efficient human-machine interface. For battery-powered computing devices, such methods and interfaces save power and increase the time between battery charges.
According to one embodiment, the electronic device having the display and associated with the first user receives a first message containing the first content from the second user and the first to the second user. Status data of, which is associated with the first message and is separate from the first content, receives the first status data, contains the first content, and the first message and the first avatar. Is displayed simultaneously on the display, where the first avatar is based on the first status data, the displayed first avatar is adjacent to the displayed first message, the first message and the first After displaying the first avatar, it receives a second message containing the second content from the second user, receives the second status data for the second user, and the second status is the second message. The second message containing the second content and the second avatar are displayed on the display, while maintaining the display of the first message and the first avatar, which is associated with and separate from the second content. The displayed and displayed second avatar is adjacent to the displayed second message, and the second avatar is different from the first avatar and the second avatar based on the second status data.
One embodiment of a temporary computer-readable storage medium stores one or more programs, one or more programs contain instructions, and the instructions are one of electronic devices including a display and one or more input devices. When executed by the above processor, the device causes the device to display the content in the application on the display, where the content is displayed while the application is in the first configuration and while displaying the content on the electronic device. The first set of content locking criteria is set in response to having image data captured by one or more image sensors, capturing the image data, then receiving a navigation request away from the content, and receiving a navigation request away from the content. The first set of content lock criteria is a captured image of a rogue user using the device, while maintaining the display of the content while preventing navigation away from the content, according to the determination that is satisfied. It includes a first criterion that is met when the data indicates, and prompts the user to navigate away from the content as requested, according to the determination that the first set of content locking criteria is not met.
According to one embodiment, the electronic device having one or more image sensors, memories, and displays is the first image data from the one or more image sensors of the electronic device, from a first viewpoint. A second image data that captures the first image data, including the first optical image data of the object, and is the second image data from one or more image sensors of the electronic device, which is different from the first viewpoint. Second optical image of the object from one viewpoint Captures the second image data, including the optical data, selects an algorithm based on the change in viewpoint from the first viewpoint to the second viewpoint, and this Based on the algorithm, it determines the additional image data needed to continue 3D modeling of the object, and provides visual feedback on the display that provides instructions to capture the additional image data determined based on the selected algorithm. Display on.
One embodiment of a temporary computer-readable storage medium stores one or more programs, one or more programs contain instructions, and the instructions are one of electronic devices including a display and one or more image sensors. When executed by the above processor, the device includes, from one or more image sensors of the electronic device, first image data, the first optical image data of the object from the first viewpoint. A second optical image of an object from a second viewpoint that is different from the first viewpoint and is the second image data from one or more image sensors of the electronic device that captures the first image data. Capture second image data, including optical data, select an algorithm based on the change in perspective from the first perspective to the second perspective, and continue 3D modeling of the object based on this algorithm. The additional image data required for this is determined and visual feedback is displayed on the display that provides instructions to capture the additional image data determined based on the selected algorithm.
According to one embodiment, an electronic device having a display and one or more image sensors displays content in the application on the display, where the content is displayed while the application is in the first configuration. And while displaying the content, it captures image data from one or more image sensors on the electronic device, captures the image data, then receives a navigation request to leave the content, and receives a navigation request to leave the content. Correspondingly, according to the determination that the first set of content lock criteria is met, the content remains visible while preventing navigation away from the content, where the first set of content lock criteria is fraudulent. Includes a first criterion that is met when the captured image data indicates that the user is using the device, and leaves the content as requested according to the determination that the first set of content locking criteria is not met. Perform navigation.
One embodiment of a temporary computer-readable storage medium stores one or more programs, one or more programs contain instructions, and the instructions are one of electronic devices including a display and one or more image sensors. When executed by the above processor, the device causes the device to display the content in the application on the display, where the content is displayed while the application is in the first configuration and while displaying the content on the electronic device. The first set of content locking criteria is set in response to having image data captured by one or more image sensors, capturing the image data, then receiving a navigation request away from the content, and receiving a navigation request away from the content. The first set of content locking criteria is captured image data that a rogue user is using the device, while maintaining the display of the content, preventing navigation away from the content, according to the determination that is satisfied. Includes a first criterion that is met when indicates, and causes navigation to move away from the content as requested, according to the determination that the first set of content locking criteria is not met.
Executable instructions that perform these functions are optionally contained within a non-temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors. Executable instructions that perform these functions are optionally contained within a temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors.
In this way, the device is provided with a faster and more efficient method and interface for extending user interaction with image data, thereby making it effective, efficient, in such a device. And increase user satisfaction. Such methods and interfaces may complement or replace other methods for extending user interaction with image data.
In order to better understand the various embodiments described, the following "forms for carrying out the invention" should be referred to in conjunction with the following drawings. Here, similar reference numbers refer to the corresponding parts throughout all of those figures.
<figref num="1A">FIG. 3 is a block diagram showing a portable multifunction device having a touch-sensitive display according to some embodiments.</figref>
<figref num="1B">It is a block diagram which shows the exemplary component for event processing which concerns on some embodiments.</figref>
<figref num="2">A portable multifunctional device having a touch screen according to some embodiments is shown.</figref>
<figref num="3">FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch sensitive surface according to some embodiments.</figref>
<figref num="4A">An exemplary user interface to a menu of applications on a portable multifunction device according to some embodiments is shown.</figref>
<figref num="4B">An exemplary user interface is shown for a multifunctional device having a touch sensitive surface separate from the display according to some embodiments.</figref>
<figref num="5A">The personal electronic devices according to some embodiments are shown.</figref>
<figref num="5B">It is a block diagram which shows the personal electronic device which concerns on some embodiments.</figref>
<figref num="6A">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6B">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6C">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6D">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6E">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6F">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6G">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6H">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6I">An exemplary user interface for communicating user status information for a message is shown.</figref><figref num="6J">An exemplary user interface for communicating user status information for a message is shown.</figref>
<figref num="7A">It is a flow chart which shows the method for communicating the user status information for a message.</figref><figref num="7B">It is a flow chart which shows the method for communicating the user status information for a message.</figref>
<figref num="8">An exemplary functional block diagram of an electronic device is shown.</figref>
<figref num="9A">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9B">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9C">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9D">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9E">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9F">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9G">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9H">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9I">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9J">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9K">An exemplary user interface for capturing data to build a 3D model is shown.</figref><figref num="9L">An exemplary user interface for capturing data to build a 3D model is shown.</figref>
<figref num="10A">It is a flow diagram which shows the method for capturing the data which constructs a 3D model.</figref><figref num="10B">It is a flow diagram which shows the method for capturing the data which constructs a 3D model.</figref>
<figref num="11">An exemplary functional block diagram of an electronic device is shown.</figref>
<figref num="12A">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12B">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12C">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12D">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12E">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12F">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12G">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12H">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12I">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref><figref num="12J">An exemplary user interface for restricting access to data and applications based on the user is shown.</figref>
<figref num="13A">It is a flow diagram which shows the method for restricting access to data and an application based on a user.</figref><figref num="13B">It is a flow diagram which shows the method for restricting access to data and an application based on a user.</figref>
<figref num="14">An exemplary functional block diagram of an electronic device is shown.</figref>
The following description describes exemplary methods, parameters, and the like. However, it should be recognized that such description is not intended as a limitation to the scope of the present disclosure and is instead provided as a description of an exemplary embodiment.
There is a need for electronic devices that provide efficient methods and interfaces for using avatars and / or image data, not just to store the visual representation of a particular moment. By using some embodiments of the techniques described below, avatars and / or image data can be used to extend user interaction with electronic devices and other users. Such techniques reduce the cognitive burden on users who use avatars and / or image data to communicate with other users and interact with their electronic devices, thereby increasing productivity. Can be done. In addition, such techniques can reduce processor and battery power that would otherwise be wasted with redundant user input.
Hereinafter, according to FIGS. 1A to 1B, 2, 2, 3, 4A to 4B, and 5A to 5B, the techniques described below that use image data to enhance user interaction are performed. An exemplary device description for is provided.
6A-6J show exemplary user interfaces for communicating user status information for messages with avatars. 7A to 7B are flow charts showing a method for communicating user status information for a message having an avatar according to some embodiments. The user interface in FIGS. 6A-6G is used to illustrate the processes described below, including the processes in FIGS. 7A-7B.
Figures 9A-9L show an exemplary user interface for capturing data that builds a 3D model. 10A to 10B are flow charts showing a method for capturing data for constructing a 3D model according to some embodiments. The user interface in FIGS. 9A-9L is used to illustrate the processes described below, including the processes in FIGS. 10A-10B.
12A-12J show exemplary user interfaces for restricting access to data and applications based on the user. 13A-13B are flow diagrams illustrating methods for restricting access to data and applications based on the user, according to some embodiments. The user interface in FIGS. 12A-12J is used to illustrate the processes described below, including the processes in FIGS. 13A-13B.
In the following description, terms such as "first" and "second" are used to describe the various elements, but these elements should not be limited by those terms. These terms are used only to distinguish one element from another. For example, the first touch can be referred to as the second touch, and similarly the second touch can be referred to as the first touch, without departing from the scope of the various embodiments described. Both the first touch and the second touch are touches, but they are not the same touch.
The terminology used in the description of the various embodiments described herein is for the sole purpose of describing a particular embodiment and is not intended to be limiting. Within the description of the various embodiments described and the appended claims, the singular "a", "an", and "the" also include the plural unless explicitly stated otherwise in the context. Is intended. Also, as used herein, the term "and / or" refers to and includes any or all possible combinations of one or more of the relevant listed items. I want to be understood. The terms "includes," "including," "comprises," and / or "comprising," as used herein, are the features described. Specifies the existence of integers, steps, actions, elements, and / or components, but the existence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof. It will be further understood not to exclude.
The term "if" is optional, depending on the context, "when" or "upon", or "in". It is interpreted to mean "response to determining" or "in response to detecting". Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" is optional. In response to the context, "upon determining" or "in response to determining", or "[state or event] is detected. (upon detecting [the stated condition or event]) or in response to detecting [the stated condition or event] It is interpreted to mean "stated condition or event])".
Embodiments of electronic devices, user interfaces for such devices, and related processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions such as a PDA function and / or a music player function. Illustrative embodiments of portable multifunction devices include iPhone®, iPod Touch®, and iPad® devices from Apple Inc. in Cupertino, CA. Not limited. Optionally, other portable electronic devices such as laptops or tablet computers with touch sensitive surfaces (eg, touch screen displays and / or touchpads) are also used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer having a touch sensitive surface (eg, a touch screen display and / or a touch pad).
The following discussion describes electronic devices including displays and touch sensitive surfaces. However, it should be understood that electronic devices optionally include one or more other physical user interface devices such as physical keyboards, mice, and / or joysticks.
Devices typically include drawing applications, presentation applications, word processing applications, website creation applications, disk authoring applications, spreadsheet applications, gaming applications, phone applications, video conferencing applications, email applications, instant messaging applications, and training. It supports a variety of applications such as support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or one or more of digital video player applications.
Various applications running on the device optionally use at least one common physical user interface device, such as a touch sensitive surface. One or more functions of the touch sensitive surface, as well as the corresponding information displayed on the device, may be adjusted and / or modified on a per-application basis and / or optionally within each application. Thus, the device's common physical architecture (such as touch-sensitive surfaces) optionally supports a variety of applications with user interfaces that are intuitive and transparent to the user.
Here, attention is paid to an embodiment of a portable device including a touch-sensitive display. FIG. 1A is a block diagram showing a portable multifunction device 100 having a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes known or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPU) 120, peripheral interface 118, RF circuit 108, audio circuit. Includes 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input control devices 116, and external port 124. Device 100 optionally includes one or more photosensors 164. The device 100 optionally includes one or more contact strength sensors 165 (eg, a touch sensitive surface such as the touch sensitive display system 112 of the device 100) that detects the strength of contact on the device 100. The device 100 optionally produces a tactile output on the device 100 (eg, on a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100 or the touchpad 355 of the device 300). Includes one or more tactile output generators 167. These components optionally communicate via one or more communication buses or signal lines 103.
As used herein and in the claims, the term "strength" of contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact on the touch-sensitive surface (eg, finger contact), or. Refers to a proxy for the force or pressure of contact on the touch-sensitive surface. The contact strength has a range of values, including at least four distinct values, and more typically hundreds (eg, at least 256) of distinct values. Contact strength is optionally specified (or measured) using different techniques and different sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch sensitive surface are optionally used to measure forces at various points on the touch sensitive surface. In some implementations, force measurements from multiple force sensors are combined to determine the estimated force of contact (eg, weighted average). Similarly, the pressure sensitive tip of the stylus is optionally used to identify the pressure of the stylus on the touch sensitive surface. Alternatively, the size and / or its change of the contact area detected on the touch sensing surface, the electrical capacity and / or its change of the touch sensing surface close to the contact, and / or the resistance of the touch sensing surface close to the contact and / Or its variation is optionally used as a substitute for contact force or pressure on the touch sensitive surface. In some embodiments, the alternative measurement for contact force or pressure is used directly to determine if the intensity threshold is exceeded (eg, the intensity threshold is the unit corresponding to the alternative measurement). Described in). In some embodiments, the surrogate measurement for contact force or pressure is converted to an estimated force or pressure and is used to determine if the estimated force or pressure has exceeded the strength threshold ( For example, the intensity threshold is a pressure threshold measured in units of pressure). By using the strength of contact as an attribute of user input, to display affordances (eg, on a touch-sensitive display) and / or user input (eg, touch-sensitive display)
As used herein and in the claims, the term "tactile output" is the physical displacement of a device with respect to the device's previous position, which will be detected by the user by the user's tactile sensation. Refers to the physical displacement of a component (eg, touch-sensitive surface) of the device with respect to another component (eg, housing) of the device, or the displacement of the component relative to the center of mass of the device. For example, in situations where the device or a component of the device is in contact with a touch-sensitive user's surface (eg, the finger, palm, or other part of the user's hand), the tactile sensation generated by physical displacement. The output will be interpreted by the user as a tactile sensation that corresponds to a perceived change in the physical properties of the device or its components. For example, the movement of a touch-sensitive surface (eg, a touch-sensitive display or trackpad) is optionally interpreted by the user as a "down-click" or "up-click" of a physical actuator button. In some cases, the user "down-clicks" or "up-clicks" even when there is no movement of the physical actuator button associated with the touch-sensitive surface that is physically pressed (eg, displaced) by the user's actions. I feel the tactile sensation. As another embodiment, the movement of the touch-sensitive surface is optionally interpreted or sensed by the user as the "roughness" of the touch-sensitive surface, even if the smoothness of the touch-sensitive surface does not change. Will be done. The interpretation of such touches by the user depends on the user's personal sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Therefore, if the tactile output is described as corresponding to a particular sensory perception of the user (eg, "up-click", "down-click", "roughness"), the generated tactile sensation, unless otherwise stated. Intellectual output corresponds to the physical displacement of a device, or component of a device, that produces a typical (or average) user's described sensory cognition.
The device 100 is merely an embodiment of a portable multifunctional device, the device 100 optionally having more or fewer components than those shown and optionally. , It should be understood that it is a combination of two or more components, or optionally has a different composition or arrangement of those components. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.
Memory 102 optionally includes fast random access memory and optionally non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Including. The memory controller 122 optionally controls access to memory 102 by other components of device 100.
Peripheral interface 118 can be used to couple device input and output peripherals to CPU 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and process data. In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally mounted on separate chips.
The RF (radio frequency) circuit 108 sends and receives RF signals, also called electromagnetic signals. The RF circuit 108 converts an electrical signal into an electromagnetic signal, or converts an electromagnetic signal into an electrical signal, and communicates with a communication network and other communication devices via the electromagnetic signal. RF circuits 108 optionally include well-known circuits for performing these functions, including, but not limited to, antenna systems, RF transmitters and receivers, and one or more amplifiers. These include tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, memory, and more. The RF circuit 108 optionally includes the Internet, intranet, and / or cellular telephone network, wireless local area network (LAN) and / or metropolitan area network, also known as the World Wide Web (WWW). (metropolitan area Communicates wirelessly with networks such as networks, MAN) and other devices. The RF circuit 108 includes a well-known circuit that optionally detects a near field communication (NFC) field, such as by a short range communication radio. Wireless communication is optional, but not limited to, Global System for Mobile Communications (GSM®), Enhanced Data GSM Environment (EDGE). , High-speed downlink packet access (HSDPA), High-speed uplink packet access (HSUPA), Evolution data only (Evolution, Data-Only, EV-DO), HSPA, HSPA +, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), short-range communication (NFC), wideband code division multiple access (wideband code division multiple access, W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth®, Bluetooth Low Energy (Bluetooth Low) Energy, BTLE®), Wireless Fidelity, Wi-Fi® (eg, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol Protocol, VoIP), Wi-MAX®, e-mail protocols (eg, Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging ( For example, extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), instant messaging and presence. Event Package (Instant Messaging and Presence Service (IMPS)) and / or Short Message Service, SMS), or any of a plurality of communication standards, protocols, and technologies, including any other suitable communication protocol, including communication protocols that have not yet been developed as of the filing date of this specification. use.
The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electric signal, and transmits the electric signal to the speaker 111. The speaker 111 converts an electrical signal into a human audible sound wave. The audio circuit 110 also receives an electrical signal converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral interface 118 for processing. The audio data is optionally acquired from the memory 102 and / or the RF circuit 108 by the peripheral interface 118 and / or transmitted to the memory 102 and / or the RF circuit 108. In some embodiments, the audio circuit 110 also includes a headset jack (eg, 212 in FIG. 2). The headset jack is between the audio circuit 110 and removable audio input / output peripherals such as output-only headphones or headsets that have both output (eg, one-ear or binaural headphones) and input (eg, microphone). Provides an interface for.
The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. Including. One or more input controllers 160 receive electrical signals from the other input control device 116 / transmit electrical signals to the other input control device 116. Other input control devices 116 optionally include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, the input controller 160 is optionally coupled (or not coupled to) any of pointer devices such as keyboards, infrared ports, USB ports, and mice. One or more buttons (eg, 208 in FIG. 2) optionally include up and down buttons for volume control of the speaker 111 and / or microphone 113. One or more buttons optionally include a push button (eg, 206 in FIG. 2).
Described in U.S. Patent Application No. 11 / 322,549, "Unlocking a Device by Performing Gestures on an Unlock Image," U.S. Pat. No. 7,657,849, filed December 23, 2005, incorporated herein by reference in its entirety. As has been done, a quick push button press optionally disengages the touch screen 112, or optionally unlocks the device using a gesture on the touch screen. To start. A longer press of a push button (eg, 206) optionally turns power on or off the device 100. The functionality of one or more of the buttons can be optionally customized by the user. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.
The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives the electrical signal from the touch screen 112 and / or transmits the electrical signal to the touch screen 112. The touch screen 112 displays a visual output to the user. This visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output corresponds to the user interface object, optionally.
The touch screen 112 comprises a touch sensing surface, a sensor, or a set of sensors that accept input from the user based on tactile and / or tactile contact. The touch screen 112 and the display controller 156 (along with any associated module and / or instruction set in memory 102) detect a contact (and any movement or interruption of contact) on the touch screen 112, and the detected contact is detected by the touch screen. 112 Convert to a interaction with the user interface object displayed above (eg, one or more softkeys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.
The touch screen 112 optionally uses LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but in other embodiments, other display technologies are also used. To. The touch screen 112 and display controller 156 optionally, but are not limited to, capacitive, resistive, infrared, and surface acoustic techniques, as well as one or more points of contact with the touch screen 112. Any of the currently known or upcoming touch sensing techniques, including other proximity sensor arrays or other elements to determine, is used to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing techniques, such as those found on the iPhone® and iPod Touch® from Apple Inc. in Cupertino, CA, are used.
The touch-sensitive displays in some embodiments of the touch screen 112 are optionally incorporated herein by reference, respectively, US Pat. Nos. 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.). ), And / or similar to the multi-touch sensitive touchpad described in US Pat. No. 6,677,932 (Westerman) and / or US Pat. No. 2002 / 0015024A1. However, the touch screen 112 displays the visual output from the device 100, whereas the touch-sensitive touchpad does not provide the visual output.
Touch-sensitive displays in some embodiments of the touch screen 112 include (1) US Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller," (2) May 6, 2004. US Patent Application No. 10 / 840,862 filed in Japan, "Multipoint Touchscreen", (3) US Patent Application No. 10 / 903,964 filed on July 30, 2004, "Gestures For Touch Sensitive Input Devices", (4) 2005 US Patent Application No. 11 / 048,264 filed January 31, 2005, "Gestures For Touch Sensitive Input Devices", (5) US Patent Application No. 11 / 038,590 filed January 18, 2005, "Mode-Based Graphical" User Interfaces For Touch Sensitive Input Devices , (6) US Patent Application No. 11 / 228,758 filed on September 16, 2005, Virtual Input Device Placement On A Touch Screen User Interface , (7) US Patent Application filed on September 16, 2005 No. 11 / 228,700, "Operation Of A Computer With A Touch Screen Interface", (8) US Patent Application No. 11 / 228,737 filed on September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard" , And (9) US Patent Application No. 11 / 367,749, filed March 3, 2006, in the application "Multi-Functional Hand-Held Device". All of these applications are incorporated herein by reference in their entirety.
The touch screen 112 optionally has a video resolution of greater than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally touches the touch screen 112 using any suitable object or accessory such as a stylus or finger. In some embodiments, the user interface is designed primarily to handle finger-based contact and gestures, which is as precise as stylus-based input due to the larger finger contact area on the touch screen. It may not be. In some embodiments, the device translates a rough finger-based input into an exact pointer / cursor position or command to perform the action desired by the user.
In some embodiments, in addition to the touch screen, device 100 optionally includes a touch pad (not shown) for activating or deactivating a particular function. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike a touchscreen, does not display visual output. The touchpad is optionally a touch sensing surface that is separate from the touch screen 112 or an extension of the touch sensing surface formed by the touch screen.
The device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (eg, battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (eg, light emitting). Includes diodes (LEDs)), as well as any other components associated with power generation, management, and distribution within portable devices.
The device 100 also optionally includes one or more optical sensors 164. FIG. 1A shows an optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 may optionally be a charge-coupled device (CCD) or a complementary metal-oxide. semiconductor, CMOS) Includes phototransistors. The light sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. The optical sensor 164, in conjunction with the imaging module 143 (also called the camera module), optionally captures a still image or video. In some embodiments, the optical sensor is located on the back of the device 100, as opposed to the touch screen display 112 on the front of the device, so the touch screen display is for the acquisition of still and / or video images. It is effective to use it as a viewfinder of. In some embodiments, the optical sensor is located in front of the device, so that the user's image is optionally obtained for the video conference and the user touch screens other video conference participants. View on the display. In some embodiments, the position of the light sensor 164 can be changed by the user (eg, by rotating the lens and sensor within the device housing), thus a single light sensor 164 can be a touch screen display. It is also used for video conferencing and acquisition of still and / or video images.
The device 100 also optionally includes one or more contact strength sensors 165. FIG. 1A shows a contact strength sensor coupled to a strength sensor controller 159 in the I / O subsystem 106. The contact strength sensor 165 may optionally include one or more piezo resistance strain gauges, capacitive force sensors, electrical force sensors, pressure power sensors, optical force sensors, capacitive touch sensing surfaces, or other strength sensors (eg,). , A sensor used to measure the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (eg, pressure information or a proxy for pressure information) from the environment. In some embodiments, the at least one contact strength sensor is juxtaposed with or close to a touch-sensitive surface (eg, touch-sensitive display system 112). In some embodiments, the at least one contact strength sensor is located on the back surface of the device 100, as opposed to the touch screen display 112, which is located on the front surface of the device 100.
The device 100 also optionally includes one or more proximity sensors 166. FIG. 1A shows the proximity sensor 166 coupled to the peripheral interface 118. Instead, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106 . Proximity sensor 166 is optionally incorporated herein by reference in its entirety, US Patent Application No. 11 / 241,839, "Proximity Detector In Handheld Device", No. 11 / 240,788, "Proximity Detector In". Handheld Device "11 / 620,702," Using Ambient Light Sensor To Augment Proximity Sensor Output ", 11 / 586,862," Automated Response To And Sensing Of User Activity In Portable Devices ", and 11 / 638,251," Methods As described in And Systems For Automatic Configuration Of Peripherals, these applications are incorporated herein by reference in their entirety. In some embodiments, the proximity sensor turns off and disables the touch screen 112 when the multifunction device is placed near the user's ear (eg, when the user is making a call).
The device 100 also optionally includes one or more tactile output generators 167. FIG. 1A shows a tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally comprises one or more electroacoustic devices such as speakers or other audio components and / or motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or the like. Includes electromechanical devices that convert energy into linear motion, such as tactile output generation components (eg, components that convert electrical signals to tactile output on the device). The contact strength sensor 165 receives a tactile feedback generation command from the tactile feedback module 133 and generates a tactile output on the device 100 that can be perceived by the user of the device 100. In some embodiments, at least one tactile output generator is located on or in close proximity to a touch-sensitive surface (eg, touch-sensitive display system 112) and optionally the touch-sensitive surface. To generate a tactile output by moving the device vertically (eg, inside / outside the surface of the device 100) or horizontally (eg, back and forth in the same plane as the surface of the device 100). In some embodiments, the at least one tactile output generator sensor is located on the back surface of the device 100, as opposed to the touch screen display 112, which is located on the front surface of the device 100.
The device 100 also optionally includes one or more accelerometers 168. FIG. 1A shows the accelerometer 168 coupled to the peripheral interface 118. Instead, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 is optionally incorporated herein by reference, both in reference to U.S. Patent Publication No. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices," and U.S. Patent Publication No. 1. 20060017692, "Methods And MFPes For Operating A Portable Device Based On An It works as described in Accelerometer. In some embodiments, the information is displayed on a touch screen display in portrait or landscape view based on the analysis of data received from one or more accelerometers. The device 100 is optional, in addition to the accelerometer (s) 168, a magnetometer (not shown) and a GPS for obtaining information about the position and orientation (eg, portrait or landscape) of the device 100. Or GLONASS or other global navigation system) receivers (not shown) included.
In some embodiments, the software components stored in memory 102 are operating system 126, communication module (or instruction set) 128, contact / motion module (or instruction set) 130, graphic module (or instruction set). ) 132, text input module (or instruction set) 134, Global Positioning Includes System, GPS) module (or instruction set) 135, and application (or instruction set) 136. Further, in some embodiments, memory 102 (FIG. 1A) or memory 370 (FIG. 3) stores device / global internal state 157, as shown in FIGS. 1A and 3. Device / global internal state 157 indicates the active application state, which application, view, or other information, if any, is currently active, in various areas of the touchscreen display 112. Includes one or more of display states indicating whether the device is occupied, sensor states including information obtained from various sensors and input control devices 116 of the device, and location information regarding the location and / or orientation of the device.
Operating system 126 (eg, Darwin®, RTXC®, LINUX®, UNIX®, OS X®, iOS®, WINDOWS, or Embedded operating systems such as VxWorks® include various software components and / or drivers that control and manage normal system tasks (eg, memory management, storage device control, power management, etc.). Facilitates communication between various hardware and software components.
The communication module 128 also facilitates communication with other devices via one or more external ports 124 and also contains various software components for processing the data received by the RF circuit 108 and / or the external port 124. Including. External ports 124 (eg Universal Serial Bus (USB), FIREWIRE®, etc.) can be directly or indirectly over a network (eg, Internet, WiFi, etc.) to other devices. Adapted to connect. In some embodiments, the external port is the same or similar to the 30-pin connector used on the iPod® (Apple Inc. trademark) device and / or is compatible with multi-pin (eg, Apple Inc.) devices. , 30 pin) connector.
The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (eg, touchpad or physical click wheel). The contact / movement module 130 determines if a contact has occurred (eg, detecting a finger-lowering event), the strength of the contact (eg, the force or pressure of the contact, or an alternative to the force or pressure of the contact). To determine (object), to determine if a contact movement is present, to track movement across the touch-sensitive surface (eg, to detect one or more finger dragging events), and Includes various software components to perform various actions related to contact detection, such as determining if a contact has stopped (eg, detecting a finger raising event or contact interruption). .. The contact / motion module 130 receives contact data from the touch sensing surface. Determining the movement of a contact point, represented by a series of contact data, is optionally the speed (magnitude), velocity (magnitude and direction), and / or acceleration (magnitude and direction) of the contact point. / Or change of direction) includes determining. These actions are optionally applied to a single contact (eg, one finger contact) or multiple simultaneous contacts (eg, "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.
In some embodiments, the contact / motion module 130 uses one or more sets of intensity thresholds to determine if an operation has been performed by the user (eg, the user "clicks" on the icon. Determine if). In some embodiments, at least one subset of the intensity threshold is determined according to software parameters (eg, the intensity threshold is not determined by the activation threshold of a particular physical actuator, but rather the physical hardware of device 100. Can be adjusted without changing the wear). For example, the mouse "click" threshold of a trackpad or touchscreen display can be set to any of a wide range of predetermined thresholds without changing the trackpad or touchscreen display hardware. In addition, in some implementations, the user of the device is provided with software settings that adjust one or more of the set of intensity thresholds (eg, a system-level click "intensity" parameter to set individual intensity thresholds. By adjusting and / or by adjusting multiple intensity thresholds at once).
The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different detected contact movements, timings, and / or intensities). Therefore, gestures are detected by optionally detecting a particular contact pattern. For example, detecting a finger tap gesture detects a finger down event, followed by a finger up (lift off) event at the same position (or substantially the same position) as the finger down event (for example, the position of the icon). Including doing. As another example, detecting a finger swipe gesture on a touch-sensitive surface detects a finger down event, followed by one or more finger drag events, followed by a finger up (lift off) event. Includes detecting.
The graphics module 132 includes components that change the visual effects of the displayed graphics (eg, brightness, transparency, saturation, contrast, or other visual characteristics) on the touch screen 112 or other display. Includes various known software components that render and display graphics in. As used herein, the term "graphic" is not limited to that, but includes, but is not limited to, characters, web pages, icons (such as user interface objects including softkeys), digital images, videos, animations, etc. Includes any object that can be displayed.
In some embodiments, the graphics module 132 stores data representing the graphics used. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives, if necessary, one or more codes that specify the graphic to be displayed, along with coordinate data and other graphic characteristic data, from an application or the like, and then outputs it to the display controller 156. Generate screen image data.
The tactile feedback module 133 is used by the tactile output generator (s) 167 to generate tactile outputs at one or more locations on the device 100 in response to user interaction with the device 100. Includes various software components for generating instructions.
The text input module 134 is an optional component of the graphic module 132 and is a variety of applications (eg, contacts 137, email 140, IM141, browser 147, and any other application that requires text input. ) Provides a soft keyboard for entering text.
The GPS module 135 locates the device and uses this information in various applications (eg, to the phone 138 for use in position-based dial calls, camera 143 as photo / video metadata. To applications that provide location-based services, such as weather widgets, regional yellow page widgets, and map / navigation widgets.
Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof. Contact Module 137 (sometimes called address book or contact list), Telephone Module 138, Video Conferencing Module 139, Email Client Module 140, Instant messaging (IM) Module 141, Training support module 142, Camera module 143 for still and / or video images, Image management module 144, Video player module, Music player module, Browser module 147, Calendar module 148, Optional Of Meteorological Widget 149-1, Stock Price Widget 149-2, Computer Widget 149-3, Alarm Clock Widget 149-4, Dictionary Widget 149-5, and Other Widgets Obtained by User, and User Created Widget 149-6 Widget module 149, which contains one or more of Widget creator module 150 to create user-created widget 149-6, Search module 151, Video and music player module 152 that integrates video player module and music player module, Memo module 153, Map module 154, and / Or Online video module 155.
Examples of other applications 136 that are optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA® compatible applications, encryption, digital writing. These include rights management, speech recognition, and speech replication.
In conjunction with the touch screen 112, display controller 156, contact / movement module 130, graphic module 132, and text input module 134, the contact module 137 optionally adds a name to the address book, address book. Remove a name (s) from, associate a phone number (s), email address (s), address (s), or other information with a name, make an image into a name Includes associating, sorting by name, providing telephone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM141, etc. It is used to manage an address book or contact list (eg, stored in application internal state 192 of contact module 137 in memory 102 or memory 370).
In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the telephone module 138 is optional. Enter the character sequence corresponding to the phone number, access one or more phone numbers in the contact module 137, correct the phone number entered, dial each phone number, hold the conversation, end the conversation It is used to hang up or hang up when you do. As mentioned above, wireless communication optionally uses any of a plurality of communication standards, protocols, and technologies.
RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphic module 132, text input module 134, contact module 137 , And in conjunction with the telephone module 138, the video conferencing module 139 includes executable instructions to start, execute, and end a video conference between the user and one or more other participants in accordance with user instructions.
In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the email client module 140 creates and sends email in response to user instructions. Includes executable instructions to receive, and manage. In connection with the image management module 144, the email client module 140 makes it very easy to compose and send an email with a still or video image taken by the camera module 143.
In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the instant messaging module 141 inputs the character sequence corresponding to the instant message, previously. Correct the characters entered, send each instant message (for example, Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephone communication-based instant messages, or Internet-based instant messages. Includes actionable instructions for receiving instant messages (using XMPP, SIMPLE, or IMPS), and viewing received instant messages. In some embodiments, the sent and / or received instant message is optionally MMS and / or Enhanced Messaging. Includes graphics, photos, audio files, video files, and / or other attachments supported by Service, EMS). As used herein, "instant messaging" uses telephone communication-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, XMPP, SIMPLE, or IMPS). (Messages sent by) and both.
In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / movement module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and music player module, the training support module 142 Create training (eg with time, distance, and / or calorie burning targets), communicate with training sensors (sports devices), receive training sensor data, and calibrate the sensors used to monitor training. Includes actionable instructions for selecting and playing music for training, displaying, storing, and transmitting training data.
In conjunction with the touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphic module 132, and image management module 144, the camera module 143 provides a still image or video (video stream). Includes actionable instructions for capturing (including) and storing in memory 102, modifying the characteristics of a still image or video, or deleting a still image or video from memory 102.
In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, and camera module 143, the image management module 144 arranges and modifies still and / or video images ( Includes executable instructions for, for example, editing) or other operations, labeling, deleting, presenting (eg, by digital slideshow or album), and memory.
In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the browser module 147 is linked to a web page or part thereof, as well as a web page. Includes actionable instructions for browsing the Internet according to user instructions, including searching, linking, receiving, and displaying attachments and other files.
In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 follows user instructions. Includes actionable instructions to create, view, modify, and store calendars and data associated with them (eg, calendar entries, module lists, etc.).
In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / movement module 130, graphic module 132, text input module 134, and browser module 147, widget module 149 is optionally downloaded and used by the user. Mini-applications (eg Meteorological Widget 149-1, Stock Widget 149-2, Computer Widget 149-3, Alarm Clock Widget 149-4, and Dictionary Widget 149-5), or mini-applications created by the user (eg , User-created widget 149-6). In some embodiments, the widget is an HTML (Hypertext Markup Language) file, CSS (Cascading Style). Includes Sheets; Cascading Style Sheets) files and JavaScript® files. In some embodiments, the widget includes an XML (extensible markup language) file and a JavaScript (registered trademark) file (eg, Yahoo! Widget).
In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / movement module 130, graphic module 132, text input module 134, and browser module 147, the widget creator module 150 optionally creates widgets. Used by users to do (eg, make a user-specified part of a web page a widget).
In conjunction with the touch screen 112, display controller 156, contact / movement module 130, graphic module 132, and text input module 134, the search module 151 follows one or more search criteria (eg, one or more users) according to user instructions. Includes executable instructions to search for characters, music, sounds, images, videos, and / or other files in memory 102 that match the designated search term).
In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, the video and music player module 152 can be user MP3 or Executeable instructions that allow you to download and play recorded music and other sound files stored in one or more file formats, such as AAC files, as well as display, presentation, or other methods of video. Includes executable instructions to play on (eg, on a touch screen 112 or on an external display connected via an external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (Apple trademark).
In conjunction with the touch screen 112, display controller 156, contact / movement module 130, graphic module 132, and text input module 134, the memo module 153 is executable to create and manage memos, tudulists, etc. according to user instructions. Includes instructions.
In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally Receive, display, modify, and store maps and data associated with maps (eg, data about driving directions, specific locations or nearby stores and other target points, and other location-based data) in accordance with user instructions. Used to do.
Online in conjunction with touch screen 112, display controller 156, contact / motion module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147. The video module 155 is connected by a user to access a particular online video, browse a particular online video, receive (eg, by streaming and / or download), play (eg, on a touch screen or via an external port 124). Includes instructions that allow you to send e-mails with links to specific online videos (on an external display), as well as other management of online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of an email client module 140 to send a link to a particular online video. Additional instructions for online video applications are incorporated herein by reference in their entirety, US Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, "Portable Multifunction Device, Method," And Graphical User Interface for Playing Online Videos , and US Patent Application No. 11 / 968,067 filed December 31, 2007, Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos .
Each of the modules and applications identified above is an executable instruction that performs one or more of the functions described above and the methods described in this application (eg, computer implementation methods and other information processing methods described herein). Corresponds to the set. These modules (eg, instruction sets) do not need to be implemented as separate software programs, procedures, or modules, so in various embodiments, various subsets of these modules are optionally combined. , Or otherwise reconstructed. For example, the video player module is optionally combined with the music player module into a single module (eg, the video and music player module 152 in FIG. 1A). In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. In addition, memory 102 optionally stores additional modules and data structures not described above.
In some embodiments, device 100 is a device in which the operation of a set of predetermined functions on the device is performed only through the touch screen and / or touch pad. Optionally, the number of physical input control devices (push buttons, dials, etc.) on the device 100 by using the touch screen and / or touchpad as the primary input control device for the device 100 to operate. Is reduced.
A set of predetermined functions performed only through the touch screen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad navigates the device 100 to the main menu, home menu, or root menu from any user interface displayed on the device 100 when touched by the user. In such an embodiment, the "menu button" is implemented using a touchpad. In some other embodiments, the menu button is not a touchpad, but a physical push button or other physical input control device.
FIG. 1B is a block diagram showing exemplary components for event handling according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) is the event sorter 170 (eg, in operating system 126) and the respective application 136-1 (eg, applications 137-151 described above). , 155, any of 380-390) and included.
The event sort unit 170 receives the event information and determines the application view 191 of the application 136-1 and the application 136-1 to which the event information is distributed. The event sort unit 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 indicating the current application view displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by the event sorter 170 to determine which application (s) are currently active, and application internal state 192 is the event information. Used by the event sorter 170 to determine which application view 191 is being delivered.
In some embodiments, the application internal state 192 is the resume information used when application 136-1 resumes execution, the information displayed or ready to be displayed by application 136-1. One or more of the user interface state information that indicates, a state queue that allows the user to return to the previous state or view of application 136-1, and a redo / undo queue for previous actions taken by the user. Contains additional information such as.
The event monitor 171 receives event information from the peripheral device interface 118. Event information includes information about sub-events (eg, user touches on the touch-sensitive display 112 as part of a multi-touch gesture). Peripheral interface 118 transmits information received from sensors such as the I / O subsystem 106, or proximity sensor 166, accelerometer (s) 168, and / or microphone 113 (via audio circuit 110). .. The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display 112 or the touch-sensitive surface.
In some embodiments, the event monitor 171 sends a request to the peripheral interface 118 at predetermined intervals. Accordingly, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when there is a significant event (eg, reception of an input that exceeds a predetermined noise threshold and / or exceeds a predetermined duration).
In some embodiments, the event sort unit 170 also includes a hit view determination module 172 and / or an active event recognition unit determination module 173.
The hit view determination module 172 provides a software procedure that determines where in one or more views a sub-event occurred when the touch-sensitive display 112 displays two or more views. The view consists of controls and other elements that the user can see on the display.
Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (for each application) for which touch is detected optionally corresponds to the program hierarchy of the application or the program level within the view hierarchy. For example, the lowest level view where a touch is detected is optionally called a hit view, and a set of events that are recognized as appropriate input is optionally an initial start of a touch gesture. Determined based at least in part on the hit view of the touch.
The hit view determination module 172 receives information related to touch-based gesture sub-events. When the application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchy to handle subevents. In most situations, the hit view is the lowest level view at which the starting subevent (eg, the first subevent in the subevent sequence that forms the event or potential event) takes place. After the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source identified as the hit view.
The active event recognizer determination module 173 determines which view (s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In another embodiment, the active event recognizer determination module 173 determines that all views, including the physical position of the sub-event, are active-related views, and thus all active-related views. Determines that a particular sequence of subevents should be received. In other embodiments, the higher views in the hierarchy remain the views that are actively involved, even if the touch subevents are completely confined to the area associated with one particular view.
The event dispatcher module 174 dispatches event information to an event recognition unit (for example, event recognition unit 180). In the embodiment including the active event recognition unit determination module 173, the event dispatcher module 174 distributes event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores event information acquired by each event receiver 182 in the event queue.
In some embodiments, the operating system 126 includes an event sorter 170. As an alternative, application 136-1 includes event sorter 170. In yet another embodiment, the event sorter 170 is part of a stand-alone module or another module stored in memory 102, such as the contact / motion module 130.
In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191 each containing instructions for processing touch events that occur within each view of the application's user interface. including. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are separate modules, such as user interface kits (not shown) or higher level objects for which application 136-1 inherits methods and other characteristics. Is part of. In some embodiments, the corresponding event processing unit 190 is one or more of data updating unit 176, object updating unit 177, GUI updating unit 178 and / or event data 179 received from event sorting unit 170. including. The event processing unit 190 optionally uses or calls the data update unit 176, the object update unit 177, or the GUI update unit 178 to update the application internal state 192. Alternatively, one or more of the application views 191 includes one or more of each event processor 190. Further, in some embodiments, one or more of the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in the corresponding application view 191.
Each event recognition unit 180 receives event information (for example, event data 179) from the event sort unit 170, and identifies an event from this event information. The event recognition unit 180 includes an event reception unit 182 and an event comparison unit 184. In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188, optionally including sub-event delivery instructions.
The event receiving unit 182 receives event information from the event sorting unit 170. Event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When a sub-event involves a touch movement, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event involves rotation of the device from one orientation to another (eg, from portrait to landscape, or vice versa), and the event information is the current orientation of the device (device). Includes corresponding information about (also called posture).
The event comparison unit 184 compares the event information with the definition of the default event or sub-event, and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparison unit 184 includes event definition 186. Event definition 186 includes definitions of events (eg, sequences of predetermined sub-events) such as event 1 (187-1) and event 2 (187-2). In some embodiments, the sub-event within event (187) includes, for example, touch start, touch end, touch movement, touch cancel, and multiple touches. In one embodiment, the definition for event 1 (187-1) is a double tap on the displayed object. Double tapping is, for example, a first touch (touch start) on the display object for a predetermined stage, a first lift-off (touch end) for a predetermined stage, and a second touch (touch) on the display object for a predetermined stage. Includes start) and a second lift-off (touch end) to a given stage. In another embodiment, the definition of event 2 (187-2) is a drag on the displayed object. Dragging includes, for example, touching (or touching) on a display object with respect to a given stage, touch movement on the touch-sensitive display 112, and lift-off (touch termination) of the touch. In some embodiments, the event also contains information about one or more associated event handlers 190.
In some embodiments, the event definition 187 includes an event definition for each user interface object. In some embodiments, the event comparison unit 184 performs a hit test to determine which user interface object is associated with the subevent. For example, in an application view in which three user interface objects are displayed on the touch-sensitive display 112, when a touch is detected on the touch-sensitive display 112, the event comparison unit 184 will perform the event comparison unit 184 among the three user interface objects. Run a hit test to determine which is associated with that touch (sub-event). When each displayed object is associated with the corresponding event processing unit 190, the event comparing unit uses the result of the hit test to determine which event processing unit 190 should be activated. For example, the event comparison unit 184 selects an event processing unit associated with a sub-event and an object that triggers a hit test.
In some embodiments, the definition for each event (187) also includes a delay action that delays the transmission of event information until after it is determined whether the sub-event sequence corresponds to the event type of the event recognizer. Including.
If each event recognizer 180 determines that the series of sub-events does not match any of the events in event definition 186, each event recognizer 180 decides that the event is impossible, the event fails, or the event ends. It goes into a state, after which it ignores subsequent subevents of the touch gesture. In this situation, if there is another event recognizer that remains active for the hit view, that event recognizer will continue to track and process sub-events of gestures with ongoing touches.
In some embodiments, the corresponding event recognizer 180 indicates configurable properties, flags, and / of to the event recognizer that is actively involved in how the event delivery system should perform subevent delivery. Or it contains metadata 183 with a list. In some embodiments, the metadata 183 is a configurable property that indicates how event recognizers interact with each other, or how event recognizers are allowed to interact with each other. Includes flags and / or lists. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists indicating whether subevents are delivered to various levels in the view hierarchy or program hierarchy.
In some embodiments, each event recognizer 180 activates an event processor 190 associated with an event when one or more specific subevents of the event are recognized. In some embodiments, each event recognition unit 180 delivers event information associated with the event to the event processing unit 190. Activating the event processor 190 is separate from sending (and deferring) sub-events to each hit view. In some embodiments, the event recognizer 180 throws in a flag associated with the recognized event, and the event processor 190 associated with the flag catches the flag and executes a default process.
In some embodiments, the event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about the sub-event without activating the event processor. Instead, the sub-event delivery instruction delivers event information to the event processor associated with the set of sub-events, or to the views that are actively involved. The event processing unit associated with a series of sub-events or views actively involved receives event information and executes a predetermined process.
In some embodiments, the data updater 176 creates and updates the data used by application 136-1. For example, the data update unit 176 updates the telephone number used by the contact module 137 or stores the video file used by the video player module. In some embodiments, the object updater 177 creates and updates the objects used by application 136-1. For example, the object update unit 177 creates a new user interface object or updates the position of the user interface object. The GUI update unit 178 updates the GUI. For example, the GUI updater 178 prepares the display information and sends the display information to the graphic module 132 for display on the touch-sensitive display.
In some embodiments, the event processing unit (s) 190 includes or has access to a data update unit 176, an object update unit 177, and a GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in a single module of the respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.
The above discussion of user touch event handling on touch-sensitive displays also applies to other forms of user input for operating the multifunction device 100 with input devices, and those user inputs are not necessarily all touch screens. It will be understood that it does not always start above. For example, mouse movement and mouse button presses, touch movements such as tapping, dragging, scrolling on the touchpad, pen stylus input, device devices, which are optionally linked to single or multiple keyboard presses or holds. Use movement, verbal commands, detected eye movements, biometric inputs, and / or any combination thereof as inputs corresponding to sub-events that optionally define the event to be recognized.
FIG. 2 shows a portable multifunction device 100 with a touch screen 112 according to some embodiments. The touch screen is optionally a user interface (user) interface, UI) Display one or more graphics in 200. In this embodiment, and other embodiments described below, the user may see, for example, one or more fingers 202 (not drawn to exact scale in the figure) or one or more stylus 203 (in the figure). Is not drawn to the exact scale), and by performing gestures on the graphics, it is possible to select one or more of those graphics. In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with the one or more graphics. In some embodiments, the gesture optionally contacts one or more taps, one or more swipes (left to right, right to left, up and / or down), and / or device 100. Includes finger rolls (right-to-left, left-to-right, up, and / or down). In some implementations or situations, unforeseen contact with a graphic does not select that graphic. For example, if the gesture corresponding to the selection is a tap, the swipe gesture that sweeps over the application icon does not optionally select the corresponding application.
The device 100 also optionally includes one or more physical buttons such as a "home" button or a menu button 204. As mentioned above, menu button 204 is used to optionally navigate to any application 136 in the set of applications running on device 100. Alternatively, in some embodiments, the menu button is implemented as a softkey in the GUI displayed on the touch screen 112.
In some embodiments, the device 100 has a touch screen 112, a menu button 204, a push button 206 that locks the device by turning the device on and off, a volume control button 208, a subscriber identification module (SIM) card slot. Includes 210, headset jack 212, and external docking / charging port 124. The push button 206 optionally presses the button down and holds the button down for a predetermined period of time to power on / off the device, so that the button is pressed down for a predetermined period of time. Used to lock the device and / or to unlock the device or initiate the unlocking process by releasing the button before doing so. In an alternative embodiment, device 100 also accepts verbal input to activate or deactivate some functions via microphone 113. The device 100 also optionally produces one or more contact strength sensors 165 that detect the strength of the contact on the touch screen 112 and / or one or more tactile outputs to the user of the device 100. Includes output generator 167.
FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop computer, desktop computer, tablet computer, multimedia playback device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (eg, home). Computer for business use or business use). The device 300 typically includes one or more processing units (CPUs) 310, one or more networks or other communication interfaces 360, memory 370, and one or more communication buses that interconnect these components. Including 320. The communication bus 320 optionally includes circuits (sometimes referred to as chipsets) that interconnect and control communication between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally produces a keyboard and / or mouse (or other pointing device) 350, a touchpad 355, and a tactile output generator 357 (tactile output generator) on the device 300. For example, light, acceleration, similar to the tactile output generator 167 described above with reference to FIG. 1A) and sensor 359 (eg, similar to the contact strength sensor 165 described above with reference to FIG. 1A). Includes proximity, touch sensing, and / or contact strength sensors). Memory 370 is DRAM, SRAM, DDR One or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage, including high-speed random access memory such as RAM, or other random access solid-state memory devices. Includes non-volatile memory such as devices. Memory 370 optionally includes one or more storage devices located remote from CPU 310. In some embodiments, the memory 370 is a program, module, and data structure similar to, or a subset of, a program, module, and data structure stored in memory 102 of the portable multifunction device 100 (FIG. 1A). Remember. In addition, memory 370 optionally stores additional programs, modules, and data structures that do not exist in memory 102 of portable multifunction device 100. For example, the memory 370 of the device 300 could optionally store a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390. In contrast, the memory 102 of the portable multifunction device 100 (FIG. 1A) optionally does not store these modules.
Each of the elements identified above in FIG. 3 is optionally stored in one or more of the aforementioned memory devices. Each module identified above corresponds to an instruction set that performs the functions described above. The modules or programs identified above (eg, instruction sets) need not be implemented as separate software programs, procedures, or modules, so in various embodiments, various subsets of these modules are optional. , Combined, or otherwise reconstructed. In some embodiments, the memory 370 optionally stores a subset of the modules and data structures identified above. In addition, memory 370 optionally stores additional modules and data structures not described above.
Next, attention is paid to an embodiment of a user interface optionally implemented, for example, on the portable multifunction device 100.
FIG. 4A shows an exemplary user interface for a menu of applications on a portable multifunction device 100 according to some embodiments. A similar user interface is optionally implemented on the device 300. In some embodiments, the user interface 400 includes the following elements, or a subset or superset thereof. Signal strength indicator (s) for wireless communication (s) such as cellular and Wi-Fi signals 402, Time 404, Bluetooth® indicator 405, Battery status indicator 406, Tray 408 with icons for frequently used applications, such as: Phone module 138 labeled "phone", optionally containing indicator 414 for the number of missed calls or voicemail messages. Icon for 416, Icon for Email Client Module 140, labeled "Mail", optionally including indicator 410 for the number of unread emails, Icon 420 for browser module 147, labeled "Browser", and For video and music player module 152, also known as iPod (Apple Inc. trademark) module 152, labeled "iPod" Icon 422, and Icons for other applications, such as Icon for IM Module 141, labeled Message 424, Icon for Calendar Module 148, labeled Calendar Icon 426, Icon for image management module 144, labeled "Photo", Icon 430 for camera module 143, labeled "Camera", Labeled as "Online Video" Apple, icon 432 for online video module 155, icon 434 for stock price widget 149-2 labeled "stock", icon 436 for map module 154 labeled "map", " Icon 438 for Meteorological Widget 149-1, labeled "Weather" Icon 440 for alarm clock widget 149-4 labeled Clock, Icon 442 for training support module 142 labeled Training Support, Labeled Memo , Icon 444 for memo module 153, and Icon 446 for a configuration application or module labeled "Settings" that provides access to configuration for device 100 and its various applications 136.
Note that the icon label shown in Figure 4A is merely an example. For example, the icon 422 for the video and music player module 152 may optionally be labeled as "music" or "music player" with other labels for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to each application icon. In some embodiments, the label of a particular application icon is different from the name of the application that corresponds to that particular application icon.
FIG. 4B is an example on a device (eg, device 300 in FIG. 3) having a touch sensitive surface 451 (eg, tablet or touchpad 355 in FIG. 3) separate from the display 450 (eg, touch screen display 112). User interface is shown. The device 300 also optionally for one or more contact strength sensors (eg, one or more of the sensors 359) that detect the strength of contact on the touch sensing surface 451 and / or the user of the device 300. Includes one or more tactile output generators 357 that generate tactile outputs.
Some of the examples below are given with reference to the inputs on the touch screen display 112 (where the touch sensing surface and display are combined), but in some embodiments the device is shown in Figure 4B. Detects inputs on the touch-sensitive surface separate from the display shown. In some embodiments, the touch sensitive surface (eg, 451 in FIG. 4B) has a spindle (eg, 452 in FIG. 4B) that corresponds to a spindle on the display (eg, 450) (eg, 453 in FIG. 4B). Have. According to these embodiments, the device is at a location corresponding to each location on the display (eg, in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470) with the touch sensing surface 451. Detect contacts (eg, 460 and 462 in Figure 4B). In this way, the user input detected by the device on the touch-sensitive surface (eg, 451 in FIG. 4B) (eg, contacts 460 and 462, and their movements) has the touch-sensitive surface separate from the display. When used by the device to manipulate the user interface on the display of the multifunctional device (eg 450 in Figure 4B). It should be understood that similar methods are optionally used for the other user interfaces described herein.
In addition, the following examples are given primarily with reference to finger input (eg, finger contact, finger tap gesture, finger swipe gesture), but in some embodiments, one or more of the finger inputs , It should be understood that it will be replaced by input from another input device (eg mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (eg, instead of a contact) followed by a cursor move along the swipe path (eg, instead of a touch move). As another example, tap gestures are optionally replaced with mouse clicks while the cursor is over the position of the tap gesture (eg, detect a contact and then stop detecting the contact). instead of). Similarly, it is understood that when multiple user inputs are detected simultaneously, multiple computer mice are optionally and simultaneously used, or mouse and finger contact is optionally and simultaneously used. I want to.
FIG. 5A shows an exemplary personal electronic device 500. The device 500 includes a main body 502. In some embodiments, device 500 may include some or all of the features described for device 100 and device 300 (eg, FIGS. 1A-4B). In some embodiments, the device 500 has a touch-sensitive display screen 504, hereafter touch screen 504. As an alternative or addition to the touch screen 504, the device 500 has a display and a touch sensitive surface. Similar to devices 100 and 300, in some embodiments, the touch screen 504 (or touch sensing surface) is optionally one or more to detect the intensity of the contact (eg, touch) being applied. Includes strength sensor. One or more intensity sensors on the touch screen 504 (or touch sensing surface) can provide output data representing the intensity of the touch. The user interface of the device 500 can respond to touches based on the intensity of the touch, which means that touches of different intensities can call different user interface actions on the device 500.
Exemplary techniques for detecting and processing touch intensities are, for example, published as International Patent No. WO / 2013/169849, each incorporated herein by reference in its entirety, filed May 8, 2013. Published as International Patent Application No. PCT / US 2013/040061, "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application", and International Patent No. WO / 2014/105276, November 2013 It can be found in the related application "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships", International Patent Application No. PCT / US 2013/069483 filed on March 11.
In some embodiments, the device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508, if included, can be physical mechanisms. Examples of physical input mechanisms include pushbuttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms attach the device 500 to, for example, hats, eyewear, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, trousers, belts, shoes, purses, backpacks, etc. Can be made possible. These attachment mechanisms allow the user to wear the device 500.
FIG. 5B shows an exemplary personal electronic device 500. In some embodiments, the device 500 may include some or all of the components described with respect to FIGS. 1A, 1B, and 3. The device 500 has a bus 512 that operably couples the I / O section 514 to one or more computer processors 516 and memory 518. The I / O section 514 can be connected to the display 504, which can have a touch-sensitive component 522 and optionally an intensity sensor 524 (eg, a contact intensity sensor). In addition, I / O section 514 is a communication that receives application and operating system data using Wi-Fi, Bluetooth®, Near Field Communication (NFC), cellular, and / or other wireless communication techniques. Can be connected to unit 530. The device 500 can include input mechanisms 506 and / or 508. The input mechanism 506 is optionally, for example, a rotatable input device or a pushable and rotatable input device. In some examples, the input mechanism 508 is optionally a button.
In some examples, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors such as GPS sensor 532, accelerometer 534, direction sensor 540 (eg, compass), gyroscope 536, motion sensor 538, and / or a combination thereof. All of these sensors can be operably connected to I / O section 514.
The memory 518 of the personal electronic device 500 can include one or more non-temporary computer-readable storage media for storing computer executable instructions, the instructions being executed by one or more computer processors 516. Then, for example, a computer processor can be made to perform the techniques described below, including the processes 700, 1000, and 1300 (FIGS. 7, 10, and 13). The personal electronic device 500 is not limited to the components and components of FIG. 5B, and may include other or additional components in a plurality of components.
As used herein, the term "affordance" is optionally displayed on the display screen of devices 100, 300, and / or 500 (FIGS. 1, 3, and 5). Refers to an interactive graphical user interface object. For example, images (eg, icons), buttons, and characters (eg, hyperlinks) each optionally constitute an affordance.
As used herein, the term "focus selector" refers to an input element that indicates the current part of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as a "focus selector" and thus the cursor is of a particular user interface element (eg, a button, window, slider, or other user interface element). When an input (eg, press input) is detected on a touch sensitive surface (eg, touch pad 355 in FIG. 3 or touch sensitive surface 451 in FIG. 4B) while located above, a particular user interface element Adjusted according to the detected input. In some embodiments, including a touch screen display that enables direct interaction with user interface elements on the touch screen display (eg, touch-sensitive display system 112 in Figure 1A or touch screen 112 in Figure 4A), on the touch screen. Detected contact acts as a "focus selector" and is therefore input at the location of a particular user interface element (eg, a button, window, slider, or other user interface element) on the touchscreen display (eg, by contact). When a press input) is detected, a particular user interface element is adjusted according to the detected input. In some implementations, without the corresponding cursor movement or touch movement on the touch screen display (eg, by using the tab key or arrow keys to move the focus from one button to another). Focus is moved from one area of the user interface to another area of the user interface. In these implementations, the focus selector moves according to the movement of focus between various areas of the user interface. Regardless of the particular form the focus selector takes, the focus selector is generally a user interface element (or contact on a touch screen display) and the user. Controlled by the user to communicate the intended interaction of the user with the user interface (eg, by showing the device an element of the user interface that the user intends to interact with). For example, the position of the focus selector (eg cursor, touch, or selection box) above the corresponding button while the press input is being detected on the touch sensitive surface (eg touchpad or touchscreen) is its position. It indicates that the user intends to activate the corresponding button (rather than the other user interface elements shown on the device's display).
As used herein and in the claims, the term "characteristic strength" of a contact refers to the characteristic of the contact based on one or more strengths of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally a predetermined number of intensity samples, i.e., a predetermined event (eg, after detecting a contact, before detecting a lift-off of a contact, before or after detecting the start of movement of a contact, A predetermined time period (eg, 0.05, 0.1) associated with before detecting the end of contact, before or after detecting an increase in contact strength, and / or before or after detecting a decrease in contact strength. , 0.2, 0. Based on a set of intensity samples collected during (5, 1, 2, 5, 10 seconds). The contact characteristic strengths are optionally the maximum value of the contact strength, the mean value of the contact strength, the average value of the contact strength, the top 10 percentile values of the contact strength, and the contact strength. It is based on one or more of the maximum half value of, the maximum 90% value of contact strength, and so on. In some embodiments, the duration of contact is used in identifying the property strength (eg, when the property strength is the average of the contact strengths over time). In some embodiments, the characteristic intensity is compared to one or more sets of intensity thresholds to determine if the operation was performed by the user. For example, a set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, a contact having a characteristic strength that does not exceed the first threshold results in a first action, and a contact that exceeds the first intensity threshold but does not exceed the second intensity threshold is the first. A contact having a characteristic strength that results in a second action and exceeds the second threshold results in a third action. In some embodiments, the comparison between the characteristic intensity and one or more thresholds is used to determine whether the first action should be performed or the second action should be performed. Rather than, it is used to determine whether one or more actions should be performed (eg, whether each action should be performed or whether each action should be stopped).
In some embodiments, a portion of the gesture is identified for the purpose of identifying characteristic intensities. For example, the touch-sensitive surface optionally receives a series of swipe contacts that transition from the start location to reach the end location, at which point the strength of the contact increases. In this example, the characteristic strength of the contact at the end location is optionally based on only a portion of the continuous swipe contact (eg, only the end location portion of the swipe contact) rather than the entire swipe contact. In some embodiments, a smoothing algorithm is optionally applied to the strength of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate a small increase or decrease in the strength of the swipe contact for the purpose of identifying characteristic strength.
The intensity of contact on the touch-sensitive surface is optionally one or more intensities, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. Characterized against a threshold. In some embodiments, the light press strength threshold corresponds to the strength with which the device typically performs the action associated with clicking a button or trackpad on the physical mouse. In some embodiments, the deep press intensity threshold corresponds to the intensity with which the device performs an action that is different from the action normally associated with clicking a button or trackpad on a physical mouse. In some embodiments, the device detects a light press when a contact with a characteristic intensity below the light press intensity threshold (eg, above the nominal contact detection intensity threshold at which no contact is detected) is detected. The focus selector is moved according to the movement of contact on the touch-sensitive surface without performing the action associated with the intensity threshold of or the intensity threshold of deep pressing. In general, unless otherwise stated, these intensity thresholds are consistent between the various sets of user interface diagrams.
An increase in the characteristic intensity of contact from an intensity below the light pressing intensity threshold to an intensity between the light pressing intensity threshold and the deep pressing intensity threshold is sometimes referred to as a "light pressing" input. Increasing the characteristic intensity of contact from an intensity below the deep pressing intensity threshold to an intensity above the deep pressing intensity threshold is sometimes referred to as a "deep pressing" input. An increase in the characteristic intensity of contact, from an intensity below the contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light pressing intensity threshold, is sometimes referred to as contact detection on the touch surface. The decrease in the characteristic strength of the contact from the strength above the contact detection strength threshold to the strength below the contact detection strength threshold is sometimes referred to as the detection of lift-off of contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.
In some embodiments described herein, each press input is detected in response to the detection of a gesture containing each press input, or by each contact (or multiple contacts). Depending on, one or more actions are performed, and each press input is detected at least in part based on the detection of an increase in the intensity of the contact (or multiple contacts) above the press input intensity threshold. .. In some embodiments, each action is performed in response to detecting an increase in the intensity of each contact above the press input intensity threshold (eg, a "downstroke" of each press input). In some embodiments, the press input includes an increase in the intensity of each contact above the press input intensity threshold, followed by a decrease in the intensity of the contact below the press input intensity threshold, and each action follows a subsequent press. It is executed in response to the detection of a decrease in the intensity of each contact below the input threshold (eg, the "upstroke" of each pressing input).
In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, which may be referred to as "jitter", and the device has a hysteresis intensity threshold that has a predetermined relationship with the pressing input intensity threshold. (For example, the hysteresis intensity threshold is an X intensity unit lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold. ). Thus, in some embodiments, the press input includes an increase in the intensity of each contact above the press input intensity threshold, followed by a decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold. Each action is performed in response to detecting a subsequent decrease in the strength of each contact below the hysteresis strength threshold (eg, the "upstroke" of each pressing input). Similarly, in some embodiments, the press input is such that the device increases the intensity of contact from an intensity below the hysteresis intensity threshold to an intensity greater than or equal to the press input intensity threshold, and optionally below the hysteresis intensity thereafter. Detected only when detecting a decrease in contact strength to the strength of the, and each action was to detect a pressing input (eg, increasing contact strength or decreasing contact strength, depending on the situation). Executed in response.
For ease of explanation, the description of the press input associated with the press input intensity threshold, or the action performed in response to the gesture containing the press input, optionally describes the contact intensity above the press input intensity threshold. Increase in contact intensity from below the hysteresis intensity threshold to above the press input intensity threshold, decrease in contact intensity below the press input intensity threshold, and / or the hysteresis intensity threshold corresponding to the press input intensity threshold. Triggered in response to any detection of a decrease in contact strength below. Further, in an example in which the action is described to be performed in response to detecting a decrease in contact strength below the press input intensity threshold, the action optionally corresponds to the press input intensity threshold. It is performed in response to the detection of a decrease in contact intensity below the hysteresis intensity threshold below it.
Next, attention is paid to embodiments of user interfaces (UI) and related processes implemented on electronic devices such as portable multifunction devices 100, devices 300, or devices 500.
6A-6J show exemplary user interfaces for communicating user status information for messages, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 7A-7B.
FIG. 6A shows device 600, which is a specific form factor for device 100, device 300, or device 500 described above in some examples. The device 600 includes a display 601 which is a touch-sensitive display in some embodiments and an image sensor 602. In addition, in some embodiments of the device 600, along with the image sensor 602, there are additional sensors (eg, depth sensors, IR sensors, etc.) and / or other components (eg, flash or IR emitters). ..
In FIG. 6A, display 601 shows messaging interface 603 for another user named "Joe". The messaging interface 603 includes a back button 604, a username 605, a user photo 606, and a contact information button 607. Depending on the user's selection of the back button 604, for example, by touching on the back button 604 on the display 601 the messaging interface 603 can display and / or communicate with other recent messages. Return to the main messaging screen where the contact is selected. Username 605 and user photo 606 show other users that the messaging interface 603 is configured to communicate with. User Photo 606 is, for example, an avatar to another user (eg, an emoji, or other graphic representation such as a non-photorealistic graphic representation), a photo of another user, or any other associated with another contact. It is an image. Depending on the user selecting the contact information button 607, the options, settings, and other actions associated with the other user can be accessed. In some examples, the contact information button 607 causes the device 600 to open a user interface page that allows the user to initiate communication with another user using another communication channel. Allows certain types of information (eg, current location) to be sent to other users, allows users to change communication settings for other users, and / or allows users to previously interact with other users. Allows you to review information about your communications.
The messaging interface 603 includes a message area 608, which is empty in FIG. 6A, and a message input area 609. The message input area includes a photo button 610, a drawing button 611, a sticker / pictogram button 612, a text input field 613, and a voice recognition button 614. The photo button 610 allows the user to either select an existing photo / video for inclusion in a message to other users or capture a new photo / video. The draw button 611 allows the user to create a sketch for inclusion in a message to other users. The sticker / emoji button 612 allows the user to select a sticker and / or emoji for inclusion in a message to another user. The text entry field 613 allows the user to enter the text that will be included in the message to other users. In FIG. 6A, the text input field 613 contains the word "iMessage®" that indicates the type of text message that will be sent to other users. The voice recognition button 614 allows the user to enter text in the text input field 613 by speaking to the device 600 instead of manually entering the text.
As shown in FIG. 6B, a keyboard 616 is displayed on display 601 in response to user input selecting a text input field 613, such as touch 615 in FIG. 6A. In addition, the text input field 613 is moved above the display and magnified by hiding the photo button 610, draw button 611, and sticker / emoji button 612 in FIG. 6A. If the user selects the magnify button 617, these buttons will reappear. In addition, in some examples, a cursor (not shown) appears inside the text input field 613.
As shown in FIG. 6C, depending on the character selection on the keyboard 616 (eg, via the touches on the keyboard 616, such as the letters "H" and "I", touches 618 and 619 in FIG. 6B, respectively). Text input field 613 is updated to include the selected characters. In addition, the voice recognition button 614 (Figure 6B) has been replaced by the send button 620, which sends the currently entered message to other users, including the entered text and other data such as image data. Send.
Figure 6D shows the messaging interface 603 after additional text input via an additional touch on keyboard 616. Text input field 613 contains additional input text.
Once the user has finished typing text in the text entry field 613, the user indicates that he or she is ready to send the message, for example by selecting the send button 620 via touch 621 on display 601. .. Correspondingly, the device 600 prepares a message containing the text in the text input field 613 in the example of FIG. 6D. Once ready to send the message, the device 600 sends the message to another user (in the example of FIGS. 6A-6F, the other user named "Joe"). Either as part of a message or as a separate data transmission, the device 600 also displays the status data of the user of the device 600 (eg, the status data associated with the user account currently associated with the device 600). Send to the user of. In some embodiments, the user's status data represents a physical condition (eg, tired, ill, hot, cold, bored) or an emotional condition (eg, happy, sad, worried, angry).
Status data can be obtained from any number of sources. In one embodiment, the status data is obtained by presenting the user with a list of statuses and allowing the user to select one of the statuses representing the content of the user or the message the user is sending. To. In other words, after the user selects the submit button 620, the device 600 prompts the user with a list of statuses that the user can choose to send in association with a message containing content containing the text in the text input field 613. In another example, the device 600 is configured to automatically determine status data for the user in order to obtain status data. In some examples, the device 600 uses one or more image sensors (eg, image sensor 602 in FIG. 6A) based on the content of the message (eg, via a messaging application) to provide user image data. Determine status data based on and / or other contextual information available to the device 600 (eg, calendar entry, weather, time zone / year, location, etc.). The device 600 uses data from the image sensor 602, or other sensor such as a biometric sensor, to identify facial expressions or to determine user status physiological data (eg, heart rate or). Face recognition can be performed to obtain body temperature). In some examples, the user status is personal to the user and is not associated with an object associated with the user (eg, an electronic device). Once the user status data is determined, the user status data is associated with the message and either with the message (eg, as part of the message) or separately from the message (eg, before sending the message, after sending the message). Or at the same time as sending the message), it will be sent to other users.
After sending the message to another user, as shown in FIG. 6E, the content of the message containing the text in the text input field 613 is displayed in the message area 608 displayed in the text box 622. The text box 622 also optionally contains other non-text content (eg, images, animations, videos, etc.) of the message. In addition to displaying the text box 622, the avatar 623 is also displayed adjacent to the text box 622 (for example, the message comes out of the avatar's mouth). In some embodiments, the avatar 623 is generated based on a previously defined (eg, baseline) avatar or avatar model (eg, wire, mesh, or structural model) that represents the user of device 600. To. In addition, Avatar 623 is based on user status data associated with the message in text box 622. For example, the user status data for the message in text box 622 was sympathetic. The messaging application has determined that a smiling avatar is best suited for this status data. Therefore, Avatar 623 was generated to represent a smile and a warm look. In some examples, the generated avatar can include accessories (eg, umbrellas in the rain, sunglasses when on the beach, costumes for Halloween, or gifts for wishing someone a happy birthday). In some examples, the generated avatar is an animated avatar.
Figure 6F shows messaging interface 603 after a message and associated status data have been received from another user named "Joe" (eg, from an electronic device associated with another user "Joe"). Shown. The content of the message is displayed in the text box 624 of the message area 608. In addition, the associated avatar 625 is displayed adjacent to the text box 624. Like avatar 623, avatar 625 is a previously defined avatar on behalf of the user on device 600, except that the previously defined (eg baseline) avatar or avatar model is for "Joe". Or based on the avatar model. The previously defined avatar or avatar model is optionally set by the user of the device 600 or provided by another user or by a remote server. The device 600 optionally stores the previously defined avatar or avatar model locally or retrieves the previously defined avatar or avatar model as needed. When the device 600 receives a message from another user, such as Joe, the messaging application on the device 600 loads the previously defined avatar or avatar model and modifies or uses it to be associated with the received message. Generate an avatar based on user status data. Similar to the transmission of user status data described above, user status data for a message received from another user may be part of the message or separately from the message (eg, before, after, or with the message). Can be received (at the same time).
User status data for other users is optionally determined in the same manner as described above for the user, except that the other user's device (eg, "Joe's" device) makes the determination. .. For the message associated with text box 624, another user's device determined that the user status was in favor and thankful.
Like Avatar 623, Avatar 625 is based on user status data. In the case of Avatar 625, the messaging application generated Avatar 625 to include smiles and thumbs up, based on user status data in favor and appreciation. In addition, avatar 625 is an animated avatar (arrows represent animated sway of the avatar's thumb).
In some embodiments, the avatar 625 is optionally stored on the device 600 and associated with other users. For example, user photo 606 is replaced with avatar 625 or an avatar subsequently generated for the contact. Whenever other user information is displayed (for example, in another application or in the contact list), the most recently received avatar representing user status data for the other user is displayed.
Figure 6G shows the messaging interface 603 after the user on device 600 has typed in and sent another message to another user named "Joe". The content of the message is displayed in the text box 626 of message area 608. In addition, the avatar 627 is displayed adjacent to the text box 626 and is generated based on the user status data of the user when the message associated with the text box 626 was sent. In this case, device 600 determines that the user status is sad (eg, as programmed by a messaging application). Based on this status, Avatar 627 was generated to have a sad look compared to Avatar 623 (eg, eyes, eyebrows, and mouth were all modified to reflect a sad look). Avatar 627 reflects new user status data compared to Avatar 623, but Avatar 623 remains unchanged. In other words, the avatar 623 remains the same even though the device 600 has received the updated user status data. In such an embodiment, the immutable status of Avatar 623 can serve as a historical record of the user's status, even if the status data is subsequently updated.
In some embodiments, avatars 623 and 627 can be updated based on the new baseline avatar or avatar model. For example, if a user of device 600 shaves his beard and updates the baseline avatar or avatar model accordingly, avatars 623 and 627 are updated to remove that beard, but look and look based on user status data. Other features remain the same (although they can be modified based on the new baseline avatar or avatar model). In some embodiments, previously generated avatars such as avatars 623 and 627 are not updated, even when the baseline avatar or avatar model is updated.
Figure 6G shows messaging interface 603 after a user on device 600 receives another message from another user named "Joe". The content of the message is displayed in the text box 628 of the message area 608. In addition, the avatar 629 is displayed adjacent to the text box 628 and is generated based on the user status data of other users associated with the messages received and represented in the text box 628. Avatar 629 was generated to have a relevant look, based on user status data indicating that other users were apologetic. Avatar 629 reflects new user status data for other users compared to Avatar 625, but Avatar 625 remains unchanged. As shown in Figure 6G, the avatar 625 remains the same even though the device 600 has received the updated user status data. In this way, it is easy to determine the status of a user when a particular message is set. As a result, it is possible to provide a more efficient interface that allows the user to search and understand the message faster.
In some embodiments, avatars 625 and 629 can be updated based on the new baseline avatar or avatar model. For example, if another user starts wearing glasses and updates the baseline avatar or avatar model accordingly, the other user will send out a new baseline avatar or avatar model so that avatars 625 and 629 add glasses. It will be updated, but the facial features and other features based on user status data will remain (although they can be modified based on the new baseline avatar or avatar model). In some embodiments, previously generated avatars, such as avatars 625 and 629, are not updated, even when the baseline avatar or avatar model for other users is updated.
FIG. 6I shows a message application interface 630 including an edit button 631, a title 632, a create button 633, and affordances 634-636. Interface 630 is optionally displayed depending on the selection of the back button 604 in FIG. 6H. Selecting the edit button 631 on interface 630 allows the user to select a specific contact affordance and perform an operation (eg delete) on the contact information associated with the selected affordance. .. Title 632 provides the name of the messaging application. By selecting the compose button 633, the user can compose a new message and select the recipient of that message.
Affordances 634-636 display contact information for different remote users. In this example, each affordance contains contact information such as a username, a summary of the most recent communications with other users, graphic elements associated with other users, and the time of the most recent communications. In some embodiments, the graphic element is the most recently generated avatar for another user. For example, the affordance 634 for "Joe Smith" includes the most recently generated avatar 629 for "Joe Smith" based on the user status received with the message associated with the text box 629 in Figure 6H. .. This avatar is a subsequently generated avatar based on subsequently received user status data so that the contact list in interface 630 always contains the avatar that reflects the most up-to-date status of other users. , Updated arbitrarily.
Selecting one of the affordances returns you to the display of messaging interface 603. For example, selecting contact affordance 634 will return to the display of messaging interface 603, as shown in Figure 6H.
Affordance 635 does not have an associated avatar. Instead, instead of displaying the avatar, Monogram 637 is displayed.
FIG. 6J shows a messaging interface 603 for another user "Joe" after a new baseline avatar or avatar model has been received for "Joe". In this example, Joe has a mustache and updates his avatar accordingly. In response to receiving a new baseline avatar or avatar model, device 600 updated avatars 625 and 629 with avatars 638 and 639 to reflect Joe's new mustache, for example via a messaging program. Avatars 628 and 629 are based on user status data received with the corresponding message. In the example shown in Figure 6J, this means that the facial expressions remain the same.
7A-7B are flow diagrams illustrating methods for an enhanced messaging interface using electronic devices, according to some embodiments. Method 700 is performed on a device having a display (eg, 100, 300 or 500). Some actions of Method 700 are optionally combined, some actions are arbitrarily modified in order, and some actions are optionally omitted.
As described below, Method 700 provides an intuitive way to communicate user status information for a message. This method reduces the user's cognitive burden of communicating status information, thereby creating a more efficient human-machine interface. For battery-powered computing devices, enabling users to understand and respond to messages faster and more efficiently saves power and increases battery charging time intervals.
An electronic device having a display (eg, display 601 of device 600) (eg, device 600 of FIGS. 6A-6J) is associated with a first user (eg, owner or user of device 600). The electronic device is the first message from the second user (eg, another user with username 605 in FIG. 6A and user photo 606) (eg, the message associated with the text box 624 in FIGS. 6F-6J). Receive (eg, SMS messages, MMS messages, iMessage®, or other types of messages) (702). The first message includes the first content (for example, the content in the text box 624 of FIGS. 6F to 6H). According to some embodiments, the second user is associated with the source electronic device that sends the first message and the second message.
The electronic device receives first status data for a second user (eg, emotional or physical condition, or other condition associated with another user or personal to another user). 704). The first status data is associated with the first message and is separate from the first content (for example, the first status data is sent as part of a message separate from the content, or , Sent separately from the message, but in a way that allows the status data to be identified as for the message). Status data can be obtained from any number of sources. In some embodiments, the status data presents the user with a list of statuses, allowing the user to select one of the statuses representing the content of the user or the message the user is sending. To be acquired. In some embodiments, in order to obtain status data, the electronic device uses, for example, one or more image sensors (eg, image sensor 602 in FIG. 6A) based on the user's image data, and / Or it is configured to automatically determine status data for the user based on other contextual information available to the electronic device (eg, calendar entry, weather, time zone / year, location, etc.). Optionally, physiological data (eg, heart rate) for identifying facial expressions or determining user status using data from image sensor 602, or other sensors such as biometric sensors. Or face recognition can be performed to obtain (or body temperature). In some embodiments, the user status is personal to the user and is not associated with an object associated with the user (eg, an electronic device). Once the user status data is determined, the user status data is associated with the message and either with the message (eg, as part of the message) or separately from the message (eg).
The display of the electronic device includes first content (eg, content in a text box 624) and first avatar (eg, avatar 624 in FIGS. 6F-6H) (eg, pictograms or other graphic representations). , Display the first message (706). The first avatar is based on the first status data, and the displayed first avatar is adjacent to the displayed first message (for example, the message comes out of the avatar's mouth).
After displaying the first message and the first avatar, the electronic device receives a second message from the second user (eg, another SMS message, MMS message, iMessage®, or some other type of message. ) Is received (708), where the second message contains the second content (eg, the context in the text box 629 of FIG. 6H).
The electronic device receives a second status data (eg, emotional or physical condition) to the sender (710), where the second status is associated with the second message and the second It is separate from the content of. In some embodiments, the second status data is generated and transmitted in a manner similar to the first status data.
While maintaining the display of the first message and the first avatar, the display of the electronic device displays the second content (eg, text box 629 in FIG. 6H) and the second avatar (eg, avatar 629 in FIG. 6H). Display a second message containing (eg pictograms or other graphic representations) (712), where the displayed second avatar is adjacent to the displayed second message (eg, the message is an avatar). The second avatar is different from the first avatar and the second avatar based on the second status data (which comes out of the mouth). By including different avatars for different messages based on different user status, the usability of the electronic device is enhanced by increasing the amount of information presented on the display. For example, a user can quickly scan a message and determine some context associated with the message based on the avatar, without having to read the textual content of the message. This improved usability of electronic devices makes it more efficient, reduces power consumption and battery life of the device by allowing users to use the device faster and more efficiently. To improve. For example, different avatars for different messages can be useful when the user is trying to find a particular message that the user knows to be associated with a particular status (for example, a sad message may be a sad avatar). Will have). In addition, if a message can be interpreted as having multiple meanings, the message-specific avatar can help the reader correctly interpret the meaning of the message.
According to some embodiments, the electronic device has contact information (eg, FIG. 6I) (eg, FIG. 6I) (eg, FIG. 6I) for a set of users, including contact information for a second user (eg, information in affordance 633 of FIG. 6J). Display phone numbers, email addresses, usernames, recent communications, etc. (720), where the second avatar (eg, avatar 629) is displayed with contact information for the second user. In some embodiments, when a change in the status of the second user is detected, the second avatar is replaced with a third avatar for the second user. By updating the second user's avatar when a new avatar is received, the user of the electronic device also has access to the avatar that represents the second user's most recent status. This provides a more efficient interface by eliminating the need for electronic device users to review old messages to determine the status of a second user during the last communication. In this way, the user needs to interact less with the electronic device in order to retrieve the desired information, so by allowing the user to use the device faster and more efficiently, the electronic device becomes more efficient. To reduce power consumption and improve device battery life.
In some embodiments, the electronic device remembers the first association of the first avatar or first status data with the contact information of the second user (eg, the first avatar or first). (Link the status data in the database with or as part of the contact information), the second avatar or the second status data, the second user's contact information. Remember the association (for example, link the first avatar or first status data with or as part of the contact information in the database).
According to some embodiments, the electronic device receives a first avatar model (eg, wire mesh, parameterized model) for a second user into the first avatar model and the first status data. Based on generating a first avatar (eg 625) (eg an avatar that looks like a second user with a first facial expression such as a smile) and based on the first avatar model and the second status data Generates a second avatar (eg, 629). (For example, an avatar that looks like a second user with a second look, such as a frown). According to some embodiments, the electronic device maps the first status data onto a given avatar model (eg, an existing avatar on the electronic device) to give the first avatar (eg, 625). create. By using the avatar model, the electronic device is more efficient with the second user avatar based on the status data received, as the baseline avatar does not have to be received or generated each time a new avatar is generated. Can be generated in. In this way, the electronic device has more efficient, more power saving and longer battery life by limiting the processing power required to generate the avatar.
According to some embodiments, the electronic device receives a second avatar model (eg, wire mesh, parameterized model) for a second user (722), a second avatar model and a first status. Generate an updated first avatar (eg, 637) based on the data (724). The electronic device generates an updated second avatar (eg, 638) based on the second avatar model and the second status data (726). The electronic device also displays an updated first avatar instead of the first avatar (eg, Figure 6J), along with a first message containing the first content (728) (eg message content). And update the displayed avatar without changing the context information related to the avatar). According to some embodiments, the first avatar (eg, 625) and the second avatar (629) represent the physical appearance of the second user (eg, the avatar represents the appearance of the second user). It reflects). By allowing the baseline avatar to be updated for the second user, the messaging interface ensures that the avatar always reflects the second user's current avatar. This eliminates the need for the user of the electronic device to recognize multiple different baseline avatars associated with the second user, thereby providing a more efficient interface. In this way, the user can interact with the electronic device more quickly and efficiently, which can make the electronic device more efficient, reduce power consumption, and improve the battery life of the device.
According to some embodiments, the first status data is based on the first biometric characteristics of the second user (eg, depth information, face recognition, heart rate, etc.) (716). According to some embodiments, the first status data is based on an optical or depth image of the sender. According to some embodiments, the first status data is when the second user composes and / or sends the first message (eg, a face map (eg, a depiction of various facial features). Based on the detected facial expression of the second user (based on) (714). According to some embodiments, the second status data is when the second user composes and / or sends a second message (eg, a face map (eg, a depiction of various facial features). Based on the detected facial expression of the second user (based on) (718). By using the biometric properties or image data of the second user to generate the first status data, by minimizing the burden on the second user to determine and / or enter the status data. , The messaging interface is extended. In addition, the use of biometric properties can provide a second user with finer and / or more accurate status compared to other methods. As such, electronic devices provide an improved man-machine interface that makes electronic devices more efficient, reduces power consumption, and improves battery life.
According to some embodiments, the electronic device is based on the first status data and has one or more properties for the first avatar (eg, eyes, mouth, and for the sad look of avatar 629 or 638). Select eyebrows (eg facial expressions or accessories) (eg select a given avatar or generate a new avatar). According to some embodiments, the electronic device selects one or more properties for the second avatar based on the second status data (eg, selects an existing avatar or generates a new avatar). The second status data is based on the second biometric characteristics (eg, depth information, face recognition, heart rate, etc.). According to some embodiments, the first status data represents the emotions of the second user (eg, happy, sad, smiling, etc.). By selecting avatar characteristics instead of selecting the entire avatar, electronic devices can more efficiently generate new avatars based on status by simply changing a subset of the characteristics associated with a particular status. In this way, the electronic device has more efficient, more power saving and longer battery life by limiting the processing power required to generate the avatar.
According to some embodiments, the first avatar is an animated avatar (eg, 629 or 638) (eg, a Quicktime® based avatar based on a series of recorded facial expressions, a GIF avatar, etc.). .. The use of animated avatars increases the range and granularity of stats that the stats of the first avatar can represent, thereby providing a more efficient first avatar and messaging interface.
According to some embodiments, the electronic device responds to a third message (eg, message 626 replying to the first message 624 of the second user) from the first user and on the electronic device. The third content (eg, the content in the text box 626 of FIGS. 6G and 6H) is received. The electronic device generates a third status data for the first user and associates the third status data with the third message (eg, send them together or cross-reference the message with the status data). To do). The electronic device sends a third message addressed to the second user, which sends the third status data to the second user. According to some embodiments, the electronic device has a third message containing a third content (eg, content in the text box 626 of FIGS. 6G and 6H) and a third avatar (eg, FIGS. 6G and 6H). Avatar 627) and are displayed on the display at the same time. The third avatar is based on the third status data, and the third message and the third avatar are displayed at the same time as the second message and the second avatar. By including different avatars for different messages based on different user status, the usability of the electronic device is enhanced by increasing the amount of information presented on the display. For example, a user can quickly scan a message and determine some context associated with the message based on the avatar, without having to read the textual content of the message. This improved usability of electronic devices makes it more efficient, reduces power consumption and battery life of the device by allowing users to use the device faster and more efficiently. To improve. For example, different avatars for different messages can be useful when the user is trying to find a particular message that the user knows is associated with a particular status (eg, sad message). Will have a sad avatar in some cases). In addition, if a message can be interpreted as having multiple meanings, a message-specific avatar can help the reader interpret the meaning of the message correctly.
According to some embodiments, displaying the first message and the first avatar (eg, 625) causes the first message to come out of the mouth of the first avatar (eg, 624). ) Including displaying as. Displaying the first message coming out of the mouth of the first avatar clarifies the association between the first avatar and the first message, even with a minimal review of the messaging interface. Make it possible.
According to some embodiments, FIG. 8 shows an exemplary functional block diagram of an electronic device 800 configured according to the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 800 are configured to perform the techniques described above. Functional blocks of device 800 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. It will be appreciated by those skilled in the art that the functional blocks described in FIG. 8 are optionally combined or separated into sub-blocks in order to implement the principles of the various embodiments described. Accordingly, the description herein optionally supports any possible combination or separation of functional blocks described herein, or further definitions.
As shown in FIG. 8, the electronic device 800 includes a display unit 802 and a processing unit 804 coupled to the display unit 802. In some embodiments, the processing unit 804 includes a receiving unit 806, a display enablement unit 808, a generating unit 810, a selection unit 812, a mapping unit 814, an association unit 816, and a transmitting unit 818.
The processing unit 804 receives the first message containing the first content (for example, using the receiving unit 806) from the second user, and is the first status data for the second user. A first message and a first message that receives the first status data (eg, using receive unit 806) and contains the first content, which is associated with the first message and separate from the first content. It is possible to display one avatar and one avatar at the same time on the display (for example, using the display enablement unit 808), where the first avatar is displayed based on the first status data. The first avatar is adjacent to the displayed first message, and after displaying the first message and the first avatar, a second message containing the second content is sent from the second user (for example, the receiving unit). Received (using 806) and received second status data for the second user (eg, using receiving unit 806), where the second status is associated with the second message, Separate from the second content, the second message containing the second content and the second avatar are displayed on the display (eg, display) while maintaining the display of the first message and the first avatar. Displayed (using enablement unit 808), where the displayed second avatar is adjacent to the displayed second message, and the second avatar is based on the second status data, the first The avatar and the second avatar are configured to be different.
In some embodiments, processing unit 804 is further configured to display contact information for a set of users, including contact information for a second user (eg, using display enablement unit 808). , The second avatar is displayed with the contact information of the second user.
In some embodiments, the processing unit 804 receives the first avatar model for the second user (eg, using the receiving unit 806) into the first avatar model and the first status data. Further configured to generate a first avatar based on (eg, using generation unit 810) and a second avatar based on the first avatar model and the second status data. There is.
In some embodiments, the processing unit 804 receives a second avatar model for the second user (eg, using the receiving unit 806) into the second avatar model and the first status data. Based on the updated first avatar (eg using generation unit 810), and based on the second avatar model and the second status data, the updated second avatar (for example) For example, instead of the first avatar, the updated first avatar (eg, display enablement unit 808) is generated with the first message containing the first content, generated (using generation unit 810). It is further configured to display (using).
In some embodiments, the first status data is based on the first biometric characteristics of the second user.
In some embodiments, the processing unit 804 is further configured to select one or more properties for the first avatar (eg, using selection unit 812) based on the first status data. ing.
In some embodiments, the processing unit 804 is further configured to select one or more properties for the second avatar (eg, using the selection unit 812) based on the second status data. , Here, the second status data is based on the second biometric characteristic.
In some embodiments, the first avatar is an animated avatar.
In some embodiments, the first status data is based on an optical or depth image of the second user.
In some embodiments, the processing unit 804 is further configured to map the first status data onto a given avatar model (eg, using the mapping unit 814) to create the first avatar. Has been done.
In some embodiments, the first status data represents the emotions of the second user.
In some embodiments, the first status data is based on the detected facial expressions of the second user when the second user composes and / or sends the first message.
In some embodiments, the second status data is based on the detected facial expression of the second user when the second user composes and / or sends the second message.
In some embodiments, the processing unit 804 receives the third content for the third message (eg, using the receiving unit 806) from the first user and on the electronic device, and the first Generate a third status data for one user (eg using generation unit 810), associate the third status data with a third message (eg using association unit 816), third Message to a second user (eg, using send unit 818) and a third status data to a second user (eg, using send unit 818), and so on. It is configured.
In some embodiments, the processing unit 804 is further configured to simultaneously display a third message containing a third content and a third avatar (eg, using the display enablement unit 808). The third avatar is based on the third status data, and the third message and the third avatar are displayed at the same time as the second message and the second avatar.
In some embodiments, the first avatar and the second avatar represent the physical appearance of the second user.
In some embodiments, the display of the first message and the first avatar comprises displaying the first message as a text bubble coming out of the mouth of the first avatar.
In some embodiments, the second user is associated with a source electronic device that sends the first message and the second message.
9A-9K show exemplary user interfaces for capturing data for building 3D models, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 10A-10B.
FIG. 9A shows device 900, which is a specific form factor of device 100, device 300, or device 500 described above in some examples. The device 900 includes a display 901, which is a touch-sensitive display in some examples, and an image sensor 902. In addition, in some embodiments of device 900, along with the image sensor 902, there are additional sensors (eg, depth sensors, IR sensors, etc.) and / or other components (eg, flash or IR emitters). ..
In FIG. 9A, display 901 displays the scanning interface 903 of the scanning program, which contains instructions on how to proceed, and the image data area 904, which shows the data that has been captured for the 3D model so far. There is. The image data area 904 includes a data addition button 905.
FIG. 9B depicts the device 900 depicted in FIG. 9A from the opposite side. On this side of the device 900 are an image sensor 906, an optical sensor 907, and a flash 908. In some examples, the image sensor 906 is a high quality visible light sensor. Optionally, it also captures other types of light, such as IR. The image sensor 906 is also optionally configured to capture depth information using flight time or other techniques. In some examples, the light sensor 907 can be used to determine the illumination characteristics and the illumination characteristics can be used to suitably configure and operate the image sensor 906. The flash 908 provides a visible light flash for the image sensor 906. In some examples, the flash 908 is also constructed based on data measured from the optical sensor 907. In some embodiments, in addition to the components depicted in FIG. 9B, there are other components (eg, additional image sensors, IR detectors, IR emitters, etc.).
FIG. 9C shows, for example, the scanning interface 903 after the add data button 905 (FIG. 9A) is selected via touch 909 on the display 901. In FIG. 9C, the scanning interface 903 includes an instruction 910 that selects one of four affordances (affordances 911 to 914) to determine how to add data. Affordance 911 allows the user to select image data already stored on the device 900. Affordance 912 allows the user to select image data that already exists on the remote device (eg, search the Internet, access a specific remote server, or select from a specific remote device). become. Affordance 913 allows the user to capture new data using, for example, image sensor 902 or image sensor 906 (FIG. 9C). Affordance 914 returns the user to the status of the interface depicted in Figure 9A.
FIG. 9D shows the scanning interface 903 after the affordance 913 (FIG. 9C) is selected, for example, via the touch 915 on the display 901. In FIG. 9D, the scanning interface 903 provides four affordances (916-919) for selecting the type of image data to capture. Affordance 916 selects a still image. Affordance 917 selects video image data. The affordance 918 selects panoramic image data. The affordance 919 returns the scanning interface to the state shown in Figure 9C.
FIG. 9E shows the scanning interface 903 after the affordance 916 (FIG. 9D) is selected, for example, via touch 920 on display 901. In FIG. 9E, a capture interface is displayed on display 901 for capturing a portion of scene 920 using image sensor 906 (FIG. 9B). The viewfinder 921 represents the current field of view of the image sensor 906 (Fig. 9B). In this example, the home button 922 on device 900 is used to capture the image data in the viewfinder 921. For example, in response to the user pressing the home button 922 (eg, pressing 923), the image data represented by the data displayed in the viewfinder 921 is captured and stored in the device 900. A similar interface is available for the other types of image capture described with respect to Figure 9D. In addition to data representing visible light or other types of light captured by image sensor 906, image data also includes position data (eg GPS data) and orientation data (eg accelerometers, gyroscopes, and / or Positional data such as (data collected from the electronic compass) is optionally included.
FIG. 9F shows the scanning interface 903 after the image data shown in FIG. 9E has been captured. Representation 924 of the captured image data as described with respect to FIG. 9E is now in the image data area 904. In FIG. 9F, the scanning interface 903 has already returned to the photo capture state through the processes described for FIGS. 9A and 9C-9D. Compared to Figure 9E, device 900 is moving relative to the target. Display 901 shows that the viewfinder 921 is aimed at a portion of scene 925 that has the same target but at a different angle compared to scene 920 (Figure 9E). In some examples, the home button 922 is used to capture the image data represented by the viewfinder 921.
FIG. 9G shows the scanning interface 903 after the image data represented by representation 926 has been captured. Based on the image data represented by representations 924 and 926, the device 900 determines, via a scanning program, a suitable algorithm for use in constructing a 3D model of the target in the image data. The scanning program takes various factors into account when determining a suitable algorithm. In one embodiment, the type of image data (eg, still image vs. video vs. panorama) is a factor. In another example, the orientation and / or position of the device 900 when the image data was captured is a factor. Other factors, such as whether the image data includes depth information and the amount of image data, are also considered in various embodiments.
FIG. 9G also includes a 3D model preview area 927 that includes the model preview 928, which can be manipulated to be viewed from different angles and viewpoints. For example, in response to touch and drag 929 on display 901, the model preview 928 is rotated as shown in Figure 9H, which means that the model preview 928 is not complete due to incomplete image data capture. Is shown. In some embodiments, the user selects the add data button 905 to add additional image data to complete the model.
FIG. 9I shows the scanning interface 903 after selection of the data addition button 905 of FIG. 9H. In Figure 9I, device 900 determined that the still image algorithm was the most suitable for building the model, based on the still image data captured so far. Therefore, the scanning interface 903 presents the user with instruction 929 to acquire additional image data of the kind that would be most useful in building the model. In another example, the scanning interface 903 provides instructions and / or suggestions for acquiring video data or panoramic data image data instead. The scanning interface 903 also optionally provides instructions on how to position the device 900 to obtain the most useful image data. The same capture options described for Figure 9C are also displayed.
Depending on the selection of affordance 912 in FIG. 9I, the scanning interface 903 displays the remote selection interface 931 shown in FIG. 9J via touch 930 on display 901. The remote selection interface 931 includes representations 932 to 934 indicating image data available from the remote location. Representations 932 to 934 are determined to represent image data that is potentially related to the model currently being built. For example, device 900 optionally sends the location to a remote server that uses that location to determine other image data captured at the same approximate location. As another example, the device 900 optionally transmits some or all of the image data that has already been captured. The remote server then performs image recognition to determine other image data that may be relevant. As yet another example, a standard Internet search is performed using the search criteria provided by the user. In some embodiments, the remote selection interface 931 is implemented as a web browser interface that displays the results of an internet search. In some examples of selection interface 931, swipe gestures on the interface allow the user to scroll through the additional image data available. Depending on the selection of image data associated with representation 932 in FIG. 9J, device 900 retrieves each image data and uses it to build a model, as shown by representation 935 in FIG. 9K. Add to.
In Figure 9K, the 3D model preview area 927 is updated with a new model, Model Preview 936, based on the available image data. FIG. 9L shows a model preview 936 after the user has rotated the model, eg, via touch on the display 901, as described above for FIGS. 9G and 9H. In some cases, the model preview 936 is constructed using a different algorithm than the device 900 via the scan program previously selected for Figure 9G.
10A-10B are flow diagrams illustrating a method for constructing a 3D model of an object using an electronic device, according to some embodiments. Method 1000 is performed on a device having a display and one or more image sensors (eg, 100, 300, 500). Some actions of Method 1000 are optionally combined, some actions are arbitrarily modified, and some actions are optionally omitted.
Method 1000 provides an intuitive way to build a 3D model of an object, as described below. This method reduces the cognitive burden on the user building a 3D model of the object, thereby creating a more efficient human-machine interface. For battery-powered computing devices, allowing users to configure 3D models of objects faster and more efficiently saves power and increases the battery charge time interval.
An electronic device (eg, 900) having one or more image sensors (eg, 902, 906), memory, and a display (eg, 901) is one or more image sensors (eg, 902, 906) of the electronic device. First image data (eg, 921, 904) from (eg, depth sensor, optical sensor, IR sensor) (eg, still image or video with optical data, depth data, and optionally position data). Capture (1002). The first image data includes the first optical image data of the object (eg, 920) from the first viewpoint (eg, FIG. 9E).
The electronic device also captures a second image data (eg, 921,926) from one or more image sensors (eg, 921,904) (eg, depth sensor, optical sensor) of the electronic device (1004). Here, the second image data includes the second optical image light data of the object from the second viewpoint (for example, FIG. 9F) different from the first viewpoint.
The electronic device selects an algorithm (eg, an algorithm that stitches different captures, an algorithm that uses panning data) based on the change in perspective from the first perspective to the second perspective (1006), and uses this algorithm. Based on this, determine the additional image data (eg, Figure 9I) needed to continue 3D modeling of the object.
Finally, the electronic device displays on the display visual feedback (929) that provides instructions to capture additional image data determined based on the selected algorithm (1010). By providing instructions to the user about the additional data needed to model the object, the electronic device has been extended to reduce the cognitive burden on the user and any additional image data (eg, for example). , The type of image data and the position of the image data) is determined. This increases the likelihood that the user will succeed in creating an accurate model.
According to some embodiments, the electronic device receives a third piece of data (eg, from an image sensor or remote server) (eg, 935) (1018). The third data includes a third optical image data of the object from the third viewpoint. The electronic device selects an updated algorithm (eg, an algorithm that stitches different captures, an algorithm that uses panning data) based on a third perspective (1020). The updated algorithm is different from the algorithm. Based on the updated algorithm, the electronic device determines the updated additional image data that is different from the additional image data required to continue 3D modeling of the object (1022). The electronic device displays on the display visual feedback that provides updated instructions for capturing the updated additional image data (1024), where the updated instructions provide the updated algorithm. Different from the instructions displayed before the selection. By choosing an updated algorithm based on the third data, the ability of the electronic device to build the model is to adjust how the model is built based on the available image data. Improve by Thus, building a model of an electronic device based on an algorithm based on captured image data is more efficient, more power saving, and improves battery life.
According to some embodiments, the electronic device uses a selected updated algorithm to provide a first image data (eg, 924), a second image data (eg, 926), and a third. Build a 3D model of the object (eg 936) based on the image data (eg 935).
According to some embodiments, the electronic device sends at least a portion of the first image data (eg, 924) (eg, location data) to the remote server and the third data (eg, 932). Receive an indicator from the remote server that the object (eg, Figure 9J) is available. In some embodiments, the third data is from data available from other users. By using data from other users, electronic devices can build models more efficiently without having to recapture data that is already available. As such, electronic devices are more efficient, more power saving, and have longer battery life. In addition, electronic devices reduce the burden on the user by eliminating the need for the user to capture additional data that is no longer available or may be difficult for the user to capture.
According to some embodiments, displaying on the display visual feedback that provides updated instructions for capturing the updated additional image data is an electronic device, the first algorithm of which is selected. According to the determination that the first instruction set (eg, 929) is displayed (1014), and according to the determination that a second algorithm different from the first algorithm is selected, the visual feedback is the first. Includes a second instruction set that is different from the instruction set (1016), including (eg, telling the user the camera, rather than rotating the camera around a fixed point or taking a still image of the object. Encourage them to slide horizontally).
According to some embodiments, the first image data includes the first depth image data of the object from the first viewpoint (1012).
According to some embodiments, the electronic device provides first position data (eg GPS, tilt, orientation data) with respect to a first viewpoint (eg, the viewpoint of FIG. 9E) (eg, orientation or relative position). Obtained (eg from sensor or image data).
According to some embodiments, the choice of algorithm is also based on the first position data.
According to some embodiments, the electronic device provides second position data (eg GPS, tilt, orientation data) with respect to a second viewpoint (eg, the viewpoint of FIG. 9F) (eg, orientation or relative position). Captured, where the second image data includes the second depth image data of the object from the second viewpoint, the algorithm selection is also based on the second position data.
According to some embodiments, the electronic device uses the selected algorithm to provide first image data (eg, 924), second image data (eg, 926), and additional image data (eg, 926). For example, based on 935), a 3D model of the object (eg, 936) is constructed and the 3D model is stored in memory. In some embodiments, a graphic representation of the 3D model (eg, 936) is displayed to the user (eg, while the user is scanning an object, the partially generated 3D model is displayed to the user. , Help the user complete the scan of the object, and / or optionally respond to detected requests to display the model, such as activation of the displayed affordance to display the 3D model. The completed 3D model is displayed to the user after the user has completed scanning the object).
According to some embodiments, choosing an algorithm indicates that the first and second image data are from a scan of an object, from a first perspective to a second perspective. It involves selecting a scan-based algorithm based on the change.
According to some embodiments, choosing an algorithm indicates that the first and second viewpoints are for discrete images (eg, the viewpoints shown in FIGS. 9E and 9F). It involves selecting a discrete image-based algorithm based on the change from the viewpoint to the second viewpoint.
According to some embodiments, the electronic device identifies the support (eg, hand or table) in the first image data touching the object and uses the selected algorithm to make the first Build a 3D model of the object based on the image data and the second image data. The 3D model does not include support for touching the first object (for example, the 3D model is at least by excluding detection points that are determined to be part of the support rather than part of the first object. Partially generated). By not including support touching the first object in the model of the object, the electronic device can generate a more useful model for the user by not including extra details that are not relevant to the user. Is extended as In this way, the user does not have to manually remove the support later, which makes the electronic device more efficient, more power saving and has a longer battery life.
According to some embodiments, the electronic device displays a first window containing a live image of the object (eg, 921) on the display of the electronic device and displays the object (eg, complete or partial). A second window containing an image of the model (eg, 927) is displayed on the display, where the model is based on the first image data and the second image data. In some embodiments, the image of the object's model is updated as additional images are captured to improve the object's model. This feedback may improve the quality of the scan, at least in some cases, by providing the user with real-time feedback on the progress of the scan and the type of device behavior, allowing the user to scan the object more accurately. It can be a more accurate model.
According to some embodiments, FIG. 11 shows an exemplary functional block diagram of an electronic device 1100 configured according to the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 1100 are configured to perform the techniques described above. The functional blocks of device 1100 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. Those skilled in the art will appreciate that the functional blocks described in FIG. 11 are optionally combined or separated into sub-blocks in order to implement the principles of the various embodiments described. Accordingly, the description herein optionally supports any possible combination or separation of functional blocks described herein, or further definitions.
As shown in FIG. 11, the electronic device 1100 includes a display unit 1102 and a processing unit 1104 coupled to the display unit 1102. In some embodiments, the processing unit 1104 is a capture unit 1106, a selective activation unit 1108, a determination unit 1110, a display enablement unit 1112, a receive unit 1114, a build unit 1116, a transmit unit 1118, an acquisition unit 1022, a storage unit. Includes 1124 and identification unit 1126.
The processing unit 1104 receives the first image data from one or more image sensors of the electronic device, including the first optical image data of the object from the first viewpoint, which is the first image data. A second image of an object from a second viewpoint that is different from the first viewpoint and is the second image data captured from one or more image sensors of the electronic device (for example, using the capture unit 1006). Optical image of the second image data, including the optical data, is captured (eg, using capture unit 1106) and the algorithm is based on the change in viewpoint from the first viewpoint to the second viewpoint. Select (using, for example, selection unit 1108) and, based on this algorithm, determine and select additional image data (eg, using determination unit 1110) needed to continue 3D modeling of the object. It is configured to display on the display (eg, using the display enablement unit 1112) visual feedback that provides instructions to capture additional image data determined based on the algorithm.
In some embodiments, the processing unit 1104 receives a third piece of data (eg, using the receiving unit 1114), including a third optical image data of the object from a third point of view. Needed to select an updated algorithm different from the one above (eg, using selection unit 1108) based on 3 perspectives and continue 3D modeling of the object based on the updated algorithm. Provides updated instructions to determine the updated additional image data that is different from the additional image data (eg, using decision unit 1110) and to capture the updated additional image data. Display the visual feedback on the display (eg, using the display enablement unit 1112), where the updated instructions are different from the instructions displayed before selecting the updated algorithm. It is further configured as follows.
In some embodiments, the processing unit 1104 uses a selected updated algorithm to provide first image data, second image data, third image data, and updated additional image data. Based on, it is further configured to build a 3D model of the object (eg, using build unit 1116).
In some embodiments, processing unit 1104 sends at least a portion of the first image data to a remote server (eg, using transmission unit 1118) and a third data is available for the object. Is further configured to receive the indicator from a remote server (eg, using receive unit 1114).
In some embodiments, the display on the display of the visual feedback that provides the updated instructions for capturing the updated additional image data is according to the determination that the first algorithm is selected. The visual feedback includes a second instruction set that is different from the first instruction set, according to displaying the first instruction set and determining that a second algorithm that is different from the first algorithm is selected. Including that.
In some embodiments, the first image data includes a first depth image data of the object from a first viewpoint.
In some embodiments, the processing unit 1104 is further configured to acquire first position data with respect to a first viewpoint (eg, using acquisition unit 1122).
In some embodiments, the choice of algorithm is also based on the first position data.
In some embodiments, the processing unit 1104 is further configured to capture the second position data with respect to the second viewpoint (eg, using the capture unit 1106), where the second image data is The selection of the algorithm, which includes the second depth image data of the object from the second viewpoint, is also based on the second position data.
In some embodiments, the processing unit 1104 uses the selected algorithm to create a 3D model of the object (eg, based on the first image data, the second image data, and additional image data). It is further configured to build (using build unit 1116) and store its 3D model in memory (eg, using storage unit 1124).
In some embodiments, choosing an algorithm indicates that the first and second image data are from a scan of an object, a change from a first perspective to a second perspective. Includes selecting a scan-based algorithm based on.
In some embodiments, the choice of algorithm is based on a change from a first perspective to a second perspective, indicating that the first and second perspectives are for discrete images. Includes selecting discrete image-based algorithms.
In some embodiments, processing unit 1104 identifies the support in the first image data touching the object (eg, using identification unit 1126) and uses the algorithm of choice. It is further configured to build a 3D model of the object (eg, using build unit 1116) based on the first image data and the second image data, and the 3D model is on the first object. Does not include touching support.
In some embodiments, the processing unit 1104 displays a first window containing a live image of the object on the display of the electronic device (eg, using the display enablement unit 1112) of the object's model. A second window containing the image is further configured to be displayed on the display (eg, using the display enablement unit 1112), and the model is based on the first image data and the second image data. ing.
12A-12J show exemplary user interfaces for restricting access to data and applications based on the user, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 13A-13B.
FIG. 12A shows a device 1200 with a display 1201, an image sensor 1202, and a home button 1204. In some embodiments, the display 1201 is a touch-sensitive display. In some embodiments, the home button 1204 is a mechanical button separate from the display 1201 and includes a fingerprint sensor for identifying the user who rests his finger on the home button 1204. In some embodiments, the home button 1204 is integrated with the display 1201 and further includes a fingerprint sensor. In FIG. 12A, device 1200 displays lock screen interface 1203 on display 1201. The lock screen interface 1203 is displayed when the device 1200 is locked with limited functionality. For example, access to most applications and other features on the device 1200 is not allowed in the locked state. However, in some examples, some applications (eg, camera applications) or features (eg, calling to emergency numbers) are available. These applications and features allowed from the locked state are optionally accessed via a button (not shown) appearing on the lock screen interface 1203 or another gesture (eg, swipe) on the lock screen interface 1203. Will be done. In some cases, the device 1200 is unlocked via authentication of fingerprint data measured by a fingerprint sensor located within the home button 1204. In other cases, the device 1200 is unlocked via authentication of the pin code entered via the pin code interface (not shown). In some embodiments, the image sensor 1202 is configured to use device 1200 to capture user image data. The image data is then processed on the device 1200 and compared with the image data or other data for a legitimate user. When the image data allows the user to authenticate
FIG. 12B shows device 1200 displaying the home screen interface 1205 after the device 1200 has been unlocked, for example, using one of the techniques described with respect to FIG. 12A. Home screen interface 1205 includes multiple icons for applications installed on device 1200, including a photo application associated with icon 1206. For example, depending on selecting icon 1206 via touch 1207 on display 1201 at position 1206, the photo application is launched, or if the photo application is already launched, the photo application is displayed. To.
FIG. 12C shows a device 1200 displaying a photo application, more specifically a photo album interface 1208, including an add button 1209, a title 1210, a search button 1211, and an edit button 1212. Select the Add button 1209 to display the Add Album interface, which allows you to create additional photo albums. Title 1210 identifies that interface as an album interface. Select the search button 1210 to display a search interface that allows you to search for photos in your photo application. Selecting the edit button 1212 enables an album editing interface that allows you to delete albums or perform other functions.
Photo Album Interface 1208 also includes thumbnails 1213-1216, respectively, representing each photo album in the photo application. In some examples, each thumbnail is a smaller version of the photo present in that album. Thumbnails 1213-1216 are also selectable, for example, via touch on display 1201.
Figure 12D shows the photographic application after selecting thumbnail 1213 (Figure 12C). A photo selection interface 1217 is displayed, including a back button 1218, a title 1219, and a selection button 1220. Select the back button 1218 to return the photo application to Album Interface 1208 (Figure 12C). Title 1219 identifies the current album displayed on the photo selection interface 1217. Selecting the select button 1220 allows the user to select multiple photos in the current album and perform operations (eg, delete) on the selected photos.
The photo album interface 1217 also includes thumbnails 1221 to 1232 representing each photo in the album. In some examples, each thumbnail is a smaller version of the photo present in that album. Thumbnails 1221 to 1232 are also selectable, for example, via touch on display 1201.
FIG. 12E shows a photographic application after selecting thumbnail 1227 (FIG. 12D). Photo interface 1234 is displayed, including back button 1235, photo information 1236, and detail button 1237. Select the back button 1235 to return the photo application to the photo selection interface 1217 (Figure 12D). Photograph Information 1235 provides information such as location and time for the currently displayed photo (1238). Selecting the details button 1237 displays a detailed interface that provides additional information about the currently displayed photo (1238), such as the associated photo or the person identified within the photo.
Photo interface 1234 also includes a display of the currently selected photo, photo 1238. The interface also includes a preview area that includes thumbnail 1239 for photo 1238, and other reduced size thumbnails 1240 that represent some of the other photos in the album. In some embodiments, for all photos in an album, if there is not enough room to display thumbnails in the preview area, the user, for example, on the preview area, swipe gestures on display 1201. You can scroll between thumbnails via.
While viewing the photo interface 1234, the user of device 1200 requests the interface to move away from photo 1238 and navigate to the next photo in the album (for example, the photo represented by thumbnail 1228 in Figure 12D). can do. In some examples, the navigation request away from the current photo is a gesture on display 1201 received on the currently selected photo (eg, photo 1238 in the case of Figure 12E). In one embodiment, swipe gesture 1241 is received. In response to receiving the swipe gesture 1241, the device 1200 uses the image sensor 1202 to capture image data. In another example, the image data is captured before receiving a navigation request away from the current photo.
Once the image data is captured and a navigation request is received to move away from the currently selected photo, the device 1200 determines if the set of content lock criteria (eg, via an authentication program) is met. .. In some examples, the set of content lock criteria includes criteria that are met if it is detected that a rogue user is using the device 1200, based on analysis of the captured image. Other examples include the absence of legitimate users (eg, by analyzing captured image data), device orientation or movement (eg, device handoff or detection of device lying flat), legitimate. Time delays since the user was last detected, and / or detection of information associated with the displayed content (eg, sensitive or personal content may have a stricter set of criteria), etc. , Other criteria are also used.
Navigation away from the currently selected content in response to a determination that the set of content lock criteria is not met (for example, based on captured image data, a rogue user is not detected as using the device) Is allowed to proceed. This is depicted in Figures 12F and 12G.
Specifically, in FIG. 12F, the photo interface 1234 transitions from photo 1238 to the next photo in the album, photo 1242, according to the swipe gesture 1241 (FIG. 12E). In addition, thumbnail 1239 associated with photo 1238 shrinks as thumbnail 1243 associated with photo 1242 grows.
Figure 12G shows the completed navigation from the previously selected content (Photo 1238 in Figure 12E) to the newly selected content (Photo 1242). The photo information in Photo Interface 1234 has been updated with Photo Information 1244 associated with Photo 1242.
On the other hand, the currently selected content depends on the determination that the set of content lock criteria is met (eg, based on the captured image data, a rogue user is detected as using the device). Navigation away from is denied. This is depicted in Figure 12H.
In FIG. 12H, device 1200 determines that the set of content lock criteria is met. For example, in response to a navigation request away from the currently selected content (Photo 1238) via the swipe gesture 1241 in Figure 12E, the device 1200 sets the next in the album, as described for Figure 12F and Figure 12G. No photo shown. In addition, the thumbnailed photo preview area 1239 is not updated on demand. In some embodiments, the photo preview area is completely hidden. In response to the swipe gesture stop, Photo 1238 returns to the center of display 1201 as shown in Figure 12E. Once the content lock criteria set is no longer met, the ability to navigate away from Photo 1238 is re-enabled.
In addition to preventing navigation away from the currently displayed or selected content, in some embodiments, other potentially different to limit or modify other features of the device 1200. A set of lock criteria is also used. For example, a set of lock criteria is associated with a device's capabilities, such as the ability to terminate an application, switch to another open application, or launch a new application. When this set of lock criteria is met, the device association feature is disabled. In another example, the set of lock criteria is associated with the configuration of the application on the device, such as a read-only configuration. When this set of lock criteria is met, the application's configuration switches (for example, the application's functionality is restricted or the application switches from a read / write configuration to a read-only configuration). In yet another example, a set of lock criteria is associated with locking all other features of the device other than the current feature (for example, if the device is viewing a photo, this is probably Other than locking the device, it is the only feature allowed on that device).
12I and 12J show another application of a set of locking criteria. In Figure 12I, the communication is received in the form of an iMessage message. Device 1200 has determined that the set of lock criteria is not met. In this way, the device 1200 indicates that the communication has been received and displays a notification 1245 to preview the content of the message and the sender. In contrast, in Figure 12J, device 1200 determined that the same set of locking criteria was met. This set of locking criteria is associated with whether to display notifications about communications, or what type of notifications are displayed. In FIG. 12J, notification 1246 merely indicates that the message or communication has been received, without indicating the content of the message or who sent the message. In some embodiments, the display of notifications is completely suppressed as the locking criteria are met.
When the set of associated locking criteria is no longer met, the limited functionality and features of the device 1200 and its applications are optionally restored. In some embodiments, when the device is unlocked using, for example, a pin code or fingerprint identification, limited functionality and features are also restored.
13A-13B are flow diagrams illustrating methods for restricting access to data and applications based on the user using the electronic device, according to some embodiments. Method 1300 is performed on a device having a display and an image sensor (eg, 100, 300, 500). Some actions of Method 1300 are optionally combined, some actions are arbitrarily modified, and some actions are optionally omitted.
As described below, Method 1300 provides an intuitive way to limit access to data and applications based on the user using the device. This method reduces the user's cognitive burden of restricting access to data and applications based on the user using the device, thereby creating a more efficient human-machine interface. For battery-powered computing devices, it saves power by allowing users to limit access to data and applications faster and more efficiently based on the user using the device. Increase the battery charge time interval.
An electronic device (eg 1200) with a display (eg 1201) and one or more image sensors (eg 1202) has content (eg 1238) in the application (eg photo, photo sharing application, messaging application). (For example, a photo) is displayed on the display. The electronic device displays the content while the application is in the first configuration (eg, the normal configuration) (1302). While displaying the content, the electronic device captures image data (eg, image and / or depth information) from one or more image sensors (eg, 1202) of the electronic device (1304). After capturing the image data, the electronic device receives a navigation request (eg, 1241) away from the content (1306). In response to receiving a navigation request away from the content (eg, swiping to a different photo or navigating to a different web page), the electronic device follows the determination that the first set of content locking criteria is met. , Preventing navigation away from the content (eg, Figure 12F) (eg, swiping to a different photo, or navigating to a different web page) while maintaining the display of the content (eg, 1238) (1310). Content Lock Criteria Set includes the first criterion that is met when the captured image data indicates that an unauthorized user is using the device, and the first Content Lock Criteria Set is not met. According to the judgment, navigation away from the content is performed according to the request (for example, Fig. 12F to Fig. 12G) (1312). Electronic devices by preventing access to other content while still allowing legitimate users to view the content they are trying to access by preventing navigation away from the content when certain criteria are met. Increased security (for example, when a legitimate user is not using the device
According to some embodiments, an electronic device that navigates away from the content translates the currently displayed content (eg, a map or list in response to scrolling inputs such as drag gestures on a touch-sensitive surface. To scroll). According to some embodiments, the electronic device that navigates away from the content includes switching between content items (eg, 1238 and 1242) in the application (eg, FIG. 12F) (eg, on a touch-sensitive surface). Switch through a page in a book or pdf, a photo in the camera roll, or a web page in the browser, based on the following content item input, such as a swipe gesture).
According to some embodiments, the electronic device navigates away from the content (eg, FIGS. 12F-12G) (eg, swipes to a different photo) according to the determination that the first set of content locking criteria is no longer met. , Or to a different web page) (1314).
According to some embodiments, the first set of locking criteria is indicated by captured image data that the legitimate user of the electronic device is not using the device (eg, the legitimate user's face is 1). Includes a second criterion that is met when (not detected in the field of view of the camera beyond a given amount of time, such as seconds, 15 seconds, or 1 minute). According to some embodiments, the first set of locking criteria includes a third criterion that is satisfied when the captured image data indicates that there are rogue users and no legitimate users. (For example, the face of a user who is not recognized as a legitimate user is detected in the field of view of the camera without the face of the legitimate user being detected in the field of view of the camera). In some embodiments, the first set of locking criteria is met when any of the included criteria are met. In some embodiments, the first set of locking criteria is met only when all the criteria included are met. By including multiple lock criteria in the first set of lock criteria, the electronic device balances the user's ability to show content to others and the prevention of access to content that users do not want others to see. Extend electronic devices by providing flexible security levels that enable them.
According to some embodiments, the first set of locking criteria is met when the captured image data indicates the presence of a rogue user with or without a legitimate user (eg,). If the first criterion is met, then the first set of locking criteria is met). According to some embodiments, the electronic device determines whether the captured image data indicates the presence of an unauthorized user of the electronic device.
According to some embodiments, the electronic device disables at least one function (FIGS. 12I and 12J) of the electronic device according to the determination that the second set of content locking criteria is met (1316). (For example, suppress the display of notifications). According to some embodiments, the electronic device continues to display content within the application according to the determination that a fourth set of content locking criteria (eg, the same as or different from other sets) is met. , Lock other functions of the electronic device (eg, all other functions except the sleep / wake button function) (1320). According to some embodiments, navigation between applications on the electronic device is prevented according to the fourth criterion being met, and navigation within the application according to the fifth criterion being met ( For example, the transition from the photo view of FIG. 12E to the album view of FIG. 12D) is prevented. For example, if the fourth criterion (and therefore the set of fifth lock criteria) is met, the device will not return to the home screen when the user presses the home button. Instead, either the application display remains, or the device is locked and the lock interface (eg, 1203) is displayed. For example, the fourth set of content locking criteria is met when the image data indicates that no legitimate user is using the electronic device. Switching between applications and / or viewing notifications is disabled as the fourth set of content locking criteria is met. Restricting access to certain features when the user is allowed others to view content on the electronic device by disabling or locking one or more features of the electronic device. Increases the security of electronic devices. For example, if a user provides an electronic device to another person to view the displayed image and the electronic device receives a call, the electronic device is the user (eg,).
According to some embodiments, the first set of lock criteria and the second set of lock criteria are different (for example, the satisfaction of the first set of lock criteria means that the image data is stored by a legitimate user. Indicates that the rogue user is using the electronic device, whether or not it exists, and the image data indicates that the legitimate user is not using the electronic device, whether or not the rogue user exists. If the second set of locking criteria is met).
According to some embodiments, the electronic device compares the application to the first configuration (eg, FIG. 12F) according to the determination that the third set of content locking criteria is met. Switch to a second configuration (eg, Figure 12G) that restricts (eg, Figure 12G) (1318) (eg, if there are rogue users and legitimate users, the navigation is locked, and if there are rogue users without legitimate users, the entire device Locked).
According to some embodiments, the electronic device is notified of the communication received on the electronic device according to the determination that the fifth set of content locking criteria (eg, the same as or different from the other set) is met. (For example, to detect the notification trigger condition and to suppress the presentation of the notification according to the determination that the content lock criterion is met / to present the notification according to the determination that the content lock criterion is not met. Prevent things (eg, via audible, tactile, or visual output) (1322).
According to some embodiments, the fifth set of locking criteria includes a fourth criterion that is met when the captured image data indicates that an unauthorized user is using an electronic device, the fifth. The set of lock criteria is met when the fourth criterion is met, and the first set of lock criteria contains a fifth criterion that is met when the captured image data indicates the absence of a legitimate user. .. For example, the fourth criterion in the fifth set of lock criteria is met when the image data captured from the electronic device indicates that the captured image data has a non-legitimate user (eg, an image). The data indicates that a rogue user (with or without a legitimate user) is in the field of view of the camera in front of the electronic device). In addition, as long as the fourth criterion is met, the fifth set of lock criteria will be met (eg, rogue users will be captured, regardless of whether the other criteria in the fifth set of lock criteria are met. No other criteria need to be met to meet the fifth set of locking criteria as long as it is present in the image data).
According to some embodiments, the image data includes optical data and depth data, and determining whether the first set of content locking criteria is met is based on the optical data and depth data. .. In some embodiments, the optical data is provided by a first camera with a first resolution and the depth data is provided by a second camera with a second resolution lower than the first resolution. In some embodiments, depth data is generated by a combination of images from two cameras that are located at a known distance apart. In some embodiments, the depth and optical data are generated by the same camera.
According to some embodiments, an electronic device that navigates away from the content switches applications to display a home screen (eg, 1205), or an application (eg, the photo application of FIGS. 12C-12J). Close (for example, switching between applications in multitasking mode triggered by repeated activation of a button such as the home button, or moving out to an application launch user interface triggered by activation of a button such as the home button). including.
According to some embodiments, the electronic device receives and locks unlock information (eg, pin code, password, or biometric information such as fingerprint or iris information) associated with a legitimate user of the electronic device. Navigation (eg, Figure 12F) (eg, Figure 12F) that determines if the unlock information is authentic and leaves the content (eg, 1238) (eg, returns the electronic device to normal operation) according to the determination that the unlock information is authentic. Allows you to swipe to a different photo or go to a different web page).
According to some embodiments, FIG. 14 shows an exemplary functional block diagram of an electronic device 1400 configured according to the principles of the various embodiments described. According to some embodiments, the functional blocks of the electronic device 1400 are configured to perform the techniques described above. Functional blocks of device 1400 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. It will be appreciated by those skilled in the art that the functional blocks described in FIG. 14 are optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Accordingly, the description herein optionally supports any possible combination or separation of functional blocks described herein, or further definitions.
As shown in FIG. 14, the electronic device 1400 includes a display unit 1402 and a processing unit 1404 coupled to the display unit 1402. In some embodiments, the processing unit 1404 is a display enablement unit 1406, a capture unit 1408, a receive unit 1410, a prevention unit 1412, a navigation unit 1414, a permit unit 1416, a disable unit 1418, a lock unit 1420, a prevention unit 1422 , Judgment unit 1424, switching unit 1126, and enablement unit 1128.
The processing unit configured in 1404 displays the content in the application (eg, using the display enablement unit 1406) on the display, where the content is displayed while the application is in the first configuration. While displaying the content, capture image data from one or more image sensors on the electronic device (eg, using capture unit 1408), capture the image data, and then make a navigation request away from the content (eg,). Content while maintaining the display of the content according to the determination that the first set of content locking criteria is met in response to receiving (using receive unit 1410) and receiving a navigation request away from the content. Prevents navigation away from (eg, using prevention unit 1412), and the first set of content lock criteria is met when the captured image data indicates that a rogue user is using the device. It is configured to include the first criterion and navigate away from the content as requested (eg, using navigation unit 1414) according to the determination that the first set of content locking criteria is not met. ..
In some embodiments, processing unit 1404 further allows navigation away from the content (eg, using authorization unit 1416) according to the determination that the first set of content locking criteria is no longer met. It is configured.
In some embodiments, the first set of locking criteria includes a second criterion that is met when the captured image data indicates that the legitimate user of the electronic device is not using the device.
In some embodiments, the first set of locking criteria includes a third criterion that is met when the captured image data indicates that there are rogue users and no legitimate users.
In some embodiments, the first set of locking criteria is satisfied when the captured image data indicates the presence of a rogue user with or without a legitimate user.
In some embodiments, processing unit 1404 disables at least one function of the electronic device (eg, using invalidation unit 1418) according to the determination that the second set of content locking criteria has been met. It is further configured to do so.
In some embodiments, the first set of lock criteria and the second set of lock criteria are different.
In some embodiments, the processing unit 1404 has a second configuration that limits the operation of the application compared to the first configuration, according to the determination that the third set of content locking criteria is met. It is further configured to switch to (eg, using switching unit 1126).
In some embodiments, the processing unit 1404 locks other features of the electronic device (eg, locks) while continuing to display content within the application, according to the determination that a fourth set of content locking criteria is met. Further configured to lock (using unit 1420).
In some embodiments, the processing unit 1404 uses a prevention unit 1412 to display notifications related to communications received on the electronic device according to the determination that the fifth set of content locking criteria has been met. It is further configured to prevent it.
In some embodiments, the fifth set of lock criteria includes a fourth lock that is met when the captured image data indicates that an unauthorized user is using an electronic device, and the fifth lock. The set of criteria is met if the fourth criterion is met, and the first set of locking criteria includes a fifth criterion that is met when the captured image data indicates the absence of a legitimate user.
In some embodiments, processing unit 1404 prevents navigation between applications on electronic devices (eg, using prevention unit 1412) according to the determination that the fourth criterion is met, and the fifth. It is further configured to prevent navigation within the application (eg, using prevention unit 1412) according to the determination that the criteria are met.
In some embodiments, processing unit 1404 is further configured to determine (eg, using determination unit 1124) whether the captured image data indicates the presence of an unauthorized user on the electronic device. ..
In some embodiments, the image data includes optical and depth data, and determining whether the first set of content locking criteria is met is based on the optical and depth data.
In some embodiments, navigation away from the content involves translating the currently displayed content.
In some embodiments, navigation away from the content involves switching between content items within the application.
In some embodiments, navigation away from the content involves switching applications or closing the application to display the home screen.
In some embodiments, processing unit 1404 receives unlock information associated with a legitimate user of the electronic device (eg, using receiver unit 1410) to determine if the unlock information is authentic (eg, using receive unit 1410). Further configured to make a decision (using, for example, decision unit 1124) and enable navigation away from the content (eg, using enabler unit 1128) according to the decision that the unlock information is authentic. ing.
The following items briefly describe the various aspects of the invention that have been described in detail above. A method of receiving a first message containing the first content from a second user on an electronic device having a display and associated with the first user, and to the second user. Receiving the first status data, which is the first status data and is associated with the first message and is separate from the first content, and the first message and the first message containing the first content. The first avatar is to be displayed on the display at the same time, the first avatar is based on the first status data, and the displayed first avatar is adjacent to the displayed first message. And, after displaying the first message and the first avatar, receiving the second message containing the second content from the second user, and the second status data for the second user. Receiving a second status, associated with the second message and separate from the second content, Displaying the second message containing the second content and the second avatar on the display while maintaining the display of the first message and the first avatar, that is, the displayed second avatar. Is adjacent to the displayed second message, the second avatar is different from the first avatar and the second avatar, based on the second status data, including the method. An item that further comprises displaying the contact information of a set of users, including the contact information of the second user, the second avatar being displayed with the contact information of the second user. The method described in 1. Receiving the first avatar model for the second user and generating the first avatar based on the first avatar model and the first status data, the first avatar model and the second status data Described in the method of item 1 or 2, further comprising generating a second avatar based on. Receiving a second avatar model for a second user and generating an updated first avatar based on the second avatar model and the first status data. Generate an updated second avatar based on the second avatar model and the second status data, and update instead of the first avatar with the first message containing the first content The method according to any one of items 1 to 3, further comprising displaying the first avatar that has been made. The first status data is based on the first biometric characteristics of the second user, any of items 1-4. The method of any of items 1-5, further comprising selecting one or more traits for the first avatar based on the first status data. Any of items 1-6, further including selecting one or more traits for the second avatar based on the second status data, the second status data is based on the second biometric trait The method described in Crab. The first avatar is an animated avatar, the method described in any of items 1-7. The method according to any one of items 1 to 8, wherein the first status data is based on an optical image or a depth image of a second user. The method of any of items 1-9, further comprising mapping the first status data to a given avatar model to create the first avatar. The method according to any one of items 1 to 10, wherein the first status data represents the emotion of the second user. The method according to any one of items 1 to 11, wherein the first status data is based on the detected facial expression of the second user when the second user composes and / or sends the first message. .. The method according to any one of items 1 to 12, wherein the second status data is based on the detected facial expression of the second user when the second user composes and / or sends the second message. .. Receiving the third content for the third message from the first user and on the electronic device, generating the third status data for the first user, and the third status data Any of items 1 to 12, further including associating with a third message, sending a third message to a second user, and sending a third status data to a second user. The method described in Crab. The third message containing the third content and the third avatar are displayed at the same time, the third avatar is based on the third status data, and the third message and the third avatar are the third. The method of item 13, further comprising being displayed at the same time as the second message and the second avatar. The method according to any one of items 1 to 14, wherein the first avatar and the second avatar represent the physical appearance of the second user. As described in any of items 1 to 16, displaying the first message and the first avatar comprises displaying the first message as a text bubble coming out of the mouth of the first avatar. Method. The method according to any of items 1 to 16, wherein the second user is associated with the source electronic device that sends the first message and the second message. Electronic devices, displays, one or more processors, one or more input devices, memory, An electronic device comprising one or more programs, the one or more programs being stored in memory and configured to be executed by one or more processors. Receives a first message from a second user that contains the first content, is the first status data for the second user, associated with the first message, and separate from the first content. A certain first status data is received, the first message containing the first content and the first avatar are displayed simultaneously on the display, and the first avatar is displayed based on the first status data. The displayed first avatar is adjacent to the displayed first message, displays the first message and the first avatar, and then receives a second message containing the second content from the second user. Receives a second status, which is the second status data for the second user, associated with the second message and separate from the second content, The second avatar, including the command, that displays the second message containing the second content and the second avatar on the display while maintaining the display of the first message and the first avatar. Is an electronic device that is adjacent to the displayed second message and the second avatar is different from the first and second avatars based on the second status data. A non-temporary computer-readable storage medium that stores one or more programs, one or more programs containing instructions, the instructions being one or more electronic devices including a display and one or more input devices. When executed by the processor, it causes the device to receive a first message containing the first content from the second user, which is the first status data for the second user and is associated with the first message. The first status data, which is separate from the first content, is received, the first message containing the first content and the first avatar are displayed simultaneously on the display, and the first avatar Is based on the first status data, the displayed first avatar is adjacent to the displayed first message, After displaying the first message and the first avatar, the second user receives the second message including the second content, which is the second status data for the second user and is the second. With a second message that contains a second content, while receiving a second status, associated with the message and separate from the second content, and maintaining the display of the first message and the first avatar. The second avatar is displayed on the display, the displayed second avatar is adjacent to the displayed second message, and the second avatar is based on the second status data, the first avatar and the second. A non-temporary computer-readable storage medium that is different from the 2 avatars. An electronic device that includes a display, one or more input devices, a means of receiving a first message containing the first content from a second user, and first status data for the second user. As a means of receiving the first status data, which is associated with the first message and separate from the first content, It is a means to display the first message including the first content and the first avatar on the display at the same time. The first avatar is based on the first status data, and the displayed first avatar is. A means adjacent to the displayed first message, and a means of receiving a second message containing a second content from a second user after displaying the first message and the first avatar. A means of receiving the second status, which is the second status data for the second user, associated with the second message and separate from the second content, and the first message and the first avatar. It is a means to display the second message containing the second content and the second avatar on the display while maintaining the display of, and the displayed second avatar is the displayed second message. Adjacent, the second avatar is an electronic device, including means, which are different from the first avatar and the second avatar, based on the second status data. An electronic device, a display, one or more processors, memory, One or more programs, and one or more programs are stored in memory and configured to be executed by one or more processors, one or more programs being described in items 1-18. An electronic device that contains instructions to perform any of the methods. A non-temporary computer-readable storage medium that stores one or more programs, one or more electronic devices in which one or more programs contain instructions and the instructions have a display and one or more input devices. A non-temporary computer-readable storage medium that, when run by a processor, causes the device to perform one of the methods described in items 1-18. An electronic device, including a display and a means of performing any of the methods described in items 1-18. A method, in an electronic device having one or more image sensors, memory, and a display, from one or more image sensors of the electronic device, which is the first image data of the object from the first viewpoint. Capturing the first image data, including the first optical image data, A second image from one or more image sensors of an electronic device, including second image data, a second optical image of the object from a second viewpoint different from the first viewpoint. Capturing the data, choosing an algorithm based on the change in perspective from the first perspective to the second perspective, and the additional necessary to continue 3D modeling of the object based on the algorithm. A method that includes determining image data and displaying on the display visual feedback that provides instructions to capture additional image data determined based on the selected algorithm. Receiving a third piece of data, including the third optical image data of the object from the third point of view, and selecting an updated algorithm different from the above algorithm based on the third point of view. Based on the updated algorithm, determine the updated additional image data that is different from the additional image data needed to continue 3D modeling of the object, The visual feedback that provides the updated instructions for capturing the updated additional image data is to be displayed on the display, and the updated instructions are displayed before selecting the updated algorithm. 25. The method of item 25, further comprising, which is different from the instructions given. Building a 3D model of an object based on the first image data, the second image data, the third image data, and the updated additional image data using the selected updated algorithm. 26. The method of item 26, further comprising. 26 or 27, further comprising sending at least a portion of the first image data to the remote server and receiving an indicator from the remote server that the third data is available to the object. the method of. Displaying visual feedback on the display that provides updated instructions to capture the updated additional image data displays the first instruction set according to the determination that the first algorithm has been selected. To do and Any of items 25-28, including that the visual feedback contains a second instruction set that is different from the first instruction set, according to the determination that a second algorithm that is different from the first algorithm has been selected. The method described in Crab. The method according to any of items 25 to 29, wherein the first image data includes the first depth image data of the object from the first viewpoint. The method according to any of items 25 to 30, further comprising acquiring the first position data with respect to the first viewpoint. The method of item 31, wherein selecting the algorithm is also based on the first position data. Further including capturing the second position data with respect to the second viewpoint, the second image data includes the second depth image data of the object from the second viewpoint, and selecting the algorithm also includes. One of the methods of items 25-31, which is based on the second location data. Building a 3D model of an object based on the first image data, the second image data, and additional image data using the selected algorithm. The method of any of items 25-31 or 46-51, further comprising storing a 3D model in memory. Choosing an algorithm is scan-based based on the change from the first viewpoint to the second viewpoint, which indicates that the first and second image data are from scanning the object. The method according to any of items 25 to 34, including selecting an algorithm. Choosing an algorithm selects a discrete image-based algorithm based on the change from the first perspective to the second perspective, which indicates that the first and second perspectives are for discrete images. The method of any of items 25-35, including: Identifying the support in the first image data touching the object and using the selected algorithm, based on the first and second image data of the object, the first The method of any of items 25-26, 28-33 or 35-36, including building a 3D model that does not include support touching an object. The first window containing the live image of the object is displayed on the display of the electronic device, and the second window containing the image of the model of the object is displayed on the display. 25. The method of any of items 25-37, further comprising being based on the image data of the first and second image data. An electronic device that includes a display, one or more processors, one or more input devices, memory, and one or more programs, and one or more programs are memory. Stored in, and configured to be executed by one or more processors, one or more programs are the first image data from one or more image sensors of an electronic device, the first The first image data, including the first optical image data of the object from the viewpoint, is captured, and the second image data from one or more image sensors of the electronic device, which is the first viewpoint. Capture the second image data, including the second optical image light data of the object from different second viewpoints, The algorithm was selected based on the change in perspective from the first perspective to the second perspective, and the algorithm was used to determine and select the additional image data needed to continue 3D modeling of the object. An electronic device, including instructions, that displays visual feedback on a display that provides instructions to capture additional image data determined based on the algorithm. A non-temporary computer-readable storage medium that stores one or more programs, one or more programs containing instructions, the instructions being one or more electronic devices containing a display and one or more input devices. When executed by a processor, the device contains first optical image data from one or more image sensors of the electronic device, which is first image data of the object from a first viewpoint. The second optical image of the object from a second viewpoint that is different from the first viewpoint and is the second image data from one or more image sensors of the electronic device. To capture the second image data, including, and to select an algorithm based on the change in viewpoint from the first viewpoint to the second viewpoint. Based on the algorithm, the display provides visual feedback that lets you determine the additional image data needed to continue 3D modeling of the object and provides instructions to capture the additional image data determined based on the selected algorithm. A non-temporary computer-readable storage medium to display on. An electronic device, a display, one or more input devices, and a first optical of an object from a first viewpoint, the first image data, from one or more image sensors of the electronic device. An object from a second viewpoint that is different from the first viewpoint and is the second image data from a means of capturing the first image data including the image data and one or more image sensors of the electronic device. Second optical image A means of capturing the second image data, including the optical data, and a means of selecting an algorithm based on the change of viewpoint from the first viewpoint to the second viewpoint, and the algorithm. Based on the means to determine the additional image data needed to continue 3D modeling of the object, An electronic device, including a means of displaying visual feedback on a display, which provides instructions to capture additional image data determined based on the selected algorithm. An electronic device that includes a display, one or more processors, memory, and one or more programs so that one or more programs are stored in memory and executed by one or more processors. An electronic device that is configured in one or more programs, including instructions that perform any of the methods described in items 25-38. A non-temporary computer-readable storage medium that stores one or more programs, one or more electronic devices in which one or more programs contain instructions and the instructions have a display and one or more input devices. A non-temporary computer-readable storage medium that, when run by a processor, causes the device to perform one of the methods described in items 25-38. An electronic device that includes a display and a means of performing any of the methods described in items 25-38. The way In an electronic device having a display and one or more image sensors, displaying the content in the application on the display, the content being displayed while the application is in the first configuration, and the content. Capturing image data from one or more image sensors on an electronic device, receiving a navigation request to leave the content after capturing the image data, and receiving a navigation request to leave the content while displaying. Depending on the determination that the first set of content locking criteria is met, the first set of content locking criteria is to prevent navigation away from the content while maintaining the display of the content. According to the determination that it contains a first criterion that is met when the captured image data indicates that a rogue user is using the device, and that the first set of content locking criteria is not met. Methods, including navigating away from the content as requested. The method of item 45, further comprising allowing navigation away from the content, according to the determination that the first set of content locking criteria is no longer met. The method of item 45 or 46, wherein the first set of locking criteria includes a second criterion that is met when the captured image data indicates that a legitimate user of the electronic device is not using the device. The first set of lock criteria is one of items 45-47, including a third criterion that is met when the captured image data indicates that there are rogue users and no legitimate users. The method described. The method of any of items 45-48, wherein the first set of locking criteria is satisfied when the captured image data indicates the presence of a rogue user with or without a legitimate user. .. The method of any of items 45-49, further comprising disabling at least one function of the electronic device in accordance with the determination that the second set of content locking criteria has been met. The method of item 50, which is different from the first set of lock criteria and the second set of lock criteria. Items 45-51, further comprising switching the application to a second configuration that limits the operation of the application compared to the first configuration, according to the determination that the third set of content locking criteria is met. The method described in either. Described in any of items 45-52, further comprising locking other functions of the electronic device while continuing to display content within the application, according to the determination that the fourth set of content locking criteria is met. the method of. The method according to any of items 45 to 53, further comprising preventing the display of notifications related to communications received on the electronic device in accordance with the determination that the fifth set of content locking criteria is met. The fifth set of lock criteria includes a fourth criterion that is met when the captured image data indicates that an unauthorized user is using an electronic device, and the fifth set of lock criteria is the fourth. 54. The method of item 54, wherein the first set of locking criteria is met if the criteria are met, including a fifth criterion that is met when the captured image data indicates the absence of a legitimate user. Preventing navigation between applications on electronic devices according to the fourth criterion being met, and preventing navigation within the application according to the fifth criterion being met. The method of item 55, including. The method of any of items 45-56, further comprising determining whether the captured image data indicates the presence of an unauthorized user of the electronic device. The image data includes optical data and depth data, and determining whether the first set of content locking criteria is met is based on the optical data and depth data, in any of items 45-57. The method described. One of items 45-58, including navigating away from the content to translate the currently displayed content. The method of any of items 45-59, wherein navigation away from the content involves switching between content items within the application. The method of any of items 45-60, wherein the navigation away from the content comprises switching applications or closing the application to display the home screen. Allows navigation away from content according to receiving unlock information associated with a legitimate user of the electronic device, determining if the unlock information is authentic, and determining that the unlock information is authentic. The method according to any of items 45 to 61, further comprising: An electronic device that includes a display, one or more processors, one or more input devices, memory, and one or more programs, and one or more programs are memory. Stored in and configured to be executed by one or more processors, one or more programs display the content in the application on the display, and the content is while the application is in the first configuration. Capture image data from one or more image sensors on an electronic device while being displayed and displaying content, After capturing the image data, the content is displayed according to the determination that the first set of content locking criteria is met in response to receiving a navigation request away from the content and receiving a navigation request away from the content. Preventing navigation away from the content while maintaining, the first set of content locking criteria includes the first criteria that are met when the captured image data indicates that a rogue user is using the device. An electronic device, including instructions, that navigates away from the content as requested, according to the determination that the first set of content locking criteria is not met. A non-temporary computer-readable storage medium that stores one or more programs, one or more programs containing instructions, the instructions being one or more electronic devices including a display and one or more input devices. When run by the processor, it causes the device to display the content within the application on the display, which is displayed while the application is in the first configuration. While displaying the content, the image data is captured from one or more image sensors of the electronic device, the image data is captured, then the navigation request to leave the content is received, and the navigation request to leave the content is received. The first set of content lock criteria prevents unauthorized users from using the device while maintaining the display of the content, according to the determination that the first set of content lock criteria is met. Including a first criterion that is met when the captured image data indicates that it is doing, and having the user navigate away from the content as requested, according to the determination that the first set of content locking criteria is not met. Non-temporary computer-readable storage medium. An electronic device, a display, one or more input devices, and a means of displaying content within an application on the display, the means by which the content is displayed while the application is in the first configuration. , A means of capturing image data from one or more image sensors of an electronic device while displaying the content, a means of receiving a navigation request to leave the content after capturing the image data, and a means to receive a navigation request to leave the content. Accordingly, a means of preventing navigation away from the content while maintaining the display of the content according to the determination that the first set of content locking criteria is met, the first set of content locking criteria. According to the means and the determination that the first set of content locking criteria is not met, including the first criterion that is met when the captured image data indicates that the rogue user is using the device. Electronic devices, including means for navigating away from content as requested. An electronic device, a display, one or more processors, memory, One or more programs, and one or more programs are stored in memory and configured to be executed by one or more processors, one or more programs being described in items 45-62. An electronic device that contains instructions to perform any of the methods. A non-temporary computer-readable storage medium that stores one or more programs, one or more electronic devices in which one or more programs contain instructions and the instructions have a display and one or more input devices. A non-temporary computer-readable storage medium that, when executed by a processor, causes the device to perform one of the methods described in items 45-62. An electronic device that includes a display and a means of performing any of the methods described in items 45-62.
The above description is given with reference to specific embodiments for purposes of explanation. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the exact forms disclosed. Many modifications and modifications are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the art and its practical application. This will allow other skilled skill in the art to make the best use of the present technology and various embodiments, using various modifications suitable for the specific application conceived.
Although the present disclosure and embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and amendments are to be understood as being within the scope of the present disclosure and examples, as defined by the claims.
As mentioned above, one aspect of the present technology is data available from various sources in order to improve delivery to the user of what the user wants or any other content that the user may be interested in. Is to collect and use. The disclosure, in some cases, includes personal information data that uniquely identifies a particular person, or personal information data that can be used to communicate with or locate a particular person. I'm thinking about getting it. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, home addresses, or any other identifying information.
The present disclosure recognizes that the use of such personal information data in the present technology may be in the interest of the user. For example, personal information data may be used to deliver content of interest to the user. Therefore, the use of such personal information data allows for the intentional control of the delivered content. Furthermore, other uses for personal information data that benefit the user are also conceived by this disclosure.
This disclosure states that the entity responsible for the collection, analysis, disclosure, transfer, storage, or other purpose of use of such personal information data complies with well-defined privacy policies and / or privacy practices. I think more. Specifically, such entities implement privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for the confidential and rigorous maintenance of personal information data. And should be used consistently. For example, personal information from users should be collected for the legitimate and legitimate use of that entity and should not be shared or sold except for their legitimate use. Furthermore, such collection should only be carried out after notifying the user and obtaining their consent. In addition, such entities shall protect and secure access to such personal information data so that others with access to that personal information data may comply with their privacy policies and procedures. We will take all necessary steps to ensure that. In addition, such entities may be evaluated by third parties to prove their compliance with widely accepted privacy policies and practices.
Despite the above, the disclosure also conceives embodiments in which the user selectively blocks the use of personal information data or access to personal information data. That is, it is conceivable that the present disclosure may provide hardware and / or software elements to prevent or prevent access to such personal information data. For example, in the case of an ad serving service, the technology is configured to allow the user to choose "opt-in" or "opt-out" to participate in the collection of personal information data during service registration. can do. In another embodiment, the user may choose not to provide location information to the target content distribution service. In yet another embodiment, the user may choose to allow the transfer of location zone information, although it does not provide accurate location information.
Therefore, although the present disclosure broadly covers the use of personal information data to implement one or more various disclosed embodiments, the present disclosure also covers such personal information data. It is conceivable that it is also possible to implement those various embodiments without requiring access to. That is, various embodiments of the present technology are not impractical due to the lack of all or part of such personal information data. For example, based on non-personal information data such as content requested by the device associated with the user or the minimum required personal information, other non-personal information available to the content distribution service, or publicly available information. By guessing the taste, the content can be selected and delivered to the user.
49 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP2016162000A | Cites | Japan |
| JP2005115480A | Cites | Japan |
| JP2003150550A | Cites | Japan |
| JP2001092783A | Cites | Japan |
| JP2007052770A | Cites | Japan |
57 members in 8 offices
Priority claims15
| Document | Office | Kind | Date |
|---|---|---|---|
| 201662399226 | United States of America | P | |
| 201662399226 | United States of America | P | |
| 62399226 | United States of America | – | |
| PA201770418 | Denmark | A | |
| PA201770418 | Denmark | A | |
| PA201770418 | Denmark | – | |
| 2017049760 | United States of America | W | |
| 2017049760 | United States of America | W | |
| 62399226 | – | – | – |
| DK2017PA70418 | – | – | – |
| DKPA201770418 | – | – | – |
| PA201770418 | – | – | – |
| US201662399226P | – | – | – |
| US2017049760 | – | – | – |
| WO2017US49760 | – | – | – |
Members57
| Document | Office | Kind | |
|---|---|---|---|
| US2018088787A1 | United States of America | A1 | |
| WO2018057268A1 | World Intellectual Property Organization (WIPO) | A1 | |
| DK201770418A1 | Denmark | A1 | |
| DK201770419A1 | Denmark | A1 | |
| WO2018057268A4 | World Intellectual Property Organization (WIPO) | A4 | |
| US2018321826A1 | United States of America | A1 | |
| WO2018212801A1 | World Intellectual Property Organization (WIPO) | A1 | |
| DK179471B1 | Denmark | B1 | |
| EP3427132A1 | European Patent Office (EPO) | A1 | |
| CN109416591A | China | A | |
| AU2017330208A1 | Australia | A1 | |
| KR20190034248A | Republic of Korea | A | |
| CN109691074A | China | A | |
| EP3485392A1 | European Patent Office (EPO) | A1 | |
| CN109981908A | China | A | |
| EP3485392A4 | European Patent Office (EPO) | A4 | |
| US10444963B2 | United States of America | B2 | |
| DK179978B1 | Denmark | B1 | |
| AU2018270420A1 | Australia | A1 | |
| AU2017330208B2 | Australia | B2 | |
| KR20190141701A | Republic of Korea | A | |
| JP2020501212A | Japan | A | |
| AU2020201721A1 | Australia | A1 | |
| JP2020520031A | Japan | A | |
| EP3427132B1 | European Patent Office (EPO) | B1 | |
| EP3757728A1 | European Patent Office (EPO) | A1 | |
| CN109981908B | China | B | |
| JP6824552B2This record | Japan | B2 | |
| EP3485392B1 | European Patent Office (EPO) | B1 | |
| JP2021073563A | Japan | A | |
| KR102257353B1 | Republic of Korea | B1 | |
| KR20210063451A | Republic of Korea | A | |
| AU2020201721B2 | Australia | B2 | |
| AU2018270420B2 | Australia | B2 | |
| AU2021250944A1 | Australia | A1 | |
| AU2021254574A1 | Australia | A1 | |
| EP3920052A1 | European Patent Office (EPO) | A1 | |
| EP3920052A4 | European Patent Office (EPO) | A4 | |
| JP2022023864A | Japan | A | |
| KR20220025212A | Republic of Korea | A | |
| KR102403090B1 | Republic of Korea | B1 | |
| US2022244838A1 | United States of America | A1 | |
| JP7158509B2 | Japan | B2 | |
| AU2021250944B2 | Australia | B2 | |
| AU2021254574B2 | Australia | B2 | |
| JP2023011639A | Japan | A | |
| AU2023200797A1 | Australia | A1 | |
| KR20230117638A | Republic of Korea | A | |
| JP2024028992A | Japan | A | |
| AU2023200797B2 | Australia | B2 | |
| JP7495913B2 | Japan | B2 | |
| US12079458B2 | United States of America | B2 | |
| US2024310988A1 | United States of America | A1 | |
| JP2024178162A | Japan | A | |
| KR102794887B1 | Republic of Korea | B1 | |
| JP7664361B2 | Japan | B2 | |
| CN120540764A | China | A |
12 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 6824552
- Publication, DOCDB
- 6824552
- Publication, EPODOC
- JP6824552B
- Application
- 2019511975
- Application, DOCDB
- 2019511975
- Application, EPODOC
- JP20190511975
Titles2
- Japanese
- 拡張されたユーザ対話のための画像データ
- English
- Image data for extended user interaction
Classification
- CPC, 13
- H04M1/72427
- G06F21/32
- G06F3/0488
- H04M1/72436
- H04M2250/52
- G06F2221/2149
- G06Q10/10
- G06F3/04817
- G06F3/0482
- G06F3/04845
- G06F3/0485
- G06V20/20
- G06T7/55
- IPC, 4
- G06T19 00
- G06F3 0481
- H04M1 72427
- H04M1 72436
