Devices, methods and graphical user interfaces for displaying an affordance on a background
11 claims: 3 independent, 8 dependent
- 1表 示生成コンポーネントと通信するコンピュータシステム によって実行される方法であって 、ビデオコンテンツが見えている間に、前記表示生成コンポーネントを介して第1のユーザインタフェースオブジェクトを表示することを含み、前記第1のユーザインタフェースオブジェクトは、前記ビデオコンテンツの少なくとも第1の部分に重なるように見えており、前記第1のユーザインタフェースオブジェクトを表示することは、前記ビデオコンテンツの少なくとも前記第1の部分の1つ以上のディスプレイ特性が第1の基準を満たすとの判定に従って、前記第1のユーザインタフェースオブジェクトの第1の変形を、前記ビデオコンテンツの少なくとも前記第1の部分に重ねて表示することであって、前記第1のユーザインタフェースオブジェクトの前記第1の変形は、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分に基づく外観を有し、かつ、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分の第1のディスプレイ特性が第2の値範囲を有する場合に、前記第1のディスプレイ特性の第1の値範囲を用いて生成される、ことと、前記ビデオコンテンツの少なくとも前記第1の部分の前記1つ以上のディスプレイ特性が、前記第1の基準とは異なる第2の基準を満たすとの判定に従って、前記第1のユーザインタフェースオブジェクトの第2の変形を、前記ビデオコンテンツの少なくとも前記第1の部分に重ねて表示することであって、前記第2の変形は、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分に基づく外観を有し、かつ、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分の前記第1のディスプレイ特性が前記第2の値範囲を有する場合に、前記第1のディスプレイ特性の第3の値範囲を用いて生成され、前記第1の値範囲は前記第3の値範囲とは異なる、ことと、を含む、方法。
- 2請求項1に記載の方法であって、前記第1のユーザインタフェースオブジェクトの前記第1の変形を表示することは、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの前記第1の部分に第1の反転量を適用することで、前記第1のユーザインタフェースオブジェクトの前記第1の変形の前記外観を得ることを含み、前記第1のユーザインタフェースオブジェクトの前記第2の変形を表示することは、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの前記第1の部分に、前記第1の反転量よりも小さい第2の反転量を適用することで、前記第1のユーザインタフェースオブジェクトの前記第2の変形の前記外観を得ることを含 み、 各反転量は、前記第1のユーザインタフェースオブジェクトと、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの前記第1の部分との間に、対応する量の外観上のコントラストを生成する 、方法。
- 3請求項1又は2に記載の方法であって、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの前記第1の部分をぼかすことで、前記第1のユーザインタフェースオブジェクトの前記外観を得ることを含む、方法。
- 4請求項1乃至3のいずれか一項に記載の方法であって、前記ビデオコンテンツの外観の時間にわたる変化を検出することと、前記ビデオコンテンツの外観の時間にわたる前記変化を検出したことに応じて、前記ビデオコンテンツの外観の時間にわたる前記変化に従って、前記第1のユーザインタフェースオブジェクトの前記外観を時間にわたって変化させることと、を含む、方法。
- 5請求項1乃至4のいずれか一項に記載の方法であって、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分の外観の変化を検出することと、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分の外観の前記変化を検出したことに応じて、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの前記第1の部分の外観の前記変化に従って、前記第1のユーザインタフェースオブジェクトの前記外観を変化させることと、を含む、方法。
- 6請求項1乃至5のいずれか一項に記載の方法であって、前記ビデオコンテンツの少なくとも前記第1の部分の前記1つ以上のディスプレイ特性の変化を検出することと、前記ビデオコンテンツの少なくとも前記第1の部分の前記1つ以上のディスプレイ特性の前記変化を検出したことに応じて、前記ビデオコンテンツの少なくとも前記第1の部分の前記1つ以上のディスプレイ特性の前記変化が第3の基準を満たすとの判定に従って、前記第1のユーザインタフェースオブジェクトの前記第1の変形の表示から、前記第1のユーザインタフェースオブジェクトの前記第2の変形の表示に遷移させることと、前記ビデオコンテンツの少なくとも前記第1の部分の前記1つ以上のディスプレイ特性の前記変化が、前記第3の基準とは異なる第4の基準を満たすとの判定に従って、前記第1のユーザインタフェースオブジェクトの前記第2の変形の表示から、前記第1のユーザインタフェースオブジェクトの前記第1の変形の表示に遷移させることと、を含む、方法。
- 7請求項1乃至6のいずれか一項に記載の方法であって、前記ビデオコンテンツの少なくとも前記第1の部分に重ねて前記第1のユーザインタフェースオブジェクトの前記第1の変形を表示することは、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分の1つ以上のディスプレイ特性の変化に基づいて、前記第1のユーザインタフェースオブジェクトの少なくとも第1の部分の少なくとも前記第1のディスプレイ特性の値を、ルールの第1のセットに従って変化させる ことで、前記第1のユーザインタフェースオブジェクトの前記第1の変形の前記外観を得る ことを含み、前記ビデオコンテンツの少なくとも前記第1の部分に重ねて前記第1のユーザインタフェースオブジェクトの前記第2の変形を表示することは、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの少なくとも前記第1の部分の前記1つ以上のディスプレイ特性の前記変化に基づいて、前記第1のユーザインタフェースオブジェクトの少なくとも前記第1の部分の少なくとも前記第1のディスプレイ特性の前記値を、ルールの前記第1のセットとは異なるルールの第2のセットに従って変化させる ことで、前記第1のユーザインタフェースオブジェクトの前記第2の変形の前記外観を得る ことを含む、方法。
- 8請求項1乃至7のいずれか一項に記載の方法であって、前記第1のユーザインタフェースオブジェクトの下にある前記ビデオコンテンツの前記第1の部分を非飽和化することで、前記第1のユーザインタフェースオブジェクトの前記外観を取得することを含む、方法。
- 9請求項1乃至8のいずれか一項に記載の方法であって、前記ビデオコンテンツが最初に表示されたときの前記ビデオコンテンツの外観に従って、前記第1のユーザインタフェースオブジェクトの前記第1の変形と前記第1のユーザインタフェースオブジェクトの前記第2の変形とのうちの1つを表示することを選択することを含む、方法。
- 10コンピュータシステムであって、表示生成コンポーネントと、1つ以上の入力デバイスと、1つ以上のプロセッサと、1つ以上のプログラムを記憶したメモリと、を備え、前記1つ以上のプログラムは、前記1つ以上のプロセッサによって実行されるように構成され、前記1つ以上のプログラムは、請求項1乃至9のいずれか一項に記載の方法を実行するための命令を含む、コンピュータシステム。
- 11命令を含むプログラムであって、当該命令は、表示生成コンポーネント及び1つ以上の入力デバイスを有するコンピュータシステムによって実行されると前記コンピュータシステムに、請求項1乃至9のいずれか一項に記載の方法を実行させる、プログラム。
Independent claims11
283 paragraphs, as filed
The present invention relates generally to electronic devices, including but not limited to electronic devices that have a display and a touch-sensitive surface and that display virtual affordances (e.g., controls, indicators, visual guides, etc.) on their touch-sensitive display screens.
The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Exemplary touch-sensitive surfaces include touchpads and touchscreen displays. Such surfaces are widely used to manipulate user interfaces on the displays and objects therein. Exemplary user interface objects include digital images, video, text, icons, and control elements such as buttons and other graphics.
Electronic computing devices often display virtual controls or visual guides on their displays. For example, keyboards, menus, dialog boxes, alerts, and other controls may be activated and manipulated (e.g., by touch input) that cause actions to be performed on a portable electronic device (e.g., a smart phone, tablet, or notebook computer). Indicators and visual guides may be overlaid on a background (e.g., an application's user interface or an operating system's user interface) that provide visual cues regarding the type of input that may be provided and/or the type of action that may be performed in association with a particular region of the background or screen.
Existing methods for displaying controls, indicators, and visual guides can be cumbersome and inefficient. For example, the controls, indicators, and visual guides may unnecessarily distract a user as the user operates the user interface, or may not be clear or prominent enough against the background, resulting in user errors and confusion as the user interacts with the device, as well as negatively impacting the energy consumption of the device. The latter problem is particularly acute in battery-operated devices.
Furthermore, certain types of affordances are displayed across diverse backgrounds and content, sometimes for long periods of time without moving. As a result, displays may exhibit ghost images (or burn-in) of the affordances after a period of use. Reducing and eliminating display burn-in has been a long-standing challenge faced by display device manufacturers. Some existing methods to solve this problem, such as screensavers or flickering icons, are not satisfactory solutions due to their side effects (e.g., causing eye strain, distraction, etc.) and lack of effectiveness in many scenarios (e.g., only usable when the device is idle).
Accordingly, electronic devices that have salient, yet undistracting, affordances (e.g., controls, indicators, visual guides, etc.) that help provide sufficient visual legibility need to guide users in providing the inputs necessary to achieve a desired result without unnecessarily diverting the user's attention from the content displayed on the application or system user interface. Additionally, for certain affordances that are displayed for extended periods of time without movement, the affordances need to be displayed in a manner that reduces or eliminates screen burn-in.
In addition, with background content constantly changing over time, either by itself or in response to user input, the appearance of the affordance must also dynamically adapt to remain efficient and effective for the above purpose. In particular, affordances that serve as controls or visual guides for gestures that trigger some frequently used system functionality are displayed in many different contexts (e.g., on user interfaces of various applications and operating systems). Sometimes, the background of an affordance in a given context (e.g., scrollable, rapidly changing, or unpredictable content) also changes dynamically, further requiring the appearance of the affordance to continually adapt to changes in the appearance of the background after the affordance is initially displayed on the background. One example of such an affordance is a system-level affordance that indicates the starting area of a home/multitasking gesture, which causes the dismissal of a currently displayed application user interface and the display of a home screen user interface, or the dismissal of a cover sheet-like system information interface (e.g., a notification center or a lock screen user interface) and the display of a previously displayed user interface (e.g., an application user interface or a home screen).
In addition, sometimes, after an affordance is displayed on the application's user interface, a change in operational context occurs in the application that may qualitatively change the user's likelihood of interacting with or needing visual guidance about the affordance. In such cases, a balance must be struck between making the affordance salient and not being too distracting to maintain the effectiveness and efficiency of the user interface.
Additionally, for certain affordances that are displayed for long periods of time without movement, there is a need to display the affordances in a way that reduces or eliminates screen burn-in.
The above demands require new methods and interfaces for displaying affordances over a background and for coordinating the appearance of the affordances (e.g., virtual controls, indicators, and visual guides). Such devices, methods, and interfaces reduce the cognitive load on users and create more efficient human-machine interfaces. In addition, such devices, methods, and interfaces reduce or eliminate screen burn-in, thereby reducing device repair costs and increasing device lifespan.
In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the device is a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also known as a "touch screen" or "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI primarily through stylus and/or finger contacts and gestures on a touch-sensitive surface. In some embodiments, the functionality optionally includes image editing, drawing, presentations, word processing, creating spreadsheets, playing games, making phone calls, video conferencing, sending emails, instant messaging, training support, digital photography, digital videography, web browsing, playing digital music, taking notes, playing digital videos, and system level operations such as displaying a home screen, locking the device, displaying a system level notification screen, displaying a system level control panel user interface, etc. Executable instructions for performing these functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.
According to some embodiments, a method comprises, in a device having a display and a touch-sensitive surface, displaying content and an affordance on the display, where the affordance is displayed over a portion of the content, a value of a display characteristic of the affordance being determined based on a value of the same display characteristic of the portion of the content over which the affordance is displayed, the value of the display characteristic of the content being allowed to vary within a first value range and the value of the display characteristic of the affordance being constrained to vary within a second value range that is smaller than the first value range; detecting, while displaying the content and the affordance, a change in appearance of the content over which the affordance is displayed; and in response to detecting the change in appearance of the content over which the affordance is displayed, changing an appearance of the affordance including, in response to a determination that the value of the display characteristic of the content has decreased, increasing the value of the display characteristic of the affordance in response to a magnitude of the change in the value of the display characteristic of the content and the second value range, and in response to a determination that the value of the display characteristic of the content has increased, decreasing the value of the display characteristic of the affordance in response to a magnitude of the change in the value of the display characteristic of the content and the second value range.
According to some embodiments, a method includes causing, on a device having a display and a touch-sensitive surface, to display a user interface of an application; and, while displaying the user interface of the application in a first mode, displaying an affordance having a first appearance on the user interface, the affordance being displayed on a portion of the user interface, where values of a set of one or more display characteristics of the affordance having the first appearance change in response to changes in values of the set of one or more display characteristics of the portion of the user interface underlying the affordance according to a first set of one or more rules; and displaying the affordance having the first appearance on the user interface while displayed in the first mode. detecting a request to transition from displaying the user interface in a first mode to displaying the user interface in a second mode while displaying the user interface on a portion of the user interface; in response to detecting the request, causing the user interface to be displayed in the second mode; and displaying an affordance having a second appearance on the user interface displayed in the second mode, wherein values of a set of one or more display characteristics of the affordance having the second appearance change according to a second set of one or more rules different from the first set of one or more rules in response to changes in values of the set of one or more display characteristics of the portion of the user interface underlying the affordance.
According to some embodiments, a method includes, in a device having a display and a touch-sensitive surface, displaying content and an affordance on the display, where the affordance is displayed over a portion of the content, and values of display characteristics of the affordance are determined based on values of the same display characteristics of the portion of the content over which the affordance is displayed, where values of the display characteristics of the content are allowed to vary within a first value range and values of the display characteristics of the affordance are constrained to vary within an affordance appearance value range that is smaller than the first value range; detecting, while displaying the content and the affordance, and while the affordance appearance value range is a second value range, a change in appearance of the content over which the affordance is displayed; and in response to detecting the change in appearance of the content over which the affordance is displayed. changing an appearance of the affordance, wherein in accordance with a determination that the appearance change of the content satisfies the range switching criterion, shifting a value range of the affordance appearance to a third value range, the third value range being different from the second value range, the third value range being smaller than the first value range; changing a value of a display characteristic of the affordance in response to a value of the same display characteristic of the content over which the affordance is displayed, the display characteristic of the affordance being constrained to vary within the value range of the affordance appearance; and in accordance with a determination that the appearance change of the content does not satisfy the range switching criterion, changing the value of the display characteristic of the affordance in response to a value of the same display characteristic of the content over which the affordance is displayed, while maintaining the value range of the affordance appearance as the second value range.
According to some embodiments, an electronic device includes a display, a touch-sensitive surface, optionally one or more sensors that detect intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, one or more processors, and a memory having stored thereon one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by an electronic device having a display, a touch-sensitive surface, optionally one or more sensors that detect intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a graphical user interface on an electronic device having a display, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, a memory, and one or more processors executing one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, and the elements are updated in response to an input as described in any of the methods described herein. According to some embodiments, the electronic device includes a display, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, and means for performing or causing to be performed any of the operations of the methods described herein. According to some embodiments, an information processing apparatus for use in an electronic device having a display, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators includes means for performing or causing to be performed any of the operations of the methods described herein.
Thus, electronic devices having a display, a touch-sensitive surface, optionally one or more sensors that detect intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, optionally one or more device orientation sensors, and optionally an audio system are provided with improved methods and interfaces for navigating through user interfaces and interacting with control objects, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can complement or replace traditional methods of displaying affordances on a background.
For a better understanding of the various embodiments described, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:
This patent or application document contains at least one printed color drawing. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
<figref num="1A">1 is a block diagram illustrating a portable multifunction device having a touch-sensitive display in accordance with some embodiments.</figref>
<figref num="1B">FIG. 2 is a block diagram illustrating example components for event processing according to some embodiments.</figref>
<figref num="2">1 illustrates a portable multifunction device having a touch screen in accordance with some embodiments.</figref>
<figref num="3">1 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface in accordance with some embodiments.</figref>
<figref num="4A">1 illustrates an exemplary user interface for a menu of applications on a portable multifunction device in accordance with some embodiments.</figref>
<figref num="4B">1 illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface separate from a display in accordance with some embodiments.</figref>
<figref num="5A">1 illustrates an example user interface including affordances whose appearance adapts to the appearance of the background according to some embodiments.</figref><figref num="5B">1 illustrates an example user interface including affordances whose appearance adapts to the appearance of the background according to some embodiments.</figref><figref num="5C">1 illustrates an example user interface including affordances whose appearance adapts to the appearance of the background according to some embodiments.</figref><figref num="5D">1 illustrates an example user interface including affordances whose appearance adapts to the appearance of the background according to some embodiments.</figref>
<figref num="5E">5A-5D show filters used to generate the appearance of the affordances of FIGS. 5A-5D based on the underlying content, according to some embodiments. </figref>
<figref num="5F">5C illustrates an example of an inversion curve for performing the inversion shown in FIG. 5E, according to some embodiments. </figref>
<figref num="5G">1 illustrates a change in appearance of an affordance of a first affordance appearance type (e.g., a "dark" affordance type), according to some embodiments.</figref><figref num="5H">1 illustrates a change in appearance of an affordance of a first affordance appearance type (e.g., a "dark" affordance type), according to some embodiments.</figref><figref num="5I">1 illustrates a change in appearance of an affordance of a first affordance appearance type (e.g., a "dark" affordance type), according to some embodiments.</figref><figref num="5J">1 illustrates a change in appearance of an affordance of a first affordance appearance type (e.g., a "dark" affordance type), according to some embodiments.</figref><figref num="5K">1 illustrates a change in appearance of an affordance of a first affordance appearance type (e.g., a "dark" affordance type), according to some embodiments.</figref>
<figref num="5L">13 illustrates a change in appearance of an affordance of a second affordance appearance type (e.g., the "light" affordance type) according to some embodiments.</figref><figref num="5M">13 illustrates a change in appearance of an affordance of a second affordance appearance type (e.g., the "light" affordance type) according to some embodiments.</figref><figref num="5N">13 illustrates a change in appearance of an affordance of a second affordance appearance type (e.g., the "light" affordance type) according to some embodiments.</figref><figref num="5O">13 illustrates a change in appearance of an affordance of a second affordance appearance type (e.g., the "light" affordance type) according to some embodiments.</figref><figref num="5P">13 illustrates a change in appearance of an affordance of a second affordance appearance type (e.g., the "light" affordance type) according to some embodiments.</figref>
<figref num="5Q">1 illustrates the difference in affordance appearance for two types of affordance appearance types, assuming the same background changes, according to some embodiments.</figref>
<figref num="5R">13 illustrates a range of values and an inverse relationship between the display characteristics of affordances and underlying content for dark and light affordance appearance types, according to some embodiments.</figref>
<figref num="5S">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5T">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5U">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5V">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5W">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5X">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5Y">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5Z">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref><figref num="5AA">1 illustrates a user interface including affordances having appearances responsive to changes in the appearance of a background and changes in an operational mode associated with the background, according to some embodiments.</figref>
<figref num="5AB">1 illustrates the difference in appearance of affordances against a background in different operational modes according to some embodiments.</figref><figref num="5AC">1 illustrates the difference in appearance of affordances against a background in different operational modes according to some embodiments.</figref>
<figref num="5AD">1 illustrates a user interface including affordances that dynamically switch between affordance appearance types based on changes in underlying content over time, according to some embodiments.</figref>
<figref num="5AE">13 illustrates value ranges and inverse relationships between display characteristics of affordances and underlying content for light, dark, and transition affordance appearance types, according to some embodiments.</figref>
<figref num="5AF">1 illustrates a gradual shift from a dark affordance appearance type to a light affordance appearance type through multiple transition affordance appearance types, according to some embodiments.</figref>
<figref num="5AG">An expanded replication of the inversion relationship between affordance display properties and underlying content for the different affordance types shown in Figure 5AF.</figref><figref num="5AH">An expanded replication of the inversion relationship between affordance display properties and underlying content for the different affordance types shown in Figure 5AF.</figref><figref num="5AI">An expanded replication of the inversion relationship between affordance display properties and underlying content for the different affordance types shown in Figure 5AF.</figref><figref num="5AJ">An expanded replication of the inversion relationship between affordance display properties and underlying content for the different affordance types shown in Figure 5AF.</figref><figref num="5AK">An expanded replication of the inversion relationship between affordance display properties and underlying content for the different affordance types shown in Figure 5AF.</figref>
<figref num="6A">FIG. 1 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the appearance of underlying content, according to some embodiments.</figref><figref num="6B">FIG. 1 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the appearance of underlying content, according to some embodiments.</figref><figref num="6C">FIG. 1 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the appearance of underlying content, according to some embodiments .</figref>
<figref num="7A">FIG. 11 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the mode of the user interface in which the underlying content and the affordance appearance changes according to some embodiments.</figref><figref num="7B">FIG. 11 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the mode of the user interface in which the underlying content and the affordance appearance changes according to some embodiments.</figref><figref num="7C">FIG. 11 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the mode of the user interface in which the affordance is displayed, according to some embodiments.</figref><figref num="7D">FIG. 11 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the mode of the user interface in which the affordance is displayed, according to some embodiments.</figref><figref num="7E">FIG. 11 is a flow diagram illustrating a method for changing the appearance of an affordance in response to changes in the mode of the user interface in which the underlying content and the affordance appearance changes according to some embodiments.</figref>
<figref num="8A">FIG. 11 is a flow diagram illustrating a method for changing affordance appearance and affordance appearance type in response to changes in the appearance of underlying content, according to some embodiments.</figref><figref num="8B">FIG. 11 is a flow diagram illustrating a method for changing affordance appearance and affordance appearance type in response to changes in the appearance of underlying content, according to some embodiments.</figref><figref num="8C">FIG. 11 is a flow diagram illustrating a method for changing affordance appearance and affordance appearance type in response to changes in the appearance of underlying content, according to some embodiments.</figref><figref num="8D">FIG. 11 is a flow diagram illustrating a method for changing affordance appearance and affordance appearance type in response to changes in the appearance of underlying content, according to some embodiments.</figref><figref num="8E">FIG. 11 is a flow diagram illustrating a method for changing affordance appearance and affordance appearance type in response to changes in the appearance of underlying content, according to some embodiments.</figref><figref num="8F">FIG. 11 is a flow diagram illustrating a method for changing affordance appearance and affordance appearance type in response to changes in the appearance of underlying content, according to some embodiments.</figref>
Affordances displayed according to conventional methods are often visually confusing and may clutter the user interface. In addition, the appearance of affordances is often fixed and does not adapt to changes occurring in the underlying content or changes in the operating mode of the underlying user interface. The following embodiments disclose a method for displaying and changing the appearance of affordances based on changes in the underlying content, where the inversion of the display characteristics of the underlying content provides a basis for determining values of the same display characteristics of the affordances. Furthermore, the value range of the display characteristics for the affordances is constrained to a subrange of the value range of the display characteristics for the underlying content. For example, different value ranges of the luminance of the affordances are provided for different affordance appearance types (e.g., dark affordance appearance type and light affordance appearance type) selected for backgrounds of different overall luminance levels (e.g., dark background and light background). This promotes visibility of the affordances against a changing background without unduly distracting the user. In some embodiments, depending on the operational mode of the user interface in which the affordance is displayed, the appearance of the affordance changes along with the appearance of the background according to different sets of rules, thereby providing a way to balance the need to maintain visibility of the affordance and the need to reduce undue distraction from the affordance as the operational context of the affordance changes. In some embodiments, both the appearance and affordance appearance type of the affordance change in response to changes in the appearance of the underlying content, including both instantaneous changes and cumulative changes over time.
In the following, Figures 1A-1B, 2, and 3 provide a description of example devices. Figures 4A-4B, 5A-5D, 5G-5P, 5S-5AA, and 5AD are examples of user interfaces with affordances that change their appearance in response to changes in the appearance of the underlying content, according to some embodiments. Figures 5E, 5Q, 5R, 5AB, 5AC, and 5AE-5AK show the difference in value ranges of affordance appearance and substrate appearance used in generating the affordances shown in Figures 5A-5D, 5G-5P, 5S-5AA, and 5AD, according to some embodiments. Figures 6A-6C, 7A-7E, and 8A-8F are flow diagrams of methods for displaying and adjusting affordance appearance, according to some embodiments. The user interfaces, affordance appearances, and value ranges shown in Figures 4A-4B, 5A-5AK are used to illustrate the processes of Figures 6A-6C, 7A-7E, and 8A-8F.
Exemplary Devices
Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments being described. However, it will be apparent to those skilled in the art that the various embodiments being described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
In this specification, terms such as first, second, etc. are sometimes used to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first contact can be referred to as a second contact, and similarly, a second contact can be referred to as a first contact, without departing from the scope of the various embodiments described. Although a first contact and a second contact are both contacts, they are not the same contact unless the context clearly indicates otherwise.
The terms used in the description of the various embodiments described herein are for the purpose of describing particular embodiments only and are not intended to be limiting. When used in the description of the various embodiments described and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is also to be understood that the term "and/or," as used herein, refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and/or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As used herein, the term "if" is interpreted to mean "when," "upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (a stated condition or event) is detected" can be interpreted to mean "upon determining," "in response to determining," "upon detecting (the stated condition or event)," or "in response to detecting (the stated condition or event)," depending on the context. "in response to detecting (a stated condition or event)."
Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile telephone, that also includes other functions, such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as laptops or tablet computers having a touch-sensitive surface (e.g., a touchscreen display and/or a touchpad), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touchscreen display and/or a touchpad).
In the following discussion, an electronic device is described that includes a display and a touch-sensitive surface, however, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, a mouse, and/or a joystick.
The device typically supports a variety of applications such as one or more of note-taking applications, drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and/or digital video player applications.
Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more features of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and/or changed for each application and/or within each application. In this way, the common physical architecture (such as the touch-sensitive surface) of the device optionally supports a variety of applications with user interfaces that are intuitive and transparent to the user.
Attention is now directed to embodiments of portable devices having touch-sensitive displays. FIG. 1A is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112, according to some embodiments. Touch-sensitive display system 112 may be conveniently referred to as a "touch screen" or simply a touch-sensitive display. Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more light sensors 164. Device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of a contact on device 100 (e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate over one or more communication buses or signal lines 103.
As used herein and in the claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that is detected by a user upon the user's touch. For example, in a situation where a device or a component of the device is in contact with a touch-sensitive user surface (e.g., a user's finger, palm, or other part of the hand), the tactile output generated by the physical displacement is interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "down-click" or "up-click" of a physical actuator button. In some cases, the user feels a tactile sensation such as a "down-click" or "up-click" even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's action. As another example, movement of the touch-sensitive surface is, optionally, interpreted or sensed by a user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. Such a user's interpretation of touch depends on the user's personal sensory perception, but there are many sensory perceptions of touch that are common to a majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "up click," "down click," "roughness"), unless otherwise noted, the generated tactile output corresponds to a physical displacement of the device, or a component of the device, that produces the described sensory perception for a typical (or average) user. Using tactile output to provide tactile feedback to a user enhances the usability of the device and makes the user device interface more efficient (e.g., by assisting the user in providing appropriate inputs when manipulating/interacting with the device and reducing user errors), as well as reducing the device's power usage and improving battery life by allowing the user to use the device more quickly and efficiently.
In some embodiments, the tactile output pattern specifies characteristics of the tactile output, such as the amplitude of the tactile output, the shape of the moving waveform of the tactile output, the frequency of the tactile output, and/or the duration of the tactile output.
When tactile outputs having different tactile output patterns are generated by the device (e.g., via one or more tactile output generators that move a movable mass to generate the tactile output), the tactile outputs can cause different tactile sensations to a user holding or touching the device. While the user's sensations are based on the user's perception of the tactile output, most users are able to distinguish changes in the waveform, frequency, and amplitude generated by the device. Thus, the waveform, frequency, and amplitude can be adjusted to indicate to the user that different operations have been performed. In that way, tactile outputs having tactile output patterns designed, selected, and/or engineered to simulate the properties (e.g., size, material, weight, stiffness, smoothness, etc.), behaviors (e.g., vibration, displacement, acceleration, rotation, magnification, etc.), and/or interactions (e.g., collision, adhesion, repulsion, attraction, friction, etc.) of objects in a given environment (e.g., a user interface including graphical features and objects, a simulated physical environment having virtual boundaries and virtual objects, a real physical environment having physical boundaries and physical objects, and/or any combination of the above) provide a user with useful feedback that, in some circumstances, reduces input errors and increases the efficiency of the user's operation of the device. In addition, tactile outputs are optionally generated to correspond to feedback not related to simulated physical properties, such as input thresholds or object selection. Such tactile outputs provide a user with useful feedback that, in some circumstances, reduces input errors and increases the efficiency of the user's operation of the device.
In some embodiments, a tactile output with an appropriate tactile output pattern serves as a cue to the occurrence of an event of interest behind the scenes in a user interface or device. Examples of events of interest include activation of an affordance (e.g., a real button, a virtual button, or a toggle switch) provided on the device or in the user interface, success or failure of a requested operation, reaching or crossing a boundary in the user interface, entering a new state, switching input focus between objects, activating a new mode, reaching or crossing an input threshold, detection or recognition of a type of input or gesture, etc. In some embodiments, a tactile output is provided to serve as a warning or alert to an impending event or outcome that will occur unless a redirection or interrupt input is detected in a timely manner. Tactile output is also used in other contexts to enhance the user experience, improve the accessibility of the device to users with visual or motor impairments or other accessibility needs, and/or improve the efficiency and functionality of the user interface and/or device. The tactile output optionally involves audio output and/or visible user interface changes that further enhance the user's experience when interacting with the user interface and/or device, better communicate information about the state of the user interface and/or device, and reduce input errors and increase efficiency in the user's operation of the device.
It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of components. The various components shown in FIG. 1A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and/or application specific integrated circuits.
Memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU(s) 120 and peripherals interface 118, is optionally controlled by a memory controller 122.
The peripherals interface 118 may be used to connect input and output peripherals of the device with the CPU(s) 120 and memory 102. The one or more processors 120 operate or execute various software programs and/or instruction sets stored in the memory 102 to perform various functions and process data for the device 100.
In some embodiments, peripherals interface 118, CPU(s) 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.
RF(radio The radio frequency (RF) circuitry 108 transmits and receives RF signals, also referred to as electromagnetic signals. The RF circuitry 108 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuitry 108 optionally includes well-known circuits for performing their functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, and memory. The RF circuitry 108 optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet, and/or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and/or metropolitan area networks (MANs), and other devices by wireless communication. The wireless communication optionally includes any of the following wireless technologies: Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and/or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and/or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any other suitable communications protocol, including communications protocols not yet developed as of the filing date of this document.
The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripherals interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data, and transmits the audio data to the peripherals interface 118 for processing. The audio data is optionally retrieved from and/or transmitted to the memory 102 and/or the RF circuit 108 by the peripherals interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212, FIG. 2). The headset jack provides an interface between audio circuitry 110 and a removable audio input/output peripheral device, such as an output-only headphone or a headset that has both an output (e.g., one- or binaural headphones) and an input (e.g., a microphone).
I/O subsystem 106 connects input/output peripherals on device 100, such as touch-sensitive display system 112, and other input or control devices 116, with peripheral interface 118. I/O subsystem 106 optionally includes one or more input controllers 160 for display controller 156, light sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and other input or control devices. One or more input controllers 160 receive electrical signals from and send electrical signals to other input or control devices 116. Other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, input controller(s) 160 are optionally coupled to any (or none) of a keyboard, an infrared port, a USB port, a stylus, and/or a pointer device such as a mouse. The one or more buttons (e.g., 208, FIG. 2) optionally include up/down buttons for volume control of speaker 111 and/or microphone 113. The one or more buttons optionally include a push button (e.g., 206, FIG. 2).
Touch-sensitive display system 112 provides an input and output interface between the device and a user. Display controller 156 receives electrical signals from and/or transmits electrical signals to touch-sensitive display system 112. Touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects. Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls. In some embodiments, some or all of the visual output corresponds to indicators and visual guides that provide visual cues indicating types of inputs and/or actions associated with various areas of a user interface or screen. Examples of indicators and visual guides include, without limitation, arrows, bars, overlays, spotlights, or other visually identified areas or shapes designed to provide visual cues to a user. As used herein, the term "affordance" refers to indicators and visual guides that are displayed on user-interactive graphical user interface objects and/or a background (e.g., as part of an application user interface or a system user interface).
Touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that receive input from a user based on haptic and/or tactile contact. Touch-sensitive display system 112 and display controller 156 (along with any associated modules and/or instruction sets in memory 102) detect contacts (and any movement or breaking of contact) on touch-sensitive display system 112 and translate the detected contacts into interactions with user interface objects (e.g., one or more soft keys, icons, web pages, or images) that are displayed on touch-sensitive display system 112. In some embodiments, the point of contact between touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.
Touch-sensitive display system 112 optionally uses liquid crystal display (LCD), light emitting polymer display (LPD), or light emitting diode (LED) technology, although other display technologies are used in other embodiments. Touch-sensitive display system 112 and display controller 156 optionally use any of a number of now known or later developed touch sensing technologies to detect contact and any movement or disruption thereof, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 112. In some embodiments, touch-sensitive display system 112 may be used with a touch-sensitive display such as an iPhone, an iPod touch, or a iPod touch from Apple Inc. of Cupertino, California. It uses projected mutual capacitance sensing technology, such as that found in the Touch and iPad.
Touch-sensitive display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen video resolution is greater than 400 dpi (e.g., 500 dpi, 800 dpi, or higher). A user optionally contacts touch-sensitive display system 112 using any suitable object or accessory, such as a stylus, finger, or the like. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger area of finger contacts on the touchscreen. In some embodiments, the device translates the coarse finger-based input into precise pointer/cursor position or commands to perform the user's desired action.
In some embodiments, in addition to a touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, a touchpad is a touch-sensitive area of a device that, unlike a touchscreen, does not display visual output. A touchpad is optionally a touch-sensitive surface separate from touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by a touchscreen.
Device 100 also includes a power system 162 for providing power to the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, power failure detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power within a portable device.
Device 100 also optionally includes one or more light sensors 164. FIG. 1A shows a light sensor coupled to light sensor controller 158 in I/O subsystem 106. Light sensor(s) 164 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. Light sensor(s) 164 receive light from the environment, projected through one or more lenses, and convert the light into data representing an image. In conjunction with imaging module 143 (also called a camera module), light sensor(s) 164 optionally captures still images and/or video. In some embodiments, the light sensor is located on the back of device 100, opposite touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for still and/or video image acquisition. In some embodiments, another light sensor is located on the front of the device so that an image of the user is captured (e.g., for a selfie, for a video conference while the user views other video conference participants on the touch screen, etc.).
Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor connected to intensity sensor controller 159 in I/O subsystem 106. Contact intensity sensor(s) 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor(s) 165 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is located on or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100, opposite touchscreen display system 112, which is located on the front of device 100.
Device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows proximity sensor 166 connected with peripherals interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I/O subsystem 106. In some embodiments, the proximity sensor turns off and disables touch-sensitive display system 112 when the multifunction device is placed near a user's ear (e.g., when the user is conducting a phone call).
Device 100 also optionally includes one or more tactile output generators 167. FIG. 1A shows a tactile output generator connected to haptic feedback controller 161 in I/O subsystem 106. In some embodiments, tactile output generator(s) 167 include one or more electroacoustic devices, such as speakers or other audio components, and/or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). Tactile output generator(s) 167 receive tactile feedback generation instructions from haptic feedback module 133 and generate tactile outputs on device 100 that can be sensed by a user of device 100. In some embodiments, at least one tactile output generator is disposed on or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112) and generates a tactile output, optionally by moving the touch-sensitive surface vertically (e.g., in/out of the surface of device 100) or horizontally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, the at least one tactile output generator sensor is located on the back of device 100, opposite touch-sensitive display system 112, which is located on the front of device 100.
Device 100 also optionally includes one or more accelerometers 168. FIG. 1A shows accelerometer 168 connected with peripherals interface 118. Alternatively, accelerometer 168 is optionally coupled to input controller 160 in I/O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from the one or more accelerometers. In addition to accelerometer(s) 168, device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the location and orientation (e.g., portrait or landscape orientation) of device 100.
In some embodiments, the software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a contact/motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and a GPS module (or instruction set) 136. 1A and 3, memory 102 stores device/global internal state 157. Device/global internal state 157 includes one or more of: active application state indicating which applications, if any, are currently active; display state indicating which applications, views, or other information are occupying various regions of touch-sensitive display system 112; sensor state including information obtained from various sensors of the device and other input or control devices 116; and position and/or orientation information regarding the location and/or orientation of the device.
Operating system 126 (e.g., an embedded operating system such as iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes various software components and/or drivers for controlling and managing overall system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.
The communications module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuitry 108 and/or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) is adapted to couple to other devices directly or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector identical to, or similar and/or compatible with, the 30-pin connector used in some iPhone, iPod Touch, and iPad devices from Apple Inc. of Cupertino, California. In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector identical to, or similar and/or compatible with, the 30-pin connector used in some iPhone, iPod Touch, and iPad devices from Apple Inc. of Cupertino, California. (b) a Lightning connector that is identical to, similar to and/or compatible with the Lightning connector used in Huawei Touch, and iPad devices.
Contact/motion module 130 optionally detects contact with touch-sensitive display system 112 (in cooperation with display controller 156), and with other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact/motion module 130 includes software components for performing various operations related to detecting contact (e.g., by a finger or stylus), such as determining if contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a proxy for the force or pressure of the contact), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining if the contact has stopped (e.g., detecting a finger up event or an interruption of the contact). Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by the set of contact data optionally includes determining the speed (magnitude), velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the contact point. These actions are optionally applied to a single contact (e.g., one finger contact or stylus contact) or multiple simultaneous contacts (e.g., "multi-touch"/multiple finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect contacts on a touchpad.
Contact/motion module 130 optionally detects gesture input by a user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timing, and/or strength of detected contacts). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event, followed by a finger up (lift off) event at the same location (or substantially the same location) as the finger down event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event, followed by one or more finger drag events, followed by a finger up (lift off) event. Similarly, taps, swipes, drags, and other gestures are optionally detected by detecting a particular contact pattern with respect to the stylus.
In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger down event and detecting a finger up event, but is not related to the strength of the finger contact between detecting a finger down event and detecting a finger up event. In some embodiments, a tap gesture is detected according to determining that the length of time between a finger down event and a finger up event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the strength of the finger contact during the tap meets a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a light or deep pressure intensity threshold. Thus, a finger tap gesture may satisfy certain input criteria that do not require that the characteristic intensity of the contact meet a given intensity threshold for the particular input criteria to be met. For clarity, the finger contact in a tap gesture typically needs to meet a nominal contact detection intensity threshold below which the contact is not detected for a finger down event to be detected. A similar analysis applies to detecting tap gestures or other contacts with a stylus. In cases where a device is capable of detecting contact of a finger or stylus hovering over the touch-sensitive surface, the nominal contact-detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.
In a similar manner, the same concepts are applied to other types of gestures. For example, swipe gestures, pinch gestures, de-pinch gestures, and/or long press gestures are optionally detected based on meeting criteria that are either unrelated to the intensity of the contacts included in the gesture or do not require the contact(s) performing the gesture to reach an intensity threshold in order to be recognized. For example, swipe gestures are detected based on the amount of movement of one or more contacts, pinch gestures are detected based on the movement of two or more contacts towards each other, de-pinch gestures are detected based on the movement of two or more contacts away from each other, and long press gestures are detected based on a duration of contact on the touch-sensitive surface that is less than a threshold amount of movement. As such, a statement that a particular gesture recognition criterion does not require that the intensity of the contact(s) meet a respective intensity threshold in order for the particular gesture recognition criterion to be satisfied means that the particular gesture recognition criterion can be satisfied when the contact(s) in the gesture do not reach the respective intensity threshold, and can also be satisfied in situations where one or more of the contacts in the gesture do not reach or exceed the respective intensity threshold. In some embodiments, a tap gesture is detected based on a determination that a finger down event and a finger up event are detected within a predefined time period, regardless of whether the contacts are above or below the respective intensity thresholds during the predefined time period, and a swipe gesture is detected based on a determination that a movement of the contact is greater than a predefined magnitude, even if the contacts exceed the respective intensity thresholds at the end of the movement of the contact. Even in implementations in which detection of gestures is affected by the strength of the contact performing the gesture (e.g., the device detects a long press more quickly when the intensity of the contact exceeds an intensity threshold, or the device is slow to detect a tap input when the intensity of the contact is higher), detection of those gestures does not require that the contact reach a particular intensity threshold, so long as criteria for recognizing the gesture can be met in situations in which the contact does not reach a particular intensity threshold (e.g., even if the amount of time it takes to recognize the gesture varies).
The contact intensity threshold, duration threshold, and movement threshold may be combined in various different combinations to create heuristics for distinguishing between two or more different gestures directed at the same input element or region, thereby enabling multiple different interactions with the same input element and providing a rich set of user interactions and responses. A statement that a particular set of gesture recognition criteria does not require that the intensity of the contact(s) meet the respective intensity thresholds in order for a particular gesture recognition criterion to be satisfied does not preclude the simultaneous evaluation of other intensity-dependent gesture recognition criteria to identify other gestures having criteria that are satisfied when the gesture includes a contact having an intensity above the respective intensity thresholds. For example, in some circumstances, a first gesture recognition criterion for a first gesture that does not require that the intensity of the contact(s) meet the respective intensity thresholds in order for the first gesture recognition criterion to be satisfied is in conflict with a second gesture recognition criterion for a second gesture that is dependent on the contact(s) reaching the respective intensity thresholds. In such a conflict, a gesture is optionally not recognized as satisfying the first gesture recognition criteria for the first gesture if the second gesture recognition criteria for the second gesture are satisfied first. For example, if the contacts reach the respective intensity thresholds before they move the predefined amount of movement, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contacts move the predefined amount of movement before they reach the respective intensity thresholds, a swipe gesture is detected instead of a deep press gesture. Even in such a situation, the first gesture recognition criteria for the first gesture still do not require that the intensity of the contact(s) meet the respective intensity thresholds for the first gesture recognition criteria to be satisfied, because if the contacts remain below the respective intensity thresholds until the end of the gesture (e.g., a swipe gesture with a contact that does not increase in intensity to above the respective intensity thresholds), the gesture will have been recognized by the first gesture recognition criteria as a swipe gesture. In that way, certain gesture recognition criteria that do not require the intensity of the contact(s) to meet a respective intensity threshold in order for the particular gesture recognition criterion to be satisfied may still be dependent on the intensity of the contact with respect to an intensity threshold in the sense that (A) in some circumstances, they may ignore the intensity of the contact with respect to an intensity threshold (e.g., for a tap gesture) and/or (B) in some circumstances, the particular gesture recognition criterion (e.g., for a long press gesture) will not function if a competing set of intensity-dependent gesture recognition criteria (e.g., for a deep press gesture) recognizes an input as corresponding to an intensity-dependent gesture before the particular gesture recognition criterion recognizes a gesture corresponding to the input (e.g., for a long press gesture that competes with a deep press gesture for recognition).
Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other display, including components for varying the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.
In some embodiments, graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 132 receives one or more codes specifying the graphics to be displayed, including coordinate data and other graphic characteristic data, as appropriate, from an application or the like, and then generates screen image data for output to display controller 156.
The haptic feedback module 133 includes various software components for generating instructions (e.g., instructions used by the haptic feedback controller 161) that use the tactile output generator(s) 167 to create a tactile output at one or more locations on the device 100 in response to user interaction with the device 100.
Text input module 134 is optionally a component of graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).
The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing, to the camera 143 as photo/video metadata, and to applications that provide location-based services such as a weather widget, a local yellow pages widget, and a maps/navigation widget).
The application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:
Contacts module 137 (sometimes called an address book or contact list), Telephone module 138, Video conferencing module 139, Email client module 140, Instant messaging (IM) module 141, Training support module 142, Camera module 143 for still and/or video images, Image management module 144, Browser module 147, Calendar module 148, Weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget a widget module 149 optionally including one or more of a widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; a widget creation module 150 for creating the user-created widgets 149-6; a search module 151; a video and music player module 152, optionally consisting of a video player module and a music player module; a notes module 153; a map module 154; and/or an online video module 155.
Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA® enabled applications, encryption, digital rights management, voice recognition, and voice duplication.
In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, contact module 137 includes executable instructions (e.g., stored in memory 102 or in the application internal state 192 of contact module 137 in memory 370) for managing an address book or contact list, including adding a name(s) to the address book, deleting a name(s) from the address book, associating phone number(s), email address(es), address(es), or other information with a name, associating an image with a name, categorizing and sorting names, providing phone numbers and/or email addresses to initiate and/or facilitate communication by telephone 138, video conference 139, email 140, or IM 141, and the like.
In cooperation with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 includes executable instructions for entering a series of characters corresponding to a telephone number, accessing one or more telephone numbers in address book 137, modifying an entered telephone number, dialing each telephone number, conducting a conversation, and disconnecting or hanging up when the conversation is completed. As discussed above, the wireless communication optionally uses any of a number of communication standards, protocols, and technologies.
In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 contains executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with the user's instructions.
In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 contains executable instructions for composing, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to compose and send emails with still or video images captured by camera module 143.
In cooperation with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant message module 141 includes executable instructions for entering a series of characters corresponding to an instant message, modifying previously entered characters, sending each instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, sent and/or received instant messages are optionally forwarded via MMS and/or Enhanced Messaging Service (EMS) protocols. and graphics, photos, audio files, video files, and/or other attachments, such as those supported by the Instant Message Service (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).
In cooperation with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module 152, the training support module 142 includes executable instructions for creating workouts (e.g., having time, distance, and/or calorie consumption goals), communicating with training sensors (in the sports device and the smart watch), receiving training sensor data, calibrating sensors used to monitor the workouts, selecting and playing music for the workouts, and displaying, storing, and transmitting the workout data.
In cooperation with touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 contains executable instructions to capture still images or videos (including video streams), store them in memory 102, modify characteristics of the still images or videos, and/or delete still images or videos from memory 102.
In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 contains executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still and/or video images.
In cooperation with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 contains executable instructions for browsing the Internet in accordance with a user's instructions, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
In cooperation with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 contains executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar items, to-do lists, etc.) according to user instructions.
In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is optionally a mini-application that is downloaded and used by a user (e.g., weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or a mini-application created by a user (e.g., user-created widget 149-6). In some embodiments, widgets are written as HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheet) files, or other markup languages (e.g., HTML, CSS, CSS3, CSS4, CSS5, CSS6, CSS7, CSS8, CSS9, CSS10, CSS11, CSS20, CSS32, CSS43, CSS54, CSS65, CSS76, CSS87, CSS98, CSS99, CSS12, CSS13, CSS14, CSS15, CSS16, CSS17, CSS18, CSS20, CSS32, CSS44, CSS55, CSS66, CSS77, CSS88, CSS99, CSS91, CSS92, CSS93, CSS94, CSS95, CSS96, CSS97, CSS98, CSS99, CSS99, CSS99, CSS99, CSS102, CSS1133, CSS1134, CSS123, CSS124, CSS125, CSS135, CSS136, CSS137, CSS140, CSS141, CSS142, CSS143, CSS Sheets (Cascading Style Sheets) files, and JavaScript files. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).
In cooperation with RF circuitry 108, touch-sensing display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget creation module 150 contains executable instructions for creating widgets (e.g., turning user-specified portions of a web page into widgets).
In cooperation with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 contains executable instructions to search memory 102 for text, music, sound, images, video, and/or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with a user's instructions.
In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 contains executable instructions that enable a user to download and play recorded music or other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions to display, present, or otherwise play videos (e.g., on touch-sensitive display system 112 or on an external display connected wirelessly or via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).
In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, notes module 153 contains executable instructions for creating and managing notes, to-do lists, and the like, in accordance with a user's instructions.
In cooperation with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 contains executable instructions to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data about stores and other points of interest at or near a particular location, and other location-based data) in accordance with a user's instructions.
In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 contains executable instructions that enable a user to access, view, receive (e.g., by streaming and/or downloading), and play (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124) online videos in one or more file formats, such as H.264, and send and otherwise manage emails with links to particular online videos. In some embodiments, instant messaging module 141 is used to send links to particular online videos, rather than email client module 140.
Each of the above-identified modules and applications corresponds to an executable instruction set that performs one or more of the functions described above and methods described in the present application (e.g., the computer-implemented methods and other information processing methods described herein). The modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of the modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 102 optionally stores additional modules and data structures not described above.
In some embodiments, device 100 is a device on which operation of a predefined set of functions is performed exclusively through a touchscreen and/or touchpad. By using the touchscreen and/or touchpad as the primary input control device for operation of device 100, the number of physical input control devices (push buttons, dials, etc.) on device 100 is optionally reduced.
The set of defined functions performed exclusively through the touchscreen and/or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 to a main menu, a home menu, or a root menu from any user interface displayed on device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.
1B is a block diagram illustrating example components for event processing, according to some embodiments. In some embodiments, memory 102 (in FIG. 1A) or 370 (in FIG. 3) includes an event sorter 170 (e.g., in operating system 126) and a respective application 136-1 (e.g., any of applications 136, 137-155, 380-390 described above).
Event sorter 170 receives the event information and determines application 136-1 and application view 191 of application 136-1 to which the event information is to be delivered. Event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192 that indicates the current application view(s) that are displayed on touch-sensitive display system 112 when the application is active or running. In some embodiments, device/global internal state 157 is used by event sorter 170 to determine which application(s) are currently active, and application internal state 192 is used by event sorter 170 to determine application view 191 to which the event information is to be delivered.
In some embodiments, application internal state 192 includes additional information such as one or more of resume information used when application 136-1 resumes execution, user interface state information indicating information being displayed or prepared for display by application 136-1, a state queue to allow the user to return to a previous state or view of application 136-1, and a redo/undo queue of previous actions taken by the user.
Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user's touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I/O subsystem 106 or sensors such as proximity sensor 166, accelerometer(s) 168, and/or microphone 113 (through audio circuitry 110). Information that peripherals interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.
In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and/or for longer than a predetermined period of time).
In some embodiments, the event sorter 170 also includes a hit view determination module 172 and/or an active event recognizer determination module 173 .
Hit view determination module 172 provides software procedures for determining where in one or more views a sub-event occurred when touch-sensitive display system 112 displays more than one view. A view consists of controls and other elements that a user can see on the display.
Another aspect of a user interface associated with an application is the set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective applications) in which the touch is detected optionally correspond to programmatic levels within the programmatic or view hierarchy of the application. For example, the lowest-level view in which the touch is detected is optionally referred to as a hit view, and the set of events that are recognized as suitable inputs is optionally determined based at least in part on the hit view of the initial touch that initiates the touch-based gesture.
Hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process the sub-events. In most situations, the hit view is the lowest level view in which the initiating sub-event occurs (i.e., the first sub-event in a series of sub-events that form an event or potential event). Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.
Active event recognizer determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that contain the physical location of the sub-event are actively participating views, and therefore all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the area associated with a particular view, views higher in the hierarchy remain actively participating views.
Event dispatcher module 174 dispatches the event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores event information obtained by each event receiver module 182 in an event queue.
In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In yet other embodiments, event sorter 170 is a stand-alone module or part of another module stored in memory 102, such as contact/motion module 130.
In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each including instructions for processing touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a higher level object from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handler 190 includes one or more of a data updater 176, an object updater 177, a GUI updater 178, and/or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Also, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.
Each event recognizer 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from the event information. Event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (optionally including sub-event delivery instructions).
The event receiver 182 receives event information from the event sorter 170. The event information includes information about a sub-event, e.g., information about a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information, such as a position of the sub-event. When the sub-event involves a movement of a touch, the event information also optionally includes a speed and a direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape or vice versa), and the event information includes corresponding information about the current orientation of the device (also called the device's attitude).
The event comparator 184 compares the event information to predefined event or sub-event definitions and determines the event or sub-event or determines or updates the state of the event or sub-event based on the comparison. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes a definition of an event (e.g., a predefined set of sub-events), such as, for example, event 1 (187-1), event 2 (187-2), etc. In some embodiments, the sub-events in the event 187 include, for example, touch start, touch end, touch movement, touch stop, and multiple touches. In one example, the definition for event 1 (187-1) is a double tap on a displayed object. The double tap includes, for example, a first touch (touch start) for a given stage on the displayed object, a first lift off (touch end) for a given stage, a second touch (touch start) for a given stage on the displayed object, and a second lift off (touch end) for a given stage. In another example, a definition of event 2 (187-2) is a drag operation on a displayed object. Drag operations include, for example, a touch (or contact) of a given stage on a displayed object, a movement of the touch across touch-sensitive display system 112, and a lift-off of the touch (end of the touch). In some embodiments, the event also includes information about one or more associated event handlers 190.
In some embodiments, event definition 187 includes a definition of the event with respect to a corresponding user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, event comparator 184 uses the results of the hit test to determine which event handler 190 to activate. For example, event comparator 184 selects an event handler associated with the sub-event and object that triggers the hit test.
In some embodiments, the definition for each event 187 also includes a delay action that delays delivery of the event information until it is determined whether a set of sub-events corresponds to the event type of the event recognizer.
If the corresponding event recognizer 180 determines that the sequence of sub-events does not match any of the events in the event definition 186, the corresponding event recognizer 180 enters an event disabled, event failed, or event finished state and thereafter ignores the next sub-event of the touch-based gesture. In this situation, other event recognizers, if any, that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.
In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and/or lists that indicate to actively participating event recognizers how the event delivery system should perform sub-event delivery. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate how event recognizers interact or are enabled to interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate how sub-events are delivered to various levels in a view or programmatic hierarchy.
In some embodiments, each event recognizer 180 activates an event handler 190 associated with an event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is separate from sending (and postponing sending) the sub-events to the respective hit view. In some embodiments, the event recognizer 180 pops a flag associated with the recognized event, and the event handler 190 associated with the flag catches the flag and performs predefined processing.
In some embodiments, the event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with a set of sub-events or to an actively participating view. The event handler associated with the set of sub-events or the actively participating view receives the event information and performs a predetermined process.
In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, data updater 176 updates phone numbers used by contacts module 137 or stores video files used by video and music player module 152. In some embodiments, object updater 177 creates and updates objects used by application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on the touch-sensitive display.
In some embodiments, event handler(s) 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of a respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.
It should be understood that the foregoing discussion regarding event processing of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 100 with input devices, not all of which are initiated on a touch screen. For example, mouse movements and mouse button presses, optionally coordinated with single or multiple keyboard presses or holds, contact movements such as tapping, dragging, scrolling on a touchpad, pen stylus input, device movements, verbal cues, detected eye movements, biometric input, and/or any combination thereof, are optionally utilized as inputs corresponding to sub-events that define the event to be recognized.
2 illustrates portable multifunction device 100 having a touch screen (e.g., touch-sensitive display system 112, FIG. 1A) in accordance with some embodiments. The touch screen optionally displays one or more graphics on a user interface. 100. In these embodiments, as well as embodiments described below, a user can select one or more of the graphics by making a gesture on the graphics, for example with one or more fingers 202 (not drawn to scale) or one or more styluses 203 (not drawn to scale). In some embodiments, selection of one or more of the graphics occurs when a user breaks contact with one or more of the graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, upward and/or downward), and/or rolling of a finger in contact with device 100 (right to left, left to right, upward and/or downward). In some implementations or situations, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture sweeping over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.
Device 100 also optionally includes one or more physical buttons, such as a "home" or menu button 204. As mentioned above, menu button 204 is optionally used to navigate to any application 136 within a set of applications executing on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on a touchscreen display.
In some embodiments, device 100 includes a touch screen display, a menu button 204 (sometimes referred to as a home button 204), push buttons 206 for powering the device on/off and locking the device, volume control button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and an external docking/charging port 124. Push buttons 206 are optionally used to power the device on/off by pressing and holding the button down for a predefined time interval, lock the device by pressing and releasing the button before the predefined time interval has elapsed, and/or unlock the device or initiate an unlocking process. In some embodiments, device 100 also accepts verbal input through microphone 113 to activate or deactivate certain features. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of a contact on touch-sensitive display system 112 and/or one or more tactile output generators 167 for generating a tactile output to a user of device 100.
FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia playback device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 300 includes an input/output (I/O) interface 330 with a display 340, which is typically a touchscreen display. I/O interface 330 also optionally includes a keyboard and/or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 (e.g., similar to tactile output generator(s) 167 described above with reference to FIG. 1A) for generating tactile output on device 300, sensors 359 (e.g., light sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and/or contact intensity sensors similar to contact intensity sensor(s) 165 described above with reference to FIG. 1A). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU(s) 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to, or a subset of, programs, modules, and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1A). Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores drawing module 380, presentation module 382, word processing module 384, website creation module 386, disk authoring module 388, and/or spreadsheet module 390, while memory 102 of portable multifunction device 100 (FIG. 1A) optionally does not store those modules.
Each of the elements in FIG. 3 identified above is optionally stored in one or more of the memory devices mentioned above. Each of the modules identified above corresponds to an instruction set that performs the functions described above. The modules or programs (i.e., instruction sets) identified above need not be implemented as separate software programs, procedures, or modules, and thus various subsets of those modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Additionally, memory 370 optionally stores additional modules and data structures not described above.
Attention is now directed to embodiments of a user interface (UI) that is optionally implemented on portable multifunction device 100.
4A shows an exemplary user interface for a menu of applications on portable multifunction device 100, according to some embodiments. A similar user interface is, optionally, implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:
signal strength indicator(s) for wireless communication(s) such as cellular and Wi-Fi signals; time; a Bluetooth indicator; a battery status indicator; a tray 408 with icons for frequently used applications, such as: an icon 416 for the phone module 138 labeled "Phone", optionally including an indicator 414 of the number of missed calls or voicemail messages; an icon 418 for the email client module 140 labeled "Mail", optionally including an indicator 410 of the number of unread emails; an icon 420 for the browser module 147 labeled "Browser", and an icon 422 for the video and music player module 152 labeled "Music", and icons for other applications, such as: an icon 424 for the IM module 141 labeled "Messages", an icon 426 for the calendar module 152 labeled "Calendar", o icon 426 for image management module 144, labeled "Photos", o icon 428 for image management module 144, labeled "Photos", o icon 430 for camera module 143, labeled "Camera", o icon 432 for online video module 155, labeled "Online Video", o icon 434 for stock prices widget 149-2, labeled "Stock Prices", o icon 436 for map module 154, labeled "Maps", o icon 438 for weather widget 149-1, labeled "Weather", o icon 440 for alarm clock widget 149-4, labeled "Clock", o icon 442 for training support module 142, labeled "Training Support", o icon 444 for notes module 153, labeled "Notes", and o icon 446 for a settings application or module that provides access to settings for device 100 and its various applications 136.
4A are merely examples. For example, other labels are optionally used for the various application icons. In some embodiments, the label for each application icon includes the name of the application that corresponds to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application that corresponds to the particular application icon.
4B shows an example user interface on a device (e.g., device 300, FIG. 3) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355, FIG. 3) separate from display 450. Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 357) for detecting the intensity of a contact on touch-sensitive surface 451 and/or one or more tactile output generators 359 for generating a tactile output to a user of device 300.
Although many of the following examples are given with reference to input on touch screen display 112 (where the touch-sensitive surface and display are combined), in some embodiments the device detects input on a touch-sensitive surface that is separate from the display, as shown in FIG. 4B. In some embodiments, the touch-sensitive surface (e.g., 451 in FIG. 4B) has a major axis (e.g., 452 in FIG. 4B) that corresponds to a major axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with touch-sensitive surface 451 (e.g., 460 and 462 in FIG. 4B) at locations that correspond to respective locations on the display (e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470). In this manner, when the touch-sensitive surface is separate from the display, user input (e.g., contacts 460 and 462, and their movements) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B) is used by the device to operate a user interface on the display (e.g., 450 in FIG. 4B) of the multifunction device. It is to be understood that similar methods are optionally used for the other user interfaces described herein.
In addition, while the following description is primarily described with reference to finger inputs (e.g., finger contacts, finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of the finger inputs are replaced with inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced with a mouse click (e.g., instead of a contact) followed by cursor movement along the path of the swipe (e.g., instead of a contact movement). As another example, a tap gesture is optionally replaced with a mouse click (e.g., instead of a contact detection followed by cessation of contact detection) while the cursor is located over the tap gesture location. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.
As used herein, the term "focus selector" refers to an input element that indicates the current portion of a user interface with which a user is interacting. In implementations that include a cursor or other location marker, the cursor serves as the "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 of FIG. 3 or touch-sensitive surface 451 of FIG. 4B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations that include a touchscreen display (e.g., touch-sensitive display system 112 of FIG. 1A or touchscreen of FIG. 4A) that allows direct interaction with user interface elements on the touchscreen display, a contact detected on the touchscreen serves as the "focus selector" such that when an input (e.g., a press input by contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one region of the user interface to another region of the user interface without corresponding cursor or contact movement on the touchscreen display (e.g., by using a tab key or arrow keys to move focus from one button to another), and in those implementations, the focus selector moves to follow the movement of focus between different regions of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or contact on a touchscreen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device which element of the user interface the user intends to interact with). For example, the position of the focus selector (e.g., cursor, contact, or selection box) over a corresponding button while a press input is detected on the touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate the corresponding button (and not other user interface elements shown on the device's display).
As used herein and in the claims, the term intensity of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or a proxy for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a value range that includes at least four different values, and more typically includes hundreds (e.g., at least 256) different values. The intensity of a contact is optionally determined (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, the force measurements from multiple force sensors are combined (e.g., weighted average or sum) to determine an estimated force of the contact. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and/or change in a contact area detected on the touch-sensitive surface, the capacitance and/or change in the capacitance of the touch-sensitive surface proximate the contact, and/or the resistance and/or change in the capacitance of the touch-sensitive surface proximate the contact, are optionally used as proxies for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the proxy measure for the force or pressure of the contact is used directly to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is described in units corresponding to the proxy measure). In some implementations, the proxy measure for the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of contact as an attribute of user input provides users with access to additional device functionality that they might not otherwise be able to easily access due to the limited area on a reduced-sized device for displaying affordances (e.g., on a touch-sensitive display) and/or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical/mechanical control such as a knob or button).
In some embodiments, the contact/motion module 130 uses a set of one or more intensity thresholds for determining whether an action has been performed by the user (e.g., for determining whether the user has clicked on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of a particular physical actuator and may be adjusted without modifying the physical hardware of the device 100). For example, the mouse click threshold of a trackpad or touchscreen display may be set to any of a wide range of predefined thresholds without modifying the hardware of the trackpad or touchscreen display. Additionally, in some implementations, a user of the device is provided with a software setting for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and/or by adjusting multiple intensity thresholds at once with a system-level click intensity parameter).
As used herein and in the claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on a plurality of intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predefined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) for a predefined event (e.g., after detecting a contact, before detecting lift-off of the contact, before or after detecting the start of movement of the contact, before detecting the end of the contact, before or after detecting an increase in the intensity of the contact, and/or before or after detecting a decrease in the intensity of the contact). The characteristic intensity of a contact is optionally based on a maximum value of the intensity of the contact, a mean value of the intensity of the contact, an average value of the intensity of the contact, a maximum ... The characteristic intensity may be based on one or more of: a top 10% value of the intensity of the contact; a half value of the maximum intensity of the contact; a 90% value of the maximum intensity of the contact; a value generated by low pass filtering the intensity of the contact over a predefined period or beginning at a predefined time; and the like. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is an average over time of the intensity of the contact). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action is performed by the user. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, a first action is performed as a result of a contact having a characteristic intensity that does not exceed the first threshold, a second action is performed as a result of a contact having a characteristic intensity that exceeds the first intensity threshold and does not exceed the second intensity threshold, and a third action is performed as a result of a contact having a characteristic intensity that exceeds the second intensity threshold. In some embodiments, the comparison between the characteristic intensity and one or more intensity thresholds is not used to determine whether to perform a first operation or a second operation, but rather to determine whether to perform one or more operations (e.g., to execute a respective option or to refrain from executing a respective operation).
In some embodiments, a portion of the gesture is identified for purposes of determining the characteristic intensity. For example, a touch-sensitive surface may receive a continuous swipe contact (e.g., a drag gesture) that transitions from a start location to an end location where the intensity of the contact increases. In this example, the characteristic intensity of the contact at the end location may be based on only a portion of the swipe contact (e.g., only the portion at the end location of the swipe contact) rather than the entire continuous swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact prior to determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. In some circumstances, these smoothing algorithms filter out small increases or decreases in the intensity of the swipe contact for purposes of determining the characteristic intensity.
The user interface diagrams described herein optionally include one or more intensity thresholds (e.g., a contact detection intensity threshold IT<sub>0</sub>, Light pressure intensity threshold IT<sub>L</sub>, deep pressure intensity threshold IT<sub>D</sub>(For example, at least initially,<sub>L</sub>higher than 0.01, and/or one or more other intensity thresholds (e.g., IT<sub>L</sub>An intensity threshold lower than IT<sub>H</sub>)) showing the current intensity of contacts on the touch-sensitive surface. The intensity diagrams are typically not part of the displayed user interface, but are provided to aid in interpretation of the diagrams. In some embodiments, the light pressure intensity threshold corresponds to an intensity at which the device performs an action typically associated with a physical mouse button or a trackpad click. In some embodiments, the deep pressure intensity threshold corresponds to an intensity at which the device performs an action different from an action typically associated with a physical mouse button or a trackpad click. In some embodiments, a slight contact detection intensity threshold IT<sub>0</sub>If a contact is detected with a characteristic intensity (greater than or equal to the light pressure intensity threshold), the device moves the focus selector according to the movement of the contact on the touch-sensitive surface without performing any action associated with the light pressure intensity threshold or the deep pressure intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different user interface sets of values.
In some embodiments, the device's response to an input detected by the device depends on criteria based on the intensity of the contact during the input. For example, for some "light press" inputs, the intensity of the contact during the input that exceeds a first intensity threshold triggers a first response. In some embodiments, the device's response to an input detected by the device depends on criteria that include both the intensity of the contact during the input and criteria based on time. For example, for some "deep press" inputs, the intensity of the contact during the input that exceeds a second intensity threshold that is greater than the first intensity threshold for a light press triggers a second response only if a delay time has elapsed between meeting the first intensity threshold and meeting the second intensity threshold. This delay time is typically less than 200 ms (milliseconds) in time (e.g., 40 ms, 100 ms, or 120 ms depending on the magnitude of the second intensity threshold, with the delay time increasing as the second intensity threshold increases). This delay time helps to avoid accidental recognition of a deep press input. As another example, for some "deep press" inputs, there is a time period of reduced sensitivity that occurs after the time the first intensity threshold is met. During the period of time when the sensitivity is reduced, the second intensity threshold is increased. This temporary increase in the second intensity threshold also helps to avoid accidental deep pressure inputs. For other deep pressure inputs, the response to the detection of a deep pressure input does not depend on a time-based criterion.
In some embodiments, one or more of the input intensity thresholds and/or corresponding outputs vary based on one or more factors, such as user settings, contact movement, input timing, running application, speed at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), position of the focus selector, etc. Exemplary factors are described in U.S. Patent Application Nos. 14/399,606 and 14/624,296, which are incorporated by reference herein in their entireties.
For example, in some embodiments, the dynamic intensity threshold varies over time based in part on the intensity of the touch input over time. The dynamic intensity threshold is the sum of two components: a first component that decays over time after a predefined delay time p1 from when the touch input is first detected, and a second component that tracks the intensity of the touch input over time. An initial high intensity threshold for the first component reduces accidental triggering of a "deep press" response while still allowing an immediate "deep press" response if the touch input provides sufficient intensity. The second component reduces unintentional triggering of a "deep press" response due to gradual intensity variations in the touch input. In some embodiments, the "deep press" response is triggered once the touch input meets the dynamic intensity threshold.
In another example, in some embodiments, a dynamic intensity threshold (e.g., intensity threshold I<sub>D</sub>) is the sum of two other intensity thresholds, namely the first intensity threshold IT<sub>H</sub>and a second intensity threshold I<sub>L</sub>In some embodiments, at or before time p2, the touch input exceeds the first intensity threshold IT<sub>H</sub>and a second intensity threshold I<sub>L</sub>is satisfied, but no response is provided until the delay time p2 has elapsed. In some embodiments, the dynamic intensity threshold is a second intensity threshold IT<sub>L</sub>This type of dynamic intensity threshold decays over time with the decay starting at a defined delay time p1 from when the response associated with the first intensity threshold IT<sub>H</sub>or the second intensity threshold I<sub>L</sub>Immediately after or simultaneously with the triggering of a response associated with a lower intensity threshold, such as a dynamic intensity threshold I<sub>D</sub>and reducing accidental triggering of the response associated with the
In another example of some embodiments, the intensity threshold IT<sub>L</sub>The response associated with is triggered after a delay time p2 has elapsed from when the touch input is first detected. At the same time, the dynamic intensity threshold (e.g., the intensity threshold I<sub>D</sub>) decays after a predefined delay p1 from when the touch input is first detected. Thus, the intensity threshold I<sub>L</sub>If the intensity of the touch input is decreased after triggering a response associated with, and then the intensity of the touch input is increased without removing the touch input, then the intensity of the touch input exceeds another intensity threshold, e.g., intensity threshold I<sub>L</sub>Even when the intensity threshold IT<sub>D</sub>A response associated with the request can be triggered.
Light pressure intensity threshold IT<sub>L</sub>From the intensity below the light pressure intensity threshold IT<sub>L</sub>and deep pressure intensity threshold IT<sub>D</sub>An increase in the characteristic intensity of a contact to an intensity between is sometimes referred to as a "light press" input.<sub>D</sub>From the intensity below the threshold IT<sub>D</sub>An increase in the characteristic intensity of a contact to an intensity above the contact-detection intensity threshold I<sub>0</sub>From the intensity below the contact detection intensity threshold IT<sub>0</sub>and light pressure intensity threshold IT<sub>L</sub>An increase in the characteristic intensity of the contact to an intensity between 0 and 1 may be referred to as detecting a contact on the touch surface.<sub>0</sub>From the intensity exceeding the contact detection intensity threshold IT<sub>0</sub>A decrease in the characteristic intensity of the contact to an intensity below 0 may be referred to as detecting lift-off of the contact from the touch surface.<sub>0</sub>is zero. In some embodiments, IT<sub>0</sub>is greater than zero. In some figures, shaded circles or ellipses are used to represent the intensity of a contact on the touch-sensitive surface. In some figures, unshaded circles or ellipses are used to represent each contact on the touch-sensitive surface without specifying the intensity of each contact.
In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a respective press input or in response to detecting a respective press input performed on a respective contact (or contacts), where the respective press input is detected based at least in part on detecting an increase in intensity of the contact (or contacts) above a press input intensity threshold. In some embodiments, the respective operations are performed in response to detecting an increase in intensity of the respective contact above the press input intensity threshold (e.g., the respective operations are performed on a "downstroke" of the respective press input). In some embodiments, the press input includes an increase in intensity of the respective contact above the press input intensity threshold and a subsequent decrease in intensity of the contact below the press input intensity threshold, where the respective operations are performed in response to detecting a subsequent decrease in intensity of the respective contact below the press input intensity threshold (e.g., the respective operations are performed on an "upstroke" of the respective press input).
In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, sometimes referred to as "jitter," and the device defines or selects a hysteresis intensity threshold that has a predefined relationship to the pressure input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units below the pressure input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the pressure input intensity threshold). Thus, in some embodiments, the pressure input includes an increase in the intensity of the respective contact above the pressure input intensity threshold, followed by a decrease in the intensity of the contact below a hysteresis intensity threshold that corresponds to the pressure input intensity threshold, and a respective action is performed in response to detecting the decrease in the intensity of the respective contact below the hysteresis intensity threshold (e.g., a respective action is performed by an "upstroke" of a respective pressure input). Similarly, in some embodiments, a pressure input is detected only when the device detects an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressure input intensity threshold, and optionally a subsequent decrease in the intensity of the contact to an intensity below the hysteresis intensity, and a respective action is performed in response to detection of that pressure input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact, as the case may be).
For ease of explanation, the description of actions performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting an increase in the intensity of the contact above the press input intensity threshold, an increase in the intensity of the contact from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, a decrease in the intensity of the contact below the press input intensity threshold, or a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to the press input intensity threshold. In addition, in examples described as performing actions in response to detecting a decrease in the intensity of the contact below a press input intensity threshold, the action is optionally performed in response to detecting a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to and lower than the press input intensity threshold. As mentioned above, in some embodiments, the triggering of those responses also depends on a time-based criterion being met (e.g., a delay time has elapsed between the first intensity threshold being met and the second intensity threshold being met).
User Interface and Related Processes
Attention now turns to embodiments of user interfaces ("UI") and associated processes that may be implemented on an electronic device, such as portable multifunction device 100 or device 300, having a display, a touch-sensitive surface, and (optionally) one or more sensors for detecting the intensity of contact with the touch-sensitive surface.
5A-5D illustrate example user interfaces having affordances (e.g., home affordances) indicating gesture initiation regions on a touch-sensitive display screen for navigation gestures (e.g., gestures to navigate to a home screen user interface), according to some embodiments. In some embodiments, affordances for controlling or providing guidance regarding other functions or operations of the device using the methods described herein are also displayed. The user interfaces in those figures are used to illustrate the processes described below, including those in FIGS. 6A-6C, 7A-7E, and 8A-8F. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., a centroid of a respective contact or a point associated with a respective contact), or a centroid of two or more contacts detected on touch-sensitive display system 112.
For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device that does not have a home button, and gestures that meet predefined criteria are used to trigger dismissal of the currently displayed user interface and display of a home screen user interface. Although not shown in Figures 5A-5D, in some embodiments, a home button (e.g., a mechanical button, a solid-state button, or a virtual button) is included on the device and is used to trigger dismissal of the currently displayed user interface and display of a home screen user interface (e.g., in response to a single press input) and/or display of a multitasking user interface (e.g., in response to a double press input).
In Figures 5A-5D, while the device is displaying any user interface, a gesture beginning at the bottom of the screen (e.g., within a defined area of the device proximate an edge region of the display (e.g., an edge region that includes a defined portion of the display near the bottom edge of the device (e.g., 20 pixels wide)) triggers a user interface navigation process that, optionally, directs navigation between multiple user interfaces based on the speed and direction of the input, and optionally based on movement parameters and characteristics of the currently displayed user interface object (e.g., a scaled-down view of the currently displayed user interface).
An exemplary user interface for an application operating on an electronic device includes an affordance (e.g., home affordance 5002) that provides visual guidance to the user as to the location of an edge region where a navigation gesture may be initiated, and, optionally, whether navigation is limited in the current operational mode of the currently displayed application (e.g., the absence of a home affordance indicates that navigation is limited and a confirming input or an extended navigation gesture (e.g., a press input followed by an up swipe, or a touch and hold input followed by an up swipe) is required to navigate through the user interface (as shown in FIGS. 5X-5AA, an initial input is required to redisplay the affordance, at which point subsequent navigation gestures may be recognized). In some embodiments, the home affordance is not activatable or does not respond directly to touch input, e.g., in a manner similar to a virtual button. In some embodiments, the home affordance or another affordance realized using the methods described herein responds to touch input, including direct contact on the affordance.
5A illustrates a web browsing user interface showing the contents of a web page 5004. A home affordance 5002 is displayed overlaying a portion of the displayed content near the bottom edge of the touch screen 112. The navigation process of the user interface is initiated by a contact 5006 that begins below, above, or near the home affordance 5002 and moves upward from the bottom of the screen, as shown in FIGS.
In Figure 5B, the web browsing user interface is replaced with a card 5008 representing the web browser user interface of Figure 5A. As the input moves upwards on the screen, in Figures 5A-5C, the card 5008 dynamically shrinks, revealing a blurred view 5010 of the home screen in the background. In some embodiments, the amount of blur applied to the home screen changes dynamically depending on the distance of the contact 5006 from the bottom of the screen.
In some embodiments, as shown in FIG. 5B , when a user interface navigation process is initiated by input through contacts 5006, home affordances 5002 cease displaying (e.g., home affordances 5006 are not directly operable by touch input and cease displaying when their purpose of providing visual guidance to the user regarding navigation gestures is achieved). If the size of card 5008 is small enough, other cards (e.g., cards 5012 and 5014, representing a system control panel user interface and a recently opened application user interface, respectively) are displayed next to card 5008. Optionally, if multiple cards are displayed, termination of input (e.g., lift-off of contacts 5006) causes the device to enter a multitasking mode (e.g., displaying an application switching user interface that allows the user to select an application, replacing the web browser application as the foreground application).
In FIG. 5C , when the contact 5006 continues to move upward and the predefined home navigation criteria are met (e.g., when a predefined characteristic (e.g., location, velocity, etc.) of the contact 5006 meets a predefined threshold), the other cards on the screen stop displaying and only the card 5008 remains on the screen. If input ceases (e.g., lift-off of the contact 5006 is detected) while only the card 5008 is displayed (e.g., a user interface state similar to FIG. 5C ), the device displays the home screen 5016 as shown in FIG. 5D (e.g., with a dashed oval indicating the lift-off location of the contact 5006). When the home screen 6016 is displayed on the touch screen 112, the home affordances 5002 are not displayed on the touch screen.
Since in many contexts (when displaying a different application or other system-level user interface (e.g., a notification center user interface, a cover sheet user interface, a control panel user interface, etc.)) it is necessary to dismiss the currently displayed user interface and go to the home screen, the home affordance 5002 must be displayed on any kind of background, which may also change spontaneously or in response to user actions, and instantaneously and over time. In addition, the user is given a large degree of freedom to decide where to start a navigation gesture along the bottom edge region of the touch screen, and therefore the home affordance 5002 is designed to have a large horizontal span to indicate the extent of the responsive area for the gesture, and a relatively small width to avoid over-cluttering the screen and unnecessarily distracting the user. As a result, even if the underlying content is stationary in time, the portion of the content underneath the home affordance 5002 may include variations in color and luminance, as well as other display characteristics, in different parts of the portion of the content. Thus, in some embodiments, the appearance of each sub-region of an affordance (e.g., each pixel or small cluster of pixels) is determined separately based on the appearance of the content directly beneath the sub-region of the affordance (and, optionally, the appearance of the content that extends slightly outside the boundaries of each sub-region of the affordance (e.g., via a blurring or averaging effect applied to the content or affordance)). As shown in Figure 5A, a home affordance 5002 is displayed over a portion of content 5004 in a webpage. That portion of content includes regions with different luminance levels, so that home affordance 5002 also includes luminance variations along its length (e.g., horizontal extent).
In some embodiments, as shown in FIG. 5E, multiple image processing filters are applied (e.g., sequentially or without restrictions on the order of filters) to the background content underlying the affordance to determine the appearance of the affordance. For example, an original full-color image of the content is desaturated to obtain a luminance map of the content. The luminance of the content is inverted (e.g., according to a predefined inversion relationship between the luminance value of the background and the luminance value of the affordance (e.g., one of the inversion relationships shown in FIG. 5F, 5R, 5AE, etc.)) to determine the luminance value of the affordance at each pixel of the affordance. The inversion relationship between the luminance of the affordance and the luminance of the underlying content is used as an example of a correlation between values of selected display characteristics of the affordance and the underlying content. Other types of display characteristics, such as gray values or luminance variations, may be used in various embodiments.
As shown in FIG. 5E, inversion creates an appearance contrast between the affordance and the underlying content. If a portion of the underlying content is lighter (e.g., has a higher luminance value), the corresponding portion of the affordance will be darker (e.g., has a lower luminance value). For example, inversion performed on different portions of a non-saturated background content with different luminance values (e.g., the portions enclosed by circles labeled 1, 2, 3, and 4 in the non-saturated strip) will result in corresponding portions of the affordance with different luminance values (e.g., the portions enclosed by circles labeled 1, 2, 3, and 4 in the inverted strip). FIG. 5F shows an example of an inversion curve used to generate affordance luminance values from corresponding background luminance values. The values of the corresponding portions of the background and affordance labeled with circles in FIG. 5E are also marked in FIG. 7F with circles labeled 1, 2, 3, and 4. The shading of the affordance is also reproduced in FIG. 7F with circles labeled 1, 2, 3, and 4. In some embodiments, after inversion of the implementation, thresholding is performed on the luminance values to reduce the dynamic range of the luminance values. For example, the luminance value of each pixel of the affordance is capped at 50% of the maximum luminance of the affordance to create a more subdued appearance with lower internal visual contrast (e.g., comparing the inverted affordance with the thresholded affordance). In some embodiments, to further reduce the internal variation and contrast within the affordance, a blur filter is applied to average the luminance variation across adjacent pixels in the content, thereby averaging the luminance variation across adjacent pixels in the affordance. Ultimately, the resulting affordance has broad stroke variations in luminance that correspond to the luminance variations in the underlying content.
When generating an affordance that stands out against a different background, a simple inversion of the luminance value creates sufficient contrast in most cases, but using the full luminance value range for the affordance would often result in a harsh appearance that could distract the user. Therefore, it is advantageous to constrain the luminance value range of the affordance to a subrange of the luminance value range of the content. In addition, depending on the expected luminance level of the underlying content, the value of the luminance range of the affordance is constrained to the "dark" affordance value range or the "light" affordance value range to generate either a "dark" or a "light" affordance. In some embodiments, after the affordance is initially displayed, the affordance appearance type (e.g., "dark" vs. "light") does not change even if the appearance of the underlying content changes from very dark to very light, and vice versa (as shown in Figures 5G-5P). In some embodiments, the affordance appearance type (e.g., "dark" vs. "light") does not change in response to an instantaneous change in content (e.g., a temporary reversal of the content's luminance level on a short time scale), but eventually changes in response to a more sustained change in content (e.g., a reversal of the content's luminance level over a longer time scale). In some embodiments, the affordance appearance type (e.g., "light" or "dark", or the value range of a particular appearance of an affordance) is selected according to the initial luminance level of the underlying content when the affordance is first displayed, and the affordance maintains that affordance appearance type until a context-switching event occurs (e.g., switching between applications, switching between applications or system user interfaces, or switching between two system user interfaces, etc.), at which point the affordance appearance type is redetermined based on the underlying content in the new context. As an example, FIG. 5R illustrates an example of an inversion relationship for a light affordance and a dark affordance (e.g., the curves are labeled LA and DA, respectively), where the content luminance value range (e.g., values along the horizontal axis) is the full range from black to white (e.g., [0, 1] for gray values in a grayscale image, [0, 255] for luminance values in a color image, or [0, 100%] for luminance or other analog display characteristics), and the affordance luminance value range (e.g., values along the vertical axis) is constrained to either an upper value range (e.g., the value range for the light affordance) or a lower value range (e.g., the value range for the dark affordance). In some embodiments, the two value ranges do not overlap (e.g., are separated by a gap). As shown in FIG. 5R, both curves (e.g., curve LA and curve DA) show that an increase in the luminance of the content leads to a decrease in the luminance of the affordance.
Figure 5F illustrates an example of a luminance inversion curve 5017 for performing the inversion shown in Figure 5E, according to some embodiments. In this example, the affordance value range constrained between upper and lower limits is more than half the value range of the background and does not include pure black (value = 0) and pure white (e.g., value = 1). In some embodiments, the inversion curve is continuous and does not include a continuum.
5G-5K illustrate changing the appearance of an affordance (e.g., affordance 5002-DA) of a first affordance appearance type (e.g., a "dark" affordance type), according to some embodiments. FIGS. 5G-5K illustrate scrolling of content 5018 shown in a web browser user interface displayed on touch screen 112. As content 5018 scrolls, a portion of the content underlying affordance 5002-DA, displayed near the bottom edge of the touch screen, changes. In other words, a different portion of content 5018 moves below affordance 5002-DA as content 5018 scrolls.
As shown in FIG. 5G, in some embodiments, when affordance 5002-DA is first displayed (e.g., when a web browser application is opened and the web browser interface is first displayed with content 5018), the overall luminance level of the portion of content 5018 is evaluated and a suitable affordance appearance type is selected for the affordance. In this particular example, the portion of content 5018 underlying the affordance is relatively dark (e.g., the overall luminance level is below a predefined luminance threshold), and an affordance appearance type (e.g., the "dark" affordance appearance type) that corresponds to the "darker" portion of the affordance luminance value range (e.g., the range [0, 0.4]) is selected as the affordance appearance type for the affordance. The affordance is generated using a set of filters, e.g., as shown in FIG. 5E, and the inversion relationship used is that of the dark affordance appearance type (e.g., the curve DA shown in FIG. 5R).
In FIG. 5H, a scroll input (e.g., movement of contact 5020 in an upward direction across the touchscreen) is detected on the touchscreen that is initiated outside (e.g., upward) of a responsive area of a user interface navigation gesture (e.g., a home/multitasking gesture by contact 5006, as shown in FIGS. 5A-5D). The scroll input causes webpage content 5018 to scroll upward, revealing a previously undisplayed portion of content 5018 beneath affordance 5002-DA. At the instant shown in FIG. 5H, the portion of content 5018 directly underlying affordance 5002-DA is completely white (e.g., has a luminance value of 1 or 100%), and correspondingly, the luminance value of affordance 5002-DA is completely black (e.g., has a luminance value of 0 or 0%), as determined based on the inversion relationship for the dark affordance appearance type (e.g., as shown by curve DA in FIG. 5R).
5I-5K show that as the scroll input continues (e.g., upward movement of contact 5020 followed by lift-off at terminal velocity of contact 5020), content 5018 scrolls upward beneath affordance 5002-DA. The appearance of affordance 5002-DA changes depending on the portion of content 5002-DA that is currently underlying affordance 5002-DA. The luminance of affordance 5002-DA is determined based on an inversion relationship for the dark affordance appearance type (e.g., as shown by curve DA in FIG. 5R).
In particular, at the moment shown in FIG. 5J, the portion of the content 5018 directly underlying affordance 5002-DA is completely white (e.g., a luminance value of 1 or 100%) on the left side and completely black (e.g., a luminance value of 0 or 0%) on the right side, and correspondingly, half of the left affordance 5002-DA is completely black (consistent with the case shown in FIG. 5H) and half of the right affordance 5002-DA is not completely white. Instead, the right half of affordance 5002-DA is gray (e.g., has an affordance luminance value greater than 0 (e.g., 0.4)) as determined based on an inversion relationship for the dark affordance appearance type (e.g., as shown by curve DA in FIG. 5R). In other words, the luminance value range of the dark affordance (e.g., 5002-DA) is constrained to a range below a maximum luminance threshold (e.g., 0.4).
5L-5P illustrate changing the appearance of an affordance of a second affordance appearance type (eg, a "light" affordance type) according to some embodiments.
Figures 5L-5P show scrolling of content 5018 shown in a web browser user interface displayed on touch screen 112. This scrolling is the reverse of the scrolling shown in Figures 5G-5K.
As shown in FIG. 5L, in some embodiments, when affordance 5002-LA is first displayed (e.g., when a web browser application is opened and the web browser interface is first displayed with content 5018), the overall luminance level of the portion of content 5018 is evaluated and a suitable affordance appearance type is selected for the affordance. In this particular example, the portion of content 5018 underlying the affordance is relatively bright (e.g., the overall luminance level exceeds a predefined luminance threshold), so an affordance appearance type (e.g., the "light" affordance appearance type) that corresponds to the "lighter" portion of the affordance luminance value range (e.g., the range [0.6, 1]) is selected as the affordance appearance type for the affordance. The affordance is generated using a set of filters, e.g., as shown in FIG. 5E, and the inverse relationship used is that of a light affordance (e.g., the curve LA shown in FIG. 5R).
5L-5M, a scroll input (e.g., movement of contact 5022 downward across the touchscreen) is detected on the touchscreen that is initiated outside (e.g., upward) of a responsive area for a user interface navigation gesture (e.g., a home/multitasking gesture with contact 5006 as shown in FIGS. 5A-5D). The scroll input causes webpage content 5018 to scroll downward, revealing a previously undisplayed upper portion of content 5018 below affordance 5002-LA.
At the moment shown in Figure 5M, the portion of content 5018 directly underlying affordance 5002-LA is completely white (e.g., a luminance value of 1 or 100%) on the left side and completely black (e.g., a luminance value of 0 or 0%) on the right side, such that correspondingly, half of the right-side affordance 5002-LA is completely white and half of the left-side affordance 5002-LA is not completely black. Instead, the left half of affordance 5002-LA is gray (e.g., has a luminance value greater than 0 (e.g., 0.6)), as determined based on an inversion relationship for the light affordance appearance type (e.g., as shown by curve LA in Figure 5R). In other words, the affordance luminance value range of the light affordance is constrained to a range above a minimum luminance threshold (e.g., 0.6). In some embodiments, as shown in Figures 5J and 5M, against the same background, the overall appearance of dark affordance 5002-DA is darker than the overall appearance of light affordance 5002-LA.
5N-5P show that as the scroll input continues (e.g., downward movement of contact 5022 followed by lift-off of contact 5022 at a terminal velocity), content 5018 scrolls downward beneath affordance 5002-LA. The appearance of affordance 5002-LA changes depending on the portion of content 5002-DA that is currently underlying affordance 5002-LA. The luminance of affordance 5002-LA is determined based on an inversion relationship for the light affordance appearance type (e.g., as shown by curve LA in FIG. 5R).
At the moment shown in Figure 5O, the portion of content 5018 directly underlying affordance 5002-LA is completely white (e.g., has a luminance value of 1 or 100%), and correspondingly, the luminance value of affordance 5002-LA is not completely black. Instead, affordance 5002-LA is gray (e.g., has a luminance value greater than 0 (e.g., 0.6)) as determined based on the inversion relationship for the light affordance appearance type (e.g., as shown by curve LA in Figure 5R). In other words, the luminance value range of the light affordance is constrained to a range above a minimum luminance threshold (e.g., 0.6).
FIG. 5Q illustrates the difference in appearance of affordance 5002 for two types of affordance appearance types (e.g., LA and DA) assuming the same change in background (e.g., content 5018), in accordance with some embodiments.
Figure 5Q shows a summary of the appearance of affordance 5002 in each of the states shown in Figures 5G-5P. The states are categorized into five groups, each corresponding to a respective state of content 5018 shown in the web browser user interface. For example, from top to bottom, the five groups correspond to (i) Figures 5G and 5P, (ii) Figures 5H and 5O, (iii) Figures 5I and 5N, (iv) Figures 5J and 5M, and (v) Figures 5K and 5L.
As shown in FIG. 5Q, for each group corresponding to a corresponding content state, affordance 5002 has an overall darker appearance (lower overall luminance) for the dark affordance appearance type than for the light affordance appearance type (e.g., comparing the DA and LA examples of affordances in the same content strip).
FIG. 5R illustrates value ranges and inverse relationships between display characteristics of affordances (e.g., luminance or gray value) and underlying content for dark and light affordance appearance types, according to some embodiments.
The difference in appearance shown in FIG. 5Q is also reflected in FIG. 5R, where the affordance luminance value range for the light affordance is entirely above the affordance luminance value range for the dark affordance, and the two values optionally do not overlap.
5S-5AA illustrate user interfaces including affordances having appearances responsive to changes in background appearance and changes in operational modes associated with the background user interface, according to some embodiments.
In Figure 5S, a web browser application is launched and a web browser user interface 5024 is displayed on the touchscreen. In this example, the web browser user interface is displayed in a landscape orientation depending on the orientation of device 100. In a first state (e.g., fully visible/high contrast state 5002-A), home affordance 5002 is displayed near the bottom edge of the touchscreen. An affordance appearance type for affordance 5002 is, optionally, selected based on an initial overall luminance level of a portion of the web page content underlying affordance 5002.
Figures 5S to 5T show media items (e.g., the movie clip "Live 5T shows that a media item ("Bright") has been selected for playback (a tap input by contact 5026 on the play icon associated with the media item in FIG. 5S). In response to the selection of the media item, a media player application is launched and a user interface of the media player application (e.g., user interface 5028) is presented on the touch screen. In FIG. 5T, the media player application is operating in a first mode (e.g., a full screen mode with displayed controls, or an interactive mode). When media playback has just begun, the user interface 5028 includes a number of control areas overlaying the media playback area (e.g., media content that takes up substantially the entire screen), a control area that includes various controls such as a media scrubber, a "Done" button for closing the media player application and returning to the web browser application user interface 5024, a volume slider control, a rewind control, a pause/play control, and a fast forward control. These controls are initially displayed over the media playback area because a user is more likely to want to first see how the media content looks and sounds before adjusting the default starting position or volume selected by the device or returning to a previous application. In some embodiments, when switching from the web browser user interface 5024 to the media player user interface 5028, a context switch event is registered by the device and the affordance appearance type of the affordance 5002 is redetermined based on the initial overall luminance level of the portion of the media content underlying the affordance 5002 when media playback was first initiated. Regardless of whether the affordance appearance type is redetermined, the affordance 5002 is initially displayed fully visible (e.g., in a high contrast state) on the user interface 5028.
FIG. 5U shows that for a first predetermined time (e.g., 10 seconds) after the start of media playback, the control region remains visible over the media content and the affordance 5002 remains fully visible (e.g., full contrast) over the user interface 5028. During this time period, the appearance of the affordance 5002 is determined according to a first set of rules. In some embodiments, the first set of rules includes a set of filters as shown in FIG. 5E and an inversion relationship as shown in FIG. 5R, where a first set of parameters for the filters and/or the shape(s) of the inversion curve are optionally preselected for a first state of the affordance. In FIGS. 5S-5U, affordance 5002-A is a gray affordance with no color information, even though the underlying content is full color.
In FIG. 5V, a first predetermined time (e.g., 10 seconds) has expired after media playback has begun. In response to the expiration of the first predetermined time, the control area ceases to be displayed over the media content. The fading of the control area may occur instantly or over a short animation. The expiration of the first predetermined time and/or the cessation of display of the control area on the media player user interface indicates that the media player user interface is operating in a second mode (e.g., a full screen display mode or a media consumption mode with no controls displayed). Additionally, in response to the expiration of the first predetermined time, the device determines the appearance of the affordance 5002 according to a second set of rules that are different from the first set of rules. In other words, the affordance is displayed in a second state (semi-visible/low contrast state 5002-A'). In some embodiments, the second set of rules includes a set of filters as shown in FIG. 5E and an inversion relationship as shown in FIG. 5R, where a second set of parameters for the filters and/or the shape(s) of the inversion curve are optionally preselected for a second state of the affordance 5002 (e.g., semi-visible/low contrast state 5002-A'). In some embodiments, in the low contrast state, the affordance 5002 preserves some color of the underlying content. For example, instead of fully desaturating the underlying content to obtain a luminance map of the content, only 70% of the color values (e.g., RGB values) are desaturated for each pixel, and 30% of the color information for each pixel in the underlying content is maintained in the final appearance of the affordance. In some embodiments, the transparency level of the affordance is adjusted so that the affordance is not fully opaque, and some color information of the underlying content is transferred to the pixels of the affordance. In some embodiments, after performing the luminance inversion, the color saturation of the remainder of the affordance is increased (e.g., by 30%) to make the affordance appear slightly more vibrant and blend better with the background. The reduced visibility or contrast for the second state of affordance 5002 takes into account that the user is less likely to interact with any controls after an initial period of time after media playback is first initiated. In some embodiments, the transition from the first state to the second state is a gradual, continuous transition, optionally across multiple intermediate states between the first and second states, as opposed to an abrupt, discontinuous transition. A gradual transition is less likely to distract the user from viewing the media content.
As shown in Figures 5V-5W, while affordance 5002 is in a second state (e.g., semi-visible/low-contrast state 5002-A'), the appearance of affordance 5002 changes in response to changes in the content underlying the affordance based on a second set of rules.
In Figure 5X, after affordance 5002 is in the second state for a second predetermined time (e.g., 5 seconds), affordance 5002 transitions from the second state to an invisible state (e.g., referred to as the third state), or in other words, the affordance fades out completely and ceases to display over the media content. In some embodiments, the transition from the second state to the third state is a gradual, continuous transition, optionally across multiple intermediate states between the second and third states, as opposed to an abrupt, discontinuous transition. A gradual transition is less likely to distract the user from viewing the media content.
5Y-5AA show that after affordance 5002 disappears over media content 5028, media playback continues until an input is detected (e.g., movement of the device, a tap or swipe input with a contact on a touchscreen, contact near the bottom edge region of the display, etc.) In response to detecting the input, affordance 5002 reappears over media content 5028.
As shown in FIG. 5Y, in response to movement of the device 100 (or other type of input, such as a tap, swipe, or touchdown of a contact near the bottom edge region of the display), the affordance 5002 is redisplayed (e.g., along with the other control regions) over the media content 5028 in a second state (e.g., semi-visible/low contrast state 5002-A'). In some embodiments, the control regions are not redisplayed in response to this input, and the device continues to operate in a full-screen display mode of the media player application without displaying the controls. In some embodiments, the affordances again cease to redisplay if another input (e.g., a tap input or a swipe input) is not detected within a threshold time. If the required input is detected within the threshold time, the affordances are redisplayed in the first state, and optionally the media control regions are also redisplayed. The user interface is returned to the first operating mode of the media player application (e.g., full-screen playback mode with displayed controls, or interactive mode). Providing affordance 5002 in a second state initially in response to a first input, as opposed to displaying affordance 5002 in a first state and immediately returning to the first operational mode, provides some indication to the user regarding the location of the Home affordances and the state of the user interface, but at the same time takes into account that the input may be unintentional and the user does not really want to divert attention from viewing the media content. If the user's intent is to use the control and/or affordance 5002, a confirmatory input from the user (e.g., a continued touch of the same contact for a threshold time, a press input by the same contact with a threshold pressure intensity, or a second tap input by a different contact) is required.
5Z and 5AA illustrate that, in some embodiments, input by a contact (e.g., contact 5030) is detected on the touch screen while affordance 5002 is not displayed (as shown in FIG. 5X) or while affordance 5002 is displayed in a second state (e.g., semi-transparent/low contrast state 5002-A') (as shown in FIG. 5Y). In response to contact 5030 input, affordance 5002 is displayed in the second state (e.g., semi-visible/low contrast state 5002-A') (if not already displayed). In addition, the media control area is also redisplayed on the media player user interface. In other words, the media player application has returned to the first operating mode. In some embodiments, continued touch input near the bottom edge region of the display causes the device to first redisplay the affordance in the second state (e.g., when contact is detected) and then redisplay the affordance in the first state (e.g., when contact is maintained for less than a threshold amount of movement and for more than a threshold time since touch down). In some embodiments, a navigation gesture is recognized by the device when movement of the contact is detected (without lift-off of the contact) after the affordance is redisplayed in the first state. In some embodiments, instead of requiring the contact to remain substantially stationary for a threshold time to redisplay the affordance in the first state, the device requires that the contact be less than a threshold amount of movement and that the intensity of the contact be above a predefined light pressure intensity threshold. In some embodiments, a navigation gesture is recognized by the device when movement of the contact is detected (without lift-off of the contact) after the affordance is redisplayed in the first state.
In FIG. 5AA, in response to the media player application returning to a first operational state (e.g., an interactive state), affordance 5002 also transitions from a second state (e.g., a semi-visible/low contrast state 5002-A') back to a first state (e.g., a fully visible/high contrast state 5002-A). In some embodiments, the transition from the second state to the first state is optionally a discontinuous transition, as opposed to a transition across multiple intermediate states between the first and second states. A sharp transition is more likely to alert the user that the operational mode of the user interface has changed, reducing the latency for the user to access controls and provide subsequent input. In some embodiments, when a confirming input is detected (e.g., a second tap after the affordance is displayed in the second state in response to a first tap, or a sustained touch with the same contact that triggered the display of the affordance in the second state), the state shown in FIG. 5Z is skipped and there is a direct transition from the state shown in FIG. 5Y to the state shown in FIG. 5AA.
5AB-5AC illustrate the difference in appearance of affordances against a background in different operational modes according to some embodiments.
In FIG. 5AB, the first state 5002-A of the affordance is opaque and is used when the media player user interface is operating in a first operational state (e.g., an interactive state) and when the media controls are displayed over the media content. The second state 5002-A is semi-transparent and is used when the media player user interface is operating in a second operational state (e.g., a protected state or a media consumption state) and the media controls are not displayed over the media content. The third state is used when the affordance is no longer displayed and the user interface continues to operate in the second operational state (e.g., a protected state or a media consumption state). Affordance 5002 transitions through these states when no input is received for a predetermined period of time after the media player is started (e.g., in full screen or landscape mode).
In FIG. 5AC, the affordance starts out in an invisible state or is not displayed over the media content (e.g., after affordance 5002 ceases to display due to an absence of user input). In response to a first input or a first portion of the input, the affordance is displayed in a second state while the media player user interface maintains a second operational state (e.g., a protected state, or a media consumption state). Then, in response to a second input or a second portion of the input, the affordance is displayed in first state 5002-A and the user interface returns to the first operational state (e.g., an interactive state). In first state 5002-A, the affordance is opaque. In second state 5002-A, the affordance is translucent.
In some embodiments, the affordance appearance type of an affordance is fixed and does not change when the underlying content changes. This provides a consistent appearance for the affordance and may be less distracting to the user. However, in some scenarios, if the content in which the affordance is displayed varies widely, a fixed affordance type may not provide sufficient contrast against the underlying background after the content changes from generally dark tones to generally light tones, or vice versa. Furthermore, sometimes the switching of content luminance levels is a short-term switch (e.g., scrolling black text on a white background), and in such cases, switching affordance types in response to such short-term changes may be inefficient for the user and may be confusing and distracting to the user. On the other hand, if the content switches luminance levels in a more permanent or long-term manner (for example, turning a full-color narrative ebook from one page (e.g., a page depicting the warm tones of an evening sky) to another (e.g., a page depicting a starry night scene)), fixing the affordance appearance type may result in insufficient visibility of the affordances for extended periods of time.
To address the above challenges, in some embodiments, balancing the need to still maintain visibility without unduly distracting the user, the device allows an affordance to switch its affordance appearance type, and correspondingly shifts the value range of the affordance appearance from one value range to another when a predefined range-switching criterion is met. In some embodiments, the range-switching criterion is met when a measurement of the overall luminance state (e.g., accumulated and integrated luminance values) of the content underlying (and optionally surrounding) the affordance crosses a predefined threshold due to a change in the content's appearance over time. In some embodiments, the measurement of the content's overall luminance state takes into account the luminance levels of the relevant portion of the content over time (e.g., using a weighted moving average), and further biases the affordance appearance type of the current affordance such that it does not change (e.g., the bias is optionally achieved by giving a higher weight to older luminance levels of the content and a lower weight to newer luminance levels of the content). The cumulative effect and bias for the current affordance appearance type satisfies both the short-term goal of stabilizing affordance appearance over time as content changes, and the long-term goal of continuing to make affordances salient enough over changed content.
FIG. 5AD illustrates a user interface including affordances that dynamically switch between affordance appearance types based on changes in underlying content over time, according to some embodiments.
FIG. 5AD shows a simple example where affordance 5002 starts out as a dark affordance (initially displayed on a user interface in the state shown in FIG. 5G). The content underlying affordance 5002 then changes to a relatively light state (e.g., the user interface state shown in FIG. 5K), e.g., through scrolling. In this example, the affordance appearance type of affordance 5002 did not change during the scrolling of the content, e.g., because the scrolling was relatively fast and the bias to maintain the currently selected affordance appearance type overcame the change in content appearance over a short timescale. After the scrolling of the content stops, the affordance remains overlaid on the content shown in a light state. The overall luminance state of the content gradually changes over time, with the current background luminance level dominating and replacing the background luminance level previously shown (e.g., when the affordance was initially displayed or during the scrolling of the content). Eventually, at time t1, the measured overall luminance state of the background crosses the predefined threshold and the range-switching trigger criteria is met. In some embodiments, in response to detecting that the range-switching trigger criteria has been met, the device immediately switches affordance appearance types and displays the affordances according to the newly selected affordance appearance type (e.g., as shown above the user interface on the right side of FIG. 5AD). The affordances have the same appearance as those shown in FIG. 5L, but in this example, the user does not need to close and restart the web browser application to have the affordances displayed as light affordances over content 5018.
In some embodiments, when the range-switching trigger criteria are met, the device initiates a gradual transition from a first affordance appearance type (e.g., a dark affordance appearance type) to a second affordance appearance type (e.g., a light affordance appearance type). For example, during a given transition period (e.g., T=t2-t1, 5 seconds), the value range of the affordance appearance passes through one or more intermediate ranges between the value ranges of the first and second appearance types. At any point during the transition period, the appearance of the affordance is determined based on the particular intermediate value range currently being used as the value range of the affordance appearance. As shown in the central user interface of FIG. 5AD, assuming the same background content, the affordance has an intermediate luminance level between the dark and light affordances.
In some embodiments, during the transition period, the measurement of the overall luminance state of the underlying content is continually updated over time, with the luminance levels of the more recent content replacing the luminance levels of the content at the previous time. If the range-switching trigger criteria are met again (e.g., at the same thresholds used in the previous switch, or at a different threshold depending on the currently selected affordance appearance type), the switch to the second affordance appearance type is not fully completed and the affordance is reverted to the first affordance appearance type. In this particular example, the content does not change and the range-switching trigger criteria are not met again during the transition period, so that the switch to the second affordance appearance type is fully completed at time t2 (e.g., the period between t1 and t2 is the defined transition period). Once the switch to the second affordance appearance type is completed, the measurement of the overall luminance state of the underlying content continues to be updated over time, and a switch to the first affordance appearance type may occur again if cumulative changes in the underlying content (e.g., context switching, scrolling, etc.) cause the range switch trigger criteria to be met again.
Figure 5AE illustrates the inversion relationships between value ranges and display characteristics of affordances (e.g., luminance) and underlying content for light affordance appearance types (A), dark affordance appearance types (C), and transition affordance appearance types (B), according to some embodiments. In some embodiments, the inversion curves illustrated in Figure 5AE are optionally used in generating the appearance of affordance 5002 in Figure 5AD.
In the example shown in 5AE, the shapes of the inversion curves for the different appearance affordance types (A), (B), and (C) are identical. Using inversion curves of the same shape allows a correlation value between background luminance values and affordance luminance values to be calculated for each point on the graph and stored in a data table, so that as the transition between affordance types ranges continuously through many intermediate value ranges, the luminance of each pixel on the affordance can be determined, at least in part, simply by a lookup in the table based on the luminance of the corresponding pixel in the background. For example, during a transition period, each of a number of evenly spaced time points is associated with a corresponding intermediate value range between the value ranges of the light affordance and the value range of the dark affordance, and the affordance appearance can be quickly determined at each of the time points based on the inversion curve of the corresponding intermediate value range for that time point, even if the content continuously changes during the transition period.
As shown in FIG. 5AE, the inversion curve 5032 includes two discontinuities. The left discontinuity 5034 is introduced to address the interference point (e.g., at background luminance 25.4%), which is the intersection of the isoluminance line 5036 (e.g., affordance luminance=background luminance) with the inversion curve without the discontinuity. The left discontinuity allows the affordance to have a luminance value that is not exactly the same as the background luminance, thereby avoiding the possibility of an "invisible" affordance in some specific cases. The right discontinuity 5038 is introduced to address the interference point (e.g., at luminance 74.51%), which is the intersection of the isoluminance line 5036 with the inversion curve without the discontinuity. The right discontinuity allows the affordance to have a luminance value that is not exactly the same as the background luminance, thereby avoiding the possibility of an "invisible" affordance in some specific cases.
In addition, special corrections are made to the luminance of the affordances near discontinuities 5036 and 5038, so that strict inversions (e.g., increasing the background luminance corresponds to decreasing the luminance of the affordances and vice versa) are not always seen. For example, in the correction region for the left discontinuity 5036, the affordance luminance on the side with the higher background luminance is relatively flat, while the affordance luminance on the side with the lower background luminance contains a small inversion of the normal inversion relationship. In the correction region for the right discontinuity 5038, the affordance luminance on the side with the lower background luminance contains a relatively large inversion of the normal inversion relationship. The design of the precise shape of these correction regions takes into account the responsiveness of human vision to luminance values in these regions, and also the need to create sufficient contrast between the affordances and the background near the interference points. For example, the width of the interference region and the adjustment of the normal inversion relationship depend on the amount of contrast required for the affordances on the background at these interference points.
Each of the graphs (A), (B), and (C) shown in FIG. 5AE includes a background luminance bar and an affordance response bar (shown below the graph area). The background luminance bar and affordance response bar in each figure show the corresponding background luminance and affordance luminance generated according to the inversion curve 5032 in the same figure. In addition, the background color for the dark affordances in graph (A) is white, which allows all shades of dark affordances (e.g., including completely black affordances, but not completely white affordances) to be shown. The background color for graph (C) for the light affordances is black, which allows all shades of light affordances (e.g., including completely white affordances, but not completely black affordances) to be shown. The background color for graph (B) for the transition affordances is gray (50% luminance). Affordance luminance relative to a background luminance of 50% is a slightly lighter shade of gray (e.g., having a luminance value below 50%) and is shown by comparing the affordance response at the midpoint along the horizontal axis to the 50% gray bar displayed at the bottom of graph (B).
FIG. 5AF illustrates a gradual shift from a dark affordance appearance type (shown in graph (A)) to a light affordance appearance type (e.g., shown in graph (C)) through multiple transition affordance appearance types (e.g., shown in graphs (B-1), (B-2), (B-3)) in accordance with some embodiments.
In some embodiments, when the range-switching trigger criteria is met at time t1, the device begins a gradual transition from a first affordance appearance type (e.g., a dark affordance appearance type) to a second affordance appearance type (e.g., a light affordance appearance type). Then, during a given transition period (e.g., T=t2-t1), the value range of the affordance appearance shifts between the value ranges of the first and second appearance types (e.g., upwards or downwards depending on the direction of the switch) through a number of intermediate ranges (e.g., the total number of intermediate ranges is the refresh rate of the display and the gap value between the upper limit of the affordance luminance for the dark affordance and 1). At any point during the transition period T, the appearance of the affordance is determined based on the particular intermediate value range currently being used as the value range of the affordance appearance.
In some embodiments, during the transition period, the measurement of the overall luminance state of the underlying content continues to be updated over time, with the luminance level of the more recent content replacing the luminance level of the content at the previous time. If the range-switching trigger criteria are met again (e.g., with the same threshold used in the previous switch, or with a different threshold depending on the currently selected affordance appearance type), the switch to the second affordance appearance type may not be fully completed, the shift in the value range of the affordance appearance may reverse direction, and the affordance may eventually be switched back to the first affordance appearance type. If the range-switching trigger criteria are not met a second time during the transition period, the switch to the second affordance appearance type reaches completion at the end of the transition period. Once the switch to the second affordance appearance type is completed, the measurement of the overall luminance state of the underlying content continues to be updated over time, and the switch to the first affordance appearance type may occur again if the range-switching trigger criteria are next met due to cumulative changes in the underlying content (e.g., context switching, scrolling, etc.).
In the example shown in 5AF, the shapes of the inversion curves for the different appearance affordance types (A), (B-1), (B-2), (B-3), and (C) are identical. Using inversion curves of identical shape allows a correlation value between the background luminance value and the affordance luminance value to be calculated for each point on the graph (or for the inversion curve for each intermediate value range) and stored in a data table, so that as the transition between affordance types ranges through many intermediate value ranges in succession, the luminance of each pixel on the affordance can be determined, at least in part, simply by a quick lookup in the data table based on the luminance of the corresponding pixel in the background. For example, during the transition period, each of multiple evenly spaced time points is associated with a corresponding intermediate value range between the value range of the light affordance and the value range of the dark affordance, and even if the content continuously changes during the transition period and the underlying content has large variations in luminance under different parts of the affordance, the affordance appearance can still be quickly determined at each of the time points based on the data previously stored in the data table.
5AG-5AK are expanded replicas of the inversion relationships between affordance display characteristics and underlying content for the different affordance types shown in FIG. 5AF.
6A-6C are flow diagrams illustrating a method 6000 for changing the appearance of an affordance in response to changes in the appearance of underlying content, according to some embodiments. Method 6000 is performed in an electronic device having a display and a touch-sensitive surface (e.g., device 300 of FIG. 3, or portable multifunction device 100, FIG. 1A). In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the touch-sensitive surface and the display are integrated into a touch-sensitive display. In some embodiments, the display is a touchscreen display, and the touch-sensitive surface is on or integrated into the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations of method 6000 are optionally combined and/or the order of some operations is optionally changed.
Method 6000 relates to displaying an affordance (e.g., an affordance indicating an acceptable starting area of a gesture for displaying a home screen) on content in a manner in which a display characteristic (e.g., a gray value or luminance value) of the affordance dynamically changes based on changes in the same display characteristic of the underlying content. Specifically, the value of the display characteristic of the affordance changes in an opposite direction to the change in the value of the same display characteristic of the underlying content (e.g., the gray value of the affordance is the inverse of the gray value of the underlying content). In addition, the value of the display characteristic of the affordance is constrained to a smaller value range than the value of the display characteristic of the underlying content. As a result, the device provides the affordance in a less distracting or intrusive manner while maintaining full visibility of the affordance as the appearance of the content continues to change (e.g., due to scrolling, context switching, and playing dynamic content). Providing affordances with dynamically changing appearances based on the appearance of the underlying content in the manner described herein increases usability of the device (e.g., by providing guidance to the user regarding the inputs required for a desired result without unduly distracting the user, thereby reducing user errors while operating the device), the interface between the user and the device becomes more efficient (e.g., by helping the user achieve the intended result with the required inputs and reducing user errors while operating/interacting with the device), and improves the battery life of the device (e.g., by helping the user use the device more quickly and efficiently). Providing affordances in the manner described herein allows on-screen affordances to effectively replace hardware buttons that provide the same functionality (e.g., displaying the home screen) in many different user interface contexts, helping to reduce the manufacturing and maintenance costs of the device. Additionally, the claimed solution constrains the value range of affordances to eliminate white-on-black and black-on-white contrasts between affordances and background content, thereby mitigating the risk of display burn-in due to prolonged display of affordances on white or black backgrounds. A known cause of screen burn-in is the prolonged display of a stationary image (e.g., a system-wide affordance such as a home gesture indicator) combined with non-uniform pixel usage (e.g., this is most severe when the contrast between foreground and background content is high). In certain usage scenarios (e.g., the realization of a system-wide home affordance), the proposed solution (e.g., reducing the extremely high contrast while keeping the affordance highly visible) effectively addresses the burn-in challenge. The problem and causes of burn-in in mobile phone displays have been well documented in the industry literature for many years and remain a problem in many commercial products. In the past, there have been attempts to address this problem by moving the affordance around on the screen or simply disappearing after a period of inactivity. However, such solutions worsen the usability of the device. The proposed solution, in which the inversion of the display properties of the underlying content is the basis for determining the same display property values of the affordance within a sub-range of display property values, allows the affordance to be used in the same place without the risk of burn-in.
The method 6000 is performed on a device having a display and a touch-sensitive surface (e.g., a touch screen display that serves as both the display and the touch-sensitive surface). The device displays (6002) on the display content (e.g., a home screen, a widget screen, a desktop, a user interface of an application, a user interface of a media player, etc.) and an affordance (e.g., a home affordance indicating an area on the display that is responsive to a home gesture), where the affordance is displayed over a portion of the content, and values of display characteristics of the affordance (e.g., gray values or luminance values of an image (e.g., a color image or a black and white image), intrinsic display parameters other than gray values or luminance values (e.g., hue, saturation, etc. of a full-color image), or derived display parameters calculated based on one or more intrinsic display parameters (e.g., gray values or luminance values of a full-color image, or variations or equivalents thereof)) are determined based on values of the same display characteristics of the portion of the content over which the affordance is displayed, and values of the display characteristics of the content over which the affordance is displayed are determined based on values of the same display characteristics of the portion of the content over which the affordance is displayed. The value of the display characteristic is allowed to vary within a first range of values (e.g., a range of [0, 1], e.g., a "range" is defined mathematically as the difference in value between the range's maximum value and the range's minimum value), and the value of the display characteristic of the affordance is constrained to vary within a second range of values that is smaller than the first range of values (e.g., one of ranges such as [0, 0.4], [0.6, 1], [0.1, 0.7], [0, 0.7], or [0.3, 1], where a "sub-range" of the "first range" is defined as a range having a maximum value smaller than the first range's maximum value and a minimum value greater than the first range's minimum value, or a range having a maximum value smaller than the first range's maximum value and a minimum value the same as the first range's minimum value, or a range having a maximum value the same as the first range's maximum value and a minimum value greater than the first range's minimum value). While displaying the content and the affordance, the device detects changes in appearance of the content over which the affordance is displayed (6004). In response to detecting a change in the appearance of the content over which the affordance is displayed, the device changes (6006) the appearance of the affordance, including: in response to a determination that the value of the display characteristic of the content has decreased, increasing a value of the display characteristic of the affordance in response to a magnitude of the change in the value of the display characteristic of the content and a second range of values (e.g., the affordance becomes lighter as the content underlying the affordance becomes darker), and in response to a determination that the value of the display characteristic of the content has increased, decreasing a value of the display characteristic of the affordance in response to a magnitude of the change in the value of the display characteristic of the content and a second range of values (e.g., the affordance becomes darker as the content underlying the affordance becomes lighter). This is shown, for example, in Figures 5F, 5G-P, 5Q, and 5R, where a display characteristic (e.g., luminance) of the affordance changes in response to a change in the same display characteristic (e.g., luminance) of the underlying content (e.g., as the content is scrolled). In addition, the value range of the affordance appearance of a display characteristic (e.g., luminance) is constrained to a subrange of the value range of the same display characteristic of the underlying content (e.g., the entire value range from black (e.g., 0 or 0%) to white (e.g., 1 or 100%)).
In some embodiments, the change in appearance of the content (6008) is due to content navigation (e.g., a portion of the content underlying the affordance changes due to scrolling, paging, etc. of the content). This is shown, for example, in FIGS. 5G-5P. Changing the value of a display characteristic of an affordance while the same display characteristic of the underlying content is changed due to content navigation may increase usability of the device (e.g., by maintaining sufficient visibility of the affordance throughout content navigation to help provide the user with the necessary input to achieve a desired outcome), make user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and improve the power efficiency and battery life of the device (e.g., by reducing user errors and helping users use the device more quickly, quickly, and efficiently).
In some embodiments, the change in appearance of the content (6010) is due to the content changing over time (e.g., the content is a playing video or animation and the displayed image of the video or animation changes over time). This is shown, for example, in FIGS. 5T-5U. Changing the values of the display characteristics of the affordances while the same display characteristics of the underlying content are changing due to the content changing over time may increase usability of the device (e.g., by maintaining sufficient visibility of the affordances across temporal changes in the content to help provide the user with the necessary input to achieve a desired result), make user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), as well as improve the power efficiency and battery life of the device (e.g., by reducing user errors and helping users use the device more quickly, quickly and efficiently).
In some embodiments, the affordance has a first variant with a first set of end values for a second value range and a second variant with a second set of end values for the second value range (6012), the second set of end values being different from the first set of end values (for a "light affordance" initially displayed on a light background, the second value range has a first start point and a first end point, and for a "dark affordance" initially displayed on a dark background, the second value range has a second start point and a second end point that are different from the first start point and the second start point, respectively). This is shown, for example, in Figures 5A and 5R. In some embodiments, the value range of the first variant does not overlap with the value range of the second variant (e.g., all values of the value range for the "light affordance" are greater than all values of the value range for the "dark affordance"). Providing two variants of an affordance, with different value ranges, allows the device to further improve the visibility of the affordance while unobtrusively maintaining the appearance of the affordance on different types of content, thereby improving device usability (e.g., by maintaining sufficient visibility of the affordance for different types of content and helping the user provide the necessary input to achieve a desired outcome) and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, a first variant of the affordance is displayed in response to a determination that the content corresponds to a first application, and a second variant of the affordance is displayed in response to a determination that the content corresponds to a second application different from the first application (6014). (e.g., the light vs. dark affordance used in the currently displayed application was selected by an application developer of the currently displayed application, and the device displays the first variant or the second variant of the affordance in response to an affordance selection parameter, or a corresponding set of end values for a second value range, as specified in the program code of the currently displayed application (e.g., using the first or second set of end values for the second value range when changing the value of a display characteristic of the affordance). Allowing different applications to use different variations of affordances helps application developers customize the appearance of affordances based on the application context, thereby further improving the fit between the appearance of affordances and the appearance of application content; improving usability of the device (by maintaining sufficient visibility of the affordances without unduly distracting the user and helping to provide the user with the necessary input to achieve the desired result); and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, the appearance of a first portion of the content over which the affordance is displayed changes (6016) a different amount than the appearance of a second portion of the content over which the affordance is displayed, where changing the appearance of the affordance includes changing the appearance of the first portion of the affordance corresponding to the first portion of the content a different amount than changing the appearance of the second portion of the affordance corresponding to the second portion of the content (e.g., the affordance changes in response to appearance changes in a corresponding portion of the content underlying the first portion of the affordance). This is shown, for example, in FIG. 5Q. For example, as the underlying content varies (e.g., if the affordance is a blurred/de-saturated/inverted version of a portion of the content), the appearance of the affordance varies, with the appearance of different portions of the affordance reflecting the appearance of the content underlying the different portions of the affordance. By being able to apply different amounts of change to the display characteristics of different portions of an affordance based on different amounts of change occurring in the display characteristics of different portions of the underlying content, usability of the device is improved (e.g., by maintaining sufficient visibility of the affordance across content changes and helping to provide the user with the necessary input to achieve a desired result), user-to-device operation is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and power efficiency and battery life of the device are improved (e.g., by reducing user errors and helping users to use the device more quickly, efficiently, and efficiently).
In some embodiments, the first value range is a continuous value range (6018), and the second value range includes at least a discontinuity corresponding to a first value of the display characteristic within the first value range. For example, in some embodiments, for a small range of values around 0.5 for the display characteristic of the content underlying the affordance, a jump from a first value below 0.5 to a second value above 0.5. In some embodiments, the device uses a discontinuity function to calculate the value of the display characteristic for the affordance based on the value of the same display characteristic for the underlying content to ensure that the appearance of the affordance is not too close to the appearance of the underlying content (to ensure that a gray affordance is not displayed over gray content that is too close to the gray value of the gray affordance). FIG. 5AE shows an inversion curve that includes two discontinuities (e.g., 5034 and 5038) for the value of the display characteristic of the affordance. Using discontinuous value ranges for the display characteristics of affordances while maintaining continuous value ranges for the value representation of the underlying content helps avoid situations where an affordance takes on a value that is too close to that of the underlying content, causing insufficient visibility of the affordance. Using discontinuous value ranges for the display characteristics of affordances can improve device operability (e.g., by maintaining sufficient visibility of the affordance across content changes and helping the user provide the necessary input to achieve a desired result), make user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and improve the power efficiency and battery life of the device (e.g., by reducing user errors and helping users use the device more quickly, quickly, and efficiently).
In some embodiments, the affordance has a first variant and a second variant (6020) (e.g., a light affordance and a dark affordance), a second value range corresponding to the first variant of the affordance and a second value range corresponding to the second variant of the affordance that do not overlap (the value range for the light affordance is separated from the value range for the dark affordance by a cutoff value range), and the device dynamically selects one of the first variant and the second variant to display on the content based on an initial value of a display characteristic of the content at a given time (e.g., when an application launches, a video scene changes, or a user interface switch occurs within the application). For example, if the content initially has dark gray values (e.g., less than 0.5), the affordance has an initial value in the light value range of the grayscale (e.g., greater than 0.6), and if the content initially has light gray values (e.g., greater than 0.5), the affordance has an initial value in the dark range of the grayscale (e.g., less than 0.4). The exact gray value of the affordance is optionally obtained by inverting the gray value of the content through a corresponding inverse function for the light or dark value range associated with the initial appearance of the affordance. In some embodiments, if the affordance is initially a light affordance (e.g., having a gray value of 0.9) displayed on dark content (e.g., having a gray value of 0.2), when the content underneath the affordance becomes increasingly lighter (e.g., increasing gray values toward 1), the affordance becomes increasingly darker (e.g., decreasing gray values toward 0), but the gray value of the affordance is constrained by a minimum threshold gray value (e.g., 0.6) that is brighter than the median gray value (e.g., 0.5) on the grayscale [0, 1]. When the content underneath the affordance becomes increasingly darker (e.g., decreasing gray values toward 0), the affordance becomes increasingly lighter (e.g., increasing gray values toward 1), until it becomes completely white (e.g., reaching an end value of 1 on the grayscale [0, 1]) when the content becomes completely black. In another example, if the affordance is initially a dark affordance (e.g., having a gray value of 0.2) displayed over light content (e.g., having a gray value of 0.9), then as the content beneath the affordance becomes darker (e.g., decreasing gray value toward 0), the affordance becomes lighter (e.g., increasing gray value toward 1), but the gray value of the affordance is constrained by a maximum threshold gray value (e.g., 0.4) that is darker than the middle gray value (e.g., 0.5) on the grayscale [0, 1]. As the content beneath the affordance becomes lighter (e.g., increasing gray value toward 1), the affordance becomes darker (e.g., decreasing gray value toward 0), until it becomes completely black (e.g., reaching an end value of 0 on the grayscale [0, 1]) when the content becomes completely white. FIGURE 5G illustrates the selection of a dark affordance based on an initial overall luminance state of relatively darker underlying content, and FIGURE 5L illustrates the selection of a light affordance based on an initial overall luminance state of relatively lighter underlying content. By allowing an application to dynamically select between two different variants of an affordance (e.g., dark and light content), the appearance of the affordance is further improved based on the application context, thereby further improving the match between the appearance of the affordance and the appearance of the underlying content, improving device usability (by maintaining sufficient visibility of the affordance without unduly distracting the user and helping provide the user with the necessary input to achieve a desired result), and making user-to-device operation more efficient (e.g., by reducing user distraction and reducing user errors while using or interacting with the device).
In some embodiments, displaying the affordance includes displaying the affordance at a first size (e.g., near a bottom edge of the device) when the device is in a first orientation (6022), the method including detecting a rotation of the device from the first orientation to a second orientation different from the first orientation while displaying the affordance at the first size (wherein the rotation of the device changes the displayed content from a first user interface orientation (e.g., portrait orientation) to a second user interface orientation (landscape orientation)), and, in response to detecting the rotation of the device from the first orientation to the second orientation, displaying the affordance at the second size different from the first size (and, optionally, displaying the affordance in a different location (e.g., near a new bottom edge of the device defined based on the second orientation of the device)). In some embodiments, a longer version of the affordance is displayed when the device is in a landscape orientation, and a shorter version of the affordance is displayed when the device is in a portrait orientation. Displaying affordances of different sizes when the device is rotated improves the visual fit between the appearance of the affordances and the device orientation (and therefore the content orientation), improves usability of the device (by maintaining sufficient visibility of the affordances without unduly distracting the user and helping to provide the user with the necessary input to achieve the desired result), and makes user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, the display characteristic is a gray value (6024) (e.g., the gray value has a full value range [0, 1], representing a gray value range from black (e.g., gray value=0) to white (e.g., gray value=1). Providing an affordance that changes gray value based on the gray value of the underlying content improves usability of the device (by maintaining sufficient visibility of the affordance without unduly distracting the user, and by helping the user provide the necessary input to achieve a desired result) and makes user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, current values of display characteristics of the content are obtained (6026) by blurring a portion of the content (e.g., by applying a blur function (e.g., a Gaussian blur function) to the content area underlying the affordance with a predefined blur radius and to the periphery of the content area directly underlying the affordance with at least one blur radius). In some embodiments, other filters, such as desaturating and/or changing opacity, are applied after blurring the content. Providing the affordance with display characteristics derived based on the same display characteristics of a blurred version of the underlying content improves operability of the device (by maintaining sufficient visibility of the affordance without unduly distracting the user, and by providing the user with the necessary input to help achieve a desired result) and makes user-to-device operation more efficient (e.g., by reducing user distraction and reducing user errors while using or interacting with the device).
In some embodiments, the current values of the display characteristics of the content are obtained (6028) by desaturating a portion of the content (e.g., by changing the color values (e.g., RGB, HSL, or HSV values) of each pixel of a region of the content underlying the affordance, and at least one blur radius around that region, to a corresponding scalar value (e.g., gray value) on a monochromatic scale (e.g., grayscale). In some embodiments, other filters, such as blurring and/or changing opacity, are applied after desaturating the content. Providing the affordance with display characteristics derived based on the same display characteristics of a desaturated version of the underlying content improves operability of the device (by maintaining sufficient visibility of the affordance without unduly distracting the user, and by helping provide the user with necessary input to achieve a desired result) and makes user-to-device operation more efficient (e.g., by reducing user distraction and reducing user errors while using or interacting with the device).
In some embodiments, the affordance shows (6030) a reaction area for initiating a predefined gesture input (e.g., a home/multitasking gesture to display a home screen and/or an application switching user interface). This is shown, for example, in FIGS. 5A-5D. In some embodiments, the affordance is not an actual button. Tapping or pressing on the affordance does not trigger any functionality of the device. In some embodiments, when the affordance is no longer displayed (e.g., after the affordance fades out in full screen mode of the application), the predefined gesture (e.g., a home/multitasking gesture) still functions as it did when the affordance was displayed. In some embodiments, the affordance reappears in full screen content display mode when a user input (e.g., a home/multitasking gesture, or a tap input on the display, etc.) is detected. The affordance is a narrow affordance with a small height to length ratio. Because reaction areas for gestures are typically more tolerant than reaction areas for buttons, affordances that indicate reaction areas for initiating predefined gestures typically do not need to have specially enhanced visibility relative to the underlying content, and thus are more beneficial to have affordances that are less distracting to the user to help avoid user mistakes while interacting with the device. Thus, using affordances with dynamically varying display characteristics in the manner described herein improves device operability (by maintaining sufficient visibility of the affordances without unduly distracting the user, and helping to provide the user with the necessary input to achieve a desired result), and makes user-to-device operation more efficient (e.g., by reducing user distraction and reducing user mistakes while using or interacting with the device).
It should be understood that the particular order described for the operations in Figures 6A-6C is merely an example, and that the described order is not intended to indicate the only order in which the operations may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 7000 and 8000) are also applicable in a similar manner to method 6000 described above with respect to Figures 6A-6C. For example, the contacts, gestures, user interface objects, application views, control panels, controls, affordances, position thresholds, directional conditions, inversion curves, filters, value ranges, navigation criteria, movement parameters, focus selectors, and/or animations described above with respect to method 6000, optionally have one or more of the features of the contacts, gestures, user interface objects, application views, control panels, controls, position thresholds, directional conditions, navigation criteria, movement parameters, focus selectors, and/or animations described herein with respect to other methods described herein (e.g., methods 7000 and 8000). For brevity, these details are not repeated here.
Operations in the information processing methods described above are optionally performed by executing one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as described above in connection with Figures 1A and 3) or an application-specific chip.
6A-6C are optionally implemented by the components depicted in FIG. 1A-1B. For example, the detection and modification operations are optionally implemented by event sorter 170, event recognizer 180, and event handler 190. Event monitor 171 of event sorter 170 detects a contact on touch-sensitive display 112, and event dispatcher module 174 delivers the event information to application 136-1. A corresponding event recognizer 180 of application 136-1 compares the event information to a corresponding event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface (or a rotation of the device) corresponds to a predefined event or sub-event, such as a selection of an object on a user interface or a rotation of the device from one orientation to another. When a corresponding predefined event or sub-event is detected, event recognizer 180 activates event handler 190 associated with the detection of the event or sub-event. Event handling unit 190 optionally uses or calls data update unit 176 or object update unit 177 to update application internal state 192. In some embodiments, event handling unit 190 accesses a corresponding GUI update unit 178 to update what is displayed by the application. Similarly, it will be clear to one skilled in the art how other processes can be implemented based on the components depicted in Figures 1A-1B.
7A-7E are flow diagrams illustrating a method of changing the appearance of an affordance in response to changes in the mode of the user interface in which the underlying content and the affordance appearance change, according to some embodiments. Method 7000 is performed in an electronic device having a display and a touch-sensitive surface (e.g., device 300 of FIG. 3, or portable multifunction device 100, FIG. 1A). In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the touch-sensitive surface and the display are integrated into a touch-sensitive display. In some embodiments, the display is a touchscreen display, and the touch-sensitive surface is on or integrated into the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 600 are optionally combined and/or the order of some operations is optionally changed.
Method 7000 relates to causing an affordance (e.g., an affordance indicating an acceptable starting area for a gesture to display a home screen) to be displayed on an application user interface in a manner in which a set of display characteristics of the affordance (e.g., gray value, luminance value, opacity, hue, saturation, etc.) varies based on a set of display characteristics of the underlying content according to two different rule sets depending on a current display mode of the application user interface. For example, when the application user interface is displayed in an interactive mode (e.g., where frequent user input is expected), the appearance of the affordance varies in a first manner based on the appearance of the underlying content (e.g., based on a first set of rules) such that the affordance is more visible, while while the application user interface is displayed in a full screen content display mode (e.g., where viewing content may be the primary objective), the appearance of the affordance varies in a second manner based on the appearance of the underlying content (e.g., based on a second set of rules) such that the affordance is less distracting to the user. Thus, providing affordances that change appearance differently based on the appearance of the underlying content and based on the display mode of the application user interface may increase usability of the device (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for a desired result without unduly distracting the user, thereby reducing user errors while operating the device), the interface between the user and the device may be more efficient (e.g., by helping the user achieve the intended result with the required inputs and reducing user errors while operating/interacting with the device), and the battery life of the device may be improved (e.g., by helping the user use the device more quickly and efficiently). Providing affordances in the manner described herein may allow on-screen affordances to effectively replace hardware buttons that provide the same functionality (e.g., displaying a home screen) in many different user interface contexts, helping to reduce manufacturing and maintenance costs of the device. Providing affordances in the manner described herein may also help to reduce or eliminate display burn-in issues.
The method 7000 is performed on a device having a display and a touch-sensitive surface (e.g., a touch screen display that serves as both the display and the touch-sensitive surface). The device displays (7002) a user interface of an application (e.g., a media player user interface, a browser user interface, an instant messaging user interface, a map user interface, a phone user interface, a gaming user interface, etc.) in a first mode (e.g., a content display mode that does not include full screen content, a windowed mode, or a default mode). While displaying a user interface of the application in a first mode, the device displays (7004) an affordance having a first appearance on the user interface (e.g., a home affordance that indicates a home gesture response area on the screen), where the affordance is displayed on a portion of the user interface (e.g., the affordance is displayed in a first defined area of the display (e.g., a home affordance display area located near a bottom central area of the display), and values of a set of one or more display characteristics of the affordance having the first appearance change in response to changes in values of the set of one or more display characteristics of the portion of the user interface underlying the affordance according to a first set of one or more rules (e.g., the set of one or more display characteristics of the affordance are obtained by applying a first set of filters to desaturate, blur, change opacity, and/or invert the luminance or gray values of an image of the portion of the user interface underlying the affordance)). While displaying an affordance having a first appearance on a portion of the user interface displayed in the first mode, the device detects (7006) a request (e.g., a request generated by an application program that is provided to the user interface or the operating system based on a current operating state (e.g., an extended period of absence of input, predefined criteria for switching operating modes, etc.) or a user request (e.g., a tap input by a contact on a touch screen, or a swipe input, etc.)) to transition from displaying the user interface in the first mode to displaying the user interface in a second mode (e.g., a full-screen content display mode). In response to detecting the request, the device displays (7008) the user interface in a second mode (e.g., in a full-screen content display mode, a portion of the original user interface is enlarged, some of the user interface elements in the user interface, such as application menu bars, scroll bars, etc., are removed from the user interface, and a system status bar that was previously displayed with the user interface is also removed from display), and the device displays an affordance having a second appearance on the user interface displayed in the second mode, and a value of a set of one or more display characteristics of the affordance having the second appearance is adjusted to match the user interface underlying the affordance. The affordance may change according to a second set of one or more rules that differ from the first set of one or more rules in response to a change in value of a set of one or more display characteristics of the portion of the interface (e.g., the set of one or more display characteristics of the affordance is obtained by applying a second set of filters to desaturate, blur, change the opacity, or invert the luminance or gray values of an image of the portion of the user interface underlying the affordance). (An affordance having a second appearance is a variation of an affordance having a first appearance, both derived from the portion of the user interface underlying the affordance but with a different set of filters or the same set of filters with different adjustment parameters). This is illustrated in FIGS. 5S-5W, where affordance 5002 is displayed in a second state (e.g., a low-contrast state in FIGS. 5V and 5W) when the device transitions from an interactive mode to a media consumption mode after the affordance is initially displayed in a first state (e.g., a fully visible state in FIGS. 5T and 5U).
In some embodiments, while displaying an affordance having a second appearance on the user interface displayed in the second mode, pursuant to a determination that the fade-out criteria are met, the device stops displaying (7010) the affordance on the user interface displayed in the second mode (e.g., while maintaining the display of the user interface in the second mode), and pursuant to a determination that the fade-out criteria are not met, maintains the display of the affordance having the second appearance on the user interface displayed in the second mode. This is shown, for example, in FIG. 5X, where affordance 5002 in a low-contrast state eventually disappears entirely after an additional period of time without user input. In some embodiments, for the fade-out criteria to be met, the fade-out criteria require that no user input has been detected on the touch-sensitive surface for at least a predefined threshold time (e.g., the fade-out criteria may be when, after entering a full-screen content display mode, the device does not detect any touch input anywhere on the touch-sensitive surface for 30 seconds, or when, after entering a full-screen content display mode, the device does not detect any touch input near the bottom central region of the touch screen display for 30 seconds (e.g., other portions of the screen may still continue to receive and respond to user input without affecting the determination with respect to the fade-out criteria)). By fading out or maintaining the visibility of affordances displayed in a predefined display mode of an application user interface based on predefined criteria, the operability of the device is improved (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for a desired result without unduly distracting the user, thereby reducing user errors while operating the device), the interface between the user and the device is more efficient (e.g., by helping the user achieve the intended result with the required inputs and reducing user errors while operating/interacting with the device), and the battery life of the device is improved (e.g., by helping the user use the device more quickly and efficiently).
In some embodiments, the user interface of the application displayed in a first mode includes (7012) a representation of content (e.g., a video, a game, a document, album art for a song) that occupies less than half of the total display area of the display (e.g., the user interface of the application displayed in the first mode is displayed simultaneously with a system status bar on the display), and the user interface of the application displayed in a second mode includes a representation of content that occupies the total display area of the display (e.g., in a full screen content display mode, a portion of the original user interface is enlarged, some of the user interface elements within the user interface, such as an application menu bar, scroll bars, etc., are removed from the user interface, and a system status bar that was previously displayed with the user interface is also removed from view). Thus, by providing affordances that change appearance based on the appearance of the underlying content in a manner different depending on whether the content is displayed in a normal display mode or a full-screen display mode, usability of the device is increased (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for a desired result without unduly distracting the user, thereby reducing user errors while operating the device), the interface between the user and the device is more efficient (e.g., by helping the user achieve intended results with the necessary inputs, reducing user errors when operating/interacting with the device), and battery life of the device is improved (e.g., by helping the user use the device more quickly and efficiently).
In some embodiments, at least one of the first appearance and the second appearance of the affordance (e.g., the luminance, intensity, or gray value of the affordance) is based on an inversion of a portion of the user interface underlying the affordance (7014). For example, in some embodiments, the portion of the user interface underlying the affordance is desaturated to obtain a monochromatic image, the monochromatic image is blurred, and then the luminance or gray values of pixels in the blurred monochromatic image are inverted to obtain the luminance or gray values of pixels in the affordance. Providing the affordance with display characteristics derived based on an inversion of the underlying content improves operability of the device (by maintaining sufficient visibility of the affordance without unduly distracting the user, and helping provide the user with necessary input to achieve a desired result) and makes user-to-device operation more efficient (e.g., by reducing user distraction and reducing user errors while using or interacting with the device).
In some embodiments, a first set of rules requires a first amount of inversion to be applied to a portion of the user interface underlying the affordance to obtain a first appearance of the affordance (7016), and a second set of rules requires a second amount of inversion to be applied to a portion of the user interface underlying the affordance to obtain a second appearance of the affordance, where the second amount of inversion is less than the first amount of inversion (e.g., the second set of rules reduces the amount of inversion of the portion of the user interface underlying the affordance to obtain a display characteristic and set of affordances). Varying the amount of inversion applied to the underlying content to obtain the affordances helps to adjust the visibility of the affordances depending on the display mode of the application user interface, thereby improving device operability (e.g., by reasonably balancing the requirements for sufficient visibility with the requirements for affordance unobtrusiveness, and helping provide the user with the necessary input to achieve the desired results) and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, at least one of the first appearance and the second appearance of the affordance (e.g., the luminance, intensity, or gray value of the affordance) is obtained by blurring (7018) a portion of the user interface underlying the affordance. For example, in some embodiments, the portion of the user interface underlying the affordance is desaturated to obtain a monochromatic image, the monochromatic image is blurred, and then the luminance or gray values of pixels in the blurred monochromatic image are inverted to obtain the luminance or gray values of pixels in the affordance. Providing the affordance with display characteristics obtained by blurring the underlying content improves operability of the device (by maintaining sufficient visibility of the affordance without unduly distracting the user, and helping provide the user with necessary input to achieve a desired result) and makes user-to-device operation more efficient (e.g., by reducing user distraction and reducing user errors while using or interacting with the device).
In some embodiments, a first set of rules requires applying a first blur amount (e.g., a Gaussian blur function) to portions of the user interface underlying the affordance to obtain a first appearance of the affordance (7020), and a second set of rules requires applying a second blur amount (e.g., a Gaussian blur function) to portions of the user interface underlying the affordance to obtain a second appearance of the affordance, and the second blur amount is less than the first blur amount (e.g., the second blur function has a smaller blur radius than the first blur function) (e.g., the second set of rules reduces blur of portions of the user interface underlying the affordance to obtain a set of display characteristics of the affordance). Varying the amount of blur applied to the underlying content to obtain an affordance helps to adjust the visibility of the affordance depending on the display mode of the application user interface, thereby improving device operability (e.g., by reasonably balancing the requirements of sufficient visibility with the requirements of unobtrusive affordances, and helping to provide the user with the necessary input to achieve the desired outcome) and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, at least one of the first appearance and the second appearance of the affordance (e.g., the luminance, intensity, or gray value of the affordance) is obtained by desaturating (7022) a portion of a user interface underlying the affordance. For example, in some embodiments, the portion of the user interface underlying the affordance is desaturated to obtain a monochromatic image, the monochromatic image is blurred, and then the luminance or gray values of pixels in the blurred monochromatic image are inverted to obtain the luminance or gray values of pixels in the affordance. Providing the affordance with display characteristics obtained by desaturating the underlying content improves operability of the device (by maintaining sufficient visibility of the affordance without unduly distracting the user, and helping to provide the user with necessary input to achieve a desired result) and makes user-to-device operation more efficient (e.g., by reducing user distraction and reducing user errors while using or interacting with the device).
In some embodiments, a first set of rules requires a first amount of desaturation to be applied to a portion of a user interface underlying the affordance to obtain a first appearance of the affordance (7024), and a second set of rules requires a second amount of desaturation to be applied to a portion of a user interface underlying the affordance to obtain a second appearance of the affordance, where the second amount of desaturation is less than the first amount of desaturation (e.g., the second set of rules reduces the amount of desaturation of the portion of the user interface underlying the affordance to obtain a set of display characteristics of the affordance). Varying the amount of desaturation applied to the underlying content to obtain the affordance helps to adjust the visibility of the affordance depending on the display mode of the application user interface, thereby improving device operability (e.g., by reasonably balancing the requirements for sufficient visibility with the requirements for affordance unobtrusiveness, and helping provide the user with the necessary input to achieve the desired result) and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, while displaying the user interface in the second mode without displaying the affordances, the device detects (7026) an input that satisfies a first affordance redisplay criterion, where the first affordance redisplay criterion is met when the input is detected on the touch-sensitive surface (e.g., at a location corresponding to the first display location), and in response to detecting the input that satisfies the affordance redisplay criterion, the device redisplays the affordances on the user interface displayed in the second mode. This is shown, for example, in FIGS. 5X-5AA. In some embodiments, when the affordances are redisplayed on the user interface displayed in the second mode, values of a set of one or more display characteristics of the affordances change according to a second set of one or more rules in response to a change in values of the set of one or more display characteristics of the portion of the user interface. In some embodiments, when the affordances are redisplayed on the user interface displayed in the first mode, values of a set of one or more display characteristics of the affordances change according to a first set of one or more rules in response to a change in values of the set of one or more display characteristics of the portion of the user interface. By re-displaying the affordances based on predefined criteria after they have faded out, the operability of the device is improved (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for the desired result without unduly distracting the user, thereby reducing user errors while operating the device), the interface between the user and the device is more efficient (e.g., by helping the user achieve the intended result with the necessary inputs, reducing user errors when operating/interacting with the device), and the battery life of the device is improved (e.g., by helping the user use the device more quickly and efficiently).
In some embodiments, while displaying the user interface in the second mode without displaying the affordances, the device detects (7028) an input that satisfies a second affordance redisplay criterion, where the second affordance redisplay criterion is satisfied when the input is a request to transition from displaying the user interface in the second mode to displaying the user interface in the first mode, and in response to detecting the input that satisfies the second affordance redisplay criterion, the device redisplays the user interface in the first mode, and the device redisplays the affordances on the user interface redisplayed in the first mode. This is shown, for example, in FIGS. 5X-5AA. In some embodiments, when the affordances are redisplayed on the user interface displayed in the first mode, values of the set of one or more display characteristics of the affordances change according to a first set of one or more rules in response to a change in values of the set of one or more display characteristics of the portion of the user interface. By re-displaying the affordances when transitioning from the second display mode to the first display mode, operability of the device is improved (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for the desired result without unduly distracting the user, thereby reducing user errors while operating the device), the interface between the user and the device is more efficient (e.g., by helping the user achieve the intended result with the necessary inputs and reducing user errors while operating/interacting with the device), and battery life of the device is improved (e.g., by helping the user use the device more quickly and efficiently).
In some embodiments, at least one of the first appearance and the second appearance of the affordance is dynamically adjusted (7030) in response to changes occurring in a portion of the user interface underlying the affordance (e.g., the portion of the user interface changes due to dynamic content changes in the user interface or navigation through content displayed in the user interface). This is shown, for example, in FIGS. 5T-5W. For example, as the user interface is scrolled, or when the user interface displayed in the second mode is a full-screen movie, game, or web page that is continuously evolving and refreshing, the appearance of the affordance is also continuously updated to reflect changes in the underlying user interface. When the user interface is displayed in the first mode, the appearance of the affordance dynamically changes according to a first set of rules, and when the user interface is displayed in the second mode, the appearance of the affordance dynamically changes according to a second set of rules. By dynamically changing the appearance of affordances based on changes in the appearance of the underlying content, the usability of the device is increased (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for a desired result without unduly distracting the user, while the content continues to change over time), the interface between the user and the device is more efficient (e.g., by helping the user achieve the intended result with the necessary inputs and reducing user errors when operating/interacting with the device), and the battery life of the device is improved (e.g., by helping the user use the device more quickly and efficiently).
In some embodiments, a first appearance is generated (7032) based on a first set of filters (e.g., blur, desaturate, and invert) applied to a portion of the user interface underlying the affordance (and edge regions surrounding that region), and a second appearance is generated based on a second set of filters applied to the portion of the user interface underlying the affordance, where for two or more filters in the second set of filters, the first set of filters includes corresponding filters of the same type but with different adjustment parameters (e.g., blur filters with different blur radii, inverted filters with different inversion curves, opacity filters with different transparency levels, desaturation filters with different desaturation ratios, etc.). Providing affordances that change their appearance differently based on the appearance of the underlying content using a set of filters with different adjustment parameters allows the affordances to maintain a relatively consistent appearance between display mode switches, reducing distraction to the user and thereby improving usability of the device (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for a desired result without unduly distracting the user, reducing user errors while operating the device) and making the interface between the user and the device more efficient (e.g., by helping the user achieve intended results with the required inputs and reducing user errors when operating/interacting with the device).
In some embodiments, in response to detecting the request, the device generates (7034) one or more intermediate affordance appearances for the affordance between the first appearance and the second appearance, where the device displays the one or more intermediate appearances of the affordance as a transition between displaying the affordance having the first appearance and displaying the affordance having the second appearance. In some embodiments, the intermediate appearances are displayed as a second mode on the user interface, where the intermediate appearances are an interpolation between the first appearance and the second appearance of the affordance. By generating intermediate affordance appearances that bridge affordance appearance transitions during display mode switching of an application user interface, the user is less distracted, thereby improving usability of the device (e.g., by providing the user with an appropriate amount of guidance regarding the inputs required for a desired result without unduly distracting the user, reducing user errors while operating the device) and the interface between the user and the device is more efficient (e.g., by helping the user achieve the intended result with the required inputs and reducing user errors while operating/interacting with the device).
In some embodiments, generating one or more intermediate appearances for an affordance between the first appearance and the second appearance includes gradually transitioning from a first set of rules to a second set of rules (7036) (e.g., varying the magnitude of one filter applied to the underlying content to generate the affordance while not changing the magnitude of another filter applied to the underlying content to generate the affordance). By gradually transitioning the set of rules used to generate the appearance of the affordance, generating intermediate appearances for the affordance may be less distracting to the user, thereby increasing usability of the device (e.g., by providing the user with an appropriate amount of guidance regarding inputs required for a desired result without unduly distracting the user, thereby reducing user errors during device operation) and the interface between the user and the device may be more efficient (e.g., by helping the user achieve an intended result with the required inputs and reducing user errors during operation/interaction with the device).
In some embodiments, the affordance having a first appearance has a first degree of opacity (7038), and the affordance having a second appearance has a second degree of opacity that is less than the first degree of opacity (e.g., but the color of the user interface is visible). Varying the opacity of the affordance based on the display mode of the application user interface helps to adjust the visibility of the affordance depending on the display mode of the application user interface, thereby improving device operability (e.g., by reasonably balancing the requirements of sufficient visibility with the requirements of affordance unobtrusiveness, and helping to provide the user with the necessary input to achieve the desired results) and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, the affordance having the first appearance and the affordance having the second appearance have the same size and position on the display (7040). Maintaining the size and position of the affordances in different display modes of an application user interface helps maintain continuity in the appearance of the affordances during switching between user interfaces, thereby improving usability of the device (e.g., by maintaining context for the user and helping the user provide necessary input to achieve a desired result) and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
In some embodiments, the first set of one or more rules creates a visual difference between the affordance and the application's user interface that is greater than the visual difference between the affordance and the application's user interface created by the second set of one or more rules (7042). Varying the visual clarity of the affordance based on the display mode of the application user interface helps to adjust the visibility of the affordance depending on the display mode of the application user interface, thereby improving device operability (e.g., by reasonably balancing the requirements of sufficient visibility with the requirements of affordance unobtrusiveness, and helping provide the user with the necessary inputs to achieve the desired results) and making user-to-device operation more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device).
It should be understood that the particular order described for the operations in Figures 7A-7E is merely an example, and that the described order is not intended to indicate the only order in which the operations may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 6000 and 8000) are also applicable in a similar manner to method 7000 described above with respect to Figures 7A-7E. For example, the contacts, gestures, user interface objects, application views, control panels, controls, affordances, position thresholds, directional conditions, inversion curves, filters, value ranges, navigation criteria, movement parameters, focus selectors, and/or animations described above with respect to method 7000, optionally have one or more of the features of the contacts, gestures, user interface objects, application views, control panels, controls, position thresholds, directional conditions, navigation criteria, movement parameters, focus selectors, and/or animations described herein with respect to other methods described herein (e.g., methods 6000 and 8000). For brevity, these details are not repeated here.
Operations in the information processing methods described above are optionally performed by executing one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as described above in connection with Figures 1A and 3) or an application-specific chip.
The operations described above with reference to Figures 7A-7E are optionally performed by the components depicted in Figures 1A-1B. For example, the detection and display operations are optionally performed by event sorter 170, event recognizer 180, and event handler 190. Event monitor 171 of event sorter 170 detects a contact on touch-sensitive display 112, and event dispatcher module 174 delivers the event information to application 136-1. A corresponding event recognizer 180 of application 136-1 compares the event information to a corresponding event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface (or a rotation of the device) corresponds to a predefined event or sub-event, such as a selection of an object on a user interface or a rotation of the device from one orientation to another. When a corresponding predefined event or sub-event is detected, event recognizer 180 activates event handler 190 associated with the detection of the event or sub-event. Event handling unit 190 optionally uses or calls data update unit 176 or object update unit 177 to update application internal state 192. In some embodiments, event handling unit 190 accesses a corresponding GUI update unit 178 to update what is displayed by the application. Similarly, it will be clear to one skilled in the art how other processes can be implemented based on the components depicted in Figures 1A-1B.
8A-8F are flow diagrams illustrating a method for changing affordance appearance and affordance appearance type in response to changes in the appearance of underlying content, according to some embodiments. Method 8000 is performed in an electronic device having a display and a touch-sensitive surface (e.g., device 300 of FIG. 3, or portable multifunction device 100, FIG. 1A). In some embodiments, the electronic device includes one or more sensors for detecting an intensity of contact with the touch-sensitive surface. In some embodiments, the touch-sensitive surface and the display are integrated into a touch-sensitive display. In some embodiments, the display is a touchscreen display, and the touch-sensitive surface is on or integrated into the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 8000 are optionally combined and/or the order of some operations is optionally changed.
The method 8000 relates to a method of displaying an affordance (e.g., an affordance indicating an acceptable starting area for a gesture to perform a predefined operation in a user interface, such as a home screen or an application switching user interface) on content in a manner where the display characteristics (e.g., gray values or luminance values) of the affordance change dynamically based on changes in the same display characteristics of the underlying content. In addition, the method requires that the value range of the affordance appearance switches between two different value ranges (e.g., a "dark affordance" value range and a "light affordance" value range) depending on whether a predefined range-switching criterion is met by a change in the display characteristics of the underlying content. As a result, to maintain visual contrast between affordances and the underlying content, the device adapts the appearance of the affordance to changes in the underlying content and can present the affordances in a less distracting or intrusive manner (e.g., by avoiding rapid flickering of the affordances that could be distracting to the user in some circumstances due to changing the affordance's appearance too rapidly) without being constrained by an initial choice of affordance appearance type (e.g., a "light" or "dark" affordance type). By providing dynamically changing appearances for affordances based on the appearance of underlying content in the manner described herein, and allowing dynamic switching of value ranges of affordance appearances over time based on changes in the underlying content, the usability of the device is increased (e.g., by providing the user with visual guidance regarding the inputs required for a desired result without unduly distracting the user, thereby reducing user errors while operating the device), the interface between the user and the device is more efficient (e.g., by helping the user achieve the intended result with the required inputs and reducing user errors while operating/interacting with the device), and the battery life of the device is improved (e.g., by helping the user use the device more quickly and efficiently). Providing affordances in the manner described herein allows on-screen affordances to effectively replace and improve hardware buttons that provide the same functionality (e.g., displaying a home screen) in many different user interface contexts, helping to reduce manufacturing and maintenance costs of the device. Providing affordances in the manner described herein also helps to reduce or eliminate display burn-in issues.
The method 8000 is performed on a device having a display and a touch-sensitive surface (e.g., a touch screen display that serves as both the display and the touch-sensitive surface). The device displays (8002) on the display content (e.g., a home screen, a widget screen, a desktop, a user interface of an application, a user interface of a media player, etc.) and an affordance (e.g., a home affordance indicating an area on the display that is responsive to a home gesture), such that the affordance is displayed over a portion of the content, and values of display characteristics of the affordance (e.g., gray values or luminance values of an image (e.g., a color image or a monochrome image), intrinsic display parameters other than gray values or luminance values (e.g., hue, saturation, etc. of a full-color image), or derived display parameters calculated based on one or more intrinsic display parameters (e.g., gray values or luminance values of a full-color image, or slight variations or equivalents thereof)) are indicative of the amount of display characteristic information that the affordance may have over the portion of the content that is displayed. is determined based on values of the same display characteristics of the portion of the content being shown, where the values of the display characteristics of the content are allowed to vary within a first value range (e.g., a range of [0, 1], optionally on a scale from 0% luminance to 100% luminance, where 0% luminance is 0 and 100% luminance is 1), and the values of the display characteristics of the affordances are constrained to vary within a value range of affordance appearance that is smaller than the first value range (e.g., optionally on a scale from black to white, where black is 0 and white is 1, an affordance appearance value range of [0, 0.4] for a dark affordance type, or [0.6, 1] for a light affordance type, both smaller than the content's value range [0, 1], optionally on a scale from 0% luminance to 100% luminance, where 0% luminance is 0 and 100% luminance is 1). While displaying the content and the affordance, and while the affordance appearance value range is in a second value range (e.g., optionally on a black-to-white scale where black is 0 and white is 1, the affordance appearance value range for a dark affordance is in [0, 0.4]), the device detects (8004) an appearance change in the content over which the affordance is displayed. In response to detecting an appearance change in the content over which the affordance is displayed, the device changes (8006) the appearance of the affordance (e.g., based on a short-time scale appearance change strategy and a long-time scale appearance change strategy) and changes the value range of the affordance appearance to a third value range pursuant to a determination that the change in content appearance satisfies a range-switching criterion (the range-switching criterion is met when a measure of the overall lightness or darkness of the content below and surrounding the affordance (e.g., a cumulative and integrated value for a display characteristic (e.g., gray value or luminance) due to a change in the appearance of the background content) crosses a first threshold). (e.g., optionally on a black-to-white scale where black is 0 and white is 1, a value range of [0.6, 1] of affordance appearance for a light affordance), the third value range being different from the second value range (e.g., the third value range includes at least one value that is not included in the second value range, and optionally does not overlap with the second value range) (e.g., when a range-switching criterion is met, a currently selected affordance type changes from a previously selected affordance type (e.g., a dark affordance) to an alternative affordance type (e.g., a light affordance)), on a black-to-white scale where black is 0 and white is 1, [0.6, 1]) is less than a first value range; changing a value of a display characteristic of the affordance in response to a value of the same display characteristic of the content over which the affordance is displayed (e.g., the value of the display characteristic for each pixel of the affordance is determined based on a first predefined value mapping that corresponds to a currently selected affordance type (e.g., a light affordance), and the display characteristic of the affordance is constrained to vary within a value range of the affordance appearance; and determining if the change in the appearance of the content satisfies the range-switching criterion. 5AD-AE. In response to the determination that the affordance type is not selected, the method includes: maintaining the value range of the affordance appearance as a second value range (e.g., when the first criterion is met, the currently selected affordance type remains the same as the previously selected affordance type (e.g., dark affordance)), while varying a value of a display characteristic of the affordance according to a value of the same display characteristic of the content over which the affordance is displayed (e.g., the value of the display characteristic for each pixel of the affordance is determined based on a second predefined transformation relationship that corresponds to the currently selected affordance type (e.g., dark affordance)). This is shown, for example, in FIGS. 5AD-AE.
In some embodiments, the range switch criteria include a range switch trigger criterion and a range switch completion criterion (8008), where the range switch trigger criterion requires that the content appearance change include a first amount of change (e.g., a biased running average of a display characteristic, such as an integrated luminance value for a lower portion of the affordance and a portion of the content below the affordance) over a first time interval that causes a predefined measurement of the content appearance to cross a predefined threshold (e.g., a first threshold when switching from a light affordance to a dark affordance and a second threshold different from the first threshold when switching from a dark affordance to a light affordance), and the range switch completion criterion requires that the content appearance change include a second amount of change over a second time interval after the first time interval to again satisfy the range switch trigger criterion after the range switch trigger criterion is met and before the predefined transition time (e.g., 5 seconds) expires. By requiring that the range-switching trigger criteria not be met again within a defined transition period after the first time they are met in order to complete a range switch of an affordance appearance between two value ranges, usability of the device is improved (e.g., by avoiding distracting the user unnecessarily by switching affordance appearance types when the underlying content change is transient), the interface between the user and the device is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and battery life of the device is improved (e.g., by reducing processor and screen demands due to range switching).
In some embodiments, the default thresholds used for the range-switching trigger criteria are selected (8010) based on a current value range being used as the value range of the affordance appearance (e.g., a value range for the currently used affordance type), including using a first threshold as the default threshold when the value range of the affordance appearance is a second value range, and using a second threshold as the default threshold when the value range of the affordance appearance is a third value range. (In some embodiments, when the value range of the affordance appearance is between a second value range (e.g., a value range associated with "dark affordance") and a third value range (e.g., a value range associated with "light affordance"), the default threshold is based on the value range most recently selected as the value range of the affordance appearance (e.g., once the "light affordance" range of affordance appearance values is selected as the target range of affordance appearance values, the "light affordance" threshold is used to determine when to switch back to the "dark affordance" range of affordance appearance values, and once the "dark affordance" range of affordance appearance values is selected as the target range of affordance appearance values, the "light affordance" threshold is used to determine when to switch back to the "dark affordance" range of affordance appearance values). (A "dark affordance" threshold is used to determine when to switch back to the "light affordance" range.) Using different thresholds as range-switching trigger criteria may increase usability of the device (e.g., by adjusting the bias toward the currently selected affordance appearance type when underlying content changes are transient, avoiding distracting the user by unnecessarily switching affordance appearance types), the interface between the user and the device may be more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and the battery life of the device may be improved (e.g., reducing processor and screen demands due to range switching).
In some embodiments, varying the appearance of the affordance in response to detecting a change in appearance of the content over which the affordance is displayed (8012) includes changing the value range of the affordance appearance to an intermediate value range different from the second value range and the third value range (e.g., optionally, an intermediate value range [0.3, 0.7] on a black-to-white scale where black is 0 and white is 1, with a start value between the second value range and a start value of the third value range) in accordance with a determination that the range-switch trigger criterion is met by a first amount of change during the first time interval and before the range-switch completion criterion is met. and shifting a display characteristic of the affordance to a second value range having a start value and an end value that is between an end value of the second value range and an end value of a third value range, and changing a value of a display characteristic of the affordance in response to a value of the same display characteristic of the content over which the affordance is displayed (e.g., the value of the display characteristic for each pixel of the affordance is determined based on a first predefined value mapping that corresponds to a currently selected affordance type (e.g., a light affordance)), where the display characteristic of the affordance is constrained to vary within the value range of the affordance appearance. In some embodiments, there are multiple intermediate value ranges between the second value range and the third value range, and the device moves sequentially through each of the multiple intermediate value ranges over a predefined transition time until a range-switching completion criterion is met. After the range-switching completion criterion is met, when the display characteristic of the affordance varies in response to any additional appearance changes of the content, the display characteristic of the affordance is constrained to be within the third value range. By shifting the value range of the affordance appearance to an intermediate value range that is different from (e.g., between) the values of the appearance ranges of the two stable affordance appearance types as a transition during switching between the two stable affordance appearance types, usability of the device is improved (e.g., by making switching between affordance appearance types more subtle and less distracting to the user), the interface between the user and the device is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and battery life of the device is improved (e.g., by allowing the user to use the device more quickly and efficiently).
In some embodiments, the content appearance change includes a third amount of change over a third time interval after the first time interval after the range-switch trigger criterion is met and before the range-switch completion criterion is met (8014). For example, after the range-switch trigger criterion is met, the device selects an intermediate content-affordance appearance inversion curve that is located between the content-affordance appearance inversion curves for the light and dark affordances, and uses the intermediate inversion curve to determine how to change the luminance of the affordance based on the change in the luminance of the content during a transition period (e.g., within 5 seconds of meeting the range-switch trigger criterion). By allowing the appearance of affordances to continue to change with the underlying content while the range of affordance appearances shifts from one stable value range to another over time, usability of the device is improved (e.g., by providing visual continuity in the appearance of affordances while switching between affordance appearance types), the interface between the user and the device is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and the battery life of the device is improved (e.g., by allowing the user to use the device more quickly and efficiently).
In some embodiments, the change in appearance of the content over which the affordance is displayed is caused by scrolling of the content (8016), and the range-switching criteria do not require an absence of scrolling of the content to be met. By allowing the appearance of the affordance to change with the underlying content while the underlying content is scrolling, usability of the device is improved (e.g., by providing visual continuity in the appearance of the affordance while the underlying content is scrolling), the interface between the user and the device is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and battery life of the device is improved (e.g., by allowing a user to use the device more quickly and efficiently).
In some embodiments, a change in appearance of the content over which the affordance is displayed is caused (8018) by movement of the content below the affordance (e.g., zooming or scrolling the content), and the range-switching criteria require that the content below the affordance moves by less than a predefined amount for at least a predetermined period of time for the range-switching criteria to be met. For example, in addition to requiring that a predefined measure of the content's appearance (e.g., a biased running average of a display characteristic, such as the integrated luminance value for the portion of the content below the affordance and the portion of the content below the affordance) crosses a predefined threshold, the range-switching criteria further require that the content remain substantially stationary for a short period of time around the time the predefined threshold is crossed. By requiring the content underlying an affordance to be substantially stationary (e.g., stop scrolling) in order to meet the range switching criteria, usability of the device is improved (e.g., to avoid unnecessarily switching back and forth between affordance appearance types when continued scrolling is causing content to change quickly), the interface between the user and the device is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and the battery life of the device is improved (e.g., by allowing the user to use the device more quickly and efficiently).
In some embodiments, changing the value of the display characteristic of the affordance in response to the value of the same display characteristic of the content includes: increasing the value of the display characteristic of the affordance in response to an amount of change (e.g., decrease) in the value of the display characteristic of the content in response to a determination that the value of the display characteristic of the content has decreased (and optionally, the value of the display characteristic of the content is within a predefined sub-range of the first value range (e.g., outside a range of two special content values where there is a discontinuity in the affordance values)), and increasing the value of the display characteristic of the affordance in response to an amount of change (e.g., increase) in the value of the display characteristic of the content in response to a determination that the value of the display characteristic of the content has increased (and, optionally, the value of the display characteristic of the content is within a predefined sub-range of the first value range (e.g., outside a range of two special content values where there is a discontinuity in the affordance values)), (e.g., the affordance becomes darker when the content underlying the affordance becomes lighter). By using value inversion (e.g., increasing affordance values as content values decrease and decreasing affordance values as content values increase) to determine values of display characteristics of affordances based on values of the same display characteristics of the underlying content, usability of the device may be improved (e.g., by providing sufficient visibility of the affordances without distraction or obstruction), the interface between the user and the device may be more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and battery life of the device may be improved (e.g., by allowing the user to use the device more quickly and efficiently).
In some embodiments, when the value of the display characteristic of the affordance varies within a second range of values (e.g., before the range-switching criterion is met) and when the value of the display characteristic of the affordance varies within a third range of values (e.g., before the range-switching criterion is met), if the value of the display characteristic of the content changes by a given amount, the value of the display characteristic of the affordance changes by the same amount (8022). This is shown, for example, in Figures 5AE and 5AF. In some embodiments, the content-affordance appearance inversion curve (e.g., affordance luminance vs. background luminance curve) between the display characteristic of the affordance and the same display characteristic of the content has the same shape for dark and light affordances, and the luminance value for each pixel of the affordance is looked up from a database of pre-stored luminance values for the currently selected affordance type. By requiring the content underlying an affordance to be substantially stationary (e.g., stop scrolling) in order to meet the range switching criteria, usability of the device is improved (e.g., to avoid unnecessarily switching back and forth between affordance appearance types when continued scrolling is causing content to change quickly), the interface between the user and the device is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and the battery life of the device is improved (e.g., by allowing the user to use the device more quickly and efficiently).
In some embodiments, the device shifts (8024) a value range of the affordance appearance from a second value range (e.g., a value range of the affordance appearance for a dark affordance) to a third value range (e.g., a value range of the affordance appearance for a light affordance), gradually shifting from the second value range to the third value range over a period of time (e.g., 5 seconds), the method including: detecting, while gradually shifting the value range of the affordance appearance from the second value range to the third value range, an additional appearance change of the content over which the affordance is displayed; and, in response to detecting the additional appearance change of the content over which the affordance is displayed, shifting the value range of the affordance appearance according to the value range of the affordance appearance. and changing the appearance of the affordance (e.g., the value range of the affordance appearance shifts incrementally over time) including beginning to incrementally shift the value range of the affordance appearance back to the second value range (e.g., shifting the start and end points of the value range over a time period of 1, 2, 3, 4, 5, 10 seconds, etc.) in accordance with a determination that the appearance change of the content meets the range-switching criterion (e.g., the range-switching criterion is met for a second time), and continuing to incrementally shift the value range of the affordance appearance from the second value range to a third value range in accordance with a determination that the appearance change of the content does not meet the range-switching criterion (e.g., the range-switching criterion is met for a second time). In some embodiments, as described above, the default threshold used to determine whether the range-switching criterion is met varies based on whether the target value range for the value range of the affordance appearance is the second value range (e.g., a dark affordance) or the third value range (e.g., a light affordance). This method allows an affordance to be gradually switched from one appearance type to another over a defined transition period, during which the appearance value of the affordance continues to change in response to changes in the appearance value of the underlying content. Furthermore, while the appearance value range is shifting, the range-switching criteria may be satisfied again (e.g., due to continued changes in the content and appearance of the affordance). As a result, the range switch that was triggered is enabled by the device to either reverse or continue and terminate depending on whether the range-switching criteria is satisfied again during the transition period. The mechanism for reversing the range switch during the transition period improves the usability of the device (e.g., to avoid unnecessary switching back and forth between affordance appearance types when content is changing quickly), the interface between the user and the device is more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and improves the battery life of the device (e.g., by allowing the user to use the device more quickly and efficiently).
In some embodiments, after changing the value of the display characteristic of the affordance within a third value range in response to determining that the change in appearance of the content satisfies the range-switching criterion, the device detects (8026) an additional change in appearance of the content over which the affordance is displayed, and in response to detecting the additional change in appearance of the content over which the affordance is displayed, the device changes the appearance of the affordance, and in response to determining that the additional change in appearance of the content has satisfied the range-switching trigger criterion for the first time and has not satisfied the range-switching completion criterion, changes a value of the display characteristic of the affordance in response to a value of the same display characteristic of the content over which the affordance is displayed (e.g., the value of the display characteristic for each pixel of the affordance is determined based on a first predefined value mapping that corresponds to a currently selected affordance type (e.g., a light affordance)), in response to determining that the additional appearance changes of the content have satisfied the range switch trigger criterion a second time without satisfying the range switch completion criterion after the first time, changing a value of a display characteristic of the affordance in response to a value of the same display characteristic of the content over which the affordance is displayed, the display characteristic of the affordance being constrained to vary within a second intermediate value range between the third value range and the first intermediate value range; and in response to determining that the additional appearance changes of the content have satisfied the range switch trigger criterion only once before satisfying the range switch completion criterion, changing a value of the display characteristic of the affordance in response to a value of the same display characteristic of the content over which the affordance is displayed, the display characteristic of the affordance being constrained to vary within the second value range. After the affordance switches from the first appearance type to the second appearance type (e.g., in response to the range-switching criteria being met for the first time), if the range-switching trigger criteria are met again, the affordance may switch from the second appearance type back to the first appearance type. During the defined transition period, the value range of the appearance continues to shift from the value range of the second appearance type to the value range of the first appearance type. While the value range of the appearance is shifting, the range-switching trigger criteria may also be met next (e.g., due to continued changes in the content and appearance of the affordance). As a result, the range switch that was triggered may be reversed from its current state and again toward the value range of the second appearance type. A mechanism for reversing range switching during transition periods may improve usability of the device (e.g., to avoid unnecessarily switching affordance appearance types back and forth when content is changing quickly), make the interface between the user and the device more efficient (e.g., by reducing user distractions and reducing user errors while using or interacting with the device), and improve the battery life of the device (e.g., by allowing the user to use the device more quickly and efficiently).
In some embodiments, when a home affordance is initially displayed (e.g., a context switch occurs that causes a change in the content displayed on the screen), the device does not accumulate a sufficient amount of information about the underlying content to actively select between a light and a dark affordance based on a measurement of the content appearance in the manner described in various embodiments. Thus, the device optionally sets the value of the affordance's display characteristics to a predefined value, such as an intermediate value (e.g., 0.5, or 50% of the full luminance value), and lets the dynamic algorithms described above update the affordance appearance value (e.g., in response to both the short-time scale strategy and the long-time scale strategy described herein). In some embodiments, when the affordance is initially displayed, the progress rate for switching affordance appearance types is temporarily increased (e.g., the bias toward maintaining the currently selected affordance type is reduced), so that the home affordance quickly adapts to either a light or a dark affordance based on the appearance of the underlying content. In some embodiments, the situation where the home affordance is initially displayed is triggered by a context-switching event in the user interface, such as when a new application user interface is displayed (e.g., via selecting a recently opened application in a multitasking user interface or launching an application from an application icon on the home screen), when a cover sheet user interface (e.g., a system-level information screen) is pulled down from the edge of the display to cover a currently displayed application user interface or home screen, or when a user interface is rotated (e.g., by rotating the device), etc. In some embodiments, the affordance is initially displayed with an animated transition that includes gradually increasing the opacity of the affordance over time, sliding the affordance up the screen from the edge of the screen over time, and/or gradually increasing the size of the affordance over time.
It should be understood that the particular order described for the operations in Figures 8A-8F is merely an example, and that the described order is not intended to indicate the only order in which the operations may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 6000 and 7000) are also applicable in a similar manner to method 8000 described above with respect to Figures 8A-8E. For example, the contacts, gestures, user interface objects, application views, control panels, controls, affordances, position thresholds, directional conditions, inversion curves, filters, value ranges, navigation criteria, movement parameters, focus selectors, and/or animations described above with respect to method 8000, optionally have one or more of the features of the contacts, gestures, user interface objects, application views, control panels, controls, position thresholds, directional conditions, navigation criteria, movement parameters, focus selectors, and/or animations described herein with respect to other methods described herein (e.g., methods 6000 and 7000). For brevity, these details are not repeated here.
Operations in the information processing methods described above are optionally performed by executing one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as described above in connection with Figures 1A and 3) or an application-specific chip.
8A-8F are optionally implemented by the components depicted in FIGS. 1A-1B. For example, the detect, shift, and change operations are optionally implemented by event sorter 170, event recognizer 180, and event handler 190. Event monitor 171 of event sorter 170 detects a contact on touch-sensitive display 112, and event dispatcher module 174 delivers the event information to application 136-1. A corresponding event recognizer 180 of application 136-1 compares the event information to a corresponding event definition 186 to determine whether a first contact at a first location on the touch-sensitive surface (or a rotation of the device) corresponds to a predefined event or sub-event, such as a selection of an object on a user interface or a rotation of the device from one orientation to another. When a corresponding predefined event or sub-event is detected, event recognizer 180 activates event handler 190 associated with the detection of the event or sub-event. Event handling unit 190 optionally uses or calls data update unit 176 or object update unit 177 to update application internal state 192. In some embodiments, event handling unit 190 accesses a corresponding GUI update unit 178 to update what is displayed by the application. Similarly, it will be clear to one skilled in the art how other processes can be implemented based on the components depicted in Figures 1A-1B.
The above description has been described with reference to specific embodiments for purposes of explanation. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various embodiments described, with various modifications suitable for the particular uses contemplated.
57 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55 Sheet 56 Sheet 57
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| US20160246475A1 | Cites | United States of America |
| US08091038B1 | Cites | United States of America |
| WO2014013613A1 | Cites | World Intellectual Property Organization (WIPO) |
40 members in 8 offices
Priority claims7
| Document | Office | Kind | Date |
|---|---|---|---|
| 201762556402 | United States of America | P | |
| 62556402 | United States of America | – | |
| PA201770711 | Denmark | A | |
| PA201770711 | Denmark | – | |
| 15878276 | United States of America | – | |
| 201815878276 | United States of America | A | |
| 2021176668 | Japan | A |
Members40
| Document | Office | Kind | |
|---|---|---|---|
| US2019079662A1 | United States of America | A1 | |
| WO2019050562A1 | World Intellectual Property Organization (WIPO) | A1 | |
| DK201770711A1 | Denmark | A1 | |
| CN109769396A | China | A | |
| DK179931B1 | Denmark | B1 | |
| EP3559795A1 | European Patent Office (EPO) | A1 | |
| CN110456979A | China | A | |
| JP2020504342A | Japan | A | |
| AU2020100060A4 | Australia | A4 | |
| KR20200023449A | Republic of Korea | A | |
| AU2020100060B4 | Australia | B4 | |
| AU2020100722A4 | Australia | A4 | |
| US10691321B2 | United States of America | B2 | |
| AU2020100722B4 | Australia | B4 | |
| US2020278787A1 | United States of America | A1 | |
| JP6775598B2 | Japan | B2 | |
| AU2020102351A4 | Australia | A4 | |
| AU2020102351B4 | Australia | B4 | |
| JP2021028829A | Japan | A | |
| US11119642B2 | United States of America | B2 | |
| CN110456979B | China | B | |
| JP6970265B2 | Japan | B2 | |
| US2021373755A1 | United States of America | A1 | |
| JP2022025111A | Japan | A | |
| CN114063842A | China | A | |
| KR102375026B1 | Republic of Korea | B1 | |
| KR20220036995A | Republic of Korea | A | |
| KR102402378B1 | Republic of Korea | B1 | |
| EP3559795B1 | European Patent Office (EPO) | B1 | |
| EP4083770A1 | European Patent Office (EPO) | A1 | |
| JP7265731B2 | Japan | B2 | |
| JP2023098962A | Japan | A | |
| CN109769396B | China | B | |
| CN117032541A | China | A | |
| JP7547542B2This record | Japan | B2 | |
| US12086398B2 | United States of America | B2 | |
| US2024310992A1 | United States of America | A1 | |
| CN114063842B | China | B | |
| EP4083770B1 | European Patent Office (EPO) | B1 | |
| US12455681B2 | United States of America | B2 |
10 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 7547542
- Application
- 62194
Titles2
- Japanese
- アフォーダンスを背景に表示するためのデバイス、方法、及びグラフィカルユーザインターフェース
- English
- DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR DISPLAYING AFFORDANCES IN A BACKGROUND - Patent application
Classification
- CPC, 11
- G06F3/0481
- G06F3/04847
- G09G5/02
- G06F9/451
- G06F3/0488
- G06F3/016
- G06F2203/04803
- G06F3/0482
- G06F3/04883
- G06F3/0484
- G06F3/041
- IPC, 1
- G06F3 0481
