User interface tools for cropping and straightening image
20 claims: 15 independent, 5 dependent
- 1少 なくとも1つの処理ユニット に画 像を編集するためのグラフィカルユーザインタフェース(GUI)を提供 させるためのコンピュータプログラムであって 、前記GUIは、 画像を表示するための表示エリアと、 (i) 擦りタッチ入力を受け取り、前記擦りタッチ入力はその擦りタッチ入力が維持されている間の前記画像の画素のセット上における異なる方向への複数の連続するスワイプ動作を含み 、(ii) 前記画像の前記画素のセットを変更する効果の前記画素のセットへの適用を、異なる方向への前記連続するスワイプ動作の各連続する動作のたびに徐々に増加させる、 ための画像処理ツールと、 を含む、コンピュータ プログラム 。
- 2前記 画像処理ツールが更に、前記効果の選択を受け取 る、 請求項1に記載のコンピュータ プログラム 。
- 3前記画像処理ツールは、前記画像処理ツールが閾値数のタッチ入力を受け取るまでのみ、前記効果 の前記適用 を 徐々 に 増加 する、請求項1に記載のコンピュータ プログラム 。
- 4前記 擦り タッチ入力 の前記連続するスワイプ動作の 1回目 における前記画素のセットへの変更 の量が、前記 擦り タッチ入力 の前記連続するスワイプ動作の 2回目 における前記画素のセットへの変更 の量と異なる、請求項1に記載のコンピュータ プログラム 。
- 5前記画像処理ツールが 前記 画像の前記 画素のセットに 適用する前記効果が、飽和効果、脱飽和効果、明色化効果、暗色化効果、シャープ化効果、及びソフト化効果のうちの少なくとも1つを含む、請求項1に記載のコンピュータ プログラム 。
- 6前記GUIが、異なる効果を表す選択可能なユーザインタフェース項目のセットを更に含み、前記画像処理ツールが、前記効果を表すユーザインタフェース項目の選択を受け取ることに応答して前記効果を選択する、請求項1に記載のコンピュータ プログラム 。
- 7画像編集アプリケーションを提供する方法であって、 表示エリア内に画像を表示 し 、 前記画像は複数の画素を含み、 前記画像の画素のセットの上における 擦りタッチ入力の異なる方向の 複数の 連続するスワイプ動作 を、 前記擦りタッチ入力が維持されている間に、 受け取 り 、 前記画像の前記画素のセットを変更する効果の前記画素のセットへの適用を、異なる方向への前記連続するスワイプ動作の各連続する動作のたびに徐々に増加させる、 こと を含む、 方法 。
- 8前記 効果の前記適用を徐々に増加させることは、前記画素のセット上への前記擦りタッチ入力の各連続するスワイプ動作のたびに 異なる量の 前記効果 を適用する こと を含む、請求項7に記載の 方法 。
- 9前記 効果の前記適用を徐々に増加させることは、前記画素のセットが前記擦りタッチ入力の前記連続するスワイプ動作の1つ を受け取るたびに適用 される前記効果 の量を線形的に 増加させ る こと を含む、請求項7に記載の 方法 。
- 10前記 効果の前記適用を徐々に増加させることは、前記擦りタッチ入力の各連続するスワイプ動作のたび に適用 される前記効果 の量を非線形的に 増加させること を含む、請求項7に記載の 方法 。
- 11前記 効果の前記適用を徐々に増加させることは、前記擦りタッチ入力の連続するスワイプ動作が 閾値回数受け取 られ るまでのみ 前記効果の前記適用を増加させること を含む、請求項7に記載の 方法 。
- 12前 記画像のそれぞれの画素にマスク値を付与する ことをさらに含み、前記効果の前記適用を徐々に増加させることは、前記画素のセットの前記マスク値を変更することを備える 、請求項7に記載の 方法 。
- 13少 なくとも1つの処理ユニット に、 画像を編集するためのグラフィカルユーザインタフェース(GUI)を提供 させるためのコンピュータプログラムであって 、前記GUIは、 複数の画素を含む画像を表示するための表示エリアと、 ツールと、を含み、前記ツールは、 (i )画 像の画素のセット を横断する、複数の連続動作としてのタッチ入力を 受け取り、(ii) 前記タッチ入力がその上を最初に移動した、前記画素の 第1のサブセット を識別 し、(iii) 前記第1のサブセットの画素に類似する画素を識別するための類似基準のセットを決定し、(iv)前記タッチ入力がその上を移動し、前記類似基準を満足する、前記画素の 第2のサブセットにのみ画像処理動作を適用 するためのツールであり、ここで前記タッチ入力がその上を移動し、前記類似基準を満たさない、前記画素のセットにおける画素には前記画像処理動作は適用されない、 コンピュータ プログラム 。
- 14前記 類似 基準のセットは、画素が、 前記第1のサブセットの画素の 画素値の範囲内に含まれる画素値を有するかどうかを含む、請求項 13 に記載のコンピュータ プログラム 。
- 15前記ツールは、 前記第2のサブセットの画素の上を前記タッチ入力が移動する たびに、前記画像処理動作 の適用 を 徐々に増加させる 、請求項 13 に記載のコンピュータ プログラム 。
- 16表示エリアに、 画像を表示 し 、 前記画像は複数の画素を備え、 前 記画像のエリア における画素のセットを横断する、複数の連続するスワイプ動作としてのタッチ入力を 受け取 り 、 前記タッチ入力の前記連続するスワイプ動作の1回目がその上を最初に移動する、前記画像の前記エリアにおける画素のサブセットの特性に基づいて、前記画素のサブセットにおける画素に類似する画素を識別するための基準のセットを決定し、 画像処理動作を、前記タッチ入力がその上を移動し、前記決定された基準を満足する画素のみに適用することを含み、ここで前記画像処理動作は、前記タッチ入力がその上を移動し、前記決定された基準を満たさない画素には適用されない、 方 法。
- 17前記基準のセットを決定することは、前記画素のサブセットにおける画素の画素値を、RGB色モデルの値からLAB空間の値へ変換することを含む 、請求項 16 に記載の方法。
- 18前記基準のセットを決定することは、さらに、前記エリアにおける画素が前記画素のサブセットから予め定められた距離内にあるかどうかを決定することを含む 、請求項 17 に記載の方法。
- 19少なくとも1つの処理ユニットに、請求項7乃至12および請求項16乃至18のいずれか1項に記載された方法を実行させるためのコンピュータプログラム。
- 20電子機器であって、 処理ユニットのセットと、 前記処理ユニットの少なくとも1つによって実行されたときに、請求項7乃至12および請求項16乃至18のいずれか1項に記載された方法を実行するコンピュータプログラムを格納したコンピュータ可読媒体と、 を備える電子機器。
Independent claims20
423 paragraphs, as filed
0001Digital graphic design and image editing applications (hereinafter collectively referred to as image editing applications) provide graphic designers, media artists and other users with the tools necessary to view and edit images. Examples of such applications include PICASA® provided by Google, PHOTOSHOP® provided by Adobe, Express Media provided by Microsoft, and the like. These applications give the user the ability to edit images in various ways.
0002With the increase in mobile devices such as tablet computers and smartphones, the usefulness of these applications has expanded to these devices. However, many image editing applications are adapted for fixed devices such as desktops and servers, and do not provide users with some of the useful features that mobile devices should be able to enjoy.
0003Some embodiments provide an image editing application with cropping and angle correction tools for editing the image. Crop and angle correction tools provide users with rotatable user interface (UI) items. The user can rotate this rotatable UI item in a particular direction (clockwise or counterclockwise), and the tool will rotate the displayed image in that particular direction accordingly. In this way, the user can angle-correct the displayed image by rotating the rotatable UI item.
0004As the rotatable UI item is rotated, the image editing application in some embodiments makes a sound to provide an audible instruction indicating that the rotatable UI item is rotated. The sound reproduced by the image editing application in some embodiments is the sound produced by the mechanical dial when the mechanical dial is rotated.
0005The rotatable UI item in some embodiments also includes a set of markings on the rotatable UI item. The marking is for the rotatable UI item to indicate the amount of rotation rotated. In some embodiments, the markings pointed to by the fixed knob indicate the amount of rotation that the rotatable UI item has rotated.
0006In some embodiments, as the image is rotated relative to the edge of the display area, the image editing application will avoid displaying the outer area of the image within the display area as much as possible of the image. Enlarge and / or crop the image to display the portion. In some cases, if the display area has already been magnified, the image editing application will not zoom in when the displayed image is rotated. In these cases, the outer area of the image will be displayed within the display area.
0007Crop and angle correction tools allow users to rotate rotatable UI items in several different ways. The user can rotate the user interface item by touching and dragging the rotatable UI item. The tools in some embodiments also allow the user to rotate a rotatable UI item by moving a device that is a device on which an image editing application is running. In these embodiments, the device comprises components capable of detecting and measuring the movement of the device. The image editing application receives the orientation value from the component and records the value before the device is moved. While the device is moving, the image editing application continues to receive orientation values from the components. The image editing application calculates the amount to rotate the rotatable UI item based on the difference between the recorded orientation value and the latest orientation value.
0008The user can also rotate the rotatable UI item by performing a rotate multi-touch gesture on the displayed image. When the user makes a rotation multi-touch gesture that exceeds the rotation threshold amount, the crop and angle correction tool rotates the rotatable UI item and the displayed image based on the rotation multi-touch gesture.
0009In some embodiments, the crop and angle correction tools provide an additional set of UI items in addition to the rotatable UI items. In some embodiments, the set of UI items is for rotating a rotatable UI item and utilizing a horizon to correct the angle of the displayed image. In these embodiments, the crop and angle correction tool detects the horizon of the image and displays the horizon in the image. The tool angle-corrects the image by rotating the rotatable UI item and leveling the horizon and the image in response to receiving a selection of one UI item in the set of UI items. The tool removes the horizon and set of UI items without rotating the rotatable UI item or angle-correcting the image in response to receiving a selection of another UI item within the set of UI items.
0010Crop and angle correction tools allow the user to crop the image in several different ways. The user can drag the edges or corners of the image to crop the image. When the user drags the edges or corners of the image toward the center of the image, the crop and angle correction tools cut off the edges or corners of the image that are outside the edges of the image. In some such embodiments, the crop and angle correction tool maintains the aspect ratio of the image before it is cropped while the edges or corners of the image are being dragged. In some embodiments, the user can magnify the image, and the crop and angle correction tools crop the portion of the image that extends outside the display area as a result of enlarging the image.
0011Some embodiments provide image editing applications that include effect tools for applying effects to selected areas of the displayed image. In some embodiments, the effect tool allows the user to select an area of the image to which the effect should be applied by rubbing the image with a finger or an equivalent input device. For some effects, the effect tool applies the effect to a selected area of the image, regardless of how many times the area of the image is rubbed. Such effects include a defect removing effect and a red eye removing effect. For some other effects, the effect tool incrementally applies the effect to the selected area based on the number of times the user rubs the area of the image. Such effects include saturation, desaturation, lightening, darkening, sharpening, softening and the like.
0012In some embodiments, the effect tool assigns a mask value to each pixel of the image. The effect tool uses mask values to process the pixels of the image. Due to the effect that the effect tool applies incrementally to the selected area of the image, the effect tool is within the selected area to adjust the amount of effect applied to the area each time the area is touched. The mask value given to each pixel of is changed.
0013In some embodiments, the effect tool applies a different amount of effect to the area each time the area is touched. That is, the amount of effect that the effect tool applies to the area the first time the area is touched is different from the amount of effect that the effect tool applies to the area the second time the area is touched. In some embodiments, the amount of effect the effect tool applies to the area is a function of the number of times the area is touched. The function may be linear, non-linear, or a combination of linear and non-linear functions.
0014In some embodiments, the effect tool selectively applies the effect to a subset of pixels within a selected area of the image. The effect tool identifies a set of evaluation criteria based on the image characteristics of a set of pixels within an area of the image. The effect tool uses evaluation criteria to identify pixels with similar characteristics. The effect tool applies the effect only to those pixels that meet the specified set of evaluation criteria. In this way, the effect tool applies the effect only to similar pixel groups in the image, leaving the other pixels in the area of the image untouched.
0015In some embodiments, the effect tool allows a touch of a certain size. That is, when the user touches the location of the displayed image, the effect tool applies the effect to pixels within a certain distance from that location. This distance is constant with respect to the size of the display area in which the image is displayed. To allow fine-grained application of effects, effect tools allow the user to magnify the image so that the touch selects fewer pixels (ie, smaller areas).
0016Some embodiments provide image editing applications that include on-image tools for applying different effects to images. The image editing application overlays some on-image tools on the displayed image. These superposed image tools are therefore visible to the user. Other on-image image tools are not visible to the user.
0017The tilt shift tool is a visible image tool. The tilt shift tool in some embodiments includes two visible horizontal lines that are superimposed on the image. The tilt shift tool blurs the area of the image above the upper horizon and the area of the image below the lower horizon. The tilt shift tool allows the user to move both horizon together by touching and dragging the space between the two horizon. The tilt shift tool allows the user to adjust the distance between two horizontal lines by dragging one or both lines vertically.
0018Vignette tools are invisible image tools for image editing applications. In some embodiments, the vignette tool darkens the pixels of an image based on the distance that each pixel of the image is away from the location of the image being touched. In some such embodiments, the vignette tool uses a sigmoid function to apply the darkening effect. Using the sigmoid function, the vignette tool applies little or no darkening effect to pixels in the image within the first distance from the location, and the maximum amount for pixels in the image a second distance from the location. Apply the darkening effect of. The vignette tool gradually applies a darkening effect to the pixels between the first and second distances from the location.
0019The vignette tool in some embodiments defines an invisible geometry around the location of the image touched by the user, darkening the area of the image outside the geometry. The user can resize the geometry and the user can touch other locations in the image and apply vignette effects based on the other locations.
0020The gradient tool is another invisible image tool for image editing applications. The gradient tool applies a gradient effect from the top of the displayed image to the location of the touched image. The user can define the area to which the gradient effect should be applied by touching other locations in the image.
0021The gradient tool in some embodiments defines and uses several different types of gradient effects. To define the type of gradient effect, the tool produces a black and white image based on the original image. In some embodiments, the tool emphasizes one color in a black and white image. The tool then produces a mixed image by mixing the black and white image with the original image. In some embodiments, the tool mixes the two images by averaging the pixel values of the corresponding pixels in the two images or by multiplying the pixel values of the corresponding pixels in the two images. To do. The tool then produces a blended image by blending the mixed image with the original image.
0022The above summary is for a brief introduction to some embodiments of the present invention. It does not mean an introduction or an outline of all the contents of the invention disclosed in the present specification. The following detailed description, and the drawings cited in the detailed description, further describe the embodiments described in the abstract as well as other embodiments. Therefore, in order to understand all the embodiments described herein, a thorough review of summaries, detailed descriptions and drawings is required. Furthermore, the claimed content is not limited by the abstract, detailed description and exemplary details in the drawings, and the claimed content can be implemented in other particular forms that do not deviate from the intent of the subject matter. Therefore, it is defined by the attached claims.
0023<figref num="1">It conceptually illustrates the use of cropping and angle correction tools.</figref><figref num="2">It is a conceptual demonstration of the use of the tilt shift tool.</figref><figref num="3">It is a conceptual demonstration of the use of vignette tools.</figref><figref num="4A">It conceptually illustrates the use of cropping and angle correction tools.</figref><figref num="4B">It conceptually illustrates the use of cropping and angle correction tools.</figref><figref num="5">Some embodiments conceptually describe the process performed to allow the user to angle-correct the image.</figref><figref num="6">It conceptually shows that the angle of an image is corrected without enlarging the image.</figref><figref num="7">It conceptually shows that the image is angle-corrected by rotating the device on which the image editing application operates.</figref><figref num="8">Several embodiments conceptually describe the process performed to allow the user to angle-correct an image by rotating the device.</figref><figref num="9">It conceptually shows that the image is angle-corrected by rotating the device on which the image editing application operates.</figref><figref num="10">It conceptually shows that the image is angle-corrected by performing a multi-touch gesture.</figref><figref num="11">Some embodiments conceptually describe the process performed to allow the user to angle-correct the image.</figref><figref num="12">It conceptually shows that the image is angle-corrected by performing a multi-touch gesture.</figref><figref num="13">It conceptually shows the use of horizontal lines to correct the angle of an image.</figref><figref num="14">It is a conceptual indication of closing the horizon.</figref><figref num="15">Some embodiments conceptually describe the process performed to allow the user to angle-correct the image by utilizing the horizon.</figref><figref num="16">It conceptually shows that an image is cropped by dragging the border of the image.</figref><figref num="17">Some embodiments conceptually represent the process performed to crop an image.</figref><figref num="18A">It conceptually shows that an image is cropped by enlarging the image.</figref><figref num="18B">It conceptually shows that an image is viewed by enlarging and moving the image.</figref><figref num="18C">It conceptually shows that an image is viewed by enlarging and moving the image.</figref><figref num="19">It conceptually shows that the image is cropped based on the selected preset aspect ratio.</figref><figref num="20">It conceptually shows that the effect is applied to the image by rubbing the image.</figref><figref num="21">Some embodiments conceptually describe the process performed to allow a user to edit an image by rubbing the device.</figref><figref num="22">It conceptually shows the enlargement of an image in order to apply the effect to the image using fine-grained control.</figref><figref num="23">It conceptually shows that the red-eye removal effect is applied to the image.</figref><figref num="24">It conceptually shows that the effect is gradually applied to the image by rubbing the image.</figref><figref num="25">Several embodiments conceptually describe the process performed to allow the user to incrementally apply the effect by rubbing the device.</figref><figref num="26">It is a conceptual demonstration of applying an effect to an image by applying different pressures while swiping the image.</figref><figref num="27">It is a conceptual demonstration of the use of smart edge detection tools.</figref><figref num="28">Some embodiments conceptually describe the process performed to allow the user to selectively apply effects to an image.</figref><figref num="29">It conceptually shows the use of the eraser tool.</figref><figref num="30">Some embodiments conceptually describe the process performed to remove an effect from an area of the displayed image.</figref><figref num="31">It conceptually shows the use of the gradient effect tool.</figref><figref num="32">Some embodiments conceptually describe the process performed to allow a user to apply a gradient effect to an image.</figref><figref num="33">It conceptually shows the architecture of an on-image gradient effect tool that applies a specific gradient effect to a portion of an image selected by the user.</figref><figref num="34">Some embodiments conceptually describe the process performed to apply a gradient effect to an image.</figref><figref num="35">It conceptually shows the architecture of an on-image gradient effect tool that applies a specific gradient effect to a portion of an image selected by the user.</figref><figref num="36">Some embodiments conceptually describe the process performed to apply a gradient effect to an image.</figref><figref num="37">It conceptually demonstrates the use of the tilt shift effect tool.</figref><figref num="38">The image shows the architecture of the tilt shift effect tool conceptually.</figref><figref num="39">Shown is a set of graphs for the original and blurred images.</figref><figref num="40">Some embodiments conceptually describe the process performed to apply a tilt shift effect to an image.</figref><figref num="41">It is a conceptual demonstration of the use of vignetting effect tools.</figref><figref num="42">Several embodiments conceptually describe the process performed to allow a user to apply a vignette effect to an image.</figref><figref num="43">Several embodiments conceptually describe the process performed to allow a user to apply a vignette effect to an image.</figref><figref num="44">Several embodiments conceptually represent the process performed to apply a vignette effect based on the location of the user's touch within the displayed image.</figref><figref num="45A">An example of selecting and deselecting cropping and angle correction tools through different GUIs of image editing applications of some embodiments is shown.</figref><figref num="45B">An example of selecting and deselecting cropping and angle correction tools through different GUIs of image editing applications of some embodiments is shown.</figref><figref num="46">It conceptually illustrates the software architecture of some embodiments of image viewing, editing, and organizing applications.</figref><figref num="47">An example of the GUI of the image editing application of some embodiments is shown.</figref><figref num="48">It conceptually shows the image data structure of some embodiments.</figref><figref num="49">It is an example of the architecture of a mobile computing device in which some embodiments are realized.</figref><figref num="50">It is a conceptual representation of an electronic system in which some embodiments are realized.</figref>
0024Some embodiments of the present invention provide a new on-image editing tool for editing an image displayed on the display area of an image editing application. These on-image tools allow the user to touch and operate on the on-image tools to apply image processing operations (eg, saturation, desaturation, defect removal, angle correction, cropping, etc.) to the image. to enable.
0025Some of these on-image tools are visible tools that some image editing applications of embodiments overlay on the displayed image. Other on-image tools are invisible tools. These invisible tools allow the user to apply image processing operations by touching the image directly.
0026FIG. 1 conceptually illustrates a graphical user interface (GUI) 100 of an image editing application of several embodiments that allows a user to angle-correct an image by using an on-image tool. Specifically, in this figure, the user touches a graphical tool to rotate the image to correct the angle of the image displayed in the image display area 125 at three different stages 105-115. Shown. FIG. 1 shows that the GUI 100 includes an image display area 125 and an angle correction tool 120.
0027The image editing application (not shown) in some embodiments is a software application for image editing. The image editing application, in some embodiments, runs on a computing device (eg, tablet computer, smartphone, etc.) that includes a touch or near touch screen to receive touch input. The image editing application allows the user to touch the image by touching the screen showing the image displayed in the display area of the image editing application.
0028The image display area 125 displays an image that the user wants to edit using the image editing application. In some embodiments, the image display area 125 displays an image having a resolution lower than that of the original image in order to fit the image within the image display area 125 and process the image quickly.
0029The angle correction tool 120 is a GUI item operated by the user to rotate the image displayed in the image display area 125. In some embodiments, the angle correction tool 120 is displayed as a geometry that is movable by the user (eg, swivel, rotatable, etc.). For example, as shown in FIG. 1, the angle correction tool 120 is a dial (eg, a partial circle). The angle correction tool 120 allows the user to rotate the image by turning the dial. In some embodiments, as the user turns the dial in one direction (eg, by dragging or swiping) in one direction (eg, left or right, clockwise, counterclockwise), the angle correction tool 120 responds accordingly. Then, the image displayed in the image display area 125 is rotated.
0030As shown in FIG. 1, the angle correction tool 120 in some embodiments overlaps the image display area 125. In other embodiments, the angle correction tool 120 and the display 125 do not overlap. Further, instead of having a geometry, or in combination with it, the angle correction tool 120 in some embodiments may allow the user to enter a numerical value (eg, the degree of angle) to rotate the image. Has a text field (not shown). In addition, the angle correction tool 120 in some embodiments also includes slider controls (eg, tracks and control knobs that move along the tracks) that the user can touch to correct the angle of the displayed image. Including.
0031From this, an operation example of an image editing application having GUI100 will be described. At step 105, the image editing application displays the image 126 within the image display area 125. Next, in step 110, the user places a finger 111 on the angle correction tool 120.
0032The third stage 115 shows the GUI 100 after the user has turned the angle correction tool 120 to the right (ie, clockwise). As the user turns the angle correction tool 120, the angle correction tool 120 in some embodiments rotates the image displayed within the image display area 125. As shown in the third stage 115, the user straightens the displayed image 126 (for example, the building and three people shown in the image are aligned with the horizontal line of the image display area 125 of the GUI 100. I swung the angle correction tool just enough (to level it).
0033FIG. 2 shows GUI200 of some embodiments of an image editing application that allow a user to apply a tilt shift effect (ie, selective focus effect) to a displayed image by touching the image. Shown conceptually. Specifically, this figure shows that the user applies the tilt shift effect to the image 225 by touching and operating the tilt shift tool 211, which is the superimposed UI control unit. Shown at different stages 205-220. As illustrated, FIG. 2 shows that the GUI 200 includes an image display area 225.
0034The image display area 225 displays an image that the user wants to edit using the image editing application. In some embodiments, the image editing application allows the user to apply a tilt shift effect to the image by touching the displayed image and manipulating the tilt shift tool 211. The image editing application superimposes the tilt shift tool 211 on the image displayed in the image display area 225.
0035In some embodiments, the tilt shift tool 211 has two parallel lines superimposed on the displayed image 230 in response to the image editing application receiving a user's touch on the displayed image 230. Includes (upper and lower parallel lines). In some embodiments, when the user moves his or her finger while touching the displayed image, the tilt shift tool displays two parallel lines. The tilt shift tool 211 arranges two parallel lines at a predetermined distance (for example, hundreds of pixels). The tilt shift tool 211 then blurs the portion of the image above the upper parallel line and the portion of the image below the lower parallel line. As a result, the portion of the image between the two lines looks sharper than the rest of the image.
0036The tilt shift tool 211 in some embodiments allows the user to adjust the distance between two parallel lines by touching the image. The user can increase the distance by performing a magnifying gesture (eg, touching the image while spreading and releasing two fingers) on the two parallel lines. The user can reduce the distance by performing a pinch gesture (eg, pulling two fingers together while touching the image). The user can also increase or decrease the distance by moving one or both of the two parallel lines (eg, by dragging). The tilt shift tool 211 also allows the user to move two parallel lines together by touching and dragging the space between the two lines.
0037Hereinafter, an operation example of an image editing application having GUI200 will be described. At step 205, the image editing application displays the image 230 within the image display area 225. The second stage 210 shows the GUI 200 after the user touches the image area just below the vertical center of the image. At this stage, the tilt shift tool 211 placed two parallel lines on the image so that the center of the touched area was equidistant from the two parallel lines. The tilt shift tool 211 also blurred the portion of the image that was vertically above the upper parallel line and the portion of the image that was vertically below the lower parallel line.
0038In step 215, the user performed a magnified gesture on the image using finger 212 and another finger 213 to spread the two parallel lines further apart. Both the upper and lower lines have moved. The upper line moved up and the lower line moved down. As a result, the portion of the image between the two parallel lines is larger, and the tilt shift tool does not blur this portion of the image.
0039The fourth stage 220 shows the GUI 200 after the user lifts the fingers 212 and 213 from the image 230. The two parallel lines remain visible to indicate that these two lines are operational. When the user exits the edit mode of the image editing application, the two parallel lines disappear from the image 230.
0040FIG. 3 conceptually illustrates GUI300 of an image editing application of several embodiments that allows a user to apply image processing operations to an image displayed within an image display area by touching the image. Shown. Specifically, this figure shows that the user applies a gradation effect to a portion of an image by touching the image at three different stages 305 to 315. FIG. 3 shows that the GUI 300 includes an image display area 325.
0041The image display area 325 displays an image that the user wants to edit using the image editing application. In some embodiments, the image editing application applies a gradient effect to the displayed image by the user touching an area of the image or swiping a finger down to the bottom of the display area 325. to enable. When the user touches the location of the image, the image editing application applies a gradient effect from the top of the image to the vertical location of the touch in the image. That is, the image editing application varies the degree of effect (eg, lightening, darkening, color application, etc.) from the top of the image to the touch location.
0042When the user swipes down the image (for example, touching the image and dragging the finger down), the image editing application applies a gradient effect from the top of the image to the lowest vertical location where the finger was. The image editing application also allows the user to swipe up or there a location that is vertically above the lowest vertical location to which the gradient effect is applied in order to reduce the area to which the gradient effect is applied. It also makes it possible to touch.
0043In some embodiments, the image editing application does not show a visible horizon that can be manipulated by the user. However, the top of the image with the gradient effect applied and the bottom of the image without the gradient effect are visually identifiable thanks to the top effect. Therefore, there is a visible boundary (or visible horizontal band) that separates the top and bottom. This boundary provides the user with visual clues and allows the user to interact with the "invisible" effects tool.
0044From this, an operation example of an image editing application having GUI300 will be described. In step 305, the image editing application displays the image 330 within the image display area 345. At step 310, the user touched a location near the top of image 330 with finger 311 or dragged finger 311 downward to that location while touching image 330. The image editing application applied a gradient effect from the top of image 330 to the location where the finger 311 is. The portion of the image above finger 311 is drawn darker than before to indicate that a gradient effect has been applied to this portion.
0045The third stage 315 shows the GUI 300 after the user touches the location of the image closer to the bottom of the image 330 or drags the finger 311 to this location while touching the image 330. The image editing application applied a gradient effect from the vertical location where the finger 311 was in the previous stage 310 to the vertical location of the finger 311 in the current stage 315. The portion of the image above finger 311 is drawn darker than before to indicate that a gradient effect has been applied to this portion.
0046The following sections describe some more detailed embodiments of the image editing tool. Section I describes image cropping and angle correction tools for some embodiments. Next, Section II describes some on-image editing tools that allow you to apply image processing operations to an image by rubbing the image or by selecting an area of the image. Section III describes some visible and invisible image tools that allow you to apply special effects to an image. This is followed by Section IV, which describes alternative UI layouts for image editing applications. Section V describes the software architecture of some embodiments of image editing applications. Next, Section VI describes that some embodiments of image editing applications are also image viewing, editing, and organizing applications. Finally, Section VII describes some electronic systems that implement some of the embodiments described herein.
0047I. Image angle correction and cropping tool A. Angle correction on the image FIG. 4A conceptually illustrates GUI400 of an image editing application of several embodiments that allows the user to angle-correct an image by using an on-image tool. Specifically, the figure shows in six different steps 401-406 that the user touches a graphical tool to rotate the image to correct the angle of the image. As illustrated, FIG. 1 shows that the GUI 400 includes an image display area 410, a collection display area 415, a tool selection pane 420, a tool display area 425, and a control pane 430.
0048The image display area 410 displays an image that the user wants to edit using the image editing application. The collection display area 415 displays a collection of thumbnails of images that are available for editing using the image editing application. When the user selects a thumbnail in the collection display area 415 (eg, by touching it), the image editing application displays the image represented by the selected thumbnail in the image display area 410. The image in some embodiments is a digital photographic image originally taken by a digital photographic device (eg, a digital camera), a digitally scanned photographic image, or any digitally created image.
0049The tool selection pane 420 in some embodiments displays a set of icons representing different editing toolsets of the image editing application. When the user selects an icon in the tool selection pane 420 (eg, by touching it), the image editing application displays the corresponding set of tools in the tool display area 425 and / or in the image display area 410. The image editing application in some embodiments provides a visual signal to indicate which set of tools is currently selected. For example, an image editing application highlights the selected icon. In some embodiments, the leftmost icon 421 displayed in the tool selection pane 420 represents an image cropping and angle correction tool. When the user selects icon 421, the image editing application in some embodiments launches a crop and angle correction tool. That is, the image editing application treats a part of the user input (for example, touch input) to the image in the image display area as the input to the activated tool. The image editing application also displays the angle correction tool 430 within the tool display area 425 in response to the selection of icon 421.
0050The angle correction tool allows the user to correct the angle of the image displayed in the image display area 410. Angle correction of an image means that objects and people in the image are tilted to one side of the photograph so that they cannot be seen. The angle correction tool in some embodiments also maintains the aspect ratio of the original image for the angle corrected image and avoids including the outer area of the original image within the final crop. To do so, as the image is angle corrected, the image is magnified and / or cropped.
0051Angle correction tools provide a number of different ways to correct the angle of a displayed image. For example, some embodiments of angle correction tools include dial 435. The angle correction tool allows the user to correct the angle of the displayed image by turning or rotating the dial 435. In some embodiments, the angle correction tool rotates the displayed image in the direction in which the dial 435 is turning or rotating (eg, clockwise or counterclockwise). The angle correction tool rotates the displayed image by the amount that the dial 435 rotates or rotates.
0052In some embodiments, the dial 435 has markings and numerals to indicate the amount of rotation or rotation of the dial 435. The dial 435 also has a fixed knob 460 that does not move as the dial 435 rotates to provide a reference system. The numbers or markings pointed to or aligned with the fixed knob 460 indicate the amount of rotation or rotation of the dial 435. The number on the dial 435 in some embodiments represents the angle in degrees or radians, or any other suitable unit that can represent the angle or amount of rotation. The sign of the number (ie, negative or positive) indicates the direction of rotation. In some embodiments, a negative number indicates that the dial 435 and the displayed image have rotated clockwise. Positive numbers indicate that the dial 435 and the displayed image have rotated counterclockwise.
0053Different embodiments define differently the relationship between the direction of the user's touch, the direction of rotation of the dial 435, the direction of rotation of the image, and the sign of the numbers shown within the dial 435. For example, in some embodiments, the image editing application rotates the displayed image in a direction opposite to the direction of rotation of the dial 435. In addition, a positive number on dial 435 may indicate a clockwise rotation of the displayed image, and a negative number on the dial 435 may indicate a counterclockwise rotation of the displayed image. ..
0054In some embodiments, the image editing application plays a sound to provide the user with an audible signal while the user is turning the dial 435. The sound played by the image editing application is, in some embodiments, the sound produced by the mechanical dial when the mechanical dial is swiveling or rotating.
0055The dial 435 in some embodiments is displayed in the tool display area 425 when the user selects icon 421 in the tool selection pane 420. In another embodiment, the image editing application displays the dial 435 in the image display area 410 so that the dial 435 partially overlaps the image displayed in the image display area 410. Image editing applications may also display the dial in different shapes and sizes within different parts of the GUI 400 . For example, the dial 410 may be displayed so as to have a perfect circular shape within the corners of the image display area 410.
0056When the image editing application activates the cropping and angle correction tool in response to the user's selection of icon 421, the image editing application in some embodiments has grid lines 450 over the image displayed within the image display area 410. Display a set of. The grid line 450 indicates that the crop and angle correction tools are activated and also provides guidance when the user crops the displayed image. The grid lines 450 in some embodiments transform the image display area 410 into a 3x3 grid. In some embodiments, the image editing application does not display the set of grid lines 450 until the user touches the displayed image. In some embodiments, the image editing application does not display the set of grid lines 450 until the dial 435 is swiveled.
0057In addition to the set of grid lines 450, the image editing application displays another set of grid lines 455 as the user begins to swivel the dial 435. This additional set of grid lines 455 provides additional visual aid when the user attempts to correct the angle of the displayed image. For example, if the object or person shown in the displayed image is orthogonal to the grid lines, the user knows that the displayed image has been angle corrected. The grid lines 450 and 455 in some embodiments together transform the image display 410 into a 9x9 grid. Those skilled in the art recognize that the grid formed by the grid lines 450 and 455 with different display areas does not necessarily have to have specific dimensions such as 3x3 or 9x9, but may have any different dimensions. Will do. In addition, image editing applications in some embodiments draw grid lines 450 and 455 in different colors. For example, grid line 450 is white and grid line 455 is yellow.
0058Control pane 430 displays a different set of icons that represent different operations performed by the image editing application in response to icon selection. The control pane 430 in some embodiments displays the name of a collection of images whose thumbnails are displayed within the collection display area 415. The control pane 430 may display the name (eg, file name) of the image displayed in the image display area 425.
0059From this, an operation example of an image editing application including GUI400 will be described. The first stage 401 shows the user's selection of thumbnail 440 by touching thumbnail 440 with finger 441. The image editing application responds to the selection of the thumbnail 440 by displaying the image 445 represented by the thumbnail 440 in the image display area 410.
0060At step 402, the user then selects the icon 421 displayed in the tool selection pane 420. In response, the image editing application activates the crop and angle correction tools in step 403. The image editing application highlights icon 421. The image editing application also displays the dial 435 in the tool display area 425 and the grid lines 450 in the image display area 410. The number on dial 435 pointed to by the fixed knob 460 indicates "0", indicating that dial 435 is not turning at all in either direction.
0061At step 404, the user places a finger 441 on a portion of dial 425 and begins turning the dial clockwise (ie, to the right). The image editing application in some embodiments displays grid lines 455 when the user places a finger 441 or the user begins to swivel the dial 435.
0062The next step 405 shows the GUI 400 after the user has turned the dial 435 clockwise. The crop and angle correction tool swiveled the displayed image 445 clockwise by the amount of rotation of the dial 435. In this example, the "-10" pointed to by the fixed knob 460, the dial 435 and the displayed image 445 are shown to have rotated 10 degrees clockwise. Step 405 also magnifies image 445 (hence in the image) so that the angle correction tool avoids displaying the outside of image 445 in the display area 410 before it is rotated. The bike looks bigger), and it also shows that it was cut out. That is, if the angle correction tool rotated the image 445 without cropping and enlarging, the portion protruding outside the image 445 would have to be displayed within the image display area 410. As illustrated, the angle correction tool in some embodiments rotates the image with respect to the edge of the display area 410 as the dial 435 is rotated. That is, only the object and the person shown in the image are rotated with respect to the edge of the image display area 410, thereby preventing the object and the person from appearing tilted.
0063The final stage 406 shows the GUI 400 after the user lifts finger 441 from dial 435 and no longer touches dial 435. Since the dial 435 is no longer touched, the image editing application has also removed the grid lines 455.
0064FIG. 4B conceptually illustrates GUI400 of an image editing application of several embodiments that allows the user to angle-correct an image by using an on-image tool. This figure shows that the user touches a graphical tool to rotate the image to correct the angle of the image in six different steps 401a-406a. Specifically, FIG. 4B shows GUI 400 of some embodiments having an inside display area 480a.
0065The image display area 410 of some embodiments includes an inner display area 480a. The image editing application of some embodiments is the displayed image while avoiding displaying the boundary of the image within the inner display area 480a while the image is being rotated clockwise or counterclockwise. Adjust the magnification level for the displayed image so that the maximum part of is displayed within the inner display area 480a. In some embodiments, the boundaries of the image are displayed within the area of the display area 410 outside the inner display area 480a while the image is being rotated. This area is called the outer area of the display area 410. Also, in some embodiments, the portion of the image that falls within the outer area of the display area 410 appears faint. The inner display area 480a is applicable to the image editing application of the embodiments described below (see, eg, FIGS. 9, 12, and 13).
0066From this, an operation example of an image editing application including GUI400 will be described. The first stage 401a and the second stage 402a are the same as the stage 401 and stage 402 described above with reference to FIG. 4A. In response to the selection of icon 421 in step 402a, the image editing application activates the crop and angle correction tools in step 403a. The image editing application displays the inside display area 480a within the display area 410. In some embodiments, the inner display area 480a is smaller than the display area 410, and the image editing application displays the inner display area 480a in the center of the display area 410.
0067The image editing application displays the image 445 in the inner display area 480a. The image editing application highlights icon 421. The image editing application also displays the dial 435 in the tool display area 425 and the grid lines 450 in the inner display area 480a. The number on dial 435 pointed to by the fixed knob 460 indicates "0", indicating that dial 435 is not swiveled in either direction at all.
0068At step 404a, the user places a finger 441 on a portion of dial 425 and begins turning the dial clockwise (ie, to the right). The image editing application in some embodiments displays grid lines 455 within the inner display area 480a when the user places a finger 441 or the user begins to swivel the dial 435.
0069The next step, 405a, shows the GUI 400 after the user has turned the dial 435 clockwise. The crop and angle correction tool swiveled the displayed image 445 clockwise by the amount of rotation of the dial 435. In this example, the "-5" pointed to by the fixed knob 460 indicates that the dial 435 and the displayed image 445 have been rotated 5 degrees clockwise. Stage 405a also allows the angle correction tool to magnify the image so that the maximum portion of image 445 is displayed within the inner display area 480a without displaying the boundaries of the image within the inner display area 480a. Show (so the bike in the image looks bigger). As shown, the rest of the image that extends out of the inner display area 480a is displayed within the outer area 485a (ie, the area of the display area 410 outside the inner display area 480a). The upper and side boundaries of image 455 are also displayed within the outer area 485a of the display area 410.
0070The final stage 406a shows the GUI 400 after the user has turned the dial 435 further clockwise. The crop and angle correction tool swiveled the displayed image 445 clockwise by the amount of rotation of the dial 435. In this example, the "-10" pointed to by the fixed knob 460 indicates that the dial 435 and the displayed image 445 have rotated clockwise by an additional 5 degrees since the previous step 405a. Step 406a also displays the maximum portion of image 445 within the inner display area 480a due to the amount of rotation of this particular image, without the angle correction tool displaying the image boundaries within the inner display area 480a. It also indicates that you have changed the magnification level of the image so that the bicycle in the image looks larger). As shown, the bicycle looks larger than it was visible in the previous stage 405a because the image editing application further magnified image 455. In some embodiments, the grid line 455 will disappear when the user raises his finger from the dial.
0071FIG. 5 allows the user to angle correct the image by manipulating the graphical user interface items of the image editing application (eg, dial 435 described above with reference to FIGS. 4A and 4B). , Conceptually illustrates the process 500 performed by some embodiments. Process 500 in some embodiments is performed by an image editing application. Process 500 begins when the image editing application runs on a computing device that has a touch-sensitive or near-touch sensitive screen to receive touch input from the user. Process 500 (in 505) begins by displaying an image within the image display area of the image editing application (eg, display area 410 or inside display area 480a described above).
0072Process 500 then displays the geometry within the area of the image editing application (at 510). In some embodiments, process 500 displays the geometry at least partially within the image display area (eg, image display area 410). In another embodiment, process 500 displays the geometry so that the geometry and the displayed image do not overlap. Different embodiments use different geometric shapes to be displayed. For example, Process 500 uses a dial shape (eg, a partial circular shape) as the geometric shape to display.
0073Process 500 (at 515) then determines if the geometry has received some touch input. That is, the process (at 515) determines whether the user has touched the geometry (by touching an area of the touch-sensitive or near-touch sensitive screen showing the geometry). If process 500 (at 515) determines that the geometry received no touch input, process 500 loops to 515 to wait for touch input on the geometry.
0074If process 500 (at 515) determines that the geometry has received a touch input, the process proceeds to 520 to move the geometry based on the touch input received. For example, if the user swipes the geometry from left to right, Process 500 rotates the geometry to the right (ie, clockwise). Similarly, when the user swipes the geometry from right to left, Process 500 rotates the geometry to the left.
0075Process 500 then (at 525) rotates the displayed image based on the movement of the geometry. In some embodiments, process 500 rotates the displayed image by an amount proportional to the amount by which the geometry is rotated (eg, the degree of angle). Then the process ends.
0076The cropping and angle correction tools of some embodiments described so far magnify the displayed image as the image is angle corrected. In some cases, cropping and angle correction tools do not zoom in as the displayed image is angle corrected.
0077Figure 6 shows the GUI400 of an image editing application. Specifically, in this figure, when the GUI400 magnifies on the displayed image 610, the angle correction tool does not magnify on the image 610 prior to angle-correcting the image. This is shown in six different stages 601-606.
0078In some embodiments, the image editing application allows the user to perform a magnifying gesture on the image (eg, sticking two fingers together to touch the image and then spreading the fingers while touching the image. By enlarging the displayed image, it is possible to crop the image displayed in the display area 410. When the image editing application enlarges the image, some parts of the image will not be displayed in the image display area 410 because those parts will protrude outside the image display area. The image editing application in some embodiments cuts off these parts of the image (ie, crops these parts from the image).
0079If the image is angle-corrected after the image editing application magnifies the displayed image, the angle-correcting tools of some embodiments allow the angle-correcting tool to rotate the image to correct the angle of the image. Sometimes do not make additional expansions. This is because when the angle correction tool corrects the angle of the image, the angle correction tool can use the cut-off portion of the image.
0080Hereinafter, an operation example of the image editing application will be described. In step 601 the image 610 is displayed in the image display area 410. Icon 421 is highlighted to indicate that the crop and angle correction tools are activated. The number on dial 435 pointed to by the fixed knob 460 is 0, indicating that dial 435 is not rotated. Grid lines 450 are also displayed in the image display area 410.
0081In step 602, the user touches image 610 with two fingers. In step 603, the user spreads and releases the finger while touching the image 610. The crop and angle correction tools magnify the image, thereby cropping parts of the image that cannot fit within the image display area 410.
0082At step 604, the user rests his finger on dial 435 and begins rotating image 610. An additional grid line 455 is displayed within the image display area 410. The next step 605 shows the GUI 400 after the user has turned the dial 435 clockwise. The crop and angle correction tools responded by swirling the displayed image 610 clockwise, but by using the portion of the image that was cropped in step 603, the image 610 was not magnified any further. The numbers indicated by the fixed knob 460 indicate that the dial 435 and image 610 have been rotated 10 degrees clockwise.
0083The final stage 605 shows the GUI 400 after the user raises finger 441 from dial 435 and no longer touches dial 435. Since the dial 435 is no longer touched, the image editing application has also removed the grid lines 455.
0084The cropping and angle correction tools of some embodiments described so far allow the user to angle correct the displayed image by touching the dial. Crop and angle correction tools provide other dial adjustment means. FIG. 7 conceptually illustrates a computing device 700 running image editing applications of some embodiments. Specifically, this figure shows that the user rotates the displayed image by rotating or rotating the computing device 700 and operating the superimposed angle correction tool. Shown in steps 705-715. FIG. 7 shows the GUI of an image editing application including an image display area 725 and an angle correction tool 730.
0085The computing device 700 has a touch-sensitive or near-touch sensitive screen for receiving touch input. Some examples of the computing device 700 are tablet computers (eg, Apple iPad®) and smartphones (eg, Apple). iPhone (registered trademark)). The computing device 700 also has one or more devices (eg, gyroscopes, accelerometers, etc.) that detect and measure the movement of the computing device 700. In some embodiments, the image editing application uses these devices to detect and measure the motion of the computing device 700. For example, an image editing application receives orientation values from a computing device 700 gyroscope (not shown). In some embodiments, these orientation values include an x-axis orientation value, a y-axis orientation value, and a z-axis orientation value. Here, the z-axis is orthogonal to the earth's surface, and the x-axis and y-axis are parallel to the earth's surface. Therefore, if the computing device 700 is rotated at the same height above the ground level (for example, if the computing device 700 is rotated on a table that is horizontal on the ground), the z-axis. Only the orientation value changes.
0086The GUI720 is similar to the GUI100 described above with reference to FIG. The image display area 725 is the same as the image display area 125, and the angle correction tool 730 is the same as the angle correction tool 120. The angle correction tool 730 has a marking 731 to indicate the amount of rotation of the image displayed within the image display area 725.
0087In some embodiments, the image editing application in some embodiments allows the user to operate the angle correction tool 730 by moving the computing device 730 (eg, turning, rotating, etc.). To enable. The image editing application records the initial orientation values for all three x, y, and z-axis or at least the z-axis of the computing device 730 before the computing device 730 is moved by the user. When the user swirls the computing device 730 (eg, changing its orientation), the image editing application in some embodiments is recorded with the initial orientation value of the computing device 730 (particularly the z-axis orientation). Keeps the orientation of the angle correction tool 130 with respect to the value). As a result, the angle correction tool 730 swivels with respect to the changing orientation of the computing device 730 as the user swivels the computing device 700.
0088The angle correction tool 730 rotates the image displayed in the image display area 725 with respect to the changing orientation of the computing device 730 according to the amount of rotation of the angle correction tool 730. As a result, some image editing applications of embodiments maintain the orientation of the displayed image relative to the initial orientation of the computing device while the computing device is in motion.
0089Hereinafter, an operation example of an image editing application having the angle correction tool 730 will be described. The first stage 705 indicates that the user holds the computing device 700 stationary and the image editing application is displaying the image 726 within the image display area 720. Here, it is assumed that the computing device 730 is placed on a table that is horizontal to the ground surface in this example. The marking 731 of the angle correction tool 730 is in their respective initial position. The image editing application records the z-axis orientation of the computing device 700 (eg, in the memory of the computing device 700) before the computing device 700 is moved.
0090At step 110, the user tilts the compute device 700 to the left as shown until the people and buildings shown in the displayed image are angle-corrected with respect to the current orientation of the compute device 700. .. As the user turns the computing device to the left, the image editing application receives or gets from the gyroscope (not shown) of the computing device 700 the amount that the computing device 700 has rotated from the initial orientation of the computing device 700. To do. The image editing application turns the angle correction tool 730 to the right with respect to the changing orientation of the computing device 730. Marking 731 is rotating to the right to indicate that the angle correction tool 730 has turned to the right. The angle correction tool 730 rotates the image 710 to the right according to the amount of rotation of the angle correction tool 730.
0091The image editing application also receives user input indicating that the user does not intend to rotate the angle correction tool 730 and the displayed image by rotating the computing device 700. Upon receiving such input, the image editing application does not rotate the angle correction tool 730 and the displayed image when the user further rotates the computing device 700.
0092In the third stage 715, the user swivels the angle correction tool 730 so that the current orientation of the computing device 730 is the same as the recorded initial orientation of the computing device 730. As shown, the displayed image was angle corrected. That is, the people and buildings shown in the displayed image are orthogonal to the edge of the image display area 725.
0093FIG. 8 shows the process 800 performed by some embodiments to allow the user to angle correct an image by manipulating graphical user interface items in an image editing application running on a computing device. Is conceptually shown. Process 800 in some embodiments is performed by an image editing application. Process 800 begins when an image editing application runs on a computing device that has one or more devices (eg, gyroscopes, accelerometers, etc.) that detect and measure the movement of the computing device. Process 800 begins by displaying an image within the image display area of the image editing application (in 805). An example of the image display area is the image display area 725 described above with reference to FIG.
0094Process 800 then displays the geometry within the area of the image editing application (in 810). In some embodiments, process 800 displays the geometry at least partially within the image display area. In another embodiment, process 800 displays the geometry so that the geometry and the displayed image do not overlap. Different embodiments display different geometries. For example, Process 800 displays a dial shape (eg, a partial circular shape) as a geometric shape.
0095Process 800 (at 815) then determines if the computing device has worked. In some embodiments, process 800 receives or obtains orientation information from the computing device's gyroscope and uses the orientation information to determine if the computing device has moved. Process 800 also holds orientation information that includes the current orientation of the computing device. Process 800 compares the current orientation of the computing device with the previously recorded orientation of the computing device to determine the behavior of the computing device with respect to a particular axis. The process determines that the computing device did not move with respect to a particular axis if the orientation with respect to that particular axis did not change.
0096If process 800 (at 815) determines that the computing device has not moved, process 800 loops to 815 to wait for the computing device to move. If process 800 (at 815) determines that the computing device has moved, the process proceeds to 820 to move the geometry based on the movement of the computing device. For example, if the user tilts the computing device to the left, process 800 rotates the geometry to the right with respect to the recorded initial rotation of the computing device. Similarly, if the user tilts the computing device to the right, process 820 rotates the geometry to the left.
0097Process 800 then (at 825) rotates the displayed image based on the movement of the geometry. In some embodiments, process 800 rotates the displayed image by an amount proportional to the amount by which the geometry is rotated (eg, the degree of angle). Then the process ends.
0098FIG. 9 conceptually illustrates a computing device 900 running image editing applications of some embodiments. Specifically, this figure shows that the user can rotate the displayed image 910 by manipulating the dial 435 by turning or rotating the computing device 900, in seven different stages 901-907. Shown in. As illustrated, FIG. 9 shows that the computing device runs an image editing application of some embodiments having a GUI 400.
0099The computing device 900 is described above with reference to FIG. 7 in that the computing device 900 includes one or more devices (eg, a gyroscope, an accelerometer, etc.) that detect and measure the movement of the computing device 900. Similar to the computing device 700 being used.
0100The image editing application also allows the user to indicate that the user intends to start or stop adjusting the dial 435 by moving the computing device 900. For example, if the crop and angle correction tools of the image editing application are activated in response to the selection of icon 421, the image editing application receives a tap on the dial 435 and the dial 435 receives a tap on the computing device 900. Enter a mode adjusted by movement. The image editing application exits that mode when it receives a new tap on dial 435. In some embodiments, the image editing application provides visual instructions to indicate that the image editing application is in that mode. For example, an image editing application highlights dial 435 or blinks dial 435 when the image editing application is in that mode.
0101From this, an operation example of an image editing application including GUI400 will be described. The first stage 901 shows the selection of the thumbnail 915 by the user by touching the thumbnail 915 with the finger 920. The image editing application responds to the selection of the thumbnail 915 by displaying the image 910 represented by the thumbnail 910 in the image display area 410.
0102In step 902, the user then selects the icon 421 displayed in the tool selection pane 420. In response, the image editing application activates the crop and angle correction tools in step 903. The image editing application highlights icon 421. The image editing application also displays the dial 435 in the tool display area 425 and the grid lines 450 in the image display area 410.
0103The user also taps on the dial 435 (eg, by quickly lowering and raising the finger) to indicate that the user wants to adjust the dial 435 by moving the computing device 900. The crop and angle correction tool records the initial orientation of the compute device 900 before it is moved. For example, the crop and angle correction tool obtains the y-axis value of the compute device 900 from the gyroscope of the compute device 900 and records at least the z-axis orientation value.
0104At stage 904, the user begins rotating the computing device 900 counterclockwise. The crop and angle correction tool maintains the orientation of the dial 435 with respect to the initial orientation of the computing device 900. This causes the crop and angle correction tool to start rotating the dial 435 clockwise. The crop and angle correction tool also displays grid lines 455 and transforms the image display area 410 into a 9x9 grid.
0105The next step 905 is to angle correct the people and buildings shown in the displayed image with respect to the current orientation of the image display area 410 (ie, the current orientation of the computing device 900). Shows the GUI 400 after the user has turned the dial 435 clockwise by turning the computing device 905 counterclockwise.
0106Step 905 also enlarges the image 910 so that the angle correction tool avoids displaying the outside of the image 910 in the display area 410 before it is rotated (hence, in the image). The person and building shown will look larger), and also indicate that they have been cut out. The next step, 906, is to indicate that the user no longer wants to adjust the dial 435 by moving the computing device 900 (eg, quickly lower and raise the finger) on the dial 435. Indicates to tap (by).
0107The final stage 907 is after the user has rotated the computing device 905, but the image 910 shows the GUI 400 which is not rotated relative to the edge of the image display area 410. The crop and angle correction tools have also removed grid lines 455, as the user showed in previous step 906 that the user no longer rotates the dial and image by rotating the computing device 900.
0108The cropping and angle correction tools of some embodiments described above and below save a portion of the image displayed in the display image area as a new image when the tool is stopped. As described above, the crop and angle correction tools "edit" the image and generate a new image.
0109In addition, some cropping and angle correction tools edit the image while it is being cropped for display within the display area as it is magnified. In another embodiment, the crop and angle correction tool enlarges and reduces the image and crops it to display an area within the display area that is not part of the image while the image is being rotated. It is used purely as a viewing aid to display as much of the image as possible within the display area. In other words, as a pure viewing aid, the crop and angle correction tools themselves do not "edit" the image.
0110The cropping and angle correction tools of some of the embodiments described above allow the user to angle correct the displayed image by manipulating UI items. Some embodiments allow the user to correct the angle by directly touching the displayed image and rotating it.
0111FIG. 10 conceptually illustrates the GUI 1000 of some embodiments of an image editing application running on a computing device. Specifically, this figure shows that the user can rotate the displayed image by touching the image with two fingers 1011 and 1012 and rotating it in four different stages 1005 to 1020. Shown in. FIG. 10 shows that the GUI 1000 includes an image display area 1025.
0112A computing device (not shown) on which an image editing application runs has a touch-sensitive or near-touch sensitive screen for receiving touch input. The image editing application receives or acquires touch inputs (eg, spatial and temporal information about touch gestures) made on a touch-sensitive or near-touch sensitive screen and uses the inputs to display the image area 1025. Rotate the image displayed inside. The image display area 1025 is similar to the image display area 125 in that the image display area 1025 displays an image that the user wants to edit using the image editing application.
0113Upon receiving the touch input, the image editing application determines whether the user wants to rotate the displayed image. In some embodiments, when the user touches the displayed image with two or more fingers and rotates the finger more than a threshold amount (eg, a few degrees) on the image, the image editing application , The user determines that he wants to rotate the displayed image. An image editing application determines the amount of finger rotation by tracking the position of the finger on the touch screen and measuring the amount of rotation produced by the line connecting the positions of the two fingers.
0114The image editing application in some embodiments does not begin to rotate the displayed image until the user's finger touching the image has rotated more than the threshold amount. This causes the image editing application to refrain from rotating the image when the user intends to perform some other action by touching the displayed image with two or more fingers. For example, when the user simply touches an image and opens two fingers further away, the media editing application magnifies the displayed image and does not rotate the displayed image. As another example, an image editing application displays different parts of an image when the user swipes the image with two or more fingers without rotating the image within the display area. In this way, the image editing application allows the user to angle-correct, crop, scale, and slide the displayed image without raising a finger from the image.
0115From this, an operation example of an image editing application having GUI1000 will be described. At stage 1005, the image editing application displays image 1026 within the image display area 1025. At step 1010, the user places two fingers 1011 and 1012 on image 1026. At step 1015, the user rotates the two fingers 1011 and 1012 slightly clockwise, but less than a predetermined threshold (eg, an angle of a few degrees). Since the amount of finger rotation did not exceed the threshold, the image editing application does not start rotating the displayed image 1026.
0116In the fourth stage 1020, the user then further rotates the fingers 1011 and 1012 clockwise, and the image editing application rotates the displayed image 1026 accordingly. The user presses fingers 1011 and 1012 just enough to straighten the displayed image 126 (for example, to level the building and three people shown in the image with the GUI1000 horizon). It was rotated.
0117FIG. 11 shows that the user can use two or more fingers to touch the displayed image and rotate it to correct the angle of the image displayed in the image display area of the image processing application. To be possible, the process 1100 performed by some embodiments is conceptually shown. Process 1100 in some embodiments is performed by an image editing application. Process 1100 begins when an image editing application runs on a computing device that has a touch-sensitive or near-touch sensitive screen to receive touch input. Process 1100 begins by displaying an image within the image display area of the image editing application (in 1105). An example of the image display area is the image display area 1025 described above with reference to FIG.
0118Process 1100 (at 1110) then determines if the displayed image has received more than one touch input. That is, process 1100 (at 1110) determines if the user has placed two or more fingers on the displayed image. If process (at 1110) determines that the displayed image did not receive more than one touch input, process 1100 loops back to 1110 to wait for multiple touch inputs.
0119If process 1100 (at 1110) determines that the displayed image has received more than one touch input, process (at 1115) determines if the touch input is rotatable. In some embodiments, if the line connecting the two locations touched by the user's finger is rotated, the process determines that the touch input is rotatable.
0120If process 1100 (at 1115) determines that the touch input is not rotatable, process 1100 terminates. Otherwise, process 1100 proceeds to 1120 to determine the amount of rotation produced by the touch input received (eg, the degree of angle).
0121Process 1100 (at 1125) then determines if the specified amount of rotation exceeds a certain threshold (eg, an angle of several degrees). If process 1100 (at 1125) determines that the amount of rotation is less than the threshold, the process terminates. Otherwise, the process proceeds to 1130 to rotate the displayed image. In some embodiments, process 1100 (at 1130) rotates the image by an amount proportional to the specified amount of touch input traveled. That is, in some embodiments, process 1100 (at 1130) rotates the displayed image by the same amount that the touch input has moved. In another embodiment, process 1100 rotates the displayed image by more or less than the amount the touch input has moved. Then the process ends.
0122FIG. 12 conceptually illustrates GUI400 of some embodiments of an image editing application running on a computing device having a touch-sensitive or near-touch sensitive screen for receiving touch input. Specifically, the figure shows that the user rotates the displayed image by touching the image and rotating it in six different stages 1205-1230.
0123From this, an operation example of an image editing application including GUI400 will be described. The first stage 1201 shows the user's selection of thumbnail 440 by touching thumbnail 440 with a finger 1230. The image editing application responds to the selection of the thumbnail 440 by displaying the image 445 represented by the thumbnail 440 in the image display area 410.
0124At stage 1202, the user then selects icon 421, which is displayed in the tool selection pane 420. In response, the image editing application activates the crop and angle correction tools at stage 1203. The image editing application highlights icon 421 to indicate that the crop and angle correction tools are activated. The image editing application also displays the dial 435 in the tool display area 425 and the grid lines 450 in the image display area 410. The numbers pointed to by the fixed knob 460 indicate that the dial 435 is not swiveled in either direction at all.
0125At step 1204, the user touches the image with two fingers 1235 and 1240 and rotates the two fingers slightly clockwise, but less than a predetermined threshold (eg, an angle of several degrees). Since the amount of finger rotation did not exceed the threshold, the crop and angle correction tools do not start rotating the displayed image 445. The crop and angle correction tools also do not start rotating dial 435 and do not show more grid lines in addition to grid lines 450.
0126In the next step 1205, the user rotated the fingers 1235 and 1240 clockwise over a predetermined threshold. The crop and angle correction tools rotate the displayed image 445 accordingly. The user rotated fingers 1235 and 1240 until the bicycle displayed in image 445 was angle corrected. In this example, the "-10" pointed to by the fixed knob 460 indicates that the dial 435 and the displayed image 445 have been rotated 10 degrees clockwise. Step 405 also magnifies image 445 so that the angle correction tool avoids showing the part that sticks out of image 445 before it is rotated (so the bike in the image looks bigger). ), It also shows that it was cut out. The crop and angle correction tools also display grid lines 455 as soon as the rotation of the finger 1235 exceeds a predetermined threshold.
0127The final stage 1206 shows the GUI 400 after the user lifts fingers 1235 and 1240 from the displayed image 445 and no longer touches the image. Since image 445 is no longer touched, the crop and angle correction tools have also removed grid lines 455.
0128In addition to allowing the user to touch the image and correct the angle of the displayed image, some image editing applications of embodiments are at least one UI control item that is superimposed on the image. It provides a UI control item that allows the user to correct the angle of the image using the UI control item.
0129FIG. 13 conceptually illustrates GUI400 of some embodiments of image editing applications. Specifically, the figure shows that the user rotates the displayed image by using the horizontal lines detected by the crop and rotate tools of the image editing application in five different stages 1301-1305.
0130In some embodiments, the crop and angle correction tool analyzes the image displayed within the display area and detects horizontal lines in the image. The crop and angle correction tools display the detected horizontal lines on top of the image, allowing the user to angle correct the image using the displayed horizontal lines. In addition to the detected horizon, the crop and angle correction tools of some embodiments provide one or more selectable user interface items for angle correction of the image. One of the user interface items provided in some embodiments indicates in which direction the image editing application will swivel the horizon to correct the angle of the image. If the user wants to angle-correct the image as suggested by the horizon and the user interface item, the user can select the user interface item to angle-correct the image. U.S. Patent Application No. 13 / 366,613, filed on 02/06/2012, details the detection of horizon. U.S. Patent Application No. 13 / 366,613 is incorporated herein by reference.
0131From this, an operation example of an image editing application including GUI400 will be described. The first stage 1301 indicates the selection of the thumbnail 1315 by the user by touching the thumbnail 1315 with the finger 1325. The image editing application responds to the selection of thumbnail 1310 by displaying the image 1320 represented by thumbnail 1315 in the image display area 410.
0132At step 1302, the user then selects icon 421, which is displayed in the tool selection pane 420. In response, the image editing application activates the crop and angle correction tools at stage 1303. The crop and angle correction tool detects the horizontal line 1345 and displays the horizontal line 1345 on top of the displayed image 1320. In addition, the crop and angle correction tools display selectable UI items 1335 and 1340 at both ends of the detected horizon 1345. UI item 1340 is for angle correction of the image and indicates the direction in which the horizon 1345 will turn when the user selects user interface item 1340. Since the horizon 1345 is tilted to the left (for example, the left edge is below the right edge), the crop and angle correction tools will rotate the horizon 1345 and image 1320 clockwise to correct the angle of the image 1320. To indicate that, the crop and angle correction tools place a down arrow in UI item 1340. User interface item 1335 is for closing the horizon 1345. If the user selects the user interface item 1335, the image editing application will erase the horizontal line 1345 and the user interface items 1335 and 1340.
0133Also, at stage 1303, the image editing application highlights icon 421. The image editing application also displays the dial 435 in the tool display area 425 and the grid lines 450 in the image display area 410. The number on dial 435 pointed to by the fixed knob 460 indicates "0", indicating that dial 435 is not turning at all in either direction.
0134At step 1304, the user selects user interface item 1340 with finger 1325. Then, in step 1305, the crop and angle correction tool responds by turning image 1330 clockwise so that the horizon 1345 is horizontal (ie, parallel to the lower edge of the image display area 1325). The image 1330 is angle-corrected by rotating it. The crop and angle correction tool swiveled the displayed image 445 clockwise by the amount of rotation of the dial 435. In this example, the "-10" pointed to by the fixed knob 460 indicates that the dial 435 and the displayed image 445 have been rotated 10 degrees clockwise. Step 1305 also magnifies image 445 so that the angle correction tool does not show the part that sticks out of image 445 before it is rotated (so the bike in the image looks bigger). ), It also shows that it was cut out.
0135FIG. 14 shows closing the detected horizon 1345 if the user wishes not to use the horizon 1345 to angle correct the image 1320. Specifically, the figure shows that the user closes the detected horizon 1345 by selecting UI item 1335 in five different stages 1401-1405.
0136As mentioned above, UI item 1335 is for closing the horizon 1345. If the user selects UI item 1335, the crop and angle correction tools will erase the horizontal line 1345 and UI items 1335 and 1340. In some embodiments, the crop and angle correction tools provide a method other than selecting UI item 1335. For example, the crop and angle correction tools will erase the horizon 1345 and UI items 1335 and 1340 when the user taps over other areas of the GUI 400. These other areas of the GUI 400 include dial 435, parts of the image display area 410 other than the UI item 1335, and the like. In some embodiments, the image editing application provides another UI item (not shown) for the user to select. When the user selects this UI item (eg, reset button), the image editing application redisplays the closed horizontal line 1345.
0137From this, an operation example of an image editing application including GUI400 will be described. The first three steps 1401 to 1403 are identical to the steps 1301 to 1303 described above with reference to FIG. At stage 1404, the user selects UI item 1435 with finger 1425. Then, in step 1405, the crop and angle correction tool responds by removing UI items 1335 and 1340 and the detected horizon 1345 from the image display area.
0138FIG. 15 shows some to allow the user to angle-correct an image displayed within the image display area of an image processing application by selecting a user interface item for angle-correcting the image. The process 1500 performed by the embodiment of is conceptually shown. Process 1500 in some embodiments is performed by an image editing application. Process 1500 begins when an image editing application runs on a computing device that has a touch-sensitive or near-touch sensitive screen to receive touch input. Process 1500 begins by identifying the horizon of the image (at 1505). U.S. Patent Application No. 13 / 366,613 details the identification of the horizon.
0139Process 1500 then (in 1510) displays the image and the identified horizon within the display area of the image editing application. An example of the image display area is the image display area 1325 described above with reference to FIG. Process 1500 also (in 1515) displays user interface items for angle correction of the displayed image. In some embodiments, process 1500 displays a user interface item within the horizon to indicate that the process will rotate the horizon and image in the direction of angle correction of the horizon and image. An example of such a user interface item is the user interface item 1340 described above with reference to FIG.
0140Process 1500 (at 1520) then determines if the user has selected the displayed user interface item. If process 1500 (at 1520) determines that the user did not select a user interface item, process 1500 loops back to 1520 to wait for the user to select a user interface item.
0141If process 1500 (at 1520) determines that the user has selected a user interface item, process 1500 proceeds to 1525 to rotate the image using the horizon. In some embodiments, process 1500 rotates the horizon with the image so that the position of the horizon with respect to the image does not change as the horizon and the image rotate. Process 1500 uses horizontal lines to rotate the displayed image in order to correct the angle of the displayed image. Then the process ends.
0142B. Crop on the image FIG. 16 conceptually illustrates GUI1600 of an image editing application of several embodiments that allows a user to crop an image by manipulating a cropping tool on the image. Specifically, the figure shows that the user crops the image 1640 by touching and dragging the corners and edges of the image 1640 at six different stages 1605-1615. FIG. 16 shows that the GUI 1600 includes an image display area 535.
0143As described above with reference to FIG. 6, the image cropping tool allows the user to crop and remove unwanted portions from the image by enlarging the image. In some embodiments, the cropping tool on the image also allows the user to crop and remove unwanted parts from the image by dragging the corners and edges of the image in and out. The crop tool also allows you to drag the corners and edges of the image with multiple touches. For example, a user can perform a diagonal pinch gesture on an image (eg, by moving two fingers toward each other on a touch-sensitive or near-touch sensitive screen) to make two corners of the image. Can be dragged toward the center of the image.
0144The image cropping tool in some embodiments allows the user to crop the image while maintaining the aspect ratio. The aspect ratio to be maintained could be one of the original image's aspect ratio before cropping, or one of the default aspect ratios (eg 2x3) that the user can choose. Also, the image cropping tool in some embodiments stretches the cropped image to match the original size of the image before cropping.
0145From this, an operation example of an image editing application having GUI1600 will be described. At stage 1605, the image editing application displays image 1640 within the image display area 1635. Next, at step 1610, the user places the finger 1611 on the upper right corner of image 1640. At step 1615, the user drags the upper right corner of image 1640 diagonally to cut off the peaks and guitarists shown in the image.
0146At stage 1620, the image editing application stretches the rest of the image (ie, the part of the image that shows the drummer). Since the aspect ratio of the rest does not match the aspect ratio of the image display area 1635, the image processing application places two gray areas on either side of the stretched rest of the image.
0147At this stage 1620, the user also places the finger 1611 on the lower edge of the rest. In the next step 1625, the user drags the bottom edge up to crop and remove more parts of image 1640. At stage 1630, the image editing application stretches the rest of the image to fill it. The final rest of the image has the same aspect ratio as the original aspect ratio of the image before it was cropped, so the image editing application does not display a gray area within the display area 1640.
0148FIG. 17 conceptually illustrates the process 1700 performed by some embodiments to allow the user of an image editing application to crop an image. Process 1700 in some embodiments is performed by an image editing application. Process 1700 begins when an image editing application runs on a computing device that has a touch-sensitive or near-touch sensitive screen to receive touch input from the user. Process 1700 begins by displaying an image within the image display area of the image editing application (in 1705).
0149Process 1700 (at 1710) then displays the geometry on top of the displayed image. In some embodiments, the geometry is the shape that first overlaps the edges of the displayed image. When the geometry is resized, the geometry defines the edges of the cropped image. Process 1700 (at 1715) then determines if the geometry has received user input. In some embodiments, if the user selects a corner or edge of a geometry by placing a finger on the corner or edge, process 1700 states that the geometry received user input. judge. Process 1700 also determines that the geometry has received user input if the user moves two or more fingers over the corners and edges of the geometry and areas within the geometry. To do.
0150Process 1700 then (in 1720) resizes the geometry based on the input received. Process 1700 interprets the gestures made by the user with one or more fingers differently. For example, when the user selects an edge of a geometry and drags it towards the center of the geometry, Process 1700 moves the edge inward. Also, when the user places his finger on the corner of the geometry and drags it towards the center of the displayed image, Process 1700 moves the corner inward. In addition, the user can use two or more fingers to select and drag the edges and corners of the image, or the user can pinch gestures over the geometry (eg, touching the inside of the geometry 2). When you perform a magnifying gesture (for example, touching the inside of a geometry and spreading and releasing two fingers) on a geometry (for example, sticking the fingers of a book), the process 1700 is one of the geometry. Move more than one edge or more than one corner at a time.
0151Process 1700 then crops the displayed image using geometric shapes (1725). In some embodiments, the process cuts off pieces of the displayed image that do not fall within the geometric shape. Then the process ends.
0152FIG. 18A conceptually illustrates GUI400 of some embodiments of image editing applications. Specifically, this figure shows that the user crops the displayed image 1810 by using the image cropping tool of an image editing application in six different stages 1801 to 1806.
0153As described above with reference to FIG. 16, some embodiments of the image cropping tool allow the user to crop unnecessary parts of the image by manipulating the edges and corners of the image. To enable. In some embodiments, the image cropping tool also allows the user to crop the displayed image by enlarging the image. The image cropping tool cuts off the portion of the image that protrudes to the outside of the image display area 410 after the image is enlarged.
0154From this, an operation example of an image editing application having GUI400 will be described. The first stage 1801 indicates the selection of the thumbnail 1810 by the user by touching the thumbnail 1810 with the finger 1815. The image editing application responds to the selection of thumbnail 1815 by displaying the image 1816 represented by thumbnail 1810 in the image display area 410.
0155At stage 1802, the user then selects icon 421, which is displayed in the tool selection pane 420. In response, the image editing application activates the image cropping tool at stage 1803. The image editing application highlights icon 421. The image editing application also displays the dial 435 in the tool display area 425 and the grid lines 450 in the image display area 410.
0156In step 1804, the user touches image 1810 with two fingers 1820 and 1825. The next step, 1805, shows the GUI 400 after the user has performed a magnified gesture on image 1810. In response, the cropping tool on the image magnifies the image 1810 and crops the portion of the image that extends outside the image display area 410. The image cropping tool stretched the cropped image to occupy the entire image display area 410. The final stage 1806 shows the GUI 400 after the user lifts fingers 1820 and 1825 from image 1810 and no longer touches image 1810.
0157Figures 18B and 18C conceptually show GUI400 of some embodiments of image editing applications. Specifically, this figure shows that the user crops the displayed image 1810 by using the image cropping tool of an image editing application in nine different stages 1801b to 1809b. Specifically, FIGS. 18B and 18C show GUI 400 of some embodiments having an inside display area 480.
0158Hereinafter, an operation example of an image editing application having GUI400 will be described. The first stage 1801b and the second stage 1802b are identical to the stages 1801 and 1802 described above with reference to FIG. 18A. In response to the selection of icon 421 in stage 1802a, the image editing application launches the crop tool in stage 1803a. The image editing application displays the inside display area 480a within the display area 410.
0159In step 1804a, the user touches image 1810 with two fingers 1820 and 1825. The next step, 1805a, shows the GUI 400 after the user has performed a magnifying gesture on image 1810 to magnify the image. In response, the Top Image Crop tool magnifies image 1810. As a result of the enlargement, the part of the image protruding to the outside of the inner display area 480a is displayed in the outer area 485a. Image boundaries are also displayed within the outer area 485a. The image cropping tool allows the user to further magnify the image, and some parts of the image may extend outside the display area 410.
0160The next step, 1806a, shows the GUI 400 after the user lifts fingers 1820 and 1825 from image 1810 and no longer touches image 1810. As shown, the image portions and image boundaries that were displayed within the outer area 485a are no longer displayed within the outer area 485a. The next step, 1807a, shows the GUI 400 after the user touches image 1810 again. The part of the image and the boundary of the image that were displayed in the outer area 485a in step 1805a reappear in the outer area 485a.
0161At step 1808a, the user drags the image to the right, so that different parts of the image 1810 are displayed within the inner display area 480a. In addition, different parts of the image protruding to the outside of the inner display area 480a are displayed in the outer area 485a this time. The next step, 1809a, shows the GUI 400 after the user lifts his finger from image 1810 and no longer touches image 1810. As shown, those parts of the image and the boundaries of the image that were displayed within the outer area 485a in the previous step 1808a are no longer displayed within the outer area 485a.
0162FIG. 19 conceptually illustrates GUI400 of some embodiments of image editing applications. Specifically, the figure shows that the user crops the displayed image 1910 by selecting a preset aspect ratio in four different stages 1901-1904.
0163The image editing application in some embodiments provides a set of aspect ratios that the user can choose from. The image editing application crops the image displayed in the image display area 410 using the selected aspect ratio. For example, the original image displayed within the image display area 410 has a 3x2 aspect ratio (eg, 3 is for width and 2 is for height), and the selected aspect ratio is 2x3. In some cases, the image editing application crops both sides of the original image and changes the aspect ratio to a 2x3 ratio.
0164The image editing application in some embodiments determines the portion to be truncated based on the location of the face (s) shown in the original image. The image editing application cuts off a portion of the image so that the rest of the image has a face (s) within the center of the image. Image editing applications use face detection techniques (eg, detect any face) and / or face recognition techniques (eg, detect faces of interest) to find faces (s).
0165If the image editing application finds multiple faces, different embodiments use different techniques to center the faces in the cropped image. For example, some image editing applications of embodiments identify the center coordinates of each found face, average the identified center coordinates, and cut out a portion of the image that has the average center coordinates. Place it in the center. As another example, some image editing applications of embodiments identify the size of each face found and place the largest face within the cropped image. As yet another example, some image editing applications of embodiments place the face of interest in the center of the cropped image.
0166In some embodiments, the image editing application provides a set of thumbnails along with a set of aspect ratios. Each of these thumbnails provides a preview of the cropped image based on the corresponding selectable aspect ratio.
0167Hereinafter, an operation example of an image editing application having GUI400 will be described. The first stage 1901 shows the GUI 400 after the user has selected an image to edit and selected icon 421. The image editing application placed the dial 435 in the tool display area 425 and highlighted the icon 421.
0168In step 1902, the user selects icon 1910 to indicate a preset aspect ratio. The image editing application displays preset aspect ratios that the user can choose from. The next step, 1903, shows the user's choice of 2x3 aspect ratio.
0169The final stage 1904 shows the GUI 400 after the image editing application crops the image 1910. The image editing application finds the face 1915 and crops the image so that the cropped image is centered on the face 1915. To make a 2x3 ratio from a 3x2 ratio, the sides of the image are cropped, not the top and bottom of the image, so the image editing application will crop the face 1915 to the left and right center of the cropped image. Place inside. The gray area displayed in the image display area 410 represents a part of the image that has been cut off.
0170Having described some embodiments of image cropping and angle correction tools, the next section is for applying image processing operations to selected areas of an image using image editing applications of some embodiments. Other image tools will be described.
0171II. Brush tool on image Image editing applications of some embodiments provide a set of on-image brush tools for applying image processing operations to selected areas of the displayed image. Some of these tools allow the user to select an area of the displayed image and apply image processing operations to the selected area. Some of these tools allow the user to select an area of the displayed image multiple times and apply image processing operations based on the number of times the image area is selected.
0172A. Applying image processing operations by selecting an area FIG. 20 conceptually illustrates GUI400 of an image editing application of several embodiments that allows a user to select an area within an image to which an image processing operation should be applied. Specifically, this figure shows in six different stages 2001-2006 that a user selects an area of an image by touching the image and applies an image processing operation to the selected area of the image.
0173As mentioned above, the tool selection pane 420 in some embodiments displays a set of icons representing different editing toolsets of the image editing application. The tool selection pane 420 includes icon 2020, which is the fourth icon from the left displayed in the tool selection pane 430 in some embodiments. The tools represented by the icon 2020 displayed in the tool selection pane 420 include the brush tools on the image. When the user selects Icon 2020, the image editing application in some embodiments will have a selectable UI item 2025 representing these image brush tools within the tool display area 425 (or any other part of the GUI 400). Display the set. When one of the tools is selected (for example, when the user selects an icon that represents that tool), that tool is launched and user input to the displayed image is input to the selected tool. Treated as.
0174The set of selectable UI items 2025 in some embodiments is a set of icons with brushes of different shapes depending on the image processing operation or effect. For example, icon 2030 represents a defect removal tool. Icon 2035 represents a red-eye removal tool. Icon 2040 represents a saturation tool. Icon 2045 represents a desaturation tool. Icon 2050 represents a brightening tool. Icon 2055 represents a darkening tool. Icon 2060 represents a sharpening tool. Icon 2065 represents a softening tool. When each tool is launched, it applies the effect to the area of the image selected by the user by touching the area.
0175Some of these tools apply the corresponding effect to an area, regardless of how many times the area is touched. That is, such tools apply the effect once when the area is touched for the first time, but do not apply the effect when the area is subsequently touched again. For example, the defect removal tool in some embodiments removes defects from the area when the user touches the area for the first time, but does not attempt to further remove the defects from the area when the user touches the area again. In some embodiments, another such tool is a red-eye removal tool.
0176Some of the tools represented by the set of selectable UI items 2025 apply the corresponding effect based on the number of times an area of the image is touched. Such tools will be described in more detail below.
0177From this, an operation example of an image editing application having GUI2000 will be described. The first stage 2001 shows the user's selection of thumbnail 2010 by touching thumbnail 2010 with a finger 2070. The image editing application displays the image 2015 represented by the thumbnail 2010 in the image display area 410 in response to the selection of the thumbnail 2010.
0178In stage 2002, the user selects the icon 2020 displayed in the tool selection pane 420. In response, the image editing application displays a set of selectable UI items 2025 within the GUI 400 in stage 2003, as illustrated. In stage 2004, the user selects a defect removal tool by touching the icon 2030. The image editing application launches a defect removal tool.
0179In stage 2005, the user uses his finger 2070 to rub the surface of the water and the tip of the oar multiple times from side to side. The defect removal tool identifies the area rubbed by the user and applies a defect removal operation to the area. U.S. Patent Application Publication No. 2009/0202170, published on 08/13/2009, provides detailed examples of defect removal operations. U.S. Patent Application Publication No. 2009/0202170 is incorporated herein by reference. In stage 2006, the image editing application removed the tip of the oar and the portion of the oar that was included in the area of the rubbed area.
0180FIG. 21 conceptualizes the process 2100 performed by some embodiments to allow a user of an image editing application to apply an image processing operation to an area of an image by touching the area. Shown in. Process 2100 in some embodiments is performed by an image editing application. Process 2100 begins when an image editing application runs on a computing device that has a touch-sensitive or near-touch-sensitive screen for receiving touch input (eg, rubbing gestures) from the user. Process 2100 begins by displaying an image within the image display area of the image editing application (in 2105).
0181Process 2100 then (at 2110) receives a selection of image processing operations to apply. In some embodiments, when the user touches a user interface item that represents an image processing operation, process 2100 receives a selection of the image processing operation through a user interface item (eg, an icon) in the image editing application. Examples of image processing operations include removal of defects, removal of red eyes, and the like.
0182Process 2100 then (at 2115) determines if an area of the image is selected. In some embodiments, process 2100 determines that an area of the image is selected when the user touches the area. If the process (at 2115) determines that no area has been selected, process 2100 loops back to 2110 to wait for the user to select an area.
0183If process 2100 (at 2115) determines that an area has been selected, process proceeds to 2120 to identify the image characteristics of the selected area. Examples of image characteristics identified by process 2100 include brightness, color, and the like. Process 2100 then (in 2125) applies the selected image processing operation to the selected area based on the identified image characteristics. Then the process ends.
0184The brush tools of some embodiments described so far use a constant brush size. That is, the width of the stroke created by using the brush tool with respect to the size of the image display area is constant. Image editing applications of some embodiments do not provide different brush sizes. In some such embodiments, the image editing application allows the effect to be applied with a finer grain size due to the brush tool having a constant brush size.
0185FIG. 22 conceptually illustrates GUI400 of some embodiments of image editing applications. Specifically, this figure enlarges the image so that the image editing application allows the user to apply the defect removal effect at a finer level with a brush that has a size that does not change with respect to the size of the image display area. It is shown in eight different stages 2201-2208 that it is possible to do so.
0186From this, an operation example of an image editing application having GUI400 will be described. At stage 2201, icon 2020 is highlighted to indicate that the image editing application is displaying a set of selectable UI items 2025 within GUI400. The user selected the defect removal tool by touching the icon 2030 with the finger 2210. The image editing application also displayed image 2215 within the image display area 410.
0187At step 2202, the user touches near the star shown in image 2215. At step 2203, the user drags his finger to the right while touching image 2215. Gray area 2220 shows the process created by the user. The height of the gray area 2220 represents the brush size. The brush size is small enough to cover the defects in the sky, but too large to apply the defect removal effect to the smaller objects shown in the image. For example, the brush size is too large to touch only the top circle of the speaker box shown in image 2215.
0188Stage 2204 indicates that the image editing application has removed two stars from the sky shown in image 2215. In step 2205, the user magnifies image 2215 by performing a magnifying gesture on image 2215. At step 2206, the user touches the upper circle of the speaker box shown in image 2215. At step 2207, the user drags his finger to the right while touching image 2215. Gray area 2225 represents a user-created process. The height of the gray area 2225 represents the brush size. The image has been magnified, so the same brush size is now small enough to touch only the top circle and eliminate the imperfections inside the top circle. The final stage 2207 shows that the defects in the upper circle have been eliminated.
0189FIG. 23 illustrates the use of another on-image brush tool that applies an image processing operation to a selected area of an image. Specifically, this figure shows that the user applies the red-eye removal effect to the image 2310 displayed in the image display area 410 of the GUI 400 at five different stages 2301 to 2305.
0190As mentioned above, some brush tools on the image apply the corresponding effect to the area, regardless of how many times the area is touched. The red-eye removal tool represented by icon 2035 is one such tool in some embodiments. Image editing applications of some embodiments may also provide an automatic red-eye removal tool that automatically removes all "red" from the red-eye in the displayed image. The red-eye removal tool represented by icon 2035 can be used before or after the automatic tool is used.
0191From this, an operation example of GUI400 using the red-eye removal tool will be described. Stage 1 2301 indicates the user's selection of thumbnail 2315 by touching thumbnail 2315 with a finger 2325. The image editing application responds to the selection of thumbnail 2315 by displaying the image 2310 represented by thumbnail 2315 in the image display area 410.
0192At stage 2302, the user selects the icon 2320 displayed in the tool selection pane 420. In response, the image editing application displays a set of selectable UI items 2325 within the GUI 400 at stage 2303, as illustrated. The user also selects the red-eye removal tool by touching the icon 2330. The image editing application launches the red-eye removal tool.
0193In step 2304, the user touches the right eye of the person shown in image 2310 (the left eye of image 2310) with his finger 2325. The red-eye removal tool in some embodiments identifies the pupil based on the area touched by the user. The red-eye removal tool then removes red from the identified pupil. In some embodiments, the red-eye removal tool uses conventional red-eye removal methods to remove red from the touched red-eye.
0194At step 2305, the user has raised his finger 2325 from the image and is no longer touching the image display area 410. This stage indicates that the image editing application has removed the "red" from the person's right eye.
0195In some embodiments, the red-eye removal tool reproduces sound when the red color is removed. Also, the red-eye removal tool in some embodiments provides visual instructions to indicate that red-eye is removed from the selected eye. For example, the red-eye removal tool displays an animation in which a circle extends from the selected eye. The red-eye removal tool cancels red-eye removal when the user selects the eye again. The red-eye removal tool plays another different sound when the tool cancels the red-eye removal.
0196B. Applying image processing operations based on the number of touches As mentioned above, some of the on-image brush tools in some embodiments apply the corresponding effect to the area based on the number of times the area is touched. Some examples of using such an image brush tool will be described below.
0197FIG. 24 conceptually illustrates GUI400 of an image editing application of several embodiments that allows a user to incrementally apply an image processing operation to a selected area of an image. Specifically, the figure shows that the user incrementally applies an image processing operation to an area of an image by rubbing the image in six different stages 2401 to 2406. FIG. 24 shows that the GUI 2400 includes an image display area 2425.
0198In some embodiments, some of the brush tools on the image allow the user to apply image processing operations by rubbing the image (eg, swiping an area of the image more than once). .. In some embodiments, such on-image brush tools include a saturation tool represented by icon 2040, a desaturation tool represented by icon 2045, and a brightening tool represented by icon 2050. Includes the darkening tool represented by icon 2055, the sharpening tool represented by icon 2060, and the softening tool represented by icon 2065.
0199Each time an area of the image is touched, the on-image brush tool applies an incremental amount of the corresponding image processing operation or effect. For example, a brightening tool gradually brightens the area of the displayed image (eg, increases the brightness value) each time the area is touched.
0200Different embodiments determine different amounts of increments of image processing operations to be applied. For example, the on-image brush tool increases or decreases the image characteristic value by a predetermined increment value based on the selected image processing operation. For example, if the selected image processing operation is a lightening operation, the on-screen brush tool increases the brightness value of the area by the default brightness value delta. In some embodiments, the on-image brush tool increases or decreases the image characteristic value by 1 percent. Also, the on-image brush tool in some embodiments uses a non-linear function to non-linearly reduce or increase the image characteristic value of the area of the image.
0201In some embodiments, the on-image brush tool uses mask values to incrementally apply image processing operations. A mask value for the image processing operation is given to the set of pixels of the displayed image. The on-image brush tool in some embodiments changes the mask value in order to apply the image processing operation to the set of pixels to which the mask value is assigned. That is, the on-image brush tool in these embodiments changes the mask value in order to incrementally apply the image processing operation.
0202In some embodiments, the number of touches the area receives is cumulative while the on-image brush tool is activated. That is, after the user has performed a rubbing operation over the area of the image (eg, swiping the area of the image at least once before raising the finger), the finger (or other device for touch, such as a stylus). If you raise and then perform a new rubbing operation again on the same area, the area is considered to be continuously rubbed. In some such embodiments, when another on-image brush tool is selected and activated to apply different corresponding effects, the number of times the area is touched is cumulative from zero or this. The brush tool on the image is last activated and accumulates from the number of times the area was touched when used.
0203Hereinafter, operation examples of the brush tool on the image of some embodiments will be described. The first stage 2401 shows how the image editing application is displaying the image 2420 in the image display area 410. The user selects the icon 2020 displayed in the tool selection pane 420.
0204At stage 2402, the image editing application displays a set of selectable UI items 2025 within the GUI 400, as illustrated. The user selects the brightening tool by touching the icon 2050. The image editing application launches a brightening tool.
0205At stage 2403, the image editing application in some embodiments highlights icon 2050 to indicate that the brightening tool has been activated. The user places his finger 2425 over the sky shown in image 2420. At step 2404, the user dragged finger 2425 to the right while touching image 2420. The brightening tool brightens the area touched by the finger 2425.
0206At step 2405, the user dragged finger 2425 to the upper left. The brightening tool brightens the area touched by the finger 2425. However, part 2430 is touched twice and therefore looks brighter than parts 2435 and 2440, which are touched only once.
0207At step 2406, the user dragged the finger 2425 back to where it was at step 2404. The brightening tool now brightens the area 2435 that was touched twice. Part 2430 has been touched three times, so part 2430 looks brighter than before.
0208FIG. 25 illustrates a process 2500 performed by several embodiments to allow a user of an image editing application to incrementally apply image processing operations to an area of an image by rubbing the area. Shown conceptually. Process 2500 in some embodiments is performed by an image editing application. Process 2500 begins when an image editing application runs on a computing device that has a touch-sensitive or near-touch-sensitive screen for receiving touch input (eg, rubbing gestures) from the user. Process 2500 begins by displaying an image within the image display area of the image editing application (in 2505).
0209Process 2500 (in 2510) then receives a selection of image processing operations to apply. In some embodiments, when the user touches a user interface item that represents an image processing operation, Process 2500 receives a selection of the image processing operation through a user interface item (eg, an icon) in the image editing application.
0210Process 2500 (at 2515) then determines if the area of the image is touched. In some embodiments, Process 2500 determines touch information (eg, a touch-sensitive screen or near-touch sensitive that the user's finger is touching) from the computing device to determine if an area of the image is touched. Receives or gets the coordinates of the area of the screen). If the process (at 2515) determines that the area has not been touched, process 2500 loops back to 2515 to wait for the user to touch the area of the displayed image.
0211If process 2500 (at 2515) determines that the area has been touched, process proceeds to 2520 to determine if the touch has been moved from the area. In some embodiments, if the touched area is no longer touched, process 2500 determines that the touch has been moved from the area. Therefore, if the user raises a finger from an area or the user moves a finger to another area of the image, Process 2500 determines that the touch has been moved from the area. If process 2500 (at 2520) determines that the touch has not been moved from the area, process 2500 loops back to 2520 to wait for the touch to move.
0212If Process 2500 (at 2520) determines that the touch has been moved from the area, Process 2500 (at 2525) incrementally applies the selected image manipulation to the area of the image. Process 2500 (at 2530) then determines if another area of the image is touched. If process 2500 (at 2530) determines that another area of the image is touched, process 2500 loops back to 2520 to wait for the touch to move out of this area. If process 2500 (at 2530) determines that the other areas of the image are not touched, process 2500 terminates.
0213Although the on-image brush tools of some embodiments that apply image processing operations based on the number of touches the area receives have been described, the next subsection describes some embodiments that apply image processing operations based on touch pressure. Explain the brush tool on the image of.
0214C. Applying touch pressure-based image processing operations FIG. 26 illustrates the use of an on-image brush tool in some embodiments that apply an image processing operation or effect based on the touch pressure applied to an area of the image. Specifically, this figure shows that the user can apply different levels of brightening effect to different areas of the displayed image by touching the image with different pressures from 6 different stages 2601 ~. Shown in 2606. FIG. 26 shows GUI 400, which is for some embodiments of an image editing application running on a computing device having a touch screen capable of measuring touch pressure, as described above.
0215In some embodiments, the on-image brush tool applies an effect to an area of the image based on touch pressure. That is, the on-image brush tool in these embodiments determines the amount of effect to be applied to the area based on the pressure applied to the area when it is touched. Therefore, the image editing application including the on-image brush tool in these embodiments must be running on a computing device having a touch-sensitive screen capable of measuring touch pressure. Such touch-sensitive screens may utilize an accelerometer to detect and measure the pressure exerted on the screen.
0216Hereinafter, operation examples of the brush tool on the image of some embodiments will be described. The first three steps 2601 to 2603 are identical to the steps 2401 to 2403 described above with reference to FIG. In step 2604, the user drags finger 2625 from where finger 2625 was in step 2603 to the right. The brightening tool brightens the area 2630 touched by the finger 2625.
0217At step 2605, the user dragged finger 2625 further to the right. However, the user pressed image 2420 harder while moving the finger to the right from where the finger 2625 was in the previous step 2604. Therefore, the lightening tool applies more lightening effects to the touched area 2635 than in the previous stage 2604. As a result, Area 2635 looks brighter than Area 2630.
0218At step 2606, the user dragged finger 2625 further to the right. However, the user pressed image 2420 even harder while moving his finger to the right from where his finger 2625 was in the previous stage 2605. Therefore, the lightening tool applies more lightening effects to areas 2640 touched after the previous stage 2605. As a result, Area 2640 looks brighter than Area 2630.
0219The on-image brush tools of some of the embodiments described so far apply the effect to the selected area. The next subsection provides a smart edge detection tool that allows the brush tool on the image to selectively apply effects to different parts of the selected area based on the image characteristics of these different parts. explain.
0220D. Smart edge detection tool Figure 27 conceptually illustrates the use of a smart edge detection tool. FIG. 27 shows GUI 400 of an image editing application of several embodiments that allows a user to selectively apply image processing operations to different parts of a selected area of an image. Specifically, the figure shows in five different stages 2701 to 2705 that the user applies the image processing operation only to those pixels in an area of the image that meets certain evaluation criteria.
0221As mentioned above, some image editing applications of embodiments allow the user to select an area to which the image processing operation should be applied. In some such embodiments, the smart edge detection tool allows the on-image brush tool to apply the corresponding image processing operation only to those pixels that meet certain criteria. For example, when the smart edge detection tool is running and the user chooses to apply a brightening operation to the selected area, the brightening tool will be blue in the selected area. Brighten the pixels, but leave the red pixels untouched.
0222In some embodiments, the smart edge detection tool identifies one or more pixels that are first touched by the user. For example, the tool first identifies the first pixel touched during the swipe operation of the user's image and uses the characteristics of the first pixel to determine the evaluation criteria. In another embodiment, the tool identifies a set of pixels (the first tens and hundreds of pixels) touched during a swipe operation on a user's image and evaluates using the characteristics of those pixels. Determine the criteria. Of all the pixels touched during the swipe operation, the only enabled on-image brush tools are those pixels that meet the evaluation criteria. The details of determining the evaluation criteria will be further described below.
0223In some embodiments, the smart edge detection tool redefines the criteria each time the image is touched (ie, each time the user's finger touches the image again after the finger has been lifted from the image). To do. In other embodiments, the smart edge detection tool does not redefine the criteria until a different on-image brush tool is selected or the smart edge detection tool stops working. That is, in these embodiments, the first determined endpoint is for all subsequent touches until a different on-image brush tool is selected or the smart edge detection tool is stopped working. Used.
0224This technique is also applicable to some image editing applications of the embodiments described above, which incrementally apply image processing operations to an area of an image each time the area is touched by the user. That is, the image editing application of some embodiments incrementally applies the image processing operation only to those pixels that meet the evaluation criteria.
0225In addition, some image editing applications use edge detection to identify different areas within the touch area that are separated by the detected edges. The image editing application of these embodiments applies an image processing operation to those pixels that meet the evaluation criteria and are within the specified area containing the first set of touched pixels.
0226From this, an operation example of the brush tool on the image while the smart edge detection tool is activated will be described. The first stage 2701 shows how the image editing application is displaying the image 2720 in the image display area 410. The user selects the icon 2020 displayed in the tool selection pane 420.
0227At stage 2702, the image editing application displays a set of selectable UI items 2025 within the GUI 400, as illustrated. The user selects the brightening tool by touching the icon 2050. The image editing application launches a brightening tool.
0228At stage 2703, the image editing application in some embodiments highlights icon 2050 to indicate that it was the brightening tool that was launched. The user selects icon 2710, which represents a smart edge detection tool. In some embodiments, the image editing application displays the icon 2710 within the tool display area 425. Also, in some embodiments, the order in which the user selects the brush tool on the image and the smart edge detection tool does not matter. That is, the smart edge detection tool enables any other on-image brush tool that is launched once it is launched.
0229At stage 2704, the user places his finger 2730 on the sea near the bottom edge of the oar shown in image 2720. The smart edge detection tool analyzes the pixels in the part of the image 2720 that was first touched by the finger 2730, and for the other pixels in the other parts of the image, the brightening tool puts them on those pixels. Set the evaluation criteria that must be met in order to apply the lightening effect. Since the part of the image that is first touched is inside the ocean shown in image 2720, the brightening tool in this example is only on those pixels that have a color similar to the ocean color. Apply a lightening effect.
0230At step 2705, the user dragged finger 2730 to the right while touching image 2720. The brightening tool brightens the area touched by the finger 2725. However, the brightening tool does not brighten the bottom edge of the oar shown in image 2720, even if the finger 2730 touches this part of the oar. This is because smart edge detection is activated and the brightening tool allows only those pixels that meet the evaluation criteria set in the previous stage 2704 to be brightened.
0231FIG. 28 illustrates a process 2800 performed by some embodiments to allow a user of an image editing application to selectively apply image processing operations to an area of an image by rubbing the area. Shown conceptually. Process 2800 in some embodiments is performed by an image editing application. Process 2800 begins when an image editing application runs on a computing device that has a touch-sensitive or near-touch-sensitive screen for receiving touch input (eg, rubbing gestures) from the user. Process 2800 begins by displaying the image within the image display area of the image editing application (in 2805).
0232Process 2800 then receives a selection of areas of the image (at 2810). Process 2800 (at 2815) then identifies certain criteria for determining the similarity between pixels. Process 2800 in some embodiments uses the image characteristics of some of the pixels in the selected area. More specifically, process 2800 in some embodiments uses a set of pixels in the selected area that was first touched by the user.
0233Different embodiments use image characteristics to specify different evaluation criteria. For example, Process 2800 in some embodiments uses the average color value of a set of pixels. Process 2800 converts the average color value of the RGB color model to the color value of the LAB color space. Process 2800 then defines the range of distances from the color values in the LAB color space as the evaluation criteria. U.S. Pat. No. 8,229,211 provides further details on the conversion of RGB color model color values to LAB color space color values. U.S. Pat. No. 8,229,211 is incorporated herein by reference.
0234Process 2800 (in 2820) then applies image processing operations (eg, saturation, desaturation, lightening, darkening, sharpening, softening, etc.) to pixels that meet the specified evaluation criteria. In some embodiments, Process 2800 applies an image processing operation to each pixel in an area that has a color value in the LAB color space that falls within a defined range. Then the process ends.
0235E. Unapply image processing operations As mentioned above, some of the on-image brush tools in some embodiments apply the corresponding effect to the area. In some embodiments, the image editing application provides an on-image brush tool that removes the effect applied to the area.
0236FIG. 29 conceptually illustrates GUI 400 of an image editing application of several embodiments that allows a user to deapply an image processing operation from an area of an image. Specifically, the figure shows in five different stages 2901-2905 that the user gradually deapplies the image processing operation applied to the image area by rubbing the image area. FIG. 29 shows that the GUI 2400 includes an image display area 2425.
0237In some embodiments, the image editing application of some embodiments has been applied to an area of the image by the user rubbing the area of the image (eg, swiping the area of the image one or more times). Provides an eraser tool that allows an image processing operation to be unapplied. For example, the eraser tool darkens a specific area of the image to which the lightening effect is applied by removing the lightening effect. That is, the eraser tool returns the area of the image to its previous state before the lightening effect was applied.
0238In some embodiments, the eraser tool incrementally deapplies the image processing operation each time an area of the image is touched. For example, the eraser tool darkens certain areas of an image to which a lightening effect is applied by an incremental amount. When a particular area is touched again, the eraser tool darkens the area by an increment. In some embodiments, the eraser tool does not deapply the image processing operation when the area of the image returns to the state it was in before the image processing operation was applied.
0239Different embodiments differently determine the increment of the image processing operation to be removed. For example, the eraser tool increases or decreases the image characteristic value by a predetermined increment value based on the selected image operation. For example, if the applied image processing operation is a lightening operation, the eraser tool reduces the luminance value of the area by a default luminance value delta. This delta may represent the same amount of brightness that some on the image brush tool applies to the area with each touch. In some embodiments, the eraser tool increases or decreases the image property value by 1 percent. Also, the eraser tool in some embodiments uses a non-linear function to non-linearly reduce or increase the image characteristic value of the area of the image.
0240The eraser tool uses a mask value to incrementally deapply the image processing operation, as the on-image brush tool of some embodiments does as described above. That is, the eraser tool in some embodiments changes the mask value in order to unapply the image processing operation to the set of pixels to which the mask value is assigned.
0241In some embodiments, the eraser tool deapplies the image processing operation from the area of the image based on the touch pressure. That is, the eraser tool in these embodiments determines the amount of image processing operations to be deapplied from the area based on the pressure applied to the area when the area is touched. Therefore, the image editing application including the eraser tool of these embodiments must be running on a computing device having a touch sensitive screen capable of measuring touch pressure. As mentioned above, such touch-sensitive screens may utilize an accelerometer to detect and measure the pressure applied to the screen.
0242In some embodiments, the eraser tool does not incrementally deapply the image processing operation. Instead, the eraser tools of these embodiments completely remove the image processing operations applied to the area of the image. That is, the eraser tool restores the area of the image to its original state before the image processing operation was applied.
0243Hereinafter, operation examples of the eraser tool of some embodiments will be described. The first stage 2901 shows how the image editing application is displaying the image 2420 in the image display area 410. In this example, the first stage 2901 shows the GUI 400 after the brightening effect has been applied to area 2910 of image 2420. The image editing application also displays the icon 2915 within the tool display area 425. Icon 2915 represents an eraser tool.
0244At stage 2902, the user selects icon 2915 and the image editing application launches the eraser tool in response to the selection of icon 2915. At stage 2903, the user places his finger 2920 over the sky shown to the left of area 2910 in image 2420. The next step, 2904, indicates that the user has touched image 2420 and dragged his finger 2920 to the right past area 2910. The eraser tool gradually removes the lightening effect from area 2925, which is the intersection of area 2910 and the area swiped by the finger 2920. As shown, the eraser tool darkened area 2925.
0245At step 2904, the user dragged finger 2920 to the left. As the user drags finger 2920 to the left, finger 2920 touches area 2925 again. The eraser tool makes Area 2925 even darker. In this example, the eraser tool completely moved the lightening effect applied to area 2925 so that area 2925 returned to the state it was in before the lightening effect was applied.
0246FIG. 30 conceptualizes a process 3000 performed by some embodiments to allow a user of an image editing application to deapply an image processing operation from an area of the image by rubbing the area. Shown in. Process 3000 in some embodiments is performed by an image editing application. Process 3000 begins when an image editing application runs on a computing device that has a touch-sensitive or near-touch-sensitive screen for receiving touch input (eg, rubbing gestures) from the user. Process 3000 starts when the displayed image has an area to which the image processing operation has been applied.
0247Process 3000 begins by receiving user input (in 3005) that causes the image editing application to launch some embodiments of the eraser tool. In some embodiments, such user input includes selecting an icon (eg, icon 2915 shown in FIG. 29 above).
0248Process 3000 then (at 3010) determines if an area of the image is touched. In some embodiments, process 3000 determines touch information (eg, a touch-sensitive screen or near-touch sensitive that the user's finger is touching) from the computing device to determine if an area of the image is touched. Receives or gets the coordinates of the area of the screen). If the process (at 3010) determines that the area has not been touched, process 3000 loops back to 3010 to wait for the user to touch the area of the displayed image.
0249If process 3000 (at 3010) determines that the area has been touched, the process proceeds to 3015 to determine if the touch has been moved from the area. In some embodiments, if the touched area is no longer touched, process 3000 determines that the touch has been moved from the area. Therefore, if the user raises a finger from an area or the user moves a finger to another area of the image, process 3000 determines that the touch has been moved from the area. If process 3000 (at 3015) determines that the touch has not been moved from the area, process 3000 loops back to 3015 to wait for the touch to move.
0250If process 3000 (at 3015) determines that the touch has been moved from the area, process 3000 (at 3020) determines whether the area overlaps the area to which the image processing operation has been applied. If process 3000 (at 3020) determines that there is an overlapping area, process 3000 (at 3025) deapplies the image processing operation from the overlapping area by an incremental amount. Process 3000 does not deapply the image processing operation if there is nothing left to deapply.
0251Different embodiments differently determine the increment of the image processing operation to be removed. For example, process 3000 in some embodiments increases or decreases the image characteristic value by a predetermined increment value based on the image processing operation applied to the overlapping area. For example, if the image processing operation applied to the overlapping areas is a lightening operation, process 3000 reduces the luminance values of the areas by a default luminance value delta. This delta may represent the same amount of brightness value that some on the image brush tool apply with each touch. In some embodiments, Process 3000 increases or decreases the image characteristic value by 1 percent. Process 3000 in some embodiments also uses a non-linear function to non-linearly reduce or increase the image characteristic value of the area of the image.
0252If process 3000 (at 3020) determines that there are no overlapping areas, process proceeds to 3030. Process 3000 then (at 3030) determines if another area of the image is touched. If process 3000 (at 3030) determines that another area of the image is touched, process 3000 loops back to 3015 to wait for the touch to move out of this area. If process 3000 (at 3030) determines that the other areas of the image are not touched, process 3000 terminates.
0253Section II described some on-image brush tools in some embodiments. The next section describes some on-image tools that apply some special effects to an image.
0254III. Image effect tool Some embodiments provide an on-image effect tool that allows the user to apply an image processing operation or effect by selecting a portion of the displayed image. These image effect tools are not brush tools. That is, the image on effect tool Lumpur in the area, applying the effect that is not defined by the brush tool.
0255A. Gradient tool on the image FIG. 31 conceptually illustrates the use of image effect tools of several embodiments that allow a user to selectively apply image processing operations to different parts of a selected area of an image. Specifically, this figure shows that the user applies a gradient effect to a portion of an image by touching the image in five different stages 3101-3105. Figure 31 shows the GUI 400.
0256As mentioned above, the tool selection pane 420 in some embodiments displays a set of icons representing different editing toolsets of the image editing application. The tool selection pane 420 includes icon 3110, which is the fifth icon from the left displayed in the tool selection pane 430 in some embodiments. Icon 3110 displayed in the tool selection pane 420 represents a group of image effect tools in some embodiments. When the user selects icon 3110, the image editing application in some embodiments has a set of selectable UI items 3125 representing these image effect tools within the tool display area 425 (or any other part of the GUI 400). Is displayed. When one of the tools is selected (when the user selects the icon that represents that tool), that tool is launched and the user input to the displayed image is treated as input to the selected tool.
0257The set of selectable UI items 3125 in some embodiments is a set of thumbnails and icons. In some embodiments, the set of selectable UI items 3125 is on top of one of the cards that expands when icon 3110 is selected. US Provisional Patent Application No. 61 / 607,574 provides further details on card deployment.
0258Thumbnails and icons in the set of selectable UI items 3125 represent a number of different effects. For example, UI item 3130 represents a dark gradient tool on the image. UI item 3135 represents a warm color gradient tool on the image. UI item 3140 represents a cool color gradient tool on the image. UI item 3145 represents a vignette tool. UI item 3150 represents a tilt shift tool. Each tool, when activated, applies an effect to an area of the image defined by the user by touching the area.
0259From this, an operation example of the gradation tool on an image will be described. The first stage 3101 shows the user's selection of thumbnail 3110 by touching the thumbnail 3110 with a finger 3170. The image editing application responds to the selection of the thumbnail 3110 by displaying the image 3115 represented by the thumbnail 3110 in the image display area 410.
0260At step 3102, the user selects the icon 3110 displayed in the tool selection pane 420. In response, the image editing application displays a set of selectable UI items 3125 within the GUI 400 at stage 3103, as illustrated. The user selects the dark gradient tool on the image by touching UI item 3130. The image editing application launches a dark gradient tool on the image.
0261At step 3104, the user touched a location near the top of image 3115 with finger 3170, or dragged finger 3170 down to that location while touching image 3115. The Dark Gradient Tool on the Image applied a dark gradient effect from the top of the image 3115 to the location where the finger 3170 is. The part of the image above the finger 3170 is drawn darker than before to indicate that a gradient effect has been applied to this part.
0262Step 3105 shows the GUI 400 after the user touches the location of the image closer to the bottom of the image 3115 or drags the finger 3170 to this location while touching the image 3115. The Dark Gradient tool applied a gradient effect from the vertical location where the finger 3170 was in the previous stage 3104 to the vertical location of the finger 3170 in the current stage 3105. The part of the image above the finger 3170 is drawn darker than before to indicate that a gradient effect has been applied to this part.
0263In some embodiments, the on-image dark gradient tool allows the user to apply a gradient effect by touching two locations in the image. When the user touches two locations, the top location and the bottom location, the Dark Gradient Tool on the image draws an invisible horizontal line that intersects each of the two locations, and the portion of the image between the two lines. Apply a gradient effect on top of it. In some embodiments, these two lines are parallel lines, and the distance between the two lines is one or both of the two fingers used to touch the two locations. Can be adjusted by dragging them closer to or farther from each other. Further, in some embodiments, the on-image dark gradient tool allows the user to rotate the two lines so that the two lines are not horizontal. In addition, the Dark Gradient Tool on the Image applies different amounts of gradient effect on the portion of the image between the two lines. For example, in some embodiments, the pixels near one of the two lines are the darkest (eg, representing a 100% gradient effect) and the pixels near the other line are the brightest (eg, 0). Represents a% gradient effect). In some such embodiments, the amount of gradient effect applied to the pixels in the portion depends on the proximity of the pixel to either of the two lines. The use of two lines is also applicable to other gradient tools described in this application (eg, warm gradient tools).
0264FIG. 32 conceptually illustrates the process 3200 performed by some embodiments to allow the user of an image editing application to apply a gradient effect to an image. Process 3200 in some embodiments is performed by an image editing application. Process 3200 begins when the image editing application runs on a computing device that has a touch-sensitive or near-touch sensitive screen to receive touch input from the user (eg, tap, swipe, etc.). Process 3200 begins by displaying an image within the image display area of the image editing application (in 3205).
0265Process 3200 then (in 3210) receives a touch input on the location in the displayed image. In some embodiments, process 3200 receives a touch input on the location when the user taps on the location. Process 3200 receives a touch input when the user touches the displayed image (for example, without raising his finger) and drags his finger from another location in the displayed image.
0266Process 3200 then applies a gradient effect to the image (at 3215). When the user taps a location in the displayed image, Process 3200 applies a gradient effect from the top of the image to that location in the image. When the user drags his finger from the first location of the image to a second location farther from the top of the image than the first location, Process 3200 runs the image vertically across the top of the image to the second location. Apply a gradient effect to the part of. If a gradient effect has already been applied to the portion of the image that spans the top of the image to the first location, Process 3200 will perform a gradient effect on the portion of the image that extends vertically from the first location to the second location. To apply.
0267Some of the different on-image gradient effect tools mentioned above apply different gradient effects to the image. When applied, these different gradient effects provide different visual features. For example, the part of the image to which the "dark gradation" effect is applied looks dark. The "warm gradient" effect makes parts of the image look "warm". The on-image gradient tool achieves these different visual features by treating the pixels of the image differently. The next four figures, Figures 33-36, show examples of two different gradient effects that the on-image gradient effect tool can apply.
0268FIG. 33 conceptually illustrates the architecture of an on-image gradient effect tool that applies a particular gradient effect to a portion of an image selected by the user. Specifically, this figure shows that the on-image gradient effect tool applies a "blue" gradient effect. As illustrated, FIG. 33 shows the gradient effect tool 3300, the original image data 3330, and the gradient image 3335.
0269The original image 3330 is image data for the selected image before the gradient effect is applied. The selected image is an image displayed in the image display area of the image editing application. The gradation image 3335 is image data for an image created by applying a gradation effect to the image. In some embodiments, the gradient image 3335 is a set of instructions to execute to generate a new image from the original image.
0270The Gradient Effect Tool 3300 applies a blue gradient effect to the original image. The gradient effect tool 3300 includes a black and white image generator 3305, a color image generator 3310, an image mixer 3315, a gradient calculator 3325, and a gradient image generator 3320.
0271The black and white (B / W) image generator 3305 generates a black and white image based on the original image 3330. In some embodiments, the B / W image generator 3305 emphasizes the blue color in the black and white image. The B / W image generator 3305 in some embodiments uses a blue RGB triplet, eg (0.02, 0.018, 0.77). The three numbers in the triplet represent the three primary color channels-red, green, and blue, respectively. Each of the three numbers has a range of 0 to 1, representing a range of colors. For example, for the first number in parentheses, 0 represents no blue and 1 represents maximum blue. Those skilled in the art recognize that other ranges, such as 0-255, 0% -100%, 0-65535, etc., may be used for RGB values.
0272To generate a B / W image that emphasizes blue, the B / W image generator 3305 calculates the dot product of each pixel in the original image and the square of the blue RGB triplet. Such a dot product can be represented by the following pseudo code. c1 = dot (image.rgb, blue.rgb<sup>2</sup>); image.rgb is the RGB triplet of the pixels in the original image. blue.rgb is a blue RGB triplet. C1 is the dot product and is also the brightness value for the corresponding pixel in the resulting B / W image (eg, brightness value, RGB sum, etc.). Brightness values for pixels in a B / W image have a (grayscale) range of 0 to 1. Here, 0 is black and 1 is white.
0273The color image generator 3310 converts the B / W image generated by the B / W image generator 3305 into a blue image. In some embodiments, the color image generator 3310 squares the brightness value of the corresponding pixel in the B / W image, multiplies the blue RGB triplet by the squared brightness value, and then multiplies. By doubling the result of, we generate an RGB triplet for each pixel of the blue image. This could be represented by the following pseudo code. blueImage = 2.0 * (c1<sup>2</sup>) * blue.rgb. blueImage is the RGB value of the pixels in the generated blue image.
0274The image mixer 3315 uses a grayscale of 0 to 1 to ensure that bright areas in the image retain their whiteness while darker areas appear blue. And the blue image are mixed. Mixing can be performed using the following pseudo-code example. mixedImage = mix (originalImage, blueImage, (1.0-c1)). A mixedImage is an RGB triplet of pixels in a mixed image. The originalImage is the RGB triplet of the corresponding pixel in the original image. mix () is a function. Different embodiments use different mix () functions. In some embodiments, the mix () function calculates the arithmetic mean of three input values / triplets. If the triplets are averaged using a single value (eg 1.0-c1), then each value is averaged using that single value.
0275The gradient calculator 3325 calculates the gradient value for each pixel of the original image. The gradient calculator 3325 in some embodiments calculates a gradient value for a particular pixel based on the location of the pixels in the original image. Since the on-image gradient effect tool in some embodiments applies a gradient effect based on the vertical location of the user's touch, the gradient calculator 3325 calculates the gradient value based on the row number of a particular pixel. The calculation of the gradient value for a particular pixel is also based on the line number of the touched location and the overall height of the original image.
0276In some embodiments, the row number of the pixel is the y coordinate of the pixel, the top pixel (ie, the top row of pixels in the image) has a y coordinate of 0, and the bottom pixel (ie, the bottom pixel (ie)). That is, the bottom row of pixels in the image) has the y-coordinate of the total height (ie, the number of pixels in the column of pixels in the image-1). That is, the y coordinate starts at the top row of the image.
0277The gradient calculator 3325 in some embodiments sets the gradient value for a particular pixel to 0 if the particular pixel is below the vertical location of the user's touch. This is represented by the following pseudo code. If row> heightTouched, gradient = 0.0; row is the y-coordinate value of a particular pixel. heightTouched is the y coordinate of the user's touch (eg, the vertical center pixel (s) of the set of touched pixels). The gradient calculator obtains the heightTouched value from touch information (ie, user input) from a user interface module (not shown) that receives the user's touch.
0278The gradient calculator 3325 in some embodiments calculates a gradient value for a particular pixel using the following pseudocode when the particular pixel is above the vertical location of the user's touch. topY = min (0.0, heightTouched.n-0.2); r = line / (height-1); r = (r-topY) / (heightTouched.n-topY); If (r <0.8), gradient = (0.1r + (0.8-r) * 0.8) /0.8; Otherwise, if (r <0.9), then gradient = ((r-0.8) * 0.03 + (0.9-r) * 0.1) /0.1; If not, gradient = ((r-0.9) * 0.03 + (1-r) * 0.1) /0.1; TopY is calculated to apply a gradual attenuation of the gradient effect for pixels near the top of the image. r is a normalized line number with a range of 0 to 1. height is the overall height of the original image (ie, the number of pixels in a row of pixels). heightTouched.n is a normalized heightTouched and has a range of 0 to 1. Here, 0 is the top pixel line of the original image, and 1 is the bottom pixel line of the original image.
0279The gradient image generator 3320 blends the mixed image with the original image using the gradient values calculated by the gradient calculator 3325. In some embodiments, the gradient image generator 3320 uses the following pseudocode to blend the mixed image with the original image to generate a new image: gradImage = originalImage * (1-gradient) + mixedImage * gradient; A gradImage is an RGB triplet of pixels in a new image generated by applying a blue gradient effect to the original image.
0280From this, an operation example of the blue gradation effect tool 3300 will be described with reference to FIGS. 33 and 34. FIG. 34 conceptually illustrates the process 3400 that some embodiments perform to apply a blue gradient effect to an image. The process 3400 in some embodiments is performed by an on-image gradient effect tool such as the gradient effect tool 3300. Process 3400 begins when the Gradient Effect Tool on Image is launched by an image editing application and receives image data for the displayed image.
0281Process 3400 begins by generating a black and white (B / W) image that emphasizes one color based on the original image (in 3405). Process 3400 in some embodiments emphasizes blue in the B / W image by using a blue RGB triplet, eg (0.02, 0.018, 0.77). The values in the blue RGB triplet may vary from embodiment to embodiment as long as blue is greater than red and green. For example, the difference between blue in a triplet and other colors may be a single digit.
0282Process 3400 in some embodiments uses RGB triplets of the corresponding pixels of the original image (ie, pixels with the same coordinates) to calculate the brightness value for each pixel of the B / W image. For example, the B / W image generator 3305 is a pixel of a B / W image by taking the dot product of the corresponding pixel RGB triplets (0.5, 0.5, 0.5) of the original image with the blue RGB triplet. Calculate the brightness value for. Then the brightness value for the pixels of the B / W image would be 0.02 * 0.5 + 0.018 * 0.5 + 0.77 * 0.5. This will be 0.404.
0283Process 3400 then (in 3410) produces a color image for the emphasized color based on the B / W image. In some embodiments, process 3400 produces a blue image by calculating an RGB triplet for each pixel of the blue image based on the brightness value of the corresponding pixel in the B / W image. For example, the color image generator 3310 calculates the RGB triplet of a pixel for a blue image using a previously generated brightness value of 0.404 for the corresponding pixel in the B / W image. The RGB triplet of the pixels in the blue image is 2.0 * (0.404)<sup>2</sup>) * (0.02, 0.018, 0.77), which is (0.00652864, 0.005875776, 0.25135264).
0284Process 3400 (at 3415) then mixes the original image and the generated color image based on the B / W image. For example, the image mixer 3315 mixes the pixels of the original image and the color image generated using the brightness value 0.404, as well as the corresponding pixels in the B / W image. As mentioned above, the original pixel has RGB triplets (0.5, 0.5, 0.5). The RGB triplet of the corresponding pixel in the blue image is (0.00652864, 0.005875776, 0.25135264). Therefore, in some embodiments, the RGB triplets of the corresponding pixels in the mixed image are ((0.5 + 0.00652864 + 0.0404) / 3, (0.5 + 0.005875776 + 0.0404) / 3, (0.5 + 0.25135264 + 0.0404)). / 3), which becomes (0.182309547, 0.182091925, 0.263917547).
0285Process 3400 (at 3420) then receives a touch input on the location in the original image. In some embodiments, process 3400 receives a touch input on the location when the user taps on the location. Process 3400 receives a touch input when the user touches the displayed image (eg, without raising his finger) and drags his finger from another location in the original image. For example, the original image has a height of 768 pixels and the touch location is the 300th pixel from the top of the image. In some embodiments, the pixel considered touched is the pixel at the center of the set of pixels touched by the user's finger. In some embodiments, the pixel considered touched is the average pixel coordinate of a set of pixels touched by the user's finger.
0286Process 3400 then calculates the gradient value for each pixel of the original image (at 3425). In some embodiments, process 3400 calculates a gradient value for a particular pixel based on the location of the pixel in the original image. Since the on-image gradient effect tool in some embodiments applies a gradient effect based on the vertical location of the user's touch, process 3400 calculates the gradient value based on the row number of a particular pixel. The calculation of the gradient value for a particular pixel is also based on the line number of the touched location and the overall height of the original image.
0287For example, the gradation calculator 3325 calculates the gradation value for the pixel that is the 200th pixel from the top of the original image. The gradation calculator 3325 calculates topY. topY is the minimum of 0.0 and (200/768) -0.2, which is 0.0. The gradation calculator 3325 calculates the normalized line number. The normalized line number is (200 / (768-1) -0.0) / (300 / 768-0.0), which is 0.668407311. Next, the gradation calculator 3325 calculates the gradation value. Since the normalized line number is less than 0.8, the gradient value is (0.1 * 0.668407311 + (0.8-0.668407311) * 0.8) /0.8, which is 0.215143603.
0288Process 3400 (at 3430) then blends the mixed image with the original image using the calculated gradient values. Process 3400 calculates the RGB triplet for each pixel of the blended image. For example, the gradient image generator 3320 calculates an RGB triplet for the pixels in the blended image. The corresponding pixel in the original image is the pixel that is the 200th pixel from the top. Therefore, the RGB triplet for the pixels in the blended image is (0.5, 0.5, 0.5) * (1-0.215143603) + (0.182309547, 0.182091925, 0.263917547) * 0.215143603, which is (0.431650931, 0.431604111, 0.44920837). Become.
0289Figure 35 conceptually illustrates the architecture of an on-image gradient effect tool that applies a particular gradient effect to a portion of an image selected by the user. Specifically, this figure shows that the on-image gradient effect tool applies a "coffee" gradient effect. As illustrated, FIG. 35 shows the gradient effect tool 3500, the original image data 3520, and the gradient image 3525.
0290The original image 3520 is image data for the selected image before the gradient effect is applied. The selected image is an image displayed in the image display area of the image editing application. The gradation image 3525 is image data for an image created by applying a gradation effect to the image. In some embodiments, the gradient image 3525 is a set of instructions to execute to generate a new image from the original image.
0291The Gradient Effect Tool 3500 applies a coffee gradient effect to the original image. The gradient effect tool 3500 includes a black and white image generator 3505, an image mixer 3510, a gradient calculator 3325, and a gradient image generator 3515.
0292The black and white (B / W) image generator 3505 generates a black and white image based on the original image 3530. The B / W image generator in some embodiments uses a B / W RGB triplet, eg (0.2, 0.7, 0.1), to generate a B / W image. Each of the three numbers in the triplet has a range of 0 to 1 that represents the range of colors. To generate a B / W image, the B / W image generator 3505 calculates the dot product of RGB triplets and B / W RGB triplets for each pixel in the original image. The B / W image generator 3505 in some embodiments also scales the dot product by 2.0 to obtain a brighter B / W image. Such a dot product can be represented by the following pseudo code. bw = 2.0 * dot (im, (0.2, 0.7, 0.1)) = 2.0 * (im.r * 0.2 + im.g * 0.7 + im.b * 0.1); bw is the brightness value of the pixels in the B / W image. im is the RGB triplet (im.r, im.g, im.b) of the corresponding pixel in the original image.
0293The image mixer 3510 mixes the original image with the B / W image by using a coffee RGB triplet that defines a brownish color. In some embodiments, the coffee RGB triplet is (0.7, 0.5, 0.35). The image mixer 3510 in some embodiments mixes the original image and the B / W image by multiplying the RGB values of the pixels of the original image by the brightness values of the pixels of the coffee RGB triplet and the B / W image. To do. Such a mixture can be represented by the following pseudo code. mixedImage = im * coffeeRGB * bw; A mixed image is an RGB triplet of pixels in a mixed image. coffeeRGB is a coffee RGB triplet. bw is the RGB triplet of the corresponding pixel in the B / W image generated by the B / W image generator 3505.
0294The gradient image generator 3520 blends the mixed image with the original image using the gradient values calculated by the gradient calculator 3325. In some embodiments, the gradient image generator 3520 uses the following pseudocode to blend the mixed image with the original image to generate a new image. gradImage = originalImage * (1-gradient) + mixedImage * gradient; A gradImage is an RGB triplet of pixels in a new image generated by applying a coffee gradient effect to the original image.
0295From this, an operation example of the coffee gradation effect tool 3500 will be described with reference to FIGS. 35 and 36. FIG. 36 conceptually illustrates the process 3600 that some embodiments perform to apply a coffee gradient effect to an image. The process 3600 in some embodiments is performed by an on-image gradient effect tool such as the gradient effect tool 3500. Process 3600 begins when the Gradient Effect Tool on Image is launched by an image editing application and receives image data for the displayed image.
0296Process 3600 begins by generating a black and white (B / W) image (in 3605). Process 3600 in some embodiments produces a B / W image by using a B / W RGB triplet, eg (0.2, 0.7, 0.1). Process 3600 in some embodiments uses RGB triplets of the corresponding pixels of the original image (ie, pixels with the same coordinates) to calculate the brightness value for each pixel of the B / W image. For example, the B / W image generator 3505 takes a B / W image by taking the dot product of the corresponding pixel RGB triplets of the original image, eg (0.3, 0.4, 0.5) and the B / W RGB triplets. Calculate the brightness value for the pixels of. Then the brightness value for the pixels of the B / W image would be 2.0 * (0.3 * 0.2 + 0.3 * 0.7 + 0.5 * 0.1). This will be 0.64.
0297Process 3600 (at 3610) then mixes the original image and the B / W image using a coffee RGB triplet, eg (0.7, 0.5, 0.35). For example, the image mixer 3515 mixes the pixels of the original image and the generated color image using a brightness value of 0.64 for the corresponding pixels in the B / W image. As mentioned above, the original pixel has RGB triplets (0.3, 0.4, 0.5). Therefore, the RGB triplet of the corresponding pixel in the mixed image is (0.3, 0.4, 0.5) * (0.7, 0.5, 0.35) * 0.64, which is (0.0768, 0.128, 0.112).
0298Process 3600 (at 3615) then receives a touch input on the location in the original image. In some embodiments, process 3600 receives a touch input on the location when the user taps on the location. Process 3600 receives a touch input when the user touches the displayed image (eg, without raising his finger) and drags his finger from another location in the original image. For example, the original image has a height of 768 pixels and the touch location is the 400th pixel from the top of the image. In some embodiments, the pixel considered touched is the pixel at the center of the set of pixels touched by the user's finger. In some embodiments, the pixel considered touched is the average pixel coordinate of a set of pixels touched by the user's finger.
0299Process 3600 then calculates the gradient value for each pixel of the original image (in 3620). In some embodiments, process 3600 calculates a gradient value for a particular pixel based on the location of the pixel in the original image. Since the on-image gradient effect tool in some embodiments applies a gradient effect based on the vertical location of the user's touch, Process 3600 calculates the gradient value based on the row number of a particular pixel. The calculation of the gradient value for a particular pixel is also based on the line number of the touched location and the overall height of the original image.
0300For example, the gradation calculator 3325 calculates the gradation value for the pixel that is the 300th pixel from the top of the original image. The gradient calculator 3525 calculates topY. topY is the minimum of 0.0 and (300/768) -0.2, which is 0.0. The gradient calculator 3525 calculates the normalized line number. The normalized line number would be (300 / (768-1) -0.0) / (400 / 768-0.0), which would be 0.750977835. Next, the gradation calculator 3325 calculates the gradation value. Since the normalized line number is less than 0.8, the gradient value is (0.1 * 0.750977835 + (0.8-0.750977835) * 0.8) /0.8, which is 0.142894394.
0301Process 3600 (at 3625) then blends the mixed image with the original image using the calculated gradient values. Process 3600 calculates the RGB triplet for each pixel of the blended image. For example, the gradient image generator 3515 calculates the RGB triplet for the pixels in the blended image. The corresponding pixel in the original image is the pixel that is the 300th pixel from the top. Therefore, the RGB triplet for the pixels in the blended image is (0.3, 0.4, 0.5) * (1-0.142894394) + (0.0768, 0.128, 0.112) * 0.142894394, which is (0.268105971, 0.361132724, 0.444556975). Become.
0302In this subsection, some image gradient effect tools have been described. The following subsections describe some image tilt shift tools.
0303B. Tilt shift tool on the image FIG. 37 conceptually illustrates the use of some embodiments of an image effect tool that allows a user to apply a tilt shift effect to a selected portion of an image by touching the image. Specifically, this figure shows that the user can apply the tilt shift effect to the image 3725 by touching and manipulating the tilt shift tool 3715, which is a tool on the visible image, in six different stages. It is shown in 3701 to 3706. FIG. 37 shows GUI400, icons 3110, and 3130-2950.
0304As described above, the image display area 410 displays an image that the user wants to edit using the image editing application. In some embodiments, the image editing application allows the user to apply a tilt shift effect to the image by touching the displayed image and manipulating the tilt shift tool 3715. The image editing application superimposes the tilt shift tool 3715 on the image displayed in the image display area 410. The tilt shift tool 3715 is similar to the tilt shift tool 211 described above with reference to FIG.
0305From this, an operation example of the tilt shift tool on the image will be described. The first stage 3701 shows the user's selection of thumbnail 3710 by touching thumbnail 3720 with a finger 3725. The image editing application responds to the selection of thumbnail 3710 by displaying the image 3710 represented by thumbnail 3720 in the image display area 410.
0306At step 3702, the user selects the icon 3110 displayed in the tool selection pane 420. In response, the image editing application displays a set of selectable UI items 3125 within the GUI 400 at stage 3703, as illustrated. The user selects the tilt shift tool on the image by touching UI item 3150. The image editing application launches the tilt shift tool on the image.
0307The next step, 3704, shows the GUI 400 after the user touches the image area just below the vertical center of the image. At this stage, the tilt shift tool 3715 placed two parallel lines on the image so that the center of the touched area was equidistant from the two parallel lines. The tilt shift tool 3715 also blurred the portion of the image that was vertically above the upper parallel line and the portion of the image that was vertically below the lower parallel line. In some embodiments, the tilt shift tool 3715 may place two parallel lines on the default location of the image with the default width.
0308At step 3705, the user performed a magnified gesture on the image using fingers 3730 and 3735 to widen and further separate the two parallel lines. Both the upper and lower lines have moved. The upper line moved up and the lower line moved down. As a result, the portion of the image between the two parallel lines is larger, and the tilt shift tool does not blur this portion of the image.
0309The next step, 3706, shows the GUI 400 after the user lifts fingers 3730 and 3735 from image 3730. The two parallel lines remain visible to indicate that these two lines are operational. When the user exits the edit mode of the image editing application, the two parallel lines disappear from the image 3710.
0310FIG. 38 conceptually illustrates the architecture of an on-image tiltshift tool for some embodiments that apply a tiltshift effect to a portion of the image selected by the user. Specifically, the figure shows that the tilt effect tool on an image applies a tilt shift effect using some pre-generated images that are blurred based on the original selected image. As illustrated, FIG. 38 shows the tilt shift tool 3800 on the image, the original image data 3830, the blurred image 3835, and the tilt shift image 3840.
0311The original image data 3830 is image data for the selected image before the tilt shift effect is applied. The selected image is an image displayed in the image display area of the image editing application. The tilt shift image 3840 is image data for an image created by applying tilt shift to the image. In some embodiments, the tiltshift image 3335 is a set of instructions to execute to generate a new image from the original image.
0312The on-image tilt shift tool 3800 includes a blur image generator 3815, a distance calculator 3805, a weight calculator 3810, and an image adder 3825. The on-image tiltshift tool in some embodiments vertically tilt-shifts the image-blurring (in other words, the location of the user's touch), except for the horizontal bands of the image that are defined based on the vertical location of the user's touch. To blur parts of the image that are not within the specified vertical distance). The Tilt Shift Tool 3800 on the Image has been modified to allow the Tilt Shift Tool 3800 to radially tilt shift parts of the image that are not within a predetermined distance from the location of the user's touch-to allow blurring. You may. The on-image tilt-shift tool 3800 in some embodiments produces a tilt-shift image by combining the original image with a blurred version of the original image.
0313The blur image generator 3815 generates some blur images based on the original image. In some embodiments, the blurred image generator 3815 produces a blurred image by convolving the original image with a Gaussian kernel. The standard deviation of the Gaussian kernel is relative to the size of the original image to be processed. Also, the standard deviation increases in proportion to the size of the original image. That is, the larger the scale of the original image, the larger the standard deviation of the Gaussian kernel. For example, if the blur image generator 3815 produces 5 blur images, for the original image with a scale of 2000 pixels, the standard deviations for the 5 Gaussian kernels are 1.6, 4, 8, 12 and 16 It is set as the size of the pixel. When the Tilt Shift Tool 3800 on Image is launched by an image editing application, the Blur Image Generator 3815 pre-generates the Blur image and stores the Blur image (eg in the cache), thereby on the original image. Upon receiving the user's touch, these blurred images can be quickly combined to generate a tilt-shift image.
0314The distance calculator 3805 receives user input from a user interface module (not shown) that receives touch input. For each pixel in the original image, the distance calculator 3805 calculates the vertical distance (eg, the difference in y-axis values) between the pixel and the pixel touched by the user. In some embodiments, the distance calculator 3805 uses this pseudo-code example to normalize the calculated vertical distance: denom = 1.0 / max (heightTouched, imageHeight-heightTouched); dist = 1.1-| (currentRow-heightTouched) | * denom * 1.25; dist = max (min (dist, 1.0), 0.0); heightTouched is the y-coordinate of the pixel touched by the user (eg, the vertical center pixel (s) of the set of touched pixels). In some embodiments, the y coordinate starts at the top row of the image. That is, the top row of pixels in the image has a y-coordinate of 0, and the bottom row of pixels in the image is the y-coordinate of the total height (eg, the number of pixels in a column of pixels in the image-1). ). imageHeight is the total height of the original image. currentRow is the y coordinate of the row to which the particular pixel currently being processed belongs. dist is the normalized distance for a particular pixel. Distances are normalized to have a range of 0 to 1.1.
0315The calculated distance is depicted as curve 3905 in Graph 3900 shown in Figure 39. The pixel touched by the user is about the 800th pixel from the top row. Graph 3900 is for an original image with a height of 2500 pixels. The vertical axis of graph 3900 represents the normalized distances for curves 3905 and the normalized weights for curves 3910, 3915, 3920, 3925, 3930, and 3935. The horizontal axis represents the top row, 0, to the bottom row, 2500. Curve 3905 looks like a pyramid with a flat top. The fourth line of pseudocode shown above sets all normalized distance values above 1.0 to 1.0, so the curve 3905 has a flat peak.
0316The weight calculator 3810 calculates the weights for each of the blurred images generated by the blurred image generator 3815, and the weights for the original image. The weight calculator in some embodiments derives the weight for each blurred image from the distance from the vertical location of the pixel touched by the user. The weight calculator 3810 is a Gaussian function whose input is a Gaussian function, which is a normalized distance, to calculate the weights for each blurred image. The Gaussian function used by the weight calculator 3810 is as follows.
0317<maths num="1"><img id="000002" he="15" wi="41" file="JP5986232B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
0318Parameter a is set to 1 in some embodiments, so the function is regulated by two parameters b and c. As is well known in the art, b is the symmetrical "bell curve" or the position of the center of the Gaussian kernel, and c adjusts the width of the "bell". The center of the Gaussian kernel 3910, which adjusts the weights for the original image, is set as the vertical pixel location of the pixel touched by the user. The centers of the other Gaussian kernels 3910-3935 are sequentially set so that one Gaussian kernel and the next Gaussian kernel intersect at or near half of them. The width of the Gaussian kernel is adjusted by the user's pinch gesture. The greater the distance between the fingers (ie, the distance between the two parallel lines of the tilt shift tool on the image), the wider the Gaussian kernel. All Gaussian kernels have the same width. The weight calculator 3810 calculates the width of the Gaussian kernel using the following pseudo code. width = 0.12 * FX height / 0.32; If (width <0.06) width = 0.06; FXheight is the normalized distance between the two parallel lines of the tilt shift tool on the image. width is the calculated width. The weight calculator 3810 normalizes the weights to ensure that the weights add up to 1.0.
0319The image adder 3825 produces a tilt-shifted image by adding the weighted original image and the weighted blurred image. That is, the image adder 3825 adds the weighted pixel value of the blurred image generated by the blurred image generator 3835 to the weighted pixel value of each pixel of the original image.
0320From this, an operation example of the tilt shift tool 3800 on the image will be described with reference to FIGS. 38 and 40. FIG. 40 conceptually illustrates the process 4000 that some embodiments perform to apply a tilt shift effect to an image. Process 4000 in some embodiments is performed by an on-image tilt shift tool such as the online shift tool 3800. Process 4000 begins when the tilt shift tool on the image is launched by the image editing application and receives image data for the displayed image.
0321Process 4000 begins by pre-generating some blurred images (in 4005) based on the original image. For example, the blur image generator 3815 generates five blurred images. The blur image generator 3815 generates a first blur image using an average of size equal to 1.6 pixels. That is, the blurred image generator 3815 in some embodiments averages the pixel values of all pixels within each 1.6 × 1.6 pixel square of the original image. The blur image generator 3815 generates a second blur image using an average of equal size of 4 pixels. That is, the blurred image generator 3815 in some embodiments averages the pixel values of all pixels within each 4 × 4 pixel square of the original image. In a similar manner, the Blur Image Generator 3815 produces third, fourth, and fifth blurred images using averaging equal to 8, 12, and 16 pixels in size, respectively.
0322Process 4000 then receives a selection of central pixels (at 4010). For example, the distance calculator 3805 receives the location of the pixel touched by the user. In this example, the pixel touched by the user is the 800th from the top of the original image, which has 2500 pixels vertically.
0323Process 4000 then (at 4015) calculates the distance from the touched pixel for each row of pixels. In some embodiments, Process 4000 also normalizes the calculated distance. For example, the distance calculator 3805 calculates the distance between the 500th row from the top of the image and the touched pixel. The distance is 1.1-(| 499-799 | * (1.0 / max (799,2499-799)) * 1.25), which is 0.879411765.
0324Process 4000 then (at 4020) calculates the weights for each of the generated blurred images and for the original image. In some embodiments, Process 4000 also normalizes the calculated weights so that all weights total 1.0. For example, the weight calculator 3810 calculates the same width for all Gaussian kernels for all blurred images. In this example, the distance between the two parallel lines of the tilt shift tool on the image is 200 pixels. This distance, when normalized, is 1.1-(| 200-799 | * (1.0 / maximum (799,2499-799)) * 1.25), which is 0.658823529. Therefore, the width is 0.12 * 0.658823529 / 0.32, which is 0.247058823. The width is not less than 0.06, so the width is 0.247058823. Then the weight for the original image is e<sup>((1.0-1.0)^2/2*(0.247058823)^2)</sup>And this becomes 1. After the weight calculator 3810 calculates the weights for each blurred image, the weight calculator 3810 normalizes the calculated weights for the original image.
0325Process 4000 then (at 4025) adds the weighted original image and the weighted blurred image. That is, process 4000 adds the weighted pixel values of the pixels in the original image and the weighted pixel values of the corresponding pixels in the blurred image.
0326In this subsection, some embodiments of the on-image tilt shift tool have been described. The following subsection describes the vignette tool on the image.
0327C. Vignette tool on the image FIG. 41 conceptually illustrates the use of some embodiments of an image effect tool that allows a user to apply a vignette effect to a selected portion of an image by touching the image. Specifically, this figure shows six different stages in which the user applies a vignette effect to image 3710 by touching and manipulating an invisible (eg, borderless) geometry. It is shown in 4101 to 4106. FIG. 41 shows GUI400, thumbnails 3720, images 3710, and icons 3110 and 3130-2950.
0328As described above, the image display area 410 displays an image that the user wants to edit using the image editing application. In some embodiments, the image editing application allows the user to manipulate the application of vignette effects to the image by moving and resizing the geometry. When the user touches the location of the image, the image editing application applies a vignette effect based on the touched location of the image. An image editing application in some embodiments draws an invisible geometry (ie, a shape without a visible border) that surrounds the touched location of the image, and the image is outside the geometry. Darken the area. As a result, the area of the image inside the geometry looks brighter than the area of the image outside the geometry. Therefore, the geometry is visible because of the difference in brightness between the outer area of the geometry and the inner area of the geometry. However, the geometry itself is still invisible in the sense that the shape does not have its own boundaries.
0329Image editing applications of some embodiments allow the user to move and resize geometry (ie, brighter areas). The user can move the geometry by touching it and drag the geometry to another location in the displayed image. The user can enlarge the geometric shape by touching the inside of the geometric shape with a plurality of fingers and spreading and releasing the fingers. The user can shrink the geometry by performing a pinch gesture on top of the geometry (eg, pulling the fingers touching the geometry closer together).
0330In some embodiments, the geometry may be any other shape. The non-comprehensive list includes circles, triangles, rectangles, etc. The shape may be an irregular shape or a familiar object shape such as a hand or a flower.
0331From this, an operation example of an image editing application having GUI4100 will be described. The first two stages 4101 and Q02 are identical to stages 3701 and 3702, respectively. In response to the user's selection of thumbnail 3110 in the previous stage 3702, the image editing application displays a set of selectable UI items 3125 within the GUI 400 at stage 4103, as illustrated. The user selects the vignette tool on the image by touching UI item 3145. The image editing application launches a vignette tool on the image.
0332At step 4104, the user touches the location of the image near the face of the person on the left in the image with finger 4110. The Vignette tool on the image draws a borderless ellipse 4120 that surrounds the touched location of the image, darkening the area outside the ellipse. As a result, the face of the person on the left appears to be in the spotlight.
0333At step 4105, the user drags finger 4110 to another location in the image, the face of the person on the right. The vignette tool on the image causes the ellipse to follow (or move with) the finger 4110 to that location. Alternatively, the user may lift the finger 4110 from the image and touch the location near the top of the image 3710 with the finger 4110, or the user is dragging the finger 4110 to the location while touching the image 3710. You may. The vignette tool on the image applied the vignette effect by darkening the area outside the ellipse, as shown.
0334Then in step 4106, the user performed a magnifying action (eg, spreading his finger while touching the image) on the ellipse 4120. The Vignette tool on the image resizes the ellipse and applies a vignette effect based on the resized ellipse size.
0335FIG. 42 conceptually illustrates the process 4200 performed by some embodiments to allow a user of an image editing application to apply a vignette effect to an image. Process 4200 in some embodiments is performed by an image editing application. Process 4200 begins when the image editing application runs on a computing device that has a touch-sensitive or near-touch sensitive screen to receive touch input from the user (eg, tap, swipe, etc.). Process 4200 begins by displaying an image within the image display area of the image editing application (in 4205).
0336Process 4200 then (at 4210) receives a touch input on the location in the displayed image. In some embodiments, process 4200 receives a touch input on the location when the user taps on the location. Process 4200 receives a touch input when the user touches the displayed image (eg, without raising his finger) and drags his finger from another location in the displayed image.
0337Process 4200 then applies a vignette effect to the image (at 4215). When the user taps a location in the displayed image, Process 4200 draws an invisible geometry and darkens the area outside the geometry.
0338FIG. 43 conceptually illustrates the process 4300 performed by some embodiments to allow a user of an image editing application to apply a vignette to an image. Process 4300 in some embodiments is performed by a vignette tool on the image. Process 4300 begins after the vignette tool on the image receives the first touch input and applies a vignette effect based on the location of the first touch input.
0339Process 4300 begins by determining if the vignette tool on the image has received a new touch input. If process 4300 determines that the vignette tool did not receive the new touch input on the image, process 4300 loops back to 4320 to wait for the new touch input.
0340If process 4300 determines that the vignette tool on the image has received a new touch input, process 4300 proceeds to 4325 to determine if the touch input is for a geometry. In some embodiments, if the touch input touches an area inside or near the geometry, process 4300 determines that the touch input is for the geometry. If process 4300 determines that the touch input is not for the geometry, process 4300 proceeds to 4360 as described further below.
0341If process 4300 determines that the touch input is for a geometry (4125), the process determines if the touch input is a multi-touch input (4130). If process 4300 determines that the touch input is not a multi-touch input, process 4300 proceeds to 4355, which is further described below. If process 4300 (at 4330) determines that the touch input is a multi-touch input, process 4300 proceeds to 4335 to determine if the multi-touch input was a pinch action.
0342If process 4300 (in 4335) determines that the multi-touch input was a pinch action, process 4300 shrinks the geometry (in 4340) and reduces the size of the geometry. If process 4300 (in 4335) determines that the multi-touch input was not a pinch action, process 4300 (in 4345) determines if the multi-touch input was an expansion action (or expansion gesture). .. If process 4300 (at 4345) determines that the multi-touch input was not a magnifying action, the process terminates. Otherwise, Process 4300 expands the geometry.
0343If process 4300 (at 4330) determines that the touch input was not a multi-touch input, process 4300 determines if the touch input was dragged (4155). If process 4300 (at 4355) determines that the touch was not dragged, process 4300 terminates. Otherwise, Process 4300 applies the effect based on the location of the touch input and the default geometry.
0344FIG. 44 conceptually illustrates the process 4400 that some embodiments perform to apply a vignette effect based on the location of the user's touch within the displayed image. Process 4400 in some embodiments is performed by a vignette tool on the image. Process 4400 begins when the vignette tool on the image receives the location of the user's touch.
0345Process 4400 begins by receiving a scale value (at 4405). In some embodiments, the vignette effect is the attenuation of a "sigmoid" shape based on a single "scale" value. The scale value corresponds to the size of the geometry (eg, the radius of the circle) that the user adjusts by performing a pinch gesture or a magnifying gesture when the vignette tool is activated on the image. In some embodiments, the scale value is normalized to have a range of 0 to 1.
0346Process 4400 then calculates the inner and outer diameters of the circular shape (at 4410). In some embodiments, Process 4400 uses the following pseudo-code example to calculate the radius. innerRadius = scale * (1-scale) * 0.8; outerRadius = scale + 0.5; innerRadius is a circular inner radius. outerRadius is a circular outer diameter. scale is the received scale value.
0347Process 4400 then (in 4415) calculates the distance between the pixels touched by the user and each pixel in the original image. In some embodiments, the distance is calculated using the Euclidean distance metric below. dist = ((x-centerX) 2+ (y-centerY)<sup>2</sup>)<sup>(1/2)</sup>; dist is the calculated distance. x is the x-coordinate of the pixel, y is the y-coordinate of the pixel, and the lower left corner of the image is the origin, (0, 0). In some embodiments, the process uses a squared distance.
0348Process 4400 (at 4420) then determines if the calculated distance is less than the calculated inner diameter of the circular shape. If the calculated distance is less than the calculated inner diameter, process 4400 sets the vignette value for each pixel of the image (at 4425) to the first default value. In some embodiments, the first default value is 1.0. The process then proceeds to 4455, which will be further described below.
0349If process 4400 (in 4420) determines that the calculated distance is not less than the calculated inner diameter of the circle, then process 4400 (in 4430) determines that the calculated distance is a circle calculation. Determine if it is larger than the outer diameter. If the calculated distance is greater than the calculated outer diameter, process 4400 sets the vignette value for each pixel of the image (at 4435) to the second default value. In some embodiments, the second default value is 0.05. The process then proceeds to 4455, which will be further described below.
0350If process 4400 (at 4430) determines that the calculated distance is not greater than the calculated outer diameter, process 4400 (at 4440) calculates the range of sigmoid shapes. The process (at 4420 and 4430) determined that the calculated distance was not less than the calculated inner diameter and not greater than the calculated outer diameter, so the distance between the pixels of the image being processed was It is between the two radii. Process 4400 in some embodiments uses the following pseudo-code example to calculate the range: range = 3.0 + 2.0 * scale;
0351Process 4400 also (in 4445) computes some normalization components using the following pseudo-code example. rangeMin = 1 / (1 + e<sup>range</sup>); rangeNorm = 1 + e<sup>-range</sup>-rangeMin; rangeMin is the normalized minimum range. rangeNorm is the normalized range.
0352Process 4400 (in 4450) is then an image whose distance from the touched pixel calculates a vignette value for each pixel of the image, which is between the calculated radii. Process 4400 calculates a vignette value for such a pixel using the following pseudo-code example. tmp = 1 / (1 + e<sup>tmp</sup>); vignette = 1-(tmp-rangeMin) * rangeNorm * 0.95; e is the Euler number. rangeMin and rangeNorm work to prevent vignette values above 1.0 or below 0.05.
0353Process 4400 then applies a vignette to the image. In some embodiments, the process applies a vignette to an image by simply multiplying each pixel value of the image by a vignette value for the pixel. However, those skilled in the art will recognize that methods other than multiplication may be used to apply vignette values to the pixels of the image.
0354IV. Alternative user interface layout The GUI of the image editing application shown in the above figure is shown on the assumption that the device on which the image editing application operates has a screen large enough to display the GUI. However, some devices running image editing applications may have a constrained screen size and cannot display UI items so that the items are displayed within the larger screen of a larger device. In some cases. Also, the larger screen of a larger device may be considered constrained when the device is held in a different orientation (eg, portrait orientation). In some embodiments, the image editing application displays different sets of different UI items in different time instances to accommodate the constrained screen size.
035545A and 45B conceptually show GUI 4500 in some embodiments. Specifically, the figure shows that image editing applications of some embodiments display different sets of different UI items in different time instances at 10 different stages 4501-4510. 45A, 45B show that GUI4500 includes a control pane 4515, an image display area 4516, a collection display area 4520, and a tool navigation pane 4525.
0356The image display area 4516 is similar to the image display area 410 described above in that the image display area 4516 displays an image that the user wants to edit using the image editing application. The collection display area 4515 is similar to the collection display area 415 in that the collection display area 4515 displays a collection of thumbnails of images that can be used for editing using an image editing application. In some embodiments, the image editing application allows the user to scroll up and down or sideways to display different thumbnails. The image editing application also provides a visual display (eg, highlighting) on the thumbnail to indicate that the thumbnail is selected and the corresponding image is displayed within the display area. In addition, some image editing applications of embodiments provide visual instructions (eg, toolbox icons) on thumbnails to indicate that the corresponding image has been edited.
0357The Tool Navigation Pane 4525 displays different sets of selectable or adjustable UI items in different time instances. In some embodiments, the set of user interface items currently displayed in the Tool Navigation Pane 4525 relies on the user's interaction with the set of UI items previously displayed in the Tool Navigation Pane 4525. For example, when a user touches a particular UI item that represents a set of editing tools, the set of UI items for the set of editing tools associated with that particular UI item is displayed in the Tool Navigation Pane 4525. Replaces the set of UI items that contain.
0358The image editing application may replace the first set of UI items displayed in the tool navigation pane 4525 with the second set of UI items in a number of different ways. For example, an image editing application overlays a second set on top of the first set as the second set slides in. Image editing applications may also slide the first set out while sliding the second set in. Further, the direction of the slide by the first and second sets may be any direction-up, down, left, right, diagonal, etc.
0359Here, an operation example of GUI4500 will be described. At stage 4501, the image editing application displays image 4517 within the image display area 4516. Image 4517 corresponds to thumbnail 4535 displayed in the collection display area 4520. That is, the image editing application displays the image 4517 in the image display area 4516 in response to the user's selection (eg, by touching) thumbnail 4535. At this stage, the user selects UI item 4530, which is an icon indicating the toolbox in this example. UI item 4530 represents a set of editing tools provided by the image editing application.
0360The next step 4502 shows that the first set of UI items 4524 displayed in the tool navigation pane 4525 will be replaced by the second set of UI items 4526. In this example, in stage 4502, the second set of UI items 4526 slides to the right and enters the tool navigation pane 4525, with the second set of UI items superimposed on the first set of UI items. Show that it is going. However, in some embodiments, as shown, the second set of UI items 4526 slides under the UI item 4530 and covers the rest of the first set of UI items. The UI item 4530 in the first set of UI items stays in place while it is being covered.
0361At stage 4503, the image editing application displays a second set of UI items 4526 within the tool navigation pane 4525. As illustrated, the second set of UI items 4526 in this example includes five UI items, including UI items 4545, 4550, and 4555. UI item 4540, which does not belong to the second set of UI items, has replaced UI item 4530. UI item 4540 is for replacing the second set of UI items with another set of UI items in the tool navigation pane 4525. UI item 4540 is also for the user to touch to indicate that the user does not want to use the editing tools represented by the second set of UI items. UI item 4545 represents a cropping and angle correction tool on the image. UI item 4550 represents the brush tool on the image. UI item 4555 represents an effect tool on the image. At this stage, the user touches UI item 4545 to select a crop and angle correction tool on the image.
0362At stage 4504, the image editing application is revealing a third set of UI items in response to the user's selection of UI item 4545 in previous stage 4503. In some embodiments, the image editing application slides the second set of UI items out to reveal the third set of UI items. The third set of UI items, which is not fully shown in stage 4504, represents a crop and angle correction tool. In some embodiments, the image editing application slides the second set of UI items to the left, slipping under UI item 4540 and out, as shown.
0363The next step 4505 shows the GUI 4500 after the image editing application has replaced the second set of UI items and the UI item 4540 with the third set of UI items. The third set of UI items represents the crop and angle correction tools and includes UI items 4560 and 4565. UI item 4550 is currently shown in the tool navigation pane 4525, indicating that it is the crop and angle correction tool that is activated. UI item 4560 is also for replacing a third set of UI items with another set of UI items in the tool navigation pane 4525. UI item 4560 is also for the user to touch to indicate that the user does not want to use the crop and angle correction tool or has finished using it. UI item 4565 is similar to dial 455 above. Also, at stage 4505, the user performed a magnifying gesture on the displayed image 4517 to crop the image.
0364The next stage, 4506, shows the GUI 4500 after the crop and angle correction tool magnifies the image 4517 and cuts off the portion of the image that could not be displayed within the image display area 4516. The user also touches UI item 4560 to exit the crop and angle correction tools.
0365At stage 4507, the image editing application is sliding in a second set of UI items in response to the user's selection of UI item 4560 in the previous stage 4506. As illustrated, in some embodiments, the image editing application covers a second set of UI items 4526 over UI item 4565, but to the left so that it slips under UI item 4560. Slide it in.
0366The next stage, 4508, shows the GUI 4500 after the image editing application has replaced the third set of UI items with the second set of UI items. UI item 4560 in the third set of UI items has been replaced by UI item 4540, as shown. The image editing application also overlays visual instructions 4560 on thumbnail 4535. Visual instruction 4560 indicates that the image represented by thumbnail 4535 has been edited. At stage 4508, the user selects UI item 4540 to exit the second set of UI items 4526.
0367At stage 4509, the image editing application slides the second set of UI items to the left to reveal the first set of UI items in response to the user's selection of UI items 4540 in the previous stage 4506. , I'm sneaking under UI item 4540 and going out. The next stage, 4510, shows the GUI 4500 after the image editing application has replaced the second set of UI items with the first set of UI items. As shown, UI item 4530 has replaced UI item 4540.
0368V. Software architecture In some embodiments, the processes described above with reference to FIGS. 5, 8, 11, 15, 17, 21, 25, 28, 30, 32, 34, 36, 40, 42, 43, and 44 , Implemented as software running on a particular device such as a computer or handheld device, or stored on a computer-readable medium. FIG. 46 conceptually illustrates the software architecture of the image editing application 4600 in some embodiments. In some embodiments, the image editing application may be a stand-alone application or integrated into another application, and in other embodiments, the application may be implemented within an operating system. Moreover, in some embodiments, the application is provided as part of a server-based solution. In some such embodiments, the application is provided by a thin client. That is, the application runs on the server and the user interacts with the application via a separate machine remote from the server. In other such embodiments, the application is provided by a thick client. That is, the application is distributed from the server to the client machine and operates on the client machine.
0369Image Editing Application 4600 includes User Interface (UI) Dialogue Module 4605, Crop and Angle Correction Tool 4610, Brush Tool Group 4615, Effect Tool Group 4620, Tilt Shift Tool 4625, Gradient Tool 4630, Vignette Tool 4635, and Launch Manager 4670. Including. The image editing application also uses the source file 4640 and the editing instruction 4645. In some embodiments, the source file 4640 stores media content (eg, text, audio, image, and video content). The editing command 4645 stores the image editing operation executed by the image editing application 4600 as a set of commands. The image editing application 4600 uses a set of these instructions to generate a new image based on the original data stored in the source file. In some embodiments, the media content data is stored in the source file 4640 as files such as .mov, .avi, .jpg, .png, gif, pdf, .mp3, .wav, .txt. In some embodiments, the memories 4640 and 4645 are all stored in one physical memory. In other embodiments, the memories are in separate physical memories, or one of the memories is in one physical memory, while the other is in a different physical memory. For example, other project data and source files will often be separate.
0370FIG. 46 also shows an operating system 4650 including an input device driver (s) 4660 and a display module 4665. In some embodiments, as illustrated, the device drivers 4655 and 4660 and the display module 4665 are part of the operating system 4655, even when the image editing application is an application separate from the operating system.
0371The input device driver 4660 may include drivers for converting signals from keyboards, mice, touchpads, tablets, touch screens, gyroscopes, accelerometers, and the like. The user interacts with one or more of these input devices, which send signals to their corresponding device drivers. The device driver then converts the signal into user input data provided in the UI dialogue module 4605.
0372The present application describes a graphical user interface that provides a user with a number of ways to perform different sets of actions and functions. In some embodiments, these actions and functions are performed based on various commands received from the user via various input devices (eg, keyboard, trackpad, touchpad, mouse, etc.). For example, the present application describes using a cursor in a graphical user interface to control (eg, select and move) an object in the graphical user interface. However, in some embodiments, objects in the graphical user interface can also be controlled or manipulated by other controls, such as touch controls. In some embodiments, touch control is implemented by an input device capable of detecting the presence and location of touch on the display of the device. An example of such a device is a touch screen device. In some embodiments, touch control allows the user to interact directly with the object by interacting with a graphical user interface superimposed on the display of the touch screen device. For example, a user can select a particular object in a graphical user interface by simply touching the particular object on the display of the touch screen device. Therefore, when touch control is utilized, in some embodiments, a cursor may not be provided to allow selection of objects in the graphical user interface. However, when a cursor is provided within a graphical user interface, in some embodiments touch control can be used to control the cursor.
0373In addition, the present application, in some embodiments, edits various commands (eg, trimming, extension of edit points, division, marking of areas of interest, etc.) to edit a media clip or series of media clips. Explain the use of hotkeys to call a command). In some embodiments, the hotkeys are tangible keys within a typical keyboard (eg, a keyboard for a desktop computer, a keypad for a smartphone, etc.). However, in other embodiments, hotkeys are provided through touch control. Hotkeys in some such embodiments are keys in a graphical keyboard rendered on a touch screen device, any user interface item that is selectable or clickable, or any other touch screen technique. ..
0374The display module 4665 converts the output of the user interface for the display device. That is, the display module 4665 receives signals indicating what should be displayed (for example, from the UI dialogue module 4605) and converts those signals into pixel information sent to the display device. The display device may be an LCD, a plasma screen, a CRT monitor, a touch screen, or the like.
0375The UI interaction module 4605 of the image editing application 4600 interprets the user input data received from the input device driver and interprets it with the crop and angle correction tool 4610, brush tool group 4615, effect tool group 4620, tilt shift tool 4625, gradient. Pass to various modules, including Tool 4630 and Vignette Tool 4635. The UI dialogue module also manages the display of the UI and outputs this display information to the display module 4665.
0376Launch Manager 4670 manages the launch and shutdown of editing tools. Launch Manager 4670 interprets the user input to the UI item to start and stop the editing tool and launches the editing tool so that the user input to the image is processed by the appropriate editing tool (s). And stop.
0377The crop and angle correction tool 4610 is similar to the crop and angle correction tools described above in that the tool 4610 performs cropping, angle correction, and enlargement operations. The brush tool group 4615 includes the above-mentioned defect removal tool, red-eye removal tool, saturation tool, desaturation tool, lightening tool, darkening tool, sharpening tool, softening tool, and smart edge detection tool. The effect tools group 4620 includes a group of tools for applying the effect to the displayed image. The tilt shift tool 4625 is similar to the tilt shift tool described above. The gradient tool group 4630 in some embodiments includes the dark gradient tool, the warm gradient tool, the cool gradient tool, the blue gradient tool, and the coffee gradient tool described above. The vignette tool 4635 is similar to the vignette tool described above.
0378Many of the features are described as being performed by one module (eg, gradient tools 4630), but those skilled in the art can divide the function into multiple modules, and one. You will recognize that performing features may also require multiple modules.
0379VI. Image viewing, editing, and organizing applications The various figures described above show various examples of GUIs for image viewing, editing, and organizing applications of some embodiments. FIG. 47 shows a detailed view of the GUI 4700 for some embodiments of viewing, editing, and organizing images. The GUI 4700 will be described with some reference to FIG. 48, which conceptually illustrates the data structure 4800 for images stored by the applications of some embodiments.
0380Data structure 4800 includes image ID 4805, image data 4810, edit instructions 4815, cached version 4840 of the image, and any additional data 4850 of the image. Image ID 4805 is, in some embodiments, a unique identifier for the image used by the collection data structure to refer to the image stored within the collection. Image data 4810 is the actual full-size pixel data that displays the image (eg, a set of color space channel values for each pixel in the image or its encoded version). In some embodiments, this data can be stored in a database of image viewing, editing, and organizing applications, or can be stored with data from another application on the same device. In some embodiments, the additional application is another image organization application that operates on a device that operates image viewing, editing, and organization.
0381Therefore, the data structure can store a pointer to a local file associated with the application, or an ID that can be used to query the database of another application. In some embodiments, when the application uses the image in the journal or edits the image, the application automatically makes a local copy of the image file containing the image data.
0382Edit instruction 4815 contains information about any edits the user has made to the image. In this way, the application stores the image in a non-destructive format so that the application can easily revert from the edited version of the image to the original at any time. For example, the user can apply a saturation effect to an image, leave the application, and then restart the application at another time to remove the effect. The edits stored in these commands can be crop and rotate, full image exposure and color adjustments, localized adjustments and special effects, and other edits that affect the pixels of the image. Some embodiments store these edit instructions in a particular order so that the user can view different versions of the image with only the application of a particular set of edits.
0383In some embodiments, the edit instruction 4815 is executed as list 4860 of the editing work. Listing 4860 includes editing tasks such as editing 4861, 4862, 4863, and 4865. For each edit task in Listing 4860, specify the parameters required to perform the edit task. For example, editing task 4865 in Listing 4860 specifies an edit to the image to which the saturation effect is applied using the color selection parameter θ.
0384In some embodiments, Listing 4860 records a sequence of editing work initiated by the user to create the final edited image. In some embodiments, Listing 4860 is edited by the image editing application to produce an output image for display, as some embodiments specify a particular order for the different possible edits made by the application. Stores edit commands in the order in which they are applied to the image. For example, some embodiments define the saturation effect as one of the editing tasks that will be applied after other editing tasks such as cropping and rotation, full image exposure, and color adjustment. Listing 4860 of some of these embodiments shows where the editing instructions for the saturation effect will be applied after some of the other editing tasks (eg, edits 4861 to 1363) (ie, edits 4861 to 1363). , Edit 4865) will remember.
0385The cached image version 4840 stores the versions of the images that are commonly accessed and displayed so that the application does not have to iteratively generate these images from the full size image data 4810. For example, an application will often store thumbnails of images as well as display resolution versions (eg, versions adjusted for the image display area). Some embodiments of the application generate a new thumbnail of the image and replace the previous thumbnail each time an edit is applied. Some embodiments store multiple display resolution versions, including the original image and one or more edited versions of the image.
0386Finally, the image data structure 4800 includes additional data 4850 (eg, location and face size, etc.) that the application may store with the image. In some embodiments, additional data may be in Exchangeable image file format (Exif) data, heading data, shared image data, tags on images, or any other format. Data can be mentioned. Exif data includes various information stored by the camera that captured the image, such as camera settings, GPS data, time stamps, and so on. The heading is a description of the image entered by the user. Tags are information that allows an application to associate with an image, such as marking an image as favorite, flagged, hidden, and so on.
0387Those skilled in the art will recognize that the image data structure 4800 is only one possible data structure that an application may use to store the required information in an image. For example, different embodiments may store more or less information, store information in a different order, and so on.
0388Returning to FIG. 47, the GUI 4700 includes a thumbnail display area 4705, an image display area 4710, a first toolbar 4715, a second toolbar 4720, and a third toolbar 4725. The thumbnail display area 4705 displays thumbnails of the images in the selected collection. Thumbnails are a small representation of a full-size image and, in some embodiments, represent only part of the image. For example, the thumbnails in the thumbnail display area 4705 are all square, regardless of the aspect ratio of the full size image. To determine which portion of the rectangular image should be used for the thumbnail, the application identifies the smaller dimension of the image and uses the center of the image in the longer direction. For example, for a 1600 x 1200 pixel image, the application would use a 4700 x 1200 square. To further refine the selected part for thumbnails, some embodiments identify all face centers in the image (using a face detection algorithm) and then utilize this location. Place the center of the thumbnail part in the direction of cutting. So if all the faces in a theoretical 1600x1200 image are on the left side of the image, the application would use the leftmost 4700 columns of pixels instead of cutting off 200 columns on each side.
0389After determining which part of the image should be used for the thumbnail, the image viewing application produces a low resolution version of the image (eg, using pixel mixing and other techniques). Some embodiments of the application store thumbnails of images as cached version 4840 of the image. Therefore, when the user selects a collection, the application identifies all of the images in the collection (via the collection data structure) and is cached in each image data structure for display in the thumbnail view area. Access thumbnails.
0390The user can select one or more images in the thumbnail display area (eg, via the various touch interactions described above, or via other user input interactions). The selected thumbnail is displayed with a highlight of the selection or other indicator. In the thumbnail display area 4705, thumbnail 4730 is selected. Further, as illustrated, the thumbnail display area 4705 of some embodiments displays some images flagged in the collection (eg, having a flag tag set to Yes). In some embodiments, this text is selectable to display only thumbnails of flagged images.
0391The application displays the selected image in the image display area 4710 at a resolution greater than the corresponding thumbnail. Images are usually not displayed in full size because they have a higher resolution than display devices. With this in mind, some embodiments of the application store a cached version 4840 of an image designed to fit in the image display area. The image in the image display area 4710 is displayed with the aspect ratio of the full size image. When one image is selected, the application displays the largest image possible within the image display area without cropping any part of the image. When multiple images are selected, the application displays the images so as to maintain visual weighting by using approximately the same number of pixels for each image, even when the images have different aspect ratios.
0392The first toolbar 4715 displays title information (for example, the name of the collection shown in the GUI, the heading added by the user to the currently selected image, etc.). Further, the toolbar 4715 includes a first set of GUI items 4735 to 1238 and a second set of GUI items 4740 to 1243.
0393The first set of GUI items includes a back button 4735, a grid button 4736, a help button 4737, and a cancel button 4738. The back button 4735 allows the user to return to the collection organization GUI where they can select from different collections of images (eg albums, events, journals, etc.). By selecting the grid button 4736, the application puts the thumbnail display area on the GUI or removes it from the thumbnail display area (eg, via slide animation). In some embodiments, the user can also place the thumbnail display area on the GUI or remove it from the thumbnail display area via a swipe gesture. Help button 4737 provides the user with a tool pelp indicator that briefly describes the tool by invoking a contextual help feature that identifies the current active toolset to the user. In some embodiments, the pelp indicator is selectable to access additional information about the tool. By selecting the cancel button 4738, the application removes the latest edits to the image, whether this edit is cropping, color adjustment, etc. To make this cancellation, some embodiments remove the latest instruction from the set of edit instructions 4815 stored with the image.
0394The second set of GUI items includes a share button 4740, an info button 4741, an original button 4742, and an edit button 4743. The share button 4740 allows users to share images in a variety of different ways. In some embodiments, the user sends the selected image to another suitable device on the same network (eg, Wi Fi® or Bluetooth® network) and sends the image to image hosting or social media. You can upload to a website and, among other things, create a journal (ie, a presentation of placed images to which you can add additional content) from a selected set of images.
0395The info button 4741 activates a display area that displays additional information about one or more selected images. The information displayed in the activated display area can include part or all of the Exif data (eg, camera settings, time stamps, etc.) stored for the image. When multiple images are selected, some embodiments display only Exif data that is common to all of the selected images. Some embodiments (i) display a map showing where an image or multiple images were captured according to GPS data, if information is available, and (ii) on any photo sharing website. Includes an additional tab in the information display area to display the comment stream for the image. To download this information from the website, the application sends this information to the website using the object ID stored for the image along with the shared image data. A comment stream and, in some cases, additional information is received from the website and displayed to the user.
0396The original button 4742 allows the user to switch between the original version of the image and the current edited version of the image. When the user selects a button, the application displays the original version of the image to which the edit command 4815 has not been applied at all. In some embodiments, an appropriately sized image is stored as one of the cached versions 4840 of the image, making this image quickly accessible. When the user selects the button 4742 again, the application displays the edited version of the image to which the edit command 4815 has been applied.
0397The edit button 4743 allows the user to enter or exit edit mode. When the user selects a set of editing tools in the toolbar 4720, the edit button 4743 returns the user to view and organize mode, as shown in Figure 47. When the user selects the edit button 4743 while in browse mode, the application returns to the last set of editing tools used in the order shown on toolbar 4720. That is, the items in the toolbar 4720 are arranged in a specific order, and the edit button 4743 activates the rightmost item that has been edited on the selected image.
0398Toolbar 4720 contains, as mentioned, five items 4745-1249 arranged in a particular order from left to right. Crop item 4745 activates a crop and rotate tool that allows the user to align the distorted image and remove unwanted parts of the image. Exposure item 4746 activates a set of exposure tools that allow the user to modify black spots, shadows, contrast, brightness, highlights, and white spots in an image. In some embodiments, this set of exposure tools is a set of sliders that work together in different combinations to modify the gradation attributes of the image. Color item 4747 activates a set of color tools that allow the user to modify saturation and vibrancy, as well as color-specific saturation (eg, blue or green pixels) and white balance. In some embodiments, some of these tools are presented as a set of sliders. Brush item 4748 activates a set of correction tools that allow the user to localize image corrections. With a brush, the user can remove red eyes and scratches and apply or remove saturation and other features to localized areas of the image by performing a rubbing action on the image. Finally, effect item 4749 activates a set of special effects that the user can apply to the image. These effects include duotone effect, graininess effect, gradation, tilt shift, non-photorealistic desaturation effect, grayscale effect, and various filters. In some embodiments, the application presents these effects as a set of items that expand from the toolbar 4725.
0399As mentioned, UI items 4745 to 1249 are arranged in a particular order. This order follows the order in which the user most commonly applies five different types of edits. Therefore, edit instructions 4815 are stored in this same order in some embodiments. When the user selects one of items 4745-1249, some embodiments apply only edits from the tool to the left of the selected tool to the displayed image (but other edits are instruction sets). Remains remembered in 4815).
0400Toolbar 4725 includes a set of GUI items 4750 to 1254 and setting items 4755. Auto-enhancement item 4750 automatically performs correction editing (eg, clear red-eye removal, color balance adjustment, etc.) on the image. Rotate button 4751 rotates any selected image. In some embodiments, the image is rotated 90 ° in a particular direction each time the rotate button is pressed. The automatic correction includes, in some embodiments, a predetermined set of editing instructions placed within the instruction set 4815. Some embodiments analyze the image and then define a set of instructions based on the analysis results. For example, the auto-enhancement tool attempts to detect red-eye in the image, but if no red-eye is detected, no instruction to correct the red-eye is generated. Similarly, the automatic color balance adjustment is based on the analysis result of the image. The rotation generated by the rotation button is also stored as an edit command.
0401The flag button 4752 tags any selected image as flagged. In some embodiments, the flagged images of the collection can be displayed without any unflagged images. The Favorites button 4753 allows the user to mark any selected image as a favorite. In some embodiments, this tags the image as a favorite and adds the image to the collection of favorite images. The hide button 4754 allows the user to tag the image as hidden. In some embodiments, the hidden image is not displayed within the thumbnail display area and / or when the user crawls the images in the collection within the image display area. As discussed above with reference to FIG. 48, many of these features are stored in the image data structure as tags.
0402Finally, the settings button 4755 launches a contextual menu that presents different menu options depending on the currently active toolset. For example, in browse mode, the menus of some embodiments present options for creating new albums, setting important photos for albums, and copying settings between photos, and other options. When different sets of editing tools are active, the menu presents options related to a particular active toolset.
0403Those skilled in the art will recognize that the image viewing and editing GUI4700 is only one embodiment of many possible graphical user interfaces for images for image viewing, editing and organizing applications. For example, various items can be located in different areas or in different orders, and some embodiments may include items with additional or different functionality. The thumbnail display area of some embodiments may display thumbnails that fit the aspect ratio, such as the corresponding full size image.
0404VII. Electronic system Many of the aforementioned functions and applications are implemented as software processes designated as instruction sets recorded on a computer-readable storage medium (also called a computer-readable medium). When these instructions are executed by one or more compute or processing units (eg, one or more processors, processor cores, or other processing units), those instructions are indicated to the processing units. Perform the action taken. Examples of computer-readable media include CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). However, it is not limited to these. Computer-readable media do not include carrier and electronic signals that travel wirelessly or by wired connection.
0405As used herein, the term "software" includes firmware stored in read-only memory or an application stored in magnetic storage that can be read into memory for processing by a processor. Also, in some embodiments, the plurality of software inventions may be implemented as subdivisions of a larger program, leaving the separate software inventions intact. In some embodiments, multiple software inventions can also be executed as separate programs. Finally, any combination of separate programs that execute the software inventions described herein together is within the scope of the invention. In some embodiments, a software program defines one or more specific machine implementations that perform the actions of the software program when installed to run on one or more electronic systems.
0406A. Mobile device The image editing and viewing applications of some embodiments run on mobile devices. FIG. 49 is an example of such a mobile computing device architecture 4900. Examples of mobile computing devices include smartphones, tablets, laptops and the like. As illustrated, the mobile computing device 4900 includes one or more processing units 4905, a memory interface 4910, and a peripheral interface 4915.
0407Peripheral interface 4915 is coupled to various sensors and subsystems, including camera subsystem 4920, wireless communication subsystem 4925, audio subsystem 4930, I / O subsystem 4935, and so on. Peripheral interface 4915 enables communication between the processing unit 4905 and various peripherals. For example, an orientation sensor 4945 (eg, a gyroscope) and an acceleration sensor 4950 (eg, an accelerometer) are coupled to the peripheral interface 4915 to facilitate alignment and acceleration functions.
0408The camera subsystem 4920 is one or more photosensors 4940 (eg, charged coupled device (CCD) photosensors, complementary metal oxide semiconductors (complementary). It is coupled to metal-oxide-semiconductor, CMOS) optical sensor, etc.). The camera subsystem 4940 combined with the optical sensor 4920 facilitates camera functions such as image and / or video data capture. The wireless communication subsystem 4925 serves to facilitate communication functions. In some embodiments, the wireless communication subsystem 4925 includes a wireless receiver and transmitter and an optical receiver and transmitter (not shown in FIG. 49). These receivers and transmitters of some embodiments are implemented to operate on one or more communication networks such as GSM networks, WiFi networks, Bluetooth networks and the like. The audio subsystem 4930 is coupled to a speaker to output audio (eg, to output different sound effects associated with different image manipulations). In addition, the audio subsystem 4930 is coupled to a microphone to facilitate voice activation functions such as voice recognition and digital recording.
0409The I / O subsystem 4935 transfers between the input / output peripherals 4905, such as displays and touch screens, and the CPU data bus via the peripheral interface 4915. The I / O subsystem 4935 includes a touch screen controller 4955 and another input controller 4960 to facilitate transfer between the I / O peripherals and the data bus of the processing unit 4905. As shown, the touch screen controller 4955 is coupled to the touch screen 4965. The touch screen controller 4955 uses one of several touch sensitivity technologies to detect contact and movement on the touch screen 4965. Other input controllers 4960 are coupled to other input / control devices, such as one or more buttons. Some embodiments include near-touch sensitive screens (near-touch sensitive screens) and corresponding controllers that can detect near-touch interactions in place of or in addition to touch interactions.
0410Memory interface 4910 can be coupled to memory 4970. In some embodiments, the memory 4970 is a combination of volatile memory (eg, fast random access memory), non-volatile memory (eg, flash memory), volatile and non-volatile memory combinations and / or any other form. Includes memory. As shown in Figure 49, memory 4970 stores operating system (OS) 4972. OS 4972 includes instructions for processing basic system services and instructions for performing hardware-dependent tasks.
0411Memory 4970 facilitates communication instructions 4974 that facilitate communication with one or more additional devices, graphical user interface instructions 4976 that facilitates graphical user interface processing, and images that facilitate image-related processing and functions. Processing instruction 4978, input processing instruction 4980 that facilitates input-related (for example, touch input) processing and functions, audio processing instruction 4982 that facilitates audio-related processing and functions, and camera-related processing and functions. Also includes camera instructions 4984 to facilitate. The instructions described above are merely exemplary, and memory 4970 includes additional and / or other instructions in some embodiments. For example, the memory of a smartphone can include telephone instructions that facilitate telephone-related processes and functions. The instructions specified above need not be executed as separate software programs or modules. Various functions of mobile computing equipment can be performed in hardware and / or software, including one or more signal processing and / or application-specific integrated circuits.
0412Although the components shown in FIG. 49 are shown as separate components, one of ordinary skill in the art will recognize that two or more components can be integrated into one or more integrated circuits. In addition, two or more components can be combined together by one or more communication buses or signal lines. Also, although many of the functions have been described as being performed by one component, those skilled in the art will recognize that the functions described with respect to FIG. 49 can be split into two or more integrated circuits.
0413B. Computer system FIG. 50 conceptually illustrates another embodiment of an electronic system 5000 in which some embodiments of the present invention are realized. The electronic system 5000 may be a computer (eg, desktop computer, personal computer, tablet computer, etc.), telephone, PDA, or other type of electronic or computing device. Such electronic systems include interfaces for various types of computer-readable media, and various other types of computer-readable media. The electronic system 5000 includes a bus 5005, a processing unit 5010, a graphics processing unit (GPU) 5015, a system memory 5020, a network 5025, a read-only memory 5030, a permanent storage device 5035, an input device 5040 and an output device 5045.
0414Bus 5005 collectively represents all system buses, peripheral buses, and chipset buses that connect a large number of internal devices of the electronic system 5000 by communication. For example, bus 5005 communicatively couples processing unit 5010 with read-only memory 5030, GPU 5015, system memory 5020, and permanent storage device 5035.
0415From these various memory units, the processing unit 5010 acquires instructions to be executed and data to be processed in order to execute the process of the present invention. The processing unit may be a single processor or a multi-core processor in different embodiments. Some instructions are passed to GPU 5015 and executed by GPU 5015. The GPU 5015 can offload various computations and complement image processing provided by the processing unit 5010. In some embodiments, such functionality may be provided using Core Image's kernel shading language.
0416Read-only memory (ROM) 5030 stores static data and instructions required by the processing unit 5010 and other modules of the electronic system. On the other hand, the permanent storage device 5035 is a read / write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 5000 is off. Some embodiments of the present invention use a large capacity storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 5035.
0417Other embodiments use removable storage devices (floppy disks, flash memory devices, etc., and their corresponding drives) as permanent storage devices. Like the permanent storage device 5035, the system memory 5020 is a read / write memory device. However, unlike the storage device 5035, the system memory 5020 is a volatile read / write memory such as a random access memory. System memory 5020 stores some of the instructions and data required by the processor at run time. In some embodiments, the process of the invention is stored in system memory 5020, permanent storage 5035, and / or read-only memory 5030. For example, various memory units include instructions for processing multimedia clips, according to some embodiments. From these various memory units, the processing unit 5010 acquires the instructions to be executed and the data to be processed to execute the processes of some embodiments.
0418Bus 5005 also connects to input device 5040 and output device 5045. The input device 5040 allows the user to send information to the electronic system and select commands. Input devices 5040 include alphanumeric keyboards and pointing devices (also referred to as "cursor control devices"), cameras (eg, webcams), microphones that receive voice commands, or similar devices. The output device 5045 displays the image generated by the electronic system or outputs the data in other ways. Output device 5045 includes display devices such as printers, cathode ray tubes (CRTs) or liquid crystal displays (LCDs), as well as speakers or similar audio output devices. Some embodiments include devices such as touch screens that act as both input and output devices.
0419Finally, as shown in FIG. 50, the bus 5005 couples the electronic system 5000 to the network 5025 via a network adapter (not shown). Thus, the computer may be part of the computer's network (such as a local area network (LAN), wide area network (WAN), or intranet), or a network of networks such as the Internet. Any or all components of the electronic system 5000 may be used in the present invention.
0420Some embodiments include electronic configurations such as a microprocessor, a storage device and memory for storing computer program instructions on a machine-readable or computer-readable medium (also referred to as a computer-readable storage medium, machine-readable medium or machine-readable storage medium). Contains elements. Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROMs), write-once compact discs (CD-R), rewritable compact discs (CD-RW), and read-only compact discs. Multipurpose discs (eg DVD-ROM, dual layer DVD-ROM), various recordable / rewritable DVDs (eg DVD-RAM, DVD-RW, DVD + RW, etc.), flash memory (eg SD card, mini) SD cards, micro SD cards, etc.), magnetic and / or solid state hard disks, read-only recordable Blu-Ray® discs, ultra-high density optical discs, any other optical or magnetic media, and floppy discs. .. A computer-readable medium may be run by at least one processing unit and may store a computer program containing an instruction set for performing various operations. Examples of computer programs or computer code include machine code as created by a compiler, and files containing high-level code executed by a computer, electronic component, or microprocessor using an interpreter.
0421The above discussion primarily refers to microprocessors or multi-core processors running software, but some embodiments include application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). Performed by one or more integrated circuits. In some embodiments, such an integrated circuit executes instructions stored in the circuit itself. In addition, some embodiments run software stored in a programmable logic device (PLD), ROM, or RAM device.
0422As used herein and in this application, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technical equipment. These terms do not include people or groups of people. By definition, the term "display or display" means display on an electronic system. As used in this specification and in the claims of this application, the terms "computer-readable medium", "computer-readable medium", and "machine-readable medium" are tangible physical storage of information in a computer-readable format. Completely restricted to objects. These terms exclude wireless signals, wired downloaded signals, and other interim signals.
0423Having described the invention in a number of specific details, one of ordinary skill in the art will appreciate that the invention can be practiced in other particular forms that do not deviate from the spirit of the invention. For example, many of the figures show various touch gestures (eg, tap, double tap, swipe gesture, hold-hold gesture, etc.). However, many of the illustrated operations use different touch gestures (eg, swipe instead of tap, etc.) or by non-touch input (eg, cursor controller, keyboard, touchpad / trackpad, near-touch sensitive screen, etc.). Can be done (using). In addition, some diagrams (Fig. 5, Fig. 8, Fig. 11, Fig. 15, Fig. 17, Fig. 21, Fig. 25, Fig. 28, Fig. 30, Fig. 32, Fig. 34, Fig. 36, Fig. 40, and Fig. 42 ~ (Including 44) conceptually describes the process. The specific operations of these processes do not have to be performed in the exact order shown and described. The specific operation does not have to be performed in one continuous series of operations, and various specific operations may be performed in different embodiments. In addition, the process may be carried out using several subprocesses or as part of a larger macro process. Thus, one of ordinary skill in the art will appreciate that the invention is not limited by these exemplary details, but is defined by the appended claims.
55 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| KR1020120010525A | Cites | Republic of Korea |
| US20090303199A1 | Cites | United States of America |
| Adobe Photoshop Express : 写真の加工は「Photoshop」の無料アプリで!Androidアプリ1420,2011年 2月13日,URL,http://octoba.net/archives/20110213-android-1420.html | Non-patent | – |
| TouchRetouch : イマイチ写真がよみがえる!"超"使える画像編集アプリ!Androidアプリ1852,2011年 7月21日,URL,http://octoba.net/archives/20110721-android-1852.html | Non-patent | – |
| Kuroobisan's Memo: iphone写真アプリ『Effect Touch』,2010年 1月,URL,http://kuroobisan.blogspot.jp/2010/01/iphoneeffect-touch.html | Non-patent | – |
126 members in 7 offices
Members126
| Document | Office | Kind | |
|---|---|---|---|
| US2013235067A1 | United States of America | A1 | |
| US2013235068A1 | United States of America | A1 | |
| US2013235069A1 | United States of America | A1 | |
| US2013235070A1 | United States of America | A1 | |
| US2013235071A1 | United States of America | A1 | |
| US2013235074A1 | United States of America | A1 | |
| US2013235076A1 | United States of America | A1 | |
| US2013236091A1 | United States of America | A1 | |
| US2013236093A1 | United States of America | A1 | |
| US2013238724A1 | United States of America | A1 | |
| US2013238747A1 | United States of America | A1 | |
| US2013238990A1 | United States of America | A1 | |
| US2013239031A1 | United States of America | A1 | |
| US2013239051A1 | United States of America | A1 | |
| US2013239055A1 | United States of America | A1 | |
| US2013239056A1 | United States of America | A1 | |
| US2013239057A1 | United States of America | A1 | |
| US2013239062A1 | United States of America | A1 | |
| US2013239063A1 | United States of America | A1 | |
| WO2013133893A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013133894A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013133895A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013133901A2 | World Intellectual Property Organization (WIPO) | A2 | |
| WO2013133905A1 | World Intellectual Property Organization (WIPO) | A1 | |
| KR20130140896A | Republic of Korea | A | |
| KR20140002064A | Republic of Korea | A | |
| KR20140002065A | Republic of Korea | A | |
| EP2681652A1 | European Patent Office (EPO) | A1 | |
| EP2685362A1 | European Patent Office (EPO) | A1 | |
| EP2685363A1 | European Patent Office (EPO) | A1 | |
| CN103677529A | China | A | |
| CN103703437A | China | A | |
| CN103729115A | China | A | |
| JP2014112380A | Japan | A | |
| JP2014112381A | Japan | A | |
| JP2014522527A | Japan | A | |
| GB201413469D0 | United Kingdom | D0 | |
| GB201413475D0 | United Kingdom | D0 | |
| GB201413675D0 | United Kingdom | D0 | |
| GB201413677D0 | United Kingdom | D0 | |
| GB2513499A | United Kingdom | A | |
| GB2513762A | United Kingdom | A | |
| GB2513763A | United Kingdom | A | |
| GB2513766A | United Kingdom | A | |
| CN104221358A | China | A | |
| CN104221359A | China | A | |
| CN104247390A | China | A | |
| CN104247392A | China | A | |
| US8963962B2 | United States of America | B2 | |
| US8971617B2 | United States of America | B2 | |
| US8971623B2 | United States of America | B2 | |
| US2015109324A1 | United States of America | A1 | |
| JP2015513747A | Japan | A | |
| US9041727B2 | United States of America | B2 | |
| JP5735090B2 | Japan | B2 | |
| JP2015517135A | Japan | A | |
| JP2015518295A | Japan | A | |
| JP2015518588A | Japan | A | |
| US2015205502A1 | United States of America | A1 | |
| US9092893B2 | United States of America | B2 | |
| US9105121B2 | United States of America | B2 | |
| JP2015149078A | Japan | A | |
| US9131192B2 | United States of America | B2 | |
| US9159144B2 | United States of America | B2 | |
| US9189876B2 | United States of America | B2 | |
| US2015331595A1 | United States of America | A1 | |
| US9202433B2 | United States of America | B2 | |
| KR101580478B1 | Republic of Korea | B1 | |
| US2016063952A1 | United States of America | A1 | |
| US9299168B2 | United States of America | B2 | |
| US9363220B2 | United States of America | B2 | |
| US2016173726A1 | United States of America | A1 | |
| US2016188131A1 | United States of America | A1 | |
| JP5965381B2 | Japan | B2 | |
| JP5986232B2This record | Japan | B2 | |
| JP5990600B2 | Japan | B2 | |
| JP6047587B2 | Japan | B2 | |
| JP6062982B2 | Japan | B2 | |
| JP2017016670A | Japan | A | |
| JP6077020B2 | Japan | B2 | |
| US9569078B2 | United States of America | B2 | |
| CN103729115B | China | B | |
| US9591181B2 | United States of America | B2 | |
| CN104247390B | China | B | |
| JP2017062797A | Japan | A | |
| US2017109023A1 | United States of America | A1 | |
| CN104221358B | China | B | |
| KR101737166B1 | Republic of Korea | B1 | |
| JP2017084387A | Japan | A | |
| CN103703437B | China | B | |
| KR101776147B1 | Republic of Korea | B1 | |
| CN107168614A | China | A | |
| JP6220021B2 | Japan | B2 | |
| CN104221359B | China | B | |
| US9886931B2 | United States of America | B2 | |
| JP6298522B2 | Japan | B2 | |
| JP6329230B2 | Japan | B2 | |
| JP2018152097A | Japan | A | |
| EP2685363B1 | European Patent Office (EPO) | B1 | |
| US10282055B2 | United States of America | B2 |
14 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 |
Numbers
- Publication
- 5986232
- Application
- 2014560908
Titles2
- Japanese
- 画像の切り抜き及び角度補正のためのユーザインタフェースツール
- English
- User interface tool for image cropping and angle correction
Classification
- CPC, 11
- H04N1/3877
- G06F3/04845
- G06T11/60
- G06F3/0484
- G06F3/16
- G06F3/0482
- G06F3/0488
- G06F3/04842
- G06F3/167
- G06T3/60
- G06T2200/24
- IPC, 1
- G06F3 0488
