Technologies for adjusting a perspective of a captured image for display

BACKGROUND

0001Augmented reality systems fuse the real-world and virtual-world environments by projecting virtual characters and objects into physical locations, thereby allowing for immersive experiences and novel interaction models. In particular, in some augmented reality systems, virtual characters or objects may be inserted into captured images of real-world environments (e.g., by overlaying a two- or three-dimensional rendering of a virtual character on a captured image or video stream of the real-world environment). In some systems, a physical object recognized in the captured image may be replaced by a virtual object associated with that physical object. For example, recognized vehicles in the captured image may be recognized and replaced with animated or cartoon-like vehicles.

0002Augmented reality systems have been implemented in both stationary and mobile computing devices. In some mobile augmented reality systems, a camera of a mobile computing device (e.g., a smart phone camera positioned opposite the display) captures images of the real-world environment. The augmented reality system then makes augmented reality modifications to the captured images and displays the augmented images in the display of the mobile computing device (e.g., in real time). In such a way, the user is able to see a virtual world corresponding with his or her actual real-world environment. However, because the user and the camera of the mobile computing device have different perspectives of the real-world environment, the immersive experience suffers due to an obstructed visual flow. For example, from the user's perspective, real-world objects (e.g., those at the periphery of the mobile computing device) are duplicated in the augmented reality renderings.

BRIEF DESCRIPTION OF THE DRAWINGS

0003The concepts described herein are illustrated by way of example and not by way of limitation in the accompanying figures. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. Where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements.

0004<figref idref="DRAWINGS">FIG. 1</figref> is a simplified block diagram of at least one embodiment of a mobile computing device for adjusting a perspective of a captured image for display;

0005<figref idref="DRAWINGS">FIG. 2</figref> is a simplified block diagram of at least one embodiment of an environment established by the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref>;

0006<figref idref="DRAWINGS">FIG. 3</figref> is a simplified flow diagram of at least one embodiment of a method for adjusting a perspective of a captured image for display by the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref>;

0007<figref idref="DRAWINGS">FIG. 4</figref> is a simplified flow diagram of at least one embodiment of a method for generating a back projection of a real-world environment of the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref>;

0008<figref idref="DRAWINGS">FIG. 5</figref> is a simplified illustration of a user holding the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref> during execution of the method of <figref idref="DRAWINGS">FIG. 4</figref>;

0009<figref idref="DRAWINGS">FIG. 6</figref> is simplified flow diagram of at least one other embodiment of a method for generating a back projection of the real-world environment of the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref>;

0010<figref idref="DRAWINGS">FIGS. 7-8</figref> are simplified illustrations of the user holding the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref> showing various angular relationships;

0011<figref idref="DRAWINGS">FIG. 9</figref> is a simplified illustration of a real-world environment of the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref>;

0012<figref idref="DRAWINGS">FIG. 10</figref> is a simplified illustration of the user holding the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref> and a corresponding captured image displayed on the mobile computing device without an adjusted perspective; and

0013<figref idref="DRAWINGS">FIG. 11</figref> is a simplified illustration of the user holding the mobile computing device of <figref idref="DRAWINGS">FIG. 1</figref> and a corresponding captured image displayed on the mobile computing device having an adjusted perspective by virtue of the method of <figref idref="DRAWINGS">FIG. 3</figref>.

DETAILED DESCRIPTION OF THE DRAWINGS

0014While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.

0015References in the specification to “one embodiment,” “an embodiment,” “an illustrative embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may or may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. Additionally, it should be appreciated that items included in a list in the form of “at least one A, B, and C” can mean (A); (B); (C): (A and B); (B and C); or (A, B, and C). Similarly, items listed in the form of “at least one of A, B, or C” can mean (A); (B); (C): (A and B); (B and C); or (A, B, and C).

0016The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).

0017In the drawings, some structural or method features may be shown in specific arrangements and/or orderings. However, it should be appreciated that such specific arrangements and/or orderings may not be required. Rather, in some embodiments, such features may be arranged in a different manner and/or order than shown in the illustrative figures. Additionally, the inclusion of a structural or method feature in a particular figure is not meant to imply that such feature is required in all embodiments and, in some embodiments, may not be included or may be combined with other features.

0018Referring now to <figref idref="DRAWINGS">FIG. 1</figref>, a mobile computing device <b>100</b> for adjusting a perspective of a captured image for display is shown. In use, as described in more detail below, the mobile computing device <b>100</b> is configured to capture an image of a user of the mobile computing device <b>100</b> and an image of a real-world environment of the mobile computing device <b>100</b>. The mobile computing device <b>100</b> further analyzes the captured image of the user to determine a position of the user's eye(s) relative to the mobile computing device <b>100</b>. As discussed below, in doing so, the mobile computing device <b>100</b> may determine the distance of the user to the mobile computing device <b>100</b> and identify/detect the position of the user's eye(s) in the captured image. Additionally, the mobile computing device <b>100</b> determines a distance of one or more objects (e.g., a principal object and/or other objects in the captured scene) in the captured real-world environment relative to the mobile computing device <b>100</b>. For example, as described below, the mobile computing device <b>100</b> may analyze the captured image of the real-world environment, utilize depth or distance sensor data, or otherwise determine the relative distance of the object depending on the particular embodiment. The mobile computing device <b>100</b> determines a back projection of the real-world environment to a display <b>120</b> of the mobile computing device <b>100</b> based on the distance of the real-world object relative to the mobile computing device <b>100</b>, the position of the user's eye(s) relative to the mobile computing device <b>100</b>, and one or more device parameters. As discussed below, the back projection may be embodied as a back projection image, a set of data (e.g., pixel values) usable to generate a back projection image, and/or other data indicative of the corresponding back projection image. As discussed below, the device parameters may include, for example, a focal length of a camera of the mobile computing device <b>100</b>, a size of the display <b>120</b> or of the mobile computing device <b>100</b> itself, a location of components of the mobile computing device <b>100</b> relative to one another or a reference point, and/or other relevant information associated with the mobile computing device <b>100</b>. The mobile computing device <b>100</b> displays an image based on the determined back projection and, in doing so, may apply virtual objects, characters, and/or scenery or otherwise modify the image for augmented reality. It should be appreciated that the techniques described herein result in an image back-projected to the display <b>120</b> such that the image visable on the display <b>120</b> maps directly, or near directly, to the real world such that the user feels as though she is looking at the real-world environment through a window. That is, in the illustrative embodiment, the displayed image includes the same content as that which is occluded by the mobile computing device <b>100</b> viewed from the same perspective as the user.

0019The mobile computing device <b>100</b> may be embodied as any type of computing device capable of performing the functions described herein. For example, the mobile computing device <b>100</b> may be embodied as a smartphone, cellular phone, wearable computing device, personal digital assistant, mobile Internet device, tablet computer, netbook, notebook, ultrabook, laptop computer, and/or any other mobile computing/communication device. As shown in <figref idref="DRAWINGS">FIG. 1</figref>, the illustrative mobile computing device <b>100</b> includes a processor <b>110</b>, an input/output (“I/O”) subsystem <b>112</b>, a memory <b>114</b>, a data storage <b>116</b>, a camera system <b>118</b>, a display <b>120</b>, one or more sensors <b>122</b>, and a communication circuitry <b>124</b>. Of course, the mobile computing device <b>100</b> may include other or additional components, such as those commonly found in a typical computing device (e.g., various input/output devices and/or other components), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory <b>114</b>, or portions thereof, may be incorporated in the processor <b>110</b> in some embodiments.

0020The processor <b>110</b> may be embodied as any type of processor capable of performing the functions described herein. For example, the processor <b>110</b> may be embodied as a single or multi-core processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit. Similarly, the memory <b>114</b> may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory <b>114</b> may store various data and software used during operation of the mobile computing device <b>100</b> such as operating systems, applications, programs, libraries, and drivers. The memory <b>114</b> is communicatively coupled to the processor <b>110</b> via the I/O subsystem <b>112</b>, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor <b>110</b>, the memory <b>114</b>, and other components of the mobile computing device <b>100</b>. For example, the I/O subsystem <b>112</b> may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem <b>112</b> may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor <b>110</b>, the memory <b>114</b>, and other components of the mobile computing device <b>100</b>, on a single integrated circuit chip.

0021The data storage <b>116</b> may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. In the illustrative embodiment, the data storage <b>116</b> may store device parameters <b>130</b> of the mobile computing device <b>100</b>. It should be appreciated that the particular device parameters <b>130</b> may vary depending on the particular embodiment. The device parameters <b>130</b> may include, for example, information or data associated with a size/shape of the mobile computing device <b>100</b>, the display <b>120</b>, and/or another component of the mobile computing device <b>100</b>, intrinsic parameters or other data regarding one or more cameras of the mobile computing device <b>100</b> (e.g., focal length, principle point, zoom information, etc.), a location of components of the mobile computing device <b>100</b> relative to a reference point (e.g., a coordinate system identifying relative locations of the components of the mobile computing device <b>100</b>), and/or other information associated with the mobile computing device <b>100</b>. Additionally, in some embodiments, the data storage <b>116</b> and/or the memory <b>114</b> may store various other data useful during the operation of the mobile computing device <b>100</b>.

0022The camera system <b>118</b> includes a plurality of cameras configured to capture images or video (i.e., collections of images or frames) and capable of performing the functions described herein. It should be appreciated that each of the cameras of the camera system <b>118</b> may be embodied as any peripheral or integrated device suitable for capturing images, such as a still camera, a video camera, or other device capable of capturing video and/or images. In the illustrative embodiment, the camera system <b>118</b> includes a user-facing camera <b>126</b> and an environment-facing camera <b>128</b>. As indicated below, each of the user-facing camera <b>126</b>, the environment-facing camera <b>128</b>, and/or other cameras of the camera system <b>118</b> may be embodied as a two-dimensional (2D) camera (e.g., an RGB camera) or a three-dimensional (3D) camera. Such 3D cameras include, for example, depth cameras, bifocal cameras, and/or cameras otherwise capable of generating a depth image, channel, or stream. For example, one or more cameras may include an infrared (IR) projector and an IR sensor such that the IR sensor estimates depth values of objects in the scene by analyzing the IR light pattern projected on the scene by the IR projector. In another embodiment, one or more of the cameras of the camera system <b>118</b> include at least two lenses and corresponding sensors configured to capture images from at least two different viewpoints of a scene (e.g., a stereo camera).

0023As described in greater detail below, the user-facing camera <b>126</b> is configured to capture images of the user of the mobile computing device <b>100</b>. In particular, the user-facing camera <b>126</b> captures images of the user's face, which may be analyzed to determine the location of the user's eye(s) relative to the mobile computing device <b>100</b> (e.g., relative to the user-facing camera <b>126</b> or another reference point of the mobile computing device <b>100</b>). The environment-facing camera <b>128</b> captures images of the real-world environment of the mobile computing device <b>100</b>. In the illustrative embodiment, the user-facing camera <b>126</b> and the environment-facing camera <b>128</b> are positioned on opposite sides of the mobile computing device <b>100</b> and therefore have fields of view in opposite directions. In particular, the user-facing camera <b>126</b> is on the same side of the mobile computing device <b>100</b> as the display <b>120</b> such that the user-facing camera <b>126</b> can capture images of the user as she views the display <b>120</b>.

0024The display <b>120</b> of the mobile computing device <b>100</b> may be embodied as any type of display on which information may be displayed to a user of the mobile computing device <b>100</b>. Further, the display <b>120</b> may be embodied as, or otherwise use any suitable display technology including, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, a cathode ray tube (CRT) display, a plasma display, a touchscreen display, and/or other display technology. Although only one display <b>120</b> is shown in the illustrative embodiment of <figref idref="DRAWINGS">FIG. 1</figref>, in other embodiments, the mobile computing device <b>100</b> may include multiple displays <b>120</b>.

0025As shown in <figref idref="DRAWINGS">FIG. 1</figref>, the mobile computing device <b>100</b> may include one or more sensors <b>122</b> configured to collect data useful in performing the functions described herein. For example, the sensors <b>122</b> may include a depth sensor that may be used to determine the distance of objects from the mobile computing device <b>100</b>. Additionally, in some embodiments, the sensors <b>122</b> may include an accelerometer, gyroscope, and/or magnetometer to determine the relative orientation of the mobile computing device <b>100</b>. In various embodiments, the sensors <b>122</b> may be embodied as, or otherwise include, for example, proximity sensors, optical sensors, light sensors, audio sensors, temperature sensors, motion sensors, piezoelectric sensors, and/or other types of sensors. Of course, the mobile computing device <b>100</b> may also include components and/or devices configured to facilitate the use of the sensor(s) <b>122</b>.

0026The communication circuitry <b>124</b> may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the mobile computing device <b>100</b> and other remote devices over a network (not shown). For example, in some embodiments, the mobile computing device <b>100</b> may offload one or more of the functions described herein (e.g., determination of the back projection) to a remote computing device. The communication circuitry <b>124</b> may be configured to use any one or more communication technologies (e.g., wireless or wired communications) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

0027Referring now to <figref idref="DRAWINGS">FIG. 2</figref>, in use, the mobile computing device <b>100</b> establishes an environment <b>200</b> for adjusting a perspective of a captured image for display on the display <b>120</b> of the mobile computing device <b>100</b>. As discussed below, the mobile computing device <b>100</b> captures an image of the user with the user-facing camera <b>126</b> and an image of the real-world environment of the mobile computing device <b>100</b> with the environment-facing camera <b>128</b>. Further, the mobile computing device determines a position of the user's eye(s) relative to the mobile computing device <b>100</b> based on the image captured by the user-facing camera <b>126</b> and a distance of an object(s) in the real-world environment relative to the mobile computing device <b>100</b> based on the image captured by the environment-facing camera <b>128</b>. The mobile computing device <b>100</b> then generates a back projection of the real-world object(s) to the display <b>120</b> and displays a corresponding image on the display <b>120</b> (e.g., including augmented reality modifications) based on the generated back projection.

0028The illustrative environment <b>200</b> of the mobile computing device <b>100</b> includes an image capturing module <b>202</b>, an eye tracking module <b>204</b>, an object distance determination module <b>206</b>, an image projection module <b>208</b>, and a display module <b>210</b>. Each of the modules of the environment <b>200</b> may be embodied as hardware, software, firmware, or a combination thereof. For example, in an embodiment, each of the modules of the environment <b>200</b> is embodied as a circuit (e.g., an image capturing circuit, an eye tracking circuit, an object distance determination circuit, an image projection circuit, and a display circuit). Additionally, in some embodiments, one or more of the illustrative modules may form a portion of another module. For example, in some embodiments, the image projection module <b>208</b> may form a portion of the display module <b>210</b>.

0029The image capturing module <b>202</b> controls the camera system <b>118</b> (e.g., the user-facing camera <b>126</b> and the environment-facing camera <b>128</b>) to capture images within the field of view of the respective camera <b>126</b>, <b>128</b>. For example, as described herein, the user-facing camera <b>126</b> is configured to capture images of the user's face (e.g., for eye detection/tracking). It should be appreciated that the mobile computing device <b>100</b> may detect and/or track one or both of the user's eyes and, therefore, in the illustrative embodiment, the images captured by the user-facing camera <b>126</b> for analysis by the mobile computing device <b>100</b> include at least one of the user's eyes. Although eye tracking and analysis is, at times, discussed herein in reference to a single eye of the user for simplicity and clarity of the description, the techniques described herein equally apply to detecting/tracking both of the user's eyes. Additionally, as described herein, the environment-facing camera <b>128</b> is configured to capture images of the real-world environment of the mobile computing device <b>100</b>. It should be appreciated that the captured scene may include any number of principal objects (e.g., distinct or otherwise important objects) although, for simplicity, such captured images are oftentimes described herein as having a single principal object.

0030The eye tracking module <b>204</b> determines the location/position of the user's eye relative to the mobile computing device <b>100</b> (e.g., relative to the user-facing camera <b>126</b> or another reference point). In doing so, the eye tracking module <b>204</b> detects the existence of one or more person's eyes in an image captured by the user-facing camera <b>126</b> and determines the location of the eye in the captured image (i.e., the portion of the image associated with the eye) that is to be tracked. To do so, the eye tracking module <b>204</b> may use any suitable techniques, algorithms, and/or image filters (e.g., edge detection and image segmentation). In some embodiments, the eye tracking module <b>204</b> determines the location of the user's face in the captured image and utilizes the location of the user's face to, for example, reduce the region of the captured image that is analyzed to locate the user's eye(s). Additionally, in some embodiments, the eye tracking module <b>204</b> analyzes the user's eyes to determine various characteristics/features of the user's eyes (e.g., glint location, iris location, pupil location, iris-pupil contrast, eye size/shape, and/or other characteristics) to determine the gaze direction of the user. The user's gaze direction may be used, for example, to determine whether the user is looking at the display <b>120</b>, to identify objects in the scene captured by the environment-facing camera <b>128</b> toward which the user's gaze is directed (e.g., principal objects), to determine a relative location or position (e.g., in three-dimensional space) of the user's eye(s), and/or for other purposes. Additionally, in some embodiments, the eye tracking module <b>204</b> may further determine the orientation of the user's head or otherwise determine the user's head pose.

0031As described below, in determining the position of the user's eye relative to the mobile computing device <b>100</b>, the eye tracking module <b>204</b> determines the distance of the user's eye relative to the mobile computing device <b>100</b> (e.g., relative to the user-facing camera <b>126</b> or another reference point). It should be appreciated that the eye tracking module <b>204</b> may utilize any suitable algorithms and/or techniques for doing so. For example, in some embodiments, the user-facing camera <b>126</b> may be embodied as a depth camera or other 3D camera capable of generating data (e.g., a depth stream or depth image) corresponding with the distance of objects in the captured scene. In another embodiment, the eye tracking module <b>204</b> may use face detection in conjunction with a known approximate size of a person's face to estimate the distance of the user's face from the mobile computing device <b>100</b>. In yet another embodiment, the eye tracking module <b>204</b> may analyze the region of the captured image corresponding with the user's eye to find reflections of light off the user's cornea (i.e., the glints) and/or pupil. Based on those reflections, the eye tracking module <b>204</b> may determine the location or position (e.g., in three-dimensional space) of the user's eye relative to the mobile computing device <b>100</b>. Further, in some embodiments, the eye tracking module <b>204</b> may utilize data generated by the sensors <b>122</b> (e.g., depth/distance information) in conjunction with the location of the user's eye in the captured image to determine the location of the user's eye relative to the mobile computing device <b>100</b>.

0032The object distance determination module <b>206</b> determines the distance of one or more objects in the real-world environment captured by the environment-facing camera <b>128</b> relative to the mobile computing device <b>100</b> (e.g., relative to the environment-facing camera <b>128</b> or another reference point of the mobile computing device <b>100</b>). As indicated above, the real-world environment within the field of view of and therefore captured by the environment-facing camera <b>128</b> may include any number of objects. Accordingly, depending on the particular embodiment, the object distance determination module <b>206</b> may determine the distance of each of the objects from the mobile computing device <b>100</b> or the distance of a subset of the objects from the mobile computing device <b>100</b> (e.g., a single object). For example, in some embodiments, the object distance determination module <b>206</b> identifies a principal object in the captured image for which to determine the distance. Such a principal object may be, for example, an object toward which the user's gaze is directed or a main object in the scene. In some embodiments, the object distance determination module <b>206</b> assumes that each of the objects in the scene is approximately the same distance from the mobile computing device <b>100</b> for simplicity. Further, in some embodiments, the object distance determination module <b>206</b> assumes or otherwise sets the distance of the object(s) to a predefined distance from the mobile computing device <b>100</b>. For example, the predefined distance may be a value significantly greater than the focal length of the environment-facing camera <b>128</b>, a value approximating infinity (e.g., the largest number in the available number space), or another predefined distance value. For ease of discussion, a number representing infinity may be referred to herein simply as “infinity.”

0033It should be appreciated that the object distance determination module <b>206</b> may determine the distance of an object in the real-world environment relative to the mobile computing device <b>100</b> using any suitable techniques and/or algorithms. For example, in some embodiments, the object distance determination module <b>206</b> may use one or more of the techniques and algorithms described above in reference to determining the distance of the user relative to the mobile computing device <b>100</b> (i.e., by the eye tracking module <b>204</b>). In particular, the environment-facing camera <b>128</b> may be embodied as a depth camera or other 3D camera, which generates depth data for determining the distance of objects in the captured image. Additionally or alternatively, the object distance determination module <b>206</b> may reference stored data regarding the size of certain objects to estimate the distance of the objects to the mobile computing device <b>100</b> in some embodiments. In yet another embodiment, the object distance determination module <b>206</b> may utilize data generated by the sensors <b>122</b> (e.g., depth/distance information) to determine the distance and/or location of the objects relative to the mobile computing device <b>100</b>. Of course, in some embodiments, the object distance determination module <b>206</b> may assign a distance of a particular object to a predefined value. For example, the object distance determination module <b>206</b> may assume the object is infinitely far in response to determining that the object's distance exceeds a predefined threshold. That is, in some embodiments, objects that are at least a threshold distance (e.g., four meters) from the mobile computing device <b>100</b> may be treated as though they are, for example, infinitely far from the mobile computing device <b>100</b>. Such embodiments may appreciate that calculation differences may become negligible (e.g., calculations based on a distance of ten meters and twenty meters may yield approximately the same result). As described below, the distance of the object(s) relative to the mobile computing device <b>100</b> (e.g., relative to the camera <b>128</b>) may be used to determine the location of the object(s) relative to the mobile computing device <b>100</b> and to generate a back projection of the real-world environment (e.g., based on the device parameters <b>130</b>).

0034The image projection module <b>208</b> generates a back projection of the real-world environment captured by the environment-facing camera <b>128</b> to the display <b>120</b>. In the illustrative embodiment, the image projection module <b>208</b> generates the back projection based on the distance of the object in the real-world environment relative to the mobile computing device <b>100</b> (e.g., infinity, a predefined distance, or the determined distance), the position/location of the user's eye relative to the mobile computing device <b>100</b>, and/or the device parameters <b>130</b> of the mobile computing device <b>100</b> (e.g., intrinsic parameters of the cameras <b>126</b>, <b>128</b>, the size of the mobile computing device <b>100</b> or the display <b>120</b>, etc.). As indicated above, by back projecting the real-world environment to the display <b>120</b> (i.e., toward the user's eye), the visual content occluded by the mobile computing device <b>100</b> is shown on the display <b>120</b> such that the user feels as though she is looking through a window. In other words, visual continuity is maintained, as objects around the periphery are not duplicated in the displayed image. It should be appreciated that the image projection module <b>208</b> may utilize any suitable techniques and/or algorithms for generating the back projection image for display on the display <b>120</b> of the mobile computing device <b>100</b>. As described below, <figref idref="DRAWINGS">FIGS. 4-8</figref> show illustrative embodiments for doing so.

0035The display module <b>210</b> renders images on the display <b>120</b> for the user of the mobile computing device <b>100</b> to view. For example, the display module <b>210</b> may render an image based on the back projection generated by the image projection module <b>208</b> on the display <b>120</b>. Of course, in some embodiments, the back projections may not be “projected” onto the display <b>120</b> in the traditional sense; rather, corresponding images may be generated for rendering on the display <b>120</b>. Further, as discussed above, in some embodiments, the display module <b>210</b> may modify the back projection image to include virtual objects, characters, and/or environments for augmented reality and render the modified image on the display <b>120</b>.

0036The communication module <b>212</b> handles the communication between the mobile computing device <b>100</b> and remote devices through the corresponding network. For example, in some embodiments, the mobile computing device <b>100</b> may communicate with a remote computing device to offload one or more of the functions of the mobile computing device <b>100</b> described herein to a remote computing device (e.g., for determination of the back projection image or modification of the images for augmented reality). Of course, relevant data associated with such analyses may be transmitted by the remote computing device and received by the communication module <b>212</b> of the mobile computing device <b>100</b>.

0037Referring now to <figref idref="DRAWINGS">FIG. 3</figref>, in use, the mobile computing device <b>100</b> may execute a method <b>300</b> for adjusting a perspective of a captured image for display by the mobile computing device <b>100</b>. The illustrative method <b>300</b> begins with blocks <b>302</b> and <b>310</b>. In block <b>302</b>, the mobile computing device <b>100</b> captures an image of the user's face with the user-facing camera <b>126</b>. Depending on the particular embodiment, the user-facing camera <b>126</b> may capture images continuously (e.g., as a video stream) for analysis or in response to a user input (e.g., a button press). In block <b>304</b>, the mobile computing device <b>100</b> identifies the user's eye(s) in the captured image. As discussed above, the mobile computing device <b>100</b> may utilize any suitable techniques and/or algorithms for doing so (e.g., edge detection and/or image segmentation). Further, depending on the particular embodiment, the mobile computing device <b>100</b> may determine and utilize the location of one or both of the user's eyes.

0038In block <b>306</b>, the mobile computing device <b>100</b> determines the position of the user's eye(s) relative to the user-facing camera <b>126</b> or another reference point of the mobile computing device <b>100</b>. In doing so, the mobile computing device <b>100</b> determines the distance of the user, or more particularly, of the user's eye(s) relative to the user-facing camera <b>126</b>. As discussed above, the mobile computing device <b>100</b> may make such a determination based on, for example, a depth image or other depth information generated by the user-facing camera <b>126</b> (i.e., if the user-facing camera <b>126</b> is a depth camera or other 3D camera), user gaze information, distance information generated by the sensors <b>122</b>, device parameters <b>130</b>, and/or other relevant data. The distance of the user relative to the user-facing camera <b>126</b> may be used in conjunction with the location of the user's eye(s) in the captured image to determine the position of the user's eye(s) relative to the user-facing camera <b>126</b> or other reference point of the mobile computing device <b>100</b>. It should be appreciated that the device parameters <b>130</b> may include information regarding the locations of the components of the mobile computing device <b>100</b> relative to one another, thereby establishing a coordinate system having a reference point as the origin. The reference point selected to be the origin may vary depending on the particular embodiment and may be, for example, the location of the user-facing camera <b>126</b>, the location of the environment-facing camera <b>128</b>, the center of the display <b>120</b>, or another suitable location.

0039As shown, in the illustrative embodiment of <figref idref="DRAWINGS">FIG. 3</figref>, blocks <b>302</b>-<b>308</b> and blocks <b>310</b>-<b>314</b> occur in parallel; however, in other embodiments, those blocks may be executed serially. In block <b>310</b>, the mobile computing device <b>100</b> captures an image of the real-world environment of the mobile computing device <b>100</b> with the environment-facing camera <b>128</b>. Similar to the user-facing camera <b>126</b>, depending on the particular embodiment, the environment-facing camera <b>128</b> may capture images continuously (e.g., as a video stream) for analysis or in response to a user input such as a button press. For example, in some embodiments, the user may provide some input to commence execution of the method <b>300</b> in which the mobile computing device <b>100</b> executes each of block <b>302</b> and <b>310</b>. As indicated above, in the illustrative embodiment, the environment-facing camera <b>128</b> is position opposite the user-facing camera <b>126</b> such that the environment-facing camera <b>128</b> has a field of view similar to the user (i.e., in the same general direction).

0040In block <b>312</b>, the mobile computing device <b>100</b> determines the distance of one or more objects in the corresponding real-world environment relative to the environment-facing camera <b>128</b> or another reference point of the mobile computing device <b>100</b>. As discussed above, the mobile computing device <b>100</b> may make such a determination based on, for example, depth information generated by the environment-facing camera <b>128</b> (i.e., if the environment-facing camera <b>128</b> is a depth camera or other 3D camera), distance information generated by the sensors <b>122</b>, device parameters <b>130</b>, and/or other relevant data. Further, the object(s) for which the relative distance is determined may vary depending on the particular embodiment. For example, as discussed above, in some embodiments, the mobile computing device <b>100</b> may determine the relative distance of each object or each principal object in the captured image, whereas in other embodiments, the mobile computing device <b>100</b> may determine only the relative distance of the main object in the captured image (e.g., the object to which the user's gaze is directed or otherwise determined to be the primary object). Further, as indicated above, the mobile computing device <b>100</b> may set the distance of the object(s) to a predefined distance in block <b>314</b>.

0041In block <b>316</b>, the mobile computing device <b>100</b> generates a back projection of the real-world environment to the display <b>120</b> based on the distance of the real-world object(s) relative to the mobile computing device <b>100</b> (e.g., the determined or predefined distance), the position of the user's eye relative to the mobile computing device <b>100</b>, and/or one or more device parameters <b>130</b> (e.g., intrinsic parameters of the cameras <b>126</b>, <b>128</b>, the size of the mobile computing device <b>100</b> or the display <b>120</b>, etc.). As indicated above, the mobile computing device <b>100</b> may generate a back projection image using any suitable algorithms and/or techniques for doing so. For example, in some embodiments, the mobile computing device <b>100</b> may generate a back projection by executing a method <b>400</b> as shown in <figref idref="DRAWINGS">FIG. 4</figref> and, in other embodiments, the mobile computing device <b>100</b> may generate the back projection by executing a method <b>600</b> as shown in <figref idref="DRAWINGS">FIG. 6</figref>. Of course, it should be appreciated that the embodiments of <figref idref="DRAWINGS">FIGS. 4 and 9</figref> are provided as illustrative embodiments and do not limit the concepts described herein.

0042After the back projection has been determined, the mobile computing device <b>100</b> displays an image on the display <b>120</b> based on the generated back projection in block <b>318</b>. In doing so, in block <b>320</b>, the mobile computing device <b>100</b> may modify the back projection or corresponding image for augmented reality purposes as discussed above. For example, the mobile computing device <b>100</b> may incorporate virtual characters, objects, and/or other virtual features into the constructed/generated back projection image for rendering on the display <b>120</b>. Of course, in some embodiments, the mobile computing device <b>100</b> may not modify the back projection for augmented reality or other purposes such that the viewer truly feels as though the display <b>120</b> is a window through which she can see the real-world environment occluded by the mobile computing device <b>100</b>.

0043Referring now to <figref idref="DRAWINGS">FIG. 4</figref>, the illustrative method <b>400</b> begins with block <b>402</b> in which the mobile computing device <b>100</b> determines whether to generate a back projection. If so, in block <b>404</b>, the mobile computing device <b>100</b> determines a ray <b>502</b> from the user's eye <b>504</b> through the next display pixel <b>506</b> of the display <b>120</b> to the real-world object(s) <b>508</b> as shown in <figref idref="DRAWINGS">FIG. 5</figref>. It should be appreciated that which display pixel <b>506</b> constitutes the “next” display pixel <b>506</b> may vary depending on the particular embodiment. In the illustrative embodiment, the mobile computing device <b>100</b> selects a display pixel <b>506</b> for which a ray <b>502</b> has not yet been determined during the execution of the method <b>400</b> as the “next” display pixel <b>506</b>. It should further be appreciated that, in other embodiments, the mobile computing device <b>100</b> may determine a ray <b>502</b> through another sub-region of the display <b>120</b> (i.e., a sub-region other than the display pixels at, for example, a different level of granularity).

0044As discussed above, the device parameters <b>130</b> of the mobile computing device <b>100</b> may include data regarding relative locations of the various components of the mobile computing device <b>100</b> and establish, for example, a three-dimensional coordinate system having some reference point as the origin. For example, in some embodiments, the environment-facing camera <b>128</b> may be the origin. It should be appreciated that every pixel/point on the display <b>120</b> is located at some point relative to the environment-facing camera <b>128</b>. Accordingly, in some embodiments, the mobile computing device <b>100</b> determines the corresponding three-dimensional coordinates of the user's eye <b>504</b> and the object(s) <b>508</b> based on the analyses described above. It should be appreciated that, armed with the coordinates or relative locations of the user's eye <b>504</b>, the display pixels <b>506</b>, and the object(s) <b>508</b>, the mobile computing device <b>100</b>, in the illustrative embodiment, determines a corresponding ray <b>502</b> from the user's eye <b>504</b> through each of the display pixels <b>506</b> to the object(s) <b>508</b>.

0045In block <b>406</b>, the mobile computing device <b>100</b> identifies the image pixel of the image of the real-world environment captured by the environment-facing camera <b>128</b> (see block <b>310</b> of <figref idref="DRAWINGS">FIG. 3</figref>) corresponding with the position/location <b>510</b> of the real-world object(s) to which the corresponding ray <b>502</b> is directed. For example, based on the device parameters <b>130</b> such as the intrinsic parameters (e.g., focal length) of the environment-facing camera <b>128</b> and the real-world coordinates or relative location of the object(s) <b>508</b>, the mobile computing device <b>100</b> may determine how the image captured by the environment-facing camera <b>128</b> is projected from the real-world environment to the captured image coordinates. In such embodiments, the mobile computing device <b>100</b> may thereby identify the image pixel associated with the real-world coordinates (i.e., the location <b>510</b>) to which the ray <b>502</b> is directed.

0046In block <b>408</b>, the mobile computing device <b>100</b> determines whether any display pixels <b>506</b> are remaining. If so, the method <b>400</b> returns to block <b>404</b> in which the mobile computing device <b>100</b> determines a ray <b>502</b> from the user's eye <b>504</b> through the next display pixel <b>506</b> to the real-world object(s) <b>508</b>. In other words, the mobile computing device <b>100</b> determines a ray <b>502</b> from the user's eye <b>504</b> through a corresponding display pixel <b>506</b> to the object(s) <b>508</b> in the real-world environment for each display pixel <b>506</b> of the display <b>120</b> (or other sub-region of the display <b>120</b>) and identifies an image pixel of the image of the real-world environment captured by the environment-facing camera <b>128</b> corresponding with a location of the object(s) in the real-world environment to which the corresponding ray <b>502</b> is directed for each determined ray <b>502</b>. In block <b>410</b>, the mobile computing device <b>100</b> constructs an image (e.g., a back projection image) from the identified image pixels for display on the mobile computing device <b>100</b>. In the illustrative embodiment, the mobile computing device <b>100</b> generates an image having the identified image pixels in the appropriate image coordinates of the generated image. In other words, the mobile computing device <b>100</b> may back project the visual content from a location to which each corresponding ray <b>502</b> is directed to the corresponding point on the display <b>120</b> through which the ray <b>502</b> is directed.

0047Referring now to <figref idref="DRAWINGS">FIG. 6</figref>, in use, the mobile computing device <b>100</b> may execute a method <b>600</b> for generating a back projection of the real-world environment of the mobile computing device <b>100</b> as indicated above. The illustrative method <b>600</b> begins with block <b>602</b> in which the mobile computing device <b>100</b> determines whether to generate a back projection. If so, in block <b>604</b>, the mobile computing device <b>100</b> determines the angular size <b>702</b> of the mobile computing device <b>100</b> from the user's perspective based on the distance <b>704</b> of the user <b>706</b> relative to the user-facing camera <b>126</b> (or other reference point of the mobile computing device <b>100</b>) and device parameters <b>130</b> as shown with regard to <figref idref="DRAWINGS">FIGS. 7-8</figref>. As indicated above, the device parameters <b>130</b> may include, for example, the size, shaped, and other characteristics of the mobile computing device <b>100</b> and/or components of the mobile computing device <b>100</b>. It should be appreciated that the angular size of an object is indicative of the viewing angle required to encompass the object from a reference point (e.g., a viewer or camera) that is a given distance from the object. In the illustrative embodiment, the angular size of an object (e.g., the mobile computing device <b>100</b>) from a perspective point (e.g., the user's eye or the environment-facing camera <b>128</b>) is determined according to

0000<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><mrow><mi>δ</mi><mo>=</mo><mrow><mn>2</mn><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo></mo><mrow><mi>arctan</mi><mo></mo><mrow><mo>(</mo><mfrac><mi>d</mi><mrow><mn>2</mn><mo></mo><mi>D</mi></mrow></mfrac><mo>)</mo></mrow></mrow></mrow></mrow><mo>,</mo></mrow></math></maths><img file="US2016078680A1_D0001.tif" />

0000where δ is the angular size of the object, d is an actual size of the corresponding object, and D is a distance between the corresponding object and the perspective point (i.e., the point from which the angular size is determined). In other embodiments, however, the angular size of an object may be otherwise determined. It should further be appreciated that, while the angular size may at times be discussed herein with respect to two dimensions, the techniques described herein may be applied to three dimensions as well (e.g., accounting for both a horizontal angular size and a vertical angular size, determining the angular size across a diagonal of the object, projecting the three-dimensional size to two dimensions, employing a three-dimensional equivalent of the angular size formula provided above, etc.).

0048In block <b>606</b>, the mobile computing device <b>100</b> determines the distance <b>708</b> of the real-world object(s) <b>710</b> relative to the user <b>706</b>. In the illustrative embodiment, the mobile computing device <b>100</b> makes such a determination based on the distance <b>704</b> of the user <b>706</b> to the user-facing camera <b>126</b> (see block <b>308</b> of <figref idref="DRAWINGS">FIG. 3</figref>) and the distance <b>712</b> of the real-world object(s) <b>710</b> to the environment-facing camera <b>128</b> (see block <b>312</b> of <figref idref="DRAWINGS">FIG. 3</figref>) or other reference point of the mobile computing device <b>100</b>. In doing so, the mobile computing device <b>100</b> may, in some embodiments, assume the user <b>706</b>, the mobile computing device <b>100</b>, and the object(s) <b>710</b> are relatively collinear and add the two previously calculated distances to determine the distance <b>708</b> between the user <b>706</b> and the real-world object(s) <b>710</b> (e.g., if the objects are far from the user). In other embodiments, the mobile computing device <b>100</b> may employ a more sophisticated algorithm for determining the distance <b>708</b> between the user <b>706</b> and the real-world object(s) <b>710</b>. For example, the mobile computing device <b>100</b> may make such a determination based on the relative locations of the mobile computing device <b>100</b>, the user <b>706</b> (or, more particularly, the user's eye(s)), and the object(s) <b>710</b> to one another or to a particular reference point (e.g., a defined origin), the mobile computing device <b>100</b> and the known distances <b>704</b>, <b>712</b> between the user <b>706</b> and the mobile computing device <b>100</b> and between the mobile computing device <b>100</b> and the object(s) <b>710</b> (e.g., based on the properties of a triangle).

0049In block <b>608</b>, the mobile computing device <b>100</b> determines the region <b>714</b> of the real-world object(s) <b>710</b> that is occluded by the mobile computing device <b>100</b> from the user's perspective. In the illustrative embodiment, the mobile computing device <b>100</b> makes such a determination based on the angular size <b>702</b> of the mobile computing device <b>100</b> from the user's perspective and the distance <b>708</b> of the real-world object <b>710</b> relative to the user <b>706</b>.

0050In block <b>610</b>, the mobile computing device <b>100</b> determines a corrected zoom magnitude of the environment-facing camera <b>128</b> based on the region <b>714</b> of the real-world object(s) occluded from the user's perspective and the distance <b>712</b> of the real-world object(s) to the environment-facing camera <b>128</b>. In other words, the mobile computing device <b>100</b> determines a zoom magnitude of the environment-facing camera <b>128</b> needed to capture an image with the environment-facing camera <b>128</b> corresponding with the region <b>714</b> of the object(s) <b>710</b> occluded by the mobile computing device <b>100</b> from the user's perspective. As discussed above, the device parameters <b>130</b> may include intrinsic parameters (e.g., the focal length, image projection parameters, etc.) of the camera <b>128</b>. It should be appreciated that such device parameters <b>130</b> may be used, in some embodiments, to identify the zoom magnitude corresponding with capturing a particular region of an environment that is a certain distance from the camera <b>128</b>. In some embodiments, the zoom magnitude is determined such that the environment-facing camera <b>128</b> captures an image having only image pixels corresponding with visual content (e.g., features of the object(s) <b>710</b>) of the object(s) <b>710</b> from the region <b>714</b> of the object(s) occluded by the mobile computing device <b>100</b> from the user's perspective.

0051In block <b>612</b> of the illustrative embodiment, to determine the corrected zoom magnitude, the mobile computing device <b>100</b> determines the angular size <b>716</b> of a region <b>718</b> of the real-world object(s) <b>710</b> from the perspective of the environment-facing camera <b>128</b> corresponding with the region <b>714</b> of the real world object(s) <b>710</b> occluded from the user's perspective. The mobile computing device <b>100</b> may make such a determination based on, for example, the device parameters <b>130</b> and/or corresponding geometry. That is, in some embodiments, the mobile computing device <b>100</b> may determine the angular size <b>716</b> based on the size of the region <b>714</b>, the distance <b>712</b>, and the angular size formula provided above. It should be appreciated that, in some embodiments, the region <b>718</b> and the region <b>714</b> are the same region, whereas in other embodiments, those reasons may differ to some extent. Similarly, the corrected zoom magnitude may diverge from the precise zoom required to generate the region <b>718</b> (e.g., based on technological, hardware, and/or spatial limitations). In block <b>614</b>, the mobile computing device <b>100</b> generates an image with the corrected zoom magnitude for display on the mobile computing device <b>100</b>. For example, in some embodiments, the mobile computing device <b>100</b> may capture a new image with the environment-facing camera <b>128</b> from the same perspective but having a different zoom magnitude. In other embodiments, the mobile computing device <b>100</b> may, for example, modify the original image captured by the environment-facing camera <b>128</b> to generate an image with the desired zoom magnitude and other characteristics.

0052Referring now to <figref idref="DRAWINGS">FIGS. 9-11</figref>, simplified illustrations of a real-world environment <b>900</b> (see, for example, <figref idref="DRAWINGS">FIG. 9</figref>) of the mobile computing device <b>100</b> and of a user holding the mobile computing device <b>100</b> (see <figref idref="DRAWINGS">FIGS. 10-11</figref>) are shown. As discussed above, the real-world environment <b>900</b> may be captured by the environment-facing camera <b>128</b> and rendered on the display <b>120</b>. Further, in circumstances in which augmented reality systems are utilized, the captured images may be modified to incorporate, for example, virtual characters, objects, or other features into the captured image for display on the mobile computing device <b>100</b>. In embodiments in which the method <b>300</b> of <figref idref="DRAWINGS">FIG. 3</figref> is not utilized (i.e., if the captured image or modified version for augmented reality is displayed on the display <b>120</b> of the mobile computing device <b>100</b>), the image <b>902</b> displayed on the display <b>120</b> includes real-world objects <b>904</b> that are also visible in the real-world environment <b>900</b> around the periphery of the mobile computing device <b>100</b> (see, for example, <figref idref="DRAWINGS">FIG. 10</figref>). In other words, certain real-world objects <b>904</b> that are visible to the user are duplicated in the displayed image <b>902</b> thereby obstructing the visual flow. However, in embodiments in which the method <b>300</b> of <figref idref="DRAWINGS">FIG. 3</figref> is utilized, the image <b>906</b> displayed on the display <b>120</b> includes the same visual content as what is occluded by the mobile computing device <b>100</b> viewed from the same perspective as the user. Because visual continuity between the displayed image <b>906</b> and the background real-world environment <b>900</b> is maintained, the user feels as though she is looking at the real-world environment <b>900</b> through a window.

EXAMPLES

0053Illustrative examples of the technologies disclosed herein are provided below. An embodiment of the technologies may include any one or more, and any combination of, the examples described below.

0054Example 1 includes a mobile computing device for adjusting a perspective of a captured image for display, the mobile computing device comprising a display; a camera system comprising a first camera and a second camera, the camera system to capture (i) a first image of a user of the mobile computing device with the first camera and (i) a second image of a real-world environment of the mobile computing device with the second camera; an eye tracking module to determine a position of an eye of the user relative to the mobile computing device based on the first captured image; an object distance determination module to determine a distance of an object in the real-world environment relative to the mobile computing device based on the second captured image; and an image projection module to generate a back projection of the real-world environment captured by the second camera to the display based on the determined distance of the object in the real-world environment relative to the mobile computing device, the determined position of the user's eye relative to the mobile computing device, and at least one device parameter of the mobile computing device.

0055Example 2 includes the subject matter of Example 1, and wherein to generate the back projection comprises to determine, for each display pixel of the display, a ray from the user's eye through a corresponding display pixel to the object in the real-world environment; identify, for each determined ray, an image pixel of the second captured image of the real-world environment corresponding with a position of the object in the real-world environment to which the corresponding ray is directed; and construct a back projection image based on the identified image pixels for display on the display of the mobile computing device.

0056Example 3 includes the subject matter of any of Examples 1 and 2, and wherein to generate the back projection comprises to determine an angular size of the mobile computing device from a perspective of the user; determine a distance of the object in the real-world environment relative to the user; determine a region of the object occluded by the mobile computing device from the user's perspective; determine a corrected zoom magnitude of the second camera based on the determined region of the object occluded by the mobile computing device and the distance of the object relative to the mobile computing device; and generate a back projection image based on the corrected zoom magnitude for display on the display of the mobile computing device.

0057Example 4 includes the subject matter of any of Examples 1-3, and wherein to determine the corrected zoom magnitude comprises to determine an angular size of a region of the object from a perspective of the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

0058Example 5 includes the subject matter of any of Examples 1-4, and wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

0059Example 6 includes the subject matter of any of Examples 1-5, and wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera having only image pixels corresponding with features of the object from the region of the object occluded by the mobile computing device from the user's perspective.

0060Example 7 includes the subject matter of any of Examples 1-6, and wherein to determine the angular size of the mobile computing device from the user's perspective comprises to determine an angular size of the mobile computing device from the user's perspective based on a distance of the user's eye relative to the mobile computing device and a size of the mobile computing device; determine the distance of the object relative to the user comprises to determine the distance of the object relative to the user based on the distance of the user's eye relative to the mobile computing device and the distance of the object relative to the mobile computing device; and determine the region of the object occluded by the mobile computing device from the user's perspective comprises to determine the angular size of the region of the object occluded by the mobile computing device based on the angular size of the mobile computing device from the user's perspective and the distance of the object relative to the user.

0061Example 8 includes the subject matter of any of Examples 1-7, and wherein angular size, δ, is determined according to

0000<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mrow><mrow><mi>δ</mi><mo>=</mo><mrow><mn>2</mn><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo></mo><mrow><mi>arctan</mi><mo></mo><mrow><mo>(</mo><mfrac><mi>d</mi><mrow><mn>2</mn><mo></mo><mi>D</mi></mrow></mfrac><mo>)</mo></mrow></mrow></mrow></mrow><mo>,</mo></mrow></math></maths><img file="US2016078680A1_D0002.tif" />

0000wherein d is an actual size of a corresponding object and D is a distance between the corresponding object and a point, the point being a perspective from which the angular size is determined.

0062Example 9 includes the subject matter of any of Examples 1-8, and wherein to capture the first image of the user comprises to capture an image of a face of the user; and determine the position of the user's eye relative to the mobile computing device comprises to identify a location of the user's eye in the image of the user's face.

0063Example 10 includes the subject matter of any of Examples 1-9, and wherein to determine the position of the user's eye relative to the mobile computing device comprises to determine a distance of the user's eye to the mobile computing device.

0064Example 11 includes the subject matter of any of Examples 1-10, and wherein to determine the position of the user's eye relative to the mobile computing device comprises to determine a position of the user's eye relative to the first camera; and determine the distance of the object in the real-world environment relative to the mobile computing device comprises to determine a distance of the object relative to the second camera.

0065Example 12 includes the subject matter of any of Examples 1-11, and wherein the first camera has a field of view in a direction opposite a field of view of the second camera about the display.

0066Example 13 includes the subject matter of any of Examples 1-12, and wherein to determine the distance of the object in the real-world environment relative to the mobile computing device comprises to set a distance of the object relative to the mobile computing device to a predefined distance.

0067Example 14 includes the subject matter of any of Examples 1-13, and wherein the predefined distance is greater than a focal length of the second camera.

0068Example 15 includes the subject matter of any of Examples 1-14, and further including a display module to display an image on the display based on the generated back projection of the real-world environment captured by the second camera.

0069Example 16 includes the subject matter of any of Examples 1-15, and wherein to display the image based on the generated back projection comprises to display an image corresponding with the back projection modified to include augmented reality features.

0070Example 17 includes the subject matter of any of Examples 1-16, and wherein the at least one device parameter comprises at least one of (i) a focal length of the second camera, (ii) a size of the display, (iii) a size of the mobile computing device, or (iv) a location of components of the mobile computing device relative to a reference point.

0071Example 18 includes a method for adjusting a perspective of a captured image for display on a mobile computing device, the method comprising capturing, by a first camera of the mobile computing device, a first image of a user of the mobile computing device; determining, by the mobile computing device, a position of an eye of the user relative to the mobile computing device based on the first captured image; capturing, by a second camera of the mobile computing device different from the first camera, a second image of a real-world environment of the mobile computing device; determining, by the mobile computing device, a distance of an object in the real-world environment relative to the mobile computing device based on the second captured image; and generating, by the mobile computing device, a back projection of the real-world environment captured by the second camera to a display of the mobile computing device based on the determined distance of the object in the real-world environment relative to the mobile computing device, the determined position of the user's eye relative to the mobile computing device, and at least one device parameter of the mobile computing device.

0072Example 19 includes the subject matter of Example 18, and wherein generating the back projection comprises determining, for each display pixel of the display, a ray from the user's eye through a corresponding display pixel to the object in the real-world environment; identifying, for each determined ray, an image pixel of the second captured image of the real-world environment corresponding with a position of the object in the real-world environment to which the corresponding ray is directed; and constructing a back projection image based on the identified image pixels for display on the display of the mobile computing device.

0073Example 20 includes the subject matter of any of Examples 18 and 19, and wherein generating the back projection comprises determining an angular size of the mobile computing device from a perspective of the user; determining a distance of the object in the real-world environment relative to the user; determining a region of the object occluded by the mobile computing device from the user's perspective; determining a corrected zoom magnitude of the second camera based on the determined region of the object occluded by the mobile computing device and the distance of the object relative to the mobile computing device; and generating a back projection image based on the corrected zoom magnitude for display on the display of the mobile computing device.

0074Example 21 includes the subject matter of any of Examples 18-20, and wherein determining the corrected zoom magnitude comprises determining an angular size of a region of the object from a perspective of the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

0075Example 22 includes the subject matter of any of Examples 18-21, and wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

0076Example 23 includes the subject matter of any of Examples 18-22, and wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera having only image pixels corresponding with features of the object from the region of the object occluded by the mobile computing device from the user's perspective.

0077Example 24 includes the subject matter of any of Examples 18-23, and wherein determining the angular size of the mobile computing device from the user's perspective comprises determining an angular size of the mobile computing device from the user's perspective based on a distance of the user's eye relative to the mobile computing device and a size of the mobile computing device; determining the distance of the object relative to the user comprises determining the distance of the object relative to the user based on the distance of the user's eye relative to the mobile computing device and the distance of the object relative to the mobile computing device; and determining the region of the object occluded by the mobile computing device from the user's perspective comprises determining the region of the object occluded by the mobile computing device based on the angular size of the mobile computing device from the user's perspective and the distance of the object relative to the user.

0078Example 25 includes the subject matter of any of Examples 18-24, and wherein angular size, δ, is determined according to

0000<maths id="MATH-US-00003" num="00003"><math overflow="scroll"><mrow><mrow><mi>δ</mi><mo>=</mo><mrow><mn>2</mn><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo></mo><mrow><mi>arctan</mi><mo></mo><mrow><mo>(</mo><mfrac><mi>d</mi><mrow><mn>2</mn><mo></mo><mi>D</mi></mrow></mfrac><mo>)</mo></mrow></mrow></mrow></mrow><mo>,</mo></mrow></math></maths><img file="US2016078680A1_D0003.tif" />

0000wherein d is an actual size of a corresponding object and D is a distance between the corresponding object and a point, the point being a perspective from which the angular size is determined.

0079Example 26 includes the subject matter of any of Examples 18-25, and wherein capturing the first image of the user comprises capturing an image of a face of the user; and determining the position of the user's eye relative to the mobile computing device comprises identifying a location of the user's eye in the image of the user's face.

0080Example 27 includes the subject matter of any of Examples 18-26, and wherein determining the position of the user's eye relative to the mobile computing device comprises determining a distance of the user's eye to the mobile computing device.

0081Example 28 includes the subject matter of any of Examples 18-27, and wherein determining the position of the user's eye relative to the mobile computing device comprises determining a position of the user's eye relative to the first camera; and determining the distance of the object in the real-world environment relative to the mobile computing device comprises determining a distance of the object relative to the second camera.

0082Example 29 includes the subject matter of any of Examples 18-28, and wherein the first camera has a field of view in a direction opposite a field of view of the second camera about the display.

0083Example 30 includes the subject matter of any of Examples 18-29, and wherein determining the distance of the object in the real-world environment relative to the mobile computing device comprises setting a distance of the object relative to the mobile computing device to a predefined distance.

0084Example 31 includes the subject matter of any of Examples 18-30, and wherein the predefined distance is greater than a focal length of the second camera.

0085Example 32 includes the subject matter of any of Examples 18-31, and further including displaying, by the mobile computing device, an image on the display based on the generated back projection of the real-world environment captured by the second camera.

0086Example 33 includes the subject matter of any of Examples 18-32, and wherein displaying the image based on the generated back projection comprises displaying an image corresponding with the back projection modified to include augmented reality features.

0087Example 34 includes the subject matter of any of Examples 18-33, and wherein the at least one device parameter comprises at least one of (i) a focal length of the second camera, (ii) a size of the display, (iii) a size of the mobile computing device, or (iv) a location of components of the mobile computing device relative to a reference point.

0088Example 35 includes a mobile computing device comprising a processor; and a memory having stored therein a plurality of instructions that when executed by the processor cause the mobile computing device to perform the method of any of Examples 18-34.

0089Example 36 includes one or more machine-readable storage media comprising a plurality of instructions stored thereon that, in response to being executed, result in a mobile computing device performing the method of any of Examples 18-34.

0090Example 37 includes a mobile computing device for adjusting a perspective of a captured image for display, the mobile computing device comprising means for capturing, by a first camera of the mobile computing device, a first image of a user of the mobile computing device; means for determining a position of an eye of the user relative to the mobile computing device based on the first captured image; means for capturing, by a second camera of the mobile computing device different from the first camera, a second image of a real-world environment of the mobile computing device; means for determining a distance of an object in the real-world environment relative to the mobile computing device based on the second captured image; and means for generating a back projection of the real-world environment captured by the second camera to a display of the mobile computing device based on the determined distance of the object in the real-world environment relative to the mobile computing device, the determined position of the user's eye relative to the mobile computing device, and at least one device parameter of the mobile computing device.

0091Example 38 includes the subject matter of Example 37, and wherein the means for generating the back projection comprises means for determining, for each display pixel of the display, a ray from the user's eye through a corresponding display pixel to the object in the real-world environment; means for identifying, for each determined ray, an image pixel of the second captured image of the real-world environment corresponding with a position of the object in the real-world environment to which the corresponding ray is directed; and means for constructing a back projection image based on the identified image pixels for display on the display of the mobile computing device.

0092Example 39 includes the subject matter of any of Examples 37 and 38, and wherein the means for generating the back projection comprises means for determining an angular size of the mobile computing device from a perspective of the user; means for determining a distance of the object in the real-world environment relative to the user; means for determining a region of the object occluded by the mobile computing device from the user's perspective; means for determining a corrected zoom magnitude of the second camera based on the determined region of the object occluded by the mobile computing device and the distance of the object relative to the mobile computing device; and means for generating a back projection image based on the corrected zoom magnitude for display on the display of the mobile computing device.

0093Example 40 includes the subject matter of any of Examples 37-39, and wherein the means for determining the corrected zoom magnitude comprises means for determining an angular size of a region of the object from a perspective of the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

0094Example 41 includes the subject matter of any of Examples 37-40, and wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

0095Example 42 includes the subject matter of any of Examples 37-41, and wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera having only image pixels corresponding with features of the object from the region of the object occluded by the mobile computing device from the user's perspective.

0096Example 43 includes the subject matter of any of Examples 37-42, and wherein the means for determining the angular size of the mobile computing device from the user's perspective comprises means for determining an angular size of the mobile computing device from the user's perspective based on a distance of the user's eye relative to the mobile computing device and a size of the mobile computing device; means for determining the distance of the object relative to the user comprises means for determining the distance of the object relative to the user based on the distance of the user's eye relative to the mobile computing device and the distance of the object relative to the mobile computing device; and means for determining the region of the object occluded by the mobile computing device from the user's perspective comprises means for determining the region of the object occluded by the mobile computing device based on the angular size of the mobile computing device from the user's perspective and the distance of the object relative to the user.

0097Example 44 includes the subject matter of any of Examples 37-43, and wherein angular size, δ, is determined according to

0000<maths id="MATH-US-00004" num="00004"><math overflow="scroll"><mrow><mrow><mi>δ</mi><mo>=</mo><mrow><mn>2</mn><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo></mo><mrow><mi>arctan</mi><mo></mo><mrow><mo>(</mo><mfrac><mi>d</mi><mrow><mn>2</mn><mo></mo><mi>D</mi></mrow></mfrac><mo>)</mo></mrow></mrow></mrow></mrow><mo>,</mo></mrow></math></maths><img file="US2016078680A1_D0004.tif" />

0000wherein d is an actual size of a corresponding object and D is a distance between the corresponding object and a point, the point being a perspective from which the angular size is determined.

0098Example 45 includes the subject matter of any of Examples 37-44, and wherein the means for capturing the first image of the user comprises means for capturing an image of a face of the user; and means for determining the position of the user's eye relative to the mobile computing device comprises means for identifying a location of the user's eye in the image of the user's face.

0099Example 46 includes the subject matter of any of Examples 37-45, and wherein the means for determining the position of the user's eye relative to the mobile computing device comprises means for determining a distance of the user's eye to the mobile computing device.

0100Example 47 includes the subject matter of any of Examples 37-46, and wherein the means for determining the position of the user's eye relative to the mobile computing device comprises means for determining a position of the user's eye relative to the first camera; and means for determining the distance of the object in the real-world environment relative to the mobile computing device comprises means for determining a distance of the object relative to the second camera.

0101Example 48 includes the subject matter of any of Examples 37-47, and wherein the first camera has a field of view in a direction opposite a field of view of the second camera about the display.

0102Example 49 includes the subject matter of any of Examples 37-48, and wherein the means for determining the distance of the object in the real-world environment relative to the mobile computing device comprises means for setting a distance of the object relative to the mobile computing device to a predefined distance.

0103Example 50 includes the subject matter of any of Examples 37-49, and wherein the predefined distance is greater than a focal length of the second camera.

0104Example 51 includes the subject matter of any of Examples 37-50, and further including means for displaying an image on the display based on the generated back projection of the real-world environment captured by the second camera.

0105Example 52 includes the subject matter of any of Examples 37-51, and wherein the means for displaying the image based on the generated back projection comprises means for displaying an image corresponding with the back projection modified to include augmented reality features.

0106Example 53 includes the subject matter of any of Examples 37-52, and wherein the at least one device parameter comprises at least one of (i) a focal length of the second camera, (ii) a size of the display, (iii) a size of the mobile computing device, or (iv) a location of components of the mobile computing device relative to a reference point.