Active stereo with satellite device or devices.

(54) Title: ACTIVE STEREOSCOPE WITH SATELLITE DEVICE OR DEVICES. (54) Title: ACTIVE STEREO WITH SATELLITE DEVICE OR DEVICES.

(57) Summary

The object description is directed to the communication of image related data between a base station and / or one or more satellite computing devices, for example, tablet computers and / or smart phones. A satellite device captures image data and communicates image related data (such as images or depth data processed therefrom) to another device, such as a base station. The receiving device uses image-related data to enhance depth data (for example, a depth map) based on image data captured from the satellite device, which may be physically closer to something on stage than the base station, for example. To more accurately capture depth data under various conditions, an active lighting pattern can be projected from the base station or another external projector, so satellite units can use the active lighting from the other source and therefore do not need consume internal power to benefit from active lighting.

(57) Abstract

The subject disclosure is directed towards communicating image-related data between a base station and / or one or more satellite computing devices, eg, tablet computers and / or smartphones. A satellite device captures image data and communicates image-related data (such as the images or depth data processed therefrom) to another device, such as a base station. The receiving device uses the image-related data to enhance depth data (eg, a depth map) based upon the image data captured from the satellite device, which may be physically closer to something in the scene than the base station, for example. To more accurately capture depth data in various conditions, an active illumination pattern may be projected from the base station or another external projector, whereby satellite units may use the other source's active illumination and thus need not consume internal power to benefit from active illumination.

I Μ ΡΙ

<img file="MX357307B_D0001.tif" />

PATENT TITLE No. 357307, γ Mor ί I

Holders): MICROSOFT TECHNOLOGY LICENSING, LLC

Address: One Microsoft Way, Redmond, Washington, 98052, USA

Name: ACTIVE STEREOSCOPE WITH SATELLITE DEVICE OR DEVICES.

Classification: CIP: H04N13 / 20; B29C64 / 00; G01B11 / 22; G06F12 / 02; G06T7 / 00; H04N13 / 00

CPC: G01B11 / S513; GO18Ú / ^; ' G01B11 / 2545; G06F3 / 0653; G06F3 / 0659;

GC &lt;3 / 0683r G06F9 / 3004, GO6F &amp; 30O43. G06F9 / 30127; G06F11 / 3024;

G06h2 / 00; G06F12 / 02; G06F12 ^ 0? D7; G06F12 / 0292; G06K9 / 62;

G06K9 / 0063; GO6 ^ S / CÜ201y- „£ 506K9 / 00536, G06T1 / 60; H04N5 / 33;

H04N5 / 332; HÓ4JMS / 2 ^ '' HQ4H0 / O45; H04N13 / 02; H04N13 / 025;

H04N13 / Q27t HQ4 ^ T7 / 0 ^ AÉ £ F1 $ / 2Í3;, G02B27 / 4233; H04N2013 / 0081 Inventor (s): AIMM G. K1RK; OLlVBRAiWHY ^ &lt;&gt; i | ÍSl ^ F0 ^ SHAHRAM IZADI

Number:

MX / a / 2015/014577

Country:

US

US

Validity: Twenty years - ,. τ

Date of Ven ^ MÜ | qito: 14 of ^ buyt ^ Date of ExpeditiVm 4 clear pKiomtoád

Date:

April 2 &amp; 13 &amp; June 21, 2013 • $ j «: Y.

7⁴¾.

Number:

61/811232

13 / 924,475) 34 ί

,₄ *,, Ζ '**? '' zi-sY - «-

The reference patent is Otí ^ gá coMSndamenJo. in the 'artteujot't ·, 2 * fWd «ion #, levy lll ^ y.69 of the Industrial Property Law.

In accordance with article 23 dg1 «üaf'de la-Ptopleded Industrial. The * Bretenfépatefke has a t / igencijrUe vefr ^ e non-extendable years, counted from the date of presentation of l »eoKciti | d igtemacionafy subject sphere« dibago dala taitafP | ra keep Yipehtes the rights.

Who subscribes to the present title has made it possible for him (* n lolísRite ^ to by Ibe «rtiamjfe δ * fra &amp; áonesftl»? · Ίιι »^ &lt;Ιβ the Industrial Property Law (Official Gazette of the Federation (DOF) '$ / Be / 199 .Retaken&gt; «®β« βΜ. 10/25/4996, 26/42/1997? 05/17/1999, 01/26/2004, 06/16/2005, 01/25/2006, 06/05/05 2009.06 / 01 / 2010,18 / 06 / 20Í0, 2ώθβ «Π &lt;0 ^ 1βΙ ^ 75⁾9^ίΙΜ ^? 1 ^^ Ηΐ ·· * ^ »'fiW ° n V Clause a), 4' and 12 ° fractions I and lll of the Regulations of the Mexican Property Institute '^^ tiM ^ jNl ^ 1 # 1« WM ^ aj | e * &lt;07/01/2002, 07/15/2004, 07/28/2004 and 09/07/2007); Articles 1, 3, 4, 5, section V, subsection a), 16 section, efPy W and aoY ^ É ^ tutoOrgánj ^ o.egl Mexican Institute of Industrial Property (DOF 12/27/1999, amended on 10/10/2002, 07/29/2004, 08/04/2004 and 13 / Qp2C) (R) ;, lvSLy 5 “item a) of the Agreement that delegates powers to the Directors General Deputies, Coordinator, Divisional Directors, Holders of the Regional Offices, Divisional Deputy Directors, Departmental Coordinators and other subordinates of the Mexican Institute of Industrial Property. (DOF 12/15/1999, amended on 02/04/2000, 07/29/2004, 08/04/2004 and 09/13/2007).

This letter is signed with an advanced electronic signature (FIEL), based on articles 7 BIS 2 of the Industrial Property Law; 3rd of its Regulations, and 1 section III, 2 section V, 26 BIS and 26 TER of the Agreement establishing the guidelines for the use of the Electronic Payment and Services Portal (PASE) of the Mexican Institute of Industrial Property, in the procedures indicated.

DIVISIONAL DIRECTOR OF PATENTS NAHANNY CANAL REYES

<img file="MX357307B_D0002.tif" />

Original string:

NAHANNY MARISOL CANAL REYES | 00001000000403252793 | Tax Administration Service | 1695 || MX / 2018/57089 | MX / a / 2015/014577 | Patent title PCT | 1223 | GAGV | Pág (s) | nfKVUTyNz4bDeQMqFTXDIIWIt0

Digital stamp:

vhUpePgb2Tx / fGaxx / JmlfC / tw26EdkRQ0Hx6NCmyM / 5IAUbymMpEXO7jWEMvjzeTDFXwhzV + Wm8L6Htj / q3vO64lw

Rmbytlnly

AUwtglklZjDc7OBI55r1V7LjtJ05PKAv7 + XcYJfRdlmSMDO1z / Hi9AC09Go8jM8B0Q0OUoog ==

<img file="MX357307B_D0003.tif" />

MX / 2018/57089

IMPI

ACTIVE STEREOSCOPE WITH GOD POSITI \ ft? P £ ffÉ $

<img file="MX357307B_D0004.tif" />

SATELLITE

BACKGROUND

In active depth detection, such as that used by active stereoscope systems, a projector projects light patterns such as Infrared (IR) dots or lines to illuminate a scene that is detected. Note that the projector can use a laser that is consumed in the order of 1W of power, which means that the projector consumes too much energy that accumulates practically in a small satellite device, such as a smartphone or tablet computer, and fact is typically part of a device plugged into an outlet.

The projected patterns are then captured by a camera / sensor (two or more in stereoscope systems), with the image (or images) processed to calculate a depth map or the like. For example, in stereoscope systems, stereoscopic cameras capture two images from different viewing points . Then, for example, one way to perform depth estimation with a stereoscopic pair of Images is to find correspondences between the Images, for example, to correlate projected and detected points in the left Image with opposite points in the right image. Once correlated, the projected patterns within the images can be correlated with each other, and

IMPI

IHSTlTUTO W &amp; XICAWO Dt INDUSTRIAL PROPERTY

<img file="MX357307B_D0005.tif" />

triangulation, possibly together with disparities between one or more characteristics of the correlated points (for example, including their intensities) can be used to estimate a depth in an object impacted by that particular projected point.

In most scenarios, cameras that capture stereoscopic images are arranged to detect a relatively large space, such as a room, which can be relatively large. As a result, an object in the distance, such as a face, appears only in a relatively small number of camera pixels. Sufficient detail and / or precision depth estimation in this way is not available when needed by many applications, such as face recognition, etc.

BRIEF DESCRIPTION OF THE INVENTION

This Brief Description is provided to introduce a selection of representative concepts in a simplified form which is further described below in the Detailed Description. This Brief Description is not intended to identify key characteristics or essential features of the claimed subject matter, nor is it intended to be used in any way that may limit the scope of the claimed subject matter.

Briefly, one or more of several aspects of the subject matter described herein are directed towards using information from a

<img file="MX357307B_D0006.tif" />

satellite device (eg mobile) to moj ^ Eer the depth datus calculated by another device, such as a base station. One or more aspects are directed toward receiving image related data from a satellite device or other device, and improving a first depth data set based at least in part on image related data and posture information from the satellite device.

In one or more aspects, a base station is configured to determine the posture information of a satellite device and capture one or more images. The base station receives image related data from the satellite device and calculates a depth map based at least in part on posture information and image related data.

One or more aspects are directed to receiving image-related data from a satellite device at a base station, and capturing actively illuminated stereoscopic images at the base station. The posture information that corresponds to the satellite device is determined. Active illuminated stereoscopic images are processed to determine a depth map from the perspective of the base station that is enhanced by depth information corresponding to image related data.

Other advantages may become apparent from the following detailed description when taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

ΙΜΡΙ

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX357307B_D0007.tif" />

The present invention is illustrated by way of example and not by way of limitation in the accompanying figures in which like reference numerals indicate like elements and in which:

Figure 1 is a representation of a base station projecting a light pattern onto a stage, in which a satellite device captures at least part of the stage, according to one or more exemplary implementations.

Figure 2 is a block diagram depicting exemplary components that can be used to project and capture images for processing on depth data, in accordance with one or more exemplary implementations.

Figure 3 is a block diagram representing exemplary processing of base station data and satellite data into depth data, according to one or more exemplary Implementations.

Figure 4 is a block diagram representing exemplary processing of data from one satellite device and data from another satellite device into depth data, according to one or more exemplary implementations.

Figure 5 is a representation of the satellite device configured to signal a base station, according to one or more exemplary implementations.

Figures 6A and 6B are representations of ways in which

MEXICAN INSTITUTE PE THE PROPERTY

INDUSTRIAL depth data can be determined from a single camera on the satellite device, according to one or more exemplary implementations.

Figure 7 is a flowchart representing exemplary steps for obtaining image related data from a satellite device for use in determining a depth map, in accordance with one or more exemplary implementations.

Figure 8 is a block diagram representing an exemplary non-limiting computing system or operating environment, in the form of a mobile device, in which one or more aspects of various modalities described herein can be implemented.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generally directed toward using satellite devices (eg, mobiles) such as smartphones and tablet computers to communicate information to and from an active lighting base station, such as to improve depth detection. , provide more image detail and / or estimate posture. At the same time, satellite devices make use of active lighting from the base station for their own active detection, so satellite devices do not need to consume battery power to project the light pattern.

In one or more alternatives, multiple satellite devices

<img file="MX357307B_D0008.tif" />

MEXICAN INSTITUTE OF PROPERTY. .... x,,, INDUSTRIAL-they can use projected light from an external source to communicate image / depth / position data calculated from it to each other. Depth and other useful data regarding a scenario can thus be made available to devices without the need for a fixed base station.

It should be understood that any of the examples herein are not limiting. For example, although satellite devices are generally exemplified herein as being mobile, an auxiliary device (or more than one) may be placed as a fixed satellite which can be calibrated with the base station if desired. This can be useful to provide a camera that is closer to a certain significant part of a stage, for example. Thus, as used herein, the term satellite need not be mobile and may include any auxiliary device. As another example, flight time cameras can be used to determine at least some data. As such, the present invention is not limited to any of the particular embodiments, aspects, concepts, structures, functionalities, or examples described herein. Rather, any of the modalities, aspects, concepts, structures, functionalities, or examples described herein are not limiting, and the present invention may be used in various ways that provide benefits and advantages in active depth detection and image capture and general processing.

Figure 1 exemplifies an implementation, in which a base station 100 includes stereoscopic cameras (eg,

IMPI

MEXICAN INSTITUTE K THE INDUSTRIAL INSTITUTE

<img file="MX357307B_D0009.tif" />

infrared) 101 and 102 that capture images of a stage 104 from different perspectives. A projector 106 illuminates stage 104 in a light pattern (eg, in the order of 100.00 dots). The light pattern can be pseudo-random, and can be generated by emitting laser light through a diffraction optical element. For the sake of brevity, the dots / components of the emitted pattern are generally referred to as dots herein, although line patterns, dots, and other shapes, etc., can in fact be generated and detected.

Cameras 102 and 103 capture the points that are reflected off the target surfaces on stage 222 and (possibly) the background. In general, points detected by one camera (eg 102) can be correlated with points detected by the other camera (eg 103) by image processing 108 (eg a system or subsystem), whereby provides (eg, by triangulation) the distance to the reflecting surface over which each point was incident. Note that Figure 1 is not intended to be to scale, nor to transport any size, distance, point distribution pattern, point density, etc.

Base station 100 may also include a clean IR camera, for example, one with a filter that blocks the active lighting pattern and / or an RGB camera that captures visible light; if presented, they can be combined into a single chamber. Camera 112 in Figure 1 represents any of these alternatives.

As shown in Figure 1, one of the objects detected in scenario 104 may be a satellite device 110, such as a

MEXICAN INSTITUTE Ot LA MONEDAD

INDUSTRIAL mobile smartphone or tablet computing device At the same time, the satellite device 110 can detect other objects on the stage, for example, including the depth therein, such as by stereoscopic IR cameras that detect the lighting pattern , as well as possibly clean IR and / or RGB images thereof.

Figure 2 shows an exemplary base station 100 in which stereoscopic cameras 102 and 103 of an image capture system or subsystem 224 capture time synchronized images (eg, cameras are synchronized). In one implementation, the cameras capture infrared (IR) images, since IR does not affect the visible appearance of the stage (which is highly advantageous, such as in video conferencing and object modeling applications). As can be seen more easily, in some scenarios such as studio environments, more than two IR depth detection cameras may be featured. Furthermore, one or more additional cameras can be presented in a given system, such as RGB cameras, and such other cameras can be used to help align images, correlate pairs of points on different stereoscopic images, etc., for example.

In Figure 2, projector 106 projects an IR pattern onto a stage. By illuminating the stage with a relatively large number of distributed infrared dots, cameras 102 and 103 capture more texture data as part of the infrared image data. Note that the placement of projector 106 may be outside of the

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX357307B_D0010.tif" />

cameras (for example, Figure 1), or between cameras (Figure 2) or at another location, such as above or below one or both of the cameras. The examples herein are in no way limiting where the cameras 102, 103 and / or the projector 106 are located relative to each other, and similarly, the cameras can be placed in different positions relative to each other.

In one implementation, the exemplary image capturing system or subsystem 224 includes a controller 228 which, through a camera interface 230, controls the operation of cameras 102 and 103, and if present, camera 112. The controller exemplified by an interface Projector 232 also controls the operation of projector 106. For example, cameras 102 and 103 are synchronized (matched) to capture stereoscopic images at the same time, such as by a controller signal (or different signals for each camera). Projector 106 may be turned on or off, pulsed, and otherwise have one or more controllably varied parameters, for example.

Images captured by cameras 102 and 103 (and camera 112, if present) are provided to the image processing system or subsystem 108 comprising logic implemented in hardware and / or software (eg, as Computer-readable Instructions) . In some implementations, the Image processing system 108 and the Image capture system or subsystem 224, or parts thereof, may be combined into a single device. For example, a home entertainment device

IMPI

WgTrruTLi MUtCriwo DE LA ΙΊΜΜΗίΛΟ

<img file="MX357307B_D0011.tif" />

· Wc i¿n raLwcmv v can include all the components shown | 5n the figure¹AND^L(asT others not shown). In other implementations, part (or tCrff) 9 "image capture system or subsystem, such as cameras and projector, may be a separate device that attaches to a game console, personal computer, satellite devices, processing device dedicated, and / or the like.

In one or more implementations, the image processing system or subsystem 108 includes a processor 240 and a memory 242 containing one or more image processing algorithms 244. One or more depth maps 246 may be obtained by algorithms 124 such as by extracting correlation characteristics (such as points and / or lines). For example, as known, as described in US published patent application no. 20130100256, different points or other projected elements have different characteristics when captured, including intensity (brightness), depending on the distance from the projector to the reflective surfaces and / or the distance from the camera to the reflective surfaces. Also as known, points in different images taken at the same time (for example, with synchronized stereoscopic cameras), can be correlated with each other, such as by correlating small patches (for example, RGB) between RGB images of the same scene captured in the same moment. In this way, with captured images, known algorithms can determine individual depth-related characteristics (depth maps) by correlating

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX357307B_D0012.tif" />

components of projected light (eg, dots) on each image, using triangulation / disparities of certain characteristics between correlated dots to determine depths. This is one way in which a depth map can be obtained by stereoscopic image processing.

Also shown in Figure 1 is one or more interfaces 132 for the base station, such as for connecting a keyboard, game controller, display, microphone with pointing device for voice commands and / or the like as appropriate for a user interact with an application or the like that uses the depth map. At least one interface 132 allows the satellite device 110 to communicate (eg, wirelessly) with the base station as described herein.

One or more implementations comprise a base station 100 in a room that emits the laser pattern to calculate depth. When the user uses a satellite device 110, such as a smartphone or tablet computer, the camera or stereoscopic cameras in the satellite device 110 can observe the pattern of transmission from the base unit. Satellite device 110 can thus calculate a stereoscope based on the externally generated laser pattern, as well as communicate information to base station 100.

As described herein, the data from the satellite device can be used to calculate a depth map at a higher resolution than the base station can calculate; (note that

ΙΜΡΙ

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX357307B_D0013.tif" />

"Higher resolution" as used herein does not refer to any of the camera's resolutions, but rather to the ability to capture an object with more pixels than a more distant camera at the base station because the satellite device is closer to the object, for example the user's face). Furthermore, not only the spatial resolution can be increased, but also the depth / precision resolution. Image information or data corresponding to the information (eg, a depth map calculated on the satellite device) can be transmitted to the base unit. As used herein, image related data refers to actual image data (from one or more actively illuminated IR and / or RGB images and / or clean IR and / or RGB images, in whole or in part ), any associated metadata, and / or any corresponding information processed from that image data, eg a depth map. In this way, Image-related data can communicate to and from a base station, and to and from satellite devices.

Additionally, images and / or depth maps taken on the satellite device and the base unit can be correlated. This provides a six degree of freedom (6DoF) posture estimate of the location of the mobile unit relative to the base unit.

Posture estimation can be determined by image alignment and / or by pattern correlation / dot correlation. As an example, if the satellite device captures an image that also

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX357307B_D0014.tif" />

can detect the base station (eg, a user or device that does not prevent the base station from seeing the same area), then the pixels (or corresponding surrounding patches of each pixel) can be mapped in known ways. Alignment can also be established by aligning depth data calculated from the base station with depth data calculated from the satellite device. The posture of the satellite device in this way can be estimated.

This is generally depicted in Figure 3, where a base station 300 projects a light pattern from a projector 306 that is detected by base station cameras 301 and 302 as well as by one or more cameras on a satellite device 330 (more from a satellite device can be presented). Cameras 301 and 302 provide base station images 332 that are fed into image processing together with data from satellite device 334, which may be images or information such as depth maps locally processed from images on each satellite device. .

Note that as indicated by the dashed lines on device 330, an optional (eg low-energy) projector may be included in any or each of one or more satellite devices. The data projected by the satellite device augments the data projected from the base station. The projector in the satellite device 330 may be low energy because it is limited in distance, it is spatially limited (for example, a very

<img file="MX357307B_D0015.tif" />

WSTITUTO MEXICANO DE LA PROPERTY

INDUSTRIAL __ sparse or only focused on a small Image ruler, and / or temporarily limited (eg emits only every few frames) '*' ™

Base station images 332 can be processed to determine the 3D posture / location of satellite device 330. With this information, image processing 308 produces depth data 336, which may be conventional depth data from cameras owned by the base station enhanced by data from satellite device 334. In general, projected dots captured by base station cameras can be correlated with dots captured by the satellite device, after adjusting for size differences and the like due to differences in resolution. Posture data 338 can also be produced, such as on other satellite devices.

Note that instead of the satellite device that the base station provides with data to enhance its image data, the base station can send an approximate depth map (along with the posture of the device) to the satellite device. This can be used to increase the precision and / or decrease the calculation required for the depth estimation property of the satellite device.

The base station depth map (as well as RGB data) can be improved by downstream sampling the highest resolution data on a partial depth map that is combined with the initial base station depth map at appropriate coordinates, after adjusting the depths for posture, to improve precision in these coordinates, for example. Another way to

<img file="MX357307B_D0016.tif" />

that the base station depth map can be improved is to keep the original depth map together with the higher resolution depth map (eg after adjusting depths based on posture), for example. In this way, an application using the depth map can “zoom in” on the finer grain depth data when desired, for example, the application does not need to limit itself to the pixel resolution of the original depth map.

Note that some mobile devices may have front and rear cameras. If so, one scenario is to use the cameras for different purposes. For example, the rear camera can be used to capture images for posture calculations, while the front camera can capture images of the user's face, for example, which can be used for enhanced depth calculations.

Figure 4 shows another alternative, in which a plurality of satellite devices 440 (1) - 440 (n) act as partners and in this way can calculate depth information for a stage without a base station. However, because satellite devices are battery powered, use can be made of a light pattern from an external 406 projector (or more than one). Note that a base station is basically just another device, except generally that it is not mobile (a base station device can be relatively compact and portable but uses a large or fixed power source, and thus is not typically transported, to

ΙΜΡΙ

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX357307B_D0017.tif" />

unlike a tablet or smartphone type device).

As shown in Figure 4, each of the 440 (1) - 440 (n) satellite devices exchanges their data with one another. One of devices 440 (2) is shown having image processing 408 that processes its own image data 442 and other satellite device data 444 to obtain depth data 446 enhanced by one or more data from other data from satellite devices; It is understood that any satellite device may have similar image processing capabilities, and / or may receive depth data from another satellite device.

In another aspect, Figure 5 shows satellite devices 550 and 552 configured to provide a signal to flash an identifier at a base station 500 using infrared light. The identifier can include or is used to query any type of information, including device type, capabilities, etc. The signal can also help more efficiently determine the posture of the satellite device.

Note that instead of the base station identifying a satellite device, and / or estimating its posture, satellite devices can identify the base station, and estimate its own posture based on that information. This allows a satellite device to determine its posture without necessarily communicating with the base station, which can be useful in some situations (for example, if there is a relatively large number of satellite devices).

IMPI

MEXICAN INSTITUTE OF PROPERTY

INDUSTRIAL

<img file="MX357307B_D0018.tif" />

In another aspect, Figure 6A shows' how a satellite device that moves over time can calculate depth data with a single camera. Because the points are stationary, any movement of the device changes the positions of the points from the previous perspective to the new camera perspective.

Figure 6B shows two projectors 606 and 607 detected by a camera on a mobile satellite device. The projected patterns can be different, so the satellite device can distinguish one source from another, or the projectors can alternate which one is projected at one time to provide two different perspectives of the same object / scene being detected.

Certainly, a projector may have knowledge of the light pattern, and if so, it is basically a reverse camera. In this way, a single projector and a single camera on the satellite device can be used to calculate stereoscopic depth data.

Figure 7 is a simplified flow chart depicting an exemplary set of steps that can be performed to obtain enhanced depth information (and possibly other information such as RGB images) by a satellite device. For example, consider that an application program running on or attached to a base station wants to produce a detailed close-up display of an object that is relatively far from the base station and that is particularly not easy or desirable to zoom in further. . Although a base station is described in stages

<img file="MX357307B_D0019.tif" />

MtXICANO INSTITUTE OF PROPERTY. - x ... INDUSTRIAL following examples, it is understood that another mobile device or set of devices can be used in place of a base station.

In step 702, a base station program (or a program coupled to it) communicates a message to a user instructing the user to target a satellite device on the object to obtain certain video frames in the foreground. In step 704, the user does so, whereby the mobile device begins transmitting data (eg, an image stream or depth data processed therefrom) to the base station. In each frame, at step 706, the base station processes its own captured images to determine the position of the satellite camera, and calculate an initial depth map.

Step 708 represents receiving data from the satellite device; Note that at least some of these steps may occur in parallel and / or in different orders, eg some or all of the data may be received from the satellite device before, during, or after base station processing. Also note that instead of per frame, a certain slower sampling rate may be used depending on the application.

Step 710 represents using the satellite data to enhance the depth map with more accurate depth data, for example, and / or to maintain a separate depth map for the captured object. Step 712 repeats the process until it is done, at which time the user is notified.

The user can be asked to capture video of the object from

MEXICAN INSTITUTE

OF THE PROPERTY

INDUSTRIAL -— different directions. If the projected light is detected by the satellite device, the user can be instructed to rotate the object so that different parts of the object confront the projector at different times (note that the user cannot see the infrared light pattern, and thus it cannot know if it is projected, so the Instruction may be to rotate the object to confront the base station device). Alternatively, multiple projectors from different directions can be presented. In this way, a full three-dimensional mesh or the like of the object can be generated (with which RGB Image data can be combined) at a significantly higher level of detail than the base station is capable of capturing.

The satellite device and the base station (or other mobile device) can operate together without human intervention. For example, a mobile device application can detect something of interest using the device's camera and communicate the image data to the base station. Periodically or in some other program, the base station may request one or more images from a satellite device. The base station (for example, as Instructed by an application program) may wish to have an improved Image of something that (as known by the posture data) that the camera of the satellite device is targeting. For example, this may be to obtain a higher resolution image whenever desired.

As another example, the data from the satellite device can be used to replace data on the image map captured by a

MEXICAN INSTITUTE

DB PROPERTY

INDUSTRIAL ^ a¡_ device or satellite devices. As a particular example. consider that something blocks the perspective of a desired part of a base station stage, for example, a person passing in front of an interesting part of a stage looking to capture the satellite device. Using a satellite device or set of satellite devices, the stage can be captured (from a different angle and possibly with a different resolution), so the depths of the stage (and clean RGB or IR data) can be recalculated if the person no longer blocks that part of the stage.

ILLUSTRATIVE OPERATING ENVIRONMENT

Figure 8 illustrates an example of a suitable mobile device 800 in which aspects of the subject matter described herein can be implemented. Mobile device 800 is only an example of a device and is not intended to suggest any limitation on the scope of use or functionality of aspects of the subject matter described herein. The mobile device 800 should not be interpreted as having any dependency or requirement with respect to any or a combination of components illustrated in the exemplary mobile device 800.

Referring to Figure 8, an exemplary device for implementing aspects of the subject matter described herein includes a mobile device 800. In some embodiments, mobile device 800 comprises a cell phone, a device

IMPI

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX357307B_D0020.tif" />

portable that allows voice communication with others, some other voice communication device, or the like. In these embodiments, the mobile device 800 may be equipped with a camera for taking pictures, although this may not be required in other embodiments. In other embodiments, the mobile device 800 may comprise a personal digital assistant (PDA), a portable gaming device, a notebook computer, printer, apparatus including a cable TV box, media center, or other apparatus, other mobile devices, or the like. In still other embodiments, mobile device 800 may comprise devices that are generally considered non-mobile such as personal computers, servers, or the like.

The mobile device may comprise a portable remote control of an apparatus or toy, with additional circuitry to provide the control logic along with a way to input data to the remote control. For example, an input connector or other data receiving sensor may allow the device to overlap with uncontrolled code data transmission. This can be accomplished without the need to store much of the data to be transmitted, for example the device can act as a data relay for another device (possibly with some buffer memory), such as a smartphone.

Components of mobile device 800 may include, but are not limited to, a processing unit 805, system memory 810, and a common driver 815 that mates with various components of the

IMPI

O0TITUTO MEJICANO MUnonwAD

<img file="MX357307B_D0021.tif" />

system including 810 system memory a^orTa *¹ umeta processing 805. The 815 OS §l¿ common driver can be any of several types of common driver structures that include a common memory driver, memory controller, a peripheral common driver, and a local common driver using any of a variety of common conductor architectures, and the like. Common conductor 815 allows data to be transmitted between various components of mobile device 800.

Mobile device 800 can include a variety of computer readable media. Computer readable media can be any available media that can be accessed by the mobile device 800 and includes volatile and nonvolatile media, and removable and nonremovable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media Includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for information storage such as computer readable instructions, data structures, programming modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROMs, digital versatile discs (DVDs) or optical disc storage, magnetic cassettes, magnetic tape, magnetic disk store or other magnetic storage devices, or any other means that can be used

ΙΜΡΙβ ^

MEXICAN INSTITUTE

OWNERSHIP OawflKksffiF to store the desired information and • yneda^1N§ ^ c'é^ldersK²by the 800 mobile device. '

The communication media typically represent computer readable instructions, data structures, programming modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any means of information distribution. The term modulated data signal means a signal that has one or more of its characteristics set or changed in such a way that they encode information in the signal. By way of example, and not limitation, the communication medium includes wired means such as a wired network or direct cable connection, and wireless means such as acoustic means, RF, Bluetooth®, USB Wireless, infrared, Wi-Fi , WiMAX, and other wireless media. Combinations of any of the above must also be included within the scope of the computer readable medium.

The 810 system memory includes computer storage media in the form of volatile and / or nonvolatile memory and can include read-only memory (ROM) and random access memory (RAM). On a mobile device, such as a cell phone, the 820 operating system code is sometimes included in the ROM, although in other modes, it is not required. Similarly, 825 application programs are often placed in RAM, although again, in other modes, application programs may be placed in ROM or other memory readable by

IMPI

OBTRUTO MEUORÜ V¿ *! ÍsjrSS &amp; J} \ _x . . , _Λ «LAreonHM» computer. Dynamic storage at Q.30 prd * pw ^ tonafNTreffroria for the state associated with the operating systemΌ20 yte8^,,scheduleⁱ825 application s. For example, the 820 operating system and 825 application programs may store variables and data structures in heap 830 during their operations.

Mobile device 800 may also include other removable / non-removable, volatile / nonvolatile memory. As an example, Figure 8 illustrates a flash card 835, a hard drive 836, and a memory stick 837. Hard drive 836 may be tiny to fit in a memory slot, for example. The mobile device 800 can be interconnected with these types of non-volatile removable memory through an 831 removable memory interface, or it can be connected through a common universal cable (USB), IEEE 8394, one or more of the 840 wired ports, or antennas 865. In these embodiments, removable memory devices 835-437 can be interconnected with the mobile device through communication modules 832. In some embodiments, not all of these types of memory can be included on a single mobile device. In other embodiments, one or more of these and other types of removable memory may be included on a single mobile device.

In some embodiments, the 836 hard drive can be connected in such a way that it is more permanently attached to the mobile device 800. For example, the 836 hard drive can be connected to an interface such as a parallel advanced technology add-on ( PATA), serial advanced technology attachment

IMPI

MEXICAN INSTITUTE Dt THE PROPERTY

<img file="MX357307B_D0022.tif" />

INDUSTRIAL (SATA) or other, which can be connected to common conductor 815. In such modalities, removing the hard disk drive may involve removing a cover of the mobile device 800 and removing screws or other fasteners that connect the 836 hard disk drive to supporting structures within mobile device 800.

The 835-437 removable memory devices and their associated computer storage media, discussed above and illustrated in Figure 8, provide storage of computer-readable instructions, program modules, data structures, and other data for the device. mobile 800. For example, the removable memory device (s) 835-437 can store images taken by mobile device 800, voice recordings, contact information, programs, data for programs, etc.

A user can enter commands and information on mobile device 800 through input devices such as an 841 keyboard and 842 microphone. In some embodiments, display 843 may be a touch screen and may allow a user to enter commands. and information in it. The keyboard 841 and display 843 can be connected to the processing unit 805 through a user input interface 850 that is coupled to the common conductor 815, although it can also be connected by other interface and common conductor structures, such as modules communication 832 and wired ports 840. Motion detection 852 can be used to determine gestures made with the

ΙΜΡΙ

MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY

<img file="MX357307B_D0023.tif" />

device 800.

A user can communicate with other users through conversation at the 842 microphone and via text messages entered on the 841 keyboard or an 843 touch screen, for example. Audio unit 855 can provide electrical signals to drive speaker 844 as well as receive and digitize audio signals received from microphone 842.

Mobile device 800 may include a video unit 860 that provides signals to drive an 861 camera. Video unit 860 can also receive images obtained by camera 861 and provide these images to processing unit 805 and / or memory included in mobile device 800. Images obtained by camera 861 may comprise video, one or more images that do not form a video, or some combination thereof.

Communication modules 832 can provide signals to and receive signals from one or more antennas 865. One of antennas 865 can transmit and receive messages for a cellular telephone network. Another antenna can transmit and receive messages via Bluetooth®. Still another antenna (or a shared antenna) can transmit and receive network messages using a network standard of

Wireless ethernet

In addition, an antenna provides location-based information, for example, GPS signals to a GPS interface and the 872 mechanism. In turn, the GPS 872 mechanism makes available the corresponding GPS data (for example, weather and institute Mexican r- V ​​&amp; rjQ

OF THE «QUEDAD» C—

INDUSTRIAL coordinates) for processing.

In some embodiments, a simple antenna can be used to transmit and / or receive messages from more than one type of network. For example, a single antenna can transmit and receive voice and packet messages.

When operating in a network connected environment, mobile device 800 can connect to one or more remote devices. Remote devices can include a personal computer, a server, a router, a network PC, a cell phone, a media player, a peer-to-peer device, or another common network node, and typically includes many or all the elements described above with respect to the mobile device

800.

Aspects of the subject matter described herein are operational with numerous other general-purpose or special-purpose computer system settings or environments. Examples of well known computer systems, environments, and / or configurations that may be suitable for use with aspects of the subject matter described herein include, but are not limited to, personal computers, server computers, portable or laptop devices, microprocessor systems, microcontroller systems, cable tv boxes, programmable consumer electronics, network PCs, minicomputers, main computers, distributed computing environments that include any of the above systems or devices, and the like.

IMPI

<img file="MX357307B_D0024.tif" />

MIXICAN INSTITUTE D £ LA MOHEDAL Aj * ·. INEWRUAl

Aspects of the subject matter described herein may be described in the general context of computer-executable instructions, such as programming modules, that are executed by a mobile device. Generally, programming modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Aspects of the subject matter described herein may also be practiced in distributed computing environments whose tasks are performed by remote processing devices that are linked through a communication network. In a distributed computing environment, programming modules can be located on local and remote computer storage media that include memory storage devices.

Furthermore, although the term server may be used herein, it will be recognized that this term may also encompass a client, a set of one or more processes distributed on one or more computers, one or more autonomous storage devices, a set of one or more additional devices, a combination of one or more of the above, and the like.

CONCLUSION

Although the invention is susceptible of various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. It should be understood, however, that there is no intention to

<img file="MX357307B_D0025.tif" />

IMPI

DELA PKWIEVAU MEXICAN INSTITUTE

INDUSTRIAL___ limit the Invention to the specific forms of S ^ -A¿a ^, _ on the contrary, the intention is to cover all modifications, alternative constructions and equivalents that fall within the spirit and scope of the Invention.