Active stereo with adaptive support weights from a separate image

In active depth sensing, as used by active stereo systems, the projector projects a pattern of light, such as infrared (IR) dots or IR lines, to illuminate the perceived scene. Note that the projector can use a laser that consumes about 1 W of power. This means that because the projector consumes a lot of power, it is almost impossible to incorporate it into a small satellite device such as a smartphone or tablet instead of making it part of a device that is typically plugged into a wall outlet. To do.

The projected pattern is then captured by a camera / sensor (two or more in a stereo system), along with an image (s) that are processed to calculate a depth map or something similar. For example, in a stereo system, a stereo camera captures two images from different perspectives. Then, for example, one way to perform depth estimation with stereo pairs of images is to find the correspondence between the images, eg correlate the projected and sensed dots in the left image with the corresponding dots in the right image. To let. When matched, the projected patterns in both images are correlated with each other, and if possible, using triangulation with differences between one or more features (including, for example, intensity) of the correlated dots. , The depth to the object illuminated by that particular projected dot can be estimated.

In most scenarios, the camera that captures the stereo image is arranged to sense a relatively large space, such as a room, and can be relatively large. As a result, distant objects such as faces appear only in a relatively small number of camera pixels. Therefore, sufficient detail and / or accurate depth estimation is not available as required by many applications such as face recognition.

The description in this "Summary of the Invention" is provided to briefly introduce the selection of representative concepts further described in the "Modes for Carrying Out the Invention" below. The description in this "Summary of the Invention" is not intended to identify the main features or essential features of the claimed subject matter, and is a method that limits the scope of the claimed subject matter. It is also not intended to be used in.

Briefly, one or more of the various aspects of the subject matter described herein are depths calculated by another device, such as a base station, using information from a satellite (eg, mobile) device. The target is to enhance the data. One or more aspects receive image-related data from a satellite device on another device and are first of depth data based on this image-related data and the pose information of the satellite device, at least in part. The target is to strengthen the set.

In one or more embodiments, the base station is configured to determine the pose information of the satellite device and capture one or more images. The base station receives image-related data from the satellite device and calculates a depth map based on pose information and image-related data, at least in part.

One or more aspects are intended to receive image-related data from a satellite device at a base station and capture an actively illuminated stereo image at the base station. The pose information corresponding to the satellite device is determined. The actively illuminated stereo image is processed from the base station's point of view to determine a depth map enhanced by the depth information corresponding to the image-related data.

Other advantages may be apparent from the following detailed description when considering the following detailed description in connection with the drawings.

The present invention is illustrated in the accompanying drawings as an example and not a limitation. In the drawings, similar reference numbers indicate similar elements.<figref num="1">It is a diagram showing a base station projecting an optical pattern onto a scene according to one or more exemplary implementations, showing how a satellite device captures at least a portion of the scene.</figref><figref num="2">It is a block diagram representing an exemplary component that can be used to project and capture an image for processing into depth data according to one or more exemplary implementations.</figref><figref num="3">It is a block diagram showing an example of processing base station data and satellite data into depth data according to one or more exemplary implementations.</figref><figref num="4">FIG. 6 is a block diagram illustrating an example of processing data from one satellite device and data from another satellite device into depth data according to one or more exemplary implementations.</figref><figref num="5">FIG. 5 represents a satellite device configured to signal a base station according to one or more exemplary implementations.</figref><figref num="6A">FIG. 5 illustrates how depth data can be determined from a single satellite device camera according to one or more exemplary implementations.</figref><figref num="6B">FIG. 5 illustrates how depth data can be determined from a single satellite device camera according to one or more exemplary implementations.</figref><figref num="7">FIG. 5 is a flow diagram illustrating an exemplary step of acquiring image-related data from a satellite device for use in determining a depth map according to one or more exemplary implementations.</figref><figref num="8">FIG. 6 is a block diagram representing an exemplary non-limiting computing system or operating environment in the form of a mobile device, which can implement one or more aspects of the various embodiments described herein.</figref>

Various aspects of the techniques described herein generally use satellite (eg, mobile) devices such as smartphones and tablet computers to improve, for example, depth sensing, provide further image details, and /. Alternatively, it is intended to communicate information with an active lighting base station for estimating a pose or the like. At the same time, the satellite device utilizes the active illumination of the base station for the active sensing of the satellite device itself, so that the satellite device does not need to consume battery power to project the light pattern.

In one or more alternatives, multiple satellite devices can use projected light from external sources to communicate image / depth / position data calculated from them with each other. Other useful data about depth and scene can be made available to these devices without the need for a fixed base station.

It should be understood that none of the examples herein are non-limiting. For example, satellite devices are generally exemplified as mobile herein, but one (or more) auxiliary device is placed as a "fixed satellite" that can be calibrated with a base station if desired. May be done. This can be used, for example, to provide a camera closer to some prominent part of the scene. Therefore, as used herein, the term "satellite" need not be mobile and may include any auxiliary device. As another example, a time-of-flight camera may be used to determine at least some data. Thus, the invention is not limited to any particular embodiment, aspect, concept, structure, function or example described herein. Rather, any of the embodiments, embodiments, concepts, structures, functions or examples described herein are non-limiting, and the present invention provides the advantages and benefits of active depth sensing and image capture and general processing. It can be used in various ways to provide an effect.

FIG. 1 illustrates an implementation in which base station 100 includes stereo (eg, infrared) cameras 102 and 103 that capture images of scene 104 from different viewpoints. The projector 106 illuminates the scene 104 with a light pattern (eg, about 100,000 points). The light pattern may be a pseudo-random number and can be generated by emitting laser light through a diffracting light element. Instead, line patterns, spots of other shapes, etc. may be generated and perceived, but for the sake of brevity, the points / components of the radiated pattern are commonly referred to as dots.

Cameras 102 and 103 capture the dots as they are reflected from the surface of the object in the scene 222 and (possibly) the background. Dots generally sensed by one camera (eg 102) are correlated with dots sensed by another camera (eg 103) via image processing 108 (eg system or subsystem), thereby making each dot The distance to the incident reflective surface can be provided (eg by triangulation). Note that Figure 1 is not intended to scale for any size, distance, dot distribution pattern, or dot density, nor is it intended to show these.

Base station 100 may also include clean IR cameras, such as IR cameras with filters that block active lighting patterns and / or RGB cameras that capture visible light, if both are present. The cameras may be combined into a single camera. Camera 112 in FIG. 1 represents one of these alternative cameras.

As shown in FIG. 1, one of the objects sensed in the scene 104 can be a satellite device 110, such as a mobile smartphone or tablet computing device. At the same time, the satellite device may be sensing other objects in the scene, eg, including its depth, which, for example, senses a lighting pattern and, in some cases, its clean IR and / or RGB image. It is done through a stereo IR camera or the like that senses.

FIG. 2 shows an exemplary base station 100, within which the stereo cameras 102 and 103 of the image capture system or subsystem 224 capture time-synchronized images (eg, these cameras It is "genlocked"). In one implementation, the camera captures infrared (IR) images because IR does not affect the visual appearance of the scene (which is very advantageous, for example in video conferencing and object modeling applications). Is). As is easily recognized, in some scenarios such as studio environments, there may be more than two IR depth sensitive cameras. In addition, one or more other cameras, such as RGB cameras, may be present in a given system and such other cameras may be used to align the images, eg, in different stereo images. It can help correlate pairs of dots, etc.

In FIG. 2, the projector 106 projects an IR pattern onto the scene. By illuminating the scene with a relatively large number of dispersed infrared dots, the cameras 102 and 103 can capture additional texture data as part of the infrared image data. The projector 106 may be placed outside the camera (eg, FIG. 2), between the cameras (eg, FIG. 1), or in another, such as above or below both or one camera. Note that it may be a position. The examples in this description do not limit where the cameras 102, 103 and / or the projector 106 are located relative to each other, and the cameras may be located at different positions relative to each other.

In one implementation, the exemplary image capture system or subsystem 224 includes a controller 228 that controls the operation of cameras 102 and 103 via camera interface 230 and, if present, also the operation of camera 112. The illustrated controller also controls the operation of the projector 106 via the projector interface 232. For example, the cameras 102 and 103 are synchronized (genlocked) so as to capture stereo images at the same time, for example, by a signal of one controller (or a signal different for each camera). Projector 106 may have one or more parameters that vary, eg, turn on or off, pulsed, or otherwise controllably.

Images captured by cameras 102 and 103 (and cameras 112, if any) are included in an image processing 108 system or subsystem with logic implemented in hardware and / or software (eg, as computer-readable instructions). Provided. In some implementations, the image processing system 108 and the image capture system or subsystem 224, or some of them, may be combined into a single device. For example, the home entertainment device may include all of the components shown in FIG. 1 (as well as components not shown). In other implementations, some (or all) of the image capture system or subsystem 224, such as cameras and projectors, is coupled to a game console, personal computer, satellite device, dedicated processing device and / or similar. It may be a device.

In one or more implementations, the image processing system or subsystem 108 includes a processor 240 and a memory 242 that includes one or more image processing algorithms 244. One or more depth maps 246 can be obtained by algorithm 124, for example by extracting matching features (such as dots and / or lines). For example, as is known, for example, as described in US Patent Application Publication No. 2013 / 0,100,256, different dots or other projected elements, when captured, are the distance from the projector to the reflective surface and / or It has different characteristics including intensity (brightness) depending on the distance from the camera to the reflecting surface. As is also known, dots in different images taken at the same time (eg by a genlocked stereo camera) are small (eg RGB) patches between RGB images of the same scene captured at the same moment. Can be correlated with each other by matching. Therefore, with captured images, known algorithms use the triangulation / difference of specific features between matched dots to determine the depth, and the projected light components (eg dots) within each image. By matching, the features (depth map) associated with each depth can be determined. This is one way the depth map can be obtained via stereo image processing.

Also shown in Figure 1 are for keyboards, game controllers, displays, voice commands, etc., as appropriate for the user to interact with one or more interfaces to the base station, such as applications that use depth maps. Interfaces for connecting pointing device microphones and / or similar are shown. At least one interface 132 allows the satellite device 110 to communicate (eg, wirelessly) with the base station as described herein.

One or more implementations include an indoor base station 100 that emits a laser pattern to calculate depth. When the user uses the satellite device 110 such as a smartphone or tablet, the camera or stereo camera on the satellite device 110 can observe the radiation laser pattern of the base unit (base station). The satellite device 110 can therefore calculate stereo based on its externally generated laser pattern and further communicate information to base station 100.

As described herein, satellite device data can be used to generate depth maps at a higher resolution than can be calculated by the base station (as used herein). When "high resolution" does not refer to the resolution of the camera itself, it refers to the object with more pixels than the distant camera on the base station because the satellite device is closer to the object, eg the user's face. Note that it refers to the ability to capture). In addition, not only the spatial resolution can be improved, but also the depth resolution / accuracy can be improved. Image information or data corresponding to the information (eg, depth maps calculated by satellite devices) may be transmitted to the base unit. As used herein, "image-related data" means (in whole or in part, one or more IR and / RGB actively illuminated images and / or clean IR and / or RGB. Refers to any corresponding information processed from the actual image data (of the image), any associated metadata and / or image data such as a depth map. Therefore, image-related data can be communicated to and from base stations, and to and from satellite devices.

In addition, images and / or depth maps taken by satellite devices and base units can be correlated. It provides six degrees of freedom (6DoF) for pose estimation of the mobile unit's position with respect to the base unit.

Pose estimation can be determined by image alignment and / or pattern matching / dot correlation. As an example, if a satellite device captures an image that the base station can also sense (eg, the user or device does not block the base station from "seeing" the same area), then the pixels (or each pixel) Corresponding surrounding patches) can be matched in a known way. Alignment can also be established by aligning the depth data calculated from the base station with the depth data calculated from the satellite device. Therefore, the pose of the satellite device can be estimated.

This is generally shown in FIG. 3, in which the base station 300 projects a light pattern from the projector 306, which light patterns are emitted by the base station cameras 301 and 203, as well as the satellite device 330. Detected by one or more cameras above (there may be more than one satellite device). Cameras 301 and 302 provide base station image 332 to be supplied for image processing along with satellite device data 334. The satellite device data 334 can be an image or information such as a depth map that is processed locally from the image on each satellite device.

Optional (eg, low power) projectors may be included in any or all of the one or more satellite devices, as indicated by the dashed line from device 330. The data projected by the satellite device emphasizes the data projected from the base station. The projector in satellite device 330 is distance-limited, spatially limited (eg, very sparse pattern or focused only in a small image area) and / or time-limited (eg,). It can be low power because it is radiated only every few frames).

Base station image 332 can be processed to determine the pose / 3D position of satellite device 330. Based on this information, image processing 308 outputs depth data 336. The depth data 336 may be conventional depth data from the camera of the base station itself, enhanced by satellite device data 334. In general, projected dots captured by a base station camera can be correlated with dots captured by a satellite device after adjusting for size differences and the like due to differences in resolution. The pose data 338 may be output to, for example, another satellite device.

Note that instead of the satellite device providing the base station with data to enhance its image data, the base station may send an approximate depth map (along with the device pause) to the satellite device. .. This can be used to improve accuracy and / or reduce the calculations required for depth estimation of the satellite device itself.

Base station depth maps (as well as RGB data) can be enhanced by downsampling high resolution data into partial depth maps. The partial depth map is combined with the depth map of the initial base station at the appropriate coordinates, and the accuracy at these coordinates can be improved, for example, after adjusting the depth for the pose. Another way a base station depth map can be enhanced is, for example, to keep the original depth map associated with a higher resolution depth map (eg after adjusting the depth based on the pose). In this way, applications that use depth maps can "zoom in" to finer-grained depth data when they wish, for example, applications need to be limited to the pixel resolution of the original depth map. Absent.

Note that some mobile devices have front and rear cameras. In such cases, the camera will be used for different purposes in some scenarios. For example, a rear camera may be used to capture an image for pose calculation, while a front camera may capture an image of the user's face, eg, an image that can be used for enhanced depth calculation.

FIG. 4 shows another alternative form, in which multiple satellite devices 440 (1)-440 (n) act as peers and thus the depth about the scene without the need for a base station. Information can be calculated. However, since satellite devices are battery powered, light patterns from one (or more than one) external projector 406 can be utilized. It should be noted that a base station is basically just another device, except that it is generally not mobile (base stations are relatively compact and portable, but large or fixed power supplies. Because it uses, unlike tablet devices and smartphones, it is typically not carried).

As shown in FIG. 4, each of the satellite devices 440 (1) to 440 (n) exchanges its data with each other. One of the devices 440 (2) is shown to have image processing 408, which processes the image data 442 of the device 440 (2) itself and the data 444 of other satellite devices. , Acquire depth data 446 enhanced by data from one or more other satellite devices. As will be appreciated, any satellite device may have similar image processing capabilities and / or may receive depth data from another satellite device.

In another aspect, FIG. 5 shows satellite devices 550 and 552 configured to provide signals, for example by flashing an identifier to base station 500 via infrared light. The identifier may include any type of information, including device type, capabilities, etc., or may be used to look up such any type of information. The signal can also help in more effectively determining the pose of the satellite device.

Note that instead of the base station identifying the satellite device and / or estimating its pose, the satellite device may identify the base station and estimate its pose based on that information. This allows satellite devices to determine their pose without having to communicate with the base station, which can be beneficial in some situations (eg when there are a relatively large number of satellite devices). obtain.

In another aspect, FIG. 6A shows how a satellite device moving over time can calculate depth data with a single camera. Since the dots do not move, any movement of the device will change the position of the dots from the previous viewpoint of the camera to the new viewpoint.

Figure 6B shows two projectors 606 and 607 sensed by one camera on a satellite mobile device. The projected patterns are different, which allows satellite devices to distinguish one source from another, or these projectors alternate so that one projector is perceived at the same time. It can be projected to provide two different perspectives on the object / scene.

In fact, the projector may have knowledge of light patterns, in which case it is basically a camera, on the contrary. Therefore, a single projector and a single satellite device camera can be used to calculate stereo depth data.

FIG. 7 is a simplified set of examples of steps that can be taken to obtain enhanced depth information (and possibly other information such as RGB images) via a satellite device. It is a flow chart. For example, a detailed close-up of an application program that runs on or is coupled to a base station is relatively far from the base station and is not particularly easy or desirable to move closer to. Consider a situation where you want to generate an up representation. The base station will be described below in the exemplary steps, but as will be appreciated, another mobile device or set of devices may be used instead of the base station.

In step 702, a base station program (or a program associated with it) communicates a message instructing the user to point the satellite device at the object in order to obtain some close-up video frames. To do. At step 704, the user then initiates transmission of data to the base station (eg, a stream of images or depth data to be processed from it). In each frame, in step 706, the base station processes the captured image of the base station itself to determine the pose of the satellite camera and calculate the initial depth map.

Step 708 represents receiving data from satellite devices, but at least some of these steps may occur in parallel and / or in a different order, eg, prior to base station processing. Note that some or all of the data may be received from the satellite device during or after processing. Also note that depending on the application, slower sampling rates may be used rather than frame by frame.

Step 710 represents using satellite data to enhance the depth map, for example with more accurate depth data, and / or maintain a separate depth map for the captured object. Step 712 repeats the process until the user is notified.

The user may be required to capture the video of the object from different directions. If the projected light is sensed by a satellite device, the user may be instructed to rotate the object so that different parts of the object face the projector at different times (users can instruct the pattern of infrared light). Note that you may not know that this is projected because you cannot see it, so the instructions can cause the object to rotate to face the base station device). Alternatively, there may be a plurality of projectors from different directions. In this way, a complete 3D mesh or similar about the object can be generated at a level of detail that is much higher than the level that the base station can capture (RGB image data combined with this). May be).

Satellite devices and base stations (or other mobile devices) may operate together without human intervention. For example, a mobile device application may detect something of interest via a device camera and communicate image data to a base station. On a periodic or other schedule, the base station may request one or more images from the satellite device. A base station may want to have an improved image of something that the camera of a satellite device (as known via pose data) is pointing at (as directed by an application program, for example). This, for example, allows you to get a higher resolution image whenever you want.

As another example, data from satellite devices may be used to replace the data in images captured by one or more satellite devices. As a concrete example, something is blocking the desired part of the scene from the base station's point of view, for example, a person walking in front of an interesting part of the scene that a satellite device is trying to capture. Think. A satellite device or a set of satellite devices is used to capture a scene (from different angles and, in some cases, at different resolutions), which makes it as if a person is not blocking that part of the scene. (And RGB or clean IR data) can be recalculated.

&lt;Exemplary Operating Environment&gt; FIG. 8 illustrates an example of a suitable mobile device 800 in which the aspects of the subject matter described herein can be implemented. The mobile device 800 is merely an example of a device and is not intended to imply any limitation with respect to the use or scope of function of the aspects of the subject matter described herein. Nor should the mobile device 800 be construed as having any dependency or requirement with respect to any one or combination of the components illustrated within the illustrated mobile device 800.

With reference to FIG. 8, exemplary devices for implementing aspects of the subject matter described herein include the mobile device 800. In some embodiments, the mobile device 800 comprises a mobile phone, a handheld device that enables voice communication to others, or some other voice communication device. In these embodiments, the mobile device 800 can be equipped with a camera for taking pictures, but in other embodiments this is not always required. In other embodiments, the mobile device 800 may include a personal digital assistant (PDA), a handheld gaming device, a notebook computer, a printer, a device including a set top, a media center or other device or other mobile device, and the like. Good. In yet another embodiment, the mobile device 800 may include devices generally considered non-mobile, such as personal computers and servers.

The mobile device may include a handheld remote control of a device or toy that has additional circuitry that provides control logic as well as a way to enter data into the remote control. For example, an input jack or other data receiving sensor allows the device to be repurposed for uncontrolled code data transmission. This can be achieved without having to store much of the data to be transmitted, for example the device acts as a data relay to another device (possibly with some buffering), such as a smartphone. Can be.

The components of the mobile device 800 may include, but are not limited to, a processing unit 805, a system memory 810, and a bus 815 that connects various system components to the processing unit 805, including the system memory 810. Bus 815 may include any of several types of bus structures, including memory buses, memory controllers, peripheral buses and local buses that use any of the various bus architectures. Bus 815 allows data to be transmitted between various components of the mobile device 800.

The mobile device 800 may include various computer readable media. The computer-readable medium can be any available medium accessible by the mobile device 800, including both volatile and non-volatile media and removable and non-removable media. As an example, but not limited to, a computer-readable medium may include a computer storage medium and a communication medium. Computer storage media are volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules or other data. Including. Computer storage media are, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic. Includes disk storage or other magnetic storage device, or any other medium that can be used to store desired information and is accessible by the mobile device 800.

The communication medium typically embodies computer-readable instructions, data structures, program modules or other data into modulated data signals such as carrier waves or other transmission mechanisms and includes any information transmission medium. The term "modulated data signal" means a signal that has one or more of the characteristics set or modified in such a way as to encode information within the signal. As an example, but not limited to, communication media include wired media such as wired networks and direct wired connections, and wireless media such as audio, RF, Bluetooth®, wireless USB, infrared, Wi-Fi, WiMAX and other wireless media. including. Any combination of the above should also be included within the scope of computer readable media.

System memory 810 includes computer storage media in the form of volatile and / or non-volatile memory, and may also include read-only memory (ROM) and random access memory (RAM). In mobile devices such as mobile phones, operating system code 820 is sometimes included in ROM, but in other embodiments this is not required. Similarly, the application program 825 is often located in RAM, but again, in other embodiments, the application program may be located in ROM or other computer-readable memory. Heap 830 provides memory with the state associated with operating system 820 and application program 825. For example, operating system 820 and application program 825 may store variables and data structures in heap 830 during their operation.

The mobile device 800 may include other removable / non-removable, volatile / non-volatile memory. As an example, FIG. 8 illustrates a flash card 835, a hard disk drive 836 and a memory stick 837. The hard disk drive 836 can be miniaturized to fit, for example, a memory slot. The mobile device 800 may interface with these types of non-volatile removable media via a removable memory interface 831 or one or more of a universal serial bus (USB), IEEE 8394, wired port 840 or Can be connected via antenna 865. In these embodiments, the non-removable memory devices 835-837 can interface with the mobile device via the communication module 832. In some embodiments, not all of these types of memory need to be contained in a single mobile device. In other embodiments, one or more of these and other types of removable memory may be included in a single mobile device.

In some embodiments, the hard disk drive 836 may be connected in such a way that it is permanently attached by the mobile device 800. For example, hard disk drive 836 is connected to an interface that can be connected to Parallel ATA (PATA), Serial ATA (SATA) or other method device 815. In such embodiments, removing the hard drive may require removing the cover of the mobile device 800 and removing the screws or other fasteners that connect the hard drive 836 to the support structure within the mobile device 800. ..

The removable memory devices 835-837 and their associated computer storage media described above and illustrated in FIG. 8 provide storage for computer readable instructions, program modules, data structures and other data for the mobile device 800. .. For example, one or more removable memory devices 835-837 may store images, audio recordings, contact information, programs, program data, etc. taken by the mobile device 800.

The user can input commands and information into the mobile device 800 via input devices such as the keypad 841 and the microphone 842. In some embodiments, the display 843 may be a touch-sensitive screen, which can allow the user to enter commands and information on it. The keypad 841 and display 843 may be connected to the processing unit 805 via a user input interface 850 coupled to bus 815, but may be connected by other interfaces and bus structures such as communication module 832 and wired port 840. .. Motion detection 852 can be used to determine gestures made with device 800.

The user may communicate with other users, for example, by speaking to the microphone 842 and by text messages entered on the keypad 841 or the touch-sensitive display 843. The audio unit 855 can provide an electronic signal for driving the speaker 844, receive an audio signal from the microphone 842, and digitize the received audio signal.

The mobile device 800 may include a video unit 860 that provides a signal to drive the camera 861. The video unit 860 may also receive images acquired by the camera 861 and provide these images to memory contained in the processing unit 805 and / or mobile device 800. The images acquired by the camera 861 may include video, one or more images that do not form video, or any combination thereof.

The communication module 832 provides signals to one or more antennas 865 and may receive signals from one or more antennas 865. One of the antennas 865 can send and receive messages for mobile network. Another antenna can send and receive Bluetooth® messages. Yet another antenna (or shared antenna) may send and receive network messages over a wireless Ethernet® network standard.

Furthermore, the antenna provides position-based information, such as GPS signals, to the GPS interface and mechanism 872. GPS mechanism 872 then makes the corresponding GPS data (eg, time and coordinates) available for processing.

In some embodiments, a single antenna may be used to send and / or receive messages for more than one type of network. For example, a single antenna can send and receive voice and packet messages.

When operated in a network environment, the mobile device 800 may connect to one or more remote devices. Remote devices may include personal computers, servers, routers, network PCs, mobile phones, media playback devices, peer devices or other common network nodes, typically described above in connection with the mobile device 800. Includes many or all of such elements.

Aspects of the subject matter described herein operate in a number of other general purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations that may be suitable for use with aspects of the subject matter described herein are, but are not limited to, personal computers, server computers, handhelds or Distributed computing environments including laptop devices, multiprocessor systems, multicontroller based systems, settop boxes, programmable home appliances, networked PCs, minicomputers, mainframe computers, any of the above systems or devices and the like. Includes.

Aspects of the subject matter described herein can be described in the general context of computer executable instructions executed by mobile devices, such as program modules. In general, a program module includes routines, programs, objects, components, data structures, etc. that perform a particular task or implement a particular abstract data type. Aspects of the subject matter described herein can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network. In a distributed computing environment, program modules may be located on both local and remote storage media, including memory storage devices.

In addition, the term server may be used herein, but the term is a client, a set of one or more processes distributed on one or more computers, one or more stand-alone storage devices, one. It may also include a set of the above other devices, a combination of one or more of the above, and the like.

&lt;Conclusion&gt; The present invention allows various modifications and alternative configurations, but specific exemplary embodiments thereof have been illustrated in the drawings and described in detail above. However, it is understood that the invention is not intended to be limited to the particular form disclosed, but conversely intended to cover all modifications, alternative configurations and equivalents within the spirit and scope of the invention. I want to be.