Wireless wrist computing and control device and method for 3D imaging, mapping, networking and interfacing

CROSS REFERENCE TO RELATED APPLICATIONS

0001This application claims priority to provisional application Ser. No. 61/796,056, filed Nov. 1, 2012, which application is incorporated herein in its entirety by this reference thereto.

TECHNICAL FIELD

0002This invention relates to the field of wearable computing. More particularly, the invention relates to devices and methods for 3D mapping, imaging, networking, communications, and multi-interface remote controlling.

DESCRIPTION OF RELATED ART

0003Existing control devices include the early mouse technology in the form of hand-held wired x-y positional input devices, such as found in U.S. Pat. No. 3,541,541, to sensor and spatial positioning systems such as U.S. Pat. No. 6,005,548, to wearable optical hand, finger and object spatial positioning systems that incorporate gesture and voice recognition and touchscreen interfacing controls.

0004Prior art such as U.S. Pat. No. 6,647,632 introduced a wireless control device worn on the wrist with light emitters and sensors placed on the inside of the hand to identify the position of the users hand and fingers, recognize pre-assigned gestures and voice commands and relay the data to a controlled device and U.S. Pat. No. 8,292,833 B2 introduced a wrist worn Finger Motion Detecting Apparatus that uses optical and ultrasonic wave signal monitoring of the wearers tendons to identify the position and movement of their hand and fingers and relay data to a controlled device. U.S. Patent Application 2009/0096783 A1 introduces an indoor three dimensional structured imaging system and body motion and gesture interfacing system using a light speckle pattern to 3D map illuminated objects and U.S. Patent Application 2011/0025827 introduces stereoscopic depth mapping using a combination of light projection and 3D color imaging, both systems are limited to depth mapping, modeling and interfacing from a fixed location.

0005A common attribute of the mouse and other handheld and wearable interfacing devices is the definition of the controllers being peripheral devices, and a positional data input accessories to remote controlled devices and computing systems. Therefore, there are many problems with the known with existing technology.

SUMMARY OF THE INVENTION

0006An apparatus and method for light and optical depth mapping, 3D imaging, modeling, networking, and interfacing on an autonomous, intelligent, wearable wireless wrist computing, display and control system for onboard and remote device and graphic user interface control. Embodiments of the invention enable augmentation of people, objects, devices, and spaces into a virtual environment and augmentation of virtual objects and interfaces into the physical world through its wireless multimedia streaming and multi-interface display and projection systems.

BRIEF DESCRIPTION OF THE DRAWINGS

0007<figref idref="DRAWINGS">FIG. 1A</figref> is a perspective view of an embodiment of the wrist console.

0008<figref idref="DRAWINGS">FIG. 1B</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref>.

0009<figref idref="DRAWINGS">FIG. 1C</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref>.

0010<figref idref="DRAWINGS">FIG. 1D</figref> is a top plan view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref>.

0011<figref idref="DRAWINGS">FIG. 2A</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting an expansion module.

0012<figref idref="DRAWINGS">FIG. 2B</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting the detached expansion module.

0013<figref idref="DRAWINGS">FIG. 2C</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting an expansion.

0014<figref idref="DRAWINGS">FIG. 2D</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> with the expansion module detached.

0015<figref idref="DRAWINGS">FIGS. 3A and 3B</figref> are perspective views illustrating an embodiment of the wrist console that incorporates one or more moving beams of light.

0016<figref idref="DRAWINGS">FIGS. 3C and 3D</figref> are perspective views depicting an embodiment of the invention that incorporates structured light imaging.

0017<figref idref="DRAWINGS">FIGS. 3E and 3F</figref> are perspective views illustrating a wrist console generating a depth and color mapped hand.

0018<figref idref="DRAWINGS">FIGS. 3G and 3H</figref> are perspective views illustrating a wrist console identifying the precise position of joints and creases of the hand.

0019<figref idref="DRAWINGS">FIGS. 3I and 3J</figref> are perspective views illustrating a wrist console generating a functional rigging of the hand.

0020<figref idref="DRAWINGS">FIGS. 3K and 3L</figref> are perspective views illustrating an embodiment incorporating a hand rig into the 3D mapped hand.

0021<figref idref="DRAWINGS">FIGS. 4A and 4B</figref> are perspective views illustrating the user's hand and fingers used to combine a gesture with a motion.

0022<figref idref="DRAWINGS">FIG. 4C</figref> is a perspective view illustrating the user using performing and selecting a gesture interface control function that involves multiple fingers.

0023<figref idref="DRAWINGS">FIG. 4D</figref> is a perspective view illustrating the user performing and selecting a gesture interface control function that involves one finger.

0024<figref idref="DRAWINGS">FIG. 4E</figref> is a perspective view illustrating the user performing and selecting a gesture interface control that involves touching a specified finger to an identified point or area on the hand.

0025<figref idref="DRAWINGS">FIG. 4F</figref> is a perspective view illustrating the user performing and selecting a gesture interface control function that involves touching one specified finger to another specified finger.

0026<figref idref="DRAWINGS">FIG. 5A</figref> is a perspective view illustrating a single point of control for a 2D or 3D computing environment on an external networked device using a single finger as a controller.

0027<figref idref="DRAWINGS">FIG. 5B</figref> is a perspective view illustrating multiple points of control for a 2D or 3D computing environment on an external networked device using multiple fingers as controllers.

0028<figref idref="DRAWINGS">FIG. 6A</figref> is a perspective view illustrating a wrist console projecting a graphic user interface (GUI) onto the user's hand and fingers.

0029<figref idref="DRAWINGS">FIG. 6B</figref> is a perspective view illustrating a user performing touch and gesture interfacing to control a projected GUI on the user's hand and fingers.

0030<figref idref="DRAWINGS">FIG. 6C</figref> is a perspective view illustrating a wrist console projecting a graphic user interface onto the users hands and fingers.

0031<figref idref="DRAWINGS">FIG. 6D</figref> is a perspective view illustrating the user typing on a projected keypad on the users hand.

0032<figref idref="DRAWINGS">FIG. 7</figref> is a perspective view illustrating a user typing on a projected keyboard mapped on to an external surface by both a left and right wrist console

0033<figref idref="DRAWINGS">FIG. 8A</figref> is a perspective view illustrating the user interfacing with two networked devices and using a wrist console to map, image and model a scanned physical object into a virtual computing environment on an external networked device.

0034<figref idref="DRAWINGS">FIG. 8B</figref> is a perspective view illustrating the user operating a left and right wrist console as a two hand gesture interface controller to interface and manipulate a 3D computer model scanned and modeled by the wrist console and wirelessly uploaded to an external networked device.

0035<figref idref="DRAWINGS">FIG. 8C</figref> is a perspective view illustrating the user selecting a file, document or program on one external networked device.

0036<figref idref="DRAWINGS">FIG. 8D</figref> is a perspective view illustrating the user operating and controlling a 3D scanned and modeled object on a wrist console touchscreen interface.

0037<figref idref="DRAWINGS">FIG. 9A</figref> is a perspective view illustrating a user wirelessly interfacing and controlling a remote device or vehicle.

0038<figref idref="DRAWINGS">FIG. 9B</figref> is a perspective view illustrating a user wirelessly sending and receiving real-time voice, data, video and multimedia content to and from a remote device.

0039<figref idref="DRAWINGS">FIGS. 10A and 10B</figref> are perspective views illustrating a wrist console full body scanning, depth mapping and imaging process where a user performs a body scan and a 3D computer model of the user is generated.

0040<figref idref="DRAWINGS">FIGS. 10C and 10D</figref> are perspective views illustrating the user in different positions and performing different body motions enabling the wrist console to map and image the body in multiple positions and analyze the flexibility and mobility of the user to more accurately generate the body rigging for the 3D computer model and replicate the users motions in a virtual computing environment.

0041<figref idref="DRAWINGS">FIGS. 10E and 10F</figref> are perspective views illustrating the 3D mapping and imaging of the user and a 3D computer model with clothing.

0042<b>11</b>A and <b>11</b>B are perspective views depicting the camera view from a top module of a wrist console.

0043<figref idref="DRAWINGS">FIGS. 11C and 11D</figref> are perspective views depicting the camera view from a bottom module of a wrist console.

0044<figref idref="DRAWINGS">FIGS. 12A and 12B</figref> are perspective views depicting the camera view from a top module of a wrist console.

0045<figref idref="DRAWINGS">FIGS. 12C and 12D</figref> are perspective views depicting the camera view from a bottom module of a wrist console.

0046<figref idref="DRAWINGS">FIGS. 13A-13C</figref> are perspective views illustrating the body rigging process.

0047<figref idref="DRAWINGS">FIGS. 14A and 14B</figref> are perspective views illustrating the users 3D computer model spatial position and location in a mapped physical environment.

0048<figref idref="DRAWINGS">FIG. 15</figref> is a perspective view illustrating a virtual internal body map of a user's anatomy and networked, sensors, implanted devices and prosthetics

0049<figref idref="DRAWINGS">FIGS. 16A and 16B</figref> are perspective views illustrating a wrist console space, object and environment 3D light and image mapping process.

0050<figref idref="DRAWINGS">FIGS. 17A and 17B</figref> are perspective views of the user standing in a residential living space.

0051<figref idref="DRAWINGS">FIG. 18</figref> is an overhead perspective view of the user and a wrist console identifying and mapping the location, device type, functions and applications, data, power and other system specifications and available networks and interfacing options for all networked devices in the in the residential living space.

0052<figref idref="DRAWINGS">FIGS. 19A and 19B</figref> are perspective views of a professional tennis player wearing a wrist console.

0053<figref idref="DRAWINGS">FIGS. 20A</figref>, <b>20</b>B, and <b>20</b>C are perspective views of a professional golfer mapping his swing, the ball, and virtualizing the player's entire game of golf in real time.

0054<figref idref="DRAWINGS">FIG. 21</figref> is a perspective view illustrating a professional golfer on a golf course.

0055<figref idref="DRAWINGS">FIG. 22</figref> is a schematic diagram of a wrist console.

DETAILED DESCRIPTION

0056Apparatus and methods are described for autonomous, intelligent wearable wireless voice, data, and video communications systems that combine onboard computing with a multi-interface wrist console, remote device and graphic user interface (GUI) controller. Apparatus and methods are also described for 3 Dimensional (3D) optical hand, body, object, sensor, and environment imaging, mapping, modeling, networking, and interfacing. Further, apparatus and methods are described for enabling augmentation of real world people, objects, and devices into a virtual environment and for enabling augmentation of virtual objects and interfaces into a physical environment.

0057Apparatus and methods are described that are advancements in wearable computing optics and interfacing over prior art, expanding from hand, finger, and object positioning to a 3D scanning, mapping, modeling, imaging, projection, and wireless interfacing systems all captured, rendered, and operated by a wrist computing and control console.

0058Apparatus and methods are described implementing a combination of light emitters; and sensors including body, motion, orientation, and location sensors. These apparatus and methods may further include optic, stereoscopic, or plenoptic lens arrays to generate depth maps and 3D imaging models, as well as virtual computing, interfacing and networking platforms by dynamically scanning and imaging the hands, body, objects and environment, indoors or outdoors in daylight or at night using one or more depth measurement and imaging methods.

0059Further, apparatus and methods are described for mobile light and optical depth mapping imaging, modeling, networking, and interfacing on an autonomous, intelligent wearable wireless wrist computing, display, and control system for onboard and remote device and graphic user interface (GUI) control. Augmentation of real world people, objects, and devices into a virtual environment and augmentation of virtual objects, interfaces, and environments into the real world through wireless multimedia streaming and multi-interface display and projection system is also described.

0060Light Mapping

0061Embodiments of the invention involve incorporating narrow or wide beam light emitters, or a structured light imaging system on the top and bottom of wrist consoles worn by a user. In embodiments, a device scans the top and bottom of the hand and fingers, the body, and any objects in the hand or in the field of the light imaging system.

0062When using narrow or wide beam emitter and sensor arrays, the emitter array may be assigned to move the focal point of the light beams up and down and back and forth across an x and y axis grid formation during the scanning process and are then assigned fixed positions to monitor hand and finger motion. The emitters form an x-y array for detecting the hand and fingers in the x and y dimensions of space and the sensors detect the presence or absence of a reflection. In embodiments, the depth (z) distance is measured by triangulation of the light beam reflection off of the target surface to the light sensor either mapping the target surface depth when the light beams are in motion and the object is stationary or dynamically identifying the position of the scanned object when the light beams are fixed and the scanned object is in motion.

0063In embodiments using a structured light imaging system light emitters and diffusers are incorporated to produce a light speckle pattern across the top and bottom of the hand and cover surrounding objects and environment. In embodiments designated camera sensors on a wrist console recognize the light speckle pattern and the directional reflection of each dot off of the target surface land on a different pixel within the camera sensor to triangulate the beam of light and determine the position, depth, and shape of the target surface.

0064Stationary structured light imaging system provides a constant point of origin for the projected light pattern with the only variables being the position, depth, and surface shape of target objects in its projection field. A wrist console introduces a wearable structured light imaging system that at most times is in motion. Even when a person holds their arm and hand steady, slight body movements can alter both the position and direction of the light emitters and projected light pattern and consequently the position and line of sight of the camera sensors.

0065In embodiments, the light mapping system is used both for initial mapping and modeling of the hand, body, objects, and environment and to dynamically monitor hand, finger, body, and object motion for gesture interfacing and control and to perform instant keyless user verification and authorization upon device sign-in, as well as performing instant user verification for payment and other secure transactions and security related functions such as keyless entry to home and vehicles and access to unique user accounts and user specific functions and applications on the device.

0066Position and Orientation Mapping

0067In embodiments, it is necessary to incorporate constant motion, position, and orientation data. When light and optical depth mapping and 3D color imaging is performed, the spatial position, directional motion, and orientation of the wrist console is acquired by any combination of onboard accelerometers, altimeters, compasses, and gyroscopes, as well as GPS and radio frequency (RF) directional signal and location data to continuously identify the precise relational position of the wrist console cameras, light emitters, and sensors to reflected and imaged surfaces to assign that data to each light point and color pixel in a depth and color map.

0068Optical Mapping and Imaging

0069In embodiments, depth mapping and 3D imaging is achieved using a stereoscopic or plenoptic multi-lens arrays. These arrays enable a wrist console's top and bottom modules to dynamically capture 3D or 4D multi-depth of field color imaging of the hand, body, surrounding objects, and environment.

0070In embodiments, when incorporating one or more stereoscopic lens arrays a wrist console performs stereo triangulation by determining the depth of two or more focal points in the scene, and determining the depths of the corresponding points in other images by matching points and features in one image to corresponding points and features in other images. To overcome the correspondence problem, the stereoscopic imaging system may select to incorporate the light imaging system to project one or more points of light on a target surface enabling the imaging system to verify the precise corresponding points in the images. Once the corresponding points have been identified, the imaging system determines the focal depths of all other points in the scene.

0071In embodiments, when incorporating a light-field plenoptic micro-lens array the wrist console captures multiple depths of field simultaneously. While stereoscopic lens arrays are limited to two or more individual lens arrays and sensors, each capturing light and color from a single depth of field, necessitating corresponding image analysis to match points in two or more images, the plenoptic micro-lens array assigns multiple lenses to a single sensor and captures the light and color from the entire field of view, while each lens captures a different depth of field enabling the camera to assign depth to all points in a captured image.

0072In embodiments, the optical imaging system is used both for initial imaging, depth and color mapping, and modeling of the hand, body, objects, and environment and to dynamically image hand, finger, body, and object motion for gesture and projection interfacing, to perform user verification and authorization and other security related functions, and to capture video and live stream user activities in 2D and 3D or 4D video and perform other imaging applications.

0073Modeling and Rigging

0074After light scanning and 3D imaging an object, the corresponding depth map is converted to a point cloud, a map of vertices with corresponding vectors in which each point is assigned an x, y and z (depth) coordinate. This process turns a grey scale depth map generated by the light scanning process or a 3D imaging of an object into a vertex in which each point or pixel in the image is identified as an x, y, and z coordinate that can be converted into metric units.

0075In embodiments, when a light scanned depth map is converted to a vertex and vector map by identifying the precise depth and directional position of each surface point, the color mapping process is enabled in which corresponding depth mapped color pixels are assigned to each point on the 3D vertex and vector map. This process converts the point cloud into a mesh in which points on a contiguous surface are connected and determines, for example, that one finger is behind the other, and they are not a single surface. The grid follows the surface shape, texture, and contours of the 3D mapped object.

0076Converting a surface mesh and 3D map of a persons hand or body into a functional character model that can be animated to mirror the movements of the wearer, incorporates a process of mapping the persons joints and assigning joint positions to the matching areas on the 3D model and generating an internal model rigging similar to the skeletal structure in the human body. Then attaching the rigging to the 3D mesh and model and assigning areas of influence to the mesh and surface of the 3D model similar to the effect of muscles and tendons on body motion and the skin.

0077In embodiments, when an existing functional character rigging exists, rather than generating a rig for each new model, the existing rig is scaled and conformed to the dimensions, shape, and physical characteristics of the mapped person. This may incorporate a program for determining body flexibility and motion based on the body type, sex, size, weight, age, health, fitness, flexibility, and other parameters of the mapped person to more accurately conform the rig and model to mimic the natural body motion and mobility of the person.

0078During the 3D light mapping and imaging process the wrist console may prompt the wearer to perform a number of hand and body motions, gestures, and positions to identify the joints, bone structure and mobility of the person. This may necessitate capturing multiple 3D scans and images of the person and then adjusting the rigging to replicate the precise body structure and mobility.

0079Sensor Mapping and Interfacing

0080In embodiments a wrist console is used to continuously map full body motion in real-time and incorporates external sensors into its 3D mapping and interfacing system. This includes body, clothing and remote wireless equipment sensors. By attaching micro sensors to the body or clothing on each of the limbs and joints or networking with embedded sensors in clothing, shoes and equipment the wrist console can identify the spatial position of one or more wireless sensors and assign those sensors to the mapped 3D model of the person, equipment and environment.

0081In embodiments the wrist console may use one or a combination of networking methods including Radio Frequency (RF), light/IR, Near Field Communication (NFC), Bluetooth, WiFi and Cellular networks for local and remote sensors and devices interfacing and control. This sensor network enables both sending and receiving data by the wrist console for wireless operation and control of remote sensors and dynamic mapping, interfacing and streaming of networked data as well as onboard or remote storage of mapped data.

0082In embodiments when the wrist console operates as a sensor hub for a wireless sensor network (WSN), the wrist console networks with each sensor directly or via a mesh network in which each sensor operates as a node and not only captures and sends data but also serves as a relay passing data on to the other nodes in the network.

0083In embodiments when monitoring body motion by identifying the 3D position, velocity, and acceleration of each joint or body part, a complete Cartesian coordinate 3D model of the body may be described mathematically with distance coordinates of x, y, and z; velocity coordinates of v_x, v_y, and v_z; and acceleration coordinates of a_x, a_y, and a_zto calculate the future position of an object in motion. Once the wrist console has identified and networked with the individual sensors or mesh sensor group, the wrist console is able to map the precise position of the sensors on the 3D character model. This process enables the wrist console to capture full body motion and acceleration as a continuous data stream and assign that data to the 3D rigged virtual model of the wearer to provide a real-time animation of the body and full body interfacing in a virtual environment.

0084In embodiments when the wrist console is used to map the internal body anatomy and interface with internal body sensors, devices and prosthetics, the wrist console incorporates a similar method of mapping, modeling, networking and interfacing with the internal body as it does with the external body. In embodiments when the wrist console is mapping the external body using the light and optical mapping and external sensors the wrist console is also performing internal mapping which incorporate the wrist console's onboard health and body sensors and then expands to all networked internal body sensors including ingested and implanted sensors, devices, prosthetics and any body or brain machine interfacing systems.

0085In embodiments the wrist console incorporates onboard body health and fitness sensors including top and bottom module wrist facing Infrared (IR) spectroscopy and pulse oximeter, heart rate monitor, thermometer, galvanic response system, Electroencephalograph (EEG), Electrocardiograph (ECG), Electromyograph (EMG), and glucose meter.

0086Projection Mapping and Interfacing

0087In embodiments, incorporating a pico projector for projecting an image onto external surfaces, the light and image mapping systems and orientation system are used to depth map surfaces, dynamically map the spatial position of the hands and fingers and the relational position of the wrist console and projector to a target surface. These processes enable the wrist console to map a projected display and graphic user interface onto any surface. The light and optical mapping systems are also used to dynamically monitor hand and finger motions and gestures enabling the user to perform touch and gesture interfacing to control the projected interface.

0088Further embodiments include an active touch screen displays, microphones, speakers, tactile feedback (haptic) sensor arrays, and front facing video cameras. These embodiments enable touch, voice, and gesture interfacing and voice command, video conferencing, and dynamic touch screen display with onboard graphic user interfaces.

0089In embodiments, the wrist console incorporates a touch screen display, one or more microphones and speakers, and tactile feedback (haptic) sensor array. These embodiments provide touch, voice, and gesture interfacing options for the user, enabling the user to select the most effective method for displaying and interfacing with a graphic user interface either on the wrist console or on one or more networked devices.

0090In embodiments, the user can map or assign a specific user interface such as voice command or gesture interfacing to a specific function, application, or device. For example, if the user is using the wrist console to interfacing with a personal computer and a television the user may assign voice command to the television while using gesture and touch on the computer.

0091In embodiments, the wrist console incorporates haptic sensor strips and/or a ring of haptic sensors on the inside of both the top and bottom wrist units. The wrist console generates very intricate positional, vibrational, and pressure responses to minute finger, hand, wrist, and body movements. The tactile response may also be incorporated into gesture command, touch screen, and device controls and other user interface applications to simulate button press on a projected keyboard, or provide a tactile response and more realism to object and/or application selection and control in a virtual 2D or 3D environment. The haptic response system may also be used to indicate an incoming or outgoing call, text or other event, locational and/or relational distance to a recognized object or person or any other assigned contextual application, alarm or monitored health status event such as alerting the wearer when their heart rate rises above a designated rate or glucose levels fall above or below a designated level or to inform the wearer of a potential oncoming seizure. Different types of vibrational and/or electro-stimulated responses may be generated and assigned to different callers, events and applications.

0092Device Mapping and Interfacing

0093In embodiments, the wrist console is capable of streaming content that is stored and playing on the device and or streaming to the wrist controller from the Internet to the screens of one or more networked devices and/or streaming multimedia content from a networked TV, game console, PC, or other networked device to one or more other devices or displays. This peer-to-peer networking, content management, distribution, and streaming can be achieved using a number of different wireless networks. Some of those include WiFi, Bluetooth, cellular, Infrared/light, Radio Frequency (RF), and NFC for rapid payments and transactions. One method for connecting all displays and devices in its field of view is through a single WiFi peer-to-peer network where each device is connected wirelessly through a multi-channel WiFi direct connect platform operating as a standalone WiFi hotspot and router, the wrist controller creates an ad-hoc peer-to-peer network with one or more wireless and/or Internet enabled devices and operates as remote wearable video game and computing console and wireless hub. The wrist console may also use any combination of networks to communicate with one or more devices

0094In embodiments, the wrist console manages content across multiple networked devices and monitors based on the position of the display in the room and the relation of the display to the wrist console and user. The wrist console is able to connect with multiple devices using multiple methods, networks, and channels.

0095Detailed Overview of the Embodiments in the Drawings

0096<figref idref="DRAWINGS">FIG. 1A</figref> is a perspective view of an embodiment of the wrist console <b>101</b> depicting the top wrist module <b>102</b> and bottom wrist module <b>103</b> which serve as housing modules for internal components of the device. The wrist modules are connected with adjustable wrist straps <b>104</b> which in embodiments contain communication cables between the two wrist modules. The device further has a wrist strap release and locking system <b>105</b>, forward facing light emitters <b>106</b> and sensors <b>107</b>, and multi-camera lens array <b>108</b>. <figref idref="DRAWINGS">FIG. 1A</figref> shows the inside view of the bottom module body sensors including light emitters and sensors <b>109</b>, Galvanic Skin Response System (GSR) <b>110</b>, and haptic feedback arrays (haptic array) <b>111</b>, a partial view of the display <b>114</b>, microphones <b>115</b>, speakers <b>116</b>, and top facing cameras <b>117</b>.

0097<figref idref="DRAWINGS">FIG. 1B</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting a view of the top module <b>102</b> body sensors including light emitters and sensors <b>109</b>, GSR <b>110</b> and Haptic Array <b>111</b>.

0098<figref idref="DRAWINGS">FIG. 1C</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting an overhead view of the rear facing light emitters <b>106</b> and sensors <b>107</b>, rear multi-camera lens array <b>108</b>, power and data ports <b>112</b> and docking ports <b>113</b>.

0099<figref idref="DRAWINGS">FIG. 1D</figref> is a top plan view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting the display <b>114</b>, microphones <b>115</b>, speakers <b>116</b> and top facing cameras <b>117</b>.

0100<figref idref="DRAWINGS">FIG. 2A</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting an expansion module <b>201</b> with display <b>202</b> and release buttons <b>203</b>, attached to the top module <b>102</b> of the wrist console <b>101</b>. This expansion module may serve as an additional power and data storage, processing and/or communication system for the device and/or an expanded display and interfacing system and may also perform expanded services such as a plug in glucose meter or other application.

0101<figref idref="DRAWINGS">FIG. 2B</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting the detached expansion module <b>201</b> with release buttons docking tabs <b>204</b> and power and data plug <b>204</b> and a button array <b>206</b> on the rear of the top module <b>102</b>.

0102<figref idref="DRAWINGS">FIG. 2C</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> depicting an expansion module <b>201</b> attached to the bottom module <b>103</b>.

0103<figref idref="DRAWINGS">FIG. 2D</figref> is a perspective view of the embodiment shown in <figref idref="DRAWINGS">FIG. 1A</figref> with an expansion module <b>201</b> detached from the bottom module <b>103</b>.

0104<figref idref="DRAWINGS">FIGS. 3A-3L</figref> are perspective views illustrating a wrist console <b>101</b> performing depth mapping and 3D imaging the hand and fingers <b>301</b>, identifying the joints and then rigging a fully functional computer model of the hand (hand model) <b>307</b>.

0105<figref idref="DRAWINGS">FIGS. 3A and 3B</figref> are perspective views illustrating an embodiment of the wrist console <b>101</b> that incorporates one or more moving beams of light <b>302</b> performing a light scan of the top and bottom of the hand and fingers <b>301</b> as a method for depth mapping. Triangulation is determined by the wrist console <b>101</b> light emitters <b>104</b> and sensors <b>106</b> as the light beams <b>302</b> move vertically and horizontally across the face of both sides of the hand and fingers <b>301</b>.

0106<figref idref="DRAWINGS">FIGS. 3C and 3D</figref> are perspective views depicting another embodiment of the invention that incorporates structured light imaging <b>303</b> into its depth mapping process by illuminating the top and bottom of the hand and fingers <b>301</b> with a speckled light pattern to light map <b>303</b> the entire hand <b>301</b> at once.

0107<figref idref="DRAWINGS">FIGS. 3E and 3F</figref> are perspective views illustrating the wrist console <b>101</b> generating a depth and color mapped hand <b>304</b> performing a combination of light mapping and imaging using 3D cameras <b>108</b> or 4D imaging using a plenoptic multi-lens array cameras <b>108</b> on both the top module and bottom of the wrist console <b>101</b>.

0108<figref idref="DRAWINGS">FIGS. 3G and 3H</figref> are perspective views illustrating the wrist console <b>101</b> identifying the precise position of joints and creases <b>305</b> on the top and bottom of the hand and fingers <b>301</b> for the purpose of generating a rig for the 3D mapped hand <b>304</b>.

0109<figref idref="DRAWINGS">FIGS. 3I and 3J</figref> are perspective views illustrating the wrist console <b>101</b> generating a functional rigging of the hand (hand rig) <b>306</b> from the top and bottom perspective.

0110<figref idref="DRAWINGS">FIGS. 3K and 3L</figref> are perspective views illustrating the invention incorporating the hand rig <b>306</b> into the 3D mapped hand <b>304</b> to create a fully functional rigged computer model of the hand (hand model) <b>307</b> capable of being animated and replicating the movements of the users hand and fingers <b>301</b> in real-time.

0111<figref idref="DRAWINGS">FIGS. 4A-4F</figref> and <figref idref="DRAWINGS">FIGS. 5A and 5B</figref> are perspective views illustrating an application of the invention where the user performs and selects control functions for different gestures. The wrist console <b>101</b> assigns those controls to the hand model <b>307</b>, which is used to carry out the users gesture inputs and commands in a 2D or 3D computing environment. <figref idref="DRAWINGS">FIGS. 4A-4F</figref> and <figref idref="DRAWINGS">FIGS. 5A and 5B</figref> represent only a few examples of gesture interfacing control options for a potentially limitless custom interface programming system.

0112<figref idref="DRAWINGS">FIGS. 4A and 4B</figref> are perspective views illustrating the user's hand and fingers <b>301</b> used to combine a gesture (making a first) with a motion (turning and moving the wrist) to a assign a customized gesture interface control function. The gesture is mapped and recorded by the wrist console <b>101</b>.

0113<figref idref="DRAWINGS">FIG. 4C</figref> is a perspective view illustrating the user using performing and selecting a gesture interface input and control function that involves multiple fingers <b>301</b>.

0114<figref idref="DRAWINGS">FIG. 4D</figref> is a perspective view illustrating the user performing and selecting a gesture interface control function that involves one finger <b>301</b>.

0115<figref idref="DRAWINGS">FIG. 4E</figref> is a perspective view illustrating the user performing and selecting a gesture interface control that involves touching a specified finger to an identified point or area on the hand <b>301</b>.

0116<figref idref="DRAWINGS">FIG. 4F</figref> is a perspective view illustrating the user performing and selecting a gesture interface control function that involves touching one specified finger to another specified finger <b>301</b>.

0117<figref idref="DRAWINGS">FIG. 5A</figref> is a perspective view illustrating a single point of control for a 2D or 3D computing environment on an external networked device <b>802</b> using a single finger as a controller <b>301</b>.

0118<figref idref="DRAWINGS">FIG. 5B</figref> is a perspective view illustrating multiple points of control for a 2D or 3D computing environment on an external networked device <b>802</b> using multiple fingers as controllers <b>301</b>.

0119<figref idref="DRAWINGS">FIG. 6A</figref> is a perspective view illustrating the wrist console <b>101</b> projecting a graphic user interface (GUI) <b>601</b> on to the user's hand and fingers <b>301</b>. The projected interface is mapped onto the hand using the light mapping and 3D imaging system.

0120<figref idref="DRAWINGS">FIG. 6B</figref> is a perspective view illustrating a user performing touch and gesture interfacing to control a projected GUI <b>601</b> on the user's hand and fingers <b>301</b>.

0121<figref idref="DRAWINGS">FIG. 6C</figref> is a perspective view illustrating the wrist console <b>101</b> projecting a graphic user interface <b>601</b> onto the users hands and fingers. In <figref idref="DRAWINGS">FIG. 6C</figref> the user has spread out their hand and fingers and the projected interface has dynamically conformed to the new position of the hand and fingers <b>301</b>. The user is selecting one of the projected icons representing an active program or application running on wrist console <b>101</b> or remotely via an Internet connection.

0122<figref idref="DRAWINGS">FIG. 6D</figref> is a perspective view illustrating the user typing on a projected keypad <b>601</b> on the users hand <b>301</b>.

0123<figref idref="DRAWINGS">FIG. 7</figref> is a perspective view illustrating a user typing on a projected keyboard <b>701</b> mapped on to an external surface by both a left and right wrist console <b>101</b>. <figref idref="DRAWINGS">FIG. 7</figref> depicts an embodiment with a coordinated dual projected interface in which both left and right wrist consoles operate in concert in mapping and projecting a dynamic interface on a projected surface. <figref idref="DRAWINGS">FIG. 7</figref> depicts the user typing a document on the projected keyboard that is displayed on an external device <b>803</b>. The left and right wrist consoles are either operating as a single input device and relaying data wirelessly to a remote control device <b>803</b> or the wrist consoles <b>101</b> are operating as primary operating and control device and streaming data to a remote display <b>803</b>.

0124Figure Sets <b>8</b>A-<b>8</b>D and <b>9</b>A and <b>9</b>B are perspective views depicting the wrist console wirelessly interfacing with external devices. In each of the figures the wrist console <b>101</b> is shown on both wrists, although a pair of consoles may be operated as a single device or device pair, each wrist console <b>101</b> may also operate autonomously and does not need a second console to perform two handed gesture interface control. A single wrist console <b>101</b> is capable of monitoring a second hand in close proximity for dual hand interfacing or may operate in concert with a second wrist console <b>101</b> enabling expanded functionality such as multi-function two-handed control, dual projection, expanded networking, processing, interfacing, power and data storage

0125<figref idref="DRAWINGS">FIG. 8A</figref> is a perspective view illustrating the user interfacing with two networked devices and using the wrist console <b>101</b> to map, image and model a scanned physical object into a virtual computing environment on an external networked device <b>802</b>.

0126<figref idref="DRAWINGS">FIG. 8B</figref> is a perspective view illustrating the user operating a left and right wrist console <b>101</b> as a two hand gesture interface controller to interface and manipulate a 3D computer model scanned and modeled by the wrist console <b>101</b> and wirelessly uploaded to an external networked device <b>802</b>.

0127<figref idref="DRAWINGS">FIG. 8C</figref> is a perspective view illustrating the user selecting a file, document or program on one external networked device <b>802</b> and with a gesture, voice or other UI command wirelessly transferring the file, document or program to a second networked device <b>803</b> using the wrist console as a data bridge between two networked devices.

0128<figref idref="DRAWINGS">FIG. 8D</figref> is a perspective view illustrating the user operating and controlling a 3D scanned and modeled object on the wrist console <b>101</b> touchscreen interface.

0129<figref idref="DRAWINGS">FIG. 9A</figref> is a perspective view illustrating a user wirelessly interfacing and controlling a remote device or vehicle <b>901</b> using a wide or local area peer-to-peer wireless network or via the Internet using a wide or local area Internet connection.

0130<figref idref="DRAWINGS">FIG. 9B</figref> is a perspective view illustrating a user wirelessly sending and receiving real-time voice, data, video and multimedia content to and from a remote device <b>901</b>, streaming the data and multimedia content in 3D to a left <b>904</b>, and right <b>903</b>, binocular heads up display <b>902</b>.

0131<figref idref="DRAWINGS">FIGS. 10A and 10B</figref> are perspective views illustrating the wrist console <b>101</b> full body scanning, depth mapping and imaging process where a user <b>1001</b> performs a body scan and a 3D computer model <b>1002</b> of the user is generated.

0132<figref idref="DRAWINGS">FIGS. 10C and 10D</figref> are perspective views illustrating the user in different positions and performing different body motions enabling the wrist console <b>101</b> to map and image the body <b>1002</b> in multiple positions and analyze the flexibility and mobility of the user to more accurately generate the body rigging for the 3D computer model and replicate the users motions in a virtual computing environment.

0133<figref idref="DRAWINGS">FIGS. 10E and 10F</figref> are perspective views illustrating the 3D mapping and imaging of the user <b>1001</b> and the 3D computer model <b>1003</b> with clothing. This may be accomplished by light mapping and 3D imaging the user's <b>1001</b> physical clothing or by mapping virtual clothing onto the 3D model <b>1003</b> in a computing environment.

0134<figref idref="DRAWINGS">FIGS. 11A-11D</figref> are perspective views illustrating the body mapping and imaging process from the perspective of each of the wrist console's <b>101</b> body facing cameras <b>108</b> with <figref idref="DRAWINGS">FIGS. 11A and 11B</figref> depicting the camera view from the top module and <figref idref="DRAWINGS">FIGS. 11C and 11D</figref> depicting the camera view from the bottom module <b>103</b>. The wrist console <b>101</b> is not shown in <figref idref="DRAWINGS">FIGS. 11A-11D</figref> because the figures depict the perspective of the cameras.

0135<figref idref="DRAWINGS">FIGS. 12A-12D</figref> are perspective views illustrating the body mapping and imaging process from the perspective of each of the wrist consoles <b>101</b> body facing cameras <b>108</b> with <figref idref="DRAWINGS">FIGS. 12A and 12B</figref> depicting the camera view from the top module <b>102</b> and <figref idref="DRAWINGS">FIGS. 12C and 12D</figref> depicting the camera view from the bottom module <b>103</b>. In <figref idref="DRAWINGS">FIGS. 12A-12D</figref> the users arms and hands are stretched out higher over the users head enabling the cameras to scan a different portion of the users body.

0136<figref idref="DRAWINGS">FIGS. 13A-13C</figref> are perspective views illustrating the body rigging process with <figref idref="DRAWINGS">FIG. 13A</figref> illustrates the surface mesh of depth and color mapped model of the user <b>1002</b>. <figref idref="DRAWINGS">FIG. 13B</figref> illustrates a full body character rigging (rig) that is conformed to the precise dimensions and characteristics of the mapped computer model of the user <b>1002</b>. And <figref idref="DRAWINGS">FIG. 13C</figref> illustrates the incorporation of the character rig

0137<figref idref="DRAWINGS">FIGS. 14A and 14B</figref> are perspective views illustrating the user's <b>1001</b> and the users 3D computer model <b>1002</b> spatial position and location in a mapped physical environment <b>1301</b> identified and mapped during the 3D body mapping and imaging process.

0138<figref idref="DRAWINGS">FIG. 15</figref> is a perspective view illustrating a virtual internal body map <b>1201</b> of the users anatomy and all networked, sensors, implanted devices and prosthetics all mapped and wirelessly controlled by the wrist console <b>101</b>. In <figref idref="DRAWINGS">FIG. 15</figref> the wrist console <b>101</b> using onboard, external, implanted or ingested networked body sensors to map each of the users body systems; Nervous System <b>1202</b>, Endocrine System <b>1203</b>, Skeletal System <b>1207</b>, Muscular System <b>1208</b>, Integumentary System <b>1209</b>, Cardiovascular System <b>1210</b>, Respiratory System <b>1211</b>, Lymphatic System <b>1212</b>, Digestive System <b>1213</b>, Urinary System <b>1214</b> and Reproductive System <b>1215</b>. The wrist console <b>101</b> also networks and interfaces with all internal data and multimedia interfacing systems, depicted in <figref idref="DRAWINGS">FIG. 15</figref> as a Brain Machine Interface (BMI) <b>1204</b>, Prosthetics depicted in <figref idref="DRAWINGS">FIG. 15</figref> as a prosthetic eye <b>1205</b>, and other implanted devices depicted in <figref idref="DRAWINGS">FIG. 15</figref> as a pacemaker <b>1206</b>.

0139<figref idref="DRAWINGS">FIGS. 16A and 16B</figref> are perspective views illustrating the wrist console <b>101</b> space, object and environment 3D light and image mapping process shown in <figref idref="DRAWINGS">FIGS. 16A and 16B</figref> as a residential living space <b>1301</b>. <figref idref="DRAWINGS">FIG. 16A</figref> depicts the user <b>1001</b> 3D mapping and imaging the living and dining section <b>1306</b> while <figref idref="DRAWINGS">FIG. 16B</figref> depicts the user <b>1001</b> mapping the kitchen section <b>1304</b> of the residential living space <b>1301</b>.

0140<figref idref="DRAWINGS">FIGS. 17A and 17B</figref> are perspective views of the user standing in a residential living space <b>1301</b>. <figref idref="DRAWINGS">FIG. 17A</figref> illustrates the user standing in the physical residential living space <b>1301</b> providing an example of a potential 3D mapped environment. <figref idref="DRAWINGS">FIG. 17B</figref> illustrates an overhead perspective view of a 3D mapped user <b>1003</b> and environment <b>1302</b> with all mapped people, objects, devices and environments stored securely on the wrist console or uploaded wirelessly to a user authorized account on the Internet or other network or database.

0141<figref idref="DRAWINGS">FIG. 18</figref> is an overhead perspective view of the user <b>1001</b> and the wrist console <b>101</b> identifying and mapping the location, device type, functions and applications, data, power and other system specifications and available networks and interfacing options for all networked devices in the in the residential living space <b>1301</b>.

0142<figref idref="DRAWINGS">FIGS. 19A and 19B</figref> are perspective views of a professional tennis player <b>1401</b>, wearing the wrist console <b>101</b>, playing tennis on a real outdoor tennis court <b>1404</b> while the user <b>1001</b> is testing his skills at home by attempting to return the tennis ball <b>1403</b> hit by the tennis player in real-time in a virtual gaming environment on the users <b>1001</b> television or other display <b>1405</b>.

0143<figref idref="DRAWINGS">FIGS. 20A-20C</figref> are perspective views of a professional golfer <b>1501</b> mapping his swing, the ball <b>1504</b>, and virtualizing the players entire game of golf in real time. <figref idref="DRAWINGS">FIG. 20A</figref> is a perspective view illustrating a professional golfer swinging at a golf ball <b>1504</b>. The golfer has sensors on or embedded in his clothing and shoes <b>1502</b> and equipment <b>1503</b> enabling the wrist console <b>101</b> to map every detail of the golfers body motion during the swing.

0144<figref idref="DRAWINGS">FIG. 21</figref> is a perspective view illustrating the professional golfer <b>1501</b> on the golf course mapping his swing and remotely mapping and monitoring the height, speed, trajectory, landing and resting position of a sensor enabled networked golf ball on the wrist console <b>101</b>.

0145<figref idref="DRAWINGS">FIG. 22</figref> is a schematic diagram of the connections of various components envisioned to be part of a wrist console device.

0146Although the invention is described herein with reference to the preferred embodiment, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. Accordingly, the invention should only be limited by the Claims included below.