Local advertising content on an interactive head-mounted eyepiece

LOCAL ADVERTISING CONTENT ON AN INTERACTIVE HEADMOUNTED EYEPIECE

This disclosure relates to augmented reality eyepieces, control techniques associated therewith, and applications for use.

The present disclosure also relates to a device that collects biological data and makes the collected data available over a network using a highly portable device.

<p>SUMMARY This disclosure is to provide an augmented reality eyepiece, control technology associated therewith, and applications for use.</p>

<p>In one embodiment, the eyepiece comprises a light source and an LCoS display, a freeform waveguide lens (two surfaces) enabling TIR bounce, a coupling lens disposed between the LCoS display and the freeform waveguide, and the projector is on may include a nano-projector (or micro-projector) with wedge-shaped optics (translucent corrective lenses) attached to a waveguide lens to allow proper viewing through the perceptual off perceptual lens. The projector may include an RGB LED module. RGB LED modules emit field sequential colors, where LEDs of different colors are turned on rapidly and continuously to form a color image that is reflected from the LCoS display. The projector may have a polarizing beam splitter or a projection collimator.</p><p>In one embodiment, the eyepiece may include a freeform waveguide lens, a freeform translucent correction lens, a display coupling lens, and a micro-projector.</p><p>In another embodiment, the eyepiece includes a freeform waveguide lens, a freeform corrective lens, a display coupling lens and a micro-projector, and has a FOV of at least 80 degrees and a visual display FOV (diagonal) of at least ˜25-30°. to provide.</p><p>In one embodiment, the eyepiece may include an optical wedge waveguide optimized to match the ergonomic factor of a human head capable of wrapping it around a human face.</p><p>In another embodiment, the eyepiece may include two free-form optical surfaces and a waveguide that can fold complex optical paths within a very thin prismatic form factor.</p><p>The present disclosure provides a method of collecting biological information from an individual. The method includes positioning a body part of the individual in front of a sensor. The sensor may be a flat type sensor for collecting fingerprints and palm prints, or it may be an optical device for collecting iris prints. Video and audio may be used to collect facial, gait, and voice information. The collected information is then processed to form an image, typically using light reflected from body parts, when the biological information is subject to visual capture. The captured image is formed by a flat plate sensor, using light reflected towards a camera positioned inside the sensor, which could also be a mosaic sensor. The collected images may be stored on a collection device or uploaded to a database of biological data.</p><p>Embodiments provide equipment for collecting biological data. The equipment includes a flat plate with a mosaic sensor, wherein the mosaic sensor has a camera disposed orthogonal to the flat plate as well as a plurality of light sources disposed around the periphery of the flat plate. The device also includes a keyboard and a strap for mounting the device to a forearm of a user. Inside, the device includes a geolocation module for identifying and recording location information and a communication module for providing a wireless interface with other communication devices. An internal clock is also included and provides time stamping of the collected biological information.</p><p>A further embodiment of the device provides a system for collecting biological information. The system includes a flat plate sensor for collecting fingerprint and palm information, an eyepiece that can be part of an augmented reality eyepiece, a video camera for collecting facial and gait information, and a computer for analyzing the collected biological data do. The collected data is then compared to a database of pre-collected information, and the comparison result is reported to the user.</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's a correction element for correcting a view of a surrounding environment, an integrated processor for handling content displayed to a user, and an integrated image source for guiding the content to the optical assembly, the displayed content comprising: an interactive control element; and an integrated camera facility for imaging the surrounding environment, wherein the user's hand gesture is identified as an interactive control element position command, wherein in response to the interactive control element position command, a change in the viewing direction of the user is related. Without it, the position of the interactive control element remains fixed relative to the object in the surrounding environment.</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's a correction element for correcting a view of a surrounding environment, an integrated processor for handling content displayed to a user, and an integrated image source for guiding the content to the optical assembly, the displayed content comprising: an interactive control element; and an integrated camera facility for imaging a body part of a user when interacting with the interactive control element, wherein the processor determines, based on the view of the user, to be co-located with the imaged user body part. A portion of the interactive control element is removed by removing the portion of the interactive control element.</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's a correction element for correcting a view of the surrounding environment, an integrated processor for handling content for display to a user, and an integrated image source for directing the content to the optical assembly. The displayed content includes an interactive keyboard control element, wherein the keyboard control element is associated with an input path analyzer, a word matching search facility, and a keyboard input interface. the user may enter text by sliding a pointing device (e.g., finger, stylus, etc.) across the letter keys of the keyboard input interface in a sliding motion through the appropriate sequence of words the user wishes to enter as text; Here, the input path analyzer determines the characters encountered in the input path, and the word matching facility finds the best word that matches the characters in the contacted sequence and inputs the matching best word as input text.</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's a calibrating element for correcting a view of the surrounding environment, an integrated processor for handling content for display to a user, an integrated image source for directing the content to the optical assembly, and an integrated camera facility for imaging an external visual cue; , the integrated processor identifies and translates the external visual cue as instructions for displaying content associated with the visual cue. The visual cue may be a sign of the surrounding environment, wherein the projected content is associated with an advertisement. Signs can be billboards, advertisements, and personalized advertisements based on the user's preference profile. The visual cue may be a hand gesture, projected content, and a projected virtual keyboard. The hand gestures may be thumb and forefinger from a first user's hand, and a virtual keyboard projected onto the palm of the first user's hand, where the user may be typing on the virtual keyboard with a second user's hand. can The hand gesture may be a combination of a thumb and forefinger gesture of the user's hand, and a virtual keyboard projected between the user's hand as set in the hand gesture, wherein the user uses the thumb of the user's hand on the virtual keyboard. can be typed in</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's a calibrating element for correcting a view of the surrounding environment, an integrated processor for handling content for display to a user, an integrated image source for directing the content to the optical assembly, and an integrated camera facility for imaging a gesture; An integrated processor identifies and translates the gesture as an indication of a command. The control instruction may provide an adjustment of content to be displayed, a command communicated to an external device, and the like.</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's a calibrating element for correcting a view of a surrounding environment, an integrated processor for handling content for display to a user, and an integrated image source for directing the content to the optical assembly, and a tactile control interface, the tactile control interface comprising: An interface is mounted to the eyepiece that accepts control input from the user via at least one of a user touching the interface and a user proximal to the interface.</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's Commands based on sensing a pre-defined head motion characteristic, and a correction element that corrects a view of the surrounding environment, an integrated processor that handles content for display to a user, and an integrated image source that directs the content to the optical assembly and at least one of a plurality of head motion sensing control devices integrated into the eyepiece for providing control commands to the processor as instructions.</p><p>The head motion feature may be a nod of the user's head such that the nod is a distinct motion different from the normal head motion. A clear motion can be a jerking motion of the head. Control instructions may be communicated to provide coordination of content for display, control external devices, and the like.</p><p>In an embodiment, the system comprises an interactive head-mounted eyepiece worn by a user, wherein the eyepiece comprises an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly being the user's a calibrating element for correcting a view of a surrounding environment, an integrated processor for handling content to display to a user, and an integrated image source for directing the content to the optical assembly, wherein the optical assembly is configured to: and an electrochromic layer that provides adjustment of display characteristics dependent on ambient environmental conditions. In embodiments, the display characteristics may be luminance, contrast, and the like. The ambient environmental condition may be a level of luminance that makes the displayed content difficult to visualize by the wearer of the eyepiece without adjusting the display characteristics, where the display characteristic adjustment may be applied to an area of the optical assembly onto which the content is projected.</p><p>In embodiments, the eyepiece may be an interactive head-mounted eyepiece worn by a user, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content. The optical assembly includes a correction element that corrects a user's view of the surrounding environment, and an integrated image source that guides the content to the optical assembly. Additionally, the eyepiece may include an adjustable wrap round extendable arm having any shape memory material to secure the position of the eyepiece on the user's head. The extendable arm may extend from one end of the eyepiece arm. The end of the wrap round extendable arm may be covered with silicone. Additionally, the extendable arms may abut or be secured to each other, or the extendable arms may independently grasp a portion of the head. In another embodiment, an extendable arm may be attached to a portion of the head mounted eyepiece to secure the eyepiece to a user's head. In embodiments, the extendable arm may extend telescopically from an end of the eyepiece arm. In another embodiment, at least one of the wrap around extendable arms is detachable from the head mounted eyepiece. The extendable arm may also be an add-on feature of the head mounted eyepiece.</p><p>In embodiments, the eyepiece may be an interactive head-mounted eyepiece worn by a user, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content. The optical assembly includes a correction element for correcting a user's view of the surrounding environment, and an integrated image source for guiding the content to the optical assembly. Additionally, the displayed content may include local advertisements, wherein the position of the eyepiece is determined by an integrated position sensor. Also, the local advertisement may be associated with the location of the eyepiece. In another embodiment, the eyepiece may include a capacitive sensor capable of detecting whether the eyepiece is in contact with human skin. A local advertisement may be sent to the user based on whether the capacitive sensor detects that the eyepiece is in contact with human skin. Local advertisements may also be sent in response to the eyepiece being powered on.</p><p>In other embodiments, the local advertisement may be displayed to the user as a banner advertisement, two-dimensional graphic, or text. Additionally, advertisements may relate to physical aspects of the surrounding environment. In another embodiment, the advertisement may be displayed as augmented reality associated with a physical aspect of the surrounding environment. Augmented reality advertisements can be two or three dimensional. Additionally, the advertisement may be animated, and it may be associated with the user's view of the surrounding environment. The local advertisement may be displayed to a user based on a web search performed by the user and displayed as content of the search result. Additionally, the content of the local advertisement may be determined based on user personal information. User personal information may be used for web applications or advertising facilities. The user's information may be used by a web application, an advertising facility, or the eyepiece to filter local advertisements based on the user's personal information. Local advertisements may be cached on a server that may be accessed and displayed to a user by at least one of an advertisement facility, a web application, and an eyepiece.</p><p>In another embodiment, the user may request additional information related to the local advertisement by any of eye movements, body movements, and other gestures. Additionally, a user may ignore the local advertisement by making any eye movements, body movements and other gestures, or by not selecting an advertisement for further interaction within a given amount of time from when the advertisement is displayed. In another embodiment, the user may choose not to allow local advertisements to be displayed by selecting such as an option on a graphical user interface. Alternatively, the user may disallow such advertisements by turning off these features via control over the eyepiece.</p><p>In one embodiment, the eyepiece may include an audio device. Additionally, the displayed content may include local advertisements and audio. The position of the eyepiece may be determined by an integrated position sensor, and the local advertisement and audio may be associated with the position of the eyepiece. Thereby, the user can audible audio corresponding to the displayed content and local advertisement.</p><p>In one aspect, an interactive head-mounted eyepiece includes an optical assembly through which a user views a surrounding environment and displayed content, wherein the optical assembly comprises a corrective element that corrects the user's view of the surrounding environment, and total reflection and an optical waveguide having first and second surfaces to enable total internal reflection. The eyepiece also includes an integrated processor for handling content for display to a user, and an integrated image source for directing the content to the optical assembly. In this aspect, the displayed content may be directed to the optical waveguide at an internal angle of incidence that does not result in total reflection. However, the eyepiece also includes a mirrored surface on the first surface of the optical waveguide to reflect the displayed content towards the second surface of the optical waveguide. Thus, the mirrored surface enables total reflection of light entering the optical waveguide or reflection of at least a portion of light entering the optical waveguide. In an embodiment, the surface may be mirrored at 100% or may be mirrored at a smaller rate. In some embodiments, at the location of the mirrored surface, an air gap between the waveguide and the correction element causes reflection of light entering the waveguide at an angle of incidence that does not generate a TIR.</p><p>In one aspect, an interactive head-mounted eyepiece includes an optical assembly through which a user views a surrounding environment and displayed content, wherein the optical assembly comprises a corrective element that corrects the user's view of the surrounding environment; and an integrated processor for handling content to be displayed to the user. The eyepiece also includes an integrated image source for guiding the content to the optical assembly from a side of the optical waveguide adjacent the arm of the eyepiece, wherein the aspect ratio of the displayed content is substantially horizontal of the major axis. It is between square and almost rectangular.</p><p>In one aspect, an interactive head-mounted eyepiece includes an optical assembly through which a user views a surrounding environment and displayed content, the optical assembly comprising a corrective element for correcting the user's view of the surrounding environment, and internal reflections a free-form optical waveguide that enables The eyepiece also includes an integrated processor for handling content for display to a user, and an integrated projector facility for projecting the content onto the optical assembly, wherein the projector facility includes a light source and an LCoS display, and Light is emitted under the control of the processor and traverses a polarizing beam splitter where it is polarized before being reflected from the LCoS to the optical waveguide. In another aspect, an interactive head-mounted eyepiece includes an optical assembly through which a user views a surrounding environment and displayed content, wherein the optical assembly comprises a corrective element for correcting the user's view of the surrounding environment, and an interior an optical waveguide to enable reflection, and a coupling lens disposed to direct an image from the optical display to the optical waveguide. The eyepiece also includes an integrated processor for handling content for display to a user, and an integrated image source for guiding the content to the optical assembly, the image source including a light source and an optical display. The correction element may be a see-through correction lens attached to the optical waveguide that enables proper viewing of the surrounding environment whether the image source or projector facility is on or off. The free-form optical waveguide may include a double free-form surface that allows for curvature and sizing of the waveguide, wherein the curvature and sizing are performed in a frame of the interactive head-mounted eyepiece. to enable the placement of the waveguide. The light source can be an RGB LED module that emits light to sequentially form a color image reflected from an optical display or an LCoS display. The eyepiece may further include a homogenizer through which light from the light source propagates to ensure that the light beam becomes uniform. The surface of the polarizing beam splitter reflects the color image from the optical or LCoS display to the optical waveguide. The eyepiece further includes a collimator that improves the resolution of the light entering the optical waveguide. Light from the light source is emitted under the control of the processor and traverses a polarizing beam splitter before being reflected from the optical display to the optical waveguide. The optical display may be at least one of an LCoS and an LCD display. The image source may be a projector, wherein the projector is at least one of a microprojector, a nanoprojector, and a picoprojector. The eyepiece further comprises a polarizing beam splitter that polarizes light from the light source prior to reflection from the LCoS display into the optical waveguide, wherein a surface of the polarizing beam splitter directs the color image from the LCoS display to the optical waveguide. reflect</p><p>In one embodiment, biological data capture equipment is provided. The biometric data may be visual biometric data, such as facial biometric data or iris biometric data, or may be audio biometric data. The equipment includes an optical assembly through which the user views the surrounding environment and the displayed content. The light assembly also includes a correction element that corrects the user's view of the surrounding environment. The integrated processor handles the content displayed to the user on the eyepiece. The eyepiece also incorporates an integrated image source to guide the content to the optical assembly. Biological data capture is achieved with an integrated optical sensor assembly. Audio data capture is achieved with an integrated endfire microphone array. Processing of the captured biological data occurs remotely and the data is transmitted using an integrated communication facility. The remote computing facility translates and analyzes the captured biometric data, generates display content based on the captured biometric data, and delivers the display content to the eyepiece.</p><p>A further embodiment provides a camera mounted to the eyepiece for obtaining a biological image of an individual proximate to the eyepiece.</p><p>A further embodiment provides a method for capturing biological data. In the method, an individual is positioned proximate to the eyepiece. This may be accomplished by moving the wearer of the eyepiece to a position that allows capture of the desired biological data. Once positioned, the eyepiece captures the biological data and transmits the captured biological data to a facility that stores the captured biological data in a database of biological data. The database of biometric data incorporates a remote computing facility that translates received data and generates display content based on the translation of the captured biometric data. This display content is then sent back to the user for display on the eyepiece.</p><p>A further embodiment provides a method for audio biometric data capture. In the method, an individual is positioned proximate to the eyepiece. This may be accomplished by moving the wearer of the eyepiece to a position that allows capture of the desired audio biometric data. Once positioned, the microphone array captures the audio biometric data and transmits the captured audio biometric data to a facility that stores the captured audio biometric data in a database of biometric data. The database of audio biometric data incorporates a remote computing facility that translates received data and generates display content based on the translation of the captured audio biometric data. This display content is then sent back to the user for display on the eyepiece.</p><p>In an embodiment, the eyepiece comprises a see-through correction lens attached to the outer surface of the optical waveguide to enable proper viewing of the surrounding environment with or without displayed content. The see-through corrective lens may be a customized prescription lens for the user's corrective eyeglass prescription. A see-through corrective lens may be polarized and attached to at least one of the frame of the optical waveguide and the eyepiece, wherein the polarized corrective lens blocks oppositely polarized light reflected from the user's eye. The see-through corrective lens may be attached to at least one of the optical waveguide and the frame of the eyepiece, wherein the corrective lens protects the optical waveguide and comprises at least one of a ballistic material and an ANSI approved polycarbonate material. can</p><p>In one embodiment, the interactive head-mounted eyepiece comprises an eyepiece worn by a user, an optical assembly mounted to the eyepiece through which the user views the surrounding environment and displayed content, wherein the optical assembly comprises: A corrective element for correcting a user's view of the surrounding environment, an integrated processor handling content displayed to a user, an integrated image source directing the content to the optical assembly, and adjusting focus of the displayed content for the user. and an electrically adjustable lens integrated with the optical assembly.</p><p>One embodiment relates to an interactive head-mounted eyepiece. Such an interactive head-mounted eyepiece includes an eyepiece worn by a user, and an optical assembly mounted to the eyepiece through which the user views the surrounding environment and displayed content, wherein the optical assembly provides a view of the user's surrounding environment. and a correction element for correcting , and an integrated processor of the interactive head-mounted eyepiece for handling content to be displayed to a user. The interactive head-mounted eyepiece also includes an electrically adjustable liquid lens integrated with the optical assembly, an integrated image source of the interactive head-mounted eyepiece for directing the content into the optical assembly, and the integrated a memory operatively coupled to the processor, the memory including at least one software program for providing correction to the displayed content by adjusting the electrically adjustable liquid lens.</p><p>Another embodiment is an interactive head-mounted eyepiece worn by a user. The interactive head-mounted eyepiece includes an optical assembly mounted to the eyepiece through which a user views a surrounding environment and displayed content, wherein the optical assembly is calibrated to correct a user's view of the displayed content. elements, and an integrated processor that handles content to display to a user. The interactive head-mounted eyepiece also includes an integrated image source for directing the content to the optical assembly, an electrically adjustable liquid lens integrated with the optical assembly to adjust the focus of the displayed content with respect to the user; and at least one sensor mounted to the interactive head-mounted eyepiece, wherein an output from the at least one sensor is output from the interactive head-mounted eyepiece using at least one of optical stabilization and image stabilization. used to stabilize the displayed content of the optical assembly.</p><p>One embodiment is a method of stabilizing an image. The method includes providing an interactive head-mounted eyepiece comprising a camera and an optical assembly through which a user views a surrounding environment and displayed content, and with the camera to capture an image of an object in the surrounding environment. and imaging the surrounding environment. The method also includes displaying the content in a fixed position relative to a user view of the imaged object via the optical assembly, sensing vibration and movement of the eyepiece, and the surrounding environment via at least one digital technique. stabilizing the displayed content with respect to a user view of</p><p>Another embodiment is a method of stabilizing an image. The method includes providing an interactive head-mounted eyepiece comprising a camera and an optical assembly through which the user views the surrounding environment and displayed content, the assembly further comprising: a processor for handling content displayed to the user; including an integrated projector for projecting the content onto the optical assembly, and imaging the surrounding environment with the camera to capture an image of an object in the surrounding environment. The method also includes displaying the content in a fixed position relative to a user view of the imaged object via the optical assembly, sensing vibration and movement of the eyepiece, and the surrounding environment via at least one digital technique. stabilizing the displayed content with respect to a user view of</p><p>One embodiment is a method of stabilizing an image. The method includes providing an interactive head-mounted eyepiece to be worn by a user, and imaging the surrounding environment with a camera to capture an image of an object in the surrounding environment, wherein the eyepiece comprises: an optical assembly through which a user views a surrounding environment and displayed content, wherein the optical assembly comprises a corrective element for correcting the user's view of the surrounding environment, an integrated processor for handling the content displayed to the user, and the optical assembly and an integrated image source guiding the content to The method also includes displaying the content in a fixed position relative to a user view of the imaged object via the optical assembly, detecting vibration and movement of the eyepiece, and receiving a signal indicative of vibration and movement of the eyepiece. transmitting to the integrated processor of the interactive head-mounted device and stabilizing the displayed content with respect to a user's view of the surrounding environment via at least one digital technology.</p><p>Another embodiment is an interactive head-mounted eyepiece. The interactive head-mounted eyepiece includes an eyepiece worn by a user, an optical assembly mounted to the eyepiece through which the user views the surrounding environment and displayed content, and an eyepiece mounted to the eyepiece that corrects the user's view of the surrounding environment. It contains a correction element. The interactive head-mounted eyepiece also includes an integrated processor for handling content for display to a user, an integrated image source for directing the content to the optical assembly, and at least one sensor mounted to the camera or eyepiece; , wherein the output from the at least one sensor is used to stabilize the displayed content of the optical assembly of the interactive head-mounted eyepiece using at least one digital technology.</p><p>One embodiment is an interactive head-mounted eyepiece. The interactive head-mounted eyepiece includes an interactive head-mounted eyepiece worn by a user, an optical assembly mounted to the eyepiece through which the user views the surrounding environment and displayed content, and content to display to the user. and an integrated processor of the eyepiece. The interactive head-mounted eyepiece also includes an integrated image source of the eyepiece for directing the content to the optical assembly, and at least one sensor mounted to the interactive head-mounted eyepiece, wherein: The output from the at least one sensor is used to stabilize the displayed content of the optical assembly of the interactive head-mounted eyepiece using at least one of optical stabilization and image stabilization.</p><p>Another embodiment is an interactive head-mounted eyepiece. The interactive head-mounted eyepiece includes an eyepiece worn by a user, an optical assembly mounted to the eyepiece through which the user views the surrounding environment and displayed content, and an integrated processor that handles the content being displayed to the user. The interactive head-mounted eyepiece also includes an integrated image source for directing the content to the optical assembly, and an integrated image source for stabilizing content for display to a user in series between the optical assembly and the integrated image source. an electro-optic lens, and at least one sensor mounted to the eyepiece or a mount for the eyepiece, wherein an output from the at least one sensor is output from the interactive head-mounted eyepiece. used to stabilize the electro-optic lens.</p><p>Aspects disclosed herein include an interactive head-mounted eyepiece worn by a user, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly comprising the user's said a correction element for correcting a view of the surrounding environment, an integrated processor for handling content for display to a user, and an integrated image source for directing the content to the optical assembly.</p><p>The eyepiece further includes a control device worn on a user's hand, the control device including at least one control component actuated by a digit of the user's hand, and the at least one control component as a command directive. provides a control command from the processor to the processor. The command directive may instruct the user to adjust the content to be displayed.</p><p>The eyepiece further includes a hand motion sensing device worn on the user's hand, and provides a control command from the motion sensing device as a command instruction to the processor.</p><p>The eyepiece further comprises an interactive optical assembly through which the user views ambient information concurrently with the displayed content and sensor information from the sensor and the displayed content transmitted via an integrated image source and an optical assembly from a processor handling the displayed content to the user. wherein the processor correlates information from the sensor with the displayed content to indicate a gaze of an eye on a projected image, and displays a user command in addition to gaze information on the projected image to cause an action use the</p><p>In the eyepiece, gaze information for the user's eye is communicated to the processor as a command indication.</p><p>The eyepiece may further include a hand motion sensing device for tracking hand gestures within the field of view of the eyepiece to provide control instructions to the eyepiece.</p><p>In one aspect, a method of social networking includes accessing a social networking website using the eyepiece, requesting information about members of the social networking website using the interactive head-mounted eyepiece. and retrieving adjacent members of the social networking website using the interactive head-mounted eyepiece.</p><p>In one aspect, a method of social networking includes accessing a social networking website using the eyepiece, requesting information about other members of the social networking website using the interactive head-mounted eyepiece. and transmitting a signal indicative of a user location of the interactive head-mounted eyepiece, and allowing access to information about the user of the interactive head-mounted eyepiece.</p><p>In one aspect, a method of social networking includes accessing a social networking website using the eyepiece, requesting information about members of the social networking website using the interactive head-mounted eyepiece. transmitting a signal indicative of the user's location of the interactive head-mounted eyepiece and at least one preference, allowing access to information on a social networking site regarding the user's preferences of the interactive head-mounted eyepiece and searching for adjacent members of the social networking website using the interactive head-mounted eyepiece.</p><p>In one aspect, a gaming method includes accessing an online gaming site using the eyepiece, starting or participating in a game of the online gaming site using the interactive head-mounted eyepiece, the interactive head- viewing the game through an optical assembly of a wearable eyepiece, and playing the game by adjusting at least one body-mounted control device using the interactive head-mounted eyepiece.</p><p>In one aspect, a gaming method comprises the steps of using the eyepiece to access an online gaming site, each member using an interactive head-mounted eyepiece system, the online gaming site and a plurality of members of the online gaming site. starting or participating in a game of; viewing game content with the optical assembly; and playing the game by adjusting at least one sensor for motion detection.</p><p>In one aspect, a gaming method includes using the eyepiece to access an online gaming site, using an interactive head-mounted eyepiece to contact at least one additional player for a game on the online gaming site, the starting a game of the online gaming site using an interactive head-mounted eyepiece, viewing the game of the online gaming site with the optical assembly of the interactive head-mounted eyepiece, and the interactive head- and playing the game by touch-free adjusting at least one control using the wearable eyepiece.</p><p>In one aspect, a method of using augmented vision comprises providing an interactive head-mounted eyepiece comprising an optical assembly through which a user views the surrounding environment and displayed content, with a black silicon short-wavelength infrared (SWIR) image sensor. scanning the surrounding environment, controlling a SWIR image sensor via movement, gesture, or user command, transmitting at least one visual image from the sensor to the processor of the interactive head-mounted eyepiece; and viewing the at least one visual image using the optical assembly, wherein the black silicon short wavelength infrared (SWIR) sensor provides night vision.</p><p>In one aspect, a method of using augmented vision comprises providing an interactive head-mounted eyepiece comprising a camera and an optical assembly through which the user views the surrounding environment and displayed content, the camera and black silicon short-wavelength infrared (SWIR) ) viewing the surrounding environment with an image sensor, controlling the camera through movement, gesture, or a user's command, transmitting information from the camera to the processor of the interactive head-mounted eyepiece, and viewing a visual image using an optical assembly, wherein the black silicon short wavelength infrared (SWIR) sensor provides night vision functionality.</p><p>In one aspect, a method of using augmented vision comprises providing an interactive head-mounted eyepiece comprising an optical assembly through which a user views the surrounding environment and displayed content, with a black silicon short-wavelength infrared (SWIR) image sensor. viewing the environment, controlling the image sensor scanning through user movements and gestures, transmitting information from the image sensor to the processor of the interactive head-mounted eyepiece, and using the optical assembly viewing a visual image with a device, wherein the optical assembly comprises a corrective element for correcting a user's view of the surrounding environment, an integrated processor for handling content to display to the user, and an optical assembly for directing the content to the optical assembly. an integrated image source, wherein the black silicon short wavelength infrared (SWIR) sensor provides night vision.</p><p>In one aspect, a method of receiving information includes accessing a database accessible using an interactive head-mounted eyepiece including an optical assembly through which a user views a surrounding environment and displayed content, the interactive head - requesting information from the accessible database using a mounted eyepiece, and viewing information from the accessible database using the interactive head-mounted eyepiece, wherein the requesting The steps and viewing steps are accomplished without the user touching the controls of the interactive head-mounted eyepiece.</p><p>In one aspect, a method of receiving information includes accessing an accessible database using the eyepiece, requesting information from the accessible database using the interactive head-mounted eyepiece, using an optical facility. and displaying the information by using a processor, and adjusting the information using a processor, wherein the requesting, displaying, and adjusting steps are performed without touching the controls of the interactive head-mounted eyepiece. is achieved</p><p>In one aspect, a method of receiving information comprises: accessing an accessible database using the eyepiece; requesting information from the accessible website using an eyepiece; allowing access to information on the accessible website without touching the controls of the interactive head-mounted eyepiece; displaying information, and manipulating the information using the processor without touching a control of the interactive head-mounted eyepiece.</p><p>In one aspect, a social networking method includes providing the eyepiece, scanning facial features of an adjacent person with an optical sensor of the head-mounted eyepiece, extracting a facial profile of the adjacent person, the interactive accessing the social networking website using the communication facility of the head-mounted eyepiece, and searching a database of the social networking site for matching the facial profile.</p><p>In one aspect, a social networking method includes providing the eyepiece, scanning facial features of an adjacent person with an optical sensor of the head-mounted eyepiece, extracting a facial profile of the adjacent person, the interactive accessing a database using the communication facility of the head-mounted eyepiece, and searching the database for a person matching the facial profile.</p><p>In one aspect, a social networking method comprises the steps of: accessing a social networking website using the eyepiece, requesting information about an adjacent member of the social networking website using the interactive head-mounted eyepiece; scanning facial features of an adjacent person identified as a member of the social networking site with an optical sensor of the head-mounted eyepiece, extracting a facial profile of the adjacent person, and at least for information about the adjacent person and searching one additional database.</p><p>In one aspect, a method of using augmented vision includes providing the eyepiece, controlling a camera through a user's movement, gesture or command, and transferring information from the camera to a processor of the interactive head-mounted eyepiece. transmitting, and viewing a visual image using the optical assembly, wherein the visual image from the camera and optical assembly has an improvement to the user in at least one of focus, brightness, and clarity and magnification. have.</p><p>In one aspect, a method of using augmented vision includes providing the eyepiece, controlling the camera through movement of a user without touching the controls of the interactive head-mounted eyepiece, and receiving information from the camera. transmitting to a processor of an interactive head-mounted eyepiece, and viewing a visual image using an optical assembly of the interactive head-mounted eyepiece, wherein the visual image from the camera and optical assembly is focused; There is an improvement for the user in at least one of brightness, sharpness, and magnification.</p><p>In another aspect, a method using augmented vision includes providing the eyepiece, controlling the camera through movement of a user of the interactive head-mounted eyepiece, and transferring information from the camera to the interactive head-mounted eyepiece. transferring to the integrated processor of the interactive eyepiece, applying image enhancement techniques using computer software and the integrated processor of the interactive head-mounted eyepiece, and using the optical assembly of the interactive head-mounted eyepiece viewing a visual image, wherein the visual image from the camera and optical assembly has an improvement to the user in at least one of focus, brightness, sharpness, and magnification.</p><p>In one aspect, a facial recognition method includes capturing an image of a subject with the eyepiece, converting the image into biological data, comparing the biological data to a database of pre-collected biological data, and pre-collected biological data. identifying biometric data that matches the data, and reporting the identified matching biometric data as displayed content.</p><p>In another aspect, a system includes an eyepiece, a facial detection facility associated with the integrated processor facility, and an integrated vibration actuator in the eyepiece, wherein the facial detection facility captures an image of faces in a surrounding environment and captures compares the imaged images to images stored in a facial recognition database, and provides a visual indication indicative of a match, wherein the visual indication corresponds to a current location of the imaged face in the surrounding environment as part of the projected content and , the vibration actuator provides a vibration output to inform the user of the match.</p><p>In one aspect of the present invention, a method for augmented vision includes collecting photons with a short-wavelength infrared sensor mounted on the eyepiece, converting the collected photons in a short-wavelength infrared spectrum into an electrical signal, the electrical signal relaying to the eyepiece for display, collecting biological data using the sensor, collecting audio data using an audio sensor, and transmitting the collected biological data and audio data to a database include</p><p>In one aspect, an object recognition method includes capturing an image of an object with an eyepiece, analyzing the object to determine whether the object is pre-captured, resolution of a captured image area that is neither pre-captured nor analyzed increasing , and lowering the resolution of an area of the captured image that is neither pre-captured nor analyzed.</p><p>In another aspect, a system comprises an eyepiece, a position determination system external to the eyepiece, in communication with an eyepiece, the processor facility to enable position information of the sensor processor facility to determine a pointing direction of a weapon, the processor facility comprising: The content is provided through a display for the user to indicate the current pointing direction of the weapon.</p><p>In one aspect, a system includes an eyepiece having a communication interface, and a control device worn on a user's hand, comprising at least one control component actuated by a finger of the user's hand, and comprising: provide control instructions from actuation of at least one control component to the processor, wherein the instruction instructions relate to identifying a target to potentially fire with a handheld weapon.</p><p>In another aspect, a system includes an eyepiece and a weapon mounting interface that accepts user input to the eyepiece and generates control instructions.</p><p>In another aspect, a system includes an eyepiece and a weapon mounting interface that accepts user input to the eyepiece and generates control instructions, wherein the displayed content relates to information about an object viewed through the eyepiece.</p><p>In one aspect, the system includes an eyepiece, wherein an optical assembly is attached to the eyepiece and can be moved out of the field of view of a user.</p><p>In one aspect, a method for collecting biological information includes positioning a body part in front of a sensor, and when the sensor is illuminated from a side orthogonal to the body part, the sensor is illuminated on the body part using light reflected from the body part. Recording biological information about the body part, forming an image using light reflected from the body part, and similarly storing the image in a database of collected biological information.</p><p>In another aspect, a device for collecting biological information includes a flat plate including a mosaic sensor, wherein the mosaic sensor includes a plurality of light sources disposed around the flat plate, a camera disposed orthogonally to the flat plate, and a keyboard , a strap for mounting on a user's forearm, a geolocation module for ensuring positional positioning, a communication module for wirelessly interfacing with other communication devices, and a watch for time stamping the collected biological information.</p><p>In another aspect, a biometric information collection system includes a flat plate sensor for collecting finger and palm information, an eyepiece for collecting iris and facial information, a video camera for collecting facial and gait information, and analyzing the collected biological data and a computer for comparing with a database of previously collected information, determining whether the collected biological information is stored in the database in advance, and providing the analysis result.</p><p>In one aspect, a method of streaming data to an eyepiece includes providing the eyepiece, coupling a communication interface to an optical train of a device, and streaming data from the device to the eyepiece.</p><p>In another aspect, a fire sight includes an optical lens that magnifies a target, a camera that captures an image of the target, a sensor that collects biological data from the target, and a wireless data transmitter that transmits the captured image and biological information to the eyepiece. includes</p><p>These and other systems, methods, objects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description of the embodiments and drawings.</p><p>All documents mentioned in the text are incorporated herein by reference in their entirety. Items in the singular are to be understood to include items in the plural, unless otherwise stated or made clear in the documentation. Unless otherwise stated or clear from context, grammatical conjunctions are intended to express any and all adjunct conjunctions and combinations of conjunctions in the joined clauses, sentences, words, etc.</p>

BRIEF DESCRIPTION OF THE DRAWINGS The following detailed description of the present disclosure and specific embodiments thereof may be understood with reference to the following drawings: 1 shows an exemplary embodiment of an optical arrangement. 2 shows an RGB LED projector. 3 shows the projector in use. 4 shows one embodiment of a waveguide and correction lens disposed in a frame. 5 shows a design for a waveguide eyepiece. 6 shows one embodiment of an eyepiece with a see-through lens. 7 shows one embodiment of an eyepiece with a see-through lens. 8A-C show one embodiment of an eyepiece arranged as a flip-up/flip-down unit. 8d and 8e show the snap-fit element of the second optic. 9 shows an embodiment of a flip-up/flip-down electro-optic module. 10 shows the benefits of the eyepiece in real-time image enhancement, keystone correction, and virtual perspective correction. 11 shows a plot of reactivity versus wavelength for three substrates. 12 shows the performance of a black silicon sensor. 13A shows an incumbent night vision system, FIG. 13B shows a night vision system of the present disclosure, and FIG. 13C illustrates the difference in reactivity between the two. 14 shows the tactile interface of the eyepiece. 14A illustrates motion in one embodiment of eyepiece featuring node control. 15 shows a ring that controls the eyepiece. 15A shows a hand mounted sensor in one embodiment of a virtual mouse. 15B shows a facial actuation sensor as mounted on the eyepiece. 15C shows the hand pointing control of the eyepiece. 15D shows the hand pointing control of the eyepiece. 15E shows an example of an eye tracking control. 15F shows the hand positioning control of the eyepiece. 16 shows the location-based application mode of the eyepiece. 17 shows the difference in image quality between A) a flexible platform of an uncooled CMOS image sensor enabling VIS/NIR/SWIR and B) an image-enhanced night vision system. 18 illustrates an augmented reality-enabled custom billboard. 19 illustrates an augmented reality-enabled custom advertisement. 20 depicts augmented reality-enabled custom artwork. 20A illustrates a method of posting a message to be sent when a viewer has reached a particular location. 21 shows an alternative arrangement of the eyepiece optics and electronics. 22 shows an alternative arrangement of the eyepiece optics and electronics. 23 shows an alternative arrangement of the eyepiece optics and electronics. 24 shows the lock position of the virtual keyboard. 25 shows a detailed view of the projector. 26 shows a detailed view of the RGB LED module. 27 shows a gaming network. 28 shows a method of playing a game using augmented reality glasses. 29 shows an exemplary electronic circuit diagram for an augmented reality eyepiece. 30 shows a control circuit for eye tracking control of an external device. 31 illustrates a communication network between users of an augmented reality eyepiece. 32 shows a flowchart for a method of identifying a person based on a person's speech as captured by a microphone of an augmented reality device. 33 illustrates a mosaic finger and palm registration system according to one embodiment. 34 illustrates a general optical approach used by other finger and palm print systems. 35 illustrates an approach used by a mosaic sensor according to one embodiment. 36 shows a device layout of a mosaic sensor according to one embodiment. 37 illustrates a camera field of view and number of cameras used in a mosaic sensor according to another embodiment. 38 shows a bio-phone and a tactical computer according to one embodiment. 39 illustrates the use of a bio-phone and tactical computer in capturing potential fingerprints and palm prints according to one embodiment. 40 illustrates a typical DOMEX collection. 41 illustrates a relationship between a bio-phone and a biological image captured using a tactical computer and a biological clock according to one embodiment. 42 depicts a pocket bio-kit according to one embodiment. 43 shows the components of a pocket bio-kit according to one embodiment. 44 illustrates a fingerprint, palm print, geo-location and POI registration device according to an embodiment. 45 illustrates a multi-modal biological collection, identification, geo-location and POI registration apparatus according to an embodiment. 46 illustrates a fingerprint, palm print, geo-location, and POI forearm wearable device according to one embodiment. 47 depicts a mobile folding biometric registration kit according to one embodiment. 48 shows a high level system diagram of a biological registration kit according to one embodiment. 49 is a system diagram of a folding biological registration device according to an embodiment. 50 illustrates a thin-film finger and palm print sensor according to one embodiment. 51 depicts a biometric collection device for collecting finger, palm, and enrollment data according to one embodiment. 52 shows a capture of a two stage long print according to one embodiment. 53 shows a capture of a tap of a fingertip according to one embodiment. 54 shows a capture of a slab and roll print according to one embodiment. 55 illustrates a system for taking a contactless fingerprint, palm print, or other biological print in accordance with one embodiment. 56 depicts the process of taking a contactless fingerprint, palm print, or other biological print. 57 shows an embodiment of an eyepiece for optical or digital stabilization. 58 shows a typical camera for a video call or conference. 59 shows one embodiment of a block diagram of a video call camera. 60 shows an embodiment of a classic Casgrangian configuration. Fig. 61 shows the configuration of the folding optical camera of the microcasgrange telescope. 62 illustrates partial image removal by the eyepiece. 63 shows the swipe process to the virtual keyboard. 64 shows a target marker process for a virtual keyboard. 65 shows the electrochromic layer of the eyepiece. 66 illustrates glasses for capturing biological data according to one embodiment. 67 illustrates iris recognition using biological data capture glasses according to one embodiment. 68 illustrates facial and iris recognition according to one embodiment. 69 illustrates the use of a dual omni-microphone according to one embodiment. 70 shows a directional improvement with multiple microphones. 71 illustrates the use of an adaptive array to steer an audio capture facility according to one embodiment. 72 shows a block diagram of a system including an eyepiece.

This disclosure relates to eyepiece electro-optics. The eyepiece includes projection optics suitable for projecting an image into a see-through or translucent lens, wherein the wearer of the eyepiece can view the surrounding environment as well as the displayed image. Projection optics known as projectors may include RGB LED modules using field sequential color. With field sequential color, a single full color image can be separated into color fields based on the primary colors of red, green, and blue and imaged individually by a silicon liquid crystal display (LCoS) optical display 210 . As each color field is imaged by the optical display 210 , the corresponding LED color is turned on. When these color fields are displayed in quick order, a full color image can be seen. With field sequential color illumination, the resulting projected image at the eyepiece can be adjusted for any chromatic aberration by shifting the red image relative to the blue and/or green image, etc. The image can then be reflected back into two surface freeform waveguides where the image light is in Total Reflection (TIR) until it reaches the active viewing area of the lens through which the user views the image. A processor, which may include memory and an operating system, may control the LED light source and optical display. The projector may also include, or be optically coupled to, a display coupling lens, a condenser lens, a polarizing beam splitter, and a field lens.

1 , an exemplary embodiment of an augmented reality eyepiece 100 may be shown. Although embodiments of the eyepiece 100 may not include all elements shown in FIG. 1 , it will be understood that other embodiments may include additional or different elements. In one embodiment, the optical assembly may be embedded in the arm portion 122 of the frame 102 of the eyepiece. An image may be projected with the projector 108 into at least one lens 104 disposed in an aperture of the frame 102 . One or more projectors 108 , such as nanoprojectors, picoprojectors, microprojectors, femtoprojectors, laser-based projectors, holographic projectors, etc., may be disposed on the arm portion of the eyepiece frame 102 . In an embodiment, both lenses 104 are see-through or translucent, while in other embodiments only one lens 104 is translucent while the other lens is opaque or absent. In one embodiment, one or more projectors 108 may be included in the eyepiece 100 .

1 , the eyepiece 100 also absorbs heat from the at least one segmented earbud 120 , the radio transceiver 118 and the LED light engine, and keeps it cool to full brightness. It may include a heat sink 114 to make it operational. There is also a TI OMAP4 (Open Multimedia Applications Processor) 112 , and a flex cable 110 with RF antenna, all of which will be further described herein.

In one embodiment and with reference to FIG. 2 , the projector 200 may be an RGB projector. The projector 200 may include a housing 202 , a heatsink 204 and an RGB LED engine or module 206 . The RGB LED engine 206 may include LEDs, dichroics, concentrators, and the like. A digital signal processor (DSP) (not shown) can convert an image or video stream into a control signal, such as a voltage drop/current modulated, pulse width modulated (PWM) signal, etc. to control the intensity, duration, and mixing of LED light. have. For example, the DSP may control the duty cycle of each PWM signal to control the average current flow through each LED producing a plurality of colors. Eyepiece's still image coprocessor employs noise-filtering, image/video stabilization, and face detection, and is capable of image enhancement. The processor at the audio rear end of the eyepiece may employ buffering, SRC, equalization, and the like.

The projector 200 may include an optical display 210 such as an LCoS display and a number of components as shown. In one embodiment, the projector 200 may be designed as a single panel LCoS display 210 ; A three-panel display is also possible. In a single panel embodiment, the display 210 is sequentially dimmed in red, blue, and green (field sequential color, as is known). In another embodiment, the projector 200 is a backlit liquid crystal display (LCD), a front-lit LCD, a transflective LCD, an organic light emitting diode (OLED), a field emission display (FED), a ferroelectric Alternative optical display technologies such as LCoS (FLCOS) may be used.

The eyepiece may be powered by any power supply, such as battery powered, solar powered, wired powered, or the like. Power may be integrated into the frame 102 , or placed external to the eyepiece 100 and in electrical communication with a powered element of the eyepiece 100 . For example, the solar energy collector may be disposed on the frame 102 , on a belt clip, or the like. Battery charging can occur using a wall charger, vehicle charger, on a belt clip, inside the eyepiece case, and the like.

The projector 200 includes an LED light engine 206, a hollow tapered light tunnel 220 that can be mounted on a heatsink 204 and a holder 208 to ensure a vibration-free mounting for the LED light engine. It may include a diffuser 212 and a condenser lens 214 . The cavity tunnel 220 helps homogenize the rapidly changing light from the RGB LED light engine. In one embodiment, the cavity light tunnel 220 includes a silver coating. The diffuser lens 212 further homogenizes and mixes the light before it reaches the condenser lens 214 . Light exits the condenser lens 214 and then enters a polarizing beam splitter (PBS) 218 . In PBS, the LED light propagates and splits into polarizing components before it is refracted by field lens 216 and LCoS display 210 . LCoS displays provide images for microprojectors. The image is then reflected from the LCoS display back through a polarizing beam splitter and then reflected at 90°. Thus, the image leaves the microprojector 200 approximately halfway through the microprojector. The light then enters a coupling lens 504 described below.

In one embodiment, a digital signal processor (DSP) may be programmed and/or configured to receive the video feed information and set the video feed to drive no matter what type of image source is used as the optical display 210 . The DSP may include a bus or other communication mechanism for communicating information, and an internal processor coupled with the bus for processing the information. DSPs include random access memory (RAM) or other dynamic storage devices (eg, dynamic RAM (DRAM), static RAM (SRAM), and synchronous DRAM (SDRAM)) coupled with a bus to store information and instructions to be executed. The DSP may include memory, for example, read-apply memory (ROM) or other static storage (eg, programmable ROM (PROM)) coupled with a bus to store static information and instructions to the internal processor. , erasable PROM (EPROM)), and electrically erasable PROM (EEPROM)). DSPs include dedicated logic devices (eg, application specific integrated circuits (ASICs)) or configurable logic devices (eg, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays). (FPGA)).

The DSP may include at least one computer-readable medium or memory for holding programmed instructions and for containing data structures, tables, records, or other data necessary to drive an optical display. Examples of computer-readable media suitable for application of this disclosure include compact disks, hard disks, floppy disks, tapes, magneto-optical disks, PROMs (EPROM, EEPROM, flash EPROM), DRAM, SRAM, SDRAM, or other magnetic media; A compact disk (eg, CD-ROM), or other optical medium, punch card, paper tape, or other physical medium having a pattern of holes, a carrier wave (described below), or other computer-readable medium can be Various forms of computer-readable media may be incorporated into the optical display 210 for execution in execution of one or more sequences of one or more instructions. The DSP may also include a communication interface to provide data communication coupled to a network link that may be coupled to, for example, a LAN, or other communication network, such as the Internet. A wireless link may also be implemented. In such implementations, a suitable communication interface may send and receive electrical, electromagnetic, or optical signals that convey digital data streams representing various types of information (such as video information) to the optical display 210 .

In another embodiment, Figures 21 and 22 show alternative positions of the waveguide and projector in exploded views. In this arrangement, the projector is placed just behind the hinge of the arm of the eyepiece, it is oriented vertically so that the initial travel of the RGB LED signal is vertical until the direction is changed by the reflective prism to enter the waveguide lens. do. A vertically positioned projection engine may have a PBS 218 in the center, an RGB LED array at the bottom, a hollow tapered tunnel with a thin film diffuser to mix colors for focusing in optics, and a condenser lens. PBS has a pre-polarizer on the entrance face. The pre-polarizer may be arranged to transmit light of a particular polarization, such as p-polarized light, and to reflect (or absorb) light of the opposite polarization, such as s-polarized light. The polarized light then passes through the PBS to the field lens 216 . The purpose of the field lens 216 may be to create adjacent telecentric illumination of the LCoS panel. The LCoS display is precisely reflective and reflects the colors sequentially with the correct timing, so that the image is displayed properly. For bright areas of the image, light can be reflected from the LCoS panel and rotated to s-polarization. The light is then refracted through the field lens 216, reflected at the internal interface of the PBS, exits the projector, and is directed to the coupling lens. The cavity tapered tunnel 220 may replace a homogeneous lenslet from another embodiment. By orienting the projector vertically and centering the PBS, space is saved and the projector can be placed in a hinged space with a small moment arm suspended from the waveguide.

Light entering the waveguide can be polarized, such as s-polarized. When this light is reflected from the user's eyes, it can be seen as a "luminous" from the user's eyes. This luminescence can be removed by attaching a lens to a frame or waveguide that is counter-polarized from light reflected from the user's eye, such as the snap-fit described herein, in this case p-polarized.

21-22 , augmented reality eyepiece 2100 includes a frame 2102 and left and right earpieces or temple pieces 2104 . A protective lens 2106, such as a ballistic lens, is mounted to the front of the frame 2102 to protect the user's eyes or, if they are prescription lenses, correct the user's view of the environment. The front view of the frame may also be used to mount a camera or image sensor 2130 and one or more microphones 2132 . Although not visible in FIG. 21 , a waveguide is mounted behind a protective lens 2106 on each side of a central or adjustable nose bridge 2138 . The front cover 2106 may be interchangeable, allowing the tint or prescription to change dramatically for a particular user of the augmented reality device. In one embodiment, each lens may be rapidly interchangeable, allowing for different prescriptions for each eye. In one embodiment, the lens is quickly replaceable with the snap-fit disclosed herein. Certain embodiments may have only a combination of a projector and a waveguide on one side of the eyepiece, while the other side may be filled with regular lenses, reading lenses, prescription lenses, and the like. Each of the left and right ear pieces 2104 vertically mounts a projector or microprojector 2114 or other image source on top of a spring-loaded hinge 2128 for easier assembly and vibration/shock protection. Each temple piece also includes a temple housing 2116 for mounting the associated electronics relative to the eyepiece, and each can also include an elastomeric head grip pad 2120 for better retention to the user. Each temple piece also includes an orifice 2126 for mounting an extended, wraparound ear bud 2112 and a headstrap 2142 .

As noted above, the temple housing 2116 contains the electronics associated with the augmented reality eyepiece. The electronics may include a microprocessor and a number of circuit boards, such as a radio 2122 , a communications system on a chip (SOC) 2124 , and an Open Multimedia Applications Processor (OMAP) processor board 2140 , as shown. have. A communications system-on-a-chip (SOC) is an electronic device for one or more communications functions, including wide area networks (WLAN), Bluetooth communications, frequency modulated (FM) radio, global positioning system (GPS), three-axis accelerometer, one or more gyroscopes, and the like. device may be included. Additionally, the right temple piece may include an optical trackpad (not shown) on the outer surface of the temple piece for user control of the eyepiece and one or more applications.

Frame 2102 is a pair of wraparound sunglasses in a general shape. The sides of the glasses include shape memory alloy straps 2134, such as nitinol straps. A nitinol or other shape-memory alloy strap is fitted to the augmented reality eyepiece user. The straps are tailored so that they assume their trained or desired shape when worn by the user and warmed to close to body temperature.

Another feature of this embodiment includes removable, noise-cancelling earbuds. As shown in the figure, the earbuds are intended to connect to the controls of the augmented reality eyepiece to deliver sound to the user's ear. The sound may include input from the wireless internet or the telecommunication function of the augmented reality eyepiece. The earbuds include a soft, deformable plastic or foam portion to protect the inside of the user's ear in a manner similar to earplugs. In one embodiment, the earbuds limit input to the user's ear by about 85 dB. This allows for the wearer's normal hearing while providing protection from gunshots or other explosive noises. In one embodiment, the control of the noise-cancelling earbuds has an automatic gain control for very quick adjustment of the cancellation feature when protecting the wearer's ears.

23 shows the layout of a vertically positioned projector 2114, where the illumination light passes one side of the PBS on its path from the bottom to the top, the display and imager backed by silicon. It passes to the (imager) board, is refracted as image light (where the light strikes the inner surface of the triangular prism constituting the polarizing beam splitter), and is reflected from the projector to the waveguide lens. In this example, the size of the projector is shown to have a width of 11 mm of the imager board, a distance of 10.6 mm from the end of the imager board to the image center line, and a distance of 11.8 mm from the image center line to the end of the LED board.

A detailed and assembled view of the components of the projector described above can be seen in FIG. 25 . This figure shows how compact the microprojector 2500 is when assembled, for example, adjacent the hinge of the augmented reality eyepiece. Microprojector 2500 includes a housing and holder 208 for mounting specific optical pieces. As each color field is imaged by the optical display 210, the corresponding LED color is lit. An RGB LED light engine 202 is shown adjacent the floor and mounted on a heatsink 204 . The holder 208 is mounted on top of the LED light engine 202 , the holder mounted light tunnel 220 , the diffuser lens 212 (to remove hot spots) and the condenser lens 214 . Light passes from the condenser lens through a polarizing beam splitter 218 to a field lens 216 . The light is then refracted to an LCoS (silicon liquid crystal display) chip 210, where an image is formed. The light for the image is then reflected back through field lens 216 , polarized and reflected at 90° through polarizing beam splitter 218 . The light then exits the microprojector for transmission to the optical display of the glasses.

26 shows an example RGB LED module. In this example, the LEDs are a 2x2 array, with 1 red, 1 blue and 2 green dies, and the LED array has 4 cathodes and a common anode. The maximum current is 0.5A per die, and the highest voltage (<img file="KR20130000401A_D0001.tif" />4V) may be required for the green and blue dies.

Fig. 3 shows an embodiment of a projector that is placed horizontally in use. The projector 300 may be disposed on the arm portion of the eyepiece frame. The LED module 302, under processor control 304, can emit a single color at a time in a fast sequence. The emitted light travels down into a light tunnel 308 where it strikes a polarizing beam splitter 312 and at least one homogenizing lenslet 310 before being redirected towards an LCoS display 314 where a full color image is displayed. pass through The LCoS display has a resolution of 1280x720p. The image is then reflected back through the polarizing beam splitter, reflected at the fold mirror 318, and travels through the collimator on its path from the projector to the waveguide. The projector may include a diffractive element to eliminate aberrations.

In one embodiment, the interactive head-mounted eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content, the optical assembly comprising a corrective element for correcting the user's view of the surrounding environment, internal reflection a free-form waveguide that enables The eyepiece further includes an integrated processor for handling content for display to a user, and an integrated image source, such as a projector facility, for directing the content to the optical assembly. In embodiments where the image source is a projector, the projector equipment includes a light source and an optical display. Light from a light source, such as an RGB module, is emitted under the control of a processor and in certain other embodiments traverses a polarizing beam splitter to an optical waveguide where the light is polarized before being reflected off an optical display, such as an LCoS display or LCD display. The surface of the polarizing beam splitter may reflect the color image from the optical display to the optical waveguide. The RGB LED module can emit light sequentially to form a color image that is reflected from the optical display. The correction element may be a see-through correction lens attached to the optical waveguide to enable proper viewing of the surrounding environment whether the image source is on or off. Such corrective lenses may be wedge-shaped corrective lenses and may be prescription, tinted, coated, or the like. A freeform optical waveguide, which may be described as a higher order polygon, may include a double freeform surface that allows for curvature and sizing of the waveguide. The curvature and sizing of the waveguide makes it possible to place itself in the frame of the interactive head-mounted eyepiece. These frames may be sized to fit the user's head in a manner similar to sunglasses or single-piece glasses. Other elements of the optical assembly of the eyepiece include a homogenizer through which light from the light source propagates to ensure that the light beam becomes homogeneous and a collimator that improves the resolution of the light entering the optical waveguide.

4 , image light, which may be polarized and collimated, may optionally traverse a display coupling lens 412 , which may or may not be the collimator itself or in addition to the collimator, and a waveguide 414 . go into In an embodiment, waveguide 414 may be a free-form waveguide, wherein the surface of the waveguide is described as a polynomial. The waveguide may be a straight line. The waveguide 414 may include two reflective surfaces. As the image light enters the waveguide 414, it may strike the first surface at an angle of incidence greater than the critical angle at which total reflection (TIR) occurs. The image light engages a TIR bounce between the first and second surfaces and consequently reaches the active viewing area 418 of the compound lens. In one embodiment, the light may engage at least three TIR bounces. The thickness of the composite lens 420 may not be homogeneous because the waveguide 414 is tapered so that the TIR bounce may eventually leave the waveguide. Distortion through the viewing area of the composite lens 420 is reduced by disposing a wedge-shaped corrective lens 410 along the length of the freeform waveguide 414 to provide a uniform thickness over at least the viewing area of the lens 420 . can be minimized. The corrective lens 410 may be a prescription lens, a tinted lens, a polarized lens, a ballistic lens, or the like.

In some embodiments, while an optical waveguide has a first surface and a second surface that enable total reflection of light entering the waveguide, light may not actually enter the waveguide at an internal angle of incidence that causes total reflection. The eyepiece may include a mirrored surface on the first surface of the optical waveguide to reflect the displayed content towards the second surface of the optical waveguide. Thus, the mirrored surface enables total reflection of light entering the optical waveguide or reflection of at least a portion of the light entering the optical waveguide. In embodiments, the surface may be 100% mirrored or mirrored at a lower rate. In some embodiments, at the location of the mirrored surface, an air gap between the waveguide and the correction element may cause reflection of light entering the waveguide at an angle of incidence that does not result in TIR.

In one embodiment, the eyepiece includes an integrated image source, such as a projector, that directs content for display to an optical assembly from the side of the optical waveguide adjacent the arm of the eyepiece. In contrast to prior art optical assemblies where image injection occurs from the top side of the optical waveguide, the present disclosure provides image injection into the waveguide from the side of the waveguide. The aspect ratio of the displayed content is between a nearly rectangular and a nearly square with a nearly horizontal longitudinal side. In an embodiment, the aspect ratio of the displayed content is 16:9. In embodiments, reaching a rectangular aspect ratio for displayed content where the longitudinal axis is nearly horizontal may be achieved through rotation of the injected image. In other embodiments, it may be achieved by enlarging the image until the desired aspect ratio is reached.

5 shows a design for a waveguide eyepiece showing sample size. For example, in this design, the width of the coupling lens 504 is 13-15 mm, and the optical display 502 is optically coupled in series. These elements are disposed on the arm of the eyepiece. Image light from the optical display 502 is projected through a coupling lens 504 into a freeform waveguide 508 . The thickness of the composite lens 520 including the waveguide 508 and the correction lens 510 may be 9 mm. In this design, the waveguide 502 enables an exit pupil diameter of 8 mm and eye clearance of 20 mm. The resulting see-through view 512 may be about 60-70 mm. The distance (size a) from the pupil to the image light path when light enters the waveguide 502 can be about 50-60 mm, which can accommodate a significant percentage of the width of a human head. In one embodiment, the field of view may be larger than the pupil. In one embodiment, the field of view cannot fill the lens. It should be understood that these sizes are for specific example embodiments and should not be considered limiting. In one embodiment, the waveguide, snap-on optics, and/or corrective lens may comprise optical plastic. In other embodiments, the waveguide snap-on optics and/or corrective lens may include glass, marginal glass, bulk glass, metallic glass, palladium-rich glass, or other suitable glass. In an embodiment, the waveguide 508 and corrective lens 510 may be made of different materials selected to have little or no chromatic aberration. Materials may include diffraction gratings, holographic gratings, and the like.

1 , the projected image may be a stereo image when two projectors 108 are used for left and right images. To enable stereo viewing, the projectors 108 may be positioned at an adjustable distance from each other, which may be adjusted based on the interpupillary distance for the individual wearer of the eyepiece.

While describing particular embodiments of the eyepiece, various additional features, applications for use 4512, control technology and external control device 4508, associated external device 4504, software, networking functions, integrated sensors 4502, An external processing facility 4510 , an associated third party facility 4514 , and the like are described. External devices 4504 for use as an eyepiece include devices useful for entertainment, navigation, computing, communications, weaponry, and the like. External control device 4508 may include ring/hand or other haptic controllers, external devices that enable gesture control (eg, non-integrated cameras, devices with built-in accelerometers), I/F to external devices, and the like. include External processing facilities 4510 include local processing facilities, remote processing facilities, I/F to external applications, and the like. Applications for use 4512 include applications for commercial, consumer, military, educational, government, augmented reality, advertising, media, and the like. Various third party facilities 4514 may be accessed by or work in conjunction with the eyepiece. The eyepiece 100 may interact with the other eyepiece 100 through wireless communication, near magnetic field communication, wired communication, or the like.

6 shows an embodiment of an eyepiece 600 with a see-through or translucent lens 602 . Projected image 618 is visible on lens 602 . In this embodiment, the image 618 projected by the lens 602 happens to be an augmented reality version of the scene viewed by the wearer, where the tag point of interest (POI) of the field of view is displayed to the wearer. The augmented reality version may be enabled by a forward facing camera built into the eyepiece (not shown in FIG. 6 ) that images what the wearer is looking at and identifying the location/POI. In one embodiment, the output of the camera or optical transmitter may be sent to an eyepiece controller or memory for storage for transmission to a remote location, or for viewing by a person wearing the eyepiece or glasses. For example, the video output may be streamed to a virtual screen viewed by the user. The video output can thus be used to help determine the location of the user, or it can be transmitted remotely or for other purposes to others to assist in locating the wearer's location. Other detection technologies, such as GPS, RFID, manual input, and the like, may be used to determine the location of the wearer. When using location or identification data, the database can be accessed by the eyepiece for information that can be displayed as overlying, projected, or viewed. Augmented reality applications and techniques will be further described herein.

In FIG. 7 , an eyepiece 700 with a translucent lens 702 and an incoming call notification on a displayed streaming medium (e-mail application) is shown. In this embodiment, the medium obscures a portion of the viewing area, but it should be understood that the displayed image may be placed anywhere in the field of view. In embodiments, the medium may be made to be somewhat transparent.

In one embodiment, the eyepiece may receive input from an external source, such as an external converter box. The source may be shown in the lens of the eyepiece. In one embodiment, when the external source is a phone, the eyepiece may use the phone's location capabilities to display location-based augmented reality, including marker overlays from marker-based AR applications. In embodiments, a VNC client running on the eyepiece's processor or associated device may be used to connect to and control a computer, where the computer's display is viewed by the wearer on the eyepiece. In one embodiment, content from any source may be streamed to the eyepiece, such as a display from a panoramic camera mounted on top of a vehicle, a user interface to the device, images from a drone or helicopter, and the like. For example, a gun-mounted camera could fire a target that is not in a straight line of sight when the camera feed is directed to the eyepiece.

The lenses may be photochromic or chromic, such as electrochromic. The electrochromic lens may include an essential chromic material or chromic coating that changes the opacity of at least a portion of the lens in response to a burst of charge applied by the processor throughout the chromic material. For example, and with reference to FIG. 65 , the chromic portion 6502 of the lens 6504 is directed toward providing greater viewability by the wearer of the eyepiece when that portion presents the displayed content to the wearer. As such, it is shown in dark colors. In embodiments, a plurality of chromic regions on a lens that can be independently controlled, such as multiple portions of a lens, a sub-portion of a projected area, a programmable area of a lens and/or a projected area, controlled at the pixel level, etc. This can be. Activation of the chromic material may be controlled via a control technique as further described herein, or automatically enabled for a specific application (eg, streaming video application, sun tracking application), or in response to a built-in frame UV sensor. can be controlled by The lens may have an angle-sensitive coating with a high angle of incidence, allowing the transmission of light-waves with low angles of incidence and reflected light, such as s-polarized light. The color-changeable coating may be controlled in part or in whole, such as by the control techniques described herein. The lens may be of variable contrast. In an embodiment, the user wears an interactive head-mounted eyepiece, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content. The optical assembly may include a corrective element that corrects the user's view of the surrounding environment, an integrated processor that handles content to display to the user, and an integrated image source that directs the content to the optical assembly. The optical assembly may include an electrochromic layer that provides adjustment of display properties dependent on the displayed content requirements and ambient environmental conditions. In an embodiment, the display characteristic may be luminance, contrast, or the like. Ambient environmental conditions may have luminance levels that make it difficult to visualize the displayed content by the wearer of the eyepiece without adjusting the display characteristics, where the display characteristic adjustments may be applied to areas of the optical assembly where the content is displayed.

In embodiments, the eyepiece may have luminance, contrast, spatial, resolution, etc. controls over the projected area of the eyepiece to alter and improve the user's view of the projected content relative to a light or dark surrounding environment. For example, the user may be using the eyepiece under bright daylight conditions, and the display area may need to be changed in brightness and/or contrast in order for the user to clearly see the displayed content. Alternatively, the viewing area surrounding the display area may be changed. Additionally, the altered area, whether within the display area or not, may be spatially oriented or controlled according to the application being implemented. For example, only a small portion of the display area may need to be changed when that portion of the display area deviates from some predetermined or predetermined contrast between the display portion of the display area and the surrounding environment. In embodiments, fixed to cover the entire display area, adjusted for only a portion of the lens, adaptable and dynamic to changing lighting conditions in the ambient environment and/or luminance-contrast in the displayed content, etc. It can be changed in luminance, contrast, spatial extent, resolution, and the like. The spatial extent (e.g., area affected by the change) and resolution (e.g., display optical resolution) are different segments for the application running, including high resolution segments, low resolution segments, single pixel segments, etc. It can vary for different parts of the lens, which can be combined to achieve seeing the object of(s). In an embodiment, the technology implementing the change in brightness, contrast, spatial range, resolution, etc. is an electrochromic material, LCD technology, beads embedded in optics, flexible display, suspension particle device (SPD) technology, colloidal technology, and the like.

In embodiments, there may be various modes of activation of the electrochromic layer. For example, the user may enter a sunglasses mode in which the composite lens appears only slightly dark, and may enter a "blackout" mode in which the composite lens is completely dark.

Examples that may be employed when implementing changes in brightness, contrast, spatial range, resolution, etc. may include an electrochromic material, a film, an ink, and the like. Electrochromism is a phenomenon exhibited by some materials that reverse their appearance when an electrical charge is applied. Various types of materials and structures may be used to construct the electrochromic device depending on the particular application. For example, the electrochromic material is tungsten oxide (WO), the main chemical used in the production of electrochromic windows or smart glasses.₃) is included. In an embodiment, an electrochromic coating may be used on the lens of the eyepiece in an alternative implementation. In another example, an electrochromic display may be used in an 'electronic paper' implementation, which is designed to mimic the outward appearance of a typical jug, where the electronic paper displays reflected light similarly to normal paper. In an embodiment, the electrochromic properties include a gyricon (consisting of a polyethylene sphere embedded in a transparent silicone sheet, each sphere suspended in an oil of oil so that it can rotate freely), an electro-moving display (applied used electric field to reposition charged pigment particles to form an image), E-ink technology, electrowetting, electro-fluidic, interference modulators, organic transistors embedded into flexible substrates, It can be implemented in a variety of applications and materials, including nano-chromic displays (NCDs) and the like.

Another example of a technique that may be employed in implementing alternatives such as brightness, contrast, spatial range, resolution, etc. may be a floating particle device (SPD). When a small voltage is applied to the SPD film, its microscopic particles, which are randomly scattered in their steady state, are aligned and allow light to pass through. The response can be immediate, uniform, and color stable throughout the film. Voltage regulation may allow the user to control the amount of light, flash, and heat passing through. The system response can range from a royal blue appearance that completely blocks light in its off state to a transparent appearance in its on state. In an embodiment, the SPD technique may be an emulsion applied onto a plastic substrate in the creation of an active film. These plastic films may be laminated (as a single sheet of glass) and suspended between two sheets of glass, plastic or other transparent material, etc.

Referring to Figure 8, in a particular embodiment, the electro-optical device is a monocular or two-part 1) electro-optical device; and 2) a binocular flip-up/flip-down arrangement with corrective lenses. 8A shows a two-part eyepiece contained within a module 802 in which electro-optical devices may be electrically connected to the eyepiece via electrical connectors 810 such as plugs, pins, sockets, wires, and the like. In this arrangement, the lenses 818 in the frame 814 may be entirely corrective lenses. The distance (IPD) between the two lens centers between the two halves of the electro-optic module 802 may be adjusted in the bridge 808 to accommodate various IPDs. Similarly, the placement of the display 812 may be adjusted via the bridge 808 . FIG. 8B shows a binocular electro-optical device module 802 with one half flipped up and the other half flipped down. The nose bridge is fully adjustable and elastomeric. In one embodiment, the lens 818 can be an ANSI-compliant, hard-coated scratch-resistant polycarbonate ballistic lens, can be chromic, have an angle-sensitive coating, include a UV sensitive material, and the like. can do.

As discussed in FIG. 8 , the augmented reality glasses may include a lens 818 for each eye of the wearer. Lenses 818 may be made to easily fit into frame 814 such that each lens is tailored for the person for which the glasses are intended. Thus, the lens can be a corrective lens, can also be tinted for use as sunglasses, or have other qualities suitable for the intended environment. Thus, the lens may be tinted to yellow, black, or other suitable color, or may be photochromic, such that the lens' transparency decreases when exposed to brighter light. In one embodiment, the lens may also be designed to snap fit into the frame, ie, a snap on lens is one embodiment.

Of course, the lens need not be a corrective lens; They can function just as sunglasses or as protection for the optical system in the frame. In a non-flip-up/flip-down arrangement, it goes without saying that the outer lens helps protect the more expensive waveguides, and is important for protecting the electronics and systems within the augmented reality glasses. At a minimum, the outer lens provides protection from scratches by the user's environment in one environment, weather sand, shrubbery, gas, etc., and flying debris, cartridge casings and shrapnel debris in another environment. Additionally, the outer lens can be decorative, perhaps with the action of changing the shape of the synthetic lens to appeal to the user's personality or fashion sense. External lenses may also help one individual user to distinguish their glasses from other users, for example when many users are gathered together.

Preferably, the lens is suitable for impacts such as ballistic impacts. Thus, in one embodiment, the lens and frame meet ANSI standard Z87.1-2010 for ballistic resistance. In one embodiment, the lens also meets the ballistic standard CE EN166B. In another embodiment, for military use, the lens and frame may meet the criteria of MIL-PRF-31013, standard 3.5.1.1 or 4.4.1.1. Each of these standards has slightly different requirements for ballistic resistance, and each is intended to protect the user's eyes from the effects of high-speed projectiles or debris. Unless a specific material is specified, a specific Lexan? Polycarbonate such grades are generally sufficient to pass the tests specified in the appropriate standards.

In one embodiment, as shown in FIG. 8D , the lens snaps in from the outside of the frame, rather than the inside, for better impact resistance because any influence from the outside of the augmented reality glasses is predicted. In this embodiment, the replaceable lens 819 has a plurality of snap-fitting arms 819a that fit into the recesses 820a of the frame 820 . The engagement angle 819a of the arm is greater than 90°, but the engagement angle 820b of the recess is also greater than 90°. Making the angle greater than a right angle has the practical effect of allowing removal of lens 819 from frame 820 . Lens 819 may need to be removed if the person's vision changes or a different lens is desired for any reason. The design of the snap fitting allows for some compression or bearing load between the lens and the frame. That is, the lens can be held securely in the frame, such as by slightly interference fit the lens within the frame.

The cantilever snap fit of Figure 8 is not the only possible way to removably snap-fit the lens and frame. For example, an annular snap fit may be used, in which the continuous sealing lip of the frame engages the enlarged edge of the lens, which then snap fits to or onto the lip. These snap fittings are commonly used to attach the cap to the ink pen. Such a configuration may have the advantage of a tighter bond with very little chance of entry of very small dust and dust particles. Possible disadvantages include very tight tolerances required all around the lens and frame, and the requirement for dimensional integrity in all three dimensions over time.

A simpler interface, which can be considered a snap-fit, may also be used. A groove is molded into the outer surface of the frame, and the lens has a protruding surface, which can be considered a tongue fitting into the groove. When the groove is semi-cylindrical, such as from about 270° to about 300°, the barrel snaps into the groove to remain fixed and still be removable through the gap remaining in the groove. In this embodiment, as shown in FIG. 8E , a cover 826 with a lens or replacement lens or barrel 828 is provided with a groove 827 in the frame 825 even if the lens or cover is not snap-fitting into the frame. can be inserted into Because the fit is nearly customizable, it functions as a snap-fit and keeps the lens fixed to the frame.

In another embodiment, the frame may consist of two parts, a lower and an upper, with conventional barrel and groove fittings. In another embodiment, this design may also utilize standard fasteners to ensure a tight grip of the lens by the frame. The design does not need to separate anything inside the frame. Accordingly, snap-on or other lenses or covers must be assembled onto, or removed from, the frame without having to go inside the frame. As disclosed elsewhere in this disclosure, augmented reality glasses have multiple component parts. Some of the assemblies and sub-assemblies may require careful alignment. Moving these assemblies and jarring can cause problems with their function, just like moving the frame and external or snap-on lenses or covers and causing them to vibrate.

In an embodiment, the flip-up/flip-down arrangement allows for a modular design for the eyepiece. For example, not only may the eyepiece be equipped with a monocular or binocular module 802 , the lens 818 may also be replaced. In embodiments, additional features may be included in module 802 associated with one or both displays 812 . For example, a monocular or binocular version of module 802 may be only a monocular 902 or binocular 904 display, or may be equipped with a forward-facing camera 908 (monocular), and binocular 910 and 912 . In some embodiments, the module may have additional integrated electronics such as GPS, laser range finder, etc. In an embodiment 912, the binocular electro-optic module 912 is a stereo Equipped with a forward-facing camera 920 and a laser range finder 918 .

In one embodiment, the electro-optical properties are as follows, which is not meant to be limiting:

<img file="KR20130000401A_D0002.tif" />

In one embodiment, the projector characteristics are as follows:

<img file="KR20130000401A_D0003.tif" />

In another embodiment, the augmented reality eyepiece may include an electrically-controlled lens as part of the microprojector, or as part of the optics between the microprojector and the waveguide . 21 shows an embodiment with such a liquid lens 2152.

The glasses also include at least one camera or optical sensor 2130 that provides an image or images viewed by the user. An image is formed by a microprojector 2114 on each side of the glasses for delivery to a waveguide 2108 on that side. In one embodiment, an additional optical element, variable focus lens 2152 is also provided. The lens is electrically adjustable by the user so that the image viewed in waveguide 2108 is focused for the user.

Variable lenses may include so-called liquid lenses provided by LensVector, Inc., Mountain View, CA, USA, or by Varioptic, SA of Lions, France. These lenses contain a core with two immiscible liquids. Generally, in these lenses, the light path through the lens, ie, the focal length of the lens, is changed, or focused by applying an electrical potential between electrodes immersed in a liquid. At least one liquid is affected by the resulting electric or magnetic field potential. Accordingly, electrowetting may occur, as described in US Patent Publication 2010/0007807 assigned to LensVector, Inc. Other techniques are described in LensVector's patent application publications 2009/021331 and 2009/0316097. All three of these disclosures are incorporated herein by reference, although each page and figure has been described as such in the text.

Another patent document from Varioptic, SA describes other devices and techniques for varifocal lenses, which also work through the electrowetting phenomenon. These documents include U.S. Patent Nos. 7,245,440 and 7,894,440, and U.S. Patent Application Publications 2010/0177386 and 2010/0295987, incorporated herein by reference, although each page and figure is incorporated herein by reference. In these documents, two liquids generally have different refractive indices and different electrical conductivity, for example, one liquid is conductive, such as an aqueous liquid, and the other liquid is insulating, such as an oleaginous liquid. When the electric potential is applied, the thickness of the lens is changed and the path of light passing through the lens is changed, thereby changing the focal length of the lens.

The electrically-adjustable lens can be controlled by the control of the spectacles. In one embodiment, focusing is done by calling a menu from the control and adjusting the focus of the lens. The lenses may be individually controlled or controlled together. Adjustments are made by physically turning on the control knobs, by gestures, or by voice commands. In another embodiment, the augmented reality glasses also include a range finder, and the focus of the electrically adjustable lens can be controlled automatically by pointing a range finder, such as a laser range finder, to an object at a desired distance from the target or user.

As discussed above in US Pat. No. 7,894,440, variable lenses may also be applied to augmented reality glasses or external lenses of the eyepiece. In one embodiment, the lens may merely replace the corrective lens. A variable lens with its own electrically-adjustable controls may be used in place of or in addition to an image source or projector-mounted lens. Whether the waveguide display is active or not, the corrective lens insert provides corrective optics for the user's environment and the outside world.

It is important to stabilize the image presented to the wearer of the augmented reality glasses or eyepiece(s), ie the image seen in the waveguide. The views or images presented are taken from one or two digital cameras or sensors mounted on the eyepiece to digital circuitry where the images are processed and, if desired, stored as digital data before they are displayed on the display of the glasses. In either case, and as discussed above, the digital data is then used to form an image using an LCOS display and a series of RGB light emitting diodes. The optical image is processed using a series of lenses, a polarizing beam splitter, an electrically-powered liquid correction lens, and at least one transition lens from the projector to the waveguide.

The process of gathering and presenting images involves a number of mechanical and optical couplings between components of the augmented reality glasses. Therefore, it is clear that some form of stabilization will be required. Since it is itself a mobile platform that is movably mounted to a mobile user, mounted on the glasses, this involves the most direct cause, the optical stabilization of the camera itself. Therefore, camera stabilization or calibration may be required. Additionally, at least some stabilization and correction must be used for liquid variable lenses. Ideally, the stabilization circuit at that point could not only correct for the liquid lens, but also correct for any aberrations and vibrations from multiple parts of the circuit upstream from the liquid lens containing the image source. One advantage of the present system is that many commercial off-the-shelf cameras are very advanced and generally have at least one image-stabilizing feature or option. Accordingly, there may be multiple embodiments of the present disclosure, each with the same or different methods of stabilizing an image or a very fast stream of images, as described below. The term optical stabilization is generally used in the text to mean physically stabilizing a camera, camera platform, or other physical object, whereas image stabilization refers to data manipulation and processing.

When a digital image is formed, one technique of image stabilization is performed on the digital image. These techniques may use pixels outside the bounds of the visible frame as buffers for unwanted operation. Also, the technique can be used as another stable area or reference in successive frames. This technique is applicable to video cameras and shifts the electronic image from frame to frame in a manner sufficient to compensate for motion. This technique does not rely on sensors and directly stabilizes images by reducing vibrations and other distracting movements from moving cameras. In some techniques, the speed of the image is slowed to add a stabilization process to the rest of the digital process, and may require more time per image. These techniques can use global motion vectors computed from frame-to-frame motion differences to determine the direction of stabilization.

Optical stabilization of an image uses gravity or an electrically-driven mechanism to move or manipulate an optical element or imaging sensor to cancel ambient vibrations. Another way to stabilize the displayed content is to provide sensing of the platform or gyroscope calibration, for example providing the user with augmented reality glasses. As mentioned above, sensors that can be used and used in augmented reality glasses or eyepieces include MEMS gyroscope sensors. These sensors capture movement and movement in three dimensions in very small units and can be used as feedback to correct the images transmitted from the camera in real time. It is clear that at least a large part of this unwanted and undesirable movement is probably due to the movement of the user and the camera itself. These larger movements may include the total movement of the user, for example, walking or running, driving a vehicle, and the like. Smaller vibrations can also occur within the augmented reality glasses, ie vibrations in the components of the electrical and mechanical coupling that form a path from the camera (input) to the image in the waveguide (output). This total motion may be more important to correct or account for than, for example, independent and small motions in the coupling of components downstream from the projector.

Motion detection can thus be used to detect motion and compensate for it, as in optical stabilization, or to calibrate an image that is captured and processed after detecting motion, as in image stabilization. A device for detecting motion and correcting an image or data is shown in Fig. 57a. In the present device, one or more types of motion sensors including angular position sensors or gyroscopes, such as accelerometers, MEMS gyroscopes, may be used. Data from the sensor may be provided to a suitable sensor interface, such as an analog-to-digital converter (ADC), or other suitable interface, such as a digital signal processor (DSP). The microprocessor then processes this information as described above and sends an image-stabilized frame to the display driver and then to the see-through display or waveguide described above. In one embodiment, the display begins with an RGB display in the microprojector of the augmented reality eyepiece.

In another embodiment, video sensors or augmented reality glasses, or other devices with video sensors, may be mounted on the vehicle. In this embodiment, the video stream may be delivered to a communication function or an Internet function to personnel in the vehicle. One application could be local tourism or tours. Another embodiment could be a local tour, expedition, or even reconnaissance. In this embodiment, gyroscope stabilization of the image sensor would be more helpful than applying a gyroscope correction to the image or digital data representing the image. An embodiment of the present technique is shown in FIG. 57B. In the present technology, a camera or image sensor 3407 is mounted on a vehicle 3401 . One or more motion sensors 3406 , such as a gyroscope, are mounted to the camera assembly 3405 . The stabilization platform 3403 receives information from the motion sensor and stabilizes the camera assembly 3405 so that jitter and shake are minimized as the camera operates. This is true optical stabilization. Alternatively, the motion sensor or gyroscope may be mounted on or within the stabilization platform itself. These techniques actually provide optical stabilization, stabilizing the camera or image sensor, which, as opposed to digital stabilization, later modifies the image by computer processing of data captured by the camera.

In one technique, the key to optical stabilization is to apply stabilization or correction before the image sensor converts the image into digital information. In one technique, feedback from a sensor, such as a gyroscope or angular velocity sensor, is encoded and sent to an actuator that moves the image sensor, much like an autofocus mechanism adjusts the focus of a lens. The image sensor is moved in such a way as to maintain the projection of the image into the image plane as a function of the focal length of the lens used. Autoranging and focus autoranging from the rangefinder of the interactive head-mounted eyepiece can be obtained through the lens itself. In another technology, an angular velocity sensor, sometimes referred to as a gyroscope sensor, may be used to sense horizontal and vertical movement, respectively. The sensed movement can then be fed back to the electromagnet to move the camera's floating lens. This optical stabilization technique, however, must be applied to each lens which is intended to make the result very expensive.

Stabilization of liquid lenses is disclosed in US Patent Application Publication No. 2010/0295987 assigned to Varioptic, SA of Lyon, France. In theory, the control of a liquid lens is relatively simple, since there is only one variable to control: the level of voltage in the conductive and non-conductive liquid of the lens is, for example, using the lens housing and cap as electrodes. applied to the electrode. Applying a voltage causes a change or tilt of the liquid-liquid interface through an electrowetting effect. This change or tilt adjusts the focus or output of the lens. In its most basic aspect, the control scheme with feedback then applies a voltage and determines the effect of the applied voltage on the result, ie the focus or astigmatism of the image. The voltages may be applied in a pattern such as, for example, equal and opposite + and - voltages, positive voltages on both sides of different magnitudes, negative voltages on both sides of different magnitudes, and the like. Such lenses are known as electrically variable optical lenses or electro-optical lenses.

Voltage may be applied to the electrodes for a short period of time and in a pattern to check for focal or astigmatism produced. The check can be made, for example, by an image sensor. Additionally, a sensor on the camera, or in this case a lens, may detect movement of the camera or lens. Motion sensors include accelerometers, gyroscopes, angular velocity sensors, or piezoelectric sensors mounted on a liquid lens or part of an optical train in close proximity to the liquid lens. In one embodiment, a table, such as this calibration table, is then configured with the applied voltage and the degree of correction or voltage required for a given level of motion. More sophistication can be added, for example, by using segmented electrodes in different parts of the liquid so that four voltages instead of two can be applied. Of course, if four electrodes are used, four voltages can be applied in more patterns than with only two electrodes. Such a pattern may include the same positive and negative voltages and opposite positive and negative voltages for opposing segments and the like. An example is shown in FIG. 57C. The four electrodes 3409 are mounted inside a liquid lens housing (not shown). Two electrodes are mounted on or adjacent to the non-conductive liquid, and two are mounted on or adjacent to the conductive liquid. Each electrode is independent in terms of the possible voltages that can be applied.

A look-up or calibration table may be constructed and placed in the memory of the augmented reality glasses. In use, the accelerometer or other motion sensor detects movement of the glasses, i.e. the movement of the camera on the glasses or the lens itself. Motion sensors, such as these accelerometers, specifically detect small vibration types of motion that interfere with the smooth transfer of images to the waveguide. In one embodiment, the image stabilization techniques described herein can be applied to electrically-controllable liquid lenses such that the image from the projector is instantly corrected. This stabilizes the output of the projector, while at least partially correcting the vibration and movement of the augmented reality eyepiece, as well as at least some movement by the user. There may also be manual controls for adjusting the gain or other parameters of the calibration. This technique can also be used to correct for near-sightedness or far-sightedness of an individual user, in addition to the focus adjustment provided by the image sensor control and discussed as part of an adjustable-focus projector. have.

Another variable focus element uses an adjustable liquid crystal cell to focus the image. These are disclosed in US Patent Application Publication Nos. 2009/0213321, 2009/0316097, and 2010/0007807, which are incorporated herein by reference in their entirety and are relied upon. In the present method, the liquid crystal material is contained within a transparent cell, preferably with a matching refractive index. The cell contains a transparent electrode, such as one made of indium tin oxide (ITO). Using one spiral electrode and a second spiral electrode or planar electrode, a spatially non-uniform magnetic field is applied. Other shapes of electrodes may be used. The shape of the magnetic field determines the rotation of molecules in the liquid crystal cell to achieve a change in refractive index to achieve a change in the focus of the lens. The liquid crystal thus changes its refractive index, so that it can be electromagnetically tuned so that the tunable liquid crystal cell acts as a lens.

In a first embodiment, a tunable liquid crystal cell 3420 is shown in FIG. 57D. The cell includes an inner layer of liquid crystal 3421 and a thin layer 3423 of a directional material such as polyimide. These materials help direct the liquid crystal in the desired direction. Transparent electrodes 3425 are on each side of the directional material. The electrode can be planar, or can be spiral as shown on the right side of FIG. 57D. The transparent glass substrate 3427 contains material within the cell. The electrodes are formed to give shape to the magnetic field. As mentioned above, spiral electrodes on one or both sides are used in one embodiment so that the two are not symmetrical. A second embodiment is shown in Fig. 57E. Tunable liquid crystal cell 3430 includes a central liquid crystal material 3431 , a transparent glass substrate wall 3433 , and a transparent electrode. The top electrode 3437 has a helical shape, while the bottom electrode 3435 is planar. The transparent electrode can be made of indium tin oxide (ITO).

Additional electrodes can be used for fast recovery of the liquid crystal to its non-shaped or natural state. A small control voltage is thus used to dynamically change the refractive index of the material through which light passes. The voltage creates a spatially non-uniform magnetic field of the desired shape so that the liquid crystal can act as a lens.

In one embodiment, the camera comprises a black silicon, short-wave infrared (SWIR) CMOS sensor described in this patent. In another embodiment, the camera is a 5 megapixel (MP) optically-stabilized video sensor. In one embodiment, the control comprises a 3 GHz microprocessor or microcontroller, and a 633 MHz digital signal processor with a 30 M polygon/sec graphics accelerator for real-time image processing on images from a camera or video sensor. includes In one embodiment, the augmented reality glasses are IEEE 802.11 compliant telecommunications capabilities for wireless Internet, radio or broadband, personal area network (PAN), local area network (LAN), wide area network (WLAN), or reachback It may include communication. In one embodiment, the provided equipment includes a Bluetooth function compliant with IEEE 802.15. In one embodiment, the augmented reality glasses include an encryption system, such as a 256-bit Advanced Encryption System (AES) encryption system or other suitable encryption program for secure communication.

In one embodiment, the wireless communication includes functionality for a 3G or 4G network, and may also include wireless Internet functionality. To extend the lifespan, the augmented reality eyepiece or glasses may also include at least one lithium ion battery, and a recharging function as described above. The rechargeable plug may include an AC/DC power converter and may use multiple input voltages such as 120 or 240 VAC. In one embodiment the controls for focusing the adjustable focus lens include a 2D or 3D wireless air mouse or other non-contact control that responds to a user's gesture or movement. 2D mice can be purchased from Logitech, Fremont, CA, USA. The 3D mouse may be used as described herein, or others such as the Cideko AVK05 available from Cideko, Taiwan.

In one embodiment, the eyepiece may include electronics suitable for controlling optics and associated systems, including central processing units, non-volatile memory, digital signal processors, 3-D graphics accelerators, and the like. The eyepiece is an inertial navigation system, camera, microphone, audio output, power source, communication system, sensor, stopwatch or chronometer function, thermometer, vibrating temple motor, motion sensor, microphone that may enable audio control of the system, photochromic Additional electronic elements or features may be provided, including UV sensors or the like that allow for contrast and dimming with the material.

In one embodiment, the central processing unit (CPU) of the eyepiece may be an OMAP 4 with dual 1 GHz processor cores. The CPU may include a DSP of 633 MHz, providing functionality for a CPU of 30,000,000 polygons/sec.

The system may also provide dual micro-SD (secure digital) slots to provide additional removable non-volatile memory.

The on-board camera provides 1.3 MP color and can record up to 60 minutes of video footage. You can transfer the recorded video wirelessly or use a mini USB transfer device as an off-road video.

A communication system-on-chip (SOC) can operate with a wide area network (WLAN), Bluetooth version 3.0, a GPS receiver, FM radio, and the like.

The eyepiece can run on a 3.6 VDC Li-ion rechargeable battery for long battery life and ease of use. Additional power may be provided via solar cells external to the frame of the system. Such solar cells can supply power and charge lithium-ion batteries.

The total power consumption of the eyepiece may be around 400 mW, but will vary depending on the features and applications used. For example, processor-intensive applications with critical video graphics will require more power and will be closer to 400 mW. A simpler, less video-intensive application will use less power. Charged operating time may vary depending on application and feature usage.

A microprojector lighting engine, also known herein as a projector, may include multiple light emitting diodes (LEDs). To provide lifelike color, Osram Red, Cree Green and Cree Blue LEDs are used. These are die-based LEDs. The RGB engine can provide tunable color output, while allowing users to optimize viewing for a variety of programs and applications.

In embodiments, lighting may be added to the glasses or may be controlled through various means. For example, LED lights and other lights can be embedded in the frame of the eyepiece, such as around a compound lens, or as a nose bridge at the temple.

The intensity of the light or the color of the light can be adjusted. Modulation may be achieved through various applications, filtering and scaling, and through various control techniques described herein.

For example, lighting may be adjusted through various control techniques described herein, such as through control knobs, gestures, eye movements, or voice commands. When the user wishes to increase the intensity of the illumination, the user can adjust the control knob on the glasses or by means of a control knob or other means in a user interface displayed on the lens. The user may use eye movement or other means to control the knob displayed on the lens. The user can adjust the lighting through hand movements or other body movements so that the intensity or color of the lighting changes based on the movement the user makes. In addition, the user can adjust the lighting via voice commands by asking to increase or decrease the lighting or by saying a phrase requesting a different color to be displayed. Additionally, illumination modulation may be accomplished through any control technique or other means described herein.

Additionally, the lighting can be adjusted depending on the specific application being run. For example, the application may automatically adjust the intensity of the light or the color of the light based on the optimal settings for that application. If the current level of lighting is not an optimal level for the running application, a message or command may be sent to provide lighting adjustments.

In embodiments, illumination modulation may be achieved through filtering or through magnification. For example, filtering techniques may be employed that allow the intensity or color of the light to be altered so that optimal or desired illumination is achieved. Further, in embodiments, the intensity of illumination may be modulated by applying a greater or lesser magnification to reach a desired illumination intensity.

A projector may be coupled to a display to output video and other display elements to a user. The display used may be an SVGA 800 x 600 dots/inch SYNDIANT Silicon Liquid Crystal Display (LCoS) display.

The target MPE size for the system can be 24mm x 12mm x 6mm.

The focus is adjustable allowing users to refine the projector output to suit their needs.

The optical system may be contained within a housing made of 6061-T6 aluminum and glass-filled ABS/PC.

The weight of the system, in one embodiment, is estimated to be 3.75 ounces, or 95 grams.

In one embodiment, the eyepiece and associated electronics provide night vision functionality. This night vision function could be enabled by a SWIR sensor in black silicon. Black silicon is a complementary metal-oxide silicon (CMOS) processing technology that improves the photoresponse of silicon by over 100 times. The spectral range extends deeply into the shortwave infrared (SWIR) wavelength range. In this technology, 300 nm deep absorbing and anti-reflective layers are added to the glasses. As shown in FIG. 11, the reactivity of black silicon is much greater than that of silicon to the visible region and NIR, and provides improved reactivity extending to the SWIR region. This technology is an improvement over the current technology, which has very high cost and performance issues, as well as high volume manufacturing issues. Incorporating these technologies into night vision optics offers the economic advantages of CMOS technology to the design.

These advantages include using active lighting only when needed. In some cases, there may be sufficient natural lighting at night, such as during a full moon. In this case, artificial night vision using active lighting may not be necessary. With a black silicon CMOS-based SWIR sensor, active illumination may not be needed and not provided during these conditions, thus improving battery life.

Additionally, black silicon image sensors can have more than eight times the signal-to-noise ratio known in expensive indium-gallium arsenide image sensors, depending on the conditions of the night sky. Better resolution is also provided by the present technology, providing a much higher resolution than is available using current technology for night vision. In general, long-wavelength images produced by CMOS-based SWIR have good thermal sensing, but have degraded resolution and are difficult to interpret. These problems are addressed by black image silicon SWIR sensors that rely on much shorter wavelengths. SWIR is highly desirable for night vision goggles in combat for this reason. 12 shows a) dust; b) fog; and c) the effect of black silicon night vision technology, providing both before and after images seen through smoke. The image in FIG. 12 shows the performance of the new VIS/NIR/SWIR black silicon sensor.

Previous night vision systems have suffered "bloom" from bright light sources such as street lights. This "bloom" is particularly strong for image enhancement techniques and is also associated with a loss of resolution. In some cases, a cooling system that increases weight and reduces battery power life is required for image enhancement technology systems. 17 shows the difference in image quality between A) a flexible platform of an uncooled CMOS image sensor capable of VIS/NIR/SWIR imaging and B) an image enhanced night vision system.

13 shows the structural differences between current vision enhancement technologies and uncooled CMOS image sensors. The current platform (FIG. 13A) is limited in deployment due to cost, weight, power consumption, spectral range, and reliability issues. The current system generally consists of a front lens 1301 , a photocathode 1302 , a microchannel plate 1303 , a high voltage power supply 1304 , a phosphor-containing screen 1305 , and an eyepiece 1306 . This is in contrast to the flexible platform (FIG. 13B) of an uncooled CMOS image sensor 1307 capable of VIS/NIR/SWIR imaging, at a fraction of cost, power consumption and weight. This much simpler sensor includes a front lens 1308 and an image sensor 1309 with digital image output.

These advantages are derived from CMOS-compatible processing technology that improves the photo-reactivity of silicon by a factor of 100 or more and extends the deep spectral range into the short-wave infrared region. The difference in reactivity is shown in FIG. 13C . While typical night vision goggles are limited to the UV, visible and near infrared (NIR) ranges down to about 1100 nm (1.1 μm), the new CMOS image sensor also includes a shortwave infrared (SWIR) spectrum down to 2000 nm (2 μm). do.

The black silicon core technology could offer significant improvements over current night vision glasses. Femtosecond laser doping can enhance the photodetection properties of silicon over a broad spectrum. Also, the light response can be improved by a factor of 100 to 10,000. Black silicon technology is a fast, scalable, CMOS-compatible technology at a very low cost compared to current night vision systems. Black silicon technology can also provide low operating bias, typically 3.3V. Additionally, uncooled performance may be possible up to 50°C. The cooling requirements of the current technology increase both weight and power consumption, and also cause inconvenience to users. As mentioned above, black silicon core technology provides a high-resolution replacement for current image enhancer technology. Black silicon core technology can provide high-speed electronic shuttering at speeds of up to 1000 frames/sec with minimal crosstalk. In certain embodiments of the night vision eyepiece, OLED displays may be preferred over other optical displays such as LCoS displays.

Additional advantages of the eyepiece may include robust connectivity. This connectivity enables downloads and transmissions using Bluetooth, Wi-Fi/Internet, cellular, satellite, 3G, FM/AM, TV, and UVB transceivers.

The eyepiece may provide its own cellular connection, such as via a personal wireless connection with a cellular system. A personal wireless connection may be available only to the wearer of the eyepiece, or it may be available to multiple adjacent users, such as in a Wi-Fi hotspot (e.g. MiFi) where the eyepiece provides a local hotspot for others to utilize. have. Such adjacent users may be other wearers of the eyepiece, users of some other wireless computing device, such as a mobile communication facility (eg, cell phone). With this personal wireless connection, the wearer may not need another cell phone or internet wireless connection to access wireless services. For example, without a personal wireless connection integrated into the eyepiece, the wearer must either find a Wi-Fi connection point or tether to their mobile communication facility to establish a wireless connection. In embodiments, the eyepiece may incorporate the functionality and user interface of a separate mobile communication device, such as a cell phone, mobile computer, etc., into the eyepiece, replacing the need to have a separate mobile communication device such as a cell phone, mobile computer, etc. have. For example, the eyepiece may have an integrated Wi-Fi connection or hotspot, a physical or virtual keyboard interface, a USB hub, a speaker (for streaming music, for example) or speaker input connection, an integrated camera, an external camera, etc. can In an embodiment, the external device connected to the eyepiece may provide one unit with a private network connection (eg, WiFi, cell phone connection), a keyboard, a control pad (eg, a touchpad), and the like.

The eyepiece is a MEMS-based inertial navigation system such as a GPS processor, accelerometer (e.g., to enable head control of the system and other functions), gyroscope, altimeter, inclinometer, speedometer/odometer, laser rangefinder, magnetometer, etc. may include, which may also stabilize the image.

The eyepiece may include integrated headphones, such as articulating ear buds 120 that provide audio output to the user or wearer.

In one embodiment, a front-facing camera (see FIG. 21 ) integrated into the eyepiece may enable basic augmented reality. In augmented reality, the viewer images what it sees and layers an augmented, edited, tagged, or parsed version on top of the base view. Alternatively, the associated data may be displayed with or over the base image. If two cameras are provided and mounted at an exact distance between the two pupils for the user, a stereo video image can be created. These features can be useful for those who need visual assistance. Many people suffer from defects in their vision, such as nearsightedness, farsightedness, etc. Cameras and very close virtual screens as described herein provide these people with an adjustable "video" of focus, closer or further away, and fully controlled by the person via voice or other commands. This feature may also be useful for people suffering from eye diseases such as cataracts, retinitis pigmentosa, and the like. As long as some organic visual functions remain, augmented reality eyepieces help people see more clearly. Embodiments of the eyepiece may feature one or more of an increase in magnification, an increase in brightness, and the ability to map content to areas of the still healthy eye. Embodiments of the eyepiece may be used as a bifocal or magnifying glass. The wearer may increase the zoom in the field of view or increase the zoom in the partial field of view. In one embodiment, the associated camera may provide an enlarged picture to the user after an image of the object has been created. The user interface may allow the wearer to point to the area where they want to zoom with a control technique as described herein so that image processing remains as opposed to just zooming in on everything within the camera's field of view.

A rear view camera (not shown) may also be integrated into the eyepiece in further embodiments. In this embodiment, the rear camera enables eye control of the eyepiece, allowing the user to make selections of applications or features by pointing their eyes at a specific item displayed on the eyepiece.

Another embodiment of a device for capturing biological data about an individual may incorporate a micro-cassegrain reflex telescope retractable optical camera into the device. The collapsible optical camera of the micro-cassegrain reflector can be mounted on a portable device such as a bio-print device, a bio-phone, and can also be mounted on glasses used as part of a bio-kit for collecting biological data.

The Cassegrangian reflector is a combination of a first concave mirror and a second convex mirror. Because these reflectors provide good light (or sound) collection in a shorter and smaller package, they are often used as optical reflectors and radio antennas.

In a symmetric Cassegranian, both mirrors are aligned with respect to the optical axis and the first mirror usually has a hole in the center, allowing light to reach a camera chip or light detection device such as an eyepiece or CCD chip. Another design, usually used in radio telescopes, places the final focus in front of the primary reflector. A further alternative design may tilt the mirrors so as not to obstruct the first or second mirror and eliminate the need for a hole in the first or second mirror. The folding optical camera of the micro-cassegrain reflector uses any of the above variants, and the final choice is determined by the optics of the desired size.

The classic Cassegranian configuration uses a parabolic reflector as the first mirror and a hyperbolic mirror as the second mirror. A further embodiment of the folding optical camera of the micro-cassegrain reflector may utilize a hyperbolic primary mirror and/or a spherical or elliptical secondary mirror. In operation, the classic Cassegrangian configuration with a parabolic first mirror and a hyperbolic second mirror reflects light back down through the aperture in the first 6000 as shown in FIG. 60 . Folding the optical path makes the design more compact, making it suitable for use with the bio-print sensors and bio-print kits described herein "in micro-scale". In a folded optical system, the beam bends, making the optical path much longer than the physical length of the system. One common example of folded optics are prismatic binoculars. In a camera lens, the second mirror may be mounted to an optically flat, optically clear glass plate that closes the lens tube. This scaffold eliminates the "star mesh" diffraction effect caused by the straight-vaned scaffold spider. This allows for a sealed closed tube and protects the first mirror even if there is some loss of light collection power.

The Cassegranian design also exploits the special properties of parabolic and hyperbolic reflectors. A concave parabolic reflector reflects all incoming rays parallel to its axis of symmetry with a single focus. A convex hyperbolic reflector has two foci and reflects all rays of light directed from one focus towards the other. The mirrors of these types of lenses are designed and arranged to share a single focus, placing the second focus of the hyperbolic mirror at the same point where the image is viewed, usually just outside the eyepiece. Parabolic mirrors reflect parallel rays incident on the lens to their own focus, which coincides with the focus of the hyperbolic mirror. The hyperbolic mirror then reflects the rays to a different focus, where the camera records the image.

Figure 61 shows the configuration of the folding optical camera 6100 of the micro-cassegrain reflector telescope. The camera may be mounted on augmented reality glasses, biophone, or other biological collection device. Assembly 6100 includes multiple reflector segments that allow the camera to expand with Cassegrain optics providing a longer optical path. Thread 3602 allows the camera to be mounted on a device, such as augmented reality glasses or other biological collection device. Although the embodiment shown in FIG. 61 uses threads, other mounting methods such as bayonet mounts, handles, or press-fits may also be used. The first reflector section 3604 also functions as an outer housing when the lens is in the fully retracted position. The camera may incorporate a motor that drives the extension and retraction of the camera. A second reflector section 3606 may also be included. Other embodiments may include varying the number of reflector sections depending on the length of the optical path required for the selected task or data to be collected. A third telescope section 3608 includes a lens and a reflecting mirror. The reflective mirror may be the primary reflector if the camera is designed according to the classic Cassegrain design. A second mirror may be included in the first reflector section 3604 .

A further embodiment may utilize a micromirror to form a camera, while providing a longer optical path through the use of folded optics. The same principle of Cassegrain design is used.

The lens 3610 provides an optical device for use with a collapsible optical device of a Cassegrain design. Lens 3610 may be selected from a variety of types and may vary depending on the application. Thread 3602 allows various cameras to be swapped according to the user's needs.

Eye control of feature and option selection may be controlled and activated by object recognition software loaded into the system processor. Object recognition software enables augmented reality, combines output recognition while querying a database, and combines output recognition with computational tools to determine dependencies/similarity and the like.

3D viewing is also possible in a further embodiment incorporating a 3D projector. Two stacked picoprojectors (not shown) can be used to generate the three-dimensional image output.

Referring to FIG. 10 , a plurality of digital CMOS sensors with redundant microscopy and DSP for each sensor array and projector are provided, including real-time image enhancement 1002 , real-time keystone correction 1004 , and real-time virtual projection correction 1008 . ), detects visible, near-infrared and short-wave infrared light to enable passive day and night operation.

The augmented reality eyepiece or glasses may be powered by any stored energy system, such as battery powered, solar powered, line powered, or the like. The solar energy collector may be placed in the frame on top of the frame, belt clip, or the like. Battery charging can occur using a wall charger, a car charger, on a belt clip, in a glasses case, and the like. In one embodiment, the eyepiece is rechargeable and is equipped with a mini USB connector for charging. In another embodiment, the eyepiece is a Powercast of Ligonia, PA, USA; and one or more remote inductive power conversion technologies, such as those supplied by Fulto Int'l, Inc. of Ader, Michigan, USA, which owns another supplier, Splashpower, Inc. of Cambridge, UK. can

The augmented reality eyepiece includes a camera and any interfaces necessary to connect the camera to circuitry. The output of the camera may be stored in memory and may also be displayed on a display usable by the wearer of the glasses. A display driver may also be used to control the display. The augmented reality device also includes a power supply, such as a battery, power management circuitry, and circuitry for charging the power supply, as shown. As noted elsewhere, charging may occur via a hard connection, eg, a mini USB connector, or by an inductor, solar panel input, etc.

When a power source, such as a battery, exhibits low power, the control system for the eyepiece or glasses may include a control algorithm to conserve power. Such conservation algorithms may include turning off power for high energy consuming applications such as lighting, cameras, or sensors that require high levels of energy, such as, for example, any sensor that requires a heater. Other conservation steps include slowing the power used by the sensor or camera, eg, slowing the sampling or frame rate, proceeding to a slower sampling or frame rate when power is low; or turning off the sensor or camera to a certain lower level. normal mode; Conservation power mode; and at least three operating modes depending on available power, such as emergency or shutdown mode.

Applications of the present disclosure may be applied via equipment of the eyepiece (eg, accelerometer, gyro, camera, optical sensor, GPS sensor, etc.) and/or equipment worn or mounted to the wearer (eg, body mounted sensor control). equipment), such as movements of one's hands, fingers, feet, head, eyes, etc., can be controlled through the wearer's movements and the wearer's direct actions. In this way, the wearer can directly control the eyepiece through movements and/or actions of their body without using a typical portable remote control. For example, the wearer may have a sensing device, such as a position sensing device, mounted on one or two hands, such as on at least one finger, on the palm of the hand, on the back of the hand, where the position sensing device is the hand. It provides the location data of the eyepiece and provides wireless communication of the location data as command information to the eyepiece. In an embodiment, a sensing device of the present disclosure provides a gyroscope device (eg, an electronic gyroscope, a MEMS gyroscope, a mechanical gyroscope, a quantum gyroscope, a ring laser gyroscope, a fiber optic gyroscope) that provides location information. , accelerometers, MEMS accelerometers, velocity sensors, force sensors, optical sensors, proximity sensors, RFID, and the like. For example, the wearer may have a position sensing device mounted on the index finger of their right hand where the device may sense the movement of the finger. In this example, the user may activate the eyepiece via some predetermined sequence of movements of the finger or some switching mechanism on the eyepiece, such as quickly moving a finger, tapping the finger against a hard surface, or the like. Note that tapping against a hard surface can be interpreted through sensing by an accelerometer, force sensor, etc. The position sensing device may transmit movement of the finger as command information, such as moving a finger in the air to move a cursor across a displayed or projected image, moving a finger in a fast motion to indicate a selection, or the like. In an embodiment, the position sensing device transmits sensed command information directly to the eyepiece for command processing or the command processing circuitry is mounted on a finger as part of an assembly that includes a sensor of the position sensing device, as in this example. , may be located together with a position sensing device.

In embodiments, the wearer may have a plurality of position sensing devices mounted on their body. For example, and in extension of the preceding example, the wearer may have a position sensing device mounted on a plurality of points on the hand, such as individual sensors on different fingers, or as a collection of devices, such as in a glove. . In this way, aggregate sensing command information from a set of sensors at different locations of the hand can be used to provide more complex command information. For example, a wearer may use a sensor device glove to play a game, and when using the present disclosure in simulations and execution of simulated games, where the glove is the gripping of a user's hand over a ball, bat, racquet, etc. (grasp) and motion detection. In embodiments, the plurality of position sensing devices may be mounted on different parts of the body to enable the wearer to transmit complex bodily motions to the eyepiece for use in applications.

In embodiments, the sensing device may include a force sensor, such as for detecting when the sensing device makes contact with an object. For example, the sensing device may include a force sensor at the fingertip of the wearer. In this case, the wearer may tap, multi-tap, sequence tap, swipe, or touch to generate a command for the eyepiece. The force sensor is used to indicate the degree of touch, grip, push, etc., where a predetermined or learned threshold determines command information that differs. In this way, the command can be delivered as a series of command information that continuously updates the command information used in the application through the eyepiece. In an example, the wearer may run a simulation, such as a gaming application, a military application, a commercial application, or the like, wherein movement and contact with an object, such as through at least one of a plurality of sensing devices, affects the simulation displayed through the eyepiece. is supplied to the eyepiece as a command to

In embodiments, the sensing device may include an optical sensor or optical transmitter as a way for movement to be interpreted as a command. For example, the sensing device may include an optical sensor mounted on the wearer's hand, and the eyepiece housing may contain an optical transmitter such that when the user moves his or her hand past the optical transmitter on the eyepiece, the movement may be interpreted as a command. may include Movement detected via the optical sensor may include swapping past in repeated motion at different rates, a combination of dwelling and movement, and the like. In embodiments, the optical sensor and/or transmitter may be mounted on the eyepiece, on the wearer (eg, on a hand, foot, glove, part of clothing), or used in combination between the wearer and different areas on the eyepiece, etc. .

In one embodiment, a number of sensors useful for monitoring the condition of the wearer or persons proximate to the wearer are mounted within the augmented reality glasses. Thanks to advances in electronic technology, sensors have become very small. Signal conversion and signal processing technologies have also made great strides in the trend of size reduction and digitization. Thus, it is possible to have an entire sensor array, not just a temperature sensor in the AR glasses. These sensors, as described above, include temperature sensors, and may also detect: pulse rate, beat-to-beat heart changes; EKG or ECG; breathing rate; core body temperature; heat flow in the body; electro skin reaction or GSR; EMG; EEG; EOG; Blood pressure; body fat; hydration level; activity level; oxygen consumption; glucose or blood sugar levels; posture; and UV radiation exposure or absorption. In addition, inter alia, retinal sensors and blood oxygen sensors (Sp0₂ sensors) may be present. Such sensors are available from Vermed, Bellows Falls, Vermont, USA; VTI in Venta, Finland; and ServoFlow of Lexington, Massachusetts, USA.

In some embodiments, it may be more useful to mount the sensor on a person or a person's equipment, rather than the glasses themselves. For example, an accelerometer, a motion sensor, and a vibration sensor may be usefully mounted on a person, a person's clothes, or equipment worn by a person. Are these sensors Bluetooth? Continuous or periodic contact with the controller of the AR glasses can be maintained via a radio transmitter or other wireless device that complies with IEEE 802.11 standards. For example, if a doctor wants to monitor the motion or shock experienced by a patient during a walking race, the sensor may be mounted directly on a person's skin rather than on glasses, or if mounted on a T-shirt worn by a person. could be more useful. In this case, a more accurate reading may be obtained by a sensor placed on the person or clothing rather than the glasses. Such a sensor need not be as small as a sensor suitable for mounting on the glasses themselves, as shown.

AR glasses or goggles may also include environmental sensors or sensor arrays. These sensors are mounted on the glasses to sample the atmosphere or air around the wearer. Such sensors or sensor arrays may be sensitive to particular substances or concentrations of substances. For example, sensors and arrays can measure concentrations of carbon monoxide, nitrogen oxides ("NO_x"), temperature, relative humidity, noise level, volatile organic chemicals (VOCs), ozone, particulates, hydrogen sulfide, atmospheric pressure and ultraviolet light, and their intensity. Suppliers and manufacturers are: Sensares, France , Crolles; Critical Environmental Technologies of Canada, Delta BC, Canada; Apollo Electronics Co., Shenzhen, China; and AV Technology Ltd., Stockport, Cheshire, UK. Many other sensors are well known. When mounted on equipment, they may also be useful, Such environmental sensors may include radiation sensors, chemical sensors, toxic gas sensors, and the like.

In one embodiment, the environmental sensor, the health monitoring sensor, or both are mounted on the augmented reality glasses frame. In other embodiments, the sensor may be mounted on a person or a person's clothing or equipment. For example, a sensor for measuring the electrical activity of a wearer's heart may be implanted with an accessory suitable for translating and transmitting signals indicative of a person's cardiac activity. signal bluetooth? It can be transmitted over very short distances via radio transmitters or other radio devices that comply with the IEEE 802.15.1 specification. Other frequencies or protocols may be used instead. The signal can then be processed by the signal monitoring and processing equipment of the augmented reality glasses, recorded and displayed on a virtual screen usable by the wearer. In another embodiment, the signal may also be transmitted via AR glasses to a friend or squad leader of the wearer. Thus, a person's health and well-being can be monitored by that person and others, and can also be tracked over time.

In another embodiment, the environmental sensor may be mounted on a person or a person's equipment. Radiation or chemical sensors may become more useful, for example, when worn on an outer garment or on a person's web belt, rather than directly on glasses. As mentioned above, the signal from the sensor can be monitored locally by a person through AR glasses. Sensor readings can be sent anywhere on demand or automatically, at set time intervals, perhaps every 15 or 30 minutes. Thus, the history of sensor readings, whether human body readings or the environment, can be made to track or follow a purpose.

In one embodiment, an RF/micropower impulse radio (MIR) sensor is associated with the eyepiece and serves as a short-range medical radar. The sensor can operate in ultra-wideband. The sensor may include an RF/impulse generator, receiver and signal processor, and may be useful for detecting and measuring cardiac signals by measuring the flow of ions in cardiac cells within 3 mm of the skin. The receiver may be a phased array antenna capable of determining the position of a signal in the spatial domain. The sensor may be used to detect and identify cardiac signals passing through obstructions such as walls, water, concrete, dust, metal, wood, and the like. For example, a user may use the sensor to determine how many people are located in a concrete structure by how many heart rates are detected. In another embodiment, the detected heart rates may serve as a unique identifier for the person allowing them to be recognized later. In one embodiment, the RF/impulse generator may be embedded in one device, such as an eyepiece or some other device, while the receiver is contained in a different device, such as another eyepiece or device. In this way, a virtual "trip wire" can be created when a heart rate is detected between the transmitter and receiver. In one embodiment, the sensor may be used as a field diagnostic or self-diagnostic tool. The EKG's are biometric identifiers that can be analyzed and stored for later use. The user may receive a notification of a detected heart rate signal and how many heart rates are presented as content displayed on the eyepiece.

29 depicts an embodiment of an augmented reality eyepiece or glasses with various sensors and communication equipment. As shown, one or more environmental or health sensors are coupled to the sensor interface, either locally or remotely via near field circuitry and antennas. Sensor interface circuitry includes any device that detects, amplifies, processes, and transmits or sends signals detected by the sensor(s). The remote sensor may include, for example, an implanted heart rate monitor or other body sensor (not shown). Other sensors may include accelerometers, inclinometers, temperature sensors, sensors suitable for detecting one or more chemicals or gases, or any other health or environmental sensor discussed in this disclosure. The sensor interface is connected to a microprocessor or microcontroller of the augmented reality device, and as shown, information collected from that point may be stored in a memory, such as random access memory (RAM) or permanent memory, read-only memory (ROM). can

In one embodiment, the sensing device may simultaneously sense the electric field through the eyepiece. Electric field (EF) sensing is a proximity sensing method that allows a computer to detect, evaluate, and work with objects that are close to it. Physical contact with the skin, such as a handshake with another person or some other physical contact with a conductive or non-conductive device or object, is sensed as a change in an electric field and enables data transfer to or from the eyepiece or data Transmission can be terminated. For example, video captured by the eyepiece may be stored on the eyepiece until a wearer of the eyepiece with a built-in electric field sensing transceiver touches an object and begins transmitting data from the eyepiece to the receiver. The transceiver includes a transmitter that includes a transmitter circuit that directs an electric field towards the body and a data sensing furnace, which detects both data transmission and reception, thereby identifying a transmit/receive mode and transmit a control signal according to the two modes to enable two-way communication. print out An instant personal network between two people can be created by contact, such as a handshake. Data may be transmitted between the user's eyepiece and the second user's data receiver or eyepiece. Additional security measures, such as facial or voice recognition, detection of eye contact, fingerprint detection, biometrics, and the like, may be used to enhance private networks.

In embodiments, in whole or in part, the eyepiece, such as access to displayed or projected content, access to limited projected content, possible functions of the eyepiece itself (eg, access to functions of the eyepiece via login), etc. There may be authentication facilities associated with accessing the functionality of Authentication may be provided through recognition of the wearer's voice, iris, retina, fingerprint, etc. or other biometric identifier recognition. The authentication system can provide for biometric input databases for multiple users, so that access control can be provided for use of the eyepiece according to policies and associated access rights for each user entered into the database. The eyepiece may provide an authentication process. For example, the authentication facility may detect when the user has taken off the eyepiece and request re-authentication when the user puts it back on. This further ensures that the eyepiece provides access only to authorized users and only to those authorized by the wearer. In one example, the authentication facility may detect the presence of an eye or head of a user on which the eyepiece is being worn. At the first level of access, the user can only access less sensitive items until authentication is complete. During the authentication process, the authentication facility may identify the user and check the user's access rights. Once these privileges are determined, the authentication facility can then provide appropriate access to the user. If an unauthorized user is detected, the eyepiece may retain access to less sensitive items, further restrict access, deny access entirely, and the like.

In one embodiment, the receiver may be coupled to an object that may enable control of that object via touch by the wearer of the eyepiece, wherein the touch enables the transmission or execution of a command signal on the object. For example, the receiver may be associated with a car door lock. When the wearer of the eyepiece touches the car, the car door may be unlocked. In another example, the receiver may be included in a vial. When the wearer of the eyepiece touches the medicament bottle, an alarm signal may be initiated. In another example, the receiver may be coupled to a wall along a sidewalk. When the wearer of the eyepiece passes through or touches the wall, an advertisement may be launched on the eyepiece or a video panel on the wall.

In one embodiment, once the wearer of the eyepiece initiates physical contact, the WiFi exchange of information with the receiver may provide an indication that the wearer may connect to an online activity, such as a game, or provide identification in an online environment. have. In embodiments, the human representation may change color or receive other visual indications in response to contact. In an embodiment, the eyepiece may include a tactile interface such as swipe, tap, touch, pressure, click, roll of a roller ball, etc., capable of haptic control of the eyepiece as in FIG. 14 . For example, the tactile interface 1402 may be mounted on the frame of the eyepiece, such as an arm, both arms, a nose pad, a top of a frame, a bottom of a frame, and the like. The wearer then taps the interface once or multiple times, passes a finger across the interface, presses and holds, presses on one or more interfaces simultaneously, etc. You can touch the tactile interface by way of In embodiments, the tactile interface may be attached to the wearer's body and to the wearer's clothing, as an attachment to clothing, as a ring 1500 , as a bracelet, as a necklace, or the like. For example, the interface provides different command information when touching different parts of the interface (e.g., touching the front, back, center, holding for a period of time, tapping, swiping, etc.); It can be attached to the body, such as on the back of the wrist. In another example, the wearer may have an interface mounted on a ring, hand piece, etc. as shown in FIG. 15 , wherein the interface has a plurality of commands, such as a tactile interface with a wireless command connection to the eyepiece, a position sensor device, etc. At least one of interface types may be provided. In one embodiment, the ring 1500 has controls that mirror a computer mouse, such as a button 1504 (single button, multi-button, and similar to mouse functions), a 2D position control 1502, a scroll wheel, etc. have. Button 1504 and 2D position control 1502 may be as shown in FIG. 15 , where the button is on the side facing the thumb and the 2D position controller is on the top. Alternatively, the button and 2D position control may be of any other configuration, both facing the thumb side, all on the top surface, or in any other combination. 2D position control 1502 may include a 2D button position controller (such as the TrackPoint pointing device built into some laptop keyboards for controlling the position of a mouse), a pointing stick, a joystick, an optical trackpad, an optical touch wheel, It can be a touch screen, touchpad, trackpad, scrolling trackpad, trackball, other positioning or pointing controller, or the like. In embodiments, control signals from a tactile interface (eg, ring tactile interface 1500 ) may be wired to or to the eyepiece where the user may conveniently provide control input with their hand, thumb, finger, or the like. It may be provided as a wireless interface. For example, the user may associate the control with his thumb, where the ring is worn on the user's index finger. In embodiments, a method or system may provide an interactive head-mounted eyepiece worn by a user, wherein the eyepiece communicates to the user an optical assembly through which the user views the surrounding environment and displayed content, content to display to the user. a processor for processing, an integrated projector facility for projecting content onto an optical assembly, and a control device worn on a user's hand, said control device comprising at least one control component actuated by a finger of the user's hand, and at least one Provides a control instruction from the operation of the control component of the processor as an instruction instruction. The command instruction instructs the user to adjust the content to be displayed. The control device may be worn on a first finger of the user's hand, and the at least one control component may be actuated by a second finger of the user's hand. The first finger may be an index finger, the second finger may be a thumb, and the first finger and the second finger may be on the same hand of the user. The control device may have at least one control component mounted on the side of the forefinger facing the thumb. The at least one control component may be a button. The at least one control component may be a 2D position controller. The control device may have at least one button actuation control component mounted on the side of the index finger facing the thumb and a two-dimensional position controller actuation control component mounted on the upper end facing the side of the index finger. The control component may be mounted on at least two fingers of the user's hand. The control device may be worn as a glove on the user's hand. The control device may be worn on the user's wrist. The at least one control component may be worn on at least one finger of the hand and the transmission facility may be worn separately on the hand. The transmission facility may be worn on the wrist. The transmission device may be worn on the back of the hand. The control component may be at least one of the plurality of buttons. At least one button may provide a function substantially similar to a conventional computer mouse button. Two of the plurality of buttons may function substantially similar to the first button of a conventional two-button computer mouse. The control component may be a scroll wheel. The control component may be a 2D position control component. The 2D position control component may be such as a button position controller, pointing stick, joystick, optical track pad, optical touch wheel, touch screen, touch pad, track pad, scrolling track pad, track ball, capacitive touch screen, and the like. The 2D position control component can be controlled with the user's thumb. The control component can be a touch screen that can implement touch controls including button-like functions and 2D manipulation functions. The control component may be actuated when the user wears the projected processor content pointing and control device. A surface sensing component of the control device for detecting motion across the surface may also be provided. The surface sensing component may be disposed on the palm side of the user's hand. The surface may be at least one of a hard surface, a soft surface, the surface of the user's skin, the surface of the user's clothes, and the like. Providing control commands may be transmitted over a wireless, wired connection, or the like. The control device may control a pointing function associated with the displayed processor content. The pointing function can control the cursor position; selection of displayed content, selecting and moving displayed content; control of magnification, pan, field of view, size and position of the displayed content; etc can be controlled. The control device may control a pointing function associated with the displayed surrounding environment. The pointing function can place the cursor on an object that is visible in the surrounding environment. The position position of the object being viewed may be determined by the processor in relation to a camera integrated with the eyepiece. Identification of the object being viewed may be determined by a processor in association with a camera integrated with the eyepiece. The control device may control the function of the eyepiece. A function may be associated with the displayed content. The function can be mode control of the eyepiece. The control device is foldable for easy storage when not worn by the user. In embodiments, the control device may be used in conjunction with an external device to control an external device associated with the eyepiece. The external device may be an entertainment device, an audio device, a portable electronic device, a navigation device, a weapon, a vehicle control, or the like.

In an embodiment, the system includes an interactive head-mounted eyepiece worn by a user, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content, wherein the optical assembly is the user's at least one of a calibrating element that calibrates a view of the surrounding environment, an integrated processor handling content for display to a user, an integrated image source directing the content to an optical assembly, and a user touching the interface and a user proximate to the interface and a tactile control interface mounted on the eyepiece for receiving a control input from a user through

In an embodiment, control of the eyepiece, and in particular of a cursor associated with content displayed to the user, is via hand control, such as a worn device 1500 as shown in FIG. 15 , the virtual computer mouse in FIG. 15A ( 1500A), and so on. For example, the worn device 1500 transmits commands via a physical interface (e.g., button 1502, scroll wheel 1504), and virtual computer mouse 1500A uses motion and the user's thumb, Commands can be interpreted by detecting movements such as fists and hands. In computation, a physical mouse is a pointing device that functions by detecting two-dimensional movement relative to its support surface. A physical mouse traditionally consists of an object held in one of the user's hands with one or more buttons. It often features other elements such as "wheels" that allow the user to perform various system-dependent operations, or additional buttons or features that can add more control or dimension inputs. Mouse movement is translated into cursor movement on the display allowing great control over the graphical user interface. For the eyepiece, the user can use a physical mouse, a virtual mouse, or a combination of both. In an embodiment, the virtual mouse includes one or more sensors attached to the user's hand, such as thumb 1502A, finger 1504A, palm 1508A, wrist 1510A, etc., where the eyepiece is It receives a signal and translates the received signal into movement of a cursor on the eyepiece for display to the user. In embodiments, the signal may be received via an external interface, such as tactile interface 1402 , via a receiver internal to the eyepiece, at a second communication interface, an associated physical mouse or worn interface, or the like. The virtual mouse may also include an actuator or other output-type element attached to the user's hand for haptic feedback to the user via vibration, force, electric shock, temperature, and the like. The sensors and actuators may be attached to the user's hand in the manner of a wrap, ring, pad, glove, or the like. Thus, the eyepiece virtual mouse allows the user to translate hand movement into movement of the cursor on the eyepiece display, where 'movement' includes slow movement, rapid movement, paroxysmal movement, position, change of position, etc. It allows users to work in three dimensions without the need for a physical surface, and including some or all of the six degrees of freedom. Note that since a 'virtual mouse' can be associated with multiple parts of a hand, a virtual mouse can be implemented as multiple 'virtual mouse' controllers, or as a distributed controller across multiple control elements of a hand. In embodiments, the eyepiece may provide for multiple virtual mouse uses, eg, one on the user's hand, one or more of the user's feet, and the like.

In embodiments, the eyepiece virtual mouse does not require a physical surface to operate, and is one of a plurality of accelerometer types (e.g., tuning fork, piezoelectric, shear mode, strain mode, capacitive, thermal, resistive, electromechanical, resonant) , magnetic, optical, acoustic, laser, three-dimensional, etc.) and determine the translational and angular displacement of the hand or part of the hand via the output signal of the sensor(s). For example, an accelerometer may generate an output signal with a magnitude proportional to the translational acceleration of the hand in three directions. The pair of accelerometers may be configured to detect rotational acceleration of the hand or portion of the hand. The translational speed and displacement of the hand or part of the hand may be determined by integrating the accelerometer output signal, and the rotational speed and displacement of the hand may be determined by integrating the difference between the output signals of the pair of accelerometers. Alternatively, other sensors may be utilized, such as ultrasonic sensors, imagers, IR/RF, magnetometers, gyro magnetometers, and the like. As accelerometers or other sensors can be mounted on various parts of the hand, the eyepiece can accommodate multiple movements of the hand, from simple movements typically associated with computer mouse movements to more complex movements such as interpretation of complex hand movements in simulation applications. can be detected. In embodiments, the user may require only small translational or rotational actions to translate this action into a movement associated with the user's intended action on the eyepiece projection on the user.

In embodiments, the virtual mouse may include a physical switch associated therewith for controlling a device, such as an on/off switch mounted on the hand, eyepiece, or other part of the body. The virtual mouse may have on/off control through a predetermined movement or action of a hand, and the like. For example, the operation of the virtual mouse may be possible through a fast forward and backward movement of the hand. In another example, the virtual mouse may be disabled through movement of the hand past the eyepiece, such as in front of the eyepiece. In embodiments, the virtual mouse for the eyepiece is a single click of a hand, a double click, a triple click, a right click, a left click, a click and drag, a combination of clicks, a roller wheel motion, generally operating in conjunction with a physical mouse control. It is possible to provide a translation of a plurality of movements familiar to the user without training such as , , and the like. In embodiments, the eyepiece may provide gesture recognition while interpreting hand gestures through mathematical algorithms.

In embodiments, gesture control recognition may be provided via a technique that utilizes capacitive changes resulting from changes in the distance of the user's hand from the conductor element as part of the eyepiece's control system, and thus may be mounted on the user's hand. No device required. In embodiments, the conductor may be mounted as part of the eyepiece on an arm or other part of the frame, or as some external interface mounted to the user's body or clothing. For example, the conductor could be an antenna, where the control system works in a manner similar to a touchless instrument known as a telemin. Theremin uses the heterodyne principle to generate an audio signal, but in the case of an eyepiece, the signal can be used to generate a control input signal. The control circuit may include multiple radio frequency oscillators, such that one oscillator operates at a fixed frequency and another frequency controlled by the other hand, such that the distance from the hand changes the input at the control antenna. . In this technique, the user's hand is part of the oscillator and serves as the ground plate of the variable capacitor (the user's body is connected to the ground) in the LC (inductance-capacitance) circuit that determines its frequency. In another example, the circuit may use a single oscillator, two pairs of heterodyne oscillators, and the like. In embodiments, there may be a plurality of different conductors used as control inputs. In embodiments, this type of control interface may be ideal for control inputs that vary over a range, such as volume control, zoom control, and the like. However, this type of control interface can be used for more discrete control signals (eg, on/off control) where a predetermined threshold determines the state change of the control input.

In embodiments, the eyepiece may interface with a physical remote control device, such as a wireless trackpad mouse, handheld remote control, body mounted remote control, remote control mounted on the eyepiece, and the like. The remote control device may be mounted on an external device for personal use, gaming, professional use, military use, and the like. For example, a remote control could be mounted on a pistol grip and on a muzzle air shroud, on a front grip, etc., while providing remote control to the soldier without the need to remove the weapon from his or her hand. It can be equipped on a weapon for The remote control may be removably mounted to the eyepiece.

In embodiments, remote control to the eyepiece may be activated and/or controlled via a proximity sensor. The proximity sensor may be a sensor capable of detecting the presence of a nearby object without physical contact. For example, a proximity sensor may emit an electromagnetic or electrostatic field, or a beam of electromagnetic radiation (infrared, for example), looking for changes in the field or return signal. The sensed object is usually referred to as the target of the proximity sensor. Different proximity sensor targets may require different sensors. For example, a capacitive or photoelectric sensor may be suitable for a plastic target; Inductive proximity sensors are required for metal targets. Other examples of proximity sensor technologies include capacitive displacement sensors, eddy currents, magnetic, photoelectric cells (reflection), lasers, passive thermal infrared, passive light, CCD, reflection of ion radiation, and the like. In embodiments, the proximity sensor may be incorporated into any of the control embodiments described herein, including physical remote controls, virtual mice, controls mounted on the eyepiece portion of the equipment (eg game controllers, weapons), and the like. .

In embodiments, control of the eyepiece, and in particular of a cursor associated with content displayed to the user, may be via facial actuation sensor 1502B sensing movement of a facial feature of the user wearing the eyepiece, clicking of teeth, movement of the jaw, etc. can be made possible For example, as shown in FIG. 15B , the eyepiece has a facial actuation sensor as an extension from the eyepiece earphone assembly 1504B, arm 1508B of the eyepiece, etc. It can detect forces and vibrations related to movement. The facial actuation sensor may also be mounted separately from the eyepiece assembly, such as part of a standalone earpiece, and the sensor output of the earpiece and the face actuation sensor may be communicated via wired or wireless communication (eg, Bluetooth or known in the art). other communication protocols) can be transmitted to the eyepiece. The facial actuation sensor may be attached around the ear, in the mouth, on the face, on the neck, or the like. A facial actuation sensor may also be comprised of a plurality of sensors to optimize the sensed movement of different facial or internal movements or actions. In embodiments, a facial actuation sensor may detect movement and interpret it as a command, or a raw signal may be sent to the eyepiece for translation. The command may be a command for a control of an eyepiece function, a control related to a cursor or pointer provided to the user as part of the display of the content, or the like. For example, a user may click on his or her teeth once or twice to indicate a single or double click, such as is commonly associated with a computer mouse click. As another example, a user may strain a facial muscle to indicate a command, such as a selection associated with a projected image. In embodiments, facial actuation sensors may utilize noise reduction processing to minimize background movement of the face, head, etc., such as through adaptive signal processing techniques. Voice activity sensors may be utilized to reduce interference from the user, from other nearby individuals, from ambient noise, and the like. In an example, the facial actuation sensor also improves communication, has multiple microphones to identify background noise, removes noise by detecting vibrations in the user's cheeks while speaking, and removes them through noise cancellation, volumetric augmentation, etc. can do.

In embodiments, the user of the eyepiece may obtain information regarding some environmental features, locations, objects, etc., viewed through the eyepiece by raising their hand into the field of view of the eyepiece and pointing to an object or location. For example, a user's pointing finger may point to an environmental feature where the finger is in the view of the eyepiece as well as in the view of the built-in camera. The system may correlate the position of the pointing finger with the position of the environmental feature as seen by the camera. In addition, the eyepiece may have position and orientation sensors such as GPS and magnetometers so that the system can know the position and gaze of the user's eyes. From this, the system provides the user with location information, overlays the location of the environmental information on a 2D or 3D map, and provides ancillary information about that location (e.g., address, name of the person at that address, at that location). Location information of environmental features may be inferred, such as by further correlating location information constructed to correlate that location information to the business name, location coordinates, etc. Referring to FIG. 15C , in the example, the user looks through the eyepiece 1502C and points in his/her field of view with his hand 1504C at home 1508C, where the built-in camera 1510C is the pointing hand 1504C. ) and house 1508C are within its field of view. In this case, the system may determine the location of the house 1508C and provide a 3D map overlaid with the location information 1514C and the user's view of the environment. In embodiments, information associated with environmental features may be provided by an external facility, communicated via a wireless communication connection, stored within the eyepiece, downloaded to the eyepiece for the current location, and the like.

In embodiments, users control the perspective of their views with respect to 3D projected images, such as 3D projected images associated with the external environment, stored and retrieved 3D projected images, 3D displayed motions (eg, downloaded for viewing), etc. can do. For example, and with reference to FIG. 15C , the user may change the perspective view of the 3D display image 1512C by turning his/her head, wherein the live external environment and the 3D display image allow the user to change his/her head They are even together when turning, moving positions, and so on. In this way, the eyepiece may provide augmented reality by overlaying overlay information, such as an overlaid 3D displayed map 1512C, location degree 1514C, etc., into the external environment viewed by the user, where the displayed map, information, etc. may be changed when the user view is changed. In another example, with a 3D movie or 3D transform movie, the viewer's perspective can be changed to put' the viewer's head into the environment of the movie with some control of the perspective view, where the user moves his/her head around and The view may change in response to a changed head position, where the user may 'walk into the image as they physically walk forward, the perspective change as the user moves the stared view of their eye, etc.' . Additionally, additional image information may be provided, such as in terms of the user's view, which may be accessed by turning the head.

15D , in an embodiment the user of the eyepiece 1502D uses his or her hand 1504D to define the field of view (FOV) 1508D of the camera 1510D for a see-through view, such as for augmented reality. can use multiple hand/finger points in For example, in the example shown, the user is using his first finger and thumb to adjust the FOV 1508D of the camera 1510D of the eyepiece 1502D. The user may use other combinations to adjust the FOV 1508D, for example, combinations of fingers, fingers and thumbs, finger and thumb combinations of both hands, use of palm(s), cupped hands, etc. can utilize The use of multiple hand/finger points allows the user to change the FOV 1508 of the camera 1510D in much the same way as a user of a touch screen, where different points of the hand/finger point of the FOV to build the desired view. build points In this case, however, there is no physical contact between the user's hand(s) and the eyepiece. Here, the camera may be instructed to associate a portion of the user's hand(s) with the establishment or alteration of the camera's FOV. The command may be any command described herein, and may be a hand motion in the FOV of the camera, a command related to the physical interface of the eyepiece, a command related to motion detected in the vicinity of the eyepiece, received from the command interface on some part of the user. commands, etc., but are not limited thereto. The eyepiece may recognize finger/hand gestures as commands, such as in some certain repetitive movements. In embodiments, the user may also utilize these techniques to adjust a portion of the projected image, wherein the eyepiece is viewed by the camera on some side of the projected image, such as hand/finger points in the user's projected image. Associates a losing image. For example, the user can view the projected image with the external environment at the same time, and the user utilizes these techniques to change the projected image area, area, magnification, etc. In embodiments, the user can zoom in and out of the screen viewed in the live environment, to change the viewing area assigned to the projected image, to change the perspective view of the environment or projected image, etc., and to change the projected image The change of the FOV may be performed for a number of reasons, including enlarging or reducing the visible portion of the .

In embodiments, the eyepiece may track the eyes through light reflected from the user's eyes to determine what the user is looking at or movement of the user's eyes. This information can be used to help correlate the user's gaze to the projected image, camera view, external environment, etc., and can be used to control the techniques described herein. For example, the user, with an external remote control, or with some detected eye movement (eg, blinking an eye), can gaze at a location on the projected image and make a selection. In this example technique, and with reference to FIG. 15E , transmitted light 1508E, such as infrared light, is reflected from the eye 1504E and detected at the optical display 502 (eg, with a camera or other optical sensor, etc.). can be The information can be analyzed to extract eye rotation from changes in reflexes. In embodiments, the eye tracking facility may use the corneal reflex and the center of the pupil as features to track over time; can use reflections from the front of the cornea and the back of the lens as features to track; You can image features from inside the eye, such as blood pressure in the retina, follow these features as the eye rotates, and so on. Alternatively, the eyepiece may be a component around the eye, such as mounted on a contact lens in the eye, using other techniques to track the movement of the eye. For example, special contact lenses may be provided to the user with built-in optical components, such as mirrors, magnetic field sensors, and the like, to measure eye movement. In another example, electrical potential is measured and monitored with electrodes placed around the eye, eg, with its positive pole and its negative pole in the retina, utilizing a stable electrical potential field from the eye as a dipole. can be In this case, an electrical signal can be derived using a contact electrode disposed on the skin around the eye, on the frame of the eyepiece, or the like. As the eye moves from its central position towards the periphery, the retina approaches one electrode while the cornea approaches the contralateral electrode. A change in direction in the dipole and as a result the electric potential field results in a change in the measured signal. By analyzing these changes, eye movements can be tracked.

In embodiments, the eyepiece may have a plurality of operating modes in which control of the eyepiece is controlled at least in part by position, shape, movement of a hand, and the like. To provide this control, the eyepiece may utilize a hand recognition algorithm that detects the shape of the hand/finger, which can be combined with hand movement as a command, and then associates these hand configurations. In reality, since there are only a limited number of hand configurations and movements that can command the eyepiece, these hand configurations may need to be reused depending on the operating mode of the eyepiece. In embodiments, specific hand configurations or movements may be assigned to transition the eyepiece from one mode to the next, allowing reuse of hand movements. For example, and with reference to FIG. 15F , the user's hand 1504F may be moved in view of the camera on the eyepiece, the movement then being a circular movement 1508F, a movement across the field of view 1510F, a back-and-forth movement. (1512F), etc., may be interpreted as different commands depending on the mode. As a simple example, suppose there are two working modes, mode 1 for panning the view from the projected image and mode 2 for magnifying the projected image. In this example, one may wish to use a finger pointing hand movement from left to right to command a panning movement to the right. However, the user may also wish to use the finger pointing hand movement from left to right to command enlargement of the image to a greater magnification. To allow this dual use of hand movements for both command types, the eyepiece may be configured to interpret hand movements differently depending on the mode the eyepiece is currently in, where a particular hand movement is assigned for mode switching. For example, a clockwise rotation movement may indicate a transition from pan to zoom mode, and a counterclockwise rotation movement may indicate a transition from zoom to pan mode. These examples are illustrative and not limiting in any way, where those skilled in the art will recognize that these general techniques can be used to perform various commands/commands using the hand(s) and finger(s), such as hand-finger constructive movements, two-hand constructive moves, and the like. You will see how it can be used to implement a mod structure.

In an embodiment, the system includes an interactive head-mounted eyepiece worn by a user, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content, wherein the optical assembly is the user's a correction element for correcting a view of the surrounding environment, an integrated processor for handling content for display to a user, and an integrated image source for directing the content to an optical assembly; and an integrated camera facility for imaging the gesture, wherein the integrated processor identifies and translates the gesture as an instructional instruction. The control command may provide manipulation of content for display, a command communicated to an external device, and the like.

In embodiments, control of the eyepiece may be possible through eye movement, eye movement, and the like. For example, blinking, repeated blinking, number of blinks, blinking rate, eye opening and closing, eye tracking, eye movement from side to side, up and down, side to side, through a sequence of positions, towards a specific position, a fixed position the wearer's eye (wherein the dwell time in the , gaze towards a stationary object (eg, the edge of an eyepiece lens), eye movement or action through the center of the lens, towards a real object, etc. can be interpreted as command information. There may be a camera on the eyepiece viewing back to the ). In addition, eye control allows the viewer to focus on a specific point in the displayed image from the eyepiece, and because the camera can correlate the viewing direction of the eye to a point on the display, the eyepiece can determine where the wearer is looking and where the wearer is looking. The command may be interpreted through a combination of actions (eg, blinking, touching the interface device, movement of the position sensing device, etc.). For example, a viewer may see an object on a display and select the object through movement of a finger possible through a position sensing device.

In some embodiments, the glasses may be equipped with an eye tracking device for tracking the user's eyes, or preferably both eyes; Alternatively, the glasses may be equipped with a sensor for 6 degrees of freedom of movement tracking, ie tracking movement of the head. Such devices or sensors are commercially available, for example, from Chronos Vision GmbH, Berlin, Germany, and ISCAN, Uburn, Massachusetts. Retinal scanners can also be used to track eye movements. Retinal scanners are also equipped with augmented reality glasses and are available from various companies such as Tobii of Stockholm, Sweden, and SMI of Telto, Germany, and ISCAN.

The augmented reality eyepiece includes a user input interface to allow the user to control the device as shown. Inputs used to control the device may include any of the sensors described above, and may also include a trackpad, one or more function keys, and other suitable local or remote devices. For example, eye tracking devices can be used to control video games or other devices such as external tracking devices. For example, FIG. 30 depicts a user having an augmented reality eyepiece equipped with an eye tracking device, as disclosed herein. The eye tracking device may enable the eyepiece to track the user's eye or, preferably, the direction of the eye, and transmit the movement to a controller of the eyepiece. Control system 3000 includes an augmented reality eyepiece for a weapon and a control device. Movement can then be transmitted to the control device for the controlled weapon by the control device, which may be within the user's field of view. The weapon may be a large caliber such as a howitzer or mortar, or a small caliber such as a machine gun.

The movement of the user's eyes is then converted by appropriate software into signals to control the movement of the weapon, such as the quadrant (range) and azimuth (direction) of the weapon. Additional controls such as the user's trackpad or function keys can be used for single or continuous discharge of the weapon. Alternatively, the weapon may be stationary and non-directional, such as an implanted mine or shape charge, and protected by a safeguard, such as requiring specific encoding commands. The user of the augmented reality device can activate the weapon by sending the appropriate code and command, without using the eye tracking feature.

In embodiments, control of the eyepiece may be enabled via gestures by the wearer. For example, the eyepiece may have a camera that looks outward (eg, forward, lateral, and downward) and interprets gestures or movements of the wearer's hand as control signals. Hand signals include pointing the hand at a real object (eg, to activate augmentation of the object, etc.), passing the hand over the camera, positioning the hand in front of the camera, or doing sign language. Hand gestures, such as moving an object, rotating an object, deleting an object, opening-closing a screen or window in an image, etc., can also be used to manipulate objects displayed on the inside of the translucent lens. Although hand gestures are used in the above examples, a part of the body or an object held or worn by the wearer may also be utilized for gesture recognition by the eyepiece.

In embodiments, head motion control may be used to send commands to the eyepiece, where motion sensors, such as an accelerometer, gyro, or any sensor described herein, may be mounted on the wearer's head, on the eyepiece, on a hat, helmet, etc. can Referring to FIG. 14A , the head movement is jerking, constant, with the head in a forward and/or backward motion 1412 , in an upward and/or downward motion 1410 , in a side-to-side motion such as a nod. It may include rapid movements of the head, such as staying in position, moving and maintaining a position relative to the side, and the like. The motion sensor may be integrated into the eyepiece by a wired or wireless connection to the eyepiece, and may be mounted on the user's head or head cover (eg, hat, helmet). In an embodiment, the user includes an interactive head-mounted eyepiece, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content. The optical assembly may include a corrective element for correcting the user's view of the surrounding environment, an integrated processor for processing the content for display to the user, and an integrated image source for directing the content to the optical assembly. At least one of the plurality of head motion sensing control devices may be incorporated into or associated with the eyepiece to provide a control command to the processor as a command instruction based on sensing a predefined head motion characteristic. The head motion feature may be a nod of the user's head such that the nod of the head can be a distinctly different motion from the normal head motion. A clear movement can be a jerking movement of the head. The control instruction may provide adjustment of content to be displayed, be communicated to control an external device, and the like. Head motion control may be used in combination with other control mechanisms as described herein, with another control mechanism activating a command and head motion executing it. For example, the wearer moves the object to the right via eye control, selects the object and activates the head motion control as disclosed herein. Then, by tipping its head to the right, the object can be commanded to move to the right, and the command can be terminated via eye control.

In embodiments, the eyepiece may be controlled via audio, such as via a microphone. The audio signal may include speech recognition, voice recognition, sound recognition, sound detection, and the like. Audio can be detected through a microphone on the eyepiece, neck microphone, jaw bone microphone, boom microphone, headphones, earbuds with microphone, etc.

In an embodiment, command input may include: turn on/off the eyepiece projector, turn on/off audio, turn on/off camera, turn on/off augmented reality projection, turn on/off GPS, turn on/off GPS, interaction with the display (e.g., multiple control functions, such as selecting/receiving a displayed function, replaying a captured image or video, etc.), and interacting with the real world (eg capturing an image or video, turning a page in a displayed book, etc.) provide, actions to built-in or external mobile devices (eg mobile phones, navigation devices, music devices, VoIP, etc.), browser controls for the Internet (eg submit, next result, etc.), email controls (eg, , e-mail reading, text display, text-to-speech, selection, etc.), GPS and navigation controls (eg, saving a location, recalling a saved location, displaying directions, viewing a location on a map), and the like.

In an embodiment, the eyepiece is a stereoscopic image, auto-stereoscopic image, computer generated holography, volumetric display image, stereogram/stereoscopic, view-sequential, electronic holographic display, parallax "two view" display and parallax It is possible to provide 3D display imaging to the user by delivering a panoramagram, a reimaging system, etc., and creating a perception of 3D depth to the viewer. Displaying the 3D image to the user may be achieved by the user's left and right eyes, such as the left and right optical paths have some optical component that differentiates the image, the projector equipment projects different images into the user's left and right eyes, etc. Different images presented may be employed. The optical path, including from the projector facility through the optical path to the user's eye, may include a graphical display device that forms a visual representation of the object in three physical dimensions. An integrated processor in the eyepiece or an integrated processor in an external facility may provide 3D image processing as at least one step of generating a 3D image for a user.

In an embodiment, holographic projection techniques such as computer generated holography (CGH), a method of digitally generating holographic interference patterns, may be used for presentation of 3D image effects to users. For example, a holographic image may be projected by a holographic 3D display, such as a display that operates based on interference of coherent light. Computer-generated holograms have the advantage that the objects to be displayed do not have to have any physical entity at all, that is, they can be completely generated as 'synthetic holograms'. There are a number of different methods for calculating interference patterns for CGH, including those from the fields of holographic information and computational reduction, as well as computational and quantization techniques. For example, Fourier transform methods and point source holograms are two examples of computational techniques. The Fourier transform method can be used to simulate the propagation of each depth of an object into a holographic plane where reconstruction of the image can occur in a distant field. For example in the process, there may be two steps: first the light field at the far observer plane is computed, then the lens whose wavefront to be reconstructed by the hologram is the superposition of the Fourier transform of each plane in depth. The field is Fourier transformed back to the plane. In another example, the target image may be multiplied by a phase pattern to which an inverse Fourier transform is applied. The intermediate holograms can then be computed by shifting these image products and combined to create a final set. The final set of holograms can then be approximated to form a kinoform for continuous display for the user, where the kinoform becomes a phase hologram in which the phase modulation of the object wavefront is recorded as a surface relief profile. In the point-source holographic method, an object is decomposed into self-luminous points where a base hologram is computed for all point sources and a final hologram is synthesized by superimposing all base holograms.

In one embodiment, 3-D or holographic images may be enabled by a dual projector system in which two projectors are stacked on top of each other for 3D image output. The hologram projection mode can be entered by the control mechanism described herein or by image or signal capture, such as by raising the palm and spreading the hand, SKU, RFID reader, etc. For example, the wearer of the eyepiece may see the letter 'X' on one cardboard causing the eyepiece to enter holographic mode and turn on a second stacked projector. Choosing which hologram to display can be done with a control technique. The projector can project a hologram onto the cardboard for the letter 'X'. Associated software may track the position of the letter 'X' and move the projected image along with the movement of the letter 'X'. In another example, the eyepiece can scan a SKU, such as the SKU on a toy building kit, and a 3-D image of the completed toy building can be accessed from an online source or non-volatile memory. This interaction with the hologram, such as rotating it, zooming in/out, etc., can be performed using the control mechanisms described herein. Scanning may be enabled by the associated bar code/SKU scanning software. In another example, the keyboard can be projected spatially or on a surface. The holographic keyboard can be used with associated applications/functions or used to control any of them.

In embodiments, the virtual keyboard provides that the eyepiece facility locks the position of the virtual keyboard against objects in the real environment that do not move when the wearer moves their head (eg, tables, walls, dashboards of vehicles, etc.) can do. In the example, and with reference to FIG. 24 , a user may be sitting at a table wearing the eyepiece 2402 and may wish to enter text into an application, such as a word processing application, a web browser, a communication application, or the like. A user may provide a virtual keyboard 2408, other interactive control elements (eg, virtual mouse, calculator, touch screen, etc.) for use in input. The user may provide a command to bring up the virtual keyboard 2408 and use the hand gesture 2404 to indicate the fixed position of the virtual keyboard 2408 . The virtual keyboard 2408 is spatially fixed relative to the external environment, such as the eyepiece fixture is fixed in position on the table 2410 maintaining the position of the virtual keyboard 2408 on the table 2410 when the user turns the head. can be That is, the eyepiece 2402 can compensate for the user's head movement to maintain the user's view of the virtual keyboard 2408 on the table 2410 . In embodiments, the user may wear an interactive head-mounted eyepiece, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and displayed content. The optical assembly may include a corrective element for correcting the user's view of the surrounding environment, an integrated processor for processing the content for display to the user, and an integrated image source for directing the content to the optical assembly. An integrated camera facility is provided, which images the surrounding environment and identifies user hand gestures as interactive control elements such as hand-finger configurations that are moved in a certain way, positioned in a certain way, and the like. The position of the interactive control element may maintain a fixed position with respect to the object in the surrounding environment in response to the interactive control element position command regardless of the user's viewing direction. In this way, the user can utilize virtual keyboards in much the same way that they can be physical keyboards, where the virtual keyboards remain in the same location. However, in the case of a virtual keyboard, there is no 'physical restriction' such as gravity, which limits the position where the user can position the keyboard. For example, the user may be standing next to a wall and positioning the keyboard on a wall lamp.

In embodiments, the eyepiece facility eliminates the portion of the virtual keyboard projection where intervening obstructions are visible (eg, the user's hand is in the way, where it is undesirable for the user's hand to project the keyboard). can provide what In an example, and with reference to FIG. 62 , the eyepiece 6202 may present a projected virtual keyboard 6208 to the wearer, such as onto a table top. The wearer may reach 'over' the virtual keyboard 6208 to enter. Since the keyboard is not a physical keyboard, but simply a projected virtual keyboard without any kind of compensation for the projected image, the projected virtual computer will be projected 'onto' the back of the user's hand. However, as in this example, the eyepiece may provide compensation for the projected image so that the portion of the wearer's hand 6204 that is preventing the intended projection of the virtual keyboard onto the table may be removed from the projection. That is, what may be visualized in the user's hand may be undesirable for the portion of the keyboard projection 6208 , so the eyepiece subtracts the portion of the virtual keyboard projection co-located with the wearer's hand 6204 . In an embodiment, the user wears an interactive head-mounted eyepiece, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and display content. The optical assembly may include a corrective element that corrects the user's view of the surrounding environment, an integrated processor for processing the content for display to the user, and an integrated image source that guides the content to the optical assembly. The displayed content may include interactive control elements (eg, virtual keyboard, virtual mouse, calculator, touch screen, etc.). The integrated camera facility image may image the user body part with which it interacts with the interactive control element, wherein the processor determines the interactive control element to be positioned with the imaged user body part based on the user view. Removes part of the interactive control element by removing part of In embodiments, this technique of partial projected image removal is applicable to other projected images and obstacles, and is not intended to limit the example of such a hand via a virtual keyboard.

In embodiments, the eyepiece facility provides the ability to determine intended text input from a series of character contacts that are swiped across the virtual keypad with a finger, stylus, or the like. For example, and referring to FIG. 63 , the eyepiece is projecting a virtual keyboard 6302 , where the user wishes to input the input word 'wind'. Typically the user separately presses the key position for 'w', then 'i', then 'n' and finally 'd', and the equipment associated with the eyepiece (camera, accelerometer, etc., as described herein) Interprets each position where is a character for that position. However, the system may monitor the movement, swiping of the user's finger or other pointing device across the virtual keyboard and determine the best match for the movement of the pointer. In the figure, the pointer starts at the letter 'w' and traverses path 6304 through the letters e, r, t, y, u, i, k, n, b, v, f, and d (stopping here). The eyepiece observes this sequence and determines the sequence through the input path analyzer, feeds the sensed sequence to the word matching search facility, and outputs 'wind' as the best-fit word, in this case text 6308 . In embodiments, the eyepiece may provide a best-fit word, a list of best-fit words, and the like. In an embodiment, the user wears an interactive head-mounted eyepiece, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and display content. The optical assembly may include a correction element for correcting the user's view of the surrounding environment, an integrated processor for processing the content for display to the user, and an integrated image source for directing the content to the optical assembly. The displayed content may include an interactive keyboard control element (eg, virtual keyboard, calculator, touch screen, etc.), wherein the keyboard control element includes an input path analyzer, a word matching search facility, and a keyboard input interface; related The user may enter text by sliding a pointing device (eg finger, stylus, etc.) across the letter keys of the keyboard input interface in a sliding motion through the approximate sequence of words the user wishes to enter as text and , where the input path analyzer determines the touched characters in the input path, and the word matching facility matches the order of the touched characters and inputs the best match of words as the input text.

In embodiments, the eyepiece facility may provide for presentation display content corresponding to an identified marker indicating an intent to display the content. That is, the eyepiece may be commanded to display specific content according to a predetermined external visual cue sensing. The visual cue can be such as an image, icon, photo, facial recognition, hand composition, body composition, and the like. The display content may be an interface device brought for use, such as navigation aids to help users find a location when people arrive at a travel location, advertisements when the eyepiece views a target image, information profiles, and the like. In embodiments, the visual marker cues and their associated content for display are stored in the eyepiece's memory and stored and imported to an external computer facility as needed (eg, geographic location, proximity to a trigger target, user's command, etc.) , created by a third party, and so on. In an embodiment, the user wears an interactive head-mounted eyepiece, wherein the eyepiece includes an optical assembly through which the user views the surrounding environment and display content. The optical assembly may include a corrective element for correcting the user's view of the surrounding environment, an integrated processor for processing content for presentation to the user, and an integrated image source for directing the content to the optical assembly. An integrated camera facility may be provided, which images the visual cue, wherein the integrated processor identifies and translates the external visual cue as instructions for presenting content associated with the visual cue. Referring to FIG. 64 , in an embodiment, a visual cue 6412 may be included in a signal 6414 in the surrounding environment, where the projected content may be associated with an advertisement. The signal may be an advertisement for a billboard, personalized advertisement based on the user's preference profile. Visual cues 6402 and 6410 may be hand gestures, and projected content projected virtual keyboards 6404 and 6408 . For example, there may be thumb and index finger gestures 6402 from a first user's hand, and a virtual keyboard 6404 projected onto the palm of the first user's hand, where the user moves to a second user's hand. You can type on the upper keyboard. The hand gesture 6410 may be a combination of the thumb and forefinger of both hands of the user, and a virtual keyboard 6408 projected between the user's hand constructed in the hand gesture, where the user uses the thumb of the user's hand to You can type on the virtual keyboard. Virtual cues provide the wearer of the eyepiece with an automatic resource for associating a predetermined external visual cue with a desired outcome by way of projected content, freeing the user from searching for the cue itself.

Eyepieces can be useful for a variety of applications and markets. It should be understood that the control mechanisms described herein may be used to control the functionality of the applications described herein. The eyepiece may execute one application at a time, or multiple applications may be executed at a time. Switching between applications may be performed using the control mechanisms described herein. Eyepieces include e-books, GPS navigation (location, direction, speed and ETA), mobile TV, athletics (pacing, ranking, and views of competitive times; receiving coaching), telemedicine, industrial inspection, aviation, shopping, inventory Can be used in military applications, games, image recognition applications to view/order management tracking, firefighting (enabled by VIS/NIRSWIR sensors that see through fog, haze, darkness), outdoor/adventure, custom advertisements, etc. . In one embodiment, the eyepiece may be used for e-mail such as GMAIL in FIG. 7 , the Internet, web browsing, viewing sports scores, video chatting, and the like. In one embodiment, the eyepiece may be used for educational/training purposes by displaying step-by-step guides, such as hands-free, wireless maintenance and repair instructions. For example, video instructions and/or instructions may be displayed in the field of view. In one embodiment, the eyepiece may be used in fashion, health and beauty. For example, a potential outfit, hairstyle, or makeup may be projected onto a mirror image of the user. In one embodiment, the eyepiece may be used for business intelligence, conferencing and conferences. For example, a user's name tags can be scanned, their faces executed through a facial recognition system, or their voice names are searched in a database for biometric information. Scanned name tags, faces, and conversations can be recorded for subsequent viewing or filing.

In one embodiment, "Mode" may be input by the eyepiece. In this mode, certain applications are available. For example, the consumer version of the eyepiece may have a tourism mode, an education mode, an internet mode, a TV mode, a game mode, a practice mode, a stylist mode, a personal assistant mode, and the like.

A user of augmented reality glasses may wish to participate in a video call or video conference while wearing the glasses. Many computers, desktops and laptops all have integrated cameras that facilitate using video calls and conferencing. Typically, software applications are used to incorporate the use of cameras with call or conference features. With augmented reality glasses that provide many of the capabilities of laptops and other computing devices, many users may wish to utilize augmented reality glasses for video calling and video conferencing while on the go.

In one embodiment, the video calling or video conferencing application may work with a Wi-Fi connection or be part of a 3G or 4G phone network associated with the user's mobile phone. A camera for a video call or conference is placed on a device controller, such as a watch or other separate electronic computing device. Deploying a video call or conferencing camera on augmented reality glasses is not feasible when such an arrangement only provides the user a view of himself and does not display other participants in the meeting or call. However, users may choose to use the forward facing camera to display their surroundings or other individuals during video calls.

58 shows a typical camera 5800 for use in a video call or conference. Such cameras are generally small and may be mounted on a watch 5802, mobile phone, or other portable computing device, including a laptop computer, as shown in FIG. 58 . Video calling can work by connecting a controller with a cell phone or other communication device. The device uses software compatible with the operating system of the glasses and the communication device or computing device. In one embodiment, the screen of the augmented reality glasses displays a list of options to make a call, and the user uses a pointing control device to make a gesture or to select a video call option on the screen of the augmented reality glasses as described in the text. Other control device technologies may be used.

59 shows an embodiment of a block diagram of a video call camera 5900 . The camera incorporates a lens 3302 , a CCD/CMOS sensor 3304 , an analog-to-digital converter for an image signal 3306 , and an audio signal 3314 . Microphone 3312 collects audio input. Both analog-to-digital converters 3306 and 3314 send their output signals to signal enhancement module 3308 . Signal enhancement module 3308 forwards the enhanced signal, which is a synthesis of video and audio signals to interface 3310 . The interface 3310 is coupled with the control module 3316 to the IEEE 1394 standard bus interface.

In operation, the video telephony camera is subject to signal capture that converts incident sound as well as incident light into electrons. For light, this process is performed by a CCD or CMOS chip 3304 . A microphone converts sound into electrical impulses.

The first step in the process of creating an image for a video call is to digitize the image. A CCD or CMOS chip 3304 analyzes the image and converts it into pixels. If the pixel has collected many photons, the voltage will be high. If the pixel has collected a small number of photons, the voltage will be lowered. These voltages are analog values. During the second stage of digitization, the voltage is converted to a digital value by an analog-to-digital converter 3306 that handles image processing. At this point, a raw digital image is available.

Audio captured by microphone 3312 is also converted to voltage. This voltage is sent to an analog-to-digital converter 3314 which converts the analog value to a digital value.

The next step is to enhance the signal so that it can be sent to the video call or conference viewer. Signal enhancement involves creating color in the image using a color filter, placed in front of a CCD or CMOS chip 3304 . These filters are red, green, or blue, change their color from pixel to pixel, and may be, in an embodiment, a color filter array, or a Bayer filter. These raw digital images are then enhanced by filters to meet aesthetic requirements. Audio data can be enhanced for a better phone experience.

In a final step before transmission, the image and audio data are compressed and output as a digital video stream, in one embodiment, using a digital video camera. When using a photo camera, a single image may be output, and in a further embodiment, a voice comment may be appended to the file. The enhancement of raw digital data may occur remotely from the camera and, in one embodiment, may occur at the computing device with which the device controller or augmented reality glasses communicate during a video call or conference.

Embodiments may also provide a portable camera for use in industry, medicine, astronomy, microscopy, and other fields requiring specialized camera use. These cameras often stop improving the signal and output a raw digital image. These cameras may be mounted on other electronic devices or on the user's hand for ease of use.

The camera interfaces to the augmented reality glasses and device controller or computing device using an IEEE 1394 interface bus. These interface buses transport time-critical data such as video and data whose integrity is critically important, including parameters or files for manipulating data or transferring images.

In addition to the interface bus, protocols define the behavior of devices involved in video calls or conferences. A camera for use with augmented reality glasses may, in embodiments, employ one of a protocol such as AV/C, DCAM, or SBP-2.

AV/C is a protocol for audio video control and defines the operation of digital video devices including video cameras and video recorders.

DCAM refers to the 1394-based digital camera specification and defines the operation of a camera that outputs uncompressed image data without audio.

SBP-2 stands for Serial Bus Protocol and defines the behavior of mass storage devices such as hard drives or disks.

Devices using the same protocol can communicate with each other. Thus, to make a video call using augmented reality glasses, the same protocol can be used by the device controller and the video camera on the augmented reality glasses. Because the augmented reality glasses, the device controller, and the camera use the same protocol, data can be exchanged between these devices. Files that can be transferred between devices include: image and audio files, image and audio data flows, parameters that control cameras, and the like.

In one embodiment, a user wishing to initiate a video call may select a video call option from a screen presented when the call process is initiated. The user makes a selection by gesturing using a pointing device, or gesture, to signal the selection of a video call option. The user then positions the camera on a device controller, wrist watch, or other detachable electronic device such that an image of the user is captured by the camera. The images are processed through the process described above and then streamed to augmented reality glasses and other participants for display to the user.

In embodiments, the camera may be mounted on a cell phone, personal digital assistant, wrist watch, pendant, or other small portable device that may be carried, worn or mounted. An image or video captured by the camera can be streamed to the eyepiece. For example, when the camera is mounted on a gun, the wearer may image a target that is not in the line of sight and wirelessly receive the image as a stream of content displayed with the eyepiece.

In an embodiment, the present disclosure may provide GPS-based content reception to the wearer, as shown in FIG. 6 . As mentioned above, the augmented reality glasses of the present disclosure may include a memory, a global positioning system, a compass or other orientation device, and a camera. The GPS-based computer programs available to the wearer may include a number of applications generally available from Apple Inc.'s App Store for iPhone use. Similar versions of these programs can be used with other brands of smart phones and are applicable to embodiments of the present disclosure. Such programs include, for example, SREngine (scene recognition engine), NearestTube, TAT Augmented ID, Yelp, Layar, and TwittARound as well as more specialized applications such as RealSki.

SREngine is a scene recognition engine that can identify objects viewed by the user's camera. It is a software engine capable of recognizing scenes such as architecture, structures, paintings, objects, rooms, etc. It can then automatically apply virtual "labels"' to structures or objects as it recognizes them. For example, as in FIG. 6 , when looking at a street scene, a program may be invoked by a user of the present disclosure. Using augmented reality glasses camera, the engine recognizes the Fontaine de la Concorde in Paris. The program will then call the virtual label shown in FIG. 6 as part of the virtual image 618 projected by the lens 602 . The label may be text only, as shown at the bottom of image 608 . Other labels applicable to this scene could include "fountain", "museum", "hotel" or the name of the columnar building in the back. Other programs of this type may include Wikitude AR Travel Guides, Yelp, and many others.

NearestTube, for example, uses the same technology to direct users to the nearest subway station in London, and other programs may perform the same or similar tasks in other cities. Layar is another application that uses your camera, compass or direction, and GPS data to identify your location and field of view. With this information, an overlay or label can appear virtually to help orient and guide the user. Although Yelp and the eyeglasses perform similar functions, their databases are somewhat more specialized, helping to direct users in a similar way to restaurants or other service providers.

The user can control the glasses and invoke these functions using any of the controls described in this patent. For example, the glasses may be equipped with a microphone to select voice commands from the user and process them using software contained in the memory of the glasses. The user can then also respond to prompts from small speakers or earbuds contained within the glasses frame. The glasses can also be equipped with a small trackpad similar to those found on smartphones. A trackpad may allow a user to move a pointer or indicator on a virtual screen in AR glasses similar to a touch screen. When the user reaches a desired point on the screen, the user depresses the trackpad to indicate his/her selection. Thus, the user can call up a program, for example a travel guide, and then find his way through a number of menus, select a country, city, and then a category. Category selection may include, for example, hotels, shopping, museums, restaurants, and the like. Users make their own choices and are then guided by the AR program. In one embodiment, the glasses also include a GPS locator, and the current country and city provide default locations that can be overridden.

In one embodiment, the eyepiece's object recognition software may process images received via the eyepiece's forward-facing camera to determine what is in the field of view. In another embodiment, the GPS coordinates of the location determined by the GPS of the eyepiece may be sufficient to determine what is in the field of view. In other embodiments, an RFID or other beacon in the environment may broadcast the location. One or a combination of the above may be used by the eyepiece to identify location and field of view.

When an object is recognized, the resolution of imaging the object may be increased or an image or video may be captured with a lower compression ratio. Also, the resolution for other objects in the user's view may be reduced and captured with a higher compression ratio to reduce the bandwidth required.

Once determined, the content relating to the point of interest in the field of view may be superimposed on real images, such as social networking content, interactive tours, local information, and the like. Information and content regarding movies, local information, weather, restaurants, restaurant availability, local events, local taxis, music, etc. can be accessed by the eyepiece and projected onto the lens of the eyepiece for the user to view and interact with. For example, when a user views the Eiffel Tower, the forward-facing camera can take an image and send it to a processor associated with the eyepiece for processing. The object recognition software may determine whether the structure in the wearer's field of view is the Eiffel Tower. Alternatively, the GPS coordinates determined by the eyepiece's GPS may be searched in a database to determine if the coordinates match those of the Eiffel Tower. In any case, the content may then be retrieved for information about Eiffel Tower visitors, restaurants nearby and on the tower itself, local weather, local metro information, local hotel information, other nearby attractions, and the like. Interacting with the content may be enabled with the control mechanisms described herein. In one embodiment, GPS-based content reception may be enabled when the eyepiece's tourism mode is entered.

In one embodiment, the eyepiece may be used to view streaming video. For example, a video may be identified through a search by GPS location, a search by object recognition of an object in the field of view, a voice search, a holographic keyboard search, and the like. Continuing with the example of the Eiffel Tower, the video database can be searched via the GPS coordinates of the tower or by the term 'Eiffel Tower' if it is determined that there is structure in view. Search results may include geo-tagged videos or videos associated with the Eiffel Tower. The video may be scrolled or flipped using the control techniques described herein. The video of interest may be played using the control techniques described herein. The video may be placed overlaid on the actual screen, or displayed on a lens that is out of view. In one embodiment, the eyepiece may be dimmed via the mechanism described herein to enable higher contrast viewing. In another example, the eyepiece may use a camera and network connection, as described herein, to provide streaming video conferencing capabilities to the wearer.

As noted, users of augmented reality may receive content from rich sources. Visitors or tourists may wish to limit their choices to local businesses or institutions; On the other hand, businesses seeking visitors or tourists may want to limit their offers or solicitations to people who are in the area or location but are not local residents. Thus, in one embodiment, a visitor or traveler may limit their search to only local businesses, which may be referred to as being within certain geographic limits. These restrictions may be set via GPS criteria or by manually indicating geographic restrictions. For example, the source from which a person streams content or advertisements may require that they be restricted within a certain radius (set number, or km or miles) of that person. Alternatively, the criteria may require that the source be limited to those within a particular city or region. When a user at home or office limits their search using a keyboard or mouse, this limit may be set by the augmented reality user; Entry for the augmented reality user may be made by voice, hand movement, or other manners described in the portions of this disclosure that disclose control.

Additionally, the available content from which the user is selected may be limited or limited by the type of provider. For example, a user may limit choices to websites operated by a government agency (.gov) or by a non-profit organization or organization (.org). In this way, tourists or visitors who may be more interested in visiting government buildings, museums, historical sites, etc. may find their choices less crowded. A person can make a choice more easily when the available choices are reduced to a more reasonable number. The ability to quickly reduce available choices is desirable in more urban areas such as Paris, or Washington DC, where there are many more options.

The user controls the glasses in any manner or mode described in this patent. For example, a user may invoke a desired program or application by voice or by indicating a selection on a virtual screen of augmented reality glasses. The augmented glasses may be responsive to a track pad mounted to the frame of the glasses, as described above. Alternatively, the glasses may correspond to one or more motion or position sensors mounted to the frame. The signal from the sensor is then sent to a microprocessor or microcontroller in the glasses, which also provides any necessary signal conversion or processing. Once program selection is initiated, the user makes the selection and responds by any method disclosed herein, such as signaling a "yes" or "no" with a head movement, hand gesture, trackpad press, or voice command. Enter

At the same time, content providers, i.e. advertisers, may also limit their offers to those who are within certain geographic areas, for example within their city limits. At the same time, an advertiser, perhaps a museum, may not want to provide content to locals, but may want to reach out to visitors or outsiders outside the city. The augmented reality device disclosed herein preferably has both a GPS function and a telecommunication function. This will be a simple problem for museums that want to provide streaming content within a limited area by limiting their broadcasting power. Museums may, however, provide content via the Internet, and their content may be available worldwide. In this case, the user may receive content advising that the museum is open today and that a tour is possible through the augmented reality device.

The user may respond to the content by the augmented reality equivalent of clicking on a link to the museum. The augmented reality equivalent may be an indication of a voice indication, hand or eye movement, or other sense of the user's selection, or may be using an associated body-mounted controller. The museum then receives a cookie indicating the identity of the user or at least the user's Internet Service Provider (ISP). If the cookie indicates or suggests an Internet service provider rather than a local provider, the museum server can then respond with advertisements or offers tailored to the visitor. Cookies may include an indication of a communication link, for example a phone number. If the phone number is not an area code, this becomes an additional queue that the person responding is a visitor. A museum or other institution can then follow up with content it wants or suggested by its own marketing department.

Another application of the augmented reality eyepiece utilizes the user's ability to control the eyepiece and its tools with minimal use of the user's hand, using gestures or gestures, in lieu of voice commands. As described above, the user may invoke the augmented reality eyepiece to retrieve information. This information may already be stored in the eyepiece's memory, but instead may be placed remotely, such as an intranet accessible only to employees of a particular company or organization, or a database accessible over the Internet. The eyepiece can thus be compared to a computer or display screen that can be viewed and heard from an extremely close range and can generally be controlled with minimal use of the user's hand.

The application can thus provide information in the field to a mechanic or an electronic technician. For example, a technician may wear glasses when trying to find information about a specific structure or problem that arises when servicing an engine or power supply. Using voice commands, he can then access the database and search within the database for specific information, such as manuals or other repair and maintenance documents. The desired information can thus be accessed and applied immediately with minimal effort, allowing technicians to perform the necessary repair or maintenance more quickly and return the equipment for service. For mission-critical equipment, this time savings can save not only repair or maintenance costs, but also lifespans.

The information communicated may include things such as repair manuals, but may also include a full range of audio-visual information, ie, the eyepiece screen may be directed to the technician or mechanic at the same time the person is attempting to perform the task. You can display a video of how to perform the task. The augmented reality device also includes a communication function, so that the technician also has the ability to call for help if there is some complexity or unexpected difficulty in the task. This educational aspect of the present disclosure is not limited to maintenance and repair, but may be applied to educational initiatives such as secondary or higher classes, continuing education courses or topics, seminars, and the like.

In one embodiment, the Wi-Fi enabled eyepiece may launch a location-based application for a user opted-in geolocation. Users can participate by logging in with an application on their phone and enabling broadcast of their location, or by enabling geo-location on their own eyepiece. The wearer of the eyepiece thus scans people, scanning their opted-in device, while the application identifies the engaged user and sends commands to the projector to project augmented reality indicators onto the selected user in the user's field of view. . For example, a green ring may be placed around people who have agreed to have their location visible. In another example, yellow rings may indicate people who agree but do not meet some criteria, such as not having a Facebook account, or having no mutual friends even if they have a Facebook account.

Some social networking, job networking, and dating applications may work with location based applications. Software residing on the eyepiece can coordinate location-based applications and data from networking and dating sites. For example, TwittARound is one program that uses an onboard camera to detect and label location-stamped tweets from other tweeters nearby. This will allow a person using the present disclosure to locate other Twitter users in the vicinity. Alternatively, the user may need to set up the device to coordinate information from various networking and dating sites. For example, the wearer of the eyepiece may want to see all E-HARMONY users broadcasting their location. Once the consenting user is identified by the eyepiece, an augmented reality indicator may be placed over the consenting user. The indicator may take on a different appearance, such as when the user is common with the wearer, has multiple things in common with the user, and so on. For example, and with reference to FIG. 16 , two persons are being viewed by the wearer. Everyone is identified as an E-HARMONY user by a ring placed around them. However, the woman shown in solid lines has one or more items in common with the wearer, while the woman shown in dashed lines does not have items in common with the wearer. Any profile information is accessed and displayed for the user.

In one embodiment, when the wearer instructs the eyepiece at the direction of the user with a networking account such as Facebook, TWITTER, BLIPPY, LINKEDIN, GOOGLE, Wikipedia, etc., the user's recent posts or profile information may be displayed to the wearer. For example, recent status updates, such as "Twitter", "blips", etc. may be displayed on TwittARound as described above. In one embodiment, when the wearer points the eyepiece in the direction of the user at the target, they indicate interest in the user if the eyepiece is pointed for a period of time, and/or a gesture, head, eye or audio control is activated. The target user may receive an indication of interest in his or her cell phone or glasses. If the target user has marked the wearer with interest but is waiting for the wearer to show interest first, the indication may immediately pop up on the eyepiece of the target user's interest. The control mechanism may be used to capture the image and store the target user's information in an associated non-volatile memory or online account.

In other applications for social networking, facial recognition programs may be used, such as TAT Augmented ID from TAT-The Astonishing Tribe, Malmö, Sweden. Such programs can be used to identify people by their facial characteristics. Such software uses facial recognition software to identify people. Other applications, such as Flickr's photo identification software, can be used to identify specific people around you, and then you can take information about that person and download the information from social networking sites. Such information may include the person's name and the profile that person has provided to sites such as Facebook, Twitter, etc. Such applications can be used to refresh a person's user's memory or to identify close people, as well as gather information about that person.

In another application for social networking, the wearer may use the location-based facility of the eyepiece to leave notes, comments, reviews, and the like at location, in relation to people, places, products, and the like. For example, a person may comment on a location they have visited, where the post may then be made available to other users via a social network. In another example, a person may post the comment at the location of the place so that others can use the comment when they reach the location. In this way, the wearer can access comments left by others when they come to the location. For example, a wearer may come to the entrance of a restaurant and access reviews for the restaurant, sorted by certain criteria (eg most recent review, age of reviewer, etc.).

The user can, as described above, use the trackpad to select the desired program by voice, by selecting from the virtual touch screen, or by any control technique described herein can start the program. The menu selection may then be changed in a similar or complementary manner. A sensor or input device mounted at a convenient location on the user's body may be used, for example the sensor and trackpad would be mounted on a wristband, glove, or inconspicuous device, perhaps the size of a smart phone or personal digital assistant. can

Applications of this disclosure may provide the wearer with Internet access, such as browsing, searching, shopping, entertainment, etc., via a wireless communication interface to the eyepiece. For example, the wearer may include a part of the wearer's body (eg, on hands, head, or feet), some component used by the wearer (eg, personal computer, smartphone, music player), furniture adjacent to the wearer (eg, For example, a web search is initiated with a control gesture through a control device worn on a chair, desk, table, lamp, etc., where an image of the web search is projected for the wearer to see through the eyepiece. The wearer can view searches through the eyepiece and control web interactions through the control facility.

In an example, the user may be wearing the embodiment configured as a pair of glasses, with the projected image of the Internet web browser provided through the glasses while maintaining the ability to simultaneously see at least part of the surrounding real environment. In this case, the user may be wearing a motion-sensitive control device over his or her hand, wherein the control device is the control motion of the user's hand against the eyepiece as a control motion for web control, similar to a mouse in a conventional personal computer configuration. Relative motion can be transmitted. It will be appreciated that a user may perform web actions in a manner similar to a conventional personal computer configuration. In this case, the image of the web search is provided through the eyepiece while the control for selection of the action to perform the search is provided through the motion of the hand. For example, a general movement of a hand may move a cursor within a projected image of a web search, a flick of a finger(s) may provide a selection action, and the like. In this manner, the wearer may perform a desired web search, or function using another Internet browser, through an embodiment connected to the Internet. In one embodiment, the user may use the computer program Yelp or Monocole available on the App Store, similar products such as NRU ("near you"), applications from Zagat for locating adjacent restaurants or other stores, Google Earth, Wiki You can download an encyclopedia (Wikipedia), etc. A person may initiate a search for information, or other suppliers for goods or services, such as restaurants or hotels, repairmen, and the like. When the desired information is found, the location is indicated, or the distance and direction to the desired location are indicated. The display may take the form of a virtual label that is positioned with the real object in the user's view.

Another application from Layar (Amsterdam, Netherlands) contains various "layers" tailored to the specific information desired by the user. A layer may contain restaurant information, information about a specific company, real estate listings, gas stations, and the like. Using the information provided by these mobile applications and software applications such as the user's global positioning system (GPS), the information can be displayed on the screen of the glasses as tags with the desired information. Using haptic controls or other controls discussed in this disclosure, a user can pivot or rotate their body and view buildings tagged with virtual tags containing information. When a user finds a restaurant, the screen displays restaurant information such as name and location. When the user is looking for a specific address, the virtual tag appears on the building in the wearer's field of view. Users can select by voice, trackpad, virtual touch screen, and the like.

Applications of this disclosure may provide a method for advertisements to be delivered to a wearer. For example, advertisements may be displayed to the viewer via the eyepiece as the viewer progresses about their daily life, such as browsing the Internet, performing a web search, walking through a store, and the like. For example, a user may perform a web search, and through the web search, the user may be targeted with advertisements. In this example, the advertisement may be projected in the same space as the projected web search and floated to the side, top, or bottom of the wearer's viewing angle. In another example, an advertisement may be triggered to deliver to the eyepiece when some advertisement presents a facility, perhaps a facility adjacent to the wearer, detects the presence of the eyepiece (via wireless connection, RFID, etc.), and sends the advertisement to the eyepiece. orientate

For example, the wearer may do window shopping in Manhattan, where stores are equipped with such advertisements providing facilities. As the wearer walks through the store, advertisements offering the facility may trigger the transmission of advertisements to the wearer based on the user's known location as determined by the eyepiece's integrated location sensor, such as GPS. In one embodiment, the user's location may be further refined through other integrated sensors, such as magnetometers, that enable very small hyperlocal augmented reality advertisements. For example, a user on the first floor of a shopping mall may receive a particular advertisement if magnetometer and GPS readings position the user in front of a particular store. As the user ascends further in the shopping mall, the GPS location may remain the same, but the magnetometer reading may indicate the user's elevation change and the user's new placement in front of a different store. In embodiments, the advertisement serving facility may store personal profile information to better match the wearer's needs, the wearer may provide preferences for advertisements, the wearer may block at least some of the advertisements, and the like. can The wearer may pass on advertisements, and related discounts, to friends. the wearer can communicate directly to a friend who is nearby and enabled with his or her eyepiece; They can also communicate with them via wireless internet connection, e-mail, SMS, etc. to friends on social networks. the wearer enables communication of advertisements from sponsors of advertisements to the wearer; feedback from the wearer to advertising facilities, sponsors of advertising, and the like; It may be connected to facilities and/or infrastructure, which may provide feedback to other users, such as friends and family, or to people in the vicinity of the wearer, to a store locally or remotely on the eyepiece, to the Internet or to a computer in the user's home, or the like. These interconnecting devices use GPS, triaxial sensors, magnetometers, gyros, accelerometers, etc. to determine the direction, speed, and posture (eg, gaze direction) of the wearer to provide the user's position and gaze direction to the eyepiece. may include equipment integrated into The interconnection facility may provide a communication facility such as a wireless link, a WiFi/MiFi bridge, and the like. For example, the wearer may communicate via an available WiFi link to a wireless cellular system, an integrated MiFi (or other individual or group's cellular link). There may be facilities for the wearer to store advertisements for later use. It may be integrated with the wearer's eyepiece or placed on a local computer facility that enables caching of advertisements, such as within a local area, wherein the cached advertisements enable delivery of advertisements when the wearer is proximate to a location associated with the advertisements. can do it For example, local advertisements may be stored on a server containing geo-location local advertisements and specials, and these advertisements may be delivered to the wearer personally as the wearer approaches a particular location, or may be entered by the wearer into a geographic area. When a set of advertisements are delivered to the wearer in bulk, the advertisements become available when the user is proximate to a particular location. A geographic location can be a city, part of a city, multiple blocks, a block, street, part of a street, etc., representing a province, region, or very narrow local area. Note that while the above discussion uses the term advertisement, one of ordinary skill in the art will understand that it can mean announcement, broadcast, prototype, commercial, sponsored communication, endorsement, notice, publicity, bulletin board, message, and the like.

18-20A illustrate a manner of delivering a custom message to a person within a short distance facility that wishes to send the message, such as a retail store. Referring to FIG. 18 , an embodiment provides a way for the wearer of the eyepiece to view a custom billboard, such as when walking or driving, by the above-described application for a search for providers of goods and services. can As shown in FIG. 18 , a sign 1800 shows an exemplary augmented reality based advertisement displayed by a seller or service provider. As shown, an example advertisement may be directed to serving beverages by a bar. For example, two drinks can be served for the cost of just one drink. With these augmented reality based advertisements and offers, the wearer's attention can be easily directed to the signage. Signage can also provide detailed information about the location of the bar, such as street address, floor number, phone number, etc. According to another embodiment, several devices other than the eyepiece may be utilized to view the sign. Such devices may include, without limitation, smart phones, iPhones, IPADs, car windows, user glasses, helmets, wrist watches, headphones, vehicle mounts, and the like. According to an embodiment, the user (the wearer if the augmented reality technology is included in the eyepiece) may automatically receive suggestions from the signage and view the screen when the user is passing or driving along the road or driving. According to another embodiment, the user may receive an offer according to his/her request or view the screen of the signboard.

19 illustrates two example roadside billboards 1900 containing offers and advertisements from vendors or service providers that may be viewed in an augmented reality manner. Augmented advertisements may provide a live and near-to-reality perception to a user or wearer.

As shown in FIG. 20 , an augmented reality enabled device, such as a camera lens provided in the eyepiece, may be utilized to receive and/or view graffiti 2000, slogans, drawings, etc., whether it is on the roadside or at the top of buildings and stores, It can be displayed on the side or front. Roadside billboards and graffiti may have visual (eg, code, shape) or wireless indicators that may link advertisements, or advertisement databases, to the billboards. As the wearer approaches and views the billboard, a projection of the billboard advertisement may be provided to the wearer. In embodiments, personal profile information may be stored so that advertisements may better match the needs of the wearer, the wearer may provide preferences for advertisements, the wearer may block at least some of the advertisements, and the like. In embodiments, the eyepiece may have brightness and contrast controls for the projected area of the signage of the eyepiece to improve readability for advertisements, such as in bright outdoor environments.

In other embodiments, a user may post information or messages about a particular location based on their GPS location or other indicator of location, such as a magnetic field reading. The intended viewer may see the message when the viewer is within a location at a certain distance, as illustrated in FIG. 20A . In a first step of the method of Figure 20a, the user determines where the message is received by the person to whom the message is to be sent. The message is then posted (2003) to be sent to the appropriate person or persons when the recipient is in proximity to the intended "viewing area". The location of the wearer of the augmented reality eyepiece is continuously updated (2005) by a GPS system that forms part of the eyepiece. When the GPS system determines that the wearer is within the desired viewing area at a certain distance, for example 10 meters, a message is sent to the viewer (2007). In one embodiment, the message may be displayed to the recipient as an email or text message, or if the recipient is wearing the eyepiece, the message may appear on the eyepiece. Because messages are sent to a person based on their location, in a sense, the message can be displayed as "graffiti" about a building or feature at or in a designated location. Certain settings may be used to determine whether all passersby to a "viewing area" can see the message or only a specific person or group of people or devices with a specific identifier can see the message. For example, a soldier clearing a village can virtually mark a house as cleared by associating a message or identifier with the house by making a large X marking the location of the house. Soldiers can indicate that only other US troops can receive location-based content. When other US troops pass the house, they automatically receive an indication by seeing a virtual 'X' on the side of the house if they have an eyepiece or other device that supports augmented reality, or by receiving a message indicating that the house has been cleared. can do. In another example, content associated with safety applications, such as alerts, target identification, communications, and the like, may be streamed to the eyepiece.

Embodiments may provide a way to view information associated with the same product in a store. The information may include nutritional information about food products, care instructions for clothing products, technical specifications for consumer electronics products, electronic coupons, promotions, price comparisons with other similar products, price comparisons with other stores, and the like. This information may be projected to a location relative to the product, to the wearer's peripheral field of view, to the storage layout, and the like. Products are visually identified through SKUs, brand tags, etc.; conveyed by the packaging of the product, via the RFID tag on the product; It may be possible, for example, to be transported by the store based on the wearer's position of the store in the relative position of the product. For example, a viewer can stroll through a clothing store, and as they walk, information is provided about the clothes on the shelf, where the information is provided via the product's RFID tag. In one embodiment, the information may be conveyed as a list of information, as a graphical representation, as an audio and/or video presentation, or the like. In another example, the wearer is shopping for food, the advertising provision facility provides information to the wearer in association with products proximate to the wearer, and the wearer is informed when they select a product and view a brand, product name, SKU, etc. . In this way, the wearer may be provided with a more informational environment to shop efficiently.

One embodiment allows a user to receive or share information about shopping or an urban area through the use of an augmented reality enabled device, such as a camera lens fitted to the eyepiece of the exemplary sunglasses. This embodiment uses an augmented reality (AR) software application as described above with a search for providers of goods and services. In one scenario, the wearer of the eyepiece may walk down a street or market for the purpose of shopping. In addition, users can activate various modes that help define user preferences for specific scenarios or environments. For example, a user may enter a navigation mode through which the wearer may be guided across streets and markets for shopping for their favorite accessories and products. Modes may be selected, and various directions may be provided by the wearer through various methods, such as text commands, voice commands, and the like. In one embodiment, the wearer may give a voice command to select a navigation mode that may bring up an augmented display in front of the wearer. Augmented information may depict information related to the location of various stores and vendors in the marketplace, providing such things as current happy hours, current date and time, and the like, at various stores and various vendors. Various kinds of options may also be presented to the wearer. The wearer can scroll through options and walk a guided street through a navigation mode. Depending on the options provided, the wearer may choose a location that best suits their shopping, such as offers and discounts. The wearer gives a voice command to navigate towards the place, and the wearer can then be guided towards him. The wearer may also receive advertisements and receive offers automatically or upon request regarding current deals and promotions and events at locations of interest, such as nearby shopping stores. Advertisements, deals and offers may appear in the wearer's vicinity, and options may be displayed to purchase desired products based on the advertisements, deals and offers. The wearer can for example select a product and purchase it through Google Checkout. A message or e-mail may be displayed on the eyepiece similar to that shown in FIG. 7 along with information that the transaction for the purchase of the product has been completed. Product delivery status/information may also be displayed. The wearer can communicate further or inform friends and relatives about offers and events via social networking platforms, and also ask them to join.

In embodiments, a user may wear a head-mounted eyepiece, wherein the eyepiece includes an optical assembly through which the user may view the surrounding environment and displayed content. The displayed content may include one or more local advertisements. The position of the eyepiece may be determined by an integrated position sensor and local advertisements may be relevant to the position of the eyepiece. For example, the user's location may be determined via GPS, RFID, manual input, or the like. Further, the user may be walking next to the coffee shop, and based on the user's proximity to the coffee shop, an advertisement representing the store brand of coffee, similar to that shown in FIG. 19 , may appear in the user's field of view. have. Users may experience similar types of local advertising as they move about their surroundings.

In another embodiment, the eyepiece may include a capacitive sensor capable of detecting whether the eyepiece is in contact with human skin. Such a sensor or group of sensors may be placed on the eyepiece and/or the eyepiece arm in such a way that it can detect when the glasses have been worn by the user. In another embodiment, the sensor may be used to determine if the eyepiece is in a position that allows them to be worn by a user, for example, when the earpiece is in an unfolded position. Additionally, local advertisements can only be sent when the eyepiece is in contact with human skin, is in a wearable position, a combination of the two, is actually worn by a user, and the like. In other embodiments, local advertisements may be sent in response to the eyepiece being powered up, or in response to the eyepiece being powered up and worn by the user, or the like. For example, when a user is in proximity to a particular facility, an advertiser may choose to send local advertisements only when the user is actually wearing glasses and the glasses are powered to allow the advertiser to target advertisements to users at the appropriate time. can

According to another embodiment, the local advertisement may be presented to the user as a banner advertisement, two-dimensional graphic, text, or the like. Also, local advertisements may be associated with a physical aspect of the user's view of the surrounding environment. Local advertisements may also be displayed as augmented reality advertisements, wherein the advertisements may be associated with physical aspects of the surrounding environment. These advertisements may be two or three dimensional. For example, a local advertisement may be associated with a physical signage further shown in FIG. 18 , where the user's attention may be directed to displayed content showing beverages being poured from the signage 1800 onto an actual building in the surrounding environment. have. Local advertisements may also include sound displayed to the user via an earpiece, audio device, or other means. Additionally, local advertisements may be animated in embodiments. For example, a user may show beverages flowing from a sign to an adjacent building, and optionally the surrounding environment. Similarly, an advertisement may display other types of workouts as desired in the advertisement. Additionally, local advertisements may be displayed as three-dimensional objects that can relate to or interact with the surrounding environment. In embodiments where the advertisement is associated with an object in the user's view of the surrounding environment, the advertisement may remain associated with or adjacent to the object even when the user turns his/her head. For example, if an advertisement such as a coffee cup as described in FIG. 19 is associated with a particular building, then the coffee cup advertisement is associated with that building even when the user turns his/her head to view other objects in his/her environment. It can be maintained in a suitable position above it.

In another embodiment, local advertisements may be presented to a user based on a web search conducted by the user in which the advertisement is displayed in the content of web search results. For example, a user may search for "happy hour" when he is walking down the street, and in the content of the search result, a local advertisement advertising the beer price of a local bar may be displayed.

Also, the content of the local advertisement may be determined based on the user's personal information. The user's information may be used for web applications, advertising facilities, and the like. In addition, the web application, advertisement facility, or user's eyepiece may filter advertisements based on the user's personal information. In general, for example, a user may store personal information regarding likes or dislikes, and such information may be used to direct advertisements to the user's eyepiece. By way of specific example, a user may store data about his preference for a local sports team, and when advertisements are available, such advertisements with his or her favorite sports team may provide preferences and push against the user. . Similarly, a user's dislike can be used to exclude viewing certain advertisements. In various embodiments, advertisements may be cached on a server where the advertisements may be accessed and presented to a user by at least one of an advertisement facility, a web application, and an eyepiece.

In various embodiments, a user may interact with any type of local advertisement in a variety of ways. The user may request additional information about the local advertisement by making at least one of eye movements, body movements and other gestures. For example, if an advertisement is displayed to the user, he may wave his hand over the advertisement within his field of view, or move his eye over the advertisement to select a particular advertisement to receive more information about the advertisement. can do it In addition, the user may choose to ignore the advertisement by any movement such as eye movement, body movement, other gestures, etc. or control techniques described herein. In addition, the user may choose to ignore the advertisement by not selecting the advertisement for further interaction within a given time period so that it can be overridden by default. For example, if the user does not choose to gesture for more information from the advertisement within 5 seconds of the advertisement being displayed, the advertisement may be ignored by default and disappear from the user's view. The user may also choose not to allow local advertisements to be displayed, such that the user selects this option on a graphical user interface, or by turning this feature off via a control on the eyepiece.

In other embodiments, the eyepiece may include an audio device. Accordingly, the presentation content may also include local advertisements and audio that allow the user to hear messages or other sound effects when associated with local advertisements. For example, and referring to FIG. 18 , as the user watches the beer being poured, the user may actually hear the audio transmission corresponding to the action in the advertisement. In this case, the user can hear the bottle open and liquid pouring from the bottle onto the roof. In other embodiments, a descriptive message may be played, or general information may be provided as part of the advertisement. In embodiments, audio may be played as desired for all advertisements.

According to another embodiment, social networking may be facilitated using an augmented reality enabled device, such as a camera lens mounted on the eyepiece. This can be utilized to connect multiple users and others who do not have an augmented reality enabled device together that can share thoughts and ideas with each other. For example, the wearer of the eyepiece may be sitting on a school campus with other students. The wearer may connect with and send a message to the first student, who may be at the coffee shop. The wearer may inquire of the first student about a person who is interested in a particular topic, such as, for example, environmental economics. As other students pass through the wearer's field of view, a camera lens fitted inside the eyepiece can track and match the students against a networking database such as 'Google me', which may contain public profiles. Profiles of interested and associated persons from public databases can be displayed and popped up in front of the wearer on the eyepiece. Some of the profiles that may not be relevant may be blocked or seen as blocked by the user. Relevant profiles can be highlighted for quick reference by the wearer. The relevant profile selected by the wearer may be of interest to the subject environmental economics, and the wearer may also be associated with them. In addition, they may also be linked to the first student. In this way, a social network may be established by the wearer with the use of an eyepiece that is enabled as a feature of augmented reality. The social networks managed by the wearer and the conversations therein may be saved for future reference.

The present disclosure may be applied to real estate scenarios using an augmented reality enabled device, such as a camera lens fitted to an eyepiece. The wearer may wish to obtain information about a location where the user may be at a specific time, such as while driving, walking, jogging, etc., according to this embodiment. The wearer, for example, may want to understand the housing benefits and losses there. He may also want to get detailed information about the facilities there. Thus, the wearer can utilize maps such as Google's online maps and recognize real estate available for rent or purchase. As described above, a user may receive information about real estate for sale or lease by using a mobile Internet application such as Layar. In one such application, information about a building within the user's field of view is projected into the interior of the glasses for consideration by the user. Options, such as a trackpad mounted on the frame of the glasses, can be displayed to the wearer on the eyepiece lens for scrolling. The wearer may select and receive information about the selected option. An augmented reality-enabled scene of the selected option may be displayed to the wearer, and the wearer may view photos and tour the facility in a virtual environment. The wearer may also receive information about the real estate agent and arrange an appointment with one of them. Email notifications or phone notifications may also be received on the eyepiece to confirm appointments. When the wearer discovers the value of the selected real estate, a transaction can be made and it can be purchased by the wearer.

According to yet another embodiment, customization and sponsored tours and excursions may be enhanced through the use of an augmented reality enabled device, such as a camera lens fitted to the eyepiece. For example, the wearer (as a tourist) may wish to receive travel and tourism related information about places that have arrived in a city such as Paris and thus plan their visits for consecutive days during their stay. The wearer may wear their eyepiece or activate other augmented reality enabled devices and issue voice or text commands regarding their request. The augmented reality-enabled eyepiece can determine the wearer's tourism preferences by locating the wearer through geo-sensing technology. The eyepiece may receive and display customized information according to a request of the wearer on the screen. Personalized tourist information limits information on art galleries and museums, monuments and historical sites, shopping complexes, nightlife and nightlife attractions, restaurants and bars, most popular tourist destinations and centers/attractions, most popular local/cultural/local attractions, etc. can be included without Depending on the user's selection of one or more of these categories, the eyepiece may prompt the user with other questions, such as time of stay, investment in tourism, and the like. The wearer may respond via voice commands and in return receive personalized travel information with commands as selected by the wearer. For example, the wearer may prioritize an art gallery through a monument. Accordingly, information may be available to the wearer. Also, the map may be shown in front of the wearer with different sets of travel options with different priorities, such as:

Priority 1: 1st tour option (Alice on Champs Elysees, Louvre, Rodin, Museum, Famous Cafe)

Priority 2: Option 2

Priority 3: Option 3

Based on the preferences indicated by the wearer, the wearer may select the first option, for example, because the first option has been ranked as the highest priority. Advertisements related to the sponsor may be popped up on the right after selection. Subsequently, the virtual tour may begin in an augmented reality manner that may be very similar to the real environment. The wearer could, for example, take a 30-second tour to a special vacation to the Atlantis Resort in the Bahamas. Virtual 3D tours may include quick viewings in rooms, beaches, public places, parks, facilities, and the like. The wearer may also experience shopping facilities in the area and receive offers and discounts at the location and stores. At the end of the day, the wearer can experience a full-day tour while sitting in their chair or hotel. Finally, the wearer can decide and schedule their own plans accordingly.

Another embodiment may use an augmented reality enabled device, such as a camera lens fitted to the eyepiece, to allow information regarding automated repair and maintenance services. The wearer may receive advertisements about auto repair shops and dealers by sending a voice command to make a request. The request may include, for example, a requirement for an oil change in the vehicle/vehicle. The eyepiece may receive information from the repair shop and display it to the wearer. The eyepiece pulls out a 3D model of the wearer's vehicle and shows the amount of oil on the left side of the vehicle via an augmented reality enabled screen/view. The eyepiece may also display other relevant information about the wearer's vehicle, such as maintenance requirements in other parts, such as brake pads. The wearer may see a 3D view of the worn brake pad and may be interested in getting it repaired or changed. Thus, the wearer can use the eyepiece's integrated wireless communication capabilities to schedule an appointment with a vendor to address this issue. Confirmation may be received via email or incoming call notification on the eyepiece camera lens.

According to another embodiment, gift shopping may benefit from the use of an augmented reality enabled device, such as a camera lens fitted to the eyepiece. Wearers can post gift requests for select events via text or voice commands. The eyepiece may ask the wearer to respond to his or her preferences, such as the type of gift, the age group of the recipient of the gift, the cost range of the gift, and the like. Various options may be presented to the user depending on the received preference. For example, the options presented to the wearer could be: cookie basket, wine and cheese basket, chocolate assortment, golfer's gift basket, etc.

The available options may be scrolled by the wearer and the best suitable option may be selected via voice command or text command. For example, the wearer may select a golfer's gift basket. A 3D view of the golfer's gift basket along with the golf course may appear in front of the wearer. An activated golfer's gift basket and virtual 3D view of the golf course, enabled through augmented reality, allows for a very close perception of the real environment. The wearer can finally respond to address, location and other similar queries prompted via the eyepiece. Confirmation may be received via email or incoming call notification to the eyepiece camera lens.

Another application that may appeal to users is mobile online gaming using augmented reality glasses. These games may be computer video games, for example World of Warcraft? (WOW), supplied by Electronic Arts Mobile, UbiSoft and Activision Blizzard. Just as games and recreational applications are played on computers at home (rather than computers at work), augmented reality glasses can use gaming applications. A screen may be displayed on the inside of the glasses so that the user can observe and participate in the game. Additionally, controls for playing the game may be provided via a virtual game controller such as a joystick, control module, or mouse as described herein. The game controller may include a sensor or other type of element attached to the user's hand for feedback from the user via acceleration, vibration, force, electrical impulse, temperature, electric field sensing, and the like. Sensors and actuators may be attached to the user's hand in such a way as a wrap, ring, pad, glove, bracelet, or the like. Thus, the eyepiece virtual mouse allows a user to translate hand, wrist and/or finger movements into movement of a cursor on the eyepiece display, where "movement" means slow movement, fast movement, jerking movement, position, position. changes, etc., and allows the user to work in three dimensions without the need for a physical surface, including some or all of the 6 degrees of freedom.

27 , the game application may use both the Internet and GPS. In one embodiment, the game is downloaded from the customer database to the user's computer or augmented reality glasses, perhaps using the Internet as its own web service as shown, through the game company. At the same time, glasses that also have a communication function can send and receive communication and telemetry signals via cellular towers and satellites. Thus, the online gaming system can access information about the user's location as well as the user's desired gaming activity.

The game can utilize the knowledge of each of these players' positions. For example, a game could be made with a feature that uses a player's location via a GPS locator or a magnetic field locator to award points for reaching the location. The game may also send a message that displays, for example, a cue, or a scene or image when the player has reached a certain location. For example, a message may go to the next destination, which is then presented to the player. A scene or image may be presented as part of a struggle or obstacle to be overcome, or an opportunity to earn game points. Thus, in one embodiment, the augmented reality eyepiece or glasses may use the wearer's location to speed up and activate computer-based video games.

One method of playing an augmented reality game is shown in FIG. 28 . In this method, the user is allowed to log into the website and access the game. The game is selected. In one embodiment, the user participates in a game, and if multiplayer gaming is possible and required, the user can create a custom game, perhaps using the special role the user desires. The game may be scheduled, and in some cases, the player may select a specific time and place for the game, distribute directions to the site where the game is played, etc. Later, the player enters the game by meeting with one or more players using augmented reality glasses. Participants can then play the game and, where applicable, game results and statistics (player's score, game time, etc.) can be saved. When the game begins, the position may change for different players in the game, sending one player to one position or another player to a different position. The game can then have different scenarios for each player or group of players depending on their GPS or magnetometer provided location. Each player may also be sent different messages or images based on their role, their location, or both. Of course, each scenario can lead to different situations, different interactions, directions to different locations, etc. In a sense, these games mix the game in which the player participates and the player's actual location.

Games may vary from simple games of the type played in the palm of a player's hand, such as small, single player games. Alternatively, more complex, multiplayer games may also be played. In the previous category are games like SkySuege, AR Drone, and Fire Fighter (360). In addition, multiplayer games can also be easily envisioned. Since all players must log in to the game, certain games can be played by friends who log in and designate other people or people. The player's location may be used via GPS or other means. Sensors in augmented reality glasses or game controllers as described above, such as accelerometers, gyroscopes or magnetic compasses, may also be used for orientation and game play. An example is AR Invaders available for iPhone applications in the App Store. Other games can be obtained from other suppliers for non-iPhone type systems such as Layar of Amsterdam, which is a supplier of AR Drone, AR Flying Ace and AR Pursuit, and Paris SA of Paris, France.

In embodiments, the game may be 3D to allow the user to experience the 3D game. For example, when playing a 3D game, the user may view a virtual, augmented reality, or other environment in which the user may control the perspective of his/her view. The user may turn their head to view various aspects of the virtual environment or other environment. Thereby, when the user turns his head and makes other movements, he can see the game environment as if he were real in that environment. For example, the user's perspective allows the user to 'enter a 3D gaming environment' with at least some control over the perspective where the user moves their head and has a change of view of the gaming environment in response to the changed head position. . Also, the user can 'walk into' the game as he or she physically walks forward, and can have a change of perspective as the user moves. Additionally, the perspective view may also change as the user shifts the gaze of his or her eyes, and the like. Additional image information may be provided, such as in terms of the user's view, which can be accessed by turning the head.

In embodiments, the 3D gaming environment may be projected onto the lenses of the glasses or viewed by other means. Additionally, the lens may be opaque or translucent. In embodiments, the 3D game image is associated with and integrated with the user's external environment, allowing the user to turn his/her head and hold the 3D image and external environment together. Additionally, these 3D game images and their associations with the external environment may be altered so that in various instances the 3D image may show the user that the 3D image is interacting with various aspects or objects of the real environment, such that the 3D image is one or more objects or objects in the external environment. to be associated with one or more parts of For example, a user may see a 3D game monster climbing up a building or car where the building or car is a real object in the user's environment. In such a game, the user can interact with the monster as part of the 3D gaming experience. The real environment around the user can be part of the 3D gaming experience. In embodiments where the lens is transparent, the user may interact in the 3D gaming environment while moving relative to his or her real environment. A 3D game can incorporate elements of the user's environment into the game, it can be made entirely by the game, or it can be a mixture of the two.

In embodiments, the 3D image may be associated with or generated by an augmented reality program, 3D gaming software, etc., or other means. In embodiments where augmented reality is employed for 3D gaming purposes, a 3D image may appear or be recognized by the user based on the user's location or other data. Such an augmented reality application may provide a user to interact with such 3D image or images to provide a 3D gaming environment when using the glasses. For example, when the user changes his location, play in the game may proceed, and various 3D elements of the game may or may not be accessible to the viewer. For example, various 3D enemies of the user's game character may appear in the game based on the user's actual location. A user may interact with, or react from, other users playing the game and/or 3D elements associated with other users playing the game. Such elements associated with the user may include weapons, messages, calls, 3D images of the user, and the like. Based on the user's location or other data, the user encounters, views, or engages the other user with 3D elements associated with the other user by any means. In embodiments, the 3D game may also be provided by software installed on or downloaded to glasses with or without user location being used.

In embodiments, the lens may be opaque to provide the user with a virtual reality or other virtual 3D gaming experience in which the user 'enters' the game where the user's movement may change the perspective of the 3D gaming environment for the user. The user may use various body, head and/or eye movements, the use of a game controller, one or more touch screens, or as described herein that may allow the user to navigate, manipulate, and interact with the 3D environment to play 3D games. You can navigate through any control technology and navigate your virtual environment through it.

In various embodiments, the user navigates through the 3D gaming environment through body, hand, finger, eye, or other movement, using one or more wired or wireless controllers, one or more touch screens, any control technology described herein, etc. , you can interact with him, manipulate him, and experience 3D games.

In an embodiment, the internal and external equipment available for the eyepiece may learn the eyepiece user's motion, and perform location-aware control, activity-aware control, predictive control, and the like for the learned motion. It can be provided to store it in the action database. For example, a user may select commands from the user, images sensed through the camera, the user's GPS location, sensor inputs over time, actions triggered by the user, communications from and to the user, user requests, web tracking of events and/or actions recorded by the eyepiece, such as activities, music to listen to, directions requested, recommendations used or provided, and the like. Such motion data, either tagged with a user identifier or voluntarily, may be stored in a motion database. The eyepiece can collect such data in learning mode, collection mode, etc. The eyepiece utilizes historical data taken by the user to inform or remind the user of what they have done before, or alternatively, the eyepiece may determine which function of the eyepiece the user needs based on past collected experiences. and data to predict whether it is an application or not. In this way, the eyepiece functions as an automated assistant to the user, for example launching applications at the normal time the user launches them, turning off augmented reality and GPS when approaching a location or entering a building, Music can be streamed when users enter a gym or the like. Alternatively, learned actions and/or actions of a plurality of eyepiece users may be voluntarily stored in a collection action database, where learned actions among the plurality of users are available to individual users based on similar conditions. For example, a user may be visiting a city, waiting for a train on a platform, and the user's eyepiece may use the city to find directions, search an area of interest, listen to specific music, view train timetables, and travel information. Accessing a collection action database to determine what others have done while waiting for the train, such as accessing websites, accessing social networking sites for entertainment in the area, and the like. In this way, the eyepiece may provide the user with an automated assistant that utilizes a number of different user experiences. In embodiments, learning actions may be used to develop preference profiles, recommendations, ad targeting, social network connections, action profiles for users or groups of users, for/for users, and the like.

In one embodiment, the augmented reality eyepiece or glasses may include one or more acoustic sensors for sound detection. An example is shown in FIG. 29 above. In one aspect, an acoustic sensor is similar to a microphone in that it detects sound. Acoustic sensors generally have one or more frequency bands to which they are more sensitive, and the sensors can therefore be selected for their intended application. Acoustic sensors are available from a variety of manufacturers, and suitable transducers and other required circuitry are available. The manufacturer is ITT Electronic System, Salt Lake City, Utah, USA; Meggitt Sensing Systems, San Juan, Capistrano, CA, USA; and National Instruments of Austin, Texas, USA. Suitable microphones include an array of microphones or microphone arrays, as well as those having a single microphone.

The acoustic sensor may include using microelectromechanical systems (MEMS) technology. Because of the very precise structure in MEMS sensors, the sensors are extremely sensitive and generally have a wide range of sensitivities. MEMS sensors are typically made using semiconductor manufacturing techniques. The element of a typical MEMS accelerometer is a moving beam structure consisting of two sets of fingers. One set is fixed to a solid ground plane on the substrate; Another set is attached to a known mass mounted on a spring that can move in response to an applied acceleration. This applied acceleration changes the capacitance between the fixed and moving beam fingers. The result is a very sensitive sensor. Such sensors are made, for example, by STMicroelectronics of Austin, Texas, USA and Honeywell International, Morristown, New Jersey, USA.

In addition to identification, the voice function of the augmented reality device may also be applied to locating the origin of the sound. As is known, at least two sound or acoustic sensors are required for locating a sound. The acoustic sensor will be equipped with appropriate transducers and signal processing circuitry, such as a digital signal processor to interpret the signal and achieve the desired goal. An application for a sound locating sensor is to determine the origin of a sound from within a location, which is an emergency situation such as a burning building, car accident, etc. Emergency workers equipped with the embodiments described herein may each have one or more acoustic sensors or microphones built into the frame. Of course, the sensor may also be worn on or attached to the person's clothing. In either case, the signal is sent to the controller of the augmented reality eyepiece. The eyepiece or glasses are equipped with GPS technology, and may also be equipped with a direction-finding function; Alternatively, with two sensors per person, the microphone can determine the direction the noise is coming from.

2 If there are more than one firefighter or other emergency responder, their location is known from their GPS function. Either, or the fire chief, or control center, then learns the location of the two responders and the direction to the noise detected from each responder. The exact point at which the noise occurred can then be determined using known techniques and algorithms. See, for example, Aucoustic Vector-Sensor Beamforming and Capon Direction Estimation, M. Hawkes and A. Nehorai, IEEE Transactions on Signal Processing, vol. 46, no. 9, September 1998, 2291-2304; See also Cramer-Rao Bounds for Direction Finding by an Acoustic Vector Sensor Under Nonideal Gain-Phase Responses, Noncollocation or Nonorthogonal Orientation, PK Tam and KT Wong, IEEE Sensors Journal, vol. 9. No. 8, August 2009, 969-982. The techniques used include timing differences (differences in the arrival times of the sensed parameters), sound velocity differences, and sound pressure differences. Of course, acoustic sensors generally measure sound pressure levels (eg, decibels), and these other parameters may be used in any suitable type of acoustic sensor, including acoustic emission sensors and ultrasonic sensors or transducers.

Appropriate algorithms and all other necessary programming may be stored in the eyepiece's microcontroller or in memory accessible to the eyepiece. Using one or more responders or multiple responders, a probable location may be determined, and the responder may attempt to locate a person to rescue. In other applications, response agents may use these acoustic functions to determine the location of a person of interest for law enforcement purposes. In other applications, multiple persons for operational action may encounter enemy fire, including direct fire (out of sight) or indirect fire (out of sight, including high-angle fire). The same techniques described herein can be used to estimate the enemy's firing position. If there are a large number of people in the area, the estimation can be more sophisticated, especially if the people are segregated over a larger area, at least to some extent. This can be an effective tool for directing counter-cannon or counter-mortar fire against the enemy. Direct fire can also be used if the target is close enough.

An example using an embodiment of an augmented reality eyepiece is shown in FIG. 31 . In this example, multiple soldiers are on patrol, each equipped with an augmented reality eyepiece, and vigilant against enemy fire. Sound detected by their acoustic sensors or microphones may be relayed to squad vehicles as shown, relayed to their platoon, or relayed to a remote tactical operations center (TOC) or combat command (CP). Alternatively, or in addition to this, the signal may also be transmitted to a mobile device, such as an airborne platform as shown. Communication between soldiers and additional locations may be utilized using local area networks, or other networks. Additionally, all transmitted signals may be protected by encryption or other safeguards. One or more of a squad vehicle, platoon commander, mobile platform, TOC or CP will have an integrated capability to combine input from multiple soldiers to determine possible locations of enemy fire. The signal from each soldier will include the soldier's location from the augmented reality glasses or GPS function unique to the eyepiece. An acoustic sensor for each soldier can indicate a possible direction of the noise. Using signals from multiple soldiers, the direction and possible location of enemy fire can be determined. The soldier can then subdue that position.

In addition to the microphone, an augmented reality eyepiece may be mounted as an earbud, which may be engaged with the earbud, and remotely attached 1403 , or equipped with an audio input jack 1401 , as described herein. can The eyepiece and earbuds transmit noise-cancelling interference, allow the user to better hear the sound delivered from the audio-video communication function of the augmented reality eyepiece or glasses, and may feature automatic gain control. The augmented reality eyepiece's speakers or earbuds also connect with the device's full audio and temporal capabilities, and can deliver high-quality, clear sound from the included telecommunications device. As described herein, this includes radio or cell phone (smartphone) audio functionality, and may also include Bluetooth functionality or associated technologies such as IEEE 802.11 for wireless personal network (WPAN).

Another aspect of augmented audio functionality includes speech recognition and identification functionality. Speech recognition is about understanding what is being said, whereas speech identification is about understanding who is speaking. Speech identification can work collaboratively with the facial recognition capabilities of these devices to more positively identify people of interest. As described herein, a camera connected as part of an augmented reality eyepiece can focus in a non-protruding manner on a desired individual, such as a single person in a crowd or multiple faces in a crowd. Using a camera and suitable facial recognition software, an image of the person may be taken. The features of the image are then separated by any number of sizes or statistics, and the results are compared against a database of known persons. An identity can then be created. In the same way, voice or voice sampling from the person of interest may be taken. Samples may be marked or tagged, for example, at specific time intervals, and labeled, for example, with a number or description of the person's physical characteristics. Voice samples can be compared to a database of known people, and if that person's voice is matched, identification can be made.

In embodiments where the camera is used for the biological identification of multiple people in a crowd, the control techniques described herein may be used to select a face or iris for imaging. For example, cursor selection using a hand-mounted control device may be used to select multiple faces in a view of the user's surrounding environment. In another example, eye tracking may be used to select which face to select for biometric identification. In another example, the hand-mounted control device may detect a gesture used to select an individual, such as pointing at each individual.

In one embodiment, an important characteristic of a particular person's speech can be understood from a sample or multiple samples of that person's voice. A sample is generally divided into segments, frames and subframes. In general, important characteristics include the fundamental frequency, energy, formant, speaking speed, etc. of the person's voice. These characteristics are analyzed by software that analyzes speech according to specific formulas or algorithms. These fields are constantly changing and improving. However, currently such classifiers may include algorithms such as neural network classifiers, k-classifiers, hidden Markov models, Gaussian mixture models, and pattern matching algorithms, among others.

A universal template 3200 for speech recognition and speaker identification is shown in FIG. 32 . A first step 3201 is to provide a speech signal. Ideally, we have a known sample from a previous encounter that we compare the signal to. The signal is then digitized in step 3202 and fragmented into pieces such as segments, frames and subframes in step 3203. Features and statistics of the speech sample are then generated and extracted in step 3204 . A classifier, or one or more classifiers, is applied in step 3205 to determine a general classification of the sample. Post-processing of the sample may then be applied at step 3206, for example, to compare the sample to a known sample for possible matching and identification. The result can then be output in step 3207 . The output is directed to the person requesting the match, and may also be logged and sent to another person or one or more databases.

In one embodiment, the audio functionality of the eyepiece includes a hearing protector with associated earbuds. The eyepiece's audio processor may enable automatic noise suppression, such as when loud noise is detected close to the wearer's head. Any of the control techniques described herein may be used with automatic noise suppression.

In one embodiment, the eyepiece may include a nitinol head strap. The head strap can either emerge from the arm of the eyepiece or be a thin band of bent metal that rotates and extends behind the head to secure the eyepiece to the head. In one embodiment, the tip of the nitinol strap may include a silicone cover to hold the silicone cover coming out of the end of the arm. In an embodiment, only one arm has a nitinol band and it is secured to the other arm to form a strap. In another embodiment, both arms are provided with nitinol bands, and both sides protrude and engage to form a strap to secure the eyepiece to the wearer's head or to independently hold a portion of the head.

Referring to FIG. 21 , the eyepiece may include one or more adjustable wrap around extendable arms 2134 . An adjustable wrap-around extendable arm 2134 may secure the position of the eyepiece to the user's head. One or more extendable arms 2134 may be made of a shape memory material. In embodiments, one or both of the arms may be made of nitinol and/or any shape memory material. In another example, an end of the at least one wraparound extendable arm 2134 may be covered with silicone. Additionally, an adjustable wraparound extendable arm 2134 may extend from an end of the eyepiece arm 2116 . They may extend telescopically and/or they may slide from the end of the eyepiece arm. They may slide from the interior of the eyepiece arm 2116 , or they may slide along the exterior surface of the eyepiece arm 2116 . Additionally, the extendable arms 2134 may meet and be secured to each other. The extendable arm may also attach to another portion of the head mounted eyepiece to create a means for securing the eyepiece to a user's head. The wraparound extendable arms 2134 may meet to secure each other, interlock, magnetically engage, or otherwise secured by other means to provide secure attachment to a user's head. In embodiments, the adjustable wraparound extendable arm 2134 may also be independently adjusted to attach to or hold a portion of a user's head. As such, the independently adjustable arm may allow the user to increase customizability for a personalized fit to secure the eyepiece to the user's head. Additionally, in embodiments at least one of the wraparound extendable arms 2134 may be removable from the head mounted eyepiece. In another embodiment, the wraparound extendable arm 2134 may be an add-on feature of the head mounted eyepiece. In this example, the user may choose to place an extendable, non-extendable, or other arm into the head mounted eyepiece. For example, the arm may be sold as a kit or part of a kit that allows the user to customize the eyepiece for his or her specific preferences. Thus, the user can customize the type of material from which the adjustable wraparound extendable arm 2134 is made by selecting different kits with a particular extendable arm tailored to their preferences. Thus, users can customize their eyepieces for their specific needs and preferences.

In another embodiment, an adjustable strap 212 may be attached to the eyepiece arm such that it extends around the back of the user's head to secure the eyepiece in place. The strap can be adjusted for a proper fit. It may be made of any suitable material including, but not limited to, rubber, silicone, plastic, cotton, and the like.

In one embodiment, the eyepiece may include security features such as M-Shield Security, Secure Content, DSM, Secure Runtime, IPSec, and the like. Other software features may include: User Interface, Apps, Framework, BSP, Codecs, Integration, Testing, System Validation, etc.

In one embodiment, the eyepiece material may be selected to allow for ruggedization.

In one embodiment, the eyepiece may have access to a 3G access point, including a 3G radio, 802.11b connection and Bluetooth connection, capable of hopping data from the device to the 3G-enabled embodiment of the eyepiece.

The present disclosure also relates to methods and apparatus for capture of biological data relating to an individual. Methods and devices provide for wireless capture of an individual's fingerprints, iris patterns, facial structures, and other unique biological features, and then transmits the data over a network or directly to the eyepiece. Data collected from individuals may also be compared to previously collected data and used to identify specific individuals.

Other embodiments of the eyepiece may be used to collect biological data and provide result reports. The biometric data may be visual biometric data, such as facial biometric data, or iris biometric data, or may be audio biometric data. 66 depicts one embodiment that provides for biometric data capture. Assembly 6600 includes eyepiece 100 described above in connection with FIG. 1 . The eyepiece 100 provides an interactive head mounted eyepiece with an optical assembly. In addition, other eyepieces that provide similar functions may be used. The eyepiece may also include a global positioning system to allow display and reporting of location information.

The optical assembly allows the user to view the wearer's surroundings, including nearby individuals. The eyepiece of one embodiment allows a user to biometrically identify people in proximity using facial and iris images, or both facial and iris images, or audio samples. The eyepiece includes a correction element that corrects the user's view of the surrounding environment and displays content provided to the user via an integrated processor and image source. The integrated image source provides the optical assembly with content to be displayed to the user.

The eyepiece also includes an optical sensor for capturing biological data. In one embodiment, the integrated optical sensor may include a camera mounted on the eyepiece. These cameras are used to capture biological images of people in the vicinity of the eyepiece user. The user can orient the optical sensor or camera towards a nearby person by positioning the eyepiece in the appropriate orientation, which can be done simply by looking at that person. The user may choose whether to capture one or more of a facial image, an iris image, or an audio sample.

The biological data that may be captured by the eyepiece shown in FIG. 66 includes a facial image for facial recognition, an iris image for iris recognition, and an audio sample for speech recognition. The eyepiece 100 includes a plurality of microphones 6602 in an endfire array disposed along both the right and left temples of the eyepiece 100 . Microphone array 6602 is specially tuned to enable capture of human voice in environments with high levels of ambient noise. Microphone 6602 provides selectable options for enhanced audio capture, including omni-directional operation or directional beam operation. Directional beam operation allows a user to record audio samples from a specific person by steering the microphone array in the direction of the specific person.

Audio biological capture is enhanced by incorporating phased array audio and video tracking for audio and video capture. Audio tracking allows for continuous capture of audio samples as the target person moves into the environment with other noise sources.

To provide power for display optics and biological data collection, the eyepiece 100 also includes a lithium-ion battery 6604 capable of operating for more than 12 hours on a single charge. The eyepiece 100 also includes a processor and solid-state memory 6606 for processing the captured biological data. The processor and memory may be configured to function with any software or algorithm used as part of a biological capture protocol or format such as the .wav format.

Another embodiment of the eyepiece assembly 6600 provides integrated communication capabilities to transmit captured biological data to a remote facility that stores the biological data in a biological data database. The biometric data database interprets the captured biometric data, interprets the data, and prepares content for display on the eyepiece.

In that operation, the eyepiece wearer who wishes to capture biological data from a nearby observed person positions himself/herself so that the person appears within the field of view of the eyepiece. After positioning, the user initiates the capture of biological information. Biological information that may be captured includes iris images, facial images, and audio data.

In that operation, the eyepiece wearer who wishes to capture audio biometric data from a nearby observed person positions himself/herself so that the person is in the vicinity of the eyepiece, particularly in the vicinity of the microphone array located in the eyepiece temple. After positioning, the user initiates the capture of audio biometric information. This audio biometric information consists of recorded samples of the target person's speaking. Audio samples may be captured along with visual biometric data such as iris and facial images.

To capture an iris image, the wearer/user looks at the desired person and positions the eyepiece so that the optical sensor assembly or camera can collect an image of the desired person's biological parameters. After capture, the eyepiece processor and solid-state memory prepare the captured image for transmission to a remote computing facility for further processing.

The remote computing facility receives the transmitted biological image and compares the transmitted image to previously captured biological data of the same type. The iris or facial image is compared to previously collected iris or facial images to determine whether the person has previously been met and identified.

After this comparison is performed, the remote computing facility transmits a report of the comparison to the wearer/user's eyepiece for display. This report may indicate whether the captured biological images match previously captured images. In this case, the user receives a report containing the person's identification, along with other identifying information or statistics. Not all captured biological data allow for an unambiguous determination of identity. In such a case, the remote computing facility may request the user to collect additional biological data, perhaps of a different type, to provide a report of the search and to aid in the identification and comparison process. Visual biometric data may be supplemented with audio biometric data as an additional aid for identification.

Facial images are captured in a manner similar to iris images. Due to the size of the image being collected, the field of view is necessarily larger. This also allows the user to be further away from the subject for which facial biometric data is being captured.

In its operation, the user may have originally captured a facial image of the person. However, facial images may be incomplete or inconclusive, as a person may wear clothing or clothing such as a hat that obscures facial features. In such cases, the remote computing facility may request that another type of biological capture be used and additional images or data be transmitted. In the case described above, the user may be instructed to obtain an iris image to supplement the captured facial image. In another example, the additionally requested data may be an audio sample of a human voice.

67 illustrates capturing an iris image for iris recognition. This figure shows the focus parameters used to analyze the image, including the geographic location of the person at the time of biometric data capture. 67 also shows a sample report displayed on the eyepiece.

68 illustrates capture of multiple types of biological data, in this example, facial and iris images. Such captures may be performed concurrently, or may be performed at the request of a remote computing facility when the first type of biological data results in inconclusive results.

FIG. 69 shows an electrical configuration of a plurality of microphone arrays included in the temple of the eyepiece of FIG. 66 . A vertical microphone array allows for better discrimination of signals and better directivity at greater distances. Signal processing is improved by including delays in the rear microphone's transmission line. The use of dual omni-directional microphones makes it possible to switch from an omni-directional microphone to a directional microphone. This allows for better direction finding for audio capture of the desired person. 70 illustrates the directional enhancement available through multiple microphones.

A plurality of microphones may be arranged in a composite microphone array. Instead of using one standard, high-quality microphone to capture audio samples, the eyepiece temple houses multiple microphones of different characteristics. One example of the use of multiple microphones is the use of microphones from cut off cell phones to reproduce the correct electrical and acoustic properties of a person's voice. These samples are stored in a database for future comparison. If the human voice is captured later, the previous sample is available for comparison, and if the acoustic properties of the two samples match, it will be reported to the eyepiece user.

71 illustrates the use of an adaptive array to enhance audio data capture. By modifying the pre-existing algorithm for audio processing, an adaptive array can be created that allows the user to steer the directivity of the antenna in three dimensions. Adaptive array processing may allow for the location of the source of the speech, tying the captured audio data to a specific person. Array processing allows simple summation of the cardioid elements of the signal to be performed digitally or using analog techniques. In typical use, a user switches the microphone between an omni-directional pattern and a directional array. The processor allows beamforming, array manipulation, and adaptive array processing performed on the eyepiece.

In one embodiment, the integrated camera may continuously record video files and the integrated microphone may continuously record audio files. The eyepiece's integration process may enable event tagging within long sections of continuous audio or video recordings. For example, a passive recording of an entire day may be tagged whenever an event, conversation, encounter, or other item of interest occurs. Tagging may be accomplished through an explicit button press, a noise or physical tap, a hand gesture, or any other control technique described herein. Markers may be placed in an audio or video file, or may be stored in a metadata header. In embodiments, markers may include GPS coordinates of events, conversations, encounters, or other items of interest. In another embodiment, the marker may be time-synchronized with the GPS log of the day. Other logic based triggers may also tag audio or video files, such as proximity relationships to other users, devices, locations, and the like.

In one embodiment, the eyepiece may be used as SigInt glasses. Using one or more of an integrated WiFi, 3G or Bluetooth radio, the eyepiece can be used to significantly and passively collect signals intelligence about devices and people in the user's proximity. Signal information can be collected automatically, or can be triggered when a specific device ID is in proximity, when a specific audio sample is detected, and when a specific geographic location is reached, etc.

In one embodiment, the fingerprint collection device is known as a bio-print device. The bioprint apparatus includes a clear platen with two beveled edges. This platen may be illuminated by one bank of LEDs and one or more cameras. A plurality of cameras are used and positioned facing and adjacent to the beveled edges of the platen. A finger or palm rests on the platen and presses on the top surface of the platen, and the camera captures the ridge pattern. This image is recorded using frustrated total internal reflection (FTIR). In FTIR, the platen exits the platen across an air gap created by the ridges and valleys of the palm or finger pressing on the platen.

Other embodiments are also possible. In one embodiment, the plurality of cameras are arranged in an inverted 'V' sawtooth pattern. In another embodiment, a rectangle is formed, using light redirection through one side, and an array of cameras capture the generated image. Light enters the rectangle through the sides of the rectangle, and the camera is positioned just below the rectangle, allowing the cameras to capture the ridges and valleys illuminated by the light passing through the rectangle.

After the images are captured, software is used to stitch together the images from multiple cameras. A custom FPGA can be used for digital image processing.

After being captured and processed, the images can be streamed to a remote display, such as a smart phone, computer, portable device, or eyepiece or other device.

The preceding description provides an overview of the operation of the method and apparatus of the present invention. Additional descriptions and explanations of these and other embodiments are provided below.

33 shows the structure and layout of a light-based fingerprint and palm print system according to one embodiment. The optical array consists of approximately 60 wafer scale cameras. Light-based systems use sequential perimeter illumination for high-resolution imaging of whorls and pores, including fingerprints or palm prints. This construction provides a low profile, lightweight, and very robust construction. Durability is enhanced through a scratch proof transparent platen.

Mosaic print sensors use a total internal reflection interference (FTIR) optical faceplate that provides images to an array of wafer-scale cameras mounted on a PCB-like substrate. The sensor can be scaled to any flat width and length with a depth of approximately ½". This size is large enough to capture the roll print of just one finger on a plate small enough to hold both hands simultaneously. They can range from plates large enough to capture fingerprints.

Mosaic print sensors allow operators to capture fingerprints and compare the collected data to an onboard database. Data can also be uploaded and downloaded over the air. This unit may operate as a standalone unit, or may be integrated with any biological system.

In operation, the mosaic print sensor provides high reliability in harsh environments with excessive sunlight. To provide this capability, multiple wafer-scale optical sensors are digitally grouped together using pixel subtraction. The resulting image is designed to exceed 500 dots per inch (dpi). Power is supplied either by a battery or by power drawn parasitically from another source using the USB protocol. Formatting conforms to EFTS, EBTS NIST, ISO, and ITL 1-2007.

34 shows a traditional optical approach used by another sensor. This approach is also based on FTIR. In this figure, a fringe contacts the prism and scatters the light. The fringe on the finger being printed is shown as a dark line, and the valley of the fingerprint is shown as a light line.

35 shows the approach used by mosaic sensor 3500 . Mosaic sensors also use FTIR. However, the plate is illuminated from the side, and internal reflections are contained within the plate of the sensor. The fringe contacts the prism and scatters the light, allowing the camera to capture the scattered light. The fringes on the fingers are shown as light lines, and the valleys are shown as dark lines.

36 shows the layout of the mosaic sensor 3600 . The LED array is arranged near the outer periphery of the plate. The camera used to capture the fingerprint image is below the plate. Images are captured on a base plate known as a capture plate. This capture plate is parallel to the sensor plate on which the finger rests. The thickness of the plate, the number of cameras, and the number of LEDs may vary depending on the size of the active capture area of the plate. The thickness of the plate can be reduced by adding a mirror that bends the light path of the camera and reducing the required thickness. Each camera covers an inch of space, with some pixels overlapping between the cameras. This allows the mosaic sensor to achieve 500 ppi. A camera may have a field of view of 60 degrees, but there may be significant distortion in the image.

37 illustrates the camera field of view and interaction of multiple cameras used in a mosaic sensor. Each camera covers a small capturing area. This area depends on the camera field of view and the distance between the camera and the top surface of the plate. α is one-half of the camera's horizontal field of view, and β is one-half of the vertical field of view of the camera.

The mosaic sensor may be integrated into a bio-phone as shown in FIG. 38 and a tactical computer. The bio phone and tactical computer are a complete mobile computer that includes a dual-core processor, DSP, 3-D graphics accelerator, 3G-4G Wi-LAN (compliant with 802.11 a/b/g/n), Bluetooth 3.0, and a GPS receiver. use. The bio phone and tactical computer deliver power equivalent to a standard laptop in a phone-sized package.

38 shows the components of a biophone and a tactical computer. The bio phone and tactical computer assembly 3800 provides a display screen 3801 , speakers 3802 , and a keyboard 3803 contained within a case 3804 . These elements are visible on the front of the bio-phone and tactical computer assembly 3800 . On the back side of the assembly 3800 are located a camera 3805 for iris imaging, a camera 3806 for facial imaging and video recording, and a bio-print fingerprint sensor 3809 .

To provide secure communication and data transfer, the device includes selectable 256-bit AES encryption, along with a COTS sensor and software for biometric pre-screening for POI acquisition. Such software may be matched or filed by any approved biometric matching software to send and receive secure "perishable" voice, video, and data communications. In addition, the Bio Phone supports Windows Mobile, Linux, and Android operating systems.

The Bio Phone is a 3G-4G capable handheld device for reaching back to web portals and the Bioavailable Watch List (BEWL) database. Such databases allow for field comparison of captured biological images and data. The device is designed to fit into a standard LBV or pocket.

The bio-phone may search for, collect, register, and verify a plurality of types of biological data, including facial, iris, and two-finger fingerprints, as well as biographic data. The device also records video, voice, gait, identification marks, and pocket litter. The belongings include various small items generally carried in a pocket or wallet, and may include items such as extra change, ID, passport, cash card, and the like. 40 shows a typical collection of this type of information. An example of a collection of belongings 4000 is shown in FIG. 40 . The types of items that may be included are personal documents and documents such as photos 4101 , books 4102 , notebooks and papers 4103 , and passports 4104 .

39 illustrates the use of a bio-phone to capture latent and palm prints. Fingerprints and palm prints are captured at 1000 dpi with active illumination from an ultraviolet diode with a scale overlay. Both the fingerprint and palm print 3900 may be captured using the bio-phone.

The data collected by the bio-phone is automatically geo-located and date and time stamped using the GPS function. Dates can be uploaded or downloaded and compared to an onboard or network database. This data transfer is facilitated by the device's 3G-4G, Wi-LAN, and Bluetooth capabilities. Data entry may be performed via a QWERTY keyboard, or other method that may be provided, such as a stylus or touch screen. Biological data is filed after collection using the most prominent image. Manual input enables partial data capture. 41 shows an interplay 4100 between a digital dossier image held in a database and a biological watch list. A biological watch list is used to compare data captured in the field with previously captured data.

Formatting may use the EFTS, EBTS NIST, ISO, and ITL 1-2007 formats to provide compatibility with the range and diversity of databases for biological data.

Details for the bio phone and tactical computer are given below.

Operating temperature: -22°C to +70°C

Connectivity I/O: 3G, 4G, WLAN a/b/g/n, Bluetooth 3.0, GPS, FM

Connectivity Output:: USB 2.0, HDMI, Ethernet

Physical Dimensions: 6.875" (H) x 4.875" (W) x 1.2" (T)

Weight: 1.75 lbs.

Processor: Dual-Core -1 GHz Processor, 600 MHz DSP, and 30M Polygon/sec 3-D Graphics Accelerator;

Display: 3.8" WVGA (800 x 480) solar readable, transreflective, capacitive touch screen, and scalable for simultaneous connection to 3x 1080p Hi-Def screens ) display output

OS: Windows Mobile, Linux, SE, Android

Storage: 128GB solid state drive

Additional storage: Dual SD card slots for additional 128GB storage

Memory: 4 GB RAM

Cameras: 3 high-def still and video cameras: face, iris, and conference (user face)

3D support: stereo 3D video output function

Camera sensor support: sensor dynamic range extension, adaptive detection pixel correction, enhanced sharpness enhancement, geographic distortion correction, improved color management, HW-based face detection, video stability

Biological: Onboard optical 2 fingerprint sensor, facial, DOMEX, and iris camera

Sensors: Can accommodate addition of accelerometer, compass, ambient light, proximity, barometric pressure, and temperature sensors, depending on requirements.

Battery: &lt;8hrs, 1400 Mah, rechargeable Li-ion, hot swap battery pack

Power: Multiple power options for continuous motion

Software Features: Face/Gesture Detection, Noise Filtering, Pixel Correction

Powerful display processor with multiple overlay, rotation, and resizing capabilities

Audio: Onboard microphone, speaker, and audio/video input

Keyboard: full tactile QWERTY keyboard with adjustable backlight

Additional devices and kits may include mosaic sensors and may work with biophones and tactical computers to provide a complete field solution to collected biological data.

One such device is the pocket bio-kit shown in FIG. 42 . The components of the pocket bio kit 4200 include a GPS antenna 4201 , a bio print sensor 4202 , and a keyboard 4204 included in a case 4203 . The details of this bio kit are given below.

Size: 6"×3"×1.5"

Weight: 2 lbs total.

Processor and memory: 1 GHz OMAP processor

650 MHz core

3-D accelerator handling up to 18 million polygons/sec

64KB L2 cache

166 MHz, 32-bit FSB

1GB internal PoP memory expandable up to 4GB NAND

64 GB solid state hard drive

Display: 75mm×50mm, 640×480 (VGA) daylight readable LCD, anti-glare, anti-reflective, anti-scratch screen treatment

Interface: USB 2.0

10/100/1000 Ethernet

Power: Battery Operation: Approximately 8 hours of continuous registration with approximately 5 minutes per registration

Built-in features: Mosaic sensor optical fingerprint reader

Digital iris camera with active IR illumination

Digital face and DOMEX camera with flash (visibility)

High-speed lock GPS

Features of the bio phone and tactical computer can also be provided in a bio kit that provides a biological data collection system that folds into a rigid and compact case. Data is collected in standard biological image and data formats that can be cross-referenced through near real-time data communication with the Department of Defense Biological Agency database.

The pocket bio kit shown in Figure 43 is capable of capturing potential fingerprints and palm prints at 1,000 dpi via active illumination from an ultraviolet diode with a scale overlay. The BioKit holds a 32GB memory storage card that can interact with a combat radio or computer for uploading and downloading of data in real-time field conditions. Power is provided by a lithium-ion battery. The components of the bio kit assembly 4200 include a GPS antenna 4201 , a bio print sensor 4202 , and a case 4203 with a base bottom 4205 .

Biometric data collections are geolocated to monitor and track human movement. Fingerprints and palm prints, iris images, facial images, potential fingerprints, and videos can be collected and registered in a database using the bio kit. Algorithms for fingerprints and palm prints, iris images, and facial images facilitate this type of data collection. To help capture the iris image and potential fingerprint image simultaneously, the accessory kit has IR and UV diodes that actively illuminate the iris or potential fingerprint. In addition, the Pocket Bio Kit is also fully EFTS/EBTS compliant, including ITL 1-207 and WSQ. The Bio Kit meets MIL-STD-810 for operation in extreme environments and uses a Linux operating system.

To capture images, the Bio Kit uses a high dynamic range camera with wave front coding for maximum depth of field, which ensures details of potential fingerprint and iris images are captured. . Once captured, real-time image enhancement software and image stabilization serve to improve readability and provide good visual discrimination.

The Bio Kit can record video and store full-motion (30 fps) color video in an "onboard "camcorder on chip".

In addition to the bio kit, the mosaic sensor can be integrated into the wrist-mounted fingerprint, palm print, geo-location, and POI enrollment devices shown in FIG. 44 . The wrist mounted assembly 4400 includes a strap 4402, a set and on/off button 4403, a sensor protective cover 4404, a pressure-actuated sensor 4405, and a keyboard and LCD screen 4406 within the case 4401. ) is included.

The fingerprint, palm print, geolocation, and POI enrollment devices include an integrated computer, QWERTY keyboard, and display. The display is designed for easy operation in strong sunlight and uses an LCD screen or LED indicator to inform the operator of successful fingerprint and palm print captures. The display uses transflective QVGA colors, along with a backlit LCD screen to improve readability. The device is lightweight and compact, weighing 16 oz. and measuring 5" x 25" in a mosaic sensor. This small size and weight allows the device to slip into the LBV pocket and can be hung on a user's forearm as shown in FIG. 44 . Through other devices including mosaic sensors, all POIs can be tagged with geographic information when captured.

The size of the sensor screen allows for capture of ten fingers, the palm of the hand, four slaps, and the tip of a finger. The sensor includes a large pressure-driven print sensor for rapid registration in any weather conditions specified in MIL-STD-810 at a speed of 500 dpi. The software algorithm supports both fingerprint and palm print capture modes, and uses the Linux operating system for device management. Capture is fast due to the 720 MHZ processor with 533 MHZ DSP. This processing capability delivers well-formatted, striking images to any existing approved system software. Additionally, the device is fully EFTS/EBTS compliant, including ITL 1-207 and WSQ.

In conjunction with other mosaic sensor devices, wireless mode communication is possible using a removable UWB wireless 256-bit AES transceiver. It also provides secure uploads and downloads to biological databases away from the device.

Power is supplied using lithium polymer or AA alkaline batteries.

The wrist mounted device described above may also be used with other devices, including the augmented reality eyepiece with data and video display shown in FIG. 45 . The assembly 4500 includes the eyepiece 100 and the bioprint sensor device 4400 . Augmented reality eyepieces offer redundant, binocular, stereo sensors and devices, and the ability to see in a variety of light conditions, from dazzling midday sunlight to very low light levels at night. The operation of the eyepiece is simple: via a rotary switch located on the leg of the eyepiece, the user can access data from a forearm computer or sensor, or a laptop device. The eyepiece also offers omni-directional earbuds for hearing protection and hearing enhancement. Additionally, a noise canceling boom microphone can be integrated into the eyepiece to provide better communication of phonetically differentiated commands.

The eyepiece can communicate wirelessly with biophone sensors and forearm-mounted devices using 256-bit AES encrypted UWB. This allows the device to communicate with a laptop or military radio, as well as networking with CPs, TOCs and biological databases. The eyepiece is ABIS, EBTS, EFTS, and JPEG 2000 compatible.

Similar to the other mosaic sensor devices described above, the eyepiece uses an RF filter array as well as networked GPS to provide a highly accurate geographic location of POIs.

In operation, a low profile forearm-mounted computer and tactical display integrates facial, iris, fingerprint, palm print, and fingertip collections and identification information. The device also records video, voice, gait, and other distinguishing characteristics. Facial and iris tracking is automatic, allowing the device to help recognize non-cooperative POIs. Through the transparent display provided by the eyepiece, the operator can also view the sensor image, movement map, and data as well as the person whose biological data is being captured.

46 illustrates another embodiment of a fingerprint, palm print, geographic location, and POI enrollment device. The device is 16 oz and uses a 5" x 2.5" active fingerprint and palm print capacitive sensor. The sensor can register 10 fingers, palm, 4 slap, and fingertip prints at 500 dpi. A 0.6-1GHz processor with a 430MHz DSP provides fast registration and data capture. The device is ABIS, EBTS, EFTS, and JPEG 2000 compatible and features networked GPS for highly accurate localization of the subject. The device also communicates wirelessly via 256-bit AES encrypted UWB, laptop, or combat radio. Database information may be stored on the device, which enables on-site comparison without uploading information. This onboard data can also be shared wirelessly with other devices such as laptops or military radios.

Another embodiment wrist-mounted bio print sensor assembly 4600 includes a bio print sensor 4601 , a wrist strap 4602 , a keyboard 4603 , and a military radio connector interface 4404 .

As the back of the hand device may use a Mil-con data storage cap for increased storage capacity, data may be stored on the back of the hand device. Data input is performed on the QWERTY keyboard, and may be performed while wearing gloves.

This display is a reflective and transparent QVGA, color, backlit LCD display designed to be read in sunlight. In addition, since the device meets the requirements of MIL-STD-810 operation in harsh environments, in order to operate in strong sunlight, the device can be operated in a wide range of environments.

The mosaic sensor described above can also be integrated into a mobile foldable biometric registration kit, as shown in FIG. 47 . The mobile foldable biometric registration kit 4700 folds itself up, is sized to fit in a tactical vest pocket, and has dimensions of 8x12x4 inches when unfolded.

48 shows how the eyepiece and forearm fixed device interface provides a complete system for biometric data collection.

49 shows a system diagram for a mobile foldable biological registration kit.

In operation, the mobile foldable biometric enrollment kit enables a user to search for, collect, identify, verify, and register facial, iris, palm print, fingerprint, and biographic data for a subject, and also voice Samples, belongings, and other visual identification marks may be recorded. After being collected, the data is automatically geo-located and date and time stamped. Collected data can be retrieved and compared in onboard and networked databases. Wireless data upload/download using a laptop or combat radio with a standard networking interface is provided to communicate with the database rather than the onboard device. Formatting complies with EFTS, EBTS, NIST, ISO, and ITL 1-2007. If the device uses any matching and registration software, the pre-screened images can be sent directly to the matching software.

The device and system integration described above provides a comprehensive solution for mobile biometric data collection, identification, and contextual awareness. The device may collect fingerprint, palm print, fingertip, facial, iris, voice, and video data for uncooperative person of interest (POI) recognition. Video is captured using high-speed video so that it can be captured in unstable situations, such as moving video. Captured information can be quickly shared, and additional data is entered via the keyboard. Additionally, all data is tagged with date, time and geographic location. This facilitates the rapid dissemination of information essential for situational awareness in a potentially volatile environment. Additional data collection is possible through more people equipped with this device, demonstrating the idea of "every soldier is a sensor". Sharing is facilitated by the integration of military radios and military computers with biological devices.

50 shows a thin film fingerprint and palm print collection device. The device can record four fingerprint slaps and rolls, palm prints, and fingerprints to NIST standards. Excellent quality fingerprint images can be captured with either wet or dry hands. The device weighs less and consumes less power when compared to other large sensors. In addition, the sensor is self-contained and hot swappable. The configuration of the present sensor may be changed to suit various needs, and the present sensor may be manufactured in various shapes and sizes.

51 shows a fingerprint, palm print, and enrollment data collection device. The device records fingertips, rolls, slaps, and palm prints. The built-in QWERTY keyboard enables input of written registration data. With the device described above, all data is tagged along with a collection of date, time, and geographic location. The built-in database provides onboard matching of potential POIs from the built-in database. Matching may also be performed through other databases on military networks. The device may be integrated with the optical biological collection eyepiece described above to support facial and iris recognition.

Details for the fingerprint, palm print, and enrollment device are given below.

Weight and Size: 16 oz. Insert with forearm strap or LBV pocket

5"×2.5" fingerprint/palmprint sensor

5.75"×2.75" QWERTY keyboard

3.5"×2.25 LCD display

one-handed operation

Environment: Sensor operates in all weather conditions from -20°C to +70°C

Waterproof: Operates at 1m for 4 hours without deterioration

Biological Collections: Fingerprint and Palmprint Collections, Identifiable Information

Keyboard and LCD display for registration of POI

Maintains &gt;30,000 full template portfolios (2 iris, 10 fingerprints, facial images, 35 biographic information items) for onboard matching of POIs

All collected biological data is tagged with time, date, and location.

Pressure capacitive fingerprint/palmprint sensor

30 fps high contrast bitmap image

1000 dpi

Wireless: Fully interoperable with military radios, portable or laptop computers, and 256-bit AES encryption

Battery: Dual 2000mAh Li-Polymer Battery

&gt; 12 hours, &lt; 15 seconds fast charging battery

Processing and memory: 3SD cards up to 32GB with 256 MB Flash and 128 MB SDRA support, each

600-1GHZ ARM Cortek A8 Processor

1 GB RAM

52-54 illustrate the use of a device comprising a sensor to collect biological data. 52 shows the capture of the palm print in two steps. 53 shows collection using fingertip taps. 54 shows slab and roll prints being collected.

The above description relates to a method of collecting biological data, such as a fingerprint or palm print, using a platen or touchscreen, as shown in FIGS. 44 and 50-54. The disclosure also includes methods and systems for fingerprinting without touch or contact using polarized light. In one embodiment, the fingerprint may be obtained by a person using a polarized light source and retrieving an image of the fingerprint using polarized light reflected in two planes. In another embodiment, the fingerprint may be obtained by a person using a light source and using multispectral processing to retrieve an image of the fingerprint using, for example, two imagers at two different locations with different inputs. Different inputs may be generated by using different filters or different sensors/imagers. Applications of this technology may include biological checks of unknown persons or subjects where human safety may be an issue during the check.

In this way, an unknown person or subject can access the checkpoint in order to be authorized to proceed further to his or her destination. As shown in system 550 shown in FIG. 55 , a person P and suitable body parts such as hands, palms P, and other parts are illuminated by a source of polarized light 551 . As is well known to those skilled in the art of optics, the source of polarized light may simply be a lamp, or it may be another illumination source with a polarizing filter to emit light polarized in one plane. The light travels to the person within the area specified for non-contact fingerprinting, so that the polarized light reaches the person P's finger or other body part. The incident polarized light is then reflected from a finger or other body part and passes in all directions from the person. Two imagers or cameras 554 receive the reflected light after the light has passed through optical elements such as lens 552 and polarizing filter 553 . The camera or imager may be installed in the augmented reality glasses as described above with respect to FIG. 9 .

The light then passes from the palm or finger or fingers of the person of interest, to two different polarizing filters 554a, 554b, and then to an imager or camera 555. Light passing through the polarizing filter may have a 90° difference in direction (horizontal and vertical), or other direction differences such as 30°, 45°, 60° or 120°. The camera may be a digital camera with a suitable digital imaging sensor for converting incident light into an appropriate signal. These signals are then processed by suitable processing circuitry 556, such as a digital signal processor. These signals may then be combined in a conventional manner, for example, by a digital microprocessor with memory 557 . A digital processor with suitable memory is programmed to produce data suitable for an image of the palm, fingerprint, or other image as desired. The digital data from the imager may then be combined in this process, using, for example, techniques such as US Pat. No. 6,249,616. The combined "image" can then be checked from a database to determine the identity of the person, as previously noted herein. The augmented reality glasses may contain such a database in memory, or may refer to signal data elsewhere 558 for comparison and checking.

A process for obtaining a fingerprint, palm print, or other biological print without contact is disclosed in the flowchart of FIG. In one embodiment, a polarized light source is provided 561 . In a second step 562, the person of interest and the selected body part are positioned for illumination by light. In other embodiments, it may be possible to use incident white light without using a polarized light source. When the image is ready to be acquired, light is reflected 563 from the person to the two cameras or imagers. A polarizing filter is placed in front of each of the two cameras so that light received from the cameras is polarized 564 in two different planes, such as horizontal and vertical planes. Each camera then detects ( 565 ) the polarized light. The camera or other sensors then convert (566) the incident light into a signal or data suitable for image preparation. Finally, the images are combined 567 to form a very sharp and reliable print. The result is a very high quality image that can be compared to digital databases to identify people and detect people of interest.

Although digital cameras have been used in these contactless systems, other imagers may be used, such as active pixel imagers, CMOS imagers, imagers that image with multiple wavelengths, CCD cameras, photo detector arrays, TFT imagers, etc. should understand It should also be understood that although polarized light was used to make two different images, other variations of the reflected light could also be used. For example, instead of using polarized light, white light may be used, and other filters may be applied to the imager, such as a Bayer filter, a CYGM filter, or an RGBE filter. In other embodiments, it may be possible to use natural or white light instead of a polarized light source, but without a polarized light source.

The use of non-touch or non-contact fingerprinting has been in development for some time, as witnessed in previous systems. For example, US Patent Application 2002/0106115 used polarized light in a non-contact system, but requires a metallic coating on the fingerprint of the person taking the fingerprint. Subsequent systems such as those described in US Pat. No. 7,651,594 and US Patent Application Publication No. 2008/0219522 require contact with a platen or other surface. The contactless system described herein does not require contact when imaging, nor does it require prior contact, eg, applying a coating or reflective coating to the body part of interest. Of course, the positions of the imager or camera relative to each other must be known for easier processing.

In its use, the contactless fingerprinting system may be used at checkpoints, such as compound entrances, building entrances, roadside checkpoints, or other convenient locations. Such a location may be a place where it is desirable to admit, deny, or even detain other interested persons. Indeed, the present system can utilize an external light source, such as a lamp, if polarized light is used. A camera or other imagers used for non-contact imaging may be installed (for a person) opposite a set of augmented reality glasses. For example, the two camera version is as shown in FIG. 9 , and the two cameras 920 are installed on the frame 914 . In this embodiment, software for at least processing the image may be included in the memory of the augmented reality glasses. Alternatively, digital data from the camera/imager can be routed to a nearby data center for proper processing. Such processing may include combining digital data to form an image of the print. The processing may also include checking a database of known people to determine if the subject is of interest.

Alternatively, one camera may be used for each of the two persons, as can be seen in camera 908 of FIG. 9 . In this configuration, the two are relatively close, such that their respective images are suitably similar for combining by the appropriate software. For example, the two cameras 555 of FIG. 55 may be mounted on two different pairs of augmented reality glasses, such as two soldiers deployed at a checkpoint. Alternatively, the camera can be mounted on the wall or fixture of the checkpoint itself. The two images may then be combined by a remote processor with memory 557, such as a computer system at a building checkpoint.

As mentioned above, people using augmented reality glasses may be in constant contact with each other via at least one of a number of wireless technologies, especially if they both work at checkpoints. Thus, data from one camera, or two camera versions, can be sent to a data center or other command post for proper processing, and then check the database for matches of palm prints, fingerprints, iris prints, etc. The data center may be conveniently located near the checkpoint. With the availability of modern computers and storage devices, the cost of providing multiple data centers and updating software over the air will not be a major cost consideration in such systems.

The non-touch or non-contact biological data collection described above can be controlled in a number of ways, such as the control techniques described elsewhere herein. For example, in one embodiment, the user may initiate a data collection session by pressing a touchpad on the glasses, or by providing a voice command. In other embodiments, the user may initiate a session using a hand gesture or gesture, or any control technique described herein. Any of these techniques may bring up a menu from which the user can select an option such as "Start data collection session", "End data collection session", or "Continue session". When a data collection session is selected, the computer-controlled menu may provide menu selections for the number of cameras, which cameras, etc., and up to the user selecting a printer. Also, there may be modes such as a polarized light mode, a color filter mode, and the like. After each selection, the system may complete the task, if appropriate, or provide another selection. In addition, user intervention may be required, such as turning on a polarized light source or other light source, applying a filter or polarizer, and the like.

After a fingerprint, palm print, iris image, or other desired data is obtained, the menu may provide a choice of which database to use for comparison, which device to use for storage, and the like. The non-touch or non-contact biological data collection system may be controlled by any of the methods described herein.

The systems and sensors have obvious uses for identifying people of potential interest, but positive military uses also exist. Fingerprint sensors can be used to recall a soldier's medical history, providing immediate, quick and easy information regarding allergies, blood type, and other time-sensitive and treatment decision data, so that appropriate treatment can be provided in combat situations. do. This is especially useful for patients who may be unconscious at the time of initial treatment and have lost the identification tag.

Another embodiment device for capturing biological data from a person may include a server for storing and processing the collected biological data. The captured biometric data may include images of a hand comprising a plurality of fingers, palm prints, facial camera images, iris images, audio samples of a human voice, and video of a human gait or movement. The data collected should be usefully accessible.

The processing of biological data may be performed locally or remotely on a separate server. Local processing may provide the option to capture the raw image or audio and make the information available upon request from the computer host via WiFi or USB link. Alternatively, another local processing method processes the image and then transmits the processed data via the Internet. Such local processing includes finding a fingerprint, evaluating the fingerprint, finding a face, and cropping it, finding and evaluating the iris, and other similar steps for audio and video data. . Local processing of data requires more complex code, but offers the advantage of reducing data transmission over the Internet.

Scanners associated with biological data acquisition devices can use code that conforms to the ISB Image Device Protocol, a widely used scanner standard. Other embodiments may use other scanner standards as needed.

When a WiFi network is used to transmit data, the bioprint device described further herein may function or appear as a web server to the network. Each of the various types of images may be made available by selecting or clicking a button or web page link from a browser client. This web server function may be a part of the bio print device included in the microcomputer function in particular.

The web server may be part of the Bio Print microcomputer host, which allows the Bio Print micro device to create web pages that expose captured data and provide some controls. A further embodiment browser application uses a web cam, Skype, or other mechanism to capture high resolution palm prints, facial images, iris images, set camera resolutions, set capture times for audio samples, And it also provides controls to enable streaming connections. These connections can be attached to audio and facial cameras.

Another embodiment provides a browser application that provides access to images and audio captured via File Transfer Protocol (FTP) or other protocols. Another embodiment browser application may provide automatic refreshes at a selectable rate to repeatedly grab the preview image.

Additional embodiments provide for local processing of captured biological data using a microcomputer, and provide additional controls for displaying a rating of captured images, allowing the user to evaluate each print found, and to process captured faces. Allows for retrieval, retrieval of cropped iris images, and allows users to rate each iris print.

Another embodiment provides a USB port compatible with the Open Multimedia Application Platform (OMAP3) system. OMAP3 is a proprietary single-chip system for portable multimedia applications. The OMAP3 device port is equipped with RNDIS (Remote Network Driver Interface Specification), a proprietary protocol that can be used on top of USB. This system provides the ability for the device to appear as an IP interface when the bioprint device is plugged into a Windows PC USB host port. This IP interface may be the same as over WiFI (TCP/IP web server). This allows data to be moved from the microcomputer host and presented to the display of the captured print.

An application to a microcomputer can implement the above by receiving data from the FPGA via a USB bus. After being received, the JPEG content is created. This content can be written to a file or written over a socket to a server running on the laptop. Alternatively, the server could receive the socket stream, pop the image, and keep it open in one window, creating a new window for each biometric capture. When the microcomputer runs a network file system (NFS), a protocol for use with solar-based systems or SAMBA, and free software that provides file and print services to Windows clients, the captured files are then transferred to NFS or system management bus (SMB). ), can be shared and accessed by any client running a PC communication bus implementation. In this embodiment, the JPEG viewer can display the file. Display clients may include laptops, augmented reality glasses, or phones running the Android platform.

A further embodiment provides a server-side application that provides the same services described above.

An alternative embodiment to a server-side application displays the results on augmented reality glasses.

A further embodiment provides a microcomputer on a removable platform, similar to a mass storage device or streaming camera. The removable platform also includes an active USB serial port.

The methods and systems described herein may be disposed in whole or in part via a machine executing computer software, program code, and/or instructions on a processor. A processor may be part of a server, client, network infrastructure, mobile computing platform, stationary computing platform, or other computing platform. A processor may be any kind of computing or processing device capable of executing program instructions, code, binary instructions, and the like. A processor is a signal processor, digital processor, embedded processor, microprocessor, or co-processor (numerical co-processor, graphics co-processor, communications co-processor, etc.) or may include them. In addition, the processor may be capable of executing a plurality of programs, threads, and codes. Threads can run concurrently to enhance processor performance and to facilitate concurrent operations of applications. As an implementation method, the method, program code, program instruction, etc. described in this specification may be implemented in one or more threads. A thread may spawn other threads that may be assigned a priority associated with the thread, and the processor may execute these threads in any other order based on priority or based on instructions provided within the program code. The processor may include a memory that stores the methods, codes, instructions and programs described herein, and the like. The processor may access the storage medium through an interface capable of storing the methods, codes, and instructions described herein, and the like. A storage medium associated with the processor for storing method, program, code, program instructions or other types of instructions that may be executed by a computing or processing device may be a CD-ROM, DVD, memory, hard disk, flash drive, RAM, ROM , one or more of the caches.

The processor may include one or more cores that may enhance the speed and performance of the microprocessor. In embodiments, the process may be a dual-core processor, a quad-core processor, another chip-level multiprocessor that combines two or more independent cores (referred to as a die), or the like.

The methods and systems described herein may be deployed in whole or in part via a machine running computer software on a server, client, firewall, gateway, hub, router, or other computer, and/or networking hardware. A software program may be coupled to a server, which may include a file server, print server, domain server, Internet server, intranet server, and other variations such as secondary servers, host servers, distributed servers, and the like. A server is one of memory, processor, computer readable media, storage media, ports (physical and virtual), communication devices, and interfaces that can access other servers, clients, machines, and devices through wired or wireless media, and the like. may include more than one. A method, program, or code or the like described herein may be executed by a server. Also, other devices required for the execution of the methods described herein may be considered part of the infrastructure connected to the server.

A server may provide an interface to other devices including, but not limited to, clients, other servers, printers, database servers, print servers, file servers, communication servers, distributed servers, social networks, and the like. Further, such couplings and/or connections may facilitate remote execution of programs across networks. Networking of some or all of these devices may facilitate parallel processing of a program or method at one or more locations. Also, any device attached to the server via the interface may include at least one storage medium capable of storing methods, programs, codes, and/or instructions. A central repository may provide program instructions to be executed on different devices. In such implementations, the remote repository can serve as a storage medium for program code, instructions, and programs.

A software program may be coupled to a client, which may include a file client, a print client, a domain client, an Internet client, an intranet client, and other variations such as auxiliary clients, host clients, distributed clients, and the like. A client may include one or more of memory, processors, computer readable media, storage media, ports (physical and virtual), communication devices, and interfaces that can access other clients, servers, machines, and devices via wired or wireless media, and the like. may include A method, program, or code or the like described herein may be executed by a client. Additionally, other devices required for the execution of the methods described herein may be considered part of the infrastructure connected to the client.

A client may provide an interface to other devices including, but not limited to, servers, other clients, printers, database servers, print servers, file servers, communication servers, distributed servers, and the like. Additionally, such couplings and/or connections may facilitate remote execution of programs across networks. Networking of some or all of these devices may facilitate parallel processing of a program or method at one or more locations. Further, any device attached to the client via an interface may include at least one storage medium capable of storing methods, programs, applications, codes, and/or instructions. A central repository may provide program instructions to be executed on different devices. In this implementation, the remote repository can serve as a storage medium for program code, instructions, and programs.

The methods and systems described herein may be deployed in whole or in part through a network infrastructure. Network infrastructure may include elements such as computing devices, servers, routers, hubs, firewalls, clients, personal computers, communication devices, routing devices, and other active and passive devices, modules and/or components well known in the art. have. Computing and/or non-computing devices coupled to the network infrastructure, apart from other components, may include storage media such as flash memory, buffers, stacks, RAM, ROM, and the like. The processes, methods, program codes, instructions, etc. described herein may be executed by one or more of the network infrastructure elements.

The methods, program codes, instructions, and the like described herein may be implemented on a cellular network having a plurality of cells. The cellular network may be either a frequency division multiple access (FDMA) network, or a code division multiple access (CDMA) network. A cellular network may include mobile devices, cell sites, base stations, repeaters, antennas, towers, and the like. The cell network may be GSM, GPRS, 3G, EVDO, mesh, or other network type.

The methods, program code, instructions, and the like described herein may be implemented on or via a mobile device. Mobile devices may include navigation devices, cell phones, mobile phones, mobile personal digital assistants, laptops, palmtops, netbooks, pagers, e-book readers, music players, and the like. Such devices, apart from other components, may include storage media such as flash memory, buffers, RAM, ROM, and one or more computing devices. A computing device associated with a mobile device may be activated to execute stored program code, methods, and instructions. Alternatively, the mobile device may be configured to execute instructions in cooperation with another device. The mobile device may interface with a server and communicate with a base station configured to execute program code. A mobile device may communicate over a peer-to-peer, network, mesh network, or other communication network. The program code may be stored in a storage medium connected to the server and executed by a computing device embedded in the server. A base station may include a computing device and a storage medium. The storage device may store program code and instructions executed by a computing device coupled to the base station.

Computer software, program code, and/or instructions may include computer components, devices, and recording media holding digital data used for calculations over several time intervals; semiconductor storage devices known as random access memory (RAM); mass storage, typically for more permanent storage, such as in the form of optical disks, hard disks, tapes, drums, cards, and other types of magnetic storage; processor registers, cache memory, volatile memory, non-volatile memory; optical storage devices such as CDs and DVDs; removable media such as flash memory (eg, USB stick or key), floppy disks, magnetic tape, paper tape, punch cards, standalone RAM disks, Zip drives, removable mass storage devices, off-line, and the like; Dynamic Memory, Static Memory, Read/Write Storage, Mutable Storage, Read-Only, Random Access, Sequential Access, Location Addressable, File Addressable, Content Addressable, Network Attached Storage, Storage Area Network (SAN) ), barcodes, magnetic inks, and the like, and/or can be stored on and/or accessed on machine-readable media including other computer memories.

The methods and systems described herein may transform a physical and/or intangible item from one state to another. The methods and systems described herein may also transform data representing physical and/or intangible items from one state to another.

Elements described and illustrated herein, included in flowcharts and block diagrams throughout the drawings, suggest logical boundaries therebetween. However, in accordance with software or hardware engineering practices, the illustrated elements and their functions may be incorporated therein as monolithic software structures, as standalone software modules, or as modules using external routines, code, services, etc., or any combination thereof. It may be implemented on a machine through a computer executable medium having a processor capable of executing stored program instructions, and all such implementation methods fall within the scope of the present invention. Examples of such machines are personal digital assistants, laptops, personal computers, mobile phones, other portable computing devices, medical equipment, wired or wireless communication devices, transceivers, chips, calculators, satellites, tablet PCs, e-books, gadgets. , electronic devices, devices with artificial intelligence, computing devices, networking equipment, servers, routers, glasses with built-in processors, and the like. In addition, the elements or any other logical component shown in the flowcharts and block diagrams may be implemented on a machine capable of executing program instructions. Therefore, although the preceding drawings and descriptions enumerate functional forms of the disclosed systems, no specific arrangement of software for implementing such functional forms should be inferred from these descriptions unless explicitly stated or clear from the context. Similarly, it should be understood that the various steps described and identified above may be varied and the order of the steps may be adjusted for the particular application of the techniques described herein. Such variations and modifications are intended to fall within the scope of the present invention. As such, the illustration and/or description of various steps in an order should not be construed as requiring a special order of performing these steps unless required by a particular application, or unless explicitly stated or clear from the context. .

The above-described methods and/or processes, and steps of the processes, may be implemented in hardware, software, or any combination of hardware and software for a particular application. Hardware may include a general purpose computer, and/or a dedicated computing device, or a special computing device, or particular form or component of a special computing device. The process may be realized in one of a microprocessor, microcontroller, embedded microcontroller, programmable digital signal processor, or other programmable device, with internal and/or external memory. The process may also, or instead, be embedded within an application specific integrated circuit, programmable gate array, programmable array logic, or any other device or combination of devices that may be configured to process an electrical signal. It should also be understood that one or more processes may be implemented as computer-executable code that may be executed on a machine-readable medium.

Computer-executable code may be a structured programming language such as C, an object-oriented programming language such as C++, or any heterogeneous combination of processors, as well as the above devices, processor architectures, or combinations of different hardware and software, or any other capable of executing program instructions. using any other high-level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and technologies) that can be stored, compiled, or interpreted for execution on one of the other machines of can be created by

Thus, in one aspect, each of the methods and combinations thereof described above may be embodied as computer-executable code that, when executed on one or more computing devices, executes their respective steps. In another aspect, the method may be embodied within a system executing the steps of the method, distributed across devices in various ways, or all functionality may be integrated into a dedicated standalone device or other hardware. . In other aspects, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such permutations and combinations are intended to fall within the scope of this invention.

While this specification includes many of the embodiments shown and described in detail, various modifications and variations thereof will readily occur to those skilled in the art. Accordingly, the spirit and scope of the present invention is not limited by the preceding examples, and should be understood as the broadest concept permitted by law.

All documents referenced in this text are incorporated herein by reference.