Combining video-based and optic-based augmented reality in a near eye display

1330-1002.3

COMBINING VIDEO-BASED AND OPTIC-BASED AUGMENTED REALITY

IN A NEAR EYE DISPLAY

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.K. Patent Application GB 1504935.6, filed 24 March, 2015, which is incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to a near eye display system, and specifically to a display able to combine video-based and optic-based augmented reality.

BACKGROUND OF THE INVENTION

A near eye display system may be used in an augmented reality situation, where a scene that is being viewed by a user of the assembly is altered, typically by being augmented or supplemented. The alteration is computer processor generated, and typically involves presenting real time video, and/or non-real time images, to the user while the user is gazing at the scene.

U. S. Patent Application 2010/0149073, to Chaum et al., whose disclosure is incorporated herein by reference, describes a near eye display system. The system includes a source of modulated light, and a proximal optic positionable adjacent to an eye of a system user to receive the modulated light. The proximal optic has a plurality of groups of optically redirecting regions.

U. S. Patent Application 2012/0068913, to Bar-Zeev et al., whose disclosure is incorporated herein by reference, describes an optical see-through head-mounted display device. The device includes a see-through lens which combines an augmented reality image with light from a real-world scene, while an opacity filter is used to selectively block portions of the real-world scene so that the augmented reality image appears more distinctly.

U. S. Patent Application 2013/0050258, to Liu et al., whose disclosure is incorporated herein by reference, describes a see-through head-mounted display device that provides an augmented reality image which is associated with a real-world object. Initially, the object is identified by a user, e.g., based on the user gazing at the object for a period of time, making a gesture such as pointing at the object and/or providing a verbal command.

U. S. Patent Application 2013/0009853, to Hesselink et al., describes an eyeglasses mounted display. The display system includes an eye-glass and a projector, comprising a display device and an optical system for directing an image from the display device to reflect off of the eye-glass and into an eye.

Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that, to the extent that any terms

1330-1002.3 are defined in these incorporated documents in a manner that conflicts with definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.

SUMMARY OF THE INVENTION

An embodiment of the present invention provides apparatus, including:

a retaining structure, configured to be positioned in proximity to an eye of a subject, the eye of the subject having a pupil with a pupil diameter;

an optical combiner mounted on the structure in front of the eye;

a pixelated screen, having an array of variably transparent pixels, coating the optical combiner;

at least one image capturing device mounted on the structure configured to capture an image of a scene viewed by the eye;

a projector mounted on the structure and configured to project at least one of a portion of the captured image and a stored image onto a section of the screen at a selected location thereof; and a processor, configured to render the section of the screen at least partially opaque, to select the location of the section in response to a region of interest in the scene identified by analysis of the captured image, and to determine a dimension of the section in response to the pupil diameter.

The processor may be configured to identify the region of interest in response to radiation received by the image capturing device from at least one marker located at the region of interest.

The apparatus may include a further image capturing device configured to identify the region of interest in response to received radiation from at least one marker located at the region of interest. The at least one image capturing device may be configured to operate in the visible spectrum, and the further image capturing device may be configured to operate in the non-visible spectrum. The apparatus may include a radiator configured to radiate radiation in the non-visible spectrum towards the region of interest.

In a disclosed embodiment the apparatus includes at least one marker positioned in proximity to the region of interest, and wherein the processor is configured to detect the marker in the captured image so as to identify the region of interest.

In a further disclosed embodiment the processor is configured to determine an initial pupil diameter in response to the dimension of the section being set by the

1330-1002.3 subject to occlude an object of known size while the subject gazes at the object in a known ambient light brightness. Typically, the processor is configured to determine a brightness of the scene in response to the captured image of the scene, and the processor is configured to determine the pupil diameter in response to the initial pupil diameter and the brightness of the scene.

In a yet further disclosed embodiment the processor is configured to determine an initial pupil diameter in response to analysis of a reflected image of the subject while the subject gazes into a mirror in a known ambient light brightness. Typically, the processor is configured to determine a brightness of the scene in response to the captured image of the scene, and the processor is configured to determine the pupil diameter in response to the initial pupil diameter and the brightness of the scene.

In an alternative embodiment the dimension of the section is determined so that the region of interest is occluded. Typically, a region surrounding the region of interest is partially occluded. A fraction of occlusion in the region surrounding the region of interest may be determined in response to the pupil diameter. The processor may be configured to derive from the captured image an image corresponding to the region surrounding the region of interest, and the projector may be configured to project the derived image onto an area of the screen surrounding the at least partially opaque section of the screen. An intensity of the projected derived image may be determined in response to the fraction of occlusion.

In a further alternative embodiment the dimension of the section is determined in response to a size of the region of interest.

The dimension of the section may be determined so that an area greater than the region of interest is occluded. Alternatively, the dimension of the section may be determined so that an area less than the region of interest is occluded.

In a yet further alternative embodiment the retaining structure is a spectacle frame. Alternatively, the retaining structure is a helmet having a head-up display.

Typically, the at least one image capturing device includes two image capturing devices capturing respective images of the scene, and the processor is configured to identify the region of interest by analysis of the respective images.

There is further provided, according to an embodiment of the present invention, a method, including:

positioning a retaining structure in proximity to an eye of a subject, the eye of the subject having a pupil with a pupil diameter;

mounting an optical combiner on the structure in front of the eye;

1330-1002.3 coating the optical combiner with a pixelated screen, having an array of variabfy transparent pixefs;

mounting at feast one image capturing device on the structure so as to capture an image of a scene viewed by the eye;

mounting a projector on the structure the projector being configured to project at feast one of a portion of the captured image and a stored image onto a section of the screen at a sefected focation thereof;

rendering the section of the screen at feast partiaffy opaque;

sefecting the focation of the section in response to a region of interest in the scene identified by anafysis of the captured image; and determining a dimension of the section in response to the pupif diameter.

There is further provided, according to an embodiment of the present invention, apparatus, inchiding:

a retaining structure, configured to be positioned in proximity to an eye of a subject;

an opticaf combiner mounted on the structure in front of the eye;

a pixefated screen, having an array of variabfy transparent pixefs, coating the opticaf combiner;

at feast one image capturing device mounted on the structure configured to capture an image of a scene viewed by the eye;

a processor, configured to render a section of the screen at feast partiaffy opaque, and a projector mounted on the structure and configured to project at feast one of a portion of the captured image and a stored image onto the section of the screen so that there is misafignment between the scene viewed by the eye through the combiner and the at feast one portion of the captured image and the stored image.

TypicaUy, for a scene at 50 cm from the eye, the misafignment is no more than 2 cm.

The projector may be configured to project the portion of the captured image and the stored image, in registration with each other, onto the section of the screen.

There is further provided, according to an embodiment of the present invention, apparatus, inchiding:

a retaining structure, configured to be positioned in proximity to an eye of a subject;

an opticaf combiner mounted on the structure in front of the eye;

1330-1002.3 a rotator connected to the optical combiner and configured to rotate the optical combiner about an axis;

a pixelated screen, having an array of variably transparent pixels, coating the optical combiner;

at least one image capturing device mounted on the structure configured to capture an image of a scene viewed by the eye; and a processor, configured to render a section of the screen at least partially opaque, and to activate the rotator so that the optical combiner is oriented to be orthogonal to a region of interest in the scene.

The processor is typically configured to select the section of the screen so as to occlude the region of interest.

The axis may be a vertical axis.

There is further provided, according to an embodiment of the present invention, a method, including:

positioning a retaining structure in proximity to an eye of a subject;

mounting an optical combiner on the structure in front of the eye;

coating the optical combiner with a pixelated screen comprising an array of variably transparent pixels;

mounting at least one image capturing device on the structure, the device being configured to capture an image of a scene viewed by the eye;

rendering a section of the screen at least partially opaque;

mounting a projector on the structure; and configuring the projector to project at least one of a portion of the captured image and a stored image onto the section of the screen so that there is misalignment between the scene viewed by the eye through the combiner and the at least one portion of the captured image and the stored image.

There is further provided, according to an embodiment of the present invention, a method, including:

positioning a retaining structure in proximity to an eye of a subject;

mounting an optical combiner on the structure in front of the eye;

connecting a rotator to the optical combiner, the rotator being configured to rotate the optical combiner about an axis;

coating the optical combiner with a pixelated screen having an array of variably transparent pixels;

1330-1002.3 mounting at least one image capturing device on the structure, the device being configured to capture an image of a scene viewed by the eye;

rendering a section of the screen at least partially opaque; and activating the rotator so that the optical combiner is oriented to be orthogonal to a region of interest in the scene.

The present disclosure will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 schematically illustrates use of an augmented reality system, according to an embodiment of the present invention;

Figs. 2 A - 2E are schematic diagrams illustrating an augmented reality assembly, as well as functions that may be implemented in the assembly, according to an embodiment of the present invention;

Figs. 3A, 3B, and 3C are schematic diagrams illustrating the assembly in different orientations with respect to a region of interest, and Figs. 3D and 3E are graphs derived from the different orientations, according to an embodiment of the present invention;

Fig. 4 is a schematic diagram illustrating derivation of the dimensions of an occlusion mask, according to an embodiment of the present invention;

Fig. 5 is a schematic diagram illustrating partial occlusion of an area around the region of interest, according to an embodiment of the present invention;

Fig. 6 illustrates an occlusion mask, according to an embodiment of the present invention;

Fig. 7 illustrates graphs of occlusion vs. distance, according to an embodiment of the present invention; and

Fig. 8 is a flowchart of steps performed in operation of the augmented reality system, according to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

OVERVIEW

An embodiment of the present invention provides a near eye assembly having a retaining structure that is configured to be positioned in proximity to the eye of a user

1330-1002.3 of the assembly. Typically, the retaining structure comprises a spectacle frame. Alternatively, the retaining structure comprises a head up-display which may be mounted on a helmet worn by the assembly user.

An optical combiner is mounted on the structure in front of the user eye. Typically, two combiners are mounted, one in front of each eye. The optical combiner at least partially transmits elements of a scene in front of the assembly through the combiner. In addition, the optical combiner may receive a visible radiation transmission derived from a scene, and/or a visual transmission such as a presentation of data or a marker, and redirects the transmission back to the user's eye.

A pixelated screen, comprising an array of variably transparent pixels, coats the optical combiner. Typically, the pixels are liquid crystal display (LCD) pixels.

There is at least one image capturing device, typically two such devices, one for each eye, mounted on the structure. The capturing device is typically a visible spectrum camera that is configured to capture an image of a scene viewed by the user's eye.

A projector, typically a micro-projector, is mounted on the structure. Typically two projectors, one for each eye, are mounted on the structure. The projector is configured to project at least one of a portion of the captured image as a video, as well as a stored image, onto a section of the screen that a processor renders at least partially opaque. The at least partially opaque section is also referred to herein as an occlusion mask, or just as a mask.

The processor is configured to select the location of the section in response to a region of interest in the scene identified by analysis of the captured image. Typically, at least one marker is positioned near the region of interest, and the processor analyzes the captured image to locate the marker and so identify the region of interest. Rendering the section opaque occludes the region of interest from the user's eye.

In addition, the processor determines a dimension of the section, typically, in the case of the section being circular, the diameter of the section. The dimension is determined in response to the pupil diameter.

By setting the dimension of the section according to the pupil diameter, embodiments of the present invention more exactly control the area of the region of interest that is occluded. In addition, because of the finite size of the pupil, there is a region surrounding region of interest that is partially occluded. In some embodiments the processor operates the micro-projector to overlay relevant portions of the captured image on the partially occluded region, so as to compensate for the partial occlusion.

1330-1002.3

As stated above, a portion of the captured image may be projected as a video onto the occlusion mask. In some embodiments the captured image portion video corresponds to the occluded region of interest. There is a non-occluded region surrounding the occluded region of interest, and this non-occluded region is visible to the user through the combiner. In embodiments of the invention the video and the visible non-occluded region are typically not in accurate registration, due to slight inevitable movements of the display relative to the user's eye.

In some embodiments a stored image, such as an image of a tool, is overlaid on, and in accurate registration with, the occluded region video.

The inventors have found that registering the stored image with the video, even though the video is not fully registered with the surrounding visible region, provides an acceptable image for the user. The inventors have found that for a non-occluded region that appears to be 50 cm from the user's eye, the video and the non-occluded region may be out of registration by up to 2 cm, while still being acceptable to the user.

Thus, in contrast to prior art augmented reality systems, embodiments of the present invention are configured to operate with mis-alignment between the visible portion of a scene and an augmented reality portion of the scene. However, there is no mis-alignment between elements within the augmented reality video, i.e., the elements projected onto the occlusion mask.

In some embodiments, the optical combiner may be rotated about an axis by the processor. In the case of two combiners, they may be independently rotated about respective axes. The independent rotations may be used to orient both combiners so that each is orthogonal to the direction of gaze of the user's eyes.

SYSTEM DESCRIPTION

Reference is now made to Fig. 1, which schematically illustrates use of an augmented reality system 20, according to an embodiment of the present invention. By way of example and for simplicity, in the following description system 20 is assumed to be used in a medical procedure during part of which the user of the system is being mentored. However, it will be understood that embodiments of the present invention may be used in non-medical and/or non-mentoring situations, such as in operating a video game, in simulating a real-world event, or in providing an aid to navigation.

System 20 is operated by a medical professional 22, who wears an augmented reality assembly 24, described in more detail below with respect to Figs. 2A - 2D. While assembly 24 may be incorporated for wearing into a number of different

1330-1002.3 retaining structures on professional 22, in the present description the retaining structure is assumed to be similar to a pair of spectacles. Those having ordinary skill in the augmented reality art will be aware of other possible structures, such as incorporation of the augmented reality assembly into a head-up display that is integrated into a helmet worn by the user of system 20, and all such structures are assumed to be comprised within the scope of the present invention.

System 20 comprises and is under overall control of a processor 26. In one embodiment processor 26 is assumed to be incorporated within a stand-alone computer 28, and the processor typically communicates with other elements of the system, including assembly 24, wirelessly, as is illustrated in Fig. 1. Alternatively or additionally, processor 26 may use optical and/or conducting cables for communication. In further alternative embodiments processor 26 is integrated within assembly 24, or in the mounting of the assembly. Processor 26 is typically able to access a database 40, wherein are stored images and other visual elements used by system 20. Software enabling processor 26 to operate system 20 may be downloaded to the processor in electronic form, over a network, for example. Alternatively or additionally, the software may be provided on non-transitory tangible media, such as optical, magnetic, or electronic storage media.

The medical procedure exemplified here is on a patient 30, and during the procedure professional 22 gazes along gaze directions 32 at a region of interest (ROI) 34. ROI 34 typically, but not necessarily, comprises a portion of the patient. In some embodiments one or more ROI acquisition markers 35, comprising marker elements 36, are positioned in, and/or in proximity to, ROI 34, and the functions of such markers are described below. Typically there are at least three marker elements 36 for a given marker 35. In a disclosed embodiment the size of ROI 34 may be predefined by professional 22, for example based on a computerized tomography (CT) image of the patient, and the position of the ROI may also be a predefined distance to the right and a predefined distance below the marker. In an alternative embodiment marker elements 36 of marker 35 define ROI 34 to be a region within a surface having elements 36 in the perimeter of the surface. Typically, a margin in an approximate range of 1 - 5 cm is added to ROI 34 to compensate for mis-alignment between a video projection and a directly viewed scene, described in more detail below.

During the procedure professional 22 may use a surgical device 38, such as a surgical knife, to perform part of the procedure. Typically device 38 comprises one or more identifying elements 39 which may be used to track the device.

1330-1002.3

Figs. 2A - 2E are schematic diagrams illustrating assembly 24, as well as functions that may be implemented in the assembly, according to an embodiment of the present invention. Fig. 2A illustrates assembly 24 with none of the active elements of the assembly, i.e., those elements requiring power, operating. As stated above, assembly 24 is configured, by way of example, as a pair of spectacles 50. Similar elements of each half of the pair of spectacles are referred to generically by an identifying numeral, and the similar elements are differentiated as necessary by adding a letter to the numeral. Thus spectacles 50 comprise planar optical combiners 52, comprising combiners 52A and 52B in front of, respectively, the left and right eyes of professional 22. Optical combiners 52 are mounted on a retaining structure 54 which holds elements of assembly 24, and which is herein assumed to comprise a spectacle frame, so that structure 54 is also referred to herein as frame 54.

In some embodiments, combiner frames 82A and 82B are fixed to retaining structure 54 and vertical retaining rods 84A and 84B attached to the combiner frames support the optical combiners, so that the combiners are able to rotate about vertical axes defined by the rods. Retaining rods 84A and 84B, and thus combiners 52A and 52B, may be rotated independently of each other about their vertical axes by respective motors 86A and 86B, fixed to frames 82A and 82B. Motors 86, typically stepper motors, are controlled by processor 26 so as to rotate their attached combiners to known, typically different, fixed orientations with respect to their respective combiner frames.

Each optical combiner 52 is configured to at least partially transmit elements of a scene through the combiner, so that a portion 56 of patient 30 (Fig. 1) is assumed to be directly visible through each combiner 52. In addition, each optical combiner 52 is configured to receive a visible radiation transmission derived from a scene, and/or a visual transmission such as a presentation of data or a marker, and to redirect or reflect the transmission back to the eye of professional 22. The redirection is such that the scene and/or data or marker presented to the professional appears to be at a distance between the near and far points of vision of the professional. Thus, any given section of the optical combiner may combine directly visible material with redirected or reflected material, and provide this combined material to the eye of the user. More detail of the functioning of combiners 52 is provided below.

Optical combiners of various types are known in the art. One known type uses a semi reflective surface which transmits an image from an image source after it has passed through a set of lenses which correct deformations caused by the semi reflective

1330-1002.3 surface of the combiner. Another known type uses a waveguide which projects the image directly to the eye of the viewer. Herein, by way of example, combiners 52 are assumed to be of the waveguide type.

In one embodiment, combiners 52 comprise LUMUS DK 32 see through glasses, produced by Lumus Optical of Rechovot, Israel.

Generally similar pixelated variable transparency screens 60A and 60B respectively coat a rear side, i.e., the side away from the eyes of professional 22, of combiners 52A, 52B. Screens 60 are active elements of system 20 and are formed of an array of pixels, the opacity of each of the pixels being controlled by processor 26.

Screens 60 are typically, but not necessarily, liquid crystal displays (LCDs) formed of a rectangular array of liquid crystal pixels. Alternatively, screens 60 are formed of MEMS (microelectromechanical systems). Further alternatively, screens 60 are formed of polymer dispersed liquid crystals (PDLCs). In the following description, by way of example, screens 60 are assumed to be formed of LCDs. LCD display pixels can typically be switched between an opaque state, where approximately 95% of the incoming light is blocked and 5% is transmitted, and a transparent state where approximately 60% of the incoming light is blocked and 40% is transmitted. The LCDs then have a transmission contrast ratio of 1:8.

Fixedly attached to arms of frame 54 are generally similar micro-projectors 64A and 64B. Each micro-projector is located and oriented so as to be able to project onto respective combiner 52A and 52B, a scene, and/or a visual indication, in a form suitable for redirection by the combiners to the left or right eye of professional 22. Micro-projectors 64 are active elements, and the projected scenes/indications are provided to the micro-projectors by processor 26. The projection and redirection are configured so that the images seen by the eyes of professional 22, absent any correcting lenses, appear to be at infinity, due to parallel light coming from the combiners and entering the pupils. In some embodiments display 24 comprises correcting lenses 88A, 88B which redirect light from combiners 52A, 52B so that the images appear to be closer than infinity to the professional's eyes. The power D in diopters of the lenses defines the distance d of the images, according to the formula d = 1/D, where d is in meters, and D is a negative number. Lenses 88A, 88B are typically located between the professional's eyes and the respective combiners. For simplicity, lenses 8 8A, 88B are not shown in other figures of the present application.

At least one image capturing device 68 is attached to frame 54. In the disclosed embodiment there are two generally similar devices 68A and 68B, respectively aligned

1330-1002.3 to be approximately orthogonal to planar combiners 52A and 52B, so as to be able to capture radiation of respective images of scenes viewed by the left and right eyes of professionaf 22. Typicaffy, devices 68 comprise cameras configured to capture images of scenes in the visible spectrum. The cameras may use rolling shutters, in which cases latency (of projection via micro-projectors 64) may be reduced by processing rows of images rather than complete frames of images. In some embodiments devices 68 may also capture non-visible portions of images, such as portions in the infra-red spectrum. The operation of devices 68 is controlled by processor 26.

In some embodiments of the present invention, assembly 24 comprises a sensor 72 which is configured to capture non-visible images of efements of a scene in front of assembfy 24. Typicaffy sensor 72 uses a projector 73 configured to project radiation in the non-visible spectrum detected by the sensor, and has a bandpass filter configured to block visible radiation, such as that projected by surgical fighting. Typicaffy, sensor 72 and projector 73 operate in the near infra-red spectrum.

In some embodiments, assembly 24 comprises a manual and/or electronic control 74 which may be operated by professional 22 to move elements of the assembfy in and out of the fiefd of view of the professionaf. Additionaffy or afternativefy, there may be a button or switch 78 which enables the professionaf to power active efements of assembfy 24, such as the capturing devices and the microprojectors. In some embodiments switch 78 may be a foot switch. Further additionally or afternativefy, assembfy 24 may be configured so that it can tilt downwards about a horizontai axis, at an angle up to 40° from the horizontaf, so that the professionaf can fook through the assembfy when fooking down.

Additionaffy, assembfy 24 may comprise a sensor 76, such as an accelerometer, which is configured to measure an inchnation of the assembfy with respect to the direction of gravity, so measuring the angle of the head of the professionaf with respect to the vertical. Processor 26 may be configured to use readings from sensor 76 to move efements of assembfy 24 in and out of the fiefd of view of the professionaf, and/or to controf whether micro-projectors 64 project images.

Fig. 2B schematically illustrates the appearance of assembfy 24 when processor 26 activates screens 60A and 60B. As described above, each screen 60 comprises an array of pixefs, and the opacity of each pixef in an individual screen may be set by processor 26. In screen 60A the processor has rendered a circular array 80A of the pixefs of the screen opaque, white the remaining pixefs of the screen are rendered transparent. The opacity of array 80A means that from the point of view of the left eye

1330-1002.3 of professional 22, circular array 80A acts as a mask occluding corresponding features of portion 56 of the patient, so that array 80A is also referred to herein as occluding mask 80A.

Similarly in screen 60B processor 26 has rendered a circular array 80B of the pixels of the screen opaque, while the remaining pixels of the screen are rendered transparent. As for array 80A, array 80B occludes sections of portion 56 from the view of the right eye of professional 22. Thus array 80B is also referred to herein as occluding mask 80B.

Fig. 2C schematically illustrates the appearance of assembly 24 when processor 26 activates screens 60 and micro-projectors 64A and 64B. Screens 60 are activated to provide occluding masks 80A and 80B, as described above with respect to Fig. 2B. Micro-projector 64A projects a prerecorded ultrasound image 90A of the patient's abdomen so as to overlay the image on mask 80A, and micro-projector 64B projects an image 90B of the abdomen so as to overlay it on mask 80B. Typically, although not necessarily, images 90A and 90B are the same. In some cases, for example if the images have been acquired in a stereoscopic manner or for correct 3D perception, images 90A and 90B may be slightly different, typically being slightly displaced horizontally with respect to each other. Micro-projectors 64 are configured to position images 90A and 90B on their respective masks so that, as seen by professional 22 and with -2 diopter lenses 8 8A, 88B present, the images are in focus at approximately 50 cm and appear to be at the location of the patient's abdomen.

In addition to projecting images 90, micro-projectors 64 also project alphanumeric data 92A and 92B onto the non-occluded region of screens 60, as well as markers 96A and 96B onto masks 80A and 80B. Images 90, data 92, and markers 96 are typically stored in database 40, and are provided from the database to microprojectors 64 by processor 26.

In a mentoring situation images 90, the contents of data 92, and the position of markers 96 are typically under control of a tutor interacting with processor 26 while mentoring professional 22. In some cases the locations of masks 80 may also be provided to processor 26 by the tutor, although typically the locations of the masks depend upon gaze directions 32 of the professional. In a non-mentoring situation, i.e. where professional 22 alone operates system 20, locations of masks 80 are typically automatically set by processor 26, as is described below. Also in a non-mentoring situation, images 90, data 92, and markers 96 may be controlled by professional 22. It will be understood that images 90, data 92 and markers 96 are examples of non-video

1330-1002.3 related visual elements that are seen by professional 22, and that the provision of such elements corresponds to an optic-based augmented reality situation implemented in system 20.

Fig. 2D schematically illustrates the appearance of assembly 24 when processor 26 activates screens 60 and micro-projectors 64, and in addition incorporates a videobased augmented reality feature into the operation of the assembly. Screens 60 and micro-projectors 64 are assumed to be activated as described above for Fig. 2C, so that masks 80, images 90, data 92 and markers 96 are in the field of view of professional 22. By way of example, the figure has been drawn to illustrate a mentoring situation, where the tutor of professional 22 wants to point to a feature of the chest of patient 30, herein assumed to comprise an unusual movement of the chest.

To point to the feature, the tutor interacts with processor 26 so that the processor enhances and emphasizes portions 100A, 100B of the video images acquired by capturing devices 68, the portions corresponding to the region of the chest where the unusual movement is occurring. Micro-projectors 64A, 64B then project portions 100A, 100B onto combiners 5 2A, 52B. It will be understood that the enhancement of portions 100A, 100B and their projection on the respective combiners is in real-time. The enhancement may take a number of forms. For example, portions 100A, 100B may comprise a wireframe image of the region of the chest having unusual movement, and/or a false-color image of the region. Other suitable methods of real-time enhancement will be apparent to those having ordinary skill in the art, and all such methods are assumed to be within the scope of the present invention.

Fig. 2E schematically illustrates an overall scene 101 as seen by professional 22, during an invasive surgical procedure being performed by the professional. For simplicity, Fig. 2E illustrates the scene as it is presented on combiner 5 2A, and it will be understood that a substantially similar scene is presented to the professional on combiner 52B. The figure illustrates a hand 102 of professional 22 holding device 38, herein assumed to comprise a pair of tweezers, at a proximal end of the device. One or more device identifying elements 39, typically reflectors and/or radiators, are attached to the tweezers, so that processor 26 is able to identify and track device 38 using images acquired by capturing devices 68.

The professional has made an incision 104 in a portion 106 of patient 30, and ROI 34, defined by marker elements 36, is assumed to be at the location of the incision. In addition, the professional has inserted a lower portion of the distal end of device 38 into the patient so that the lower portion is no longer visible.

1330-1002.3

Processor 26 has formed mask 80A on combiner 52A so as to occlude ROI 34, and the portion of incision 104 comprised in the ROI. Mask 80A also includes a margin 83, typically corresponding to a margin of approximately 1 - 5 cm at the ROI. Thus, all elements of the scene outside mask 80A, comprising hand 102 and the proximal end of device 38, are directly visible through combiner 52A by the professional. However, elements of the scene within mask 80A, including a portion of incision 104 and an upper portion of the distal end of device 38 that is outside the patient, are not visible to the professional, since they are occluded by the mask.

Processor 26 overlays on mask 80A a captured image 110 of the ROI and the region corresponding to margin 83, which includes the portion of incision 104 occluded by the mask and which also includes a video image 114 of the upper portion of the distal end of device 38 (outside the patient) that has been captured by image capturing device 68. In addition, the processor overlays on the occlusion mask a stored image 112 corresponding to the lower portion of the distal end of device 38 (within the patient). Stored image 112 is a virtual elongation of image 114 and is retrieved from database 40. The section of the distal end corresponding to image 112 is not visible to capturing device 68.

The processor registers the two overlaid images, image 110 and image 112, with each other, and the registration is possible since by tracking device 38 the processor is aware of the location of the device distal end with respect to the captured image. Thus, there is no misalignment between stored image 112, corresponding to the lower portion of the distal end, and image 114 of the upper portion of the distal end, which is included in captured image 110.

However, there is typically misalignment between the two registered overlaid images 110, 112 and the directly visible portion of scene 101, including the directly visible portion of incision 104, as is illustrated in the figure. The misalignment occurs because while the captured image of the ROI is close to that seen by the professional (in the absence of the occlusion mask), it is not exactly in registration with the viewed scene. The inventors have found that a misalignment of up to 2 cm, in a scene that is 50 cm from the eye of the professional, is acceptable.

Figs. 3A and 3B are schematic diagrams illustrating assembly 24 in different orientations with respect to ROI 34, Fig. 3C is a schematic diagram illustrating angles of the assembly for the different orientations, and Figs 3D, 3E are graphs of the angles, according to an embodiment of the present invention. For simplicity, combiner frames 82 are not shown in the diagrams. In Figs. 3A and 3B processor 26 has positioned

1330-1002.3 masks 80 so that they act to occlude ROI 34 from eyes 120A, 120B of professional 22, specifically from pupils 124A, 124B of the professional's eyes. Fig. 3A illustrates a situation where ROI 34 is approximately directly in front of professional 22. In this case the processor forms mask 80A' to be on a straight line with pupil 124A and ROI 34, while simultaneously forming mask 80B' to be on a straight line with pupil 124B and the region of interest.

Fig. 3B illustrates a situation where ROI 34 is not directly in front of professional 22, but is towards one side of the professional. As for the situation of Fig. 3A the processor forms mask 80A to be on a straight line with pupil 124A and ROI 34, and forms mask 80B to be on a straight line with pupil 124B and the region of interest. In all cases masks 80 act as occlusion masks, and, as is illustrated by the differences in the positions of the masks, the processor changes the locations of the masks to account for changes in orientation of the region of interest with respect to assembly 24.

A ring 130 surrounding ROI 34 is described in more detail below.

Fig. 3C schematically illustrates the two situations of Figs. 3A and 3B, when ROI 34 is at a distance L from eyes 120A, 120B of the professional. Eyes 120A and 120B are separated by a distance w. For the situation of Fig. 3A, where the region of interest is directly in front of the professional, ROI 34 is at a location 103. For the situation of Fig. 3B, ROI 34 is to the left of the professional, at a location 104 that is a distance R from location 103.

For the first situation, where professional 22 is looking at location 103, the directions of gaze, ocr, ocr of the professional are shown by lines 103R and 103L. ocr, ocr are angles that are measured with respect to lines orthogonal to a line connecting eyes 120A, 120B, and their values are given by the following equations:

+ (^w\ a, = — arctan — , ^L \2lJ (A)

For the first situation processor 26 rotates combiners 52A and 52B (for clarity the combiners are not shown in the figure for the first situation), within their respective frames 80A and 80B, so that they are orthogonal to lines 103L and 103R. Thus the orientation of the combiners to their frames is given by equations (A).

For the second situation, where professional 22 is looking at location 105, the directions of gaze of the professional are shown by lines 105L and 105R. These

1330-1002.3 directions are respectively changed from the straight ahead directions by βρ, βρ. The values of βρ, βρ are given by equations (B):

/ /1Λ7Χ 2 _ P \ fi_L = acos (B) β_η = acos

4L²w²

For the second situation processor 26 rotates combiners 52A and 52B, within their respective frames 80A and 80B, so that they are orthogonal to lines 105L and 105R. Thus the orientation of the combiners to their frames is given by equations (B), and these orientations are illustrated in the figure.

Fig. 3D is a graph of angles βρ, βρ vs. R for values of L = 512 mm, w = 60 mm. Fig. 3E is a graph of absolute angles γρ, yr, of the angles made by combiners 5 2A, 52B with their frames where

Yl, = Pl + a_L, Yr, = Pr + Vr (C)

From the above equations, as well as from the graphs, it is apparent that the angles made by combiners 5 2A, 52B with their respective frames are different, as professional 26 gazes at a region of interest. In addition, if the professional changes his/her gaze, the changes of the combiner angles to maintain orthogonality with the gaze directions are also different.

It will be understood that calculations based on equations herein, including equations (A), (B), and (C), assume that combiners 52A, 52B transmit rays that are orthogonal to the combiners. Those having ordinary skill in the art will be able to adapt the calculations, mutatis mutandis, for situations where the combiners transmit nonorthogonal rays.

1330-1002.3

Fig. 4 is a schematic diagram illustrating derivation of the dimensions of occlusion mask 80, according to an embodiment of the present invention. Fig. 4 is derived from a section of Fig. 3B, specifically the section illustrating eye 120A, with its pupil 124A, being occluded by mask 80A while the eye is gazing at ROI 34. Fig. 4 illustrates a cross-section of the eye, the occlusion mask, and the region of interest. The figure has been drawn on xy axes with an origin O of the axes at the center of pupil 124 and the directions of the x and y axes being respectively orthogonal to and in the plane of the pupil. Mask 80A and ROI 34 are assumed to be orthogonal to, and to be symmetrically disposed with respect to, the x-axis. Pupil 124 is assumed to be substantially circular. For simplicity, mask 80A and ROI 34 are also assumed to be substantially circular. However, those having ordinary skill in the art will be able to adapt the following description, mutatis mutandis, for regions of interest and occlusion masks that are non-circular, so that the scope of the present invention is assumed to comprise both circular and non-circular regions of interests and masks.

The diagram has been drawn assuming that mask 80A just completely occludes ROI 34. Thus a ray HB, from an upper edge H of ROI 34 to an upper edge B of pupil 124A touches an upper edge F of mask 80A. Similarly, a ray GA, from a lower edge G of ROI 34 to a lower edge A of pupil 124A touches a lower edge E of mask 80A. Rays HB and GA are assumed to cross at an imaginary point J. A line from upper pupil edge B parallel to the x-axis cuts mask 80A at K and ROI 34 at M.

In the description below:

p is the apparent diameter of pupil 124A, as measured externally to eye 120A, corresponding to AB; and d is the diameter of mask 80A, corresponding to EF; d = d! for a realistic case of p > 0, d = άθ is the diameter of the mask for a theoretical pinhole case of p = 0.

In addition,

D is the diameter of ROI 34 (which is occluded by mask 80A), corresponding to GH;

L is the distance from pupil 124A to ROI 34;

1! is the distance from pupil 124A to point J; and is the distance from pupil 124A to mask 80A.

In Fig. 4 AJFE III AJHG, so that d _ 1+1!

D ^— L+l!

(1) (2) (3)

1330-1002.3

From equation (1), d = -^-D

L+l^

If 11 = 0, (for the theoretical case of p = 0), then d = d₀=l׳D

J_l

If 1! > 0, for the realistic case of p > 0, then d = d! = EL . D(4)

FK FM

ΔΒΕΚ III ΔΒΗΜ, so that — = —(5)

BK BM

For p > 0 (so d = d!) and substituting values of dp p, 1, and L for FK, BK, FM, and BM in equation (5) gives:

d! pD_p

2 2 2

⁽⁶⁾ T ־ Γ־

Equation (6) rearranges to:

י l(D-p) di = —— + p(7)

J_l

Equation (7) gives dimensions of mask 80A, i.e., its diameter dp in terms of the diameter D of ROI34, the distance 1 of the mask from the pupil, the diameter of the pupil, and the distance L of the ROI from the pupil.

For typical values of 1 = 2 cm, L = 50 cm, p = 0.3 cm, and D = 15 cm the diameter of mask 80A to just give complete occlusion is, from equation (7), approximately 0.9 cm. For the same values but with p = 0.15, the mask diameter is approximately 0.7 cm.

While, as described above, mask 80A completely occludes ROI 34, there are regions outside ROI 34 that are partly occluded by the mask. The partial occlusion follows from the finite, non-zero diameter of the pupil of eye, in the example described here pupil 124A, and is described in more detail with reference to Fig. 5 below.

Fig. 5 is a schematic diagram illustrating partial occlusion of an area around ROI 34, by mask 80A, according to an embodiment of the present invention. Fig. 5 is based upon Fig. 4, but for clarity some of the elements of Fig. 4 are omitted in Fig. 5,

1330-1002.3 while other elements are added in. Thus, a line through point A, parallel to the x-axis, cuts ROI 34 at N. A point Q, lying in the same plane as ROI 34, and at a distance R from the x-axis, is assumed to project two rays - a lower ray 150 which touches lower edge A of the pupil, and an upper ray which, but for the presence of mask 80A, would touch upper edge B of the pupil. Point Q is thus partly occluded by mask 80A.

In Fig. 5 at a distance x! from the pupil lower ray 150 is assumed to be a distance fl(x!) from the x-axis, and upper ray 160 is assumed to be a distance f2(x!) from the x-axis. A line parallel to the y-axis, at x!, cuts BM at S, AN at T, upper ray 160 at V and lower ray 150 at W. Upper ray 160 cuts mask 80A at V, and lower ray 150 cuts a plane containing the mask at W'.

At mask 80A the distances of lower ray 150 and of upper ray 160 from the xaxis are respectively fl(l) and f2(l), and the width of the beam between the upper and lower rays is:

fl(l) - f2(l)(8)

From the diagram, partial occlusion occurs if:

fl(l) > | and f2(l) < |(9) no occlusion occurs if:

f2(l) > |(10) and full occlusion, corresponding to the situation illustrated by Fig. 4, occurs if:

fl(D&lt;|(11)

From expressions (8) and (9), and inspection of Fig. 5, an equation for the fraction F2d of occlusion occurring is:

^-12(1) ^2D fl®־ f2(l) (The subscript 2D indicates that the fraction considered here is for the twodimensional case illustrated in Figs. 4 and 5. A fraction for the three-dimensional case is referred to below.)

Since AATW III AANQ fl(D=L(R-E)₊E (13)

1330-1002.3

Since ΔΒSV III ABMQ f2(l) = Vr + -)-^{v 7} L\ 27 2 (14)

From equations (13 and (14) the diameter of the cone cross-section from Q at mask 80A, which is fl(l)-f2(l), is given by:

fl(l) - f2(l) = V׳W׳ = p(l - d

J_l (15)

Substituting equations (14) and (15) into equation (12) gives the following expression for F2d:

Inspection of equation (16) indicates that the fraction of occfusion at point Q is a function of pupif diameter p, and afso decreases hnearfy as R increases.

HG is a cross-section of circufar ROI 34, so that it wiU be understood that GQ is a cross-section of a circufar, partiaffy occfuded circufar ring 130 surrounding ROI 34. As illustrated in Fig. 5, there is a point Q', having the same distance R as Q from the x-axis (but on the opposite side of the axis), and in the same pfane as ROI 34, so that HQ' is also a cross-section of ring 130.

The rays from point Q define a cone of rays emanating from Q, and this cone cuts mask 80A in a circfe having a diameter VW', the diameter being given by equation (15). The cutting of mask 80A by the cone of rays from Q is described with reference to Fig. 6 befow.

Fig. 6 iUustrates mask 80A drawn in a pfane orthogonaf to the x-axis, according to an embodiment of the present invention. A circfe 170, which has a diameter given by equation (15), is the circle cutting the plane containing mask 80A. A portion 174 of this circfe is occfuded by circufar mask 80A, and a portion 178 is transmitted. The expression for the two-dimensionaf fraction of occfusion F2d given by equation (16) corresponds to anafysis afong a fine FV'EW'.

There is a corresponding equation for a three-dimensionaf fraction of occfusion F3d, given by the fohowing expression:

(17)

1330-1002.3 where A is the area of portion 174, and Al is the area of circle 170.

F3D may also be written as:

<img file="IL247922A_D0001.tif" />

<img file="IL247922A_D0002.tif" />

(18)

From equation (18), F3□ is a function of pupil diameter p, and the equation provides numerical values of F3□ for selected values of d, R, p, 1, and L.

Fig. 7 illustrates graphs of occlusion vs. distance, according to an embodiment of the present invention. The graphs have been drawn assuming the following values:

L = 50 cm = 2 cm

P = 0.3 cm

D = 15 cm

From equation (7) the diameter of the occlusion mask to fully occlude an ROI with diameter D of 15 cm is d = 0.888 cm. The graphs of Fig. 7 have been drawn with d set at 0.894 cm.

From equation (15) the diameter of circle 170 is 0.288 cm, so that the value of the area Al of the circle is 0.065144 cnA

1330-1002.3

A solid line graph 200 illustrates the full and partial occlusion vs. distance (from the center of the ROI) for the three-dimensional case comprising equation (18). The measurements of occlusion have been normalized, so that for an LCD screen a full occlusion of 95% is normalized to 1, and a full transparency (of 60% occlusion) is normalized to 0. A broken line graph 204 illustrates the full and partial occlusion vs. distance for the two-dimensional case comprising equation (16). As is apparent from both graphs, there is full occlusion, for a mask of diameter d = 0.894 cm, for a region 208 up to approximately 8 cm from the center of the ROI, and partial occlusion in a region 212 from approximately 8 cm to approximately 15 cm. The fraction of partial occlusion decreases monotonically in region 212.

Fig. 8 is a flowchart of steps performed in operation of system 20, according to an embodiment of the present invention. The steps are assumed to be performed by processor 26 and, as necessary, professional 22 for use in a procedure on patient 30 performed by the professional using device 38 (Fig. 1). In an initial calibration step 300, frame 54 is adjusted to center combiners 52 with respect to the eyes of the professional. The sizes of the combiners and the distances of the combiners from the eyes of the professional are also measured and recorded by the processor (the use of the measurements is described further below). In step 300 assembly 24 is calibrated, i.e. optical elements of the assembly are registered with each other. Thus, devices 68 are oriented on frame 54 to capture generally similar images from the region in front of combiners 52. If sensor 72 is present it is also aligned to capture a corresponding nonvisible image from the region.

Processor 26 also orients the images from micro-projectors 64, by registering the images projected by the micro-projectors onto combiners 52 with the scene viewed by professional 22 through the combiners. The registration may be accomplished by the professional viewing a scene through combiners 52, together with an image of the same scene as it is captured by devices 60 and projected by the micro-projectors onto the combiners. The professional then adjusts the orientation of the micro-projectors and/or the capturing devices so that the projected image and the viewed scene coincide.

Typically the registration and adjustment of the micro-projectors and the capturing devices is performed for different regions of combiners 52, such as the left and right peripheral regions, the upper and lower peripheral regions, and a central region. In addition, the registration and adjustment may be performed for different scenes according to the distance of the scene from the combiner, such as a scene of relatively near elements, typically up to Im from the combiner, and a scene of

1330-1002.3 relatively far efements, typicaffy greater than fm from the combiner. The registrations and adjustments of the micro-projectors and the capturing devices are typicaffy different for the different regions of the combiners, as weff as for scenes at different distances from the combiners. Processor 26 stores the different registration data acquired during the cafibration step for use when the professionaf is using assembfy 24.

During the cafibration step the sizes of the pupifs of the eyes of professionaf 22 are measured. In one embodiment professional 22 gazes at a circular object of a known diameter and at a known distance from the professionaf, and processor 26 presents an occfusion mask on screens 60 to the professionaf. The professionaf then adjusts a diameter of the occfusion mask untif compfete occfusion of the object is achieved. As is apparent from equation (7), the diameter of the compfetefy occfuding mask provides a vafue for the pupif diameter, since άμ 1, L and D (terms in equation (7)) are aff known.

Afternativefy or additionaffy, the professionaf may fook into a mirror white image capturing devices 60 acquire images of the reflected scene, in this case the professional wearing assembly 24. Processor 26 analyzes the acquired images, by processes that are well known in the art, to identify the pupifs of the professionaf as weff as the outfines of combiners 52. The processor then compares the diameters of the pupifs with the known dimensions of the combiners, so as to determine vafues for the diameters.

The measurements of the pupif diameters are taken for different ambient fight brightnesses, and the ambient brightness vafues may be determined from the signaf fevefs of the images acquired by devices 68. Processor 26 stores the vafues of the pupif diameters, and the corresponding brightness fevefs.

As stated above, processor 26 is configured to track device 38, using the one or more identifying efements 39 (Fig. 1). In calibration step 300 the processor initiates tracking of device 38, and professionaf 22 confirms that the tracking is acceptabfe.

In an ROI defining step 302, ROI acquisition marker 35 (Fig. 1) is positioned on patient 30, so as to define a region of interest of the patient sefected by the professionaf, herein assumed to be ROI 34. As explained above, marker elements 36 of marker 35 define the position of ROI 34, and the size of the ROI may be defined by the professionaf. Typicaffy there are at feast three marker efements 36, afthough more may be used, and characteristics of the efements, such as their cofor and/or shape, are sefected so that they may be easify distinguished from patient 30. If assembfy 24

1330-1002.3 comprises sensor 72 with an infra-red projector, marker elements 36 may be configured as retro-reflectors which selectively reflect only infra-red radiation.

In an imaging step 304, image capturing devices 68 acquire images of the scene in front of assembly 24. Sensor 72, if present, also captures a corresponding image of the scene. Processor 26 analyzes the images to identify marker elements 36, and from the identified elements determines the orientation of ROI 34 with respect to assembly 24, and also the distance of the ROI from the assembly. Even if sensor 72 is not present, it will be understood that having two devices 68 acquiring respective images of the scene simplifies the analysis needed to be performed by the processor to identify elements 36. In addition, having two capturing devices 68 reduces the number of elements 36 required to accurately determine the orientation and distance of the ROI with respect to assembly 24, compared to the number required if only one capturing device 68 is used. With two capturing devices 68 the inventors have found it is sufficient to have one marker with three marker elements to accurately locate the ROI with respect to assembly 24. If sensor 72 is present, its image alone may be sufficient to identify elements 36, although typically processor 26 uses the images from devices 68 to improve the accuracy of the orientation and distance measures of the ROI determined by the sensor.

Processor 26 also analyzes the images acquired by devices 68 in order to determine a measure of the brightness of the scene in front of assembly 24.

In a frame orientation step 305, the processor rotates combiners 52A and 52B with respect to their respective frames so that the combiners are orthogonal to the gaze directions of the professional towards the ROI. The processor uses equations (A), (B) and/or (C) to determine the angles of rotation of the combiners.

In a masking step 306, the processor generates circular occlusion masks 80 in screens 60. The processor, using the orientation of the ROI measured in step 304 and the central adjustment of combiners 52 in step 300, determines positions for the masks that will occlude ROI 34. From the brightness measured in step 304, and from the correspondence between pupil size and brightness stored in initial step 300, the processor estimates a value of the pupil diameter of the professional.

In one embodiment the processor sets the diameter of masks 80 according to equation (7), i.e., inter alia, according to the professional's pupil size, so that the masks fully occlude ROI 34. In this case partially occluded ring 130 surrounds ROI 34, the fraction of partial occlusion within the ring being given by equations (12) and (18).

1330-1002.3

In some embodiments the processor determines sections of the scene corresponding to partially occluded ring 130, and as acquired by devices 68. The processor then configures micro-projectors 64 to overlay video of the acquired sections onto the partially occluded ring, so as to compensate for the partial occlusion. Typically, processor configures the intensity of the projected video to be the inverse of the fraction of the occlusion.

In an alternative embodiment, rather than setting the diameter of the masks to be according to equation (7), the processor sets the diameter to be reduced from the value determined by the equation. The reduction is typically determined by professional 22. In one embodiment the diameter is set to be 90% of the value determined by equation (7).

In a further alternative embodiment, the processor, using instructions from professional 22, sets the diameter of the masks to be larger than the diameter of equation (7). In one embodiment the diameter is set to be 110% of the value determined by equation (7).

In a mask projection step 308 processor 26 uses micro-projectors 64 to project augmented video onto occlusion masks 80. In the case of the augmented video including two or more types of images being projected onto the masks, processor 26 registers the images with each other. However, the images are not necessarily registered, and are typically misaligned, with the scene surrounding and outside the masks. Thus, as exemplified by Fig. 2E and the description of the figure, a video image 114 of the upper portion of the distal end of device 38, together with a stored image 112 corresponding to the lower portion of the distal end, are registered together and are projected onto masks 80. As is also illustrated in Fig. 2E, the images on masks 80 are typically misaligned with the visible scene outside the masks.

In a further projection step 310, processor 26 uses micro-projectors 64 to project augmented video onto the partially occluded ring surrounding the masks, and/or the non-occluded section of combiners 52. As in step 308, multiple image types are registered together, but are typically misaligned with the visible scene of the nonoccluded section.

Typical images that are projected in steps 308 and 310 include, but are not limited to, those described above with respect to Figs. 2C, 2D, and 2E, and the choice and positioning of the images is typically under the overall control of professional 22. In a mentoring situation, at least some of the images are typically under control of a mentor of professional 22.

1330-1002.3

It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, 5 as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.