Visual information system.

1. FIELD OF THE INVENTION

This invention relates to the field of image processing systems, more particularly to head-mounted image processing and display systems.

2. BACKGROUND OF THE INVENTION

There are many instances when it is advantageous to use photographs or electronically stored images to aid a viewer in observing the features and spatial relationships of an object. The features the viewer would like to locate or see are sometimes obscured or are difficult to locate with the unaided eye.

One example is during a surgical procedure. A surgeon may take a photograph or an electronic image of an area before surgery begins. The image may be from previous diagnostic test such as magnetic resonance imaging (MRI) or computerized axial tomography (CAT). The surgeon will refer to this image during surgery as a guide for the procedure. For example, an image may be taken of a tumor before starting a removal procedure. During the procedure, the surgeon can refer back to the image to assist in locating the tumor or to ensure that the entire tumor has been removed.

SUMMARY OF THE INVENTION

The present invention provides a method and device for providing visual information to a viewer without blocking the vision of the viewer or requiring the viewer to look away from the object of interest. One embodiment of the invention to provides an imaging system that displays a computer generated image in the field of view of the user, without obstructing the view of the user. This allows the user to see the image without looking away from the object of interest. Furthermore, the image the user sees will be manipulated to have the correct perspective or viewpoint. The computer generated image may be superimposed on the actual image that the user sees to help the user correlate information from the computer generated image with the users actual view of the object.

It is a further advantage of the invention that the movement of the user or object can be monitored by the system and the displayed images changed to compensate for the movement. This allows the user to see the object from different perspectives by simply moving his head. This three dimensional effect is important in that the image may show internal features that may not be visible by the user. For example, during a surgical procedure locations of major arteries can be marked to allow the surgeon to avoid them.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of one embodiment of a visual information system.

FIG. 2 is a schematic view of a first embodiment of a display mechanism of the visual information system.

FIG. 3 is a pictorial view of a second embodiment of a display mechanism of the visual information system.

FIG. 4 is a pictorial view of one embodiment of the visual information system.

FIG. 5 is a pictorial view of a patients hand as seen through one embodiment of the disclosed visual information system.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

There are several drawbacks associated with using stored images to assist a viewer in identifying objects in his view. First, the viewer must look away from the object of interest in order to see the stored images, interrupting the work being performed and breaking the viewers concentration. Second, the images presented to the viewer have a fixed perspective, that is the viewer cannot usually manipulate the viewpoint of the image. Depending on the circumstances, the image may be difficult to correlate with the prespective of the viewer. For example, when testing an electronic assembly, the image of a component to be probed or removed may not be of much help when the component is surrounded by a large number of very similar, or worse yet, identical components.

A much better solution is to superimpose the image on the object being viewed, in effect highlighting the features of interest in the object. There are many applications for the vision enhancement imaging system disclosed, for example, three dimensional information from the images used to guide the surgical procedure discussed above could be used to generate enhanced images and projected into the line of sight of the surgeon to allow the surgeon to see the images without having to look away from the patient.

Figure 1 shows a schematic view of one embodiment of the disclosed imaging system. The video camera <b>22</b> monitors the subject, in this case a surgery patient, and transmits an electronic representation of the image to the processor <b>28</b> via channel <b>34</b>. The choice of video camera depends on the information required and is not critical to the invention. For example, the camera could be monochrome or color, digital or analog, and could be receptive to light outside the visible spectrum. In some applications it may be advantageous to have more than one camera to provide the processor with additional perspectives to aid in the generation of three-dimensional images.

Optional patient position sensor <b>24</b> transmits information to the processor <b>28</b> to allow the image processing and display functions to compensate for patient motion. Note that the object position and motion information can be extracted from the image obtained via the camera and the use of vision algorithms. The processor <b>28</b> may determine position and motion information by tracking features of the patient. If no suitable features exist naturally on the patient, high contrast marks could be drawn on the patient prior to beginning a procedure to allow the patient to be tracked optically by the processor <b>28</b>.

Viewer position sensor <b>26</b> tracks the position of the viewer to allow the viewer to move or turn to see different views of an image. Not only does the position sensor <b>26</b> allow the processor <b>28</b> to compensate for the viewing angle, it allows the processor <b>28</b> to scale the image as the distance from the object to the viewer changes and to move the image across the screen as the viewer scans the object. If the initial positions of the viewer and object are known, then motion sensors may be used instead of position sensors. Motion compensation prevents images from being warped or smeared as the patient moves.

The processor <b>28</b> applies image processing algorithms to manipulate the image from the camera <b>22</b> and other image sources. The other sources are not shown in Figure 1, but could include scanned photographs, MRI or CAT scans, other medical diagnostic equipment, prior surgical procedures, or any image device. Image processing functions that would be desirable in some applications are image conditioning, feature extraction, and matching or correlation algorithms. The computer generated images may be wire-frame representations or fully rendered three-dimensional images or a hybrid combination of the two. For example, a tumor may be fully rendered and colorized while the surrounding tissue is represented by a wire-frame model. The processor could also use false coloring to highlight certain features or objects. Textual data may also be displayed to inform the surgeon of the patients vital signs, duration of surgery, etc.

In addition to manipulating the image data, the processor <b>28</b> may also receive inputs from the surgeon to allow the surgeon to mark areas of interest and to build a graphical database for later analysis. Inputs from the surgeon could also be used with stored information to aid feature recognition. The surgeon could use any available means to input data into the processor including a keyboard, mouse, touch screen, trackball, light pen, or voice recognition.

Processor <b>28</b> can be any type of processor or group of processors capable of performing the necessary image processing and display control functions required of the imaging system. For example, the processor could be designed around the Texas Instruments Multimedia Video Processor (MVP) image processing chip. The processor should have sufficient through-put to process the images, object motion, viewer position, and viewer input data in real-time. The required frame rate will vary depending on the application but should preferably be in the range of 60 to 100 frames per second, although 30 or fewer frames per second may be acceptable in some circumstances, especially situations with very little movement. The frame rate may be dynamically varied as image complexity and image processing algorithms vary the demands on processing power.

The processed images are sent to the display device <b>30</b> via display control bus <b>40.</b> The display device could be any type of spatial light modulator capable of displaying an image such that it appears superimposed on an object being viewed. Optics <b>33</b> are used as required to focus and project the image. Light source <b>32</b> and optics <b>33</b> may not be required in all applications depending on the type of spatial light modulator used.

One embodiment of a display system <b>68</b> comprising the display device, light-source, and optics is shown in Figure 2. Light from source <b>54</b> is focused and collimated as necessary by lens <b>58.</b> The focused light <b>56</b> is modulated by a spatial light modulator, in this case a DMD array <b>60</b>. The reflected light <b>62</b> is focused and magnified as necessary by lens <b>64</b>. The light is then reflected off of the lens <b>50.</b> At the same time, light <b>48</b> from the object <b>46</b> is allowed to pass through the lens <b>50.</b> The light <b>52</b> from the object and the DMD array exits the display system <b>68</b> and is seen by the viewer.

The lens <b>50</b> may be worn like a pair of goggles, or a face shield, to allow the viewer to see the object <b>46</b> through the lens <b>50</b> and also see the image projected onto the lens. Lens <b>50</b> may be partially silvered to reflect light or may simply be any material with a partially reflective surface. Lens <b>50</b> is typically designed to allow approximately 90% of the light from the object to pass through the lens <b>50</b> while reflecting 10% of the reflected light <b>66</b> from the DMD. The amount of light reflected and transmitted by lens <b>50</b> depends on the application and the amount of light available from the object and the imager. The DMD <b>60</b> is particularly useful for this applications because it allows the brightness of the projected image to be adjusted over a wide range. This wide adjustment allows the viewer to strike an acceptable balance between image brightness and object visibility.

A second embodiment of a display system according to the present invention simply attenuates light from the object of interest. This second embodiment, shown in Figure 3, may use a liquid crystal display (LCD) instead of a DMD. Because no light source or projection optics are required, the LCD <b>67</b> may be the actual lens of the goggles <b>69</b> worn by the viewer. Cable <b>71</b> carries the signals necessary to operate the LCD. The use of an LCD instead of the reflective lens of Figure 2 attenuates more of the light from the image and reduces the visibility of the viewer. Areas of interest may be marked by attenuating light from the areas of interest, or by attenuating light from all areas except those of interest.

A pictorial view of one embodiment of the surgical imaging system is shown in Figure 4. Viewer <b>72</b> looks though display system <b>68</b> at the object <b>46.</b> In this embodiment both the display system <b>68</b> and the viewer position sensor <b>26</b> are head mounted on the viewer <b>72</b>. Object position sensor <b>24</b> is attached to the object <b>46</b> without obstructing the view of either the viewer <b>72</b> or the camera <b>22</b>. Processor <b>28</b> receives data from the camera <b>22</b>, the viewer position sensor <b>26</b>, and the object position sensor <b>24</b> and sends data to the display system <b>68.</b>

Figure 5 shows one example of the view through the imaging system during an operation to remove a foreign object from a patient. In Figure 5 a surgeon is attempting to remove a foreign object <b>80</b> from a patient's hand <b>82.</b> Textual information <b>84,</b> such as the patient's vital signs, distances between objects, and elapsed time may be displayed. Objects of concern, in this case a nerve <b>86</b> that the surgeon is attempting to avoid, and the entrance path <b>88</b> of the object <b>80,</b> may be highlighted. It should be appreciated that as the surgeon moves in relation to the patient, the perspective of the displayed image and the patient's hand will change in unison allowing the surgeon to determine the best method of removing the object <b>80</b>.

Thus, although there has been disclosed to this point a particular embodiment for a surgical imaging system, it is not intended that such specific references be considered as limitations upon the scope of this invention except in-so-far as set forth in the following claims. Furthermore, having described the invention in connection with certain specific embodiments thereof, it is to be understood that further modifications may now suggest themselves to those skilled in the art, it is intended to cover all such modifications as fall within the scope of the appended claims.