System and method for augmentation of endoscopic surgery

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Industrial application field]

The present invention relates to a surgical procedure using an endoscope. More specifically, it relates to obtaining accurate information about the anatomical structure in the patient's body and using that information to align the endoscopic camera and surgical instruments in the patient's body to the correct position. ..

【0002】

[Conventional technology]

Various systems have been developed to allow surgeons to see images that internally treat anatomical structures within the patient, augmenting the surgeon's capabilities. Generally, these systems consist of specialized shaped cameras or medical telescopes. In addition, these systems, including endoscopic or laparoscopic instruments, have reduced the invasion of pathogens that affect healthy tissues associated with many surgical procedures.

【0003】

This type of system has some notable common features: First, surgeons using these systems cannot directly handle the patient's anatomy. Second, surgeons cannot see directly what they are doing. Instead, surgeons have to rely on instruments that are inserted into the body through a trocar needle or endoscopic tube. Often, the surgeon's hands and attention are full of procedures, so during the procedure you must rely on an assistant to operate the endoscopic camera.

【0004】

In order to improve the inconvenience of such a device, a robotic augmentation device for endoscopy has been developed. One such device is described in detail in the co-pending U.S. Patent Application No. 714,816 (filed June 13, 1991), "Systems and Methods for Surgical Enhancement." The contents of this patent application shall be referred to herein as necessary.

【0005】

Robotic augmentation devices have the potential to greatly assist surgeons in their practice. The robotic device does not get tired. Medical telescopes and surgical instruments can be aligned very accurately, with the potential for accurate realignment and repetitive functioning. However, in order to realize these advantages of robotic devices, a number of problems need to be solved. The surgeon needs to determine the action to be taken by the robotic device and requires a means of communication with the computer controlling the robot. The operation of the device can be determined in advance, such as bone shaping by a plastic surgery machine, the amount of excision determined in advance by medical imaging equipment (CT scan or MRI scan, etc.), planned tissue excision, etc. This is the case. However, in other cases, the surgeon needs to look directly at the patient's tissue and specify the action to be taken with respect to the anatomical tissue structure and medical telescope. In these cases, all that is required is the exact placement of the anatomical tissue structure and surgical instruments relative to the medical telescope and with respect to each other, and to use this information to enhance robotics. Is to control.

【0006】

Specialized robotic devices have been developed that operate a prostate obstruction resection endoscope according to pre-programmed procedures. However, the system provides the surgeon with a convenient means of controlling the field of view seen by the endoscopic device, and interacts with the surgical instrument in response to images of operational conditions within the body and other perceptual information. It does not address the problem of providing or needing surgeons with means to operate in a methodical manner.

【0007】

There was an example of an attempt to control a flexible endoscope by voice. In this method, a servomotor directly attached to a control knob of a commercially available flexible endoscope is activated in response to a surgeon's command. This attempt has the following drawbacks. (a) The surgeon (or assistant) still needs to determine the direction in which the tip of the endoscope is directed to obtain the required field of view. As a result, the distance between the tip of the endoscope and the observed anatomical tissue must be kept constant. (b) The above adjustments must be made continuously, distracting the surgeon's attention from what is more important. (c) The use of voice commands for the purpose of endoscopic control can distract the surgeon and is prone to negligence. It can also make voice communication between the surgeon and operating room personnel more difficult.

【0008】

There are several studies on the improved mechanism of flexible endoscopes. However, these devices cannot simplify the problem that the surgeon himself must control the endoscope by instructing a skilled operator to obtain the required field of view.

【0009】

[Problems to be Solved by the Invention]

Unfortunately, the field of view of medical telescopes used in surgical procedures with minimal invasion to the body is limited. As a result, only a small portion of the anatomical tissue structure hidden within the patient's body is seen at one time. At any given time, it is possible to observe from only one observation point at a time, and it is difficult to obtain the required field of view.

【0010】

Usually, to compensate for such a limited field of view, a surgical assistant manipulates and turns the telescope to capture the anatomical tissue structure in different fields of view. During this operation, the assistant must maintain the relative orientation of the telescope and the anatomical tissue structure so that the telescope can be aimed quickly and accurately according to the surgeon's requirements. It doesn't become. In addition, the assistant must correctly interpret the surgeon's request, which is not always clear from the words spoken by the surgeon.

【0011】

This causes a number of problems. This type of surgical procedure also requires a highly skilled assistant to assist the surgeon in operating the medical telescope. This is because the surgeon has both hands occupied by other tasks. The interaction between the surgeon and his assistant increases the likelihood of negligence during surgery. The telescope cannot capture an image of the entire affected tissue. Therefore, the surgeon (and assistant) must keep in mind the appearance of the entire affected tissue that is not visible. The field of view provided by many telescopes, whether flexible or fixed, is tilted. That is, the direction of the field of view does not coincide with the main axis of the telescope. This exacerbates the difficulty of correctly aiming the telescope to see the required field of view. It also increases the chances that the surgeon or assistant will misunderstand the image shown or lose the telescope's orientation towards the affected tissue. Physical fatigue degrades the telescope's alignment and / or the ability of the telescope to interpret the transferred image.

【0012】

There is a need for accurate and reliable information about anatomical tissue that is hidden and invisible in the body. There is also a need for a device that accurately aligns and orients surgical instruments within the body to provide accurate location information for hidden anatomical tissue. In addition, there is a need to provide an accurate interface between the surgeon and the surgical instrument used so that they can be accurately aligned with the anatomical tissue in the body without taking the instrument off. ..

【0013】

An object of the present invention is to provide an improved method of obtaining and displaying accurate information about the location of anatomical tissue in a patient.

【0014】

It is also an object of the present invention to provide an improved method for locating endoscopic cameras and other surgical instruments within a patient.

【0015】

A further object of the present invention is to provide an interface for the surgeon to accurately position the endoscopic camera and other surgical instruments within the patient without taking his hands off the instrument.

【0016】

[Means for solving problems]

The present invention is a method and an apparatus for determining the position information of a target object and using the information to align an instrument with respect to the target object. The present invention can be applied in various ways, but is particularly effective when the target is hidden from the field of view or in a position where it is difficult to approach. One of the preferred examples, which is used in endoscopic surgical procedures, can determine the location of designated anatomical tissue hidden within the patient's body. This information is used to align the surgical instrument with respect to the anatomical tissue in the body.

【0017】

In the present invention, an instrument inserted into the patient's body, such as a surgical instrument, is first aligned with the desired position with respect to the target (anatomical tissue). The instrument is capable of transmitting an image of the target to the computer, which then uses various types of image processing to determine location information about the target. The information is then given to the person (eg, surgeon) or computer that controls the robotic device. This position information is used to align or realign the transmitter or other device with respect to the specified target.

【0018】

To facilitate the use of the present invention, a number of different output modes are provided for information transmission from imaging devices and computers to operating room personnel.

【0019】

In order to facilitate the use of the present invention, an input device is incorporated into the device to be inserted into the body so that the user can input a request to the system at the same time while operating the device. Other methods of inputting requests into the system, such as voice recognition systems, can also be incorporated so that interaction with the system does not interfere with the operation of the device.

【0020】

[Example]

FIG. 1 shows a computerized augmentation system for laparoscopic and similar practice. This system typically includes a manipulator or robot 242, a computer 243, a drive monitor interface 244, a monoscopic monitor 247 and a compatible image processor 245, a graphic adapter 246, and a stereoscopic monitor 272. It consists of a compatible stereo display system 271 and a terminal 248 that connects an additional input device to the computer 243.

【0021】

A manipulator similar to the manipulator 242 used in the examples discussed herein is detailed in US Patent Application No. 714,816.

【0022】

As can be seen in FIGS. 1 and 2, the manipulator 242 is composed of a linear manipulator 6 closer to the base and a manipulator 240 at the tip having a center point of operation at a distant position. The manipulator 6 is composed of parts 1, 2, and 3 that slide at right angles to each other, and each of them moves in the X, Y, and Z directions. Parts 1, 2 and 3 are equipped with a computer-controlled monitor drive 4 connected to monitor interface 244. It also comes with a manual lock clamp 51. The manipulator 240 has a rotational motion θ._p, Θ_x, Θ_y, And rotation at a distance θ_zIt is composed of rotating parts 250, 251, 252, and a slide motor 253 attached to a shaft-shaped slide device 254. A computer-controlled motorized drive 249 mounted on these parts is connected to the motor interface 244 and is equipped with a manual lock clamp 255. Each moving part of the manipulator 242 can be activated either manually or by computer control and can be optionally locked with a manual locking device. Motorized devices 4 and 249 are controlled by computer 243 via motor interface 244.

【0023】

Figure 2 shows a rotary manipulator 240 that performs microscopic movements at the ends and a device 241 that is inserted into the patient's body through the incision. In this example, the terminal manipulator 240 has 5 degrees of freedom (θ)._p, Θ_x, Θ_y, Θ_z, D), which acts as a wrist with a central point of motion at a distance, supported by the aforementioned alignment system closer to the base, which has three degrees of freedom of linear motion orthogonal to each other. ing. Utilizing the degree of freedom of linear movement near the base, the movement center point M of the wrist portion having the movement center point at a distant position is aligned with the insertion point P into the patient's body. In addition, if the machine has a sufficient degree of freedom (SCRA manipulator, etc., manufactured and sold by IBM), it can be a substitute for the manipulator 6.

【0024】

With 4 degrees of freedom of rotation and 1 degree of freedom of sliding at the end of the manipulator 240, the surgeon has a spherical working area with 5 degrees of freedom centered on the insertion point M. Selective locking of any of these degrees of freedom or independent movement (either manually or by computer control) helps the surgeon to achieve the exact alignment needed. In addition, small movements within the work area can be obtained with slight movements in independent individual axial directions. No matter how the terminal manipulator 240 moves, its effect does not reach point M (the point at which the surgical device is inserted into the patient's body). Therefore, it is not necessary to move or increase the size of the insertion point into the patient's body in order to move this manipulator within the work area.

【0025】

One result of this design is that the manipulator 6 closer to the base does not need to be operated unless the patient is moved. Therefore, in one of the preferred embodiments, the manipulator 6 is stopped and / or the drive motor is stopped by a manual lock whenever the device is inserted into the patient's body. In this state, the control computer 243 interprets the instruction requesting the operation of the manipulator 242 as follows. When a certain operation is requested, the control computer 243 tries to meet the request only by the operation of the terminal manipulator 240. If there are several ways to achieve that movement, the movement of the rotational movement part 7 closest to the center is minimal (ie θ)._pSelect an action that minimizes the movement of. There are various ways to minimize error in the position of the tip of device 241, error in direction, and errors in their various combinations. If this error is greater than a pre-specified threshold, the control computer will notify the surgeon by synthetic voice, alarm sound or other means, and the surgeon will not explicitly instruct the surgeon to continue using voice recognition or other input methods. As long as it is stopped. In an alternative embodiment, when a surgical device supported by a straight manipulator is inserted into the patient's body, the nail is prohibited from moving the manipulator 6 from the base, and the total amount of movement of the terminal manipulator 240 is applied. The purpose is to minimize it. However, in other embodiments, the manipulator is allowed to move slightly from the base as long as the center point M of motion remains within a specified threshold distance (eg, 3 millimeters).

【0026】

If necessary, a flexible tip can be attached to the end of the device 241 to provide more freedom of movement. When using a display device such as device 254, readjustable mirrors or prisms can be attached to the ends of the device to give them the freedom to adjust the viewing direction.

【0027】

Looking back at FIG. 1, in this embodiment, the light source 277 is connected to the device 254 via the video camera 259 and the fiberscope cable 278. The video signal output of camera 259 is input to graphic adapter 246, where it can be freely mixed with graphic output from computer 243 and displayed on monitor 247. The video output of the camera is also optionally input to the image processing system 245, where it analyzes the image of the camera and sends the relative position information of the surgical device, the camera and the anatomical tissue to the computer 243. .. The video output of the camera can also be optionally input to the stereo display system 271. Here, a stereoscope image of the anatomical tissue can be assembled and displayed on the stereoscopic monitor 272 from two or more images obtained at different points.

【0028】

In one of the preferred embodiments, the stereo display system is StereoGraphics CrystalEyes (a registered trademark of StereoGraphics, Inc). In this case, 2 this video signal is displayed on the stereoscopic monitor, which alternately displays the image of the left eye and the image of the right eye at a frequency of 120 Hz, and the video information of each eye is displayed 60 times per second. It is updated one by one. The surgeon's stereoscope liquid crystal (LC) goggles 273 are synchronized with the monitor and alternately block the light coming from the left and right eyes so that the left eye receives only the video signal from the camera on the left. , The right eye receives only the video signal from the camera on the right. The frequency at which the images from the left and right alternate is high enough for the surgeon to perceive a continuous stereoscopic image of the anatomical tissue without flicker. There are other stereo display technologies, and they may be used.

【0029】

In the examples shown here, the surgeon is using a second surgical device 260 in the patient's body. It has a visible sign. These signs are marks placed on the device and are selected so that they can be easily located by the image processing system 245 in the image supplied by the camera 259.

【0030】

In FIG. 1, the set of input / output devices attached to the input / output interface 248 of computer 243 is sterilized on a computer speech recognition and synthesis system 267, a control stick 268 attached to a surgical device, and a monitor 247. Includes touch screen 269. In this embodiment, the control stick is a small device that has the same functionality as a 2D or 3D mouse. However, it is designed so that it can be attached directly to a surgical device and that pressure can be applied to the small protrusions of the stick to specify at least two degrees of freedom of movement. In one embodiment of such a device, a strain gauge is used to convert the applied pressure or force into displacement or velocity information. In another example, Spaceball (Spaceball) Using an input device with 6 degrees of freedom, such as (registered trademark of Technologies, Inc.), any operation related to any of the 6 degrees of freedom can be specified. This type of device can be mounted on a surgical device or manipulator, or in any other convenient location. One of the advantages of attaching an input device, such as a small control stick, to the surgical device is that the surgeon can easily operate the control stick without taking his or her hand off the surgical device. Therefore, the surgeon can give the computer the necessary information (eg, the direction in which he wants to move the medical telescope) without interrupting his work as a doctor.

【0031】

The speech recognition and synthesis system 267 includes means of inputting information to the system, such as the microphone 275 (preferably worn on the head), and means of transmitting information to the surgeon, such as the speaker 274. The speech recognition system 267 can understand the instructional terms spoken by the surgeon and can convey information about the received instruction to the computer 243. Surgeons may use these methods separately or in combination, thereby aligning objects on monitor 247, selecting instructions or operating modes from the menu, and instructing the operation of the manipulator 242. To do.

【0032】

FIG. 3 shows another example of a computerized augmentation system for laparoscopic surgical procedures or similar surgical procedures. In this embodiment, the surgical device 254a is a medical stereoscope camera that combines two independent lens systems or optical fibers to transmit two identical images of the patient's body together. it can. The two lenses are separated by a small known distance to give a stereoscopic image. An example of such a device consists of two parallel fiber optic bundles or lens systems and one fiber optic optical channel. This assembly is covered with a suitable cylindrical cover. The video signals from the two cameras 259a and 259b are input to the stereo display system 271 and displayed to the surgeon on the stereoscope display monitor 272. Using interface hardware known in the art, both video signals are optionally input to the image processing system 245 and the graphic adapter 246.

【0033】

Figure 4 shows another embodiment of this system, which consists of two manipulators 240a and 240b, each with surgical devices 241a and 241b. In one embodiment, one surgical device is a medical telescope and the other device is a surgical device, such as forceps. Since both devices are attached to the robotic device, they can be aligned by computer control. In this case, since the manipulator has only one arm, one or both of the robots can be manually controlled by loosening, adjusting, and relocking the joint axes one at a time. Both surgical devices 241a and 241b consist of a medical telescope or other means of transmitting images outside the patient's body. In such an embodiment, one device (eg, 241a) may also include a miniaturized surgical device such as forceps. In this case, the image information captured from the two observation points can be combined to provide accurate 3D information that helps to place the surgical device on the targeted portion of the anatomical tissue.

【0034】

See again in Figure 1, the image processing system 245 can be used to show the surgeon the structure of interest on the anatomical tissue. The surgeon points out the structure of interest by one of a number of means described below. Given the surgeon's instructions via the appropriate input device attached to terminal 248, computer 243 commands the image processing system 245 to capture the image and pinpoint the location of the indicated tissue. In one embodiment, according to the surgeon's instructions, a reference image of the indicated structure is captured and stored. Image correlation techniques are used to indicate its location throughout the surgical procedure. An alternative embodiment creates a composite reference image from the medical image obtained in the previous operation and the computer reconstruction of the model. Once the structure is located, the manipulator 242 can be moved to place the structure anywhere in the camera's field of view. If necessary, more images can be captured, the position of the structure can be repositioned, and further adjustments can be made to improve the position of the camera. This procedure is repeated many times so that the aim is determined with the accuracy required for the structure.

【0035】

I would like to touch on the terminology problem, but in the following sentences, the position in two dimensions (2D) and three dimensions (3D) is described. When writing about an image captured by a single monoscope camera, the "position in the image" is a two-dimensional position. As shown with reference to FIG. 5a, position A (in 2D image 800) is given as a set of coordinates (x, y). If the image is captured by a stereoscope, the "position in the image" is the 3D position. As shown with reference to FIG. 5b, position B is described by a set of three coordinates (x, y, z). It also mentions the location of the anatomical tissue structure. Such structures are part of the anatomical tissue within the patient's body, and all references to "structure location" are the 3D location information of the structure in question.

【0036】

In order to use and manipulate images of anatomical tissue, one must first capture the image. Explained with reference to FIG. 1, this is done by sending a raw video signal from the camera 259 to the image processing system 245, which consists of at least one video digitizer. A video digitizer is a device capable of converting an analog video signal into a digital signal, and the converted digital signal is stored in a computer memory, and the computer makes arbitrary corrections. Conversely, a video digitizer can also convert a digitized (and perhaps modified) video signal back into an analog signal for display on a standard monitor.

【0037】

If position information is extracted from the image captured by the camera / lens system, to correlate points in the image (points on the camera's image plane) with points in the corresponding space (3D position in the real environment). In addition, we need a mathematical model that represents a camera or lens. A good approximation is the pinhole system shown in Figure 6 which can fully model the camera / lens system. In the figure, a camera with a lens 600 is drawn at a distance f from the image plane 601. The distance f is called the focal length of the lens. The point W = (x, y, z) on the plane 602 at the distance d = -z in front of the lens is reflected at the position C = (x', y') on the image plane 601. = x'/ f and y / d = y'/ f hold.

【0038】

Given the coordinates (x', y') on the image for a point in real space, the above equation gives two equations for three unknowns (x, y, z). However, it is not enough to reproduce the 3D coordinates of the corresponding point W in real space. Explaining with reference to FIG. 7, the radiation 605a defined by the information obtained from one image 601a captured at the first observation point 600a is the point C on the image._aStarting from, it passes through the center of the lens 600a and extends infinitely. By definition, the point W in real space should be somewhere on this straight line, but more information is needed to determine its exact position. If the second image 601b is captured at the second observation point 600b (the position and direction with respect to the first observation point 600a is known), the corresponding point C on the second image_{b b}And the second radiation 605b are defined from the second observation point 600b, and the point W in the real space also exists on this straight line. Using a known mathematical method, two radiations are calculated at the same coordinates, and the intersection is calculated by a computer to obtain the 3D position (x, y, z) of the point W.

【0039】

Most camera lenses bring in distortion, which causes the correspondence between points in the actual section and points on the image to deviate from the pinhole model described above. Calibration of the camera / lens system allows evaluation of the nature and amount of such distortion, and the resulting mathematical model can be used to avoid distorting points on the image. In this way, the pinhole model can be applied to distortion-free images. There are many methods for calibrating a camera / lens system.

【0040】

As an example of interacting with a 2D image of anatomical tissue displayed on a monitor to a surgeon, the surgeon may want to indicate a specific position within the displayed image. The surgeon can indicate a particular position on the image by any of the means shown below. (a) Align the surgical device with a clearly visible sign. Align the image of the sign on the display with the position on the image you want to point to. (b) Manipulate an object on the screen using an input device attached to the surgical device (such as the control stick 268 or similar device in FIGS. 1 and 3). (c) Manipulate objects on the screen using a conventional mouse. Visible markers in method (a) are brightly colored dots at known locations on the device, or well-known geometric patterns consisting of such dots (eg, FIGS. 1 and 3). The pattern inside 276). The use of vibrant colors that are different from the colors originally found in the patient greatly simplifies the difficulty of locating the markers and reduces the likelihood of mistaking the location of these markers. These points on the surgical device can be located by known image processing techniques. Image processing techniques include threshold processing (separating points from the rest of the image), computer determination of the center point of the resulting region, and the like. In methods (b) and (c), the position of the anatomical tissue structure of interest is the final position of the object in the image.

【0041】

When the 2D coordinates of the position on the image are specified to the computer 243, the computer confirms the specified position by overlaying some object on the image and marking the position. As one of the specific examples of this confirmation method, a 2D crosshair cursor or a 2D box-shaped cursor can be used to indicate the position of interest in the image. The "image" referred to here may be an image of the anatomical tissue of interest from a TV camera or a computer-generated graphic image.

【0042】

So far, we have described various methods for surgeons to indicate a specific 2D position in a monoscope image. Next, a method for determining the three-dimensional anatomical tissue structure in the patient's body and the position information regarding the surgical device, such as image processing, will be described.

【0043】

With reference to FIGS. 1 and 3, consider the case where stereoscopic display (live or still image) of anatomical tissue is possible during a surgical procedure. At this time, the surgeon can point out the structure of the 3D anatomical tissue of interest by manipulating the 3D stereoscopic graphic cursor on the stereoscopic display 272. However, only if the graphic cursor matches the anatomical tissue structure. To direct the movement of the graphic cursor in space by the stereoscope in the image, use any of the appropriate input devices and method 248 (surgical device with control stick or trackball, audio, etc.) described above. You may.

【0044】

If the actual size of the indicated object is known, the distance from the observation device can be evaluated from the image of the object. As shown in Fig. 7, the structure of the anatomical tissue is known to be on the radiation passing through the observation point starting from the center of the image of the structure, so the position of the anatomical tissue structure with respect to the observation device. Can be calculated by a computer. Let l be the size of the structure on the image, s be the actual size, and f be the focal length of the camera. The distance z from the camera lens to the target structure can be calculated as z = (f × s) / l.

【0045】

In one implementation order shown in FIG. 8, a visible marker 701 on the surgical device 700 is utilized to locate a 3D structure (eg, the surgical device 700) in the following procedure. On the surgical device 700, there are at least three (not aligned) round dots 701 with a diameter of s (Fig. 8a). Surgical devices generally take any direction with respect to the camera, so these points appear as an ellipse 705 on the image plane. The diameter of the corresponding circular image when the distances of these round points from the lens are left unchanged so that the image plane is perpendicular to the direction the camera is facing is equal to the major axis l of the ellipse described above. The distance from the camera lens to the round point can be calculated by a computer from the formula z = (f × s) / l. After making this calculation for at least three round points, knowing the position of the round point pattern with respect to the tip of the surgical device would be sufficient to calculate the 3D position of the tip of the surgical device with respect to the camera. Using other methods known in the art, the position and orientation of a five-point pattern with respect to the camera can be calculated from the 2D position information of the centroids of those points on the image. Similarly, other patterns and other visual markers can be used. The 3D position of the surgical device tip with respect to the camera can then be calculated from the position of the surgical device tip with respect to a known marker.

【0046】

In addition, stereo image processing may be used to accurately locate the 3D anatomical tissue structure. As an embodiment, image processing can be used in conjunction with a stereoscope camera to determine the location of anatomical tissue structures. The surgical device 254a shown in FIG. 3 is a medical stereoscope camera, which consists of two independent lens systems or optical fibers and is capable of transmitting two images at the same time from inside the patient. As shown in FIG. 9, these lenses are separated by a small (known) distance d. The 3D position of the anatomical tissue structure with respect to the camera tip is calculated from the pinhole camera model (Fig. 6). In particular, the positions of the centers of the structures of interest on the two image planes are f, as shown in FIG.₁= (x₁, y₁), F₂= (x₂, y₂), The distance from the camera lens to the center of the structure can be calculated as z = (f × d) / c. here, [Number 1]

<img file="JP2575586B2_D0001.tif" />【0047】

And f represents the focal length of the camera. Image correlation methods or other image processing methods known in the art can also be used to locate anatomical tissue structures in images.

【0048】

Another embodiment will be described again with reference to FIG. 9, but even if only the monocular camera is used, the position of the anatomical tissue structure in three dimensions is determined by using the image processing technique according to the following procedure. can do. Capture the first image 601a of the anatomical tissue structure and store a reference display (such as a multi-resolution image pyramid display known in the field of image processing technology). Using the manipulator 242, the tip of the camera lens is moved laterally by a known distance d to obtain the second image 601b. The position of the center W of the anatomical tissue structure on the second image is determined using a reference representation of the structure and a correlation method (such as the multi-resolution normalized correlation method known in the art), as described above. The amount of 3D displacement from the camera tip of the structure in can be calculated. In particular, the position W of the center of the structure of interest on images 601a and 601b is f, respectively.₁= (x₁, y₁), F₂= (x₂, y₂), The distance from the tip of the camera to the center of the structure can be calculated as z = (f × d) / c. here, [Number 1]

<img file="JP2575586B2_D0002.tif" />【0049】

And f represents the focal length of the camera.

【0050】

In another embodiment, the telescope is moved laterally to obtain a second image, as described above, with physical binding that minimizes the movement of the telescope to the insertion slot into the patient's body. Is prevented. To explain with reference to FIG. 7, in this embodiment, first, the first image is captured from the first observation point, and the position C of the center W of the structure of interest on the image is taken._aTo decide. The telescope is then rotated slightly (by a known amount) around the insertion slot into the body. At this time, the anatomical tissue structure is still within the field of view of the telescope, and a second image 601b is obtained. Note that the position of the second observation point is also different from that of the first observation point. Position C on the second image of the center W of the structure_{b b}Is determined. Then, the 3D position of the center W of the structure can be obtained by calculating the intersection of the radiations 605a and 605b as described above. As described above, the position of the structure on the image can be determined by using the image correlation method or other known image processing method. Alternatively, the surgeon may point out the location of the center of the structure on the two images using any of the means pointing to the location of the images described above.

【0051】

Once the 3D anatomical tissue structure has been pointed out and its 3D position can be calculated, the computer 243 confirms the specified position by overlaying some stereoscopic graphic object on the image and marking the position. I take the. As one of the specific examples of this confirmation method, a 3D crosshair cursor or a 3D box-shaped cursor can be used to indicate an interesting position in the space viewed with a stereoscope. The "image" referred to here may be an image of the anatomical tissue of interest from a TV camera or a computer-generated graphic image.

【0052】

Once the 3D position of the anatomical tissue structure is stored in computer 243, this information can be used to control the position and orientation of the camera tip with respect to that structure and obtain any field of view as desired.

【0053】

See Figure 1. As one method, the surgeon can indicate the first and second 2D positions in the image using any of the aforementioned means of pointing out the 2D position. The surgeon commanded the manipulator 242 (using the appropriate input device described above) to reposition the camera tip 266, which was in the first 2D position before the camera was moved.₁Make sure that the image of is in the second 2D position after moving the camera. Anatomical tissue structure f₁The distance from the camera tip 266 to the camera tip 266 is kept constant during this operation. As a special case of this method, the second 2D position may be in the center of the image. In this case, reposition the camera so that the anatomical tissue structure is centered on the displayed image. That is, the camera is centered on the anatomical tissue structure.

【0054】

Alternatively, the surgeon specifies a series of 2D positions in the image, commands the manipulator 242 to move the camera tip 266 while holding it at a certain height, and is defined by the series of 2D positions in the image. Make the camera follow the trajectory. In one embodiment, this series of 2D positions is the position of an image of a small anatomical tissue structure that stands out in the field of view of the camera. In other examples, this series of 2D positions corresponds to the position of the image of the boundary of a large anatomical tissue structure, such as a blood vessel. This camera realignment specifies the desired apparent movement of the anatomical tissue structure with respect to the image (corresponding to the last position in the specified set of 2D positions). The phrase "apparent movement of the anatomical tissue structure" emphasizes that the anatomical tissue structure does not physically move, but appears to move relative to the image due to the movement of the camera. To do. Execution of this method proceeds as follows. Computer 243 processes the position of a series of 2D images into a continuous trajectory by interpolation. The camera is then centered to the anatomical tissue structure that corresponds to the position of the first designated 2D image, as described in the previous paragraph. The camera is further repeatedly and aligned in the field of view so that it is centered on each of the 2D positions determined by continuous interpolation, and actually follows the trajectory defined by the surgeon. The surgeon directly controls the direction and speed of movement of the camera by means of a control stick or other suitable input means attached to the surgical device.

【0055】

In yet another method, the surgeon specifies an increment of movement along the camera's visual axis and realigns the camera along this axis by a specified distance. The "visual axis" here is a straight line connecting the center of the camera lens and the point p on the anatomical tissue visible at the center of the image of the camera. This method realizes a zoom function for 3D anatomical tissue structure, and the zoom factor (the required enlargement or reduction ratio of the image of the anatomical tissue structure) is interactively performed by the surgeon. specify. To achieve this method, in particular, the surgeon should be able to interact with the graphic cursors on the screen. The surgeon specifies the desired zoom factor by zooming in or out on the cursor with respect to the reference cursor. The size of the reference cursor does not change while the zoom factor is specified. Any suitable input device 248 is used for this cursor operation. Computer 243 then uses the relative positional relationship of these two cursors to calculate the direction and magnitude of the increment of camera movement. This is the amount required to realize the zoom function specified by the zoom factor. When the increment of camera movement is calculated, the computer 243 commands the manipulator 242 to move the camera tip 266 along the visual axis by the calculated amount (slowly) to achieve the desired zoom factor. Note that the previously defined point p remains in the center of the image during the zoom process.

【0056】

In yet another method, the surgeon can control the direction of movement of the camera's observation point directly from a device attached to the input device. In this embodiment, the input device is a control stick with 6 degrees of freedom. With such a control stick, the surgeon can simultaneously reposition and reorient the camera for all six degrees of freedom. You can choose different combinations from all 6 degrees of freedom to achieve a number of useful control modes. In particular, in the camera motion control mode that can be realized when the control capability of the control stick with 6 degrees of freedom becomes ineffective or when only the input device with 3 degrees of freedom is available, the movement of the camera tip is currently targeted. It is limited to a virtual spherical surface centered on the anatomical tissue structure and whose radius is the distance from that structure to the current position of the camera tip. In another embodiment where only an input device with two degrees of freedom is available, the device can control any two of the six degrees of freedom at any time. For example, pushing down a control stick with 2 degrees of freedom toward the tip of the device to which it is attached means "zoom in", and pushing it up in the opposite direction to "zoom out". Can mean. When you take your hand off the control stick, it means "stop". Similarly, when pressure or force is applied to the control stick with 2 degrees of freedom in the direction perpendicular to the long axis of the camera, the computer 243 keeps the camera at its current height and the direction of the applied pressure. It is interpreted to mean moving to the side. Furthermore, it is possible to make the speed of movement of the camera proportional to the magnitude of the pressure applied to the control stick.

【0057】

In another embodiment, the surgeon can manipulate a graphic object overlaid on an image of the anatomical tissue to specify the desired field of view containing the tissue structure of interest. The camera is then automatically aligned to provide the desired field of view. An example of realizing this method is as follows. First, an image of the anatomical tissue is captured and displayed on a monitor for the surgeon. The surgeon then points out the tissue structure of interest on a 2D or 3D image. However, this is limited to cases where the structure has not been pointed out and is not often seen. The surgeon can then interactively manipulate the graphic objects (cursor, slider, etc.) overlaid on the image of the anatomical tissue to specify the desired field of view containing the tissue structure of interest. For example, by specifying this field of view, the observation point of the camera can be specified at any point on the sphere with a given radius centered on the structure of interest. The computer 243 then calculates the appropriate displacement of the camera and commands the manipulator 242 to move the camera to obtain the desired field of view with the tissue structure of interest.

【0058】

If the surgical augmentation system consists of two independently controlled robotic systems, as shown in Figure 4, the surgical device is repositioned using 3D position information about the anatomical tissue structure. There is a way to match. In this case, the device to be realigned is not a surgical telescope, but a second surgical device. In one embodiment of the invention, the second surgical device may be surgical forceps, the jaw of which matches the current 3D anatomical tissue structure, and when the jaw is closed, the tissue of that structure. Aligned so that specimens can be taken.

【0059】

See Figure 10. In situations where the telescope captures a side view rather than straight forward (α = 0 °), it is especially important to specify a view that contains the particular 3D structure of interest and give the computer a new medical telescope. It is the ability to calculate various positions. Laparoscopic and similar surgical procedures typically use a telescope with a directional view in the range of 30 ° to 135 ° (relative to the long axis of the device). The telescope shown in FIG. 10 has a field of view α = 45 °. Manually aligning such a telescope to obtain the desired field of view is very difficult even for a skilled camera operator. Coordinate transformations between telescopes, anatomical tissues, and images are complex and unintuitive. However, by adding one rigid transformation to the calculation in the computer software, the direction of the field of view is not 0 °. In one particular implementation, the coordinate type F_cCombined with a telescope with a 0 ° field of view, the computer keeps track of rigid transformations between manipulators, cameras, and various anatomical tissue structures. Mathematical methods of rigid transformation are well known in the fields of robotics and computer graphics. The movement of the camera required to achieve a particular zoom factor is the coordinate system F of this camera._cCalculated with respect to. For non-straight telescopes, as shown in Figure 10, the new coordinate system F_{c ^}Is the coordinate system F_cIs defined by rotating -α with respect to a straight line passing through the tip of the lens and parallel to the X axis of Fc. New camera coordinate system F_{c ^}The original 0 ° camera coordinate system F_cRigid transformation related to^cT_{c ^}Is used to solve the problem that the direction of the field of view is not 0 °. Coordinate system F when calculating the new position of the telescope to obtain a specific field of view_{c ^}F_cAs a result of using instead of, correct realignment is possible regardless of the viewing direction of the telescope.

【0060】

There are a number of ways to display to the surgeon the visual information transmitted from the patient's body and freely augmented by image processing and computer graphics.

【0061】

See Figure 1. In one way to display information, an image of the anatomical tissue is presented to the surgeon as a combination of a raw image and a still image (a raw image is an image captured by a camera that gives new information. It keeps updating one after another, while still images are not updated). In one embodiment of this method, the image displayed on the monoscopic monitor 247 is generated as follows. A wide-angle monoscope image of the anatomical tissue is captured using the surgical device 254 and displayed as a still image on the monitor 247. The camera then zooms in to get a closer view of the structure of the tissue of interest and overlays a portion of this live TV image on top of the still wide-angle image. Therefore, the static monoscopic field of view of the entire area of interest provides information about the surroundings of the anatomical tissue being observed, and the raw image magnifies the perimeter of the structure of current interest. And show you in detail.

【0062】

In another embodiment of this display method, the static wide-angle image information is a computer graphic display of the anatomical tissue. This graphic information was created from a computer model of anatomical tissue constructed based on information collected by prior imaging and scanning examinations. As in the precedent, some of the images around the anatomical tissue structure of current interest are replaced with live TV images that magnify this area. At this time, the computer-generated image and the actual live TV image must be combined and superimposed on one display image. Numerous techniques for aligning and superimposing images are known in the art. The simplest implementation is to identify the 3D positions of some known tissues in a computer model using 3D image processing techniques. The 3D positions of these landmarks can then be used to computationally obtain a perspective view suitable for displaying the graphic model.

【0063】

In yet another embodiment of this display method, the static wide-angle image information is a computer graphic display of the anatomical tissue, as in the example above. Similarly, some of the images around the anatomical tissue structure of current interest are replaced with live TV images that magnify this area. Further, the raw TV image of the detail part can be augmented by superimposing stationary edge information. This edge information can be obtained both from a computer graphic model and as a result of image processing (edge detection) of a TV image. The advantage of this display method is that by overlaying the edges, the ongoing changes in the details in the live TV image are emphasized relative to the previous (stationary) situation of that part. ..

【0064】

In another embodiment of the method of displaying information to the surgeon, a static wide-angle field of view of the entire region of interest can be displayed as a static stereoscope image. With reference to FIG. 1, it is shown that this method can be realized as follows. Using the surgical device 254 and camera 259, a still image of the entire region of interest is captured from the first observation point. Next, the camera tip 266 is displaced by a known minute amount, and the second still image of the region of interest is captured from the observation point after the displacement. These two images are input to the stereo display system 271 and displayed on a stereoscopic monitor as a static stereoscope wide-angle image of the entire area of interest. If only the terminal manipulator 240 moves to displace the camera, there can be slight angular discrepancies between the two images obtained by the method described above. Experiments have shown that such discrepancies are negligible, as the human visual system is so skillfully fused with slightly misaligned images. Alternatively, image conversion methods known in the art can be used to significantly compensate for this discrepancy. Then, as mentioned above, the camera zooms in to get a closer view of the structure of interest, and a portion of the still wide-angle image is a magnified raw monoscope image of the anatomical tissue structure of interest. Replace with. In the resulting image, the overall surrounding information is a static stereoscope image, giving the surgeon an overall three-dimensional relationship in the field of view. On the other hand, the area around the anatomical tissue structure of interest, where the surgeon's attention is focused, is a magnified and displayed monoscope image.

【0065】

In the proposed modification of the display method described above, the raw TV image of the detail portion can be augmented by superimposing stationary edge information. This edge information can be obtained both from a computer graphic model and as a result of image processing (edge detection) of a TV image. As mentioned earlier, the advantage of this display method is that by overlaying the edges, the ongoing changes in the details in the live TV image are emphasized over the previous (stationary) situation of that part. It means that.

【0066】

See Figure 3. In another embodiment of the present invention relating to the display of visual information to a surgeon, a stereoscope camera 254a is used to capture a static stereoscope wide-angle image of the entire area of anatomical tissue of interest. As mentioned above, the camera zooms in to get a closer view of the structure of interest and replaces part of the still wide-angle image with a magnified raw stereoscopic image of the anatomical tissue structure of interest. This TV image is transmitted from inside the patient by cameras 259a and 259b.

【0067】

To emphasize the changes that occur in the detail portion, the edge information corresponding to the previous state of that portion is overlaid on the raw stereoscope image, as described above.

【0068】

See Figure 3 again. In another embodiment of the invention, a stereoscope medical camera 254a is used with stereoscopic computer graphics to display anatomical tissue. In this example, a static stereoscopic image of the entire area of anatomical tissue of interest is created from a computer model of that tissue and 3D stereoscopic on monitor 272 via the stereo display system 271. It is displayed as a graphic image. Then, as mentioned above, the stereoscope camera zooms in to get a closer view of the 3D structure of interest and magnifies the area around the structure of interest in the still graphic image. Replace with TV image. This TV image is transmitted from inside the patient by cameras 259a and 259b.

【0069】

To emphasize the changes that occur in the detail portion, the edge information corresponding to the previous state of that portion is overlaid on the raw stereoscope image, as described above.

【0070】

See Figure 1. The method of displaying anatomical tissue information to the surgeon uses a monoscope camera 254 to provide the surgeon with a raw stereoscopic image of the patient's anatomical tissue. In this method, the information provided to the surgeon's eye is obtained from a computer model of the anatomical tissue and is laterally displaced by a known small distance from the current observation point of the surgical device 254. It is a graphic image calculated by. The information supplied to the other eye is a raw image of the anatomical tissue, obtained by camera 259 attached to surgical device 254. Therefore, in this method, one eye receives a computer-generated image of the patient's body, and one eye receives a raw image transmitted by the camera from a slightly displaced observation point. When the computer graphic model is properly superimposed on the actual anatomical tissue, the human brain fuses the two images into the correct 3D stereoscope image.

【0071】

Another embodiment of the method of displaying anatomical tissue information to the surgeon uses image processing with raw video information to give the surgeon a raw stereoscopic display. Referring to FIG. 1, in this embodiment, the first image of the anatomical tissue is captured and transmitted to the image processing system 245. Next, the tip 266 of the camera is displaced laterally by a known minute amount, the second image is captured from the second observation point, and the second image is transferred to the image processing system 245. Using this image processing system and a known image processing method, edge information is detected from two images. For the stereo display system 271, one input channel (for left / right eye) is given only edge information, and the other channel (for right / left eye) is given a raw video signal with edge information, and is stereo. Generate a scope display. Subsequently, only the information for one eye is updated with the raw video information transmitted by the camera 259. This allows the human brain to provide enough information to make up for the missing information and interpret the image as a correct stereoscope 3D image.

【0072】

Alternatively, the above-mentioned display method can be used even when the edge information is obtained from a computer graphic model of the anatomical tissue instead of being obtained by image processing.

【0073】

Apart from visual information, the surgeon can also receive non-visual information about the location of the tissue and the overall state of the system. One of the non-visual channels that connects the surgeon to the system is the speech recognition and speech synthesis subsystem (267 in Figure 1). For example, the system can emit a synthetic voice message to inform the surgeon of the exact location of the surgical device with respect to the anatomical tissue structure of interest. Similarly, a synthetic sound message can confirm that a voice command has been received and ensure that the surgeon has correctly interpreted his command. The overall state of the system and its changes can also be communicated to the surgeon using synthetic speech. As an example of this, a synthetic speech message could tell the surgeon the exact distance traveled by the camera during the zooming operation.

【0074】

Tactile feedback is an alternative way to convey non-visual information to the surgeon. In one embodiment, the surgeon is interested in a graphic object or surgical device by giving feedback to the surgeon's tactile sensation through an input device (such as a control stick) that is held in the hand or attached to the instrument. It can inform that it is aligned near a certain anatomical tissue structure. To cause the tactile degree back to the surgeon's fingers (those touching the control stick), the control stick can be fitted with a computer-controlled vibrator. When the vibrator is activated by the computer, the control stick begins to vibrate at the appropriate frequency and amplitude. Its frequency is easily noticed by the surgeon, but does not interfere with or affect his alignment work.

【0075】

[Effect of the invention]

INDUSTRIAL APPLICABILITY According to the present invention, accurate information can be obtained and displayed regarding the position of anatomical tissue in a patient's body. It also allows the surgeon to accurately position the endoscopic camera and other surgical instruments within the patient without taking his hands off the instrument.

[Simple explanation of drawings]

[Figure 1]

FIG. 5 is a diagram of a system used for computer augmentation of surgical procedures.

[Figure 2]

FIG. 1 is a detailed view showing a rotatable manipulator at the end that makes micro-movements.

[Fig. 3]

It is a figure which shows one Example of this invention using a stereoscope image system.

[Fig. 4]

It is a figure which shows one Example of this invention composed of two robotic manipulators.

[Fig. 5]

It is a figure explaining the position in a 2D and 3D Cartesian coordinate system.

[Fig. 6]

It is a figure which shows the pinhole model which represents a camera.

[Fig. 7]

It is a figure which shows the method of calculating a three-dimensional position by capturing an image from two non-parallel observation points.

[Fig. 8]

It is a figure which shows the method of determining the position of a surgical device using a visual marker.

[Fig. 9]

It is a figure which shows the method of calculating a three-dimensional position by capturing an image from two parallel observation points.

[Fig. 10]

It is a figure which shows the usage method of the medical telescope facing diagonally.

[Explanation of symbols]

4 Monitor drive 6 Manipulator 240 manipulator 240a manipulator 240b manipulator 241 Surgical device inserted into the body 241a Surgical device inserted into the body 241b Surgical device inserted into the body 242 Manipulator device or robot 243 computer 244 Drive monitor interface 245 image processor 246 graphics adapter 247 Monoscorpic Monitor 248 terminal 251 Manual lock clamp 253 slide motor 254 Slide device 255 Manual lock clamp 259 camera 260 Second surgical device 266 Camera tip 267 Speech recognition and synthesis system 268 Control stick 269 touch screen 271 Stereo display system 272 Stereo Scorpic Monitor 272 276 Geometric pattern 278 Fiberscope cable 277 Light source 600 image plane 601 lens 700 surgical equipment 701 sign 705 Statue of sign 701