System for providing visual guidance for steering a tip of an endoscopic device towards one or more landmarks and assisting an operator in endoscopic navigation

FIELD OF THE INVENTION

The present invention generally relates to steerable endoscopes and in particular, to a method and system for providing visual guidance for steering a tip of an endoscopic device towards one or more landmarks and assisting an operator in endoscopic navigation.

BACKGROUND OF THE INVENTION

Minimally invasive surgical procedures are known to offer many benefits over traditional open surgery techniques, including less pain, shorter hospital stays, quicker return to normal activities, minimal scarring, reduced recovery time, and less injury to tissue. To perform such procedures, entry into the patient may be made through an incision or a natural body orifice. Both robotic and manually operated minimally invasive surgical devices may be used.

One example of a minimally invasive surgical robotic system is the da Vinci® Surgical System from Intuitive Surgical, Inc., of Sunnyvale, California. The da Vinci® Surgical System has a number of robotic arms that move attached medical devices, such as an image capturing device and Intuitive Surgical's proprietary EndoWrist® articulating surgical instruments, in response to movement of input devices by a surgeon viewing images captured by the image capturing device at a surgical site. Each of the medical devices may be inserted through its own minimally invasive incision into the patient and positioned to perform a medical procedure at the surgical site. The incisions are placed about the patient's body so that the surgical instruments may be used to cooperatively perform the medical procedure and the image capturing device may view it without their robotic arms colliding during the procedure.

An endoscope is a medical device that allows physicians to capture images of and diagnose problems with internal body organs by inserting the device either through a natural orifice or a surgeon created opening and guiding it to a target site within a patient. In some cases, it may also be used to perform medical procedures on the internal body organs. It may be steerable so that its distal tip is controllably oriented for navigation purposes. An image capturing device such as a stereoscopic or monoscopic camera may be provided at its distal tip so that images captured by the camera from that perspective may be viewed on a display screen by the surgeon. To perform various medical procedures at the target site, surgical tools, such as those used for cutting, grasping, cauterizing, etc., may extend out of the endoscope's distal tip.

The endoscope may be rigid such as those used in laparoscopy or it may be flexible so that it is capable of following the curvatures of body lumens. It may also be rigidizable and/or robotic. A rigidizable endoscope is an endoscope that has at least one section of its flexible body that can be made substantially rigid by a mechanical locking mechanism. A robotic endoscope is a flexible endoscope that has at least one section that bends under a computer controlled servo mechanism. It may also be a capsule like the EndoCapsule by Olympus or a tethered capsule in which case it is not controllably oriented for navigation purposes and gets moved in the patient anatomy by gravity, forces applied by the anatomy or by other surgical devices.

Natural Orifice Transluminal Endoscopic Surgery ("NOTES") may employ a steerable endoscope to perform surgical procedures on patients. As an example, a flexible endoscope may be guided through one of the body's orifices and enter the abdomen from the inside of the patient, rather than through a minimally invasive incision from the outside. For example, in "transgastric" surgery, instruments are passed through the mouth and into the stomach. A gastrotomy is then performed so that the instruments may enter the abdomen and be used by the surgeon to perform a medical procedure within the abdominal cavity. Once the procedure is completed, the instruments are withdrawn along with any tissue removed during the procedure, and the entry point is closed back up. Because no incisions are made in the patient to accommodate entry of the endoscope, NOTES may be even less painful than surgery using minimally invasive incisions. Also, since it uses a natural body orifice instead of incisions to enter the body, it may result in reduced needs for general anesthetics and faster recovery times.

During the operation of a steerable endoscope, such as in a NOTES application, the endoscope tip may be turned multiple times and in different directions while moving towards a target site. As a consequence, the flexible endoscope may wind up looping around itself and disorienting the operator so as to make it difficult for the operator to keep track of the current direction of the endoscope tip if its captured image fails to clearly indicate its current direction with respect to the target site. In contrast, the operator may have a relatively rigid connection to the view in conventional laparoscopy.

If the operator accidentally moves the endoscope tip in the wrong direction, the tip may inadvertently perforate or otherwise damage tissue causing harm to the patient. Even if such harm is avoided by carefully moving the endoscope tip, additional time is required to repeatedly ascertain the true direction of the endoscope relative to the target site in the patient. Thus, the time required to perform the procedure is lengthened which adds to the cost of the surgery and increases health safety concerns.

OBJECTS AND SUMMARY OF THE INVENTION

Accordingly, an object of one or more aspects of the present invention is a method, and a system for performing the method, of providing visual guidance to an operator for steering an endoscopic device towards one or more landmarks in a patient.

Another object of one or more aspects of the present invention is a method, and a system for performing the method, of providing navigation guidance for assisting an operator in steering an endoscope towards a site in a patient.

Another object of one or more aspects of the present invention is a method, and a system for performing the method, of providing information to an operator of an endoscope that visually indicates the direction that the endoscope tip is currently pointing towards.

Still another object of one or more aspects of the present invention is a method, and a system for performing the method, of providing information to an operator of an endoscope that visually indicates the current position and shape of the endoscope relative to the anatomy of a patient.

These and additional objects are accomplished by the various aspects of the present invention, wherein briefly stated, one aspect is a computer implemented method for providing guidance to steer an endoscopic device towards one or more landmarks, the method comprising: determining a current position of a reference point on the endoscopic device relative to a reference frame; determining a first vector from the current position of the reference point to a position of a first landmark relative to the reference frame; transforming the first vector from the reference frame to an image reference frame that corresponds to the perspective of an image capturing device disposed at a tip of the endoscopic device; and displaying a graphical representation of the transformed first vector along with a current image captured by the image capturing device on a display screen so as to indicate a direction to steer the tip of the endoscopic device towards the first landmark.

Another aspect is a system comprising: a display screen; an endoscopic device; and one or more processors adapted to determine a current position of a reference point on the endoscopic device relative to a reference frame, determine a first vector from the current position of the reference point to a position of a first landmark relative to the reference frame, transform the first vector from the reference frame to an image reference frame that corresponds to the perspective of an image capturing device disposed at a tip of the endoscopic device, and display a graphical representation of the transformed first vector along with a current image captured by the image capturing device on a display screen so as to indicate a direction to steer the tip of the endoscopic device towards the first landmark.

Another aspect is a computer implemented method for assisting an operator in endoscopic navigation, the method comprising: referencing a computer model of a patient to a reference frame; determining a current position and shape of an endoscopic device relative to the reference frame; generating a computer model of the endoscopic device according to the determined current position and shape; and displaying the computer models of the endoscopic device and patient on a display screen so as to provide the endoscopic navigation assistance to the operator.

Another aspect is a system comprising: a display screen; an endoscopic device; and a processor adapted to reference a computer model of a patient to a reference frame, determine a current position and shape of the endoscopic device relative to the reference frame, generate a computer model of the endoscopic device according to the determined current position and shape, and display the computer models of the endoscopic device and patient on the display screen so as to provide navigation assistance to an operator of the system.

Additional objects, features and advantages of the various aspects of the present invention will become apparent from the following description of its preferred embodiment, which description should be taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

<ul id="ul0001" list-style="none"><li><figref idref="f0001"><b>FIG. 1</b></figref> illustrates a steerable endoscope system utilizing aspects of the present invention.</li><li><figref idref="f0002"><b>FIG. 2</b></figref> illustrates a block diagram of a processing portion of a steerable endoscope system utilizing aspects of the present invention.</li><li><figref idref="f0002"><b>FIG. 3</b></figref> illustrates a block diagram of a teleoperative portion of a steerable endoscope system utilizing aspects of the present invention.</li><li><figref idref="f0003"><b>FIG. 4</b></figref> illustrates a flow diagram of a computer implemented method for assisting an operator in endoscopic navigation, utilizing aspects of the present invention.</li><li><figref idref="f0002"><b>FIG. 5</b></figref> illustrates parts of a position determining portion of a steerable endoscope system utilizing aspects of the present invention.</li><li><figref idref="f0004"><b>FIG. 6</b></figref> illustrates a primary display screen displaying an image captured by a steerable endoscope as viewed in a system utilizing aspects of the present invention.</li><li><figref idref="f0004"><b>FIG. 7</b></figref> illustrates an auxiliary display screen displaying an anterior-posterior view of registered patient and endoscopic device models as generated by a system utilizing aspects of the present invention.</li><li><figref idref="f0005"><b>FIG. 8</b></figref> illustrates an auxiliary display screen displaying an oblique view of patient and endoscopic device models as generated by a system utilizing aspects of the present invention.</li><li><figref idref="f0005"><b>FIG. 9</b></figref> illustrates an auxiliary display screen displaying a lateral-medial view of patient and endoscopic device models as generated by a system utilizing aspects of the present invention.</li><li><figref idref="f0006"><b>FIG. 10</b></figref> illustrates a primary display screen displaying an image captured by an endoscopic device in a main window area and an oblique view of patient and endoscopic device models in a picture-in-picture window area as generated by a system utilizing aspects of the present invention.</li><li><figref idref="f0006"><b>FIG. 11</b></figref> illustrates a primary display screen displaying an image captured by an endoscopic device in a first area and an oblique view of patient and endoscopic device models in a second area as generated by a system utilizing aspects of the present invention.</li><li><figref idref="f0007"><b>FIG. 12</b></figref> illustrates a flow diagram of a computer implemented method of providing directional guidance to maneuver an endoscopic device to one or more landmarks, utilizing aspects of the present invention.</li><li><figref idref="f0007"><b>FIG. 13</b></figref> illustrates a flow diagram of a computer implemented method for establishing landmarks using an endoscopic device as it moves by the landmarks, utilizing aspects of the present invention.</li><li><figref idref="f0008"><b>FIG. 14</b></figref> illustrates an oblique view of a patient torso with an endoscopic device inserted therein and examples of various landmarks as employed in a system utilizing aspects of the present invention.</li><li><figref idref="f0008"><b>FIG. 15</b></figref> illustrates a primary display screen displaying a current image captured by an endoscopic device and graphical representations of various vectors indicating directions to corresponding landmarks along with an image associated with one of the landmarks, according to a method utilizing aspects of the present invention.</li></ul>

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

<figref idref="f0001"><b>FIG. 1</b></figref> illustrates, as an example, a system <b>100</b> including a steerable endoscope <b>110</b>, one or more fiber optic cables <b>120</b> inserted in the endoscope <b>110</b>, a position processor <b>130</b>, an image processor <b>140</b>, an image capturing device <b>141</b>, a display processor <b>150</b>, a primary display screen <b>600</b>, and an auxiliary display screen <b>160</b>. Although shown as separate units, the position processor <b>130</b>, image processor <b>140</b> and display processor <b>150</b> may each be implemented as hardware, firmware, software or a combination thereof, which interact with or are otherwise executed by one or more computers, such as the one or more computers <b>200</b> depicted in <figref idref="f0002"><b>FIG. 2</b></figref>. The primary and auxiliary display screens, <b>600</b> and <b>160</b>, are preferably computer monitors capable of displaying three-dimensional images to an operator of the system <b>100</b>. However, for cost considerations, either or both of the primary and auxiliary display screens, <b>600</b> and <b>160</b>, may be a standard computer monitor capable of only displaying two-dimensional images.

The endoscope <b>110</b>, in this example, has a flexible body <b>114</b>, a steerable tip <b>112</b> at its distal end <b>111</b> and a handle or electromechanical interface <b>116</b> at its proximal end <b>115</b>. Control cables (not shown) or other control means typically extend from the handle or electromechanical interface <b>116</b> to the steerable tip <b>112</b> so that the tip <b>112</b> may be controllably bent or turned as shown for example by dotted line versions of the bent tip <b>112</b>. Although a steerable endoscope is described in this example, the present invention is not to be limited to endoscopes of this type and may be practiced with other endoscopic devices (such as rigid, rigidizable, robotic, or capsule endoscopes) as well.

A stereoscopic or monoscopic camera <b>141</b> is provided at the distal end <b>111</b> for capturing images that are transmitted to and processed by the image processor <b>140</b> and/or display processor <b>150</b> and displayed on the primary display screen <b>600</b> and/or auxiliary display screen <b>160</b> according to the various aspects of the invention as described herein. One of a plurality of fiber optic cables <b>120</b> may be coupled at its proximal end to a light source (not shown) for illumination purposes at the distal end <b>111</b>. Others of the fiber optic cables <b>120</b> may be configured with bend or shape sensors such as Fiber Bragg Gratings (or other strain sensors such as those employing Rayleigh scattering) so that light passing through the fiber optic cable is processed by the position processor <b>130</b> to determine a current position and shape of the endoscope <b>110</b> including the orientation of its distal tip <b>112</b>. In addition to the fiber optic cables <b>120</b> extending through the endoscope <b>110</b>, one or more additional fiber optic cables (not shown) configured with strain sensors may be attached to the endoscope <b>110</b> so as to provide position information of the endoscope <b>110</b> at the attachment point.

When the steerable endoscope <b>110</b> is manipulated manually by an operator, a handle <b>116</b> is used with appropriate levers or other control mechanism(s) for controllably orienting the endoscope tip <b>112</b> by pulling appropriate cables (not shown) which couple such control mechanisms of the handle <b>116</b> to the tip <b>112.</b> On the other hand, when the steerable endoscope <b>110</b> is manipulated teleoperatively by an operator, an electromechanical interface <b>116</b> is used instead of a handle.

As shown in <figref idref="f0002"><b>FIG. 3</b></figref>, the electromechanical interface <b>116</b> allows the endoscope <b>110</b> to be electrically and mechanically coupled to a robot arm <b>301</b> whose movement is controlled by a controller <b>302</b> in response to operator manipulation of an input device <b>303</b>. Either or both the interface <b>116</b> and the robot arm <b>301</b> include motors that drive the cables used to steer the endoscope tip <b>112</b>. In addition to controlling bending of the endoscope's tip <b>112</b>, the robot arm <b>301</b> may also be configured to insert/retract the endoscope <b>110</b> into and out of an aperture (e.g., natural orifice or minimally invasive incision) in the patient, rotate the endoscope <b>110</b> about its central axis, and/or rotate the electromechanical interface <b>116</b> about a pivot point at the aperture. The controller <b>302</b> is preferably implemented as hardware, firmware or software (or a combination thereof) in the one or more computers <b>200</b> along with the processors <b>130</b>, <b>140</b>, <b>150</b>.

Details on the determination of the endoscope's position and bending using Fiber Bragg Gratings may be found, for examples, in <patcit id="pcit0001" dnum="US20070156019A1"><text>U.S. 2007/0156019 A1</text></patcit> entitled "Robotic Surgery System Including Position Sensors Using Fiber Bragg Gratings", <patcit id="pcit0002" dnum="US20080212082A1"><text>U.S. 2008/0212082 A1</text></patcit> entitled "Fiber Optic Position and/or Shape Sensing Based on Rayleigh Scatter", <patcit id="pcit0003" dnum="US20080218770A1"><text>U.S. 2008/0218770 A1</text></patcit> entitled "Robotic Surgical Instrument and Methods using Bragg Fiber Sensors", and <patcit id="pcit0004" dnum="US12164829B"><text>U.S. Applic. Serial No. 12/164,829</text></patcit> entitled "Fiber Optic Shape Sensor, each of which is incorporated herein by reference. Details on a conventional steerable endoscope may be found for example in <patcit id="pcit0005" dnum="US6869396B2"><text>U.S. 6,869,396 B2</text></patcit> entitled "Steerable Endoscope and Improved Method of Insertion".

<figref idref="f0003"><b>FIG. 4</b></figref> illustrates, as an example, a flow diagram of a computer implemented method <b>400</b> for assisting an operator in endoscopic navigation of the steerable endoscope <b>110</b> towards a site in a patient. The computer in this case is preferably the one or more computers <b>200</b> that include the processors <b>130</b>, <b>140</b>, <b>150</b>.

In <b>401</b>, a patient computer model stored in memory <b>155</b> is referenced to a fixed reference frame (i.e., one that does not move during the medical procedure to be performed using the endoscope <b>110</b>) by first referencing points on a live patient who is generally prone on an operating table with the fixed reference frame, then associating the referenced points with corresponding points on the patient computer model.

For example, using a pointer device whose tip position may be readily determined in the fixed reference frame (such as the endoscope's tip <b>112</b>), the operator may move the pointer device so that it first touches the mouth of the patient to establish a first reference point corresponding to the mouth of the patient computer model. The operator may then move the pointer device so that it next touches the right and left crest of the iliac of the patient to establish a second reference point corresponding to the right and left crest of the iliac of the patient computer model. Other points such as the navel and the left and right arm pits may also be used to establish more reference points. By referencing the patient computer model to the patient in this manner, the proper size (i.e., length and width) of the patient computer model may also be established as well as its alignment with the patient and its position in the fixed reference frame. Thus, with just a few points, the patient computer model may be properly scaled and aligned (i.e., conformed) with the patient using model information in an anthropometry database in the memory <b>155</b>.

Several patient computer models may be stored in memory <b>155</b>. For example, a number of generic patient computer models may be stored so that a different model may be used depending on the age, sex, size, etc., of the patient. Preferably, each of the models is a three-dimensional computer model including at least a head and torso (such as seen in <figref idref="f0004 f0005"><b>FIGS. 7-9</b></figref>) so that all natural orifices are included as well as potential target sites for performing diagnostic or surgical procedures on a patient. Three-dimensional computer models of the actual patient may also be stored and used with the present system if available, such as may be generated using a Computed Tomography (CT) scan. Alternatively, the surgeon or assistant may move the distal tip <b>112</b> over the surface of the patient's body to map its profile and provide sufficient information to the position processor <b>130</b> to generate information for a three-dimensional computer model of the patient which it stores in the memory <b>155</b>.

In <b>402</b>, a determination is made whether or not the operator has issued a start guidance command. The operator may issue the start guidance command in any one of a number of ways such as depressing an input button or flipping a switch on the input device <b>303</b> when the steerable endoscope <b>110</b> is being controlled teleoperatively, or depressing an input button or flipping a switch on the handle <b>116</b> when the steerable endoscope <b>110</b> is being controlled manually. Other ways may include voice actuation using a microphone and voice recognition processor (not shown) that may be implemented in or operatively coupled to the one or more computers <b>200</b>; a foot pedal (not shown) that may be depressed by the operator so as to turn on or off the guidance; and a graphical user interface using the auxiliary display screen <b>160</b>, the primary display screen <b>600</b> or another display screen (not shown) that the operator may interact with using an input device.

If the determination in <b>402</b> is NO (i.e., navigation guidance mode has not been initiated by the operator), then the method loops through <b>402</b> each process cycle (or other programmed cycle) until a YES determination is made, at which time, the method proceeds to <b>403</b>.

In <b>403</b>, the current position and shape (e.g., bending) of the endoscope <b>110</b> is determined by the position processor <b>130</b> using information received from one or more optical fibers configured with shape sensors that has been inserted in the flexible body <b>114</b> of the endoscope <b>110</b>. As an example of one such optical fiber, referring to <figref idref="f0002"><b>FIG. 5</b></figref>, the distal end of an optical fiber <b>501</b> extends to the endoscope's tip <b>112</b> and its proximal end is attached to a base point <b>502</b> whose position in the fixed reference frame is known or readily determinable by the position processor <b>130</b>. Preferably, the position processor <b>130</b> is also attached or otherwise coupled to the base point <b>502</b> so that the processor <b>130</b> may receive information from the bend sensors of the optical fiber <b>501</b> at that point.

Since the length of the fiber optic cable <b>501</b> is known, its proximal end is fixed to a known location, and its distal end extends out to the endoscope's tip <b>112</b>, the current position and orientation of the tip <b>112</b> in the fixed reference frame may be readily determined from the current bending of the optical fiber <b>501</b> as indicated by information received from the bend sensors. Although only one fiber optic cable <b>501</b> is shown for illustrative purposes, multiple fibers or a single fiber with multiple cores configured with strain sensors are preferably employed for determining the shape of the endoscope body <b>114</b> and distal tip <b>112</b>, such as described using shape sensors in previously incorporated by reference <patcit id="pcit0006" dnum="US20070156019A1"><text>U.S. 2007/0156019 A1</text></patcit>, <patcit id="pcit0007" dnum="US20080212082A1"><text>U.S. 2008/0212082 A1</text></patcit>, and <patcit id="pcit0008" dnum="US164829A"><text>U.S. Application Serial No. 12/164,829</text></patcit>.

In <b>404</b>, a computer model that is indicative of the current configuration of the flexible endoscope <b>110</b> is generated using a model of the endoscope stored in the memory <b>155</b> and modifying it to reflect the shape information received from the optical fibers configured as shape sensors. Preferably, both the stored and generated computer models of the endoscope <b>110</b> are three-dimensional models.

In <b>405</b>, the patient computer model generated in <b>401</b> and the endoscope computer model generated in <b>404</b> are displayed together properly registered with respect to each other according to their respective positions in the fixed reference frame on the auxiliary display screen <b>160</b>. Thus, the operator of the steerable endoscope <b>110</b> is provided a view of the current shape of the endoscope <b>110</b> relative to the patient's body in order to provide guidance to the operator for navigating the endoscope <b>110</b> to a target site within the patient. A visual indication of the target site may also be displayed as well as computer models or other indications of any anatomic structures or body lumens that the endoscope <b>110</b> may encounter or pass through in its path to the target site.

The manner in which the patient and endoscope computer models are displayed on the auxiliary display screen <b>160</b> is preferably selectable by the operator using any one or more of a number of well known techniques such as clickable icons and pull down menus provided on the auxiliary display screen <b>160</b> and/or primary display screen <b>600</b>. For example, icons may be provided on the auxiliary display screen <b>160</b> so that the operator may select an anterior-posterior view <b>750</b>, oblique view <b>751</b>, or lateral-medial view <b>752</b> of the patient and endoscope <b>110</b>, such as shown respectively in <figref idref="f0004"><b>FIGS. 7</b></figref>, <figref idref="f0005"><b>8</b> and <b>9</b></figref>, by clicking on the appropriate icon using a computer mouse or other suitable device.

Referring to <figref idref="f0004"><b>FIG. 7</b></figref>, the anterior-posterior view <b>750</b> of the patient computer model <b>720</b> and endoscope computer model <b>710</b> is shown along with indications of various landmarks along the endoscope's path (e.g., mouth entry, esophagus, stomach entry, and colon) on the auxiliary display screen <b>160</b>. In addition, real-time images captured by the endoscope <b>110</b> (such as the captured image <b>650</b> shown in <figref idref="f0004"><b>FIG. 6</b></figref> being displayed on a primary display screen <b>600</b>) may also be displayed on the auxiliary display screen <b>160</b> adjacent to (or appearing to emanate from) the distal end <b>711</b> of the endoscope computer model <b>710</b>.

In addition to (or alternative of) displaying the patient and endoscope computer models, <b>720</b> and <b>710</b>, on the auxiliary display screen <b>160</b>, the computer models may be displayed on the primary display screen <b>600</b> along with the captured image <b>650.</b> For example, the oblique view <b>751</b> may be shown as a picture-in-picture insert to the captured image <b>650</b> on the primary display screen <b>600</b> as shown in <figref idref="f0006"><b>FIG. 10</b></figref> or it may be shown in an adjacent window to that of the captured image <b>650</b> on the primary display screen <b>600</b> as shown in <figref idref="f0006"><b>FIG. 11</b></figref><b>.</b>

Referring back to <figref idref="f0003"><b>FIG. 4</b></figref>, in <b>406</b>, a determination is made whether or not the operator has issued a stop guidance command. The operator may issue the stop guidance command in any one of a number of ways similar to, but complementary of those in which the operator issues the start guidance command.

If the determination in <b>406</b> is NO (i.e., navigation guidance mode has not been stopped by the operator), then the method loops back to <b>403</b> to update the current position and bend of the endoscope <b>110</b> and display a computer model of it along with the computer model of the patient for the next process cycle (or other programmed cycle). On the other hand, if the determination in <b>406</b> is YES (i.e., navigation guidance mode has been stopped by the operator), then the method proceeds to <b>407</b> in which the display processor <b>150</b> either causes the computer models of the patient and endoscope to no longer be displayed on the auxiliary display screen <b>160</b> and/or primary display screen <b>600</b> or freezes them on the display screen.

During the whole endoscopic procedure, starting from the insertion of the endoscope <b>110</b> in the patient's body, the proposed method and invention stores in the memory device <b>155</b> digital versions of images periodically captured by the camera <b>141</b> (at the distal tip <b>112</b> of the endoscope <b>110</b>) together with the three-dimensional (3-D) position and orientation information of the camera <b>141</b> (as determined by the position processor <b>130</b>) with respect to the patient anatomy at the times (e.g., at a processing rate of 30 HZ) that the images were captured. Timestamps indicating the times are also stored in the memory device <b>155</b>. The resulting database provides endoscopic images tagged by their respective 3-D locations in the fixed reference frame that can be accessed at any successive time by the display processor <b>150</b> for the purpose of computing synthetic two-dimensional (2-D) and three-dimensional (3-D) views of the visited anatomy. Thus, a full recording of the endoscopic procedure may be stored in the memory device <b>155</b> (preferably a mass storage device, such as a hard drive, for this application) that may be played back on the display screens <b>600</b> and <b>160</b> for educational and/or archival purposes.

A first synthetic 2-D view can be computed by assembling (stitching together) multiple endoscopic images taken from neighboring 3-D locations into a single larger panoramic image according to the well known Computer Vision technique known as mosaicking (see for instance the seminal work by <nplcit id="ncit0001" npl-type="s"><text>R. Szeliski, H.Y. Shum, Creating full view panoramic image mosaics and environment maps, In Proc. of ACM SIGGRAPH, 1997</text></nplcit>.) Endoscopic panoramas allow improved endoscopic navigation by providing a larger field of view then the endoscopic camera is capable of hence delivering greater awareness of the surrounding anatomy to the surgeon. Thus, the captured images shown in various figures herein on the primary display screen <b>600</b> are preferably panoramic images generated by the display processor <b>150</b> by mosaicking images retrieved from the memory device <b>155</b> which are representative of the most recently captured images by the camera <b>141</b>.

As an additional endoscopic navigation tool, graphical indications showing steering directions to previously defined landmarks in the patient are also provided as an aspect of the present invention.

<figref idref="f0007"><b>FIG. 12</b></figref> illustrates, as an example, a flow diagram of a computer implemented method <b>1200</b> for providing directional guidance to maneuver the steerable endoscope <b>110</b> to one or more landmarks in a patient. The computer in this case is preferably the one or more computers <b>200</b> that include the processors <b>130</b>, <b>140</b>, <b>150</b>.

In <b>1201</b>, a determination is made whether or not the operator has turned on a landmarks directional guidance mode. The operator may turn on and/or off this mode in any one of a number of ways such as depressing an input button or flipping a switch on the input device <b>303</b> when the steerable endoscope <b>110</b> is being controlled teleoperatively, or depressing an input button or flipping a switch on the handle <b>116</b> when the steerable endoscope <b>110</b> is being controlled manually. Other ways may include voice actuation using a microphone and voice recognition processor (not shown) that may be implemented in or operatively coupled to the one or more computers <b>200</b>; a foot pedal (not shown) that may be depressed by the operator so as to turn on or off the mode; and a graphical user interface (GUI) <b>202</b> using the display screen <b>600</b> that the operator may interact with using an input device <b>201</b> as shown in <figref idref="f0002"><b>FIG. 2</b></figref> to turn on and off the mode as well as specify other aspects or parameters of the method as described herein.

If the determination in <b>1201</b> is NO (i.e., landmark directional guidance is turned off), then the method loops through <b>1201</b> each process cycle (or other programmed cycle) until a YES determination is made, at which time, the method proceeds to <b>1202</b>.

In <b>1202</b>, the current position of the endoscope tip <b>112</b> in a fixed reference frame (i.e., a reference frame that does not move during the performance of a medical procedure using the endoscope <b>110</b>) is determined by the position processor <b>130</b> using information received from the optical fibers configured with strain sensors (e.g., optical fiber <b>501</b> in <figref idref="f0002"><b>FIG. 5</b></figref>) that have been inserted in the flexible body <b>114</b> of the endoscope <b>110</b>.

The current position and orientation of the endoscope tip <b>112</b> may be readily determined in the fixed reference frame (for example, as previously explained in reference to <b>403</b> of <figref idref="f0003"><b>FIG. 4</b></figref>) from the current bending of the optical fibers as indicated by information received from their strain sensors since their lengths are known, their proximal ends are fixed to a known location (such as fiber <b>501</b> to base point <b>502</b> as shown in <figref idref="f0002"><b>FIG. 5</b></figref>), and their distal ends extend to the endoscope's tip <b>112</b>.

In <b>1203</b>, a vector connecting the current position of the endoscope tip <b>112</b> to the position of each landmark to which guidance indications are to be provided is determined by the display processor <b>150</b> using the endoscope tip position determined in <b>1202</b> and landmark positions stored in the memory device <b>155</b>.

Typically, the landmarks are established by the operator as he or she guides the steerable endoscope <b>110</b> along its path from an entry point to a target site in the patient. As an example, <figref idref="f0008"><b>FIG. 14</b></figref> shows an oblique view of a patient's head and torso where the steerable endoscope <b>110</b> has entered through the patient's mouth and its tip <b>112</b> guided to a target site. In this example, the endoscope <b>110</b> has been guided through the esophagus to the patient's stomach. The stomach was then entered and the endoscope <b>110</b> guided to the target site. Along the way, landmarks are established at the mouth entry and stomach entry using a method such as described below in reference to <figref idref="f0007"><b>FIG</b>. <b>13</b></figref>.

Alternatively, the positions of landmarks within the patient may be based upon preoperative measured data such as generated by Computer Axial Tomography (CAT), Magnetic Resonance Imaging (MRI), or X-rays. In this case, the patient's body (as he or she is lying down on the operating room table) is referenced to the fixed reference frame and the landmarks are registered with the patient's body so that their positions are determinable in the fixed reference frame by their respective positions in the patient's body. The landmark positions are then stored in the memory device <b>155</b> (or other storage device accessible to the display processor <b>150</b>). Although establishing landmarks in this manner may be time consuming and expensive, it allows the method described herein to provide directional guidance to landmarks in front of the endoscope tip <b>112</b> (i.e., between the current position of the endoscope tip and the target site), not just behind it (i.e., between the entry point into the patient and the current position of the endoscope tip).

In <b>1204</b>, the vectors determined in <b>1203</b> are then transformed from the fixed reference frame to a reference frame associated with the endoscope tip <b>112</b> (or camera <b>141</b>) by the display processor <b>150</b>.

In <b>1205</b>, 3-D arrows (or other graphical representations) indicative of the transformed vectors determined in <b>1204</b> are generated by the display processor <b>150</b>. The directions of the 3-D arrows are referenced to the endoscope tip's reference frame so that they correspond to directions that the operator should steer the endoscope's tip <b>112</b> using the handle or electromechanical interface <b>116</b> as previously described.

The sizes of the 3-D arrows, on the other hand, may be generated so as to be proportional to the magnitudes of the translated vectors so that landmarks further away are represented by larger arrows. As an alternative, the sizes may be generated so as to be inversely proportional to the magnitudes of the translated vectors so that landmarks further away are represented by smaller arrows. As yet another alternative, the sizes may be generated so as to each have the same length (such as shown by arrows <b>1511</b>, <b>1512</b> and <b>1513</b> in <figref idref="f0008"><b>FIG. 15</b></figref>).

The 3-D arrows may also be color-coded so that the color of each arrow is uniquely associated with a particular anatomic structure. For example, referring to <figref idref="f0008"><b>FIG. 15</b></figref>, a 3-D arrow <b>1511</b> indicating a steering direction towards the mouth entry landmark may be colored red, a 3-D arrow <b>1512</b> indicating a steering direction towards the stomach entry landmark may be colored yellow, and 3-D arrow <b>1513</b> indicating a steering direction towards the colon entry landmark may be colored blue. Alternatively, the edges of the display screen <b>600</b> may be colored to correspond to landmarks. The color schemes used herein may be operator defined or correspond to the color scheme used on MR viewers.

In <b>1206</b>, the 3-D arrows (or other graphical representations) generated in <b>1205</b> and a current image (as currently captured at the endoscope tip <b>112</b> and processed by the image processor <b>140</b> or a panoramic image as previously described using mosaicking of the current and previously captured images) are displayed on the primary display screen <b>600</b> by the display processor <b>150</b>. In addition, previously captured images of the landmarks may also be selectively or automatically retrieved from the memory device <b>155</b> and displayed adjacent or otherwise proximate to their respective 3-D arrows. The positions of the 3-D arrows may be displayed on the display screen <b>600</b> so as to relate to the orientations of the translated vectors so that, for example, arrows pointing to the left are positioned on the left side of the image and arrows pointing to the right are positioned on the right side of the image. Alternatively, the 3-D arrows may all be positioned near the same side of the image or spaced around the image. In the case where edges of the display screen <b>600</b> may be colored to correspond to landmarks, the 3-D arrow corresponding to each landmark is displayed adjacent its respective colored edge so that the association of the 3-D arrow and landmark is clear.

In addition to showing the landmarks as 3-D arrows on or near the current endoscope image on the primary display screen <b>600</b>, the landmarks can be displayed together with the 3-D patient and endoscope computer models being displayed on the auxiliary display screen <b>160</b> wherein positions of stored landmarks may be shown in their proper locations relative to the patient computer model. The landmarks may be indicated, for example, by a text label with the landmark name, a thumbnail image with the stored endoscope image captured with the landmark, a small graphic icon related to the type of landmark, or with an approximate 3-D model of the landmark.

Selection of which, if any, landmark images are to be displayed may be made by the operator, for example, using the input device <b>201</b> and GUI <b>202</b> to select one display option from a menu including: all landmark images to be displayed, no landmark images displayed, and only nearest landmark image to be displayed. Alternatively, the operator may not be provided an option and the choice pre-selected and programmed into the display processor <b>150</b>. Position, size, color and other options for displaying the 3-D arrows on the display screen <b>600</b> may be selected by the operator using the input device <b>201</b> and GUI <b>202</b> as shown in <figref idref="f0002"><b>FIG</b>. <b>2</b></figref>.

As an example of such displaying, <figref idref="f0008"><b>FIG. 15</b></figref> shows the display screen <b>600</b> having a current image <b>1501</b> (i.e., an image currently captured by the endoscope <b>110</b> or a panoramic image as described elsewhere herein) displayed on it along with arrows <b>1511</b>, <b>1512</b> and <b>1513</b> respectively providing directional guidance to the mouth entry, stomach entry and colon entry landmarks. In addition, a previously captured image <b>1523</b> of the colon entry landmark is shown being displayed in a window adjacent to its associated arrow <b>1513</b>, which happens to be the nearest landmark to the current position of the endoscope tip <b>112</b>.

In <b>1207</b>, a determination is made whether or not the operator has turned off the landmark directional guidance mode. The operator may turn off the mode in any one of a number of ways similar to, but in an opposite manner of those in which the operator turned on the landmark guidance mode (e.g., placing a switch in an "off" vs. "on" position).

If the determination in <b>1207</b> is NO (i.e., landmark directional guidance mode has not been turned off by the operator), then the method loops back to <b>1202</b> to update the landmark directional vectors (e.g., <b>1511</b>, <b>1512</b>, <b>1513</b> in <figref idref="f0008"><b>FIG. 15</b></figref>) as the endoscope tip <b>112</b> is being guided to the target site and display updated graphical representations of the translated vectors on the display screen <b>600</b> along with a current image captured by the endoscope <b>110</b>. On the other hand, if the determination in <b>1207</b> is YES (i.e., landmark direction guidance mode has been turned off by the operator), then the method proceeds to <b>1208</b> in which the display processor <b>150</b> causes the graphical representations of the translated vectors and any corresponding landmark captured images to no longer be displayed on the display screen <b>600</b>.

It is to be appreciated that even though landmark directional guidance mode may be turned off, the operator may still be guiding the steerable endoscope <b>110</b> towards the target site and continually updated images captured at the endoscope tip <b>112</b> (and processed by image processor <b>130</b>) may still be displayed on the display screen <b>600</b> by the display processor <b>150</b>. For example, referring to <figref idref="f0008"><b>FIG</b>. <b>15</b></figref>, the display screen <b>600</b> may still have the current or panoramic image <b>1501</b> displayed on it, but the arrows <b>1511</b>, <b>1512</b>, <b>1513</b> and landmark image <b>1523</b> are no longer displayed after the operator turns off the landmark directional guidance mode.

It is also to be appreciated that the operator may interactively establish landmarks for which directional guidance is to be provided at any time while guiding the steerable endoscope <b>110</b> towards the target site. In particular, landmark establishment may be performed while landmark directional guidance mode is turned on as well as off.

<figref idref="f0007"><b>FIG. 13</b></figref> illustrates, as an example, a computer implemented method <b>1300</b> for establishing anatomic structure landmarks as the steerable endoscope <b>110</b> is being guided through a patient's body. As in the case of the method <b>1200</b> described in reference to <figref idref="f0007"><b>FIG</b>. <b>12</b></figref>, the computer in this case is preferably the one or more computers <b>200</b> that include the processors <b>130</b>, <b>140</b>, <b>150</b>. In particular, it is preferably performed by the display processor <b>150</b> concurrently with other activities performed by the display processor <b>150</b> as described herein.

In <b>1301</b>, a landmark establishment request is received from the operator. The operator may initiate the request in any one of various well known interactive techniques including those previously described in reference to turning on and off the landmark directional guidance mode (but preferably a different one so as not confuse the two actions).

In <b>1302</b>, the current position of the endoscope tip <b>112</b> in the fixed reference frame is determined. Note that if the landmark directional guidance mode is turned on, this action is the same as <b>1202</b> of method <b>1200</b> and therefore, it does not have to be performed since the result from <b>1202</b> may be used in this case (i.e., it does not have to be performed a second time or twice in the same process cycle).

In <b>1303</b>, the current position of the endoscope tip <b>112</b> is then stored in the memory device <b>155</b> and referenced as the position of the landmark whose establishment has been requested. In <b>1304</b>, a current image captured at the endoscope tip <b>112</b> is identified as an image of the landmark and in <b>1305</b>, information of the image is also stored in the memory device <b>155</b> and referenced as the image of the landmark whose establishment has been requested.

As may be appreciated in performing the method <b>1300</b>, the order in which actions <b>1302</b> and <b>1304</b> occur is not important. In fact, the image processor <b>140</b> may process captured images at a different rate than that in which the position processor <b>130</b> determines current positions of the endoscope tip <b>112</b>. Therefore, the two actions may differ in the time as well as in the order in which they may each occur. In any event, upon receiving the landmark establishment request in <b>1301</b>, the next determined position of the endoscope tip <b>112</b> is stored in the memory device <b>155</b> and the next captured image received from the image processor <b>130</b> is stored in the memory device <b>155</b> without concern about which one occurs first.

In addition to the establishment of landmarks while navigating the endoscope <b>110</b> towards a target site within the patient, anatomic structures (such as the esophagus, stomach, colon, etc.) may be measured using position information of the endoscope tip <b>112</b> as it moves from one end of the anatomic structure to the other. Information of the measured anatomic structure may than be used to align and size 3-D models or preoperative measured data of the anatomic structures to their proper positions inside the patient's body and display the structures together with the 3-D patient computer model on the auxiliary display screen <b>160</b>. As an example, <figref idref="f0008"><b>FIG</b>. <b>14</b></figref> shows how the measured positions of the mouth and stomach entry landmarks have been used, together with the intermediate shape of the endoscope, to align and size a 3-D model of the esophagus connecting the mouth to the stomach entry. The esophagus model is then displayable as a 3-D curved tube <b>1401</b>, together with the 3-D models of the patient and the endoscope <b>110</b> on the auxiliary display screen <b>160</b>.

Although the various aspects of the present invention have been described with respect to one or more preferred embodiments, it will be understood that the invention is entitled to full protection within the full scope of the appended claims. The present application also includes the following numbered clauses: <ol id="ol0001"><li>1. A computer implemented method for providing guidance to steer an endoscopic device towards one or more landmarks, the method comprising: <ul id="ul0002" list-style="none" compact="compact"><li>determining a current position of a reference point on the endoscopic device relative to a reference frame;</li><li>determining a first vector from the current position of the reference point to a position of a first landmark relative to the reference frame;</li><li>transforming the first vector from the reference frame to an image reference frame that corresponds to the perspective of an image capturing device disposed at a tip of the endoscopic device; and</li><li>displaying a graphical representation of the transformed first vector along with a current image captured by the image capturing device on a display so as to indicate a direction to steer the tip of the endoscopic device towards the first landmark.</li></ul></li><li>2. The method according to clause 1, wherein one or more optical fibers configured with strain sensors extend through the endoscopic device to provide position and shape information of the endoscopic device.</li><li>3. The method according to clause 1, wherein the current image is a stereoscopic image, the display is a three-dimensional display, and the graphical representation of the transformed first vector is a three-dimensional arrow pointing towards the direction to steer the tip of the endoscopic device towards the first landmark.</li><li>4. The method according to clause 1, further comprising prior to determining the first vector: <ul id="ul0003" list-style="none" compact="compact"><li>receiving an indication to establish the first landmark;</li><li>determining a position of the reference point at the time of receiving the indication to establish the first landmark;</li><li>storing the position of the reference point at the time of receiving the indication to establish the first landmark in a memory device; and</li><li>referencing the stored position as the position of the first landmark.</li></ul></li><li>5. The method according to clause 4, further comprising prior to determining the first vector: <ul id="ul0004" list-style="none" compact="compact"><li>capturing an image of the first landmark at a first opportunity after receiving the indication to establish the first landmark; and</li><li>storing the captured image of the first landmark in the memory device;</li><li>wherein the displaying of the graphical representation of the transformed first vector along with the current image on the display further includes displaying the captured image of the first landmark on the display proximate to the graphical representation of the transformed first vector.</li></ul></li><li>6. The method according to clause 1, further comprising: <ul id="ul0005" list-style="none" compact="compact"><li>determining a second vector from the current position of the reference point to a position of a second landmark relative to the reference frame;</li><li>transforming the second vector from the reference frame to the image reference frame; and</li><li>displaying a graphical representation of the transformed second vector along with the current image on the display so as to indicate a direction to steer the tip of the endoscopic device towards the second landmark.</li></ul></li><li>7. The method according to clause 1, further comprising: periodically repeating the determination of the current reference point position, the determination of the first vector, the transformation of the first vector, and the display of the graphical representation of the transformed first vector along with the current image on the display until an indication to stop such periodically repeated actions is received.</li><li>8. The method according to clause 1, wherein a size of the graphical representation of the transformed first vector is related to the magnitude of the transformed first vector.</li><li>9. The method according to clause 1, wherein a color of the graphical representation of the transformed first vector is a color associated with the first landmark.</li><li>10. The method according to clause 1, wherein the displaying of the graphical representation of the transformed first vector comprises displaying the graphical representation adjacent a side of the displayed current image that is indicative of the direction to steer the tip of the endoscopic device towards the first landmark.</li><li>11. The method according to clause 1, further comprising: storing positions of a plurality of landmarks including the position of the first landmark in a memory device prior to determining vectors from the current position of the reference point to the positions of the plurality of landmarks.</li><li>12. The method according to clause 11, wherein the positions of the plurality of landmarks are based upon preoperative measured data.</li><li>13. The method according to clause 1, further comprising: displaying computer models of the patient and the first landmark on a second display.</li><li>14. The method according to clause 13, wherein a color of the graphical representation of the transformed first vector is a same color as the computer model of the first landmark being displayed on the second display.</li><li>15. The method according to clause 1, further comprising: displaying computer models of the patient and the first landmark on the display.</li><li>16. The method according to clause 15, wherein a color of the graphical representation of the transformed first vector is a same color as the computer model of the first landmark being displayed on the display.</li><li>17. A system comprising: <ul id="ul0006" list-style="none" compact="compact"><li>a display;</li><li>an endoscopic device; and</li><li>one or more processors adapted to determine a current position of a reference point on the endoscopic device relative to a reference frame, determine a first vector from the current position of the reference point to a position of a first landmark relative to the reference frame, transform the first vector from the reference frame to an image reference frame that corresponds to the perspective of an image capturing device disposed at a tip of the endoscopic device, and display a graphical representation of the transformed first vector along with a current image captured by the image capturing device on the display so as to indicate a direction to steer the tip of the endoscopic device towards the first landmark.</li></ul></li><li>18. The system according to clause 17, wherein the one or more processors includes a position processor, the system further comprising: one or more optical fibers configured with strain sensors, wherein the one or more optical fibers extend through the endoscopic device to provide position and shape information of the endoscopic device to the position processor.</li><li>19. The system according to clause 17, wherein the current image is a stereoscopic image, the display is a three-dimensional display, and the graphical representation of the transformed first vector is a three-dimensional arrow pointing towards the direction to steer the tip of the endoscopic device towards the first landmark.</li><li>20. The system according to clause 17, wherein the one or more processors includes a display processor, the system further comprising: <ul id="ul0007" list-style="none" compact="compact"><li>a memory device;</li><li>wherein the display processor is adapted to receive an indication to establish the first landmark, determine a position of the reference point on the endoscopic device within a process cycle of receiving the indication to establish the first landmark, store the determined position of the reference point in the memory device, and reference the stored position as the position of the first landmark.</li></ul></li><li>21. The system according to clause 20, wherein the display processor is adapted to store an image captured by the image capturing device of the first landmark in the memory device within a process cycle of receiving the indication to establish the first landmark, and subsequently retrieve and display the stored captured image of the first landmark proximate to the graphical representation of the transformed first vector when the transformed first vector is being displayed along with the current image on the display.</li><li>22. The system according to clause 17, wherein the processor is adapted to determine a second vector from the current position of the reference point on the endoscopic device to a position of a second landmark relative to the reference frame, transform the second vector from the reference frame to the image reference frame, and display a graphical representation of the transformed second vector along with the current image on the display so as to indicate a direction to steer the tip of the endoscopic device towards the second landmark.</li><li>23. The system according to clause 17, wherein the one or more processors is adapted to perform in each of a sequence of process cycles the determining of the current reference point position, the determining of the first vector, the transforming of the first vector, and the displaying of the graphical representation of the transformed first vector along with the current image on the display.</li><li>24. The system according to clause 17, wherein the one or more processors is adapted to display the graphical representation of the transformed first vector so that a size of the graphical representation is related to the magnitude of the transformed first vector.</li><li>25. The system according to clause 17, wherein the one or more processors is adapted to display the graphical representation of the transformed first vector so that a color of the graphical representation is a color associated with the first landmark.</li><li>26. The system according to clause 17, wherein the one or more processors is adapted to display the graphical representation of the transformed first vector adjacent a side of the displayed current image that is indicative of the direction to steer the tip of the endoscopic device towards the first landmark.</li><li>27. The system according to clause 17, further comprising: <ul id="ul0008" list-style="none" compact="compact"><li>a memory device;</li><li>wherein the one or more processors is adapted to store positions of a plurality of landmarks including the position of the first landmark in the memory device prior to determining vectors from the current position of the reference point to the positions of the plurality of landmarks.</li></ul></li><li>28. The system according to clause 27, wherein the positions of the plurality of landmarks are based upon preoperative measured data.</li><li>29. The system according to clause 17, wherein the one or more processors is adapted to display computer models of a patient and the first landmark on a second display.</li><li>30. The system according to clause 29, wherein the one or more processors is adapted to display the graphical representation of the first vector on the display so that a color of the graphical representation of the transformed first vector is a same color as the computer model of the first landmark being displayed on the second display.</li><li>31. The system according to clause 17, wherein the one or more processors is adapted to display computer models of a patient and the first landmark on the display.</li><li>32. The system according to clause 31, wherein the one or more processors is adapted to display the graphical representation of the first vector on the display so that a color of the graphical representation of the transformed first vector is a same color as the computer model of the first landmark being displayed on the display.</li><li>33. A computer implemented method for assisting an operator in endoscopic navigation, the method comprising: <ul id="ul0009" list-style="none" compact="compact"><li>referencing a computer model of a patient to a reference frame;</li><li>determining a current position and shape of an endoscopic device relative to the reference frame;</li><li>generating a computer model of the endoscopic device according to the determined current position and shape; and</li><li>displaying the computer models of the endoscopic device and patient on a display so as to provide the endoscopic navigation assistance to the operator.</li></ul></li><li>34. The method according to clause 33, wherein the referencing of the computer model of the patient to the reference frame comprises: <ul id="ul0010" list-style="none" compact="compact"><li>referencing one or more points on the patient to the reference frame; and</li><li>associating a corresponding point of the computer model of the patient with each of the referenced one or more points on the patient.</li></ul></li><li>35. The method according to clause 34, wherein the computer model of the patient is a three-dimensional computer model and the referencing of the computer model of the patient to the reference frame comprises: aligning the computer model of the patient with the patient in the reference frame using the associated reference points of the patient and the patient computer model.</li><li>36. The method according to clause 34, wherein the computer model of the patient is a three-dimensional computer model and the referencing of the computer model of the patient to the reference frame comprises: conforming the computer model of the patient with the patient by overlaying the associated reference points of the patient computer model and the patient.</li><li>37. The method according to clause 34, wherein the referencing of the one or more points on the patient to the reference frame comprises: determining a position in the reference frame of a point on the endoscopic device when the point is touched to a corresponding one of the one or more points on the patient.</li><li>38. The method according to clause 33, wherein one or more optical fibers configured with strain sensors extends through the endoscopic device so as to provide shape information of the endoscopic device.</li><li>39. The method according to clause 33, wherein one or more optical fibers configured with strain sensors is attached to the endoscopic device so as to provide position information of the endoscopic device.</li><li>40. The method according to clause 33, wherein the endoscopic device has a distal tip and an image capturing device positioned so as to capture images from a perspective of the distal tip, the method further comprising: <ul id="ul0011" list-style="none" compact="compact"><li>periodically capturing images with the image capturing device and storing the captured images in a memory device; and</li><li>determining positions of the image capturing device at the times of such image capturing and storing the positions in the memory device.</li></ul></li><li>41. The method according to clause 40, further comprising: storing timestamps corresponding to the times of such image capturing in the memory device.</li><li>42. The method according to clause 40, further comprising: computing a panoramic image from the captured images stored in the memory device by mosaicking a plurality of the captured images together.</li><li>43. The method according to clause 42, further comprising: <ul id="ul0012" list-style="none" compact="compact"><li>displaying the panoramic image in a first area on the display; and</li><li>displaying the computer models of the endoscopic device and patient in a second area on the display.</li></ul></li><li>44. The method according to clause 42, further comprising: displaying the panoramic image on the display; and displaying the computer models of the endoscopic device and patient as a picture-in-picture over the panoramic image on the display.</li><li>45. The method according to clause 42, further comprising: displaying the panoramic image so as to appear to project from a distal tip of the computer model of the endoscopic device being displayed on the display.</li><li>46. The method according to clause 33, further comprising: <ul id="ul0013" list-style="none" compact="compact"><li>determining positions in the reference frame for at least two points on an anatomic structure within the patient; and</li><li>displaying a computer model of the anatomic structure along with the computer models of the patient and endoscopic device on the display according to the determined positions for at least two points on the anatomic structure.</li></ul></li><li>47. A system comprising: <ul id="ul0014" list-style="none" compact="compact"><li>a display;</li><li>an endoscopic device; and</li><li>a processor adapted to reference a computer model of a patient to a reference frame, determine a current position and shape of the endoscopic device relative to the reference frame, generate a computer model of the endoscopic device according to the determined current position and shape, and display the computer models of the endoscopic device and patient on the display so as to provide endoscopic navigation assistance to an operator of the system.</li></ul></li><li>48. The system according to clause 47, wherein the processor is adapted to reference the computer model of the patient to the reference frame by: <ul id="ul0015" list-style="none" compact="compact"><li>receiving information of one or more points on the patient and processing the received information so as to reference the one or more points to the reference frame; and</li><li>associating a corresponding point of the computer model of the patient with each of the referenced one or more points on the patient.</li></ul></li><li>49. The system according to clause 48, wherein the computer model of the patient is a three-dimensional computer model and the processor is adapted to reference the computer model of the patient to the reference frame by: aligning the computer model of the patient with the patient in the reference frame using the associated reference points of the patient and the patient computer model.</li><li>50. The system according to clause 48, wherein the computer model of the patient is a three-dimensional computer model and the processor is adapted to reference the computer model of the patient to the reference frame by: conforming the computer model of the patient with the patient by overlaying the associated reference points of the patient computer model and the patient.</li><li>51. The system according to clause 48, wherein the processor is adapted to reference the one or more points on the patient to the reference frame by: determining a position in the reference frame of a point on the endoscopic device when the point is touched to a corresponding one of the one or more points on the patient.</li><li>52. The system according to clause 47, further comprising: one or more optical fibers configured with strain sensors extending through the endoscopic device so as to provide shape information of the endoscopic device.</li><li>53. The system according to clause 47, further comprising: one or more optical fibers configured with strain sensors and attached to the endoscopic device so as to provide position information of the endoscopic device.</li><li>54. The system according to clause 47, further comprising: <ul id="ul0016" list-style="none" compact="compact"><li>an image capturing device positioned at a distal tip of the endoscopic device so as to capture images from a perspective of the distal tip; and</li><li>a memory device;</li><li>wherein the processor is adapted to periodically capture images with the image capturing device and store the captured images in the memory device, and determine positions of the image capturing device at the times of such image capturing and storing the positions in the memory device.</li></ul></li><li>55. The system according to clause 54, wherein the processor is adapted to store timestamps corresponding to the times of such image capturing in the memory device.</li><li>56. The system according to clause 54, wherein the processor is adapted to compute a panoramic image from the captured images stored in the memory device by mosaicking a plurality of the captured images together.</li><li>57. The system according to clause 56, wherein the processor is adapted to display the panoramic image in a first area on the display and the computer models of the endoscopic device and patient in a second area on the display.</li><li>58. The system according to clause 56, wherein the processor is adapted to display the panoramic image on the display and the computer models of the endoscopic device and patient as a picture-in-picture over the panoramic image on the display.</li><li>59. The system according to clause 56, wherein the processor is adapted to display the panoramic image so as to appear to project from a distal tip of the computer model of the endoscopic device being displayed on the display.</li><li>60. The system according to clause 47, wherein the processor is adapted to determine positions in the reference frame for at least two points on an anatomic structure within the patient and display a computer model of the anatomic structure along with the computer models of the patient and endoscopic device on the display according to the determined positions for the at least two points on the anatomic structure.</li></ol>