Spatial registration of tracking system with an image using two-dimensional image projections

SPATIAL REGISTRATION OF TRACKING SYSTEM WITH AN IMAGE USING TWO-DIMENSIONAL IMAGE PROJECTIONS

Technical Field

[0001] This disclosure relates generally to systems and methods for registering a tracking system with an image space using two-dimensional image projections.

Background

[0002] Image registration is the process of determining a spatial transformation to bring two or more image coordinate systems into alignment with one another. One example is registration of computed tomography (CT) and cone beam computed tomography (CBCT) images. For instance, the CT image may be obtained in advance of a procedure, such as days or weeks prior to a procedure (e.g., study or treatment). The CBCT image may be obtained just before or at the time of the procedure. The CBCT image set is mathematically processed to spatially register it with the prior CT image. This sort of CT and CBCT registration is used for a variety of purposes, such as planning, diagnosis and treatment (e.g., interventional radiology, image guided therapy and the like). However, equipment to acquire CBCT images may not be available. Additionally, CBCT imaging typically involves a significant of radiation energy for patients and health care personnel.

Summary

[0003] This disclosure relates generally to systems and methods for registering a tracking system with an image space using two-dimensional image projections [0004] As an example, a method includes acquiring a first two-dimensional projection image using a medical imaging modality. The first image includes a twodimensional field of view that includes a patient and a multi-modal marker. The method also includes acquiring a second two-dimensional projection image using the medical image modality. The second image includes the patient and the multimodal marker and is along a non-coincident angle with respect to the first projection image. Predetermined portions of the multi-modal marker are visible in the first projection image and the second projection image and have a known location and orientation with respect to at least one sensor detectable by a tracking system. The method also includes estimating a three-dimensional position for predetermined portions of the multi-modal marker with respect to a coordinate system of the medical imaging

-1 Date Reçue/Date Received 2023-02-15 modality according to locations of the predetermined portions in each of the respective first and second two-dimensional projection images. The method also includes determining an affine transformation for registering a three-dimensional coordinate system of the tracking system with a three-dimensional coordinate system of the medical imaging modality based on the estimated position for the respective predetermined portions of the multi-modal marker and a known relationship of the at least one sensor and the predetermined portions of the multimodal marker.

[0005] As another example, a system includes one or more non-transitory computer-readable media to store data and instructions executable by a processor. The data includes prior three-dimensional image data acquired for a patient. The data also includes two-dimensional image data that includes at least one twodimensional image acquired by a medical imaging modality to include the patient and a multi-modal marker. Predetermined portions of the multi-modal marker are visible in the at least one two-dimensional image and have a known location and orientation with respect to at least one tracking sensor detectable by a tracking system. The instructions are programmed to perform a method comprising:

estimating a three-dimensional position for predetermined portions of the multi-modal marker with respect to a coordinate system of the medical imaging modality according to respective locations of the predetermined portions in each of the at least one two-dimensional image; and determining an affine transformation for registering a three-dimensional coordinate system of the tracking system with a three-dimensional coordinate system of the medical imaging modality based on the estimated position for the respective predetermined portions of the multi-modal marker and a known relationship of the at least one tracking sensor and the predetermined portions of the multi-modal marker.

[0006] As a further example, a method includes acquiring a first twodimensional projection image using a medical imaging modality, the first image including a two-dimensional field of view that includes a patient and a multi-modal marker. The method also includes acquiring a second two-dimensional projection image using the medical image modality, the second image including the patient and the multimodal marker and being along a non-coincident angle with respect to the

-2Date Reçue/Date Received 2023-02-15 first projection image. Predetermined portions of the multi-modal marker being visible in the first projection image and the second projection image and having a known location and orientation with respect to at least one sensor detectable by a tracking system. The method also includes deriving respective forward projections from the prior three-dimensional image that correspond to each of the first twodimensional projection image and the second two-dimensional projection image. The method also includes determining an affine transformation from the coordinate system of the medical imaging modality to the coordinate system of the prior threedimensional image based on registering the first and second projection images with the respective forward projections.

[0007] As yet another example, a system may include one or more nontransitory computer-readable media to store data and instructions executable by a processor. The data includes: prior three-dimensional image data acquired for a patient. The data also includes two-dimensional image data that includes at least one two-dimensional image acquired by a medical imaging modality to include the patient and a multi-modal marker. Predetermined portions of the multi-modal marker are visible in the at least one two-dimensional image and have a known location and orientation with respect to at least one tracking sensor detectable by a tracking system. The instructions are programmed to perform a method comprising:

deriving respective forward projections from the prior three-dimensional image that correspond to each of the two-dimensional images; and determining an affine transformation between the coordinate system of the medical imaging modality and the coordinate system of the prior threedimensional image based on registering each of the two-dimensional images with the respective forward projections.

Brief Description of the Drawings

[0008] FIG. 1 is a flow diagram depicting an example of a method to register sensors of a tracking system into a spatial coordinate system of a medical imaging modality.

[0009] FIG. 2 is a flow diagram for registering from a spatial coordinate system of a medical imaging modality to a spatial coordinate system of a prior threedimensional image.

-3Date Reçue/Date Received 2023-02-15

[0010] FIG. 3 depicts an example of a marker.

[0011] FIGS. 4A and 4B depict an example of a multi-modal marker.

[0012] FIG. 5 depicts an example of another multi-modal marker.

[0013] FIG. 6 demonstrates an example of marker pad device.

[0014] FIGS. 7A and 7B depict a schematic example of a medical imaging modality.

[0015] FIG. 8 depicts an example of a system for generating affine transformations.

[0016] FIG. 9 depicts an example of a registration manager to control use or corrections to one or more affine transformations.

[0017] FIGS. 10A and 10B depict graphical representations of information registered in a given domain according to an affine transformation.

Detailed Description

[0018] This disclosure relates generally to methods and systems for registering a tracking system with an interventional image space. The approaches disclosed herein can be implemented using reduced ionizing radiation compared to many existing approaches.

[0019] The method utilizes a marker device (e.g., a multi-modal marker) that includes fiducial markers detectable by more than one modality. For example, the marker device may include a first fiducial marker to provide a pattern that is visible in an image generated by a medical imaging modality. The medical imaging modality may be fluoroscopy, X-ray or another modality such as ultrasound, configured to provide the image as a two-dimensional projection, which includes a part of a patient and the first fiducial marker. In some examples, the marker device may be configurable to select whether or not the first pattern is visible in subsequent images acquired by the medical imaging modality. The marker device also includes one or more second fiducial markers (e.g., one or more sensors) detectable by a threedimensional spatial tracking system. The second fiducial markers are arranged in a predetermined spatial position and orientation known with respect to the spatial position of first fiducial marker, which is discernable from the image acquired by the medical imaging modality.

[0020] As a further example, the imaging modality can be used to acquire twodimensional image projections and to store the images in memory (e.g., as DICOM

-4Date Reçue/Date Received 2023-02-15 image files). Each ofthe images includes a field of view that includes the patient and the marker pattern corresponding to the first fiducial marker ofthe marker device.

Thus, during acquisition, the marker is configured (e.g., radiopaque) to enable the marker pattern to be visible in the acquired images. Each of the images is processed to locate and identify predetermined portions ofthe pattern in each respective image. The identified portions (e.g., points or regions) ofthe marker pattern are converted to corresponding three-dimensional locations in a threedimensional spatial coordinate system ofthe medical imaging modality. An affine transform is computed to map the sensor position to the coordinate system ofthe medical imaging modality (or to map the marker pattern locations in the imaging modality coordinate system to the coordinate system ofthe tracking system) based on the spatial relationship between each of the first and second fiducial markers of the marker device. As mentioned, the spatial relationship is already known (and stored in memory) based on the geometry ofthe pattern ofthe first fiducial marker and sensors of the second fiducial marker implemented on the tracking device. Thus, the transform enables the systems and methods to register position and orientation information provided by the tracking system into the coordinate system of the medical imaging modality (or the inverse thereof) based on a small number (e.g., two or three) of projection images, which results in greatly reduced ionizing radiation compared to many existing approaches (e.g., cone beam computed tomography). [0021] After such transform is determined, a second affine transform may be determined to map from the spatial coordinate system ofthe medical imaging modality to another three-dimensional spatial coordinate system, such as corresponding to a pre-operative three-dimensional image scan. For example, the pre-operative three-dimensional image scan may be a high-resolution imaging technique, such as computed tomography or magnetic resonance imaging, which may be performed hours, days or even weeks in advance of a procedure. The second affine transform may be determined by registering the first and second (or more) projection images, which were acquired using the medical imaging modality described above, with corresponding projections (e.g., along the same angles) derived from the pre-operative three-dimensional image scan.

[0022] FIG. 1 is a flow diagram depicting an example of a method 100 for registering a coordinate system of a tracking system into a coordinate system of a

-5Date Reçue/Date Received 2023-02-15 medical imaging modality (e.g., a fluoroscopy or x-ray system). One or more sensors are integrated into a multi-modal marker device, which includes an arrangement of fiducial markers having a known spatial relationship with respect to each other. The fiducial markers include a set of markers (e.g., sensors) that are detectable by the tracking system and another set of markers (e.g., radiopaque markers) that are visible in images acquired by the imaging modality. The method thus will be described with respect to a tracking system and a medical imaging modality.

[0023] As an example, the medical imaging modality is configured to provide image data (e.g., DICOM image file) representing a two-dimensional image projection. For example, the medical imaging modality can be an X-ray system, such as a flat panel X-ray system, a biplane X-ray system, a C-arm fluoroscopy system, an ultrasound imaging system or the like.

[0024] As a further example, the marker device includes one or more sensors configured to indicate a three-dimensional position in a coordinate system of the tracking system. For example, the tracking system is an electromagnetic tracking system that generates an electromagnetic field. Each sensor provides a sensor signal based on the electromagnetic field, which is converted into position and orientation information for each respective sensor. An example electromagnetic field tracking system is commercially available from Northern Digital, Inc., of Ontario, Canada. The tracking system can provide the tracking data at an output sample rate (e.g., sixty samples per second) for each sensor sufficient to enable substantially real time determination of sensor location (e.g., to provide a vector describing sensor position and orientation). The tracking system thus can process each frame of tracking data such that the tracking data can likewise represent real time tracking data acquired by the tracking system, which can be registered into a coordinate system of an imaging system, as disclosed herein. In some examples, each sensor can be detectable by the tracking system to enable tracking the sensor in five or six degrees of freedom. Other types of sensors and tracking system may be used in other examples.

[0025] At 102, the method 100 includes acquiring a first two-dimensional projection image using the medical imaging modality. The first image thus includes a two-dimensional image projection for a field of view that includes a patient (a region

-6Date Reçue/Date Received 2023-02-15 of interest of the patient’s body) and a multi-modal marker. As disclosed herein, the multi-modal marker includes a sensor to determine a three-dimensional position in a tracking system coordinate space and a second marker (e.g., fiducial pattern) that is detectable by the medical imaging modality. The sensor and second marker may be co-located or have other fixed relative positions that are known with respect to each other a priori.

[0026] At 104, the method includes acquiring a second two-dimensional projection image using the medical image modality. The second image also includes the patient and the multimodal marker though is taken along a non-coincident angle (e.g., less than or equal to about 90 degrees) with respect to angle along which the first image is acquired. There can be any number of total images; although the number is kept low, such as less than ten (e.g., 5 or 2 images) to reduce the exposure to ionizing radiation compared to other modalities, such as CBCT. As an example, the imaging modality can be configured to acquire (at 102 and 104) the images to include a right anterior oblique projection, a left anterior oblique projection as well as other projections (e.g., an anterior-posterior projection). In each of the images, predetermined portions of a radiopaque fiducial marker of the multi-modal marker device are visible. There may be one or more radiopaque fiducial markers on the multi-modal marker device, each having a known location and orientation with respect to one or more respective tracking sensors detectable by a tracking system. As used herein, radio opaque refers to the inability of ionizing electromagnetic radiation to pass through sufficient to make the objects visible in a corresponding image obtained by an imaging modality (e.g., two-dimensional (2D) medical imaging modality 456). Thus, the radio opaque objects can be radio-dense materials with respect to the imaging modality. In an example where two images are acquired (e.g., anterior-posterior and lateral projections), the multi-modal marker device may include three spaced apart radio-opaque markers (e.g., spheres) and the markers are arranged so as to not all be co-linear in each of the images. In another example, a single marker could be used; however, the medical imaging modality would need to acquire at least three images along different relative projection angles.

[0027] At 106, a three-dimensional position is estimated for respective predetermined portions of the multi-modal marker with respect to a coordinate system of the medical imaging modality based on the locations of such

-7Date Reçue/Date Received 2023-02-15 predetermined portions in each of the respective images. At 106, the locations for each of the predetermined portions of the multimodal marker are also determined in the tracking coordinate system, such as to provide a set of points in tracking system space. Thus, the same portions of one or more markers (e.g., coordinates in the respective spatial domains) are identified in both the intraoperative imaging coordinate system and the tracking system coordinate system. The number of points will vary depending on the construction and number of marker devices that are used.

[0028] As one example, the fiducial marker(s), which is represented in the images acquired from the imaging modality at 102 and 104, may include a radiopaque material in the form of a rectangular-shaped marker border having respective comers where edges thereof meet. For example, the radiopaque material is provided on the marker device in a form to represent an Arllco marker (see, e.g., Open Source Computer Vision Library: http://opencv.org). In this way, an established image processing algorithm (e.g., detectMarkers() function of the OpenCV library) may be implemented to detect and identify respective corners from the image projection (images acquired at 102 and 104) that includes a representation of the ArUco-type fiducial marker. Other related parameters may also be determined by the image processing algorithm. An example of such an ArUco marker is show in FIGS. 3,4A, 7A and 7B. FIGS. 7A and 7B and the corresponding description demonstrate an example of how respective corners of such marker may be located in three-dimensional coordinates of an intraoperative medical imaging modality to enable registration between an intraoperative spatial coordinate system of the medical imaging modality and a pre-operative spatial coordinate system (of a preoperative image space).

[0029] As a second example, which may be an alternative or in addition to the ArUco type marker, the radiopaque material may be in the form of three-dimensional objects on or embedded within the marker device, such as in the form of spheres, cones or the like. Spheres are an example of objects that exhibit the same given shape regardless of the viewing angle for a projection 2D image. In this example, each of the spheres has a known spatial position with respect to the coordinates and orientation of the tracking sensor that is also fixed with respect to the marker device. Thus, points for each of the spheres (or other 3D objects) system may be computed

-8Date Reçue/Date Received 2023-02-15 at 106 in the tracking coordinate by multiplying a transform for the sensor that defines a fixed, known offset with respect to each respective sphere. In the example where there are three marker devices, each containing three spheres, the estimation at 106 determines three sets of three points each in the tracking system coordinate system. Similarly, the coordinates location of each sphere is determined for each (at least two) of the 2D images that are acquired (at 102 and 104). For example, the spheres are located for each tracking sensor (e.g., using a ClosestPoint method as disclosed herein), and the spheres location (e.g., the center of each sphere) is stored in memory linked to the respective sensor. This results in the same number of points (for each sphere) as determined in the tracking coordinate system.

[0030] At 108, an affine transform is determined to map the coordinate system of the tracking system to the coordinate system of the medical imaging modality (also referred to herein as an intraoperative imaging modality). For example, the transform is determined by applying a co-registration method to align the sets of points provided at 106 for each of the tracking coordinate system and the intraoperative coordinate system. As disclosed herein (see, e.g., transform calculators in FIG. 8), some examples of co-registration methods include an error minimization function (e.g., single value decomposition), a change of basis function as well as other functions that may be used to determine the transform at 108. [0031] In some examples, the medical imaging modality includes a C-arm having a center of rotation, an origin of the coordinate system of the medical imaging modality thus may be defined as the center of rotation of the C-arm. The medical imaging system may be modelled as a pinhole camera having corresponding spatial parameters that are used to define a spatial coordinate system for medical imaging modality (e.g., a C-arm). For example, spatial parameters of a C-arm (e.g., center of rotation, rotation angle of C-arm, radius of C-arm) may be measured manually, provided by a manufacture and/or derived from fields of a DICOM image file. The transform determined at 108 can be based on the estimated position for the predetermined portions of the marker(s) determined at 106 and based on known fixed spatial relationship of the predetermined portions of the marker(s) and the tracking sensor(s) of the marker device. As an example, the fixed relationship of the predetermined portions of the marker(s) and sensors may be determined during manufacturing and printed on the marker. As another example, the relationship may

-9Date Reçue/Date Received 2023-02-15 be measured (e.g., manually) and entered into a computer (e.g., via user interface) that is programmed to determine the transform at 108. In other examples, the coordinates for each of the points may be determined automatically as relative coordinates in the image data according to pixel locations (e.g., at the centroid or center) of each marker.

[0032] FIG. 2 is a flow diagram of a method 150 for registering from a spatial coordinate system of a medical imaging modality to a spatial coordinate system of a prior three-dimensional image, such as is stored in memory (e.g., as DICOM or other image file). For example, the prior three-dimensional image can be acquired by preoperatively for a given patient by a three-dimensional medical imaging modality. As an example, the preoperative image data can correspond to a preoperative arterial CT scan for a region of interest of the patient, such as can be acquired weeks or months prior to a corresponding operation. Other imaging modalities can be used to provide three-dimensional image data, such as MRI, ultrasonography, positron emission tomography or the like. Such scans are common part of preoperative planning in a surgical workflow to help size prostheses and to plan surgery or other interventions.

[0033] In some examples, one or more anatomical structures captured in the preoperative image data may be converted to a respective three-dimensional model in the coordinate system of preoperative image. As an example, the model is an implicit model that mathematically describes a tubular anatomic structure (e.g., a patient's vessels), such as including a centerline and surface of the tubular structure. The implicit model may include a small set of parameters such as corresponding to a lofted b-spline (basis spline) function for the elongated anatomical structure. As one example, the anatomical model generator can be programmed to compute the implicit model data according to the disclosure of U.S. Patent Publication No.

2011/0026793 entitled Automated Centerline Extraction Method and Generation of Corresponding Analytical Expression and Use Thereof. Another example of generating an implicit model for tubular anatomical structures is disclosed in Analytical centerline extraction and surface fitting using CT scans for aortic aneurysm repair, Goel, Vikash R, Master’s Thesis, Cornell University (2005). Other types of geometric representations can also be utilized to provide the implicit model. For example, parameters representing lofted ellipses or triangular meshes can be

-10Date Reçue/Date Received 2023-02-15 generated to provide the anatomical model data representing the patient’s anatomical structure of interest in three dimensional coordinate system. The threedimensional mesh that is generated may be stored in memory in addition or as an alternative to the three-dimensional image acquired by the preoperative image modality.

[0034] At 152, the method includes deriving respective forward projections from the prior three-dimensional image volume based on the acquired images (e.g., first and second images acquired at 102 and 104). For example, each of the acquired 2D images is registered with corresponding forward 3D projections from the like angles. As an example, two-dimensional LAO and RAO images from a fluoroscopy system are registered by aligning such images (e.g., through translation and rotation) with corresponding LAO and RAO projections derived from a CT image volume. The registration of projection angles may be implemented through manual alignment and/or be automated.

[0035] In an example where a three-dimensional mesh has been generated to model an anatomical structure (e.g., vessel) in preoperative three-dimensional image, the mesh may be used to align the with respect to the intraoperative images (e.g., as acquired at 102 and 104). A 3D projection matrix may be applied to the mesh that was generated from the pre-operative image. For example, a ’’perspective'’ projection matrix is used if the appropriate parameters for modeling the intraoperative imaging system as a pinhole camera are known. In other examples, a parallel project matrix is used. In an example, where the marker device includes radiopaque spheres, the application of the parallel projection matrix should be sufficiently accurate because the spheres on the marker device are close together relative to the focal length of the intraoperative imaging system. If the angle of the C-arm is known for each of the intraoperative images, one 3D projection of the mesh is performed to match the angle for each intraoperative image. If the angle of the Carm is not known, multiple 3D projections may be generated along different angles and there may be a manual or automated selection of a best fit match between the respective 3D projections and the respective two-dimensional image. The end result of this will be pairs of images, in which each image pair includes a 2D intraoperative image and a 2D projection derived from the three-dimensional mesh.

-11 Date Reçue/Date Received 2023-02-15

[0036] At 154, the method includes determining an affine transformation from the coordinate system of the medical imaging modality to the coordinate system of the prior three-dimensional image based on registering the firstand second projection images (e.g., acquired at 102 and 104) with the respective forward projections determined at 152. The transform that is generated at 154 can be stored in memory.

[0037] As an example, a co-registration method may be implemented (e.g., as instructions executable by a processor) to determine the transform at 154 for mapping spatial data from the intraoperative 2D image space to the preoperative 3D image space. For example, the transform is determined ay 154 by applying a coregistration method to align sets of points identified for each of the 2D medical images and prior 3D image coordinate systems. In each of the 3D projection images (e.g., derived at 152) and the images that are acquired by the 2D medical imaging modality (at 102 and 104), one or more sets of common points is identified. In an example, the points may be anatomical landmarks or other identifiable fiducial points, such as bony landmarks on the spine, bits of calcification visible in both image set images, or points on vessels such as when contrast is used in both the intraoperative and preoperative images. Because the preoperative image is in threedimensional space, the user can identify the points using 3 orthogonal views (axial, coronal, and sagittal) to directly measure the x, y, and z locations in the preoperative 3D coordinate system. The points may be identified in each of the images manually or automated methods of feature extraction may be used.

[0038] Similar to the determination at 108, examples of co-registration methods that may be used to determine the transform at 154 include an error minimization function (e.g., single value decomposition), a change of basis function as well as other functions. In an example, the same functions may invoked to generate each of the respective transforms to provide more efficient storage and computations.

[0039] By way of example, each of the transforms generated at 108 and 154 may be stored as separate transforms or aggregated together to enable registration of sensors in the tracking system coordinate system to be represented and visualized in the 3D coordinate system of the image volume. In examples where the transforms (determined at 108 and 154) are kept separate, it affords the ability to change one of the transforms without impacting the other. For instance, if the

-12Date Reçue/Date Received 2023-02-15 second step (e.g., fluoroscopy to pre-op CT) changes from a manual registration to an automatic registration the approach is still the same and the transform generation method 100 of FIG. 1 does not have to be modified. Additionally, if the transform generation method of FIG. 1 changes, the other transform generation method 150 does not have to be modified. In examples where these transforms are kept separate, an output visualization may be generated by multiplying the two transforms together when rendering an object based on tracking sensor data. The details of the multiplication (e.g., order and whether you multiply by the transform or the inverse transform) depends on whether the view is being rendered in tracking system space, in the imaging modality space, or in coordinate space of the prior three-dimensional (e.g., CT) image space.

[0040] As a further example when rendering an output visualization in preoperative CT space, models for the bones and vasculature (e.g., generated in the pre-op CT image space) may be rendered with no transform and anything tracked in EM space (catheters, guidewires, etc.) would have both transformations applied. In an example, when rendering in tracking system space, the models for the bones and vasculature (being in the pre-op CT image space) would have the inverse of both transforms applied and anything tracked in tracking system space (e.g., objects having one or more tracking sensors, such as catheters, guidewires, etc.) would have no transform applied. Additionally, in an example when rendering in the imaging modality space (e.g., fluoroscopy or the like), the models for the bones and vasculature would have the inverse of the imaging modality to pre-op CT transform (e.g., determined at 154) applied and anything tracked in EM space by tracking sensors (e.g., catheters, guidewires, etc.) would have the tracking system to imaging modality transform (e.g., transform determined at 108) applied.

[0041] FIG. 3 depicts an example of a fiducial marker 200. As shown in this example, the marker includes black and white colors (e.g., binary) and includes a thick black rectangular (e.g., square) border 202 along each side of its entire peripheral edge (e.g., having a thickness t, such as one or more pixels thick). An interior of the marker 200 includes symbols 204 and 206 that can be used to define an orientation and/or other identifying feature for the marker, such as according to an Acllco library.

-13Date Reçue/Date Received 2023-02-15

[0042] FIGS. 4A and 4B depict an example of a multi-modal marker device 250. The multi-modal marker can be fixed with respect to a patient (e.g., attached to the patient’s body) during the acquisition of the first and second images using a 2D imaging modality, such as fluoroscopy or x-ray. FIG. 4A shows one side surface 252 of the marker device 250 that includes a fiducial marker (e.g., the marker of FIG. 3) 254 located within a white colored border 256 to provide contrast between the white border and a thick black border 258 of the fiducial marker (e.g., extending between dotted line and the white border 256). Symbols 260 and 262 are on the fiducial marker spaced apart from the black border 258.

[0043] As shown in FIG. 4B, as viewed from the other side showing surface 268 of the marker device 250, one or more tracking sensors (e.g., electromagnetic sensors) 270 are attached to the marker device 250 at known positions and orientations relative to the corners 264 of the fiducial marker 254. In one example, the one or more sensors 270 can respectively spatially sense a plurality of degrees of freedom (DOF). For example, the one or more sensors 270 can be configured to sense six (6) DOF, such as disclosed herein. In one example, the sensors 270 can be localized using an electromagnetic tracking system. The tracking system allows for determination of position and orientation of each sensor 270 based on a sensor signal provided from the sensor to the tracking system in response to an electromagnetic field. Other types of tracking systems configured to track the position and orientation of each sensor in three-dimensional space may be used in other examples.

[0044] In another example, the multi-modal marker may include a chamber between its side surfaces 252 and 268 that is configured to hold a volume of material (e.g., radiopaque material, such as a radiopaque contrast agent) to render a representation black portion of the marker visible in images acquired by the medical imaging modality. The chamber extends along and defines a rectangular-shaped thick border for the marker to define respective corners where edges thereof meet, which corners may be located in acquired images, as disclosed herein.

[0045] For example, the marker device 250 includes at least one port (e.g., in one of the surfaces 252,268 or an edge of the device) to access the chamber such as to add and remove the radiopaque material with respect to the chamber. Each chamber thus can be filled with contrast to create an identifiable pattern in the two

-14Date Reçue/Date Received 2023-02-15 dimensional projection images acquired by the medical imaging modality. Thus, when the material is in the chamber, the black portion of the fiducial marker is visible in medical images and when the radiopaque is removed, the black portion is visualized substantially the same as the white portion. For example, after the registration is complete (e.g., at least after the images have been acquired at 102 and 104 for registration), the contrast material would be drained out, and potentially flushed with clear saline solution, so that it does not interfere with subsequent imaging during the procedure. In other examples, the radiopaque material remains throughout the procedure.

[0046] Other examples of a multi-modal marker device 300, which does not include the AruCo-type marker, are demonstrated in FIGS. 5 and 6. In the example of FIG. 5, the combination marker device 300 includes a plurality of radiopaque fiduciary objects 302 disposed within a substrate 304. Each of the radiopaque fiduciary objects 302 has a predetermined geometry and is arranged with a predetermined geometric relationship relative to each other. For example, the radio opaque objects 302 can be implemented as spheres or other shapes such as having a predetermined angular orientation and spatial arrangement (e.g., configured as a scalene right triangle). Thus, each of the radio opaque objects 302 can be identified in a corresponding 2D imaging modality (e.g., obtained intraprocedurally via a fluoroscopy, bi-plane x-ray or the like).

[0047] As mentioned, the type of material utilized for the respective objects 302 can vary depending upon the imaging modality being utilized (e.g., to acquire 2D images, such as at 102 and 104). The marker device 300 is a multimodal marker and, as such, also includes one or more sensors 306 detectable by the tracking system. Each sensor 306 can be dimensioned and located to have a predetermined spatial relationship (e.g., distance and angle) relative to the geometry of the respective radio opaque objects 302. For example, the sensor 306 can include a sensor coil that is positioned at the origin of a pair of axes that can be computed with respect to the geometric relationship of the objects 302. Additionally, the sensor itself 306 can extend along an axis 308 or be parallel to an axis defined by the respective radio opaque objects 302.

[0048] As a further example, the marker device 300 defines a coordinate system that includes X and Z axes lying in a virtual plane that extends through each

-15Date Reçue/Date Received 2023-02-15 of the objects 302 arranged in a triangle (of the page), with a corresponding Y axis extending perpendicular to the virtual plane (e.g., the page in which the figure is demonstrated). In an example, the central axis 308 of the sensor 306 extends along the Z axis of the coordinate system. A geometric center of a body of the sensor 306, thus may define the origin of the X, Y, and Z axes. As mentioned, the sensor 306 can be configured as an elongated coil that extends axially along a length of the Z axis, and is detectable by the tracking system. For example, the sensor 306 is implemented as a coil of electrically conductive material within the marker device 300 and a center of the sensor coil is located the origin of a corresponding coordinate system. The sensor 306 provides a sensor signal (e.g., an induced current signal) that is communicated to the tracking system in response to an electromagnetic field generated by a field generator of the tracking system.

[0049] FIG. 6 demonstrates an example of marker pad device 320 that can help protect the patient’s skin from the hard surface of the combination markers. One or more of the multi-modal marker devices 300 (FIG. 5) can be implemented within the pad device 320 to enable co-registration between the domain of the tracking system and domain of the medical imaging modality, such as disclosed herein (e.g., fluoroscopy or the like). For example, the pad 320 can contain a gel or other soft flexible material to provide a cushion around each combination marker and the pad may be attached to the patient's skin. In other examples, the pad 320 is placed adjacent to the patient, such as on a bed next to or beneath the patient.

[0050] In the example of FIG. 6, the pad device 320 includes three of the multimodal markers 300 distributed in a spaced apart arrangement with respect to each other. The pad device 320 can be configured to hold each of the combination markers in a substantially fixed spatial relationship while allowing flexibility to accommodate patient movement. Each of the marker devices 300 also includes a corresponding connection 310 that can be coupled to the tracking system. For example, the tracking system can be implemented as an electromagnetic tracking system, such as disclosed herein, and each of the connections 115 thus includes an electrical connector to provide an electrical signal to the tracking system, representing induced current in response to an electromagnetic field that is generated by a transmitter of the tracking system and detected by the respective sensing coil. In other examples, the connections can be wireless and the sensors

-16Date Reçue/Date Received 2023-02-15 can communicate via RF or other wireless technology. The tracking system can convert the sensor signals into corresponding tracking system data, which can be analyzed as disclosed herein. For example, the tracking data can include a position and orientation of a point in a three-dimensional coordinate space of the tracking system (also referred to herein as tracking system space) for each marker device 104.

[0051] FIGS. 7A and 7B depict a schematic example of a medical imaging modality 350 that can be used to acquire two-dimensional projection images from different viewing angles. For example, the medical imaging modality 350 is configured to acquire images using an ionizing radiation (e.g., a fluoroscopy system, a portable X-ray system or the like). In this example, the imaging modality 350 includes an X-ray source 352 configured to provide ionizing radiation (e.g., X-rays) and a detector 354 that are attached and held at a desired spaced apart position by a moveable C-arm 356. A patient can be position along with one or more marker devices 360 between the source 352 and detector 354. Thus, images acquired can vary depending on the viewing angle (e.g., axis) of the source and detector as adjusted by adjusting the position of the C-arm. For registration purposes, the viewing angle includes a region of interest of the patient and the marker 360.

[0052] By way of example, the registration is performed by modeling the X-ray source 352 as an ideal pinhole camera (i.e., assuming no distortion), where each pixel in the resulting image is formed by projecting 3D points into the image plane

<img file="CA3131071C_D0001.tif" />

<img file="CA3131071C_D0002.tif" />

where:

X, Y, and Z are the coordinates of a 3D point in the common coordinate system;

u and v are the coordinates of the projection point in the camera image in pixels;

fx and fy are the focal lengths in pixel units;

ex and cy is the image center in pixel units; and

-17Date Reçue/Date Received 2023-02-15 r## and 1# define the position and orientation, respectively, of the X-ray detector in the common coordinate system.

[0053] To create the vector v1 or v2, the comers of the Aril co marker 360 are located in the image as u and v. The remaining values of the equation can be filled in based on the known spatial locations, and the equation is solved for X and Y at the focal length (e.g., distance between the detector and the respective comer location). The vector is then computed by subtracting the detector position (p1 or p2) from this new location. For example, points p1 and p2 are defined based on the rotation angle of the C-arm for the two projections, where the center of rotation is defined as the origin and the distance from the center is based on the C-arm radius. The focal length of the camera is computed from the pixel dimensions stored in the associated DI COM files.

[0054] The 3D position of the corner of the Arllco marker can then be computed by finding the intersection (or nearest approach) of the two vectors v1 and v2. The position and orientation of the ArUco marker in the common coordinate system is computed by repeating this process for all 4 corner locations identified for the fiducial marker in each of the respective images. By way of example, intersection (or nearest approach) of the two vectors may be computed according to a vector crossing function implementing a closest point function. As an example, the following pseudo-code implements a closest point function to determine a closest point between respective vectors:

vector ClosestPoint(vector p1, vector v1, vector p2, vector v2) { /1 normalize direction vectors v1 = nomnalize(vl);

v2 = normalize(v2);

// check that the vectors are not co-incident (parallel) float projDir = dot_product(v1, v2);

if (absolute_value(projDir) > 0.9999f) { // vectors are nearly co-incident (parallel) return p1 ;

}

-18Date Reçue/Date Received 2023-02-15

H compute nearest point float proj 1 = dot_product(p2 - p1, v1 );

float proj2 = dot_product(p2 - p1, v2);

float distl = (proj 1 - (projDir * proj2)) /(1- (projDir * projDir)); floatdist2 = (proj2 - (projDir* proj 1))/ ((projDir* projDir) -1); vector pointOnLinel = p1 + (distl * v1);

vector pointOnLine2 = p2 + (dist2 * v2);

return HnearJnterpolate(pointOnLine1, pointOnLine2,0.5f); }

[0055] In the example of FIGS. 7A and 7B, the marker 360 is demonstrated as an AruCo type radiopaque marker (e.g., corresponding to marker device 200, 250). In other examples, the medical imaging modality may be use with different types of marker devices, as disclosed herein, namely, the multi-modal marker device 300 and combination marker system 320. The respective transforms, such as determined at 108 and 154 (e.g., respective transformation matrices) thus can be used to enable rendering one or more visualizations based on the 2D image data acquired by the medical imaging modality 350, 3D preoperative image data and real-time tracking data.

[0056] FIG. 8 depicts an example of a system 450 for generating affine transformations. In this example, the affine transformations are demonstrated as transform matrices 452 and 453 for registering tracking data and image data, as disclosed herein. The system 450 is described in the context of data and instructions and a processor can access the data and execute the instructions to perform the functions disclosed herein. It is to be understood that not all functions may be required to implement the system. For example, each of the different transform matrices may be separately generated, which affords advantages when an imaging modality changes or is replaced in another implementation, the entire system does not need to be modified.

[0057] In the example of FIG. 8, the system 450 is configured for generating a first transform matrix (T1 ) 452. The transform matrix T1 may be configured to transform from a tracking system coordinate system of a tracking system 454 into a coordinate system of a medical imaging modality 456 (e.g., a 2D imaging system such as fluoroscopy or x-ray) and/or from the coordinate system of the medical imaging modality to the tracking coordinate system. The tracking system 454 is

-19Date Reçue/Date Received 2023-02-15 configured to provide tracking data 458 to represent a position and orientation of one or more sensors 466 being positioned within a patient’s body 460.

[0058] A combination marker system 462 (e.g., including one or more multimodal marker of FIGS. 4A, 4B, 5 or 6) can be attached to the patient’s body 460 or placed near the patient’ body. In the example of FIG. 8, the combination marker system 462 can include one or more tracking sensors 464 that provide respective sensor signals to the tracking system 454 representative of a location of the combination marker within the coordinate system of the tracking system 454. In an example, the one or more object sensors 466 can be affixed relative to an object that is movable within the patient’s body 460 for identifying a location of such sensor in the coordinate system of the tracking system. Each such object sensor 466 thus can also provide a signal to the tracking system 454 based on which the tracking system can compute corresponding tracking data representative of the position and orientation of such sensor in the tracking system coordinate system. As mentioned, the tracking data 458 thus represents a position and orientation of each respective object tracking sensor 466 as well as marker tracking sensors 464 of the multi-modal marker system 462.

[0059] In some examples, such as for purposes of generating the transform matrix 452, the sensor(s) 466 and corresponding tracking data 458 may be ignored (or omitted). In other examples, the sensor466 may be placed at a known location with respect to the patient’s body 460 (e.g., a known anatomical landmark within or external to the patient’s body) to provide additional data points, in both the tracking system spatial domain (e.g., provided by tracking data 458) and the spatial domain of the imaging modality 456 (e.g., provided by 2D image data 472 at the known location) that may be used to facilitate generating the transform matrix T1 452.

[0060] By way of example, the tracking system 454 can include a transmitter (e.g., an electromagnetic field generator) that provides a non-ionizing field, demonstrated at 455, which is detected by each sensor 464 and 466 to provide a corresponding sensor signal to the tracking system. An example tracking system 454 is the AURORA spatial measurement system commercially available from Northern Digital, Inc., of Ontario, Canada. The tracking system 454 can provide the tracking data 458 at an output sample rate (e.g., sixty samples per second) for each sensor sufficient to enable substantially real time determination of sensor location

-20Date Reçue/Date Received 2023-02-15 (e.g., to provide a vector describing sensor position and orientation). A tracking processing subsystem thus can process each frame of tracking data such that the tracking data can likewise represent real time tracking data acquired by the tracking system that can be registered into another coordinate system by applying one or more ofthe generated transforms 452 and/or 453 to facilitate generating a graphical representation in a given domain, as disclosed herein.

[0061] The tracking system 454 may provide the tracking data 458 with an output sample rate to enable computation of real time positioning and visualization of the object to which the sensor is attached as well as the combination marker system. Since the marker system 462 is attached to the patient’s body 460, the tracking system 454 computes the tracking data 458 to accommodate for movement in the patient’s body 460 in the coordinate system ofthe tracking system 454.

[0062] A sensor transform 470 is configured to convert the tracking data 458 into locations for radiopaque objects implemented on each respective marker device, such as disclosed herein. Each of locations are 3D spatial coordinates in tracking system coordinate space and may remain fixed if the marker device does not move in the tracking space or may vary over time if the marker device moves in tracking space. For example, in the tracking coordinate system, each ofthe radiopaque markers of a given marker device are at fixed, known offsets (e.g., a 3D vector) from the location of the tracking sensor 464 that is part of the given marker device of marker system 462. As mentioned, the marker system may include a plurality of multi-modal marker devices, such as AruCo type (e.g., device 250), or other marker configurations (e.g., device 300) as disclosed herein.

[0063] The sensor transform 470 thus is configured to compute the points (e.g., 3D coordinates for marker locations) in the tracking system space based on the tracking data 458 and the known offsets for each tracking sensor relative to the predetermined marker locations. For the AruCo type multi-modal marker device, the marker locations may be a set of four points (e.g., emPoint_1, emPoint_2 , emPoint_3, emPoint_4) at the corners of the marker, such as disclosed herein. For example, the points in tracking system space for a set of marker locations of the AruCo type marker device having a sensor providing tracking data may be computed for a given marker device by multiplying the sensor transform (TS), which includes tracking sensor 3D coordinates, and the respective offset, as follows:

-21 Date Reçue/Date Received 2023-02-15 emPoint_1 = mult(Ts, offset___1), emPoint_2 = mult(Ts, offset_2), emPoint_3 = mult(Ts, offset_3), and emPoint_4 = mult(Ts, offset_4)

[0064] For the example of a marker device (e.g., for marker device 300) that includes an arrangement of spherical radiopaque markers, there are 3 spherical markers at known offsets distributed around each tracking sensor. Accordingly, the sensor transform will generate three points for each marker device in the marker system 462. For example, the transform 470 can determine marker locations at points (e.g., emPoint_1, emPoint_2 , emPoint_3) located at the center of each of the spherical marker based on multiplying the respective transform and the known offset (e.g., 3D offset vector) between the tracking sensor location (e.g., a 3D point) and the respective radiopaque objects, such as follows:

emPoint_1 = mult(Ts, offset_1), emPoint_2 = mult(Ts, offset_2), and emPoint_3 = mult(Ts, offset_3).

Other deterministic locations having fixed offsets associated with the radiopaque markers may be used in other examples. In some examples the points may be arranged in a set of point for each marker device or as a single set that contains all the points.

[0065] The medical imaging modality 456 is configured to generate 2D image data 472 that includes at least two images (e.g., radiographs) representing objects within a field of view 475 of the imaging modality 456. For example, the imaging modality may include a fluoroscopy scanner (e.g., the system of FIGS. 7A and 7B) configured to acquire the 2D image data for a small number of (e.g., at least two, three or four) 2D projection images acquired at different viewing angles relative to the patient’s body 460. Each of the images in the image data 472 may be acquired to include a 2D projection for radiopaque markers in each marker device of the marker system 462 and a region of the patient’s body 460 within the field of view 475. In some examples, the region of the patient's body may be a region of interest in which the object sensor 466 is to be moved, such as part of a surgical procedure.

-22Date Reçue/Date Received 2023-02-15

[0066] A marker identification function 474 can be configured to locate each radiopaque marker (e.g., AruCo marker and/or other object marker) in each image provided in the image data 472. The radiopaque markers will be visible in the images due to their opacity with respect to the ionizing radiation emitted by the imaging modality 456. For the example of the combination marker that invoke an AruCo detection function to locate each respective marker. For an example combination marker that includes a radiopaque object other than an AruCo type marker, a periphery of each such marker may thus be localized by image thresholding as well as other image processing techniques applied to values of image pixels. The marker identification function 474 may be fully automated and/or be user-interactive in response to a user input identifying the markers. The identified markers (e.g., pixel locations in the respective images) may be stored in memory for further processing.

[0067] A marker point generator 476 is programmed to generate spatial coordinates for each marker identified in the (e.g., two or more) images provided by the image data 472. For the example of the combination marker that includes a radiopaque AruCo type marker, the spatial coordinates may be generated for each of the corners of each marker, namely, coordinates for a set of four points surrounding each tracking sensor. For spherically shaped radiopaque markers, the spatial coordinates for each marker are provided as 2D coordinates at a center of the circular projection (e.g., the periphery identified by marker identification function 474) in each 2D image for the viewing angle provided by the field of view 475 relative to the marker system 462. In an example where three spherical markers surround each tracking sensor for a given marker device, the marker point generator is programmed to provide coordinates for a set of three points for the given marker device. Regardless of the type and configuration of radiopaque marker, the marker point generator for example, is programmed to execute a closest point function such as disclosed herein, to locate the set of points around each respective tracking sensor for the marker device. In this way, each set of points can be linked together and associated with a respective one of the tracking sensors to facilitate generating the first transform matrix 452.

[0068] A first transform calculator 478 is programmed to compute the first transform matrix 452 based on the points provided by the marker point generator

-23Date Reçue/Date Received 2023-02-15

476 and the sensor transform function 470. For example, the transform calculator 478 is applied to align the sets of points that have been measured in the spatial coordinate systems. Examples of such co-registration algorithm to co-registerthe points in the respective domains (e.g., tracking system coordinate system and medical imaging coordinate system) may include an error minimization function or a change of basis function.

[0069] As one example, the transform calculator 478 is programmed to implement an error minimization function. Given the ordered set of points, the transform calculator 478 is to determine unknown transform T1 that minimizes the distance between the projected location and the measured location. For example, for T1 we want to find the transform that minimizes the distance between points, such as follows:

sum(n = 1..i, distance(mult(T1, imPoint_n), emPoint_n)^A2) where: n denotes a given one of i points (i is the number of points for a given multi-modal marker;

imPoint_n is the spatial coordinates in image space for point n; and emPoint_n is the spatial coordinates in tracking space for point n.

In an example, the error minimization can be solved through Single Value Decomposition or any number of error minimization algorithms.

[0070] As another example, the transform calculator 478 is programmed to implement a change of basis function. If the points used are arranged in a way that allows us to generate a set of basis vectors (x, y, and z unit vectors that define the coordinate space) then a simpler solution is possible compared to error minimization. For example, rather than minimizing the errors, the transform calculator 478 is programmed to find the basis vectors in both coordinate systems and apply them at a common point. This is computationally more efficient than the error minimization approached mentioned above, but requires a specific arrangement of points.

[0071] By way of example, to unambiguously define the basis vectors, the arrangement needed is 3 points at a 90 degree angle, with enough additional information to allow us to identify which point is which (for example, having the legs of the triangle created by the 3 points be different lengths). Both the ArUco-type marker of FIGS. 3 and 4 and the marker devices of FIGS. 5 and 6, have

-24Date Reçue/Date Received 2023-02-15 arrangements of points sufficient enable the use of such change of basis function, with the caveat being that for the marker device of FIGS. 5 and 6, each set of 3 points needs to be treated separately.

[0072] In each coordinate system, the transform calculator 478 constructs the basis vectors from 3 points. For example, given point_1, point_2, and point_3 (e.g., vertices of a right triangle), provides two segments, one from point_2 to point_1 and another from point_2 to point_3, which segments are the legs of a right triangle.

These points and segments provide the following basis vectors:

basis_z = normalize(point_1 - point_2) basis_x = normalize(point_3 - point_2) basis_y = cross(basis_x, basis_z)

[0073] From the basis vectors, the transform calculator 478 is programmed to create a matrix (e.g., a 4x4 matrix) that defines the position and orientation of point_2 as follows:

matrix =

[ basis_x.x, basis_y.x, basis_z.x, point_2.x, basis_x.y, basis_y.y, basis_z.y, point_2.y, basis_x.z, basis_y.z, basis_z.z, point_2.z, 0, 0, 0, 1]

[0074] With that matrix defined in each coordinate system, the transform calculator 478 can compute the transform 452 between the two coordinate systems. For example, for the transform matrix T1 :

im_Matrix is the matrix defined from the basis vectors in the medical imaging (e.g., intraoperative) coordinate system; and em_Matrix is the matrix defined from the basis vectors in the tracking coordinate system.

From the above, the transform calculator 478 may determine the transform matrix (T1) 452 by multiplying the basis vector tracking matrix (em_Matrix) and the inverse of the basis vector imaging matrix (inv(im_Matrix)), such as follows:

-25Date Reçue/Date Received 2023-02-15

T1 = mult(em_Matrix, inv(im_Matrix))

The transform matrix may be stored in memory and used for transforming from the tracking system space to the medical imaging space. For example, the position of the object sensor 466 within the patient’s body, as represented by tracking data 458, may be registered into the medical imaging space by applying the transform T1 to the position and orientation information of the tracking data.

[0075] As mentioned, the system 450 also is configured to generate the second transform (T2) 453 for use in transforming between the medical imaging coordinate system and a coordinate system of prior 3D image data 480. For example, the prior 3D image data 480 may be stored in memory (e.g., as a DICOM image set) and include a 3D image from a preoperative scan (e.g., CT scan) of the patient’s body 460 that is performed at a time prior to when the medical imaging modality 456 generates its image data 472 (e.g., intraoperatively, such as corresponding to images acquired at 102 and 104).

[0076] A projection calculator 482 (e.g., corresponding to the function 152 of FIG. 2) is programmed to generate a respective projection from the 3D image data 480 for each of the images (e.g., two images) provided in the 2D image data 472. The projection calculator 482 implements a function to map the points from the 3D image space onto a two-dimensional plane. For example, the projection calculator derives forward projections that are aligned with the viewing angles of the images in the 2D image data 472. The registration of projection angles for each of the 3D projections may be implemented through manual alignment and/or be automated. In an example, the alignment may be automated, such as based on image metadata (demonstrated as included in the arrow from the 2D image data 472 to projection calculator 482) in the image data 472 that describes the angle of each of the 2D images. For example, the metadata includes data specifying the projection angle, such as AP, LAO, RAO, such has may be known from the angle of a C-arm and/or be provided in response to a user input when the imaging modality 456 acquires the image data 472.

[0077] In some examples, as disclosed herein the 3D image data may include a model of one or more anatomical structure, such as in the form of a 3D mesh corresponding to a surface of a vessel. A 3D projection matrix (e.g., perspective or parallel projection matrix) may be applied to the mesh that was generated from the -26Date Reçue/Date Received 2023-02-15 pre-operative image 480, such as disclosed herein. If the angle of the C-arm is known for each of the intraoperative images, one 3D projection of the mesh is performed to match the angle for each intraoperative image. If the angle of the Carm is not known, multiple 3D projections may be generated along different angles, and there may be a manual or automated selection of a best fit match between the respective 3D projections and the respective two-dimensional image.

[0078] A point generator 484 is programmed to generate spatial points in each of the 2D images (provided by image data 472) and the corresponding projections of the 3D image (provided by projection calculator 482). Rather than working with spheres or comers of markers, the points are selected as features that are visible in both the 2D image data 472 and the 3D image data 480. For example, the features include structures such as bony landmarks on the spine, bits of calcification that are visible in both types of images, or points on vessels in an example when contrast is used in both images. Other feature or fiducial points may be used in other examples. In some examples, a common set of features may be located in an automated method (e.g., feature extraction). Additionally or alternatively, one or more such features may be selected in response to a user input provided through a user interface 486, such as graphical user interface interacting with the respective images and projections provided to the point generator. For instance, a user may see a common visible structure among the different views and select/tag it (e.g., through a mouse, keyboard, gesture or other input) in each view. The point generator 484 thus generates points for each predetermined feature and/or user selected feature. The point generator thus operates similarly to the marker point generator 476, just using a different set of landmarks. Since the image data 480 are in 3D, in some examples, the user can identify selected points (through user interface 486) using a set of orthogonal views (e.g., axial, coronal, and sagittal views) of the 3D images of image data 480 to directly measure the x, y, and z locations in the 3D coordinate system of the image data 480. Each of these locations may be converted to two-dimensional coordinates and provided as such in the forward projections provided by the projection calculator 482. The point generator 484 is programmed to locate the same points in the 2D image data, such as by using a vector-crossing function applied to the 2D images, such as the closest point function disclosed herein.

-27Date Reçue/Date Received 2023-02-15

[0079] The resulting points in the respective images are provided to a second transform calculator 488 for generating the transform matrix 453. The transform calculator 488 is programmed to compute the transform matrix that aligns the images of the second image data with the 3D image data 480 based on the common points provided by the point generator 484. For example, the transform calculator 488 constructs the transform matrix (T2) 453 by implementing an error minimization function with respect to the common set of points, such as single value decomposition described with respect to the first transform calculator 478. Other error minimization functions may be used in other examples.

[0080] In some examples, the system 450 includes a transform correction function 490 programmed to implement manual corrections to one or both of the transform matrices 452 and 453 based on instructions provided via a correction user interface 492. Manual corrections can be applied even if we start with an estimate of theT1 or T2 transform. For example, if the image data 480 and/or 472 does not have a well-defined set of measured points (e.g., on the spine or other anatomic structure) to work from to perform the registration, the system may define an initial estimate for the transform T2 or, in some examples, an arbitrary T2 transform (e.g. an ‘identity’ matrix) and allow the user to make corrections through the correction function 490 to generate the final T2 transform 453.

[0081] By way of further example, with reference to FIG. 9, a registration manager 500 is used to control user corrections to one or both transforms T1 and T2, 452 and 453, respectively. The registration manager may be implemented as part of the system 450 of FIG. 8 or as a separate function. Accordingly, for consistency, functions and data introduced in FIG. 8 are depicted in FIG. 9 using the same reference numbers. Reference may be made back to FIG. 8 and the corresponding description for further information about such functions and data.

[0082] The registration manager 500 includes the transform correction function 490 as well as the first and second transform matrices 452 and 453, respectively. In this example, it is assumed that one or both of the transform matrices 452 and 453 may be in need of correction. The need for correction may be made manifest to a user by applying a transform to register two or more domains and provide a resulting visualization on a display 510. For example, an output generator 512 is configured to render a visualization in a selected domain, such as

-28Date Reçue/Date Received 2023-02-15 may be the coordinate system of the tracking system, the coordinate system of the medical imaging modality 456 or the coordinate system of the prior 3D image data 480. In an example, the manager 500 includes a domain selector 514 programmed to select which domain the output visualization is being rendered based on a user input instruction received via a user interface 520. Additionally, based on the selected domain, the registration manager applies one or both of the transforms T1 orT2 accordingly. As an example, the following table provides a description of which one or more transforms are applied to the image data 472, 480 or tracking data 458 for each selected domain to which the output visualization is being rendered by the output generator 512. The registration manager 500 further may be used to control the application of the respective transforms to provide a visualization in a selected domain, such as by applying one or more transforms or inverses of such transforms as set forth in the table.

<td></td><td> Tracking</td><td> Medical Imaging</td><td> Prior 3D</td>

<td> to Tracking:</td><td> [identity]</td><td> inv(T1)</td><td> inv(T1) inv(T2)</td>

<td> to Medical Imaging :</td><td> T1</td><td> [identity]</td><td> inv(T2)</td>

<td> to Prior 3D:</td><td> T2 T1</td><td> T2</td><td> [identity]</td>

[0083] As a further example, manual corrections to either transform 452 or 453 can be provided by multiplying the respective transform matrix T1 orT2 by a correction matrix, such as follows:

correctedTl = mult(correctionMatrix, T1) or correctedT2 = mu Incorrection Matrix, T2)

In an example, the supported types of corrections include translation, rotation and scaling, such as may be applied in the form of matrices, as follows:

translationMatrix = [ 1, 0, 0, translation.x, 0,1,0, translation.y, 0, 0,1, translation.z, 0, 0, 0,1]

-29Date Reçue/Date Received 2023-02-15 scalingMatrix = [ scale, 0, 0, 0, 0, scale, 0, 0, 0, 0, scale, 0, 0, 0, 0,1] rotation Matrix = (depends on axis of rotation)

[0084] By way of further example, a user initiates corrections using mousedown/drag/mouse-up actions or other actions on the user interface 516. The values used in the correction matrix may be set based on the projection matrix used to display the viewport on the display 510. For example, a translation initiated from an AP view would result in the X and Y mouse movements being used to set translation.x and translation.z values (translation.y would be 0). Such transformations thus allow the user to change the view of a single image or the alignment of multiple images.

[0085] As a further example, such as when implementing corrections for transform T2, the domain registration manager 500 applies the transform T2 to the image data 472 and the output generator 512 provides a visualization of the 2D images registered in the 3D image based on the transform T2. If the landmarks are properly aligned, as shown on the display 510, no correction may be needed.

However, if the locations of landmarks in the 2D image do not align with their respective locations in the 3D image, correction may be needed to T2. A user thus can adjust the alignment of the 2D image with respect to the 3D image (or the forward projection thereof) through the user interface 516. As mentioned, the adjustments may include translation in two dimensions, rotation and/or scaling in response to instructions entered through the user interface using an input device (e.g., mouse or keyboard). The output generator 512 may update the visualization shown in the display to show the image registration in response each adjustment (e.g., in real time). Once a desired alignment is visualized, the user can employ the user interface 516 to apply and store the corrections to the transform T2, and an updated T2 may be stored in memory for subsequent applications. Similar types of adjustments may be made with respect to the first transform matrix 452.

[0086] FIGS. 10A-10B demonstrate examples of images 600 and 630 that may be generated and visualized in a display (display 510) according to application

-30Date Reçue/Date Received 2023-02-15 of one or more transforms before and after manual correction (e.g., implemented by correction function 490). In both FIGS. 10A and 10B, the images include a common set of anatomical features and marker features. For example, the anatomical feature includes vertebrae of a spine 602. The marker features for a given marker device 610 include locations of radiopaque markers 604 and a tracking sensor 606 that have been mapped by applying a transform from one image space (e.g., the tracking system coordinate system) to another (e.g., the medical imaging coordinate system). The marker features for the marker device 610 also include locations of radiopaque markers 608 in their original domain (e.g., the medical imaging coordinate system). The images also include marker feature for another marker device 618, including locations of radiopaque markers 612 and a tracking sensor 614 that have been mapped by applying a transform from one image space (e.g., the tracking system coordinate system) to another (e.g., the medical imaging coordinate system) as well as locations of radiopaque markers 616 in their original domain (e.g., the medical imaging coordinate system).

[0087] As shown in FIG. 10A, the respective marker features and anatomical feature are not aligned due to misregistration. As disclosed herein, the output display is interactive and includes a GUI element 622 that may be used to adjust (e.g., translate, rotate or scale) one of the images relative to the other. In response to such adjustments the correction function 490 generates one or more correction matrices, such as disclosed herein. Once a desired alignment is achieved, such as shown in the visualization 630 of FIG. 10B, the user can activate a Freeze GUI element (a button) 624 to apply the corrections to the appropriate transform(s), and the updated transform can be stored in memory for further applications.

[0088] In some examples, after the first step is performed with an AP image, the system should have registration in two dimensions - patient left-to-right and head-to-foot. The procedure above may be repeated with a lateral or oblique view to get the third dimension - patient anterior to posterior. In a further example, if a user were to advance a catheter or guidewire (e.g., including a tracking sensor) near the aortic bifurcation or other identifiable landmark, a rough registration in that third dimension could be performed without a second 2D medical image.

[0089] By way of example, the two-dimensional image data may include one two-dimensional image acquired by the medical imaging modality (e.g.,

-31 Date Reçue/Date Received 2023-02-15 intraoperatively) to include the patient and the multi-modal marker, such that predetermined portions of the multi-modal marker being visible in the 2D image and have a known location and orientation with respect to at least one tracking sensor detectable by a tracking system. A three-dimensional position for predetermined portions of the multi-modal marker may be estimated with respect to a coordinate system of the medical imaging modality according to respective locations of the predetermined portions of the mark in each one (or more) of the two-dimensional images. An affine transformation is determined for registering the three-dimensional coordinate system of the tracking system with the three-dimensional coordinate system of the medical imaging modality based on the estimated position for the respective predetermined portions of the multi-modal marker and the known (a priori) relationship of the at least one tracking sensor (e.g., including the tracking sensor on the guidewire or catheter at the known landmark) and the predetermined portions of the multi-modal marker. Thus, a second 2D image may be omitted and still enable registration.

[0090] In view of the foregoing structural and functional description, those skilled in the art will appreciate that portions of the systems and method disclosed herein may be embodied as a method, data processing system, or computer program product such as a non-transitory computer readable medium. Accordingly, these portions of the approach disclosed herein may take the form of an entirely hardware embodiment, an entirely software embodiment (e.g., in one or more nontransitory machine-readable media), or an embodiment combining software and hardware. Furthermore, portions of the systems and method disclosed herein may be a computer program product on a computer-usable storage medium having computer readable program code on the medium. Any suitable computer-readable medium may be utilized including, but not limited to, static and dynamic storage devices, hard disks, optical storage devices, and magnetic storage devices.

[0091] Certain embodiments have also been described herein with reference to block illustrations of methods, systems, and computer program products. It will be understood that blocks of the illustrations, and combinations of blocks in the illustrations, can be implemented by computer-executable instructions. These computer-executable instructions may be provided to one or more processor of a general-purpose computer, special purpose computer, or other programmable data

-32Date Reçue/Date Received 2023-02-15 processing apparatus (or a combination of devices and circuits) to produce a machine, such that the instructions, which execute via the processor, implement the functions specified in the block or blocks.

[0092] These computer-executable instructions may also be stored in computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture including instructions that implement the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0093] What have been described above are examples. It is, of course, not possible to describe every conceivable combination of components or methodologies, but one of ordinary skill in the art will recognize that many further combinations and permutations are possible. Accordingly, the invention is intended to embrace all such alterations, modifications, and variations that fall within the scope of this application, including the appended claims. As used herein, the term “includes” means includes but not limited to, the term “including” means including but not limited to. The term “based on” means based at least in part on. Additionally, where the disclosure or claims recite a, an, a first, or another element, or the equivalent thereof, it should be interpreted to include one or more than one such element, neither requiring nor excluding two or more such elements.

-33Date Reçue/Date Received 2023-02-15