System and method for rapidly identifying pathogens, bacteria and abnormal cells
Summary by NHIP
Pathogen identification system
The method processes specimen images using a parallel computing platform to isolate features via recursive hierarchical segmentation. It then extracts data with a mining algorithm and compares results against reference images using an optical recognition algorithm.
Claim Score by NHIP
Abstract
The present invention achieves rapid identification of pathogens, bacteria, cancer cells and other abnormal human and animal cells. In one embodiment, the system of the present invention comprises a first subsystem that obtains and processes images of specimens of pathogens, bacteria, and other abnormal cells, and a second subsystem that accepts the images, isolates the particular features of the image using advanced image segmentation, and then rapidly and accurately identifies the pathogens, bacteria and other abnormal cells by using a pattern recognition process wherein the segmented or isolated features of the original image are compared to known reference images.

Term
Term ended
Expired 9 January 2025, 1.7 years ago.
- Priority
- Filed
- Granted
- Expired
- Today
12 claims: 2 independent, 10 dependent
- 1Broadest claimClaim Score 61, broad(NHIP)A method for processing an image of a specimen or dental x-ray to identify a pathogen within the specimen or dental x-ray, comprising the steps of:providing an image of a specimen or dental x-ray;providing a parallel processing computing platform;implementing a recursive hierarchical segmentation algorithm on the parallel processing computing platform and processing the image with the recursive hierarchical segmentation algorithm to isolate at least one segment of the provided image that has a feature that is of interest;processing the isolated segment with a data mining algorithm to extract particular image data from the isolated segment;and processing the extracted particular image data and at least one reference image with an optical recognition algorithm to determine if the extracted image data matches the at least one reference image.
- 9A system for processing an image of a specimen or dental x-ray to identify a pathogen within the specimen or dental x-ray, comprising:a device to provide an image of a specimen or dental x-ray: a digitizer to digitize the provided image;a first data storage device to store the digitized images;a second data storage device having at least one reference image stored therein;a parallel processing computing platform configured to implement a recursive hierarchical segmentation algorithm, a data mining algorithm and an optical recognition algorithm;a work station computer in data communication with the parallel processing computing platform, the work station computer comprising electronic data communication hardware and software that enables the work station computer to control the parallel processing computing platform to (i) process the digitized image with the recursive hierarchical segmentation algorithm to isolate at least one segment of the digitized image that has a feature that is of interest (ii) process the isolated segment with the data mining algorithm to extract particular image data from the isolated segment, and (iii) process the extracted particular image data and each of the reference images with the optical recognition algorithm to determine if the extracted image data matches any of the reference images;and a display device in data communication with the work station computer to display the extracted image data and the results of processing the extracted image data and the reference image with the optical recognition algorithm.
Independent claims2
52 paragraphs in 6 sections, as filed
CROSS-REFERENCE TO RELATED APPLICATIONS
0001This application is the national stage of International Application PCT/US04/09172, filed Mar. 25, 2004, which claims the benefit of commonly owned and copending U.S. Provisional Application Ser. Nos. 60/458,769, filed Mar. 27, 2003, and 60/505,944, filed Sep. 25, 2003.
TECHNICAL FIELD
0002The present invention generally relates to a system and method for identifying pathogens and abnormal cells.
BACKGROUND ART
0003The timely diagnosis of pathogens, bacteria, abnormal cell and infectious diseases is often complicated by the need to use cultures as the means to identify the bacteria and select the optimum treatment. Currently, identification of pathogens often takes days and involves complicated procedures, a situation that may unduly delay effective treatment such as the appropriate selection of an optimal antibiotic. Similar problems exist in detecting bacterial contamination in food, especially in beef, poultry and fish. The delay in identifying the presence of harmful bacteria in food products could result in contaminated products being released for distribution and consumption with dire consequences. In some instances, these delays have proved to be fatal to patients or have caused unnecessary suffering. According to 1999 statistics provided by the Center for Disease Control, there were 1,194,959 reported cases of infectious diseases caused by bacteria. Furthermore, there were many instances of food poisoning that were not subject to mandatory reporting to the Center for Disease Control. A common practice in treating infected patients is the use of broad-spectrum antibiotics. However, due to the problem of bacterial resistance to many antibiotics, broad-spectrum antibiotics may not be effective. Many of these cases of infectious diseases could have been prevented or promptly treated if rapid and accurate diagnosis was available. Rapid identification of pathogens, bacteria and abnormal cells is also critical in dealing with bio-terrorism and with biological agents during warfare.
DISCLOSURE OF THE INVENTION
0004The present invention achieves rapid identification of pathogens, bacteria and other abnormal human and animal cells. In one embodiment, the present invention is directed to a non-invasive system and method for automatically and rapidly identifying pathogens. In accordance with one embodiment of the invention, the system comprises a first subsystem that obtains and processes images of specimens of pathogens, bacteria or other abnormal cells, and a second subsystem that accepts the images of the specimens, isolates the particular features of each image using advanced image segmentation, and then rapidly and accurately identifies the pathogens, bacteria or abnormal cell structure using pattern recognition processing on the particular isolated features.
0005In one embodiment, the first subsystem described in the foregoing description comprises an image capturing system that comprises a microscope and a video camera. The image capturing system captures or acquires an image of a specimen of a pathogen, bacteria or abnormal cell structure, and then enhances, digitizes and temporarily stores the pertinent parts of the captured or acquired image of the specimen. The first subsystem further comprises a communication system that transmits the processed image to the second subsystem via any one of a variety of suitable communication schemes such as satellite links, the Internet, or telephone lines. In a preferred embodiment, the first subsystem further includes a computer, microprocessor or other controller to control the operation of the first subsystem. In a preferred embodiment, the first subsystem is configured to have automatic operation so as to minimize the manual effort in processing the image of the specimens.
0006In one embodiment, the second subsystem is typically located at a central location. The second subsystem receives the processed image transmitted by the first subsystem. The second subsystem comprises an image processing system that processes the images received from the first subsystem so as to isolate certain features of the image of the specimens that are of interest. This image processor effects image segmentation to isolate the aforementioned features of the image. The second subsystem comprises a database that contains known reference images. Such a data base functions as a library of images of known pathogen cells, bacteria cells and abnormal cells. Each reference image is associated with a known pathogen, bacteria or abnormal cell structure. The image processing system implements a data mining program that extracts particular image data from the isolated features and a pattern recognition program that compares the extracted image data to the known reference images in the database in order to determine if the isolated feature corresponds to or matches any of the known reference images.
0007The system and method of the present invention can also be used as a diagnostic radiology and imaging tool in the medical and dental field. Specifically, the system and method of the present invention can be configured to analyze medical images such as images of soft tissue, mammograms, x-rays (bone and dental), ultrasounds, MRI images, and CAT scans.
0008In another embodiment, the system is configured so that the first subsystem and second subsystem are joined together to form one main system that is located at one location. Such a configuration would be suitable for a large city hospital or one of the many teaching hospitals in the United States and throughout the world.
0009Thus, the present invention is directed to, in one aspect, a method for identifying pathogens, comprising providing an image, processing the provided image with an image segmentation algorithm to isolate at least one segment of the provided image that has a feature that is of interest, and comparing the isolated segment of the provided image to a plurality of reference images to determine if the isolated segment corresponds to any of the reference images.
0010In a related aspect, the present invention is directed to a system for identifying pathogens, comprising a device to provide an image, a data base having at least one reference image stored therein, and an image processing resource to (i) process the provided image with an image segmentation algorithm to isolate at least one segment of the provided image that has a feature of interest, and (ii) to compare the isolated segment of the provided image to the reference image to determine if the isolated segment corresponds to the reference image.
BRIEF DESCRIPTION OF THE DRAWINGS
0011The features of the invention are believed to be novel. The figures are for illustration purposes only and are not drawn to scale. The invention itself, however, both as to organization and method of operation, may best be understood by reference to the detailed description which follows taken in conjunction with the accompanying drawings in which:
0012<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of the system of the present invention.
0013<figref idref="DRAWINGS">FIG. 2</figref> is a perspective view of one embodiment of an imaging subsystem shown in <figref idref="DRAWINGS">FIG. 1</figref>.
0014<figref idref="DRAWINGS">FIG. 3</figref> is a perspective view of the rear side of the imaging subsystem of <figref idref="DRAWINGS">FIG. 2</figref>.
0015<figref idref="DRAWINGS">FIG. 4</figref> is a flow chart illustrating the operation of the imaging subsystem shown in <figref idref="DRAWINGS">FIG. 1</figref>.
0016<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of an image management diagnostic subsystem shown in <figref idref="DRAWINGS">FIG. 1</figref>.
0017<figref idref="DRAWINGS">FIGS. 5A-5D</figref> show a flow chart illustrating the operation of the image management diagnostic subsystem shown in <figref idref="DRAWINGS">FIG. 5</figref>.
0018<figref idref="DRAWINGS">FIG. 6</figref> is a flow chart illustrating a cluster scheduling process used by the image management diagnostic subsystem shown in <figref idref="DRAWINGS">FIG. 5</figref>.
MODES FOR CARRYING OUT THE INVENTION
0019Referring to <figref idref="DRAWINGS">FIG. 1</figref>, there is shown a block diagram of a system for rapid identification of pathogens, bacteria and abnormal cell structures in accordance with the invention. System <b>100</b> generally comprises imaging subsystem <b>100</b><i>a </i>and image management diagnostic subsystem <b>100</b><i>b</i>. Subsystem <b>100</b><i>a </i>generally comprises computer or controller <b>101</b>, staining module <b>102</b>, microscope <b>104</b>, digital color video camera <b>106</b>, image memory <b>108</b> and communications module <b>110</b>. As will be apparent from the ensuing description, computer <b>101</b> controls the operation and the sequence of operation of microscope <b>104</b>, digital color video camera <b>106</b>, image memory <b>108</b> and communications system <b>110</b>.
0020Referring to <figref idref="DRAWINGS">FIG. 1</figref>, staining module <b>102</b> stains the slides of specimens of pathogens, bacteria and abnormal cells that are affixed to slides. The slides are stained prior to being viewed with microscope <b>104</b>. In a preferred embodiment, staining module <b>102</b> is a commercially available immune staining procedures module. One such suitable commercially available immune staining procedures module is known in the art as motorized furescence filters for accurate color imaging of the stained cells. In a preferred embodiment, between five and ten different stains are selected to stain a predetermined number of slides for a given specimen in order to ensure that at least one of these slides has a pathogen, bacteria or abnormal cell stained to produce an acceptable image.
0021In one embodiment, statistical analysis is used to determine a sufficient number of specimen slides that are needed to ensure that at least one of the slides contains the offending pathogen, bacteria, etc. Staining module <b>102</b> is configured to utilize a standard set of stains to cover the range of pathogens, bacteria, etc. of interest.
0022Referring to <figref idref="DRAWINGS">FIG. 1</figref>, microscope <b>104</b> is configured to provide sufficient magnification and includes an oil immersion objective, an optical port for video camera <b>106</b>, an auto stage mechanism, and an auto focus mechanism. The auto stage mechanism comprises a shallow well for the convenient placement of the specimen slides. The automatic stage mechanism performs a raster scan of each slide while the auto focus mechanism maintains the image in focus. The auto stage mechanism is configured to stop briefly at each step to allow an image to be acquired. Each acquired image is assigned the x-y coordinates of the position of the auto stage mechanism. These x-y coordinates are automatically added in an appropriate format to the acquired image of the specimen.
0023Referring to <figref idref="DRAWINGS">FIG. 1</figref>, video camera <b>106</b> is controlled by computer or controller <b>101</b> to capture or acquire a color image of the specimen at each stop of the auto stage mechanism of microscope <b>104</b>. Video camera <b>106</b> is configured to provide adequate resolution and stability. Video camera <b>106</b> digitizes the acquired image. The digitized image is then transferred to image memory <b>108</b>. Image memory <b>108</b> is a temporary memory having sufficient data storage capacity to temporarily store the acquired images generated by video camera <b>106</b>.
0024In a preferred embodiment, microscope <b>104</b> and video camera <b>106</b> are realized as a single, commercially available compact unit which combines the functions of both microscope <b>104</b> and video camera <b>106</b>. One such commercially available unit is the Leica Model DMRXA2 Microscope. Other suitable, commercially available devices that combine a microscope and video camera into a single unit may be used as well.
0025In an alternate embodiment, the acquired images are pre-screened and presorted for useful and relevant content. This is accomplished by a screening processor and display device (both of which not being shown) that is in electronic data communication with image memory <b>108</b>. This pre-screening and presorting function ensures that further analysis is performed only on images having relevant information. The screening processor utilizes predetermined criteria (descriptors) to determine whether the images have relevant content.
0026Referring to <figref idref="DRAWINGS">FIG. 1</figref>, computer <b>101</b> controls image memory <b>108</b> to transfer stored digitized images into communications module <b>110</b>. In one embodiment, communications module <b>110</b> includes RF (radio frequency) antenna <b>111</b>. However, communications module <b>110</b> is preferably configured to transmit the digitized images to second subsystem <b>100</b><i>b </i>via any one of a variety of suitable communications modes, e.g. telephone lines, the Internet, dedicated lines or RF communication or communication through satellite communication. In accordance with the invention, the communications link between first subsystem <b>100</b><i>a </i>and second subsystem <b>100</b><i>b </i>is bi-directional. In a preferred embodiment, the communication between first subsystem <b>100</b><i>a </i>and second subsystem <b>10</b><i>b </i>is real time. In one embodiment, communications module <b>110</b> is realized as a DSL Speedstream Model 5260.
0027In a preferred embodiment, a suitable, commercially available PC (personal computer) high end system is used to realize control module <b>101</b> and image memory <b>108</b>.
0028In an alternate embodiment, subsystem <b>100</b><i>a </i>can be realized by separate, suitable commercially available components. For example, microscope <b>104</b> can be realized by a suitable, commercially available electronic or digital microscope. Similarly, video camera <b>106</b> can be realized by a suitable video camera that can provide a color image based on the image provided by the digital microscope.
0029Referring to <figref idref="DRAWINGS">FIGS. 2 and 3</figref>, there is shown imaging subsystem <b>100</b><i>a </i>in accordance with an alternate embodiment of the invention. In this embodiment, all the components of subsystem <b>100</b><i>a </i>are combined into a single unit that is portable, compact, robust, and capable of battery-power operation or AC power to allow for mobile operation or operation in remote locations. This embodiment of image subsystem <b>100</b><i>a </i>has housing <b>120</b>, control panels <b>122</b> and <b>123</b>, and interface <b>124</b>. Interface <b>124</b> comprises RS <b>232</b> interface <b>126</b>, video data ports <b>128</b> and <b>130</b>, USB port <b>132</b> and external power input <b>134</b>. Rechargeable battery pack <b>136</b> supplies power to all other components. Screen <b>138</b> allows capture of air samples that are to be analyzed thereby allowing airborne pathogens, bacteria, etc. to be analyzed. Slide insertion device <b>140</b> enables a user to insert a specimen slide <b>142</b> into housing <b>120</b>. Fluid inlet <b>144</b> and fluid outlet <b>146</b> allow for the ingress and egress of fluids (e.g. water) that is to be analyzed. In an alternate embodiment, the particular embodiment of subsystem <b>100</b><i>a </i>shown in <figref idref="DRAWINGS">FIGS. 2 and 3</figref> is configured to operate with power from a land vehicle's battery.
0030Referring to <figref idref="DRAWINGS">FIGS. 1 and 4</figref>, there is shown a flow chart illustrating the operation of imaging subsystem <b>100</b><i>a</i>. In step <b>150</b>, a user activates computer <b>101</b>. In step <b>152</b>, any required data stored in a master system (not shown) is loaded into computer <b>101</b>. In step <b>154</b>, there occurs the development of the sample or specimen, preparations and protocols. In this step, the specimen is stained by staining module <b>102</b>. In step <b>156</b>, microscope <b>104</b> and video camera <b>106</b> are activated by computer <b>101</b>, and a stained specimen slide is provided to microscope <b>104</b>. Next, in steps <b>158</b>, <b>160</b> and <b>162</b>, it is determined whether the imaging of the specimen slides is going to be controlled manually (i.e. locally). If it is decided that there will be manual control, the user inputs manual input commands into computer <b>101</b> in order to control microscope <b>104</b> and video camera <b>106</b> according to the data defined by such commands. Next, in step <b>164</b>, an image of the specimen is produced. In step <b>166</b>, the produced image of the specimen is displayed on an external display device (not shown) such as computer screen or LCD which may be connected to either computer <b>101</b> or video camera <b>106</b>. Included in steps <b>164</b> and <b>166</b> are the steps of pre-screening and pre-sorting of the images in order to determine if the image contains relevant information. In one embodiment, medical personnel pre-screen the images by visual inspection. In step <b>168</b>, the relevant images are collected and organized in image memory <b>108</b>. In step <b>170</b>, the relevant images are stored in image memory <b>108</b> or in an external data storage device (not shown) such as a ROM or CD-ROM. In one embodiment, the external data storage device is an external device that is in electronic data communication with image memory <b>108</b>. In step <b>172</b>, the relevant collected and organized images are sent to an output buffer memory and then, routed to communications module <b>110</b>. In step <b>174</b>, these images are then communicated to image management diagnostic subsystem <b>100</b><i>b </i>with communication module <b>110</b>.
0031Referring to <figref idref="DRAWINGS">FIG. 1</figref>, in one embodiment of the invention, image management diagnostic subsystem <b>100</b><i>b </i>is centrally located. In a preferred embodiment, subsystem <b>100</b><i>b </i>is configured to serve a plurality of subsystems <b>100</b><i>a </i>and provide diagnosis information in near real time. Subsystem <b>100</b><i>b </i>generally comprises communications module <b>180</b>, antenna <b>181</b>, temporary image memory <b>182</b> and image processing system <b>190</b>. Communications module <b>180</b> receives the digitized image data transmitted by communications module <b>110</b> of subsystem <b>100</b><i>a</i>. In one embodiment, communications module <b>180</b> is realized by the commercially available DSL Speedstream Model 5260 described in the foregoing description. This received digitized image data is then transferred to temporary image memory <b>182</b>. The stored digitized image is then transferred from temporary image memory <b>182</b> to image processing system <b>190</b>. Referring to <figref idref="DRAWINGS">FIG. 5</figref>, there is shown a block diagram of image processing subsystem <b>190</b>. Image processing system <b>190</b> comprises work stations <b>200</b>, <b>202</b> and <b>204</b> which are in electronic data communication with common hub <b>206</b>. In one embodiment, work stations <b>200</b>, <b>202</b> and <b>204</b> are commercially available Pentium™ class computers which are manufactured by Linux™, Sun™, and Microsoft™, respectively. In one embodiment, common hub <b>206</b> is configured as a commercially available switch such as a Hewlett Packard or compatible 10/100/1000 hub. Image processing system <b>190</b> further comprises master node <b>208</b> and firewall <b>210</b> between master node <b>208</b> and common hub <b>206</b>. Master node <b>208</b> comprises data processing modules that effect implementation and execution of the particular image processing and analysis computer programs that are described in the ensuing description. In a preferred embodiment, master node <b>208</b> is configured to implement high-speed parallel processing. In one embodiment, master node <b>208</b> comprises a Scyld Beowulf Computer Cluster which has a parallel processor comprising 64 nodes. The Scyld Beowulf Computer Cluster is known in the art and was developed by the NASA Goddard Space Flight Center. Image processing subsystem <b>190</b> further comprises central hub <b>212</b>. In one embodiment, central hub <b>212</b> is configured as a commercially available switch such as a Hewlett Packard or compatible 10/100/1000 hub. Image processing subsystem <b>190</b> further comprises a plurality of slave nodes <b>214</b> that are in electronic data communication with central hub <b>212</b>. In one embodiment, there are sixty-four slave nodes <b>214</b> and each slave node <b>214</b> is configured as a PC Pentium class computer having a minimum of 128 MB of RAM. Image processing system <b>190</b> further comprises database server <b>220</b>. Database server <b>220</b> stores the image data that originated from subsystem <b>100</b><i>a </i>(see <figref idref="DRAWINGS">FIG. 1</figref>) and which is to be analyzed by subsystem <b>100</b><i>b</i>. Data base servers are known in the art and need not be discussed herein in detail. Image processing system <b>190</b> further comprises file server image repository <b>222</b> which has sufficient data storage capacity. Repository <b>222</b> has first and second sections. The first section is for storing images of known pathogens, bacteria and abnormal cells. Specifically, the first section contains a large library of reference images of pathogens, abnormal cell structures, bacteria, etc. with several different views of each type to account for rotation and other apparent differences. Preferably, the referenced images are compressed to minimize the memory requirements. Each reference image has corresponding identification information that provides information about the reference image such as the name of the pathogen, bacteria, cell, etc. The second section of repository <b>222</b> is for the storage of segments of images produced by a hierarchical segmentation process that is described in the ensuing description.
0032Referring to <figref idref="DRAWINGS">FIGS. 1 and 5</figref>, images outputted by temporary image memory <b>182</b> are inputted into database server <b>220</b>. Images in database server <b>220</b> are routed to master node <b>208</b> by using any of the workstations <b>200</b>, <b>202</b> and <b>204</b>. Master node <b>208</b> performs several functions. Master node <b>208</b> performs a pre-scan of the digitized images received from database server <b>220</b> to determine if the digitized images contain relevant and useful information. If the images do not contain relevant and useful information, the images are either discarded (i.e. deleted) or stored in a designated area in file server image repository <b>222</b>. If the images do contain relevant and useful information, the images are then subjected to further processing. Specifically, master node <b>208</b> performs segmentation on the image. In one embodiment, master node <b>208</b> is programmed to execute a segmentation process described in pending U.S. patent application Ser. No. 09/839,147 entitled “Method For Implementation Of Recursive Hierarchical Segmentation On Parallel Computers”, the disclosure of which is incorporated herein by reference. The aforementioned pending U.S. application Ser. No. 09/839,147 was published on May 1, 2003 having Patent Application Publication No. US 2003/0081833. Publication No. US 2003/0081833 is incorporated herein by reference. The segmentation process isolates particular features of the digitized image. Specifically, this segmentation process effects a sequential set of image segmentations at different levels of segmentation detail in which the segmentations at a relatively coarser level of detail is produced from simple mergers of regions from segmentations of finer levels of detail. A unique feature of the hierarchical image segmentation process is that the segmented region boundaries are maintained at the full image spatial resolution at all levels of segmentation details in the hierarchy. The result of the process is that regions of similar characteristics are isolated (segmented) and identified. Thus, the image of a pathogen that has features distinct from the background and debris can be isolated using certain assigned criteria, e.g. color, shape, size, etc.
0033Master node <b>208</b> then performs a fast analysis on the isolated feature based on a few descriptors such as size and shape of the isolated feature. Master node <b>208</b> includes a memory for storing criteria that is used in the fast analysis to determine whether or not a particular image of an isolated feature has useful information. If the particular image has useful information, the particular image is retained and made available for further analysis. If it is determined that the particular image does not have useful information, the particular image is discarded. If a particular image of an isolated feature does have useful information, master node <b>208</b> performs further processing on that image. Specifically, master node <b>208</b> implements and executes a computer program that effects optical recognition and data mining. In one embodiment, this computer program is configured as the computer program referred to as “Continuously Scalable Template Matching” developed by NASA Jet Propulsion Laboratories and CalTech. This computer program comprises a first portion that effects data mining and a second portion that effects optical recognition. The data mining portion is configured as the computer program known as “Diamond Eye” which is known in the art and developed by NASA's Jet Propulsion Laboratory. The “Diamond Eye” computer program is based on a distributed applet/server architecture that provides platform-independent access to image mining services. A database associated with “Diamond Eye” computer program provides persistent storage and enables querying of the “mined” information. The computational engine carries out parallel execution of the most demanding parts of the data-mining task: image processing, object recognition, and querying-by-content operations. The purpose of the data mining process is to extract desired, particular image data from the isolated feature or features of the subject image that result from the segmentation process described in the foregoing description. The user inputs particular data that defines the parameters of the image data that is to be mined from the isolated feature or features of the subject image.
0034The optical recognition portion of the computer program executed by master node <b>208</b> comprises a pattern recognition program that determines whether the mined data obtained by the data mining portion of the computer program matches or corresponds to any reference images in the reference library portion of file server image repository <b>222</b>. The optical recognition program can detect patterns that differ in size but are otherwise similar to a specified (reference) pattern. If a match or correspondence exists, the reference image, the subject isolated feature which matches or corresponds to the reference image, and any information associated with the reference image, are displayed on the displays of work stations <b>200</b>, <b>202</b> and <b>204</b>. Master node <b>208</b> also effects execution and implementation of an image analysis program that performs statistical analysis on the subject isolated feature to identify areas of interest which aids medical personnel in making a diagnosis. One suitable image analysis program is the ImageJ program developed at the National Institute of Health. As a result, medical personnel can make a diagnosis upon viewing the resulting information at any of work stations <b>200</b>, <b>202</b> and <b>204</b>. If there is no matching or corresponding reference image for a subject isolated feature, then such information is displayed at work stations <b>200</b>, <b>202</b> and <b>204</b>.
0035Master node <b>208</b> also implements and executes a scheduling program, described in detail in the ensuing description, which effects cost and time efficient scheduling of all of the nodes of image processing system <b>190</b>. Thus, whether there are 16, 64 or 128 nodes in image processing system <b>190</b>, the nodes will be used efficiently to achieve optimum operation in a cost efficient manner.
0036Referring to <figref idref="DRAWINGS">FIGS. 5A-5D</figref>, there is shown a flow chart of the image processing method implemented by image processing system <b>190</b>. The method starts in step <b>300</b> upon a command inputted by a user into any of work stations <b>200</b>, <b>202</b> and <b>204</b>. In step <b>302</b>, a user uses any of the work stations <b>200</b>, <b>202</b> and <b>204</b> to retrieve an image from database server <b>220</b>. The image retrieved is the image that is to be processed and analyzed by master node <b>208</b>. As described in the foregoing description, the retrieved image can be in JPEG, TIFF or other format. In step <b>304</b>, master node <b>208</b> converts the retrieved image into raw data that is suitable for processing by master node <b>208</b>. In step <b>306</b>, the user may input commands into work stations <b>200</b>, <b>202</b> and <b>204</b> such as parameter data and recursive level data for use by the hierarchical segmentation process implemented by master node <b>208</b>. The parameter data includes the number of regions in which the subject image is to be divided. Each region defines a specific portion of the image in which medical personnel are interested in analyzing. The recursive level data defines the desired bit resolution and the bandwidth required to process the images. In an alternate embodiment, the parameter data and recursive level data are not inputted by the users but rather, are preset within the software. Next, step <b>307</b> effects implementation of a cluster scheduling program that schedules use of the clusters within master node <b>208</b> in order achieve time and cost efficient operation and use of the clusters. Thus, step <b>307</b> ensures that all clusters are always performing tasks at any given moment and that no clusters are idle. Step <b>307</b> also schedules time and efficient operation and use of file server image repository <b>222</b> and database server <b>220</b>. The scheduling program is described in the ensuing description. Next, in step <b>308</b>, it is determined if the method is to proceed with the hierarchical segmentation process. If the method is not to proceed with hierarchical segmentation, then the method ends at step <b>309</b>. If the method is to proceed with hierarchical segmentation, the method proceeds to steps <b>310</b>, <b>312</b> or <b>314</b>. Step <b>310</b> determines whether the retrieved image shall be formatted into RGB (Red, Green, Blue) format prior to the retrieved image being segmented by hierarchical segmentation. If RGB format is desired, the method shifts to step <b>318</b> wherein the hierarchical segmentation process begins. If RGB format is not desired, the method shifts to step <b>312</b>. In step <b>312</b>, it is determined whether the retrieved image shall be formatted into eight (8) bit format prior to the retrieved image being segmented by hierarchical segmentation. If eight (8) bit is desired, the method shifts to step <b>318</b> wherein the hierarchical segmentation process begins. If eight (8) bit format is not desired, the method shifts to step <b>314</b>. In step <b>314</b>, it is determined whether the retrieved image shall be formatted into sixteen (16) bit format prior to the retrieved image being segmented by hierarchical segmentation. If sixteen (16) bit format is not desired, then the method shifts to step <b>315</b> which resets the parameters. The method then shifts to step <b>316</b> which causes the method to return to the beginning, step <b>300</b>. If sixteen (16) bit format is desired, the method shifts to step <b>318</b> wherein the hierarchical segmentation process begins. As is apparent from the foregoing description, the decision process performed by steps <b>310</b>, <b>312</b> and <b>314</b> depends upon the recursive levels inputted in step <b>306</b>. In step <b>318</b>, the hierarchical segmentation process begins and breaks the retrieved image into segments. Each segment defines a particular region of the retrieved image (retrieved in step <b>302</b>). In step <b>320</b>, it is determined whether the segments are to undergo further processing or whether the segments are to be stored in repository <b>222</b>. If step <b>320</b> determines that the segments of the particular regions are not to undergo further processing, then step <b>322</b> effects storage of these images of the particular regions in repository <b>222</b>. If step <b>320</b> determines that the segments are to undergo further processing, then the method shifts to step <b>324</b> wherein the regions defined by the segments are mapped. Specifically, step <b>324</b> effects mapping or assignment of labels to each region. In step <b>325</b>, the labeled regions are stored in repository <b>222</b>.
0037Next, in step <b>326</b>, the users input data defining desired CSTM (Continuously Scalable Template Matching) models into master node <b>208</b> via any of the work stations <b>200</b>, <b>202</b> and <b>204</b>. Specifically, this data defines the desired models that are to be created based on the reference images stored in image repository <b>222</b>. These models are based on specific features and characteristics of certain pathogens, bacteria or other disease. Next, step <b>327</b> then determines if the CSTM models exist in the labeled regions stored in repository <b>222</b>. This step is accomplished by execution of the “Continuously Scalable Template Matching” program described in the foregoing description. If the CSTM models do not exist in the labeled regions stored in repository <b>222</b>, then the method shifts to step <b>328</b> which sends data to work stations <b>200</b>, <b>202</b> and <b>204</b> that indicates that no match has been found. If step <b>327</b> determines that there are CSTM models that match or correspond to labeled regions stored in repository <b>222</b>, then the method shifts to step <b>330</b> which effects retrieval of the labeled images defining the particular region or regions to which the CSTM model or models correspond. In step <b>332</b>, the retrieved labeled images are displayed at work stations <b>200</b>, <b>202</b> and <b>204</b> so as to enable medical personal to review the retrieved image and make a diagnosis. The method then ends at step <b>334</b>.
0038Referring to <figref idref="DRAWINGS">FIG. 6</figref>, there is shown a flow chart of the cluster scheduling program of step <b>307</b>. In step <b>400</b>, it is determined whether the cluster scheduling program is to be executed. If the cluster scheduling program is not to be initiated, the cluster scheduling program is terminated and the method implemented by master node <b>208</b> shifts to step <b>308</b> (see <figref idref="DRAWINGS">FIG. 5A</figref>). If the cluster scheduling program is to be executed, then the program shifts to step <b>402</b>. Step <b>402</b> determines the number of nodes that are being requested to process the subject images. Thus, step <b>402</b> determines if four (4), sixteen (16), sixty four (64), one hundred twenty (128) or more nodes are requested. In step <b>404</b>, it is determined if fast nodes or slow nodes are being requested for processing the subject retrieved images. Whether fast or slow nodes are used depends upon the amount of images to be processed and the time factors dictated by any particular situation, e.g. emergency, chemical warfare scenario, etc. In step <b>406</b>, it is determined whether there will be a time delay associated with any of the required nodes. Specifically, step <b>406</b> determines if there will be a time delay before particular nodes are available for processing the subject retrieved image. The time delay is the amount of time needed by that node to complete its other task. Thus, if a particular node is busy on another task, master node <b>208</b> will schedule that node to be used for processing the subject retrieved image upon expiration of the amount of time needed by that node to complete its other task. Similarly, master node <b>208</b> schedules nodes to commence new tasks upon completion of the current tasks. Whether there will be time delays depends upon many factors such as the recursive levels, the desired number of nodes, and whether fast or slow nodes are required. Next, step <b>408</b> calculates the cost factor for this particular processing task. The cost function depends upon the recursive levels, the desired number of nodes, whether the fast or slow nodes are required, and any time delays. Thus, the cost factor can be varied if any of these preceding factors are varied. The cost factor information is displayed on any of work stations <b>200</b>, <b>202</b> and <b>204</b>. Mathematical algorithms known in the art are used in determining the cost factor. In step <b>410</b>, the cluster scheduling program terminates and the overall process implemented by master node <b>208</b> resumes at step <b>308</b>.
0039The particular hierarchical segmentation and template matching computer programs and algorithms described in the foregoing description are examples of suitable programs and algorithms that facilitate realization and working of the invention. Thus, it is to be understood that other suitable segmentation and template matching programs may also be used as well.
0040The present invention provides many advantages and benefits, such as:
0041a) elimination of the need for cultures;
0042b) provides for rapid and accurate identification of pathogens, bacteria, infectious diseases and abnormal cells;
0043c) allows separation of the image acquisition subsystem from the image processing and identification subsystem to allow remote operation under demanding conditions;
0044d) uses multiple data transmission paths to take advantage of the available communication systems;
0045e) uses a relatively low-cost parallel processing computer system to achieve near real-time operation;
0046f) combats infectious diseases, reduces morbidity and mortality, and provides high-level medicine to remote areas of the nation and the world;
0047g) effects diagnosis of infectious diseases due to bacteria, and detection of bacterial contamination of foodstuffs;
0048h) subsystem <b>100</b><i>a </i>can be located in small hospitals and clinics, particularly in rural or remote areas such as Appalachia and Indian Reservations, as well as in Third World countries with limited access to healthcare facilities;
0049i) subsystem <b>100</b><i>a </i>can be located in large slaughterhouses, meat and poultry processing facilities, large dairy farms and other agribusinesses in order to enable detection of bacteria before such meat, poultry and dairy products are shipped to consumers; and
0050j) subsystem <b>100</b><i>a </i>can be located at research laboratories, the Center for Disease Control, and pharmaceutical manufacturers to aid in research and in the development of new antibiotics.
0051Although the foregoing description is in terms of the present invention being directed to the rapid identification of pathogens, bacteria and abnormal cells, the system and method of the present invention can be used as a diagnostic radiology and imaging tool in the medical and dental field. Specifically, the system and method of the present invention can be configured to analyze medical images such as images of soft tissue, mammograms, x-rays (bone and dental), ultrasounds, MRI images, and CAT scans. In such an embodiment, the aforementioned images are segmented to generate regions for identification in generally the same manner as the digital microscope images described in the foregoing description. Specifically, the image is transferred to image processing system <b>190</b> wherein workstations <b>200</b>, <b>202</b>, and <b>204</b> are used to compress the images. In a preferred embodiment, loss-less compression software programs, known in the art, are used. Preferably, the compression software is certified for use on medical images. Suitable compression software is GZIP and BZIT2. Other suitable compression software can be used. Next, the compressed image is stored into file server image repository <b>222</b>. The compressed image is stored in repository <b>222</b> and is subsequently retrieved so it can be segmented and/or compared against another image, segment or region. After the compressed image is retrieved from repository <b>222</b>, the compressed image is prepared for segmentation using the recursive hierarchical segmentation program described in the foregoing description. Preferably, the aforementioned recursive hierarchical segmentation program is performed on a parallel computing platform as described in the foregoing description (e.g. master node <b>208</b>). As described previously herein, the image segmentation process comprises partitioning an image into sections or regions. These regions may be subsequently associated with normal, abnormal or deviations in various tissues, however, the segmentation process simply gives generic labels to each region. The regions consist of groups of multi-spectral or hyper-spectral image pixels that have similar data feature values. These data feature values may be the multi-spectral or hyper-spectral data values themselves and/or may be derivative features such as band ratios or textural features. Simultaneously, regional images that have been segmented into their sections or regions and masked segmented images that have been labeled are stored in repository <b>220</b>. The images stored in repository <b>220</b> can be recalled by the scalable template matching program, described in the foregoing description, for either viewing or matching known or defined segmented regions that have been associated with normal, abnormal or deviations in the radiological images.
0052The principles, preferred embodiments and modes of operation of the present invention have been described in the foregoing specification. The invention which is intended to be protected herein should not, however, be construed as limited to the particular forms disclosed, as these are to be regarded as illustrative rather than restrictive. Variations in changes may be made by those skilled in the art without departing from the spirit of the invention. Accordingly, the foregoing detailed description should be considered exemplary in nature and not limited to the scope and spirit of the invention as set forth in the attached claims.
Contents6
10 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US11099175B2 | Cited by | United States of America | Applicant |
| US11808758B2 | Cited by | United States of America | Applicant |
| US11872333B1 | Cited by | United States of America | Applicant |
| US2010052892A1 | Cited by | United States of America | Pre-grant |
| US9717896B2 | Cited by | United States of America | Applicant |
| US2009287093A1 | Cited by | United States of America | Pre-grant |
| US11614609B2 | Cited by | United States of America | Applicant |
| US10843190B2 | Cited by | United States of America | Applicant |
| US11721018B2 | Cited by | United States of America | Applicant |
| US11609413B2 | Cited by | United States of America | Applicant |
| US7657070B2 | Cited by | United States of America | Search report |
| US10831013B2 | Cited by | United States of America | Applicant |
| US2024024551A1 | Cited by | United States of America | Search report |
| US11100637B2 | Cited by | United States of America | Applicant |
| US2006050984A1 | Cited by | United States of America | Pre-grant |
| US11584950B2 | Cited by | United States of America | Applicant |
| US10488644B2 | Cited by | United States of America | Applicant |
| US11100634B2 | Cited by | United States of America | Applicant |
| US11914133B2 | Cited by | United States of America | Applicant |
| US11921272B2 | Cited by | United States of America | Applicant |
| US8254696B2 | Cited by | United States of America | Search report |
| US11307196B2 | Cited by | United States of America | Applicant |
| US11295440B2 | Cited by | United States of America | Applicant |
| US10663712B2 | Cited by | United States of America | Applicant |
| US11434515B2 | Cited by | United States of America | Applicant |
| US11801336B1 | Cited by | United States of America | Applicant |
| US10482595B2 | Cited by | United States of America | Applicant |
| US2008304752A1 | Cited by | United States of America | Pre-grant |
| US11199690B2 | Cited by | United States of America | Applicant |
| US11796788B2 | Cited by | United States of America | Applicant |
| US2009318773A1 | Cited by | United States of America | Pre-grant |
| US2010056873A1 | Cited by | United States of America | Pre-grant |
| US11733150B2 | Cited by | United States of America | Applicant |
| US2010052897A1 | Cited by | United States of America | Pre-grant |
| US2010052898A1 | Cited by | United States of America | Pre-grant |
| US11803964B2 | Cited by | United States of America | Applicant |
| US7697759B2 | Cited by | United States of America | Search report |
| US2007172100A1 | Cited by | United States of America | Pre-grant |
| WO2024025727A1 | Cited by | World Intellectual Property Organization (WIPO) | International search |
| US10640807B2 | Cited by | United States of America | Applicant |
| US11262571B2 | Cited by | United States of America | Applicant |
| US2002165837A1 | Cites | United States of America | Applicant |
| US2003026493A1 | Cites | United States of America | Search report |
| US2006257053A1 | Cites | United States of America | Search report |
| US5742700A | Cites | United States of America | Search report |
| US5995668A | Cites | United States of America | Search report |
| US6895115B2 | Cites | United States of America | Applicant |
14 priority claims, no other members on record
Priority claims14
| Document | Office | Kind | Date |
|---|---|---|---|
| 45876903 | United States of America | P | |
| 45876903 | United States of America | P | |
| 50594403 | United States of America | P | |
| 50594403 | United States of America | P | |
| 2004009172 | United States of America | W | |
| 2004009172 | United States of America | W | |
| 55007604 | United States of America | A | |
| 60458769 | – | – | – |
| 60505944 | – | – | – |
| PCTUS2004009172 | – | – | – |
| US20030458769P | – | – | – |
| US20030505944P | – | – | – |
| US20040550076 | – | – | – |
| WO2004US09172 | – | – | – |
39 transactions on the USPTO file
Allowed without a rejection on record.
- Non-final rejections
- 0
- Final rejections
- 0
- RCEs
- 0
- Appeals
- 0
Over time
Point at a mark for the transactionTransactions
| Event | Code | |
|---|---|---|
| Electronic Information Disclosure StatementEIDS. | EIDS. | |
| Recordation of Patent Grant MailedPGM/ | PGM/ | |
| Patent Issue Date Used in PTA CalculationAllowedPTAC | PTAC | |
| Issue Notification MailedAllowedWPIR | WPIR | |
| Dispatch to FDCD1935 | D1935 | |
| Application Is Considered Ready for IssuePILS | PILS | |
| Issue Fee Payment VerifiedN084 | N084 | |
| Issue Fee Payment ReceivedIFEE | IFEE | |
| Mail Notice of AllowanceAllowedMN/=. | MN/=. | |
| Mail Examiner's AmendmentMEX.A | MEX.A | |
| Notice of Allowance Data Verification CompletedAllowedN/=. | N/=. | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Examiner Interview Summary Record (PTOL - 413)EXIN | EXIN | |
| Examiner's Amendment CommunicationEX.A | EX.A | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Reference capture on IDSRCAP | RCAP | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| Date Forwarded to ExaminerFWDX | FWDX | |
| Response to Election / Restriction FiledELC. | ELC. | |
| Mail Restriction RequirementMCTRS | MCTRS | |
| Restriction/Election RequirementCTRS | CTRS | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| PG-Pub Issue NotificationPG-ISSUE | PG-ISSUE | |
| Case Docketed to Examiner in GAUDOCK | DOCK | |
| IFW TSS Processing by Tech Center CompleteTSSCOMP | TSSCOMP | |
| Application Dispatched from OIPEOIPE | OIPE | |
| Cleared by OIPE CSRL194 | L194 | |
| Information Disclosure Statement consideredIDSC | IDSC | |
| Reference capture on IDSRCAP | RCAP | |
| Request for Foreign Priority (Priority Papers May Be Included)RQPR | RQPR | |
| Information Disclosure Statement (IDS) FiledM844 | M844 | |
| Information Disclosure Statement (IDS) FiledWIDS | WIDS | |
| 371 Completion Date371COMP | 371COMP | |
| Additional Application Filing FeesADDFLFEE | ADDFLFEE | |
| A statement by one or more inventors satisfying the requirement under 35 USC 115, Oath of the ApplicOATHDECL | OATHDECL | |
| Initial Exam Team nnIEXX | IEXX |
8 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Lapsed due to failure to pay maintenance feeLapsedFP | FP | |
| Lapse for failure to pay maintenance feesLapsedPATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITYLAPS | LAPS | |
| Information on status: patent discontinuationPATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362STCH | STCH | |
| Fee payment procedureMAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITYFEPP | FEPP | |
| Fee paymentFPAY | FPAY | |
| Fee paymentFPAY | FPAY | |
| Information on status: patent grantGrantedPATENTED CASESTCF | STCF | |
| AssignmentAS | AS |
Numbers
- Publication
- 07346205
- Publication, DOCDB
- 7346205
- Publication, EPODOC
- US7346205
- Application
- 10550076
- Application, DOCDB
- 55007604
- Application, EPODOC
- US20040550076
Titles
- English
- System and method for rapidly identifying pathogens, bacteria and abnormal cells
Patent term adjustment
- A delay
- +297 daysthe office missed an examination deadline
- Applicant delay
- −7 days
- Net adjustment
- 290 days
Classification
- CPC, 5
- G06T7/0012
- G06T2207/30024
- G06V20/69
- G01N2015/019
- G01N15/1433
- IPC, 9
- G06K9 00
- A61B
- C12M1 34
- C12M3 00
- C12Q1 02
- C12Q1 04
- G01N15 00
- G01N15 14
- G06T7 00
- USPC, 2
- 382133000
- 382173000