Edge enhancement for thermal-visible combined images and cameras

BACKGROUND

0001Thermal imaging cameras are used in a variety of situations. For example, thermal imaging cameras are often used during maintenance inspections to thermally inspect equipment. Example equipment may include rotating machinery, electrical panels, or rows of circuit breakers, among other types of equipment. Thermal inspections can detect equipment hot spots such as overheating machinery or electrical components, helping to ensure timely repair or replacement of the overheating equipment before a more significant problem develops.

0002Depending on the configuration of the camera, the thermal imaging camera may also generate a visible light image of the same object. The camera may display the infrared image and the visible light image in a coordinated manner, for example, to help an operator interpret the thermal image generated by the thermal imaging camera. Unlike visible light images which generally provide good contrast between different objects, it is often difficult to recognize and distinguish different features in a thermal image as compared to the real-world scene. For this reason, an operator may rely on a visible light image to help interpret and focus the thermal image. For example, overlapping and/or combining the visible light image and the thermal image can provide some guidance for the operator. However, in some situations, it can still be difficult to distinguish edges and boundaries of objects in the thermal image.

SUMMARY

0003Aspects of the present disclosure are directed toward systems and methods for combining corresponding visible light and infrared image data. In some embodiments, a system includes a memory capable of storing one or more sets of visible light image data and infrared image data. A system can include a processor configured to process a first set of visible light image data to determine an edge factor value for each pixel in the first set of visible light image data. The magnitude of the edge factor value may be representative of the strength of an edge at a particular location. In some examples, the edge factor value is based on the visible light image data and an edge gain input. The system can include a user interface by which the edge gain input can be adjustable. In some embodiments, processing the visible light image data can include performing an embossing process. The embossing process can include determining a pseudo-luminance value for each of a plurality of pixels and processing the pseudo-luminance values via a processing kernel to determine the emboss value for at least one pixel associated with the kernel.

0004In some examples, the processor can be configured to combine the edge factor value associated with each pixel in the first set of visible light image data with an edge midscale value to create a first set of modified visible light image data. Accordingly, in some such examples, the modified VL image data represents the edge midscale value offset or otherwise adjusted by the edge factor value at each pixel. In some examples, the edge midscale value can be a scalar value (e.g., a luminance value) or a vector (e.g., having R, G, and B components). The edge midscale value can include gray or any other color. In some examples, the edge midscale color may be selectable or adjustable via the user interface.

0005In some embodiments, the processor can be configured to combine the first set of modified visible light image with a first set of infrared image data that corresponds to the first set of visible light image data to generate a first set of combined image data. The resulting combination can include infrared image data combined with values representative of the strength of visible light images at corresponding pixels. As such, edges present in the visible light image data may be enhanced or otherwise visible in the combined image.

0006The infrared image data can include a scalar infrared values (e.g., infrared intensities) or vector values (e.g., having R, G, and B components) based on a palettization scheme, for example. In some examples, the palettization scheme of the infrared image data may be selectable via a user interface. In some examples, combining the infrared and visible light image data can include blending the infrared and visible light image data, for example, using a blending ratio. In some such embodiments, blending ratio is variable across the combined image. For instance, in some examples, the blending ratio at each pixel in the combined image is based on the edge factor value at corresponding visible light pixels.

0007In various examples, visible light and infrared image data can be acquired from a variety of sources. For instance, one or both of visible light and infrared image data can comprise image data recalled from memory. Additionally or alternatively, one or both of visible light and infrared image data can comprise a stream of image data, for example, acquired from one or more camera modules or other sensors. In some examples, the system can include one or both of visible light and infrared camera modules for acquiring visible light image data and infrared image data, respectively.

0008Embodiments of some such systems enable a variety of user-adjustable features. For example, in some embodiments, a user may adjust at least one of an edge gain, an edge color midscale value, an infrared palettization scheme, and at least one aspect of combining the visible light and infrared image data. Such customization allows a user to optimize a combined infrared and visible light image in order to

0009In some examples, a processor may perform one or more such functions according to instructions on a non-transitory computer readable medium. Additionally or alternatively, various such components can be enclosed in or otherwise supported by a housing. For example, a thermal imaging camera may include infrared and visible light camera modules capable of generating infrared and visible light image data, respectively. The camera can further include a processor for processing the infrared and visible light image data to generate a combined image, and a display on which to present the combined image. The camera could be used to perform thermographic processes with the benefit of edges from the visible light image data in the thermographic images.

BRIEF DESCRIPTION OF THE DRAWINGS

0010<figref idref="DRAWINGS">FIG. 1</figref> is a perspective front view of an example thermal imaging camera.

0011<figref idref="DRAWINGS">FIG. 2</figref> is a perspective back view of the example thermal imaging camera of <figref idref="DRAWINGS">FIG. 1</figref>.

0012<figref idref="DRAWINGS">FIG. 3</figref> is a functional block diagram illustrating example components of the thermal imaging camera of <figref idref="DRAWINGS">FIGS. 1 and 2</figref>.

0013<figref idref="DRAWINGS">FIG. 4</figref> is a high-level schematic diagram illustrating an exemplary technique for enhancing edges in an IR image.

0014<figref idref="DRAWINGS">FIGS. 5A and 5B</figref> are schematic diagrams illustrating exemplary VL image processing techniques used for generating a modified VL image.

0015<figref idref="DRAWINGS">FIG. 6A</figref> is a schematic diagram illustrating the combining of an edge factor value with an edge midscale value in RGB color space to determine a modified VL image pixel.

0016<figref idref="DRAWINGS">FIG. 6B</figref> is a schematic diagram illustrating the combining of an edge factor value with an edge midscale value in YCbCr color space to determine a modified VL image pixel.

0017<figref idref="DRAWINGS">FIGS. 7A and 7B</figref> are schematic diagrams illustrating exemplary techniques for combining the modified VL image and a corresponding IR image.

0018<figref idref="DRAWINGS">FIGS. 8A and 8B</figref> are schematic diagrams illustrating a process for generating a blended image.

0019<figref idref="DRAWINGS">FIGS. 9A and 9B</figref> are a flow diagram showing a series of example images used in creating a blended image.

0020<figref idref="DRAWINGS">FIG. 10</figref> is a series of images illustrating adjustability of edge gain parameters.

0021<figref idref="DRAWINGS">FIG. 11</figref> is a process-flow diagram illustrating an exemplary process for creating a blended image.

DETAILED DESCRIPTION

0022The following detailed description is exemplary in nature and is not intended to limit the scope, applicability, or configuration of the invention in any way. Rather, the following description provides some practical illustrations for implementing various embodiments of the present invention. Examples of constructions, materials, dimensions, and manufacturing processes are provided for selected elements, and all other elements employ that which is known to those of ordinary skill in the field of the invention. Those skilled in the art will recognize that many of the noted examples have a variety of suitable alternatives.

0023A thermal imaging camera may be used to detect heat patterns across a scene, including an object or objects, under observation. The thermal imaging camera may detect infrared radiation given off by the scene and convert the infrared radiation into an infrared image indicative of the heat patterns. In some embodiments, the thermal imaging camera may also capture visible light from the scene and convert the visible light into a visible light image. Depending on the configuration of the thermal imaging camera, the camera may include infrared optics to focus the infrared radiation on an infrared sensor and visible light optics to focus the visible light on a visible light sensor.

0024Various embodiments provide methods and systems for producing thermal images with reduced noise using averaging techniques. To further improve image quality and eliminate problems that may arise from averaging (e.g. blurring, ghosting, etc.), an image alignment process is performed on the thermal images prior to averaging.

0025<figref idref="DRAWINGS">FIGS. 1 and 2</figref> show front and back perspective views, respectively of an example thermal imaging camera <b>100</b>, which includes a housing <b>102</b>, an infrared lens assembly <b>104</b>, a visible light lens assembly <b>106</b>, a display <b>108</b>, a laser <b>110</b>, and a trigger control <b>112</b>. Housing <b>102</b> houses the various components of thermal imaging camera <b>100</b>. The bottom portion of thermal imaging camera <b>100</b> includes a carrying handle <b>118</b> for holding and operating the camera via one hand. Infrared lens assembly <b>104</b> receives infrared radiation from a scene and focuses the radiation on an infrared sensor for generating an infrared image of a scene. Visible light lens assembly <b>106</b> receives visible light from a scene and focuses the visible light on a visible light sensor for generating a visible light image of the same scene. Thermal imaging camera <b>100</b> captures the visible light image and/or the infrared image in response to depressing trigger control <b>112</b>. In addition, thermal imaging camera <b>100</b> controls display <b>108</b> to display the infrared image and the visible light image generated by the camera, e.g., to help an operator thermally inspect a scene. Thermal imaging camera <b>100</b> may also include a focus mechanism coupled to infrared lens assembly <b>104</b> that is configured to move at least one lens of the infrared lens assembly so as to adjust the focus of an infrared image generated by the thermal imaging camera. Additionally or alternatively, the focus mechanism may move the FPA relative to one or more lenses of the infrared lens assembly.

0026In operation, thermal imaging camera <b>100</b> detects heat patterns in a scene by receiving energy emitted in the infrared-wavelength spectrum from the scene and processing the infrared energy to generate a thermal image. Thermal imaging camera <b>100</b> may also generate a visible light image of the same scene by receiving energy in the visible light-wavelength spectrum and processing the visible light energy to generate a visible light image. As described in greater detail below, thermal imaging camera <b>100</b> may include an infrared camera module that is configured to capture an infrared image of the scene and a visible light camera module that is configured to capture a visible light image of the same scene. The infrared camera module may receive infrared radiation projected through infrared lens assembly <b>104</b> and generate therefrom infrared image data. The visible light camera module may receive light projected through visible light lens assembly <b>106</b> and generate therefrom visible light data.

0027In some examples, thermal imaging camera <b>100</b> collects or captures the infrared energy and visible light energy substantially simultaneously (e.g., at the same time) so that the visible light image and the infrared image generated by the camera are of the same scene at substantially the same time. In these examples, the infrared image generated by thermal imaging camera <b>100</b> is indicative of localized temperatures within the scene at a particular period of time while the visible light image generated by the camera is indicative of the same scene at the same period of time. In other examples, thermal imaging camera may capture infrared energy and visible light energy from a scene at different periods of time.

0028Visible light lens assembly <b>106</b> includes at least one lens that focuses visible light energy on a visible light sensor for generating a visible light image. Visible light lens assembly <b>106</b> defines a visible light optical axis which passes through the center of curvature of the at least one lens of the assembly. Visible light energy projects through a front of the lens and focuses on an opposite side of the lens. Visible light lens assembly <b>106</b> can include a single lens or a plurality of lenses (e.g., two, three, or more lenses) arranged in series. In addition, visible light lens assembly <b>106</b> can have a fixed focus or can include a focus adjustment mechanism for changing the focus of the visible light optics. In examples in which visible light lens assembly <b>106</b> includes a focus adjustment mechanism, the focus adjustment mechanism may be a manual adjustment mechanism or an automatic adjustment mechanism.

0029Infrared lens assembly <b>104</b> also includes at least one lens that focuses infrared energy on an infrared sensor for generating a thermal image. Infrared lens assembly <b>104</b> defines an infrared optical axis which passes through the center of curvature of lens of the assembly. During operation, infrared energy is directed through the front of the lens and focused on an opposite side of the lens. Infrared lens assembly <b>104</b> can include a single lens or a plurality of lenses (e.g., two, three, or more lenses), which may be arranged in series. In some examples, the infrared lens assembly <b>104</b> may include lenses having diffractive or reflective properties or elements. Additional optical components such as mirrors (e.g., Fresnel mirrors) and the like may be included within or otherwise proximate to the infrared lens assembly <b>104</b>.

0030As briefly described above, thermal imaging camera <b>100</b> includes a focus mechanism for adjusting the focus of an infrared image captured by the camera. In the example shown in <figref idref="DRAWINGS">FIGS. 1 and 2</figref>, thermal imaging camera <b>100</b> includes focus ring <b>114</b>. Focus ring <b>114</b> is operatively coupled (e.g., mechanically and/or electrically coupled) to at least one lens of infrared lens assembly <b>104</b> and configured to move one or both of the FPA and the at least one lens to various focus positions so as to focus the infrared image captured by thermal imaging camera <b>100</b>. Focus ring <b>114</b> may be manually rotated about at least a portion of housing <b>102</b> so as to move the at least one lens to which the focus ring is operatively coupled. In some examples, focus ring <b>114</b> is also operatively coupled to display <b>108</b> such that rotation of focus ring <b>114</b> causes at least a portion of a visible light image and at least a portion of an infrared image concurrently displayed on display <b>108</b> to move relative to one another. In different examples, thermal imaging camera <b>100</b> may include a manual focus adjustment mechanism that is implemented in a configuration other than focus ring <b>114</b>, or may, in other embodiments, simply maintain a fixed focus.

0031In some examples, thermal imaging camera <b>100</b> may include an automatically adjusting focus mechanism in addition to or in lieu of a manually adjusting focus mechanism. An automatically adjusting focus mechanism may be operatively coupled to at least one lens of infrared lens assembly <b>104</b> and configured to automatically move the at least one lens to various focus positions, e.g., in response to instructions from thermal imaging camera <b>100</b>. In one application of such an example, thermal imaging camera <b>100</b> may use laser <b>110</b> to electronically measure a distance between an object in a target scene and the camera, referred to as the distance-to-target. Thermal imaging camera <b>100</b> may then control the automatically adjusting focus mechanism to move the at least one lens of infrared lens assembly <b>104</b> to a focus position that corresponds to the distance-to-target data determined by thermal imaging camera <b>100</b>. The focus position may correspond to the distance-to-target data in that the focus position may be configured to place the object in the target scene at the determined distance in focus. In some examples, the focus position set by the automatically adjusting focus mechanism may be manually overridden by an operator, e.g., by rotating focus ring <b>114</b>.

0032During operation of thermal imaging camera <b>100</b>, an operator may wish to view a thermal image of a scene and/or a visible light image of the same scene generated by the camera. For this reason, thermal imaging camera <b>100</b> may include a display. In the examples of <figref idref="DRAWINGS">FIGS. 1 and 2</figref>, thermal imaging camera <b>100</b> includes display <b>108</b>, which is located on the back of housing <b>102</b> opposite infrared lens assembly <b>104</b> and visible light lens assembly <b>106</b>. Display <b>108</b> may be configured to display a visible light image, an infrared image, and/or a combined image that includes a simultaneous display of the visible light image and the infrared image. In different examples, display <b>108</b> may be remote (e.g., separate) from infrared lens assembly <b>104</b> and visible light lens assembly <b>106</b> of thermal imaging camera <b>100</b>, or display <b>108</b> may be in a different spatial arrangement relative to infrared lens assembly <b>104</b> and/or visible light lens assembly <b>106</b>. Therefore, although display <b>108</b> is shown behind infrared lens assembly <b>104</b> and visible light lens assembly <b>106</b> in <figref idref="DRAWINGS">FIG. 2</figref>, other locations for display <b>108</b> are possible.

0033Thermal imaging camera <b>100</b> can include a variety of user input media for controlling the operation of the camera and adjusting different settings of the camera. Example control functions may include adjusting the focus of the infrared and/or visible light optics, opening/closing a shutter, capturing an infrared and/or visible light image, or the like. In the example of <figref idref="DRAWINGS">FIGS. 1 and 2</figref>, thermal imaging camera <b>100</b> includes a depressible trigger control <b>112</b> for capturing an infrared and visible light image, and buttons <b>116</b>, which form part of the user interface, for controlling other aspects of the operation of the camera. A different number or arrangement of user input media are possible, and it should be appreciated that the disclosure is not limited in this respect. For example, thermal imaging camera <b>100</b> may include a touch screen display <b>108</b> which receives user input by depressing different portions of the screen.

0034<figref idref="DRAWINGS">FIG. 3</figref> is a functional block diagram illustrating components of an example of thermal imaging camera <b>100</b>. Thermal imaging camera <b>100</b> includes an IR camera module <b>200</b>, front end circuitry <b>202</b>. The IR camera module <b>200</b> and front end circuitry <b>202</b> are sometimes referred to in combination as front end stage or front end components <b>204</b> of the infrared camera <b>100</b>. Thermal imaging camera <b>100</b> may also include a visible light camera module <b>206</b>, a display <b>108</b>, a user interface <b>208</b>, and an output/control device <b>210</b>.

0035Infrared camera module <b>200</b> may be configured to receive infrared energy emitted by a target scene and to focus the infrared energy on an infrared sensor for generation of infrared energy data, e.g., that can be displayed in the form of an infrared image on display <b>108</b> and/or stored in memory. Infrared camera module <b>200</b> can include any suitable components for performing the functions attributed to the module herein. In the example of <figref idref="DRAWINGS">FIG. 3</figref>, infrared camera module <b>200</b> is illustrated as including infrared lens assembly <b>104</b> and infrared sensor <b>220</b>. As described above with respect to <figref idref="DRAWINGS">FIGS. 1 and 2</figref>, infrared lens assembly <b>104</b> includes at least one lens that takes infrared energy emitted by a target scene and focuses the infrared energy on infrared sensor <b>220</b>. Infrared sensor <b>220</b> responds to the focused infrared energy by generating an electrical signal that can be converted and displayed as an infrared image on display <b>108</b>.

0036Infrared sensor <b>220</b> may include one or more focal plane arrays (FPA) that generate electrical signals in response to infrared energy received through infrared lens assembly <b>104</b>. Each FPA can include a plurality of infrared sensor elements including, e.g., bolometers, photon detectors, or other suitable infrared sensor elements. In operation, each sensor element, which may each be referred to as a sensor pixel, may change an electrical characteristic (e.g., voltage or resistance) in response to absorbing infrared energy received from a target scene. In turn, the change in electrical characteristic can provide an electrical signal that can be received by a processor <b>222</b> and processed into an infrared image displayed on display <b>108</b>.

0037For instance, in examples in which infrared sensor <b>220</b> includes a plurality of bolometers, each bolometer may absorb infrared energy focused through infrared lens assembly <b>104</b> and increase in temperature in response to the absorbed energy. The electrical resistance of each bolometer may change as the temperature of the bolometer changes. With each detector element functioning as a sensor pixel, a two-dimensional image or picture representation of the infrared radiation can be further generated by translating the changes in resistance of each detector element into a time-multiplexed electrical signal that can be processed for visualization on a display or storage in memory (e.g., of a computer). Processor <b>222</b> may measure the change in resistance of each bolometer by applying a current (or voltage) to each bolometer and measure the resulting voltage (or current) across the bolometer. Based on these data, processor <b>222</b> can determine the amount of infrared energy emitted by different portions of a target scene and control display <b>108</b> to display a thermal image of the target scene.

0038Independent of the specific type of infrared sensor elements included in the FPA of infrared sensor <b>220</b>, the FPA array can define any suitable size and shape. In some examples, infrared sensor <b>220</b> includes a plurality of infrared sensor elements arranged in a grid pattern such as, e.g., an array of sensor elements arranged in vertical columns and horizontal rows. In various examples, infrared sensor <b>220</b> may include an array of vertical columns by horizontal rows of, e.g., 16×16, 50×50, 160×120, 120×160, or 650×480. In other examples, infrared sensor <b>220</b> may include a smaller number of vertical columns and horizontal rows (e.g., 1×1), a larger number vertical columns and horizontal rows (e.g., 1000×1000), or a different ratio of columns to rows.

0039In certain embodiments a Read Out Integrated Circuit (ROIC) is incorporated on the IR sensor <b>220</b>. The ROIC is used to output signals corresponding to each of the sensor pixels. Such ROIC is commonly fabricated as an integrated circuit on a silicon substrate. The plurality of detector elements may be fabricated on top of the ROIC, wherein their combination provides for the IR sensor <b>220</b>. In some embodiments, the ROIC can include components discussed elsewhere in this disclosure (e.g. an analog-to-digital converter (ADC)) incorporated directly onto the FPA circuitry. Such integration of the ROIC, or other further levels of integration not explicitly discussed, should be considered within the scope of this disclosure.

0040As described above, the IR sensor <b>220</b> generates a series of electrical signals corresponding to the infrared radiation received by each infrared detector element to represent a thermal image. A “frame” of thermal image data is generated when the voltage signal from each infrared detector element is obtained by scanning all of the rows that make up the IR sensor <b>220</b>. Again, in certain embodiments involving bolometers as the infrared detector elements, such scanning is done by switching a corresponding detector element into the system circuit and applying a bias voltage across such switched-in element. Successive frames of thermal image data are generated by repeatedly scanning the rows of the IR sensor <b>220</b>, with such frames being produced at a rate sufficient to generate a video representation (e.g. 30 Hz, or 60 Hz) of the thermal image data.

0041The front end circuitry <b>202</b> includes circuitry for interfacing with and controlling the IR camera module <b>200</b>. In addition, the front end circuitry <b>202</b> initially processes and transmits collected infrared image data to a processor <b>222</b> via a connection therebetween. More specifically, the signals generated by the IR sensor <b>220</b> are initially conditioned by the front end circuitry <b>202</b> of the thermal imaging camera <b>100</b>. In certain embodiments, as shown, the front end circuitry <b>202</b> includes a bias generator <b>224</b> and a pre-amp/integrator <b>226</b>. In addition to providing the detector bias, the bias generator <b>224</b> can optionally add or subtract an average bias current from the total current generated for each switched-in detector element. The average bias current can be changed in order (i) to compensate for deviations to the entire array of resistances of the detector elements resulting from changes in ambient temperatures inside the thermal imaging camera <b>100</b> and (ii) to compensate for array-to-array variations in the average detector elements of the IR sensor <b>220</b>. Such bias compensation can be automatically controlled by the thermal imaging camera <b>100</b> or software, or can be user controlled via input to the output/control device <b>210</b> or processor <b>222</b>. Following provision of the detector bias and optional subtraction or addition of the average bias current, the signals can be passed through a pre-amp/integrator <b>226</b>. Typically, the pre-amp/integrator <b>226</b> is used to condition incoming signals, e.g., prior to their digitization. As a result, the incoming signals can be adjusted to a form that enables more effective interpretation of the signals, and in turn, can lead to more effective resolution of the created image. Subsequently, the conditioned signals are sent downstream into the processor <b>222</b> of the thermal imaging camera <b>100</b>.

0042In some embodiments, the front end circuitry <b>202</b> can include one or more additional elements for example, additional sensors <b>228</b> or an ADC <b>230</b>. Additional sensors <b>228</b> can include, for example, temperature sensors, visual light sensors (such as a CCD), pressure sensors, magnetic sensors, etc. Such sensors can provide additional calibration and detection information to enhance the functionality of the thermal imaging camera <b>100</b>. For example, temperature sensors can provide an ambient temperature reading near the IR sensor <b>220</b> to assist in radiometry calculations. A magnetic sensor, such as a Hall Effect sensor, can be used in combination with a magnet mounted on the lens to provide lens focus position information. Such information can be useful for calculating distances, or determining a parallax offset for use with visual light scene data gathered from a visual light sensor.

0043An ADC <b>230</b> can provide the same function and operate in substantially the same manner as discussed below, however its inclusion in the front end circuitry <b>202</b> may provide certain benefits, for example, digitization of scene and other sensor information prior to transmittal to the processor <b>222</b> via the connection therebetween. In some embodiments, the ADC <b>230</b> can be integrated into the ROIC, as discussed above, thereby eliminating the need for a separately mounted and installed ADC <b>230</b>.

0044In some embodiments, front end components can further include a shutter <b>240</b>. A shutter <b>240</b> can be externally or internally located relative to the lens and operate to open or close the view provided by the IR lens assembly <b>104</b>. As is known in the art, the shutter <b>240</b> can be mechanically positionable, or can be actuated by an electro-mechanical device such as a DC motor or solenoid. Embodiments of the invention may include a calibration or setup software implemented method or setting which utilize the shutter <b>240</b> to establish appropriate bias levels for each detector element.

0045Components described as processors within thermal imaging camera <b>100</b>, including processor <b>222</b>, may be implemented as one or more processors, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic circuitry, or the like, either alone or in any suitable combination. Processor <b>222</b> may also include memory that stores program instructions and related data that, when executed by processor <b>222</b>, cause thermal imaging camera <b>100</b> and processor <b>222</b> to perform the functions attributed to them in this disclosure. Memory may include any fixed or removable magnetic, optical, or electrical media, such as RAM, ROM, CD-ROM, hard or floppy magnetic disks, EEPROM, or the like. Memory may also include a removable memory portion that may be used to provide memory updates or increases in memory capacities. A removable memory may also allow image data to be easily transferred to another computing device, or to be removed before thermal imaging camera <b>100</b> is used in another application. Processor <b>222</b> may also be implemented as a System on Chip that integrates some or all components of a computer or other electronic system into a single chip. These elements manipulate the conditioned scene image data delivered from the front end stages <b>204</b> in order to provide output scene data that can be displayed or stored for use by the user. Subsequently, the processor <b>222</b> (processing circuitry) sends the processed data to a display <b>108</b> or other output/control device <b>210</b>.

0046During operation of thermal imaging camera <b>100</b>, processor <b>222</b> can control infrared camera module <b>200</b> to generate infrared image data for creating an infrared image. Processor <b>222</b> can generate a digital “frame” of infrared image data. By generating a frame of infrared image data, processor <b>222</b> captures an infrared image of a target scene at substantially a given point in time. That is, in some examples, a plurality of pixels making up the infrared image may be captured simultaneously. In other embodiments, sets of one or more pixels may be captured serially until each pixel has been captured.

0047Processor <b>222</b> can capture a single infrared image or “snap shot” of a target scene by measuring the electrical signal of each infrared sensor element included in the FPA of infrared sensor <b>220</b> a single time. Alternatively, processor <b>222</b> can capture a plurality of infrared images of a target scene by repeatedly measuring the electrical signal of each infrared sensor element included in the FPA of infrared sensor <b>220</b>. In examples in which processor <b>222</b> repeatedly measures the electrical signal of each infrared sensor element included in the FPA of infrared sensor <b>220</b>, processor <b>222</b> may generate a dynamic thermal image (e.g., a video representation) of a target scene. For example, processor <b>222</b> may measure the electrical signal of each infrared sensor element included in the FPA at a rate sufficient to generate a video representation of thermal image data such as, e.g., 30 Hz or 60 Hz. Processor <b>222</b> may perform other operations in capturing an infrared image such as sequentially actuating a shutter <b>240</b> to open and close an aperture of infrared lens assembly <b>104</b>, or the like.

0048With each sensor element of infrared sensor <b>220</b> functioning as a sensor pixel, processor <b>222</b> can generate a two-dimensional image or picture representation of the infrared radiation from a target scene by translating changes in an electrical characteristic (e.g., resistance) of each sensor element into a time-multiplexed electrical signal that can be processed, e.g., for visualization on display <b>108</b> and/or storage in memory. When displayed on a display <b>108</b>, an infrared image can comprise a plurality of display pixels. Display pixels can have any defined relationship with corresponding sensor pixels. In some examples, each sensor pixel corresponds to a display pixel in an image representation of infrared data. In other examples, a plurality of sensor pixels may be combined (e.g., averaged) to provide infrared information for a single display pixel. In still other examples, a single sensor pixel may contribute to a plurality of display pixels. For example, a value from a single sensor pixel may be replicated at nearby pixels, such as in a simple upsampling procedure. In other examples, neighboring or otherwise nearby pixels may be averaged to create a new pixel value, such as in an interpolation procedure. Because relationships between display pixels and sensor pixels are defined with respect to camera operation, the generic term “pixel” may refer to the sensor pixel, the display pixel, or the data as it is processed from the sensor pixel to the display pixel unless otherwise stated. Processor <b>222</b> may perform computations to convert raw infrared image data into scene temperatures (radiometry) including, in some examples, colors corresponding to the scene temperatures.

0049Processor <b>222</b> may control display <b>108</b> to display at least a portion of an infrared image of a captured target scene. In some examples, processor <b>222</b> controls display <b>108</b> so that the electrical response of each sensor element of infrared sensor <b>220</b> is associated with a single pixel on display <b>108</b>. In other examples, processor <b>222</b> may increase or decrease the resolution of an infrared image so that there are more or fewer pixels displayed on display <b>108</b> than there are sensor elements in infrared sensor <b>220</b>. Processor <b>222</b> may control display <b>108</b> to display an entire infrared image (e.g., all portions of a target scene captured by thermal imaging camera <b>100</b>) or less than an entire infrared image (e.g., a lesser port of the entire target scene captured by thermal imaging camera <b>100</b>). Processor <b>222</b> may perform other image processing functions, as described in greater detail below.

0050Independent of the specific circuitry, thermal imaging camera <b>100</b> may be configured to manipulate data representative of a target scene so as to provide an output that can be displayed, stored, transmitted, or otherwise utilized by a user.

0051Thermal imaging camera <b>100</b> includes visible light camera module <b>206</b>. Visible light camera modules are generally well known. For examples, various visible light camera modules are included in smartphones and numerous other devices. In some embodiments, visible light camera module <b>206</b> may be configured to receive visible light energy from a target scene and to focus the visible light energy on a visible light sensor for generation of visible light energy data, e.g., that can be displayed in the form of a visible light image on display <b>108</b> and/or stored in memory. Visible light camera module <b>206</b> can include any suitable components for performing the functions attributed to the module herein. In the example of <figref idref="DRAWINGS">FIG. 3</figref>, visible light camera module <b>206</b> is illustrated as including visible light lens assembly <b>106</b> and visible light sensor <b>242</b>. As described above with respect to <figref idref="DRAWINGS">FIGS. 1 and 2</figref>, visible light lens assembly <b>106</b> includes at least one lens that takes visible light energy emitted by a target scene and focuses the visible light energy on visible light sensor <b>242</b>. Visible light sensor <b>242</b> responds to the focused energy by generating an electrical signal that can be converted and displayed as a visible light image on display <b>108</b>. In some examples, the visible light module <b>206</b> is configurable by a user, and can provide output, for example, to display <b>108</b>, in a variety of formats. Visible light camera module <b>206</b> may include compensation functionality for varying lighting or other operating conditions or user preferences. The visible light camera module may provide a digital output including image data, which may include data in a variety of formats (e.g., RGB, CYMK, YCbCr, etc.).

0052Visible light sensor <b>242</b> may include a plurality of visible light sensor elements such as, e.g., CMOS detectors, CCD detectors, PIN diodes, avalanche photo diodes, or the like. The number of visible light sensor elements may be the same as or different than the number of infrared light sensor elements.

0053In operation, optical energy received from a target scene may pass through visible light lens assembly <b>106</b> and be focused on visible light sensor <b>242</b>. When the optical energy impinges upon the visible light sensor elements of visible light sensor <b>242</b>, photons within the photodetectors may be released and converted into a detection current. Processor <b>222</b> can process this detection current to form a visible light image of the target scene.

0054During use of thermal imaging camera <b>100</b>, processor <b>222</b> can control visible light camera module <b>206</b> to generate visible light data from a captured target scene for creating a visible light image. The visible light data may include luminosity data indicative of the color(s) associated with different portions of the captured target scene and/or the magnitude of light associated with different portions of the captured target scene. Processor <b>222</b> can generate a “frame” of visible light image data by measuring the response of each visible light sensor element of thermal imaging camera <b>100</b> a single time. By generating a frame of visible light data, processor <b>222</b> captures visible light image of a target scene at a given point in time. Processor <b>222</b> may also repeatedly measure the response of each visible light sensor element of thermal imaging camera <b>100</b> so as to generate a dynamic thermal image (e.g., a video representation) of a target scene, as described above with respect to infrared camera module <b>200</b>. In some examples, the visible light camera module <b>206</b> may include its own dedicated processor or other circuitry (e.g., ASIC) capable of operating the visible light camera module <b>206</b>. In some such embodiments, the dedicated processor is in communication with processor <b>222</b> for providing visible light image data (e.g., RGB image data) to processor <b>222</b>. In alternative embodiments, a dedicated processor for the visible light camera module <b>206</b> may be integrated into processor <b>222</b>.

0055With each sensor element of visible light camera module <b>206</b> functioning as a sensor pixel, processor <b>222</b> can generate a two-dimensional image or picture representation of the visible light from a target scene by translating an electrical response of each sensor element into a time-multiplexed electrical signal that can be processed, e.g., for visualization on display <b>108</b> and/or storage in memory.

0056Processor <b>222</b> may control display <b>108</b> to display at least a portion of a visible light image of a captured target scene. In some examples, processor <b>222</b> controls display <b>108</b> so that the electrical response of each sensor element of visible light camera module <b>206</b> is associated with a single pixel on display <b>108</b>. In other examples, processor <b>222</b> may increase or decrease the resolution of a visible light image so that there are more or fewer pixels displayed on display <b>108</b> than there are sensor elements in visible light camera module <b>206</b>. Processor <b>222</b> may control display <b>108</b> to display an entire visible light image (e.g., all portions of a target scene captured by thermal imaging camera <b>100</b>) or less than an entire visible light image (e.g., a lesser port of the entire target scene captured by thermal imaging camera <b>100</b>).

0057In some embodiments, one or both of infrared <b>200</b> and visible light <b>206</b> camera modules for acquiring IR and VL image data may be included in an image acquisition module <b>280</b>. The image acquisition module may be in wired or wireless communication with a processing module <b>290</b> that includes a processor such as <b>222</b>. Processing module <b>290</b> may receive image data from the image acquisition module <b>280</b> and perform subsequent processing steps as will be described herein. In some examples, processing module <b>290</b> may include portable processing devices, such as a smartphone, a tablet, a stand-alone computer such as a laptop or desktop PC, or the like. In some such embodiments, various components of front end circuitry <b>202</b> may be included in the image acquisition module <b>280</b>, the processing module <b>290</b>, or both.

0058In these and other examples, processor <b>222</b> may control display <b>108</b> to concurrently display at least a portion of the visible light image captured by thermal imaging camera <b>100</b> and at least a portion of the infrared image captured by thermal imaging camera <b>100</b>. Such a concurrent display may be useful in that an operator may reference the features displayed in the visible light image to help understand the features concurrently displayed in the infrared image, as the operator may more easily recognize and distinguish different real-world features in the visible light image than the infrared image. In various examples, processor <b>222</b> may control display <b>108</b> to display the visible light image and the infrared image in side-by-side arrangement, in a picture-in-picture arrangement, where one of the images surrounds the other of the images, or any other suitable arrangement where the visible light and the infrared image are concurrently displayed.

0059For example, processor <b>222</b> may control display <b>108</b> to display the visible light image and the infrared image in a combined arrangement. In such an arrangement, for a pixel or set of pixels in the visible light image representative of a portion of the target scene, there exists a corresponding pixel or set of pixels in the infrared image, representative of substantially the same portion of the target scene. In various embodiments, the size and/or resolution of the IR and VL images need not be the same. Accordingly, there may exist a set of pixels in one of the IR or VL images that correspond to a single pixel in the other of the IR or VL image, or a set of pixels of a different size. Similarly, there may exist a pixel in one of the VL or IR images that corresponds to a set of pixels in the other image. Thus, as used herein, corresponding does not require a one-to-one pixel relationship, but may include mismatched sizes of pixels or groups of pixels. Various combination techniques of mismatched sized regions of images may be performed, such as up- or down-sampling one of the images, or combining a pixel with the average value of a corresponding set of pixels. Other examples are known and are within the scope of this disclosure.

0060Thus, corresponding pixels need not have a direct one-to-one relationship. Rather, in some embodiments, a single infrared pixel has a plurality of corresponding visible light pixels, or a visible light pixel has a plurality of corresponding infrared pixels. Additionally or alternatively, in some embodiments, not all visible light pixels have corresponding infrared pixels, or vice versa. Such embodiments may be indicative of, for example, a picture-in-picture type display as previously discussed. Thus, a visible light pixel will not necessarily have the same pixel coordinate within the visible light image as does a corresponding infrared pixel. Accordingly, as used herein, corresponding pixels generally refers pixels from any image (e.g., a visible light image, an infrared image, a combined image, a display image, etc.) comprising information from substantially the same portion of the target scene. Such pixels need not have a one-to-one relationship between images and need not have similar coordinate positions within their respective images.

0061Similarly, images having corresponding pixels (i.e., pixels representative of the same portion of the target scene) can be referred to as corresponding images. Thus, in some such arrangements, the corresponding visible light image and the infrared image may be superimposed on top of one another, at corresponding pixels. An operator may interact with user interface <b>208</b> to control the transparency or opaqueness of one or both of the images displayed on display <b>108</b>. For example, the operator may interact with user interface <b>208</b> to adjust the infrared image between being completely transparent and completely opaque and also adjust the visible light image between being completely transparent and completely opaque. Such an exemplary combined arrangement, which may be referred to as an alpha-blended arrangement, may allow an operator to adjust display <b>108</b> to display an infrared-only image, a visible light-only image, of any overlapping combination of the two images between the extremes of an infrared-only image and a visible light-only image. Processor <b>222</b> may also combine scene information with other data, such as radiometric data, alarm data, and the like. In general, an alpha-blended combination of visible light and infrared images can comprise anywhere from 100 percent infrared and 0 percent visible light to 0 percent infrared and 100 percent visible light. In some embodiments, the amount of blending can be adjusted by a user of the camera. Thus, in some embodiments, a blended image can be adjusted between 100 percent visible light and 100 percent infrared.

0062Additionally, in some embodiments, the processor <b>222</b> can interpret and execute commands from user interface <b>208</b>, and/or output/control device <b>210</b>. This can involve processing of various input signals and transferring those signals to the front end circuitry <b>202</b> via a connection therebetween. Components (e.g. motors, or solenoids) proximate the front end circuitry <b>202</b> can be actuated to accomplish the desired control function. Exemplary control functions can include adjusting the focus, opening/closing a shutter, triggering sensor readings, adjusting bias values, etc. Moreover, input signals may be used to alter the processing of the image data that occurs in the processor <b>222</b>.

0063Processor can further include other components to assist with the processing and control of the infrared imaging camera <b>100</b>. For example, as discussed above, in some embodiments, an ADC can be incorporated into the processor <b>222</b>. In such a case, analog signals conditioned by the front-end stages <b>204</b> are not digitized until reaching the processor <b>222</b>. Moreover, some embodiments can include additional on board memory for storage of processing command information and scene data, prior to transmission to the display <b>108</b> or the output/control device <b>210</b>.

0064An operator may interact with thermal imaging camera <b>100</b> via user interface <b>208</b>, which may include buttons, keys, or another mechanism for receiving input from a user. The operator may receive output from thermal imaging camera <b>100</b> via display <b>108</b>. Display <b>108</b> may be configured to display an infrared-image and/or a visible light image in any acceptable palette, or color scheme, and the palette may vary, e.g., in response to user control. In some examples, display <b>108</b> is configured to display an infrared image in a monochromatic palette such as grayscale. In other examples, display <b>108</b> is configured to display an infrared image in a color palette such as, e.g., amber, ironbow, blue-red, or other high contrast color scheme. Combinations of grayscale and color palette displays are also contemplated. In some examples, the display being configured to display such information may include processing capabilities for generating and presenting such image data. In other examples, being configured to display such information may include the ability to receive image data from other components, such as processor <b>222</b>. For example, processor <b>222</b> may generate values (e.g., RGB values, grayscale values, or other display options) for each pixel to be displayed. Display <b>108</b> may receive such information and map each pixel into a visual display.

0065While processor <b>222</b> can control display <b>108</b> to concurrently display at least a portion of an infrared image and at least a portion of a visible light image in any suitable arrangement, a picture-in-picture arrangement may help an operator to easily focus and/or interpret a thermal image by displaying a corresponding visible image of the same scene in adjacent alignment.

0066A power supply (not shown) delivers operating power to the various components of thermal imaging camera <b>100</b> and, in some examples, may include a rechargeable or non-rechargeable battery and a power generation circuit.

0067During operation of thermal imaging camera <b>100</b>, processor <b>222</b> controls infrared camera module <b>200</b> and visible light camera module <b>206</b> with the aid of instructions associated with program information that is stored in memory to generate a visible light image and an infrared image of a target scene. Processor <b>222</b> further controls display <b>108</b> to display the visible light image and/or the infrared image generated by thermal imaging camera <b>100</b>.

0068As noted, in some situations, it can be difficult to identify and differentiate between real-world (visible) features of the target scene in a thermal image. In addition to supplementing the infrared image with visible light information, in some embodiments, it can be useful to emphasize visible edges within the target scene. While in some instances the detection of visible light edges can be difficult to perform in an infrared image (e.g., across a uniform thermal scene), known edge detection methods can be performed on a corresponding visible light image of the same target scene. Because of the corresponding relationship between the infrared image and the visible light image, visible light pixels determined to represent a visible edge in the target scene correspond to infrared pixels also representing the visible edge in the infrared image. It will be appreciated that, as used herein, “edges” need not refer to the physical boundary of an object, but may refer to any sufficiently sharp gradient in the visible light image. Examples may include physical boundaries of an object, color changes within an object, shadows across a scene, and the like.

0069<figref idref="DRAWINGS">FIG. 4</figref> is a high-level schematic diagram illustrating an exemplary technique for enhancing edges in an IR image. In the illustrated embodiment, a visible light image <b>400</b> is processed <b>402</b>, which may include a variety of processing steps. For instance, in some examples, processing may include filtering the image for noise reduction, enhancing edges, or any other processing techniques. In some examples, processing may include an embossing or other edge detection processes for emphasizing edges from the VL image.

0070In the illustrated embodiment, the resulting processed VL image is combined with an edge midscale value <b>404</b> to generate a modified VL image <b>406</b>. The edge midscale value <b>404</b> may include, for instance, a specific color. In some embodiments, the edge midscale value <b>404</b> may be selectable by a user. In some examples, combining the processed VL image <b>402</b> with the edge midscale value <b>404</b> comprises combining the processed VL image with an image consisting exclusively of the edge midscale value <b>404</b>. That is, in some embodiments, a processed VL image is created and an edge midscale value “image” is created or recalled from memory. These two images are combined by any appropriate combination method to generate the modified VL image <b>506</b>.

0071In other embodiments, the combining can include combining each pixel of the processed VL image <b>402</b> with the edge midscale value. That is, the combination is not done image-wide at once, but rather, on a pixel-by-pixel or cluster-by-cluster basis. The camera need not store or create an edge midscale value “image”, but rather may perform pixel-wise combinations of processed VL image data and the edge midscale value to determine modified VL pixel data. The modified VL image <b>406</b> may include emphasized edge information as will be described below. The modified VL image <b>406</b> can be combined with an IR image <b>408</b> corresponding to the VL image <b>400</b> to create a blended final image <b>410</b>. The resulting blended final image <b>410</b> may include both IR image data and emphasized edge information from the VL image <b>400</b>. As described herein, combining (e.g., the processed VL image <b>402</b> and the edge midscale value <b>404</b>, or the IR image <b>408</b> and the modified VL image <b>406</b>) may be performed in any of a variety of known image combination techniques, such as addition, averaging, alpha-blending, and the like.

0072<figref idref="DRAWINGS">FIG. 5A</figref> is a schematic diagram illustrating an exemplary VL image processing technique used for generating a modified VL image. According to the illustrated embodiments, VL image data <b>500</b> is directed to a VL processing engine <b>502</b>. In some embodiments, processing engine <b>502</b> can be part of processor <b>222</b> or may function separately from processor <b>222</b>. In some examples, the VL processing engine <b>502</b> can determine a pseudo-luminance value <b>512</b> for one or more pixels in the VL image. The pseudo-luminance value <b>512</b> of a pixel can include the true luminance (i.e., the Y component in the YCbCr representation of the pixel) or any other single-valued scalar representation of the pixel. For instance, in some examples, the pseudo-luminance value <b>512</b> for a pixel includes a linear combination of the R, G, and B components of the pixel representation in the RGB color space. In an exemplary embodiment, the pseudo-luminance value for a given pixel, i, can be determined by the equation:

0000<br />PseudoLum(<i>i</i>)=4×<i>R</i>(<i>i</i>)+8×<i>G</i>(<i>i</i>)+4×<i>B</i>(<i>i</i>)

0073Once a pseudo-luminance value <b>512</b> is determined for one or more pixels, a processing kernel <b>514</b> may be applied across the VL image, for example, across one or more pixels in the VL image <b>500</b>. In some examples, the processing kernel <b>514</b> can be used to determine an edge factor value <b>516</b> for one or more pixels within the kernel that is generally representative of the “strength” of an edge at a given pixel. In some examples, determining the edge factor value <b>516</b> may include performing an embossing process for determining a value for each pixel representative of the presence and/or strength of a VL edge at that pixel. With reference to the processing kernel <b>514</b> in <figref idref="DRAWINGS">FIG. 5A</figref>, each entry (1-9) in the kernel <b>514</b> may correspond to a single pixel in the VL image, and each such pixel may have an associated pseudo-luminance value <b>512</b>. In other examples, each entry in the processing kernel <b>514</b> may correspond to a plurality of pixels within the VL image (e.g., a defined group of pixels such as a square or rectangle that may be averaged together, such as in a filtering or downsampling process). In some embodiments, the processing kernel <b>514</b> as illustrated in <figref idref="DRAWINGS">FIG. 5A</figref> may be used to determine an edge factor value (EFV) <b>516</b> for the center pixel of the kernel <b>514</b>. For example, with reference to the illustrated embodiment, the difference in the pseudo-luminance values of diagonally opposite corner pixels may be used to determine the edge factor value of the center pixel. Note that an edge factor value determined by taking a difference between values could be positive or negative. In some examples, the edge factor value <b>516</b> may be scaled by a user-adjustable edge gain <b>518</b> value. Thus, in an exemplary embodiment, the edge factor value <b>516</b> for the center pixel, 5, in the processing kernel <b>514</b> can be determined by the equation:

0000<br />EFV(5)=edgeGain×[PseudoLum(1)−PseudoLum(9)]

0074The resulting edge factor value corresponds to the “strength” of an edge at that pixel. That is, an edge factor value having a large magnitude generally corresponds to a sharp edge or contour within the VL image <b>500</b>. It will be appreciated that a variety of equations or methods may be suitable for determining an edge factor value <b>516</b> of one or more pixels in the processing kernel <b>514</b>. For example, rather than subtracting associated values for pixels <b>1</b> and <b>9</b>, similar pixel comparison (e.g., subtraction) may be performed for other pairs or groups of pixels. In some embodiments, a user may select which pixels are used for calculating the edge factor value. By doing so, a user may influence how strongly edges in certain directions are reflected in the edge factor values. Similarly, defining which pixels are used in determining the edge factor value may allow a user to ensure that edges in a particular direction are less likely to be missed by the EFV calculation. Exemplary comparisons may include

0000<br />PseudoLum(3)−PseudoLum(7)

0000<br />PseudoLum(6)−PseudoLum(4)

0000<br />PseudoLum(2)−PseudoLum(8)

0000and the like. Additionally, the kernel need not be limited to a 3×3 kernel as shown. Rather, the processing kernel <b>514</b> may be any size or shape, such as an n×n square or an m×n rectangle.

0075In various examples, different equations or methods for determining edge factor values <b>516</b> may be best suited for processing kernels of particular size and/or shape. For example, in some embodiments, a plurality of kernel entries may be compared (e.g., via subtraction) and used in determining the EFV. In an exemplary embodiment, a plurality of subtractions may be performed (e.g., opposite corners, top and bottom coordinates, left and right coordinates, etc.) and the subtractions averaged for determining the EFT. In another example, the largest difference magnitude from the set of subtractions is used to determine the EFV. Additionally or alternatively, in some examples, a single processing kernel <b>514</b> may be used to determine edge factor values for multiple pixels. Once the processing engine <b>502</b> has determined the edge factor value <b>516</b> for a pixel, the edge factor value <b>516</b> may be combined with the edge midscale value <b>504</b> as described above for generating a corresponding pixel in the modified VL image <b>506</b>. In general, the processing kernel <b>514</b> may be moved across the entire VL image in order to determine an edge factor value for each pixel, or may be moved in order to determine an edge factor value for a subset of the pixels in the VL image.

0076<figref idref="DRAWINGS">FIG. 5B</figref> is a schematic diagram illustrating another exemplary VL image processing technique used for generating a modified VL image. The process of <figref idref="DRAWINGS">FIG. 5B</figref> is similar to that of <figref idref="DRAWINGS">FIG. 5A</figref>, though various steps may be performed differently or in different orders. For instance, in the embodiment of <figref idref="DRAWINGS">FIG. 5B</figref>, the VL image data <b>500</b> is processed in processing kernel <b>514</b>. Similarly to the kernel described above with regard to <figref idref="DRAWINGS">FIG. 5A</figref>, kernel <b>514</b> of <figref idref="DRAWINGS">FIG. 5B</figref> may be any size, and may be used to process a subset of VL image pixels at a time.

0077In the illustrated example of <figref idref="DRAWINGS">FIG. 5B</figref>, the processing kernel <b>514</b> receives the edge midscale value <b>504</b> and edge gain <b>518</b> parameters. The processes shown as being performed in in <figref idref="DRAWINGS">FIG. 5A</figref> may be performed on pixels in the kernel <b>514</b> of <figref idref="DRAWINGS">FIG. 5B</figref>. That is, the pseudo-luminance values <b>512</b> may be determined for the pixels within the kernel and used in conjunction with the edge gain <b>518</b> to determine an edge factor value <b>516</b> for one or more pixels within the kernel. The one or more edge factor values determined within the kernel <b>514</b> may be combined with the edge midscale value <b>504</b> within the kernel <b>514</b>.

0078In some such embodiments, the processing kernel <b>514</b> may output one or more pixels for inclusion in the modified VL image <b>506</b>. That is, in some embodiments, a subset of VL image pixels are entered into processing kernel <b>514</b>, in which processing techniques are performed only on that subset of pixels. The processing may result in one or more pixels for inclusion in the modified VL image <b>506</b>. After the processing is complete, the kernel <b>514</b> may be moved relative to the VL image <b>500</b> (or, equivalently, a different subset of pixels of the VL image <b>500</b> are applied to the processing kernel <b>514</b>) for further processing and generating one or more additional pixels for inclusion in the modified VL image <b>506</b>. The process may be repeated for a plurality of subsets of VL image pixels until the entire modified VL image <b>506</b> is constructed.

0079<figref idref="DRAWINGS">FIG. 6A</figref> is a schematic diagram illustrating the combining of an edge factor value with an edge midscale value in RGB color space to determine a modified VL image pixel. In the illustrated embodiment, the edge factor value <b>616</b> for a given pixel is combined with each of the R, G, and B values of the edge midscale value <b>604</b>. In general, the edge midscale value may be any color. In various embodiments, the edge midscale value may be a fixed color stored in memory, or may be definable or otherwise selectable by a user. Since the edge factor value <b>616</b> as described above comprises a single value for each pixel, the same edge factor value <b>616</b> may be added to each RGB channel of the edge midscale value <b>604</b>, effectively adjusting the R, G, and B values of the edge midscale value <b>604</b> to create the modified VL image <b>606</b>.

0080In some examples, the edge factor value <b>616</b> needs to be scaled to an appropriate size in order to combine the value with each of the RGB channels. Thus, in the illustrated embodiment, the edge factor value is scaled <b>620</b> prior to being added to the various channels. In various embodiments, the number of bits associated with each channel may not be the same. For example, in an exemplary embodiment, the R and B channels may be 5 bits while the G channel may be 6 bits, using 16 bits total. Any number of bits per channel may be used, and any number of channels may have like or different numbers of bits associated therewith.

0081For a given pixel, adding the EFV <b>616</b> to the RGB values of the edge midscale value <b>604</b> defines the resulting color for the corresponding pixel in the modified VL image <b>606</b>. For example, in the illustrated embodiment, the edge factor value <b>616</b> is scaled and added to the R channel of the edge midscale value <b>604</b> (R_emv), resulting in the R channel of the modified VL pixel <b>606</b> (R_VL′). The same is performed for the G and B channels. That is:

0000<br /><i>R</i>_emv+EFV=<i>R</i>_<i>VL′</i>

0000<br /><i>G</i>_emv+EFV=<i>G</i>_<i>VL′</i>

0000<br /><i>B</i>_emv+EFV=<i>B</i>_<i>VL′</i>

0000wherein the edge factor value may be scaled differently for each channel.

0082In general, values in the R, G, and B channels are confined to one or more certain ranges. For example, in 24-bit color representation, each of the R, G, and B channels may include 8 bits of information, ranging, for example, between 0 and 255. In other examples, such as a 16-bit color representation, each of the R, G, and B channels may be confined to different ranges. For instance, two channels may be limited to 5 bits (e.g., 0-31) while the third channel is limited to 6 bits (e.g., 0-63). Thus, the sum of an edge midscale value channel and an edge factor value may fall outside of the range (e.g., below zero if the edge factor value is negative or above 255 if the edge factor value is positive). In some examples, this results in saturation of the channel, and the resulting sum is defined to be the limit of the range. For example, if the R channel of the edge midscale value <b>604</b> (R_emv) is 240, and the scaled edge factor value <b>616</b> to be added to R_emv is 50,

0000<br /><i>R</i>_<i>VL′=R</i>_emv+EFV=240+50=295→255

0000That is, even though the sum (295) is above the upper limit of the R_VL′ range (255), the resulting value in the modified VL pixel <b>606</b> is saturated at the upper limit of the range (255).

0083Generally, adding the same value (e.g., the edge factor value <b>616</b>) to each of the R, G, and B channels of a pixel results in a change in the luminance of the pixel while preserving its chrominance. However, as described above, the edge midscale value <b>604</b> may be any color. In various embodiments, the edge midscale value <b>604</b> may be a specific color programmed into a camera, selectable from a predetermined list, or customizable by a user. Thus, in some situations, the R, G, and B values of the edge midscale value <b>604</b> are not necessarily equal. As a result, adding the edge factor value <b>616</b> to each of the R, G, and B channels may result in saturation of some channels but not other channels. For example, if R_emv=240, G_emv=100, and B_emv=100, and the edge factor value <b>616</b> is 50, then,

0000<br /><i>R</i>_<i>VL′=</i>240+50=295→255

0000<br /><i>G</i>_<i>VL′=</i>100+50=150

0000<br /><i>B</i>_<i>VL′=</i>100+50=150

0000Thus, in adding the edge factor value <b>616</b> to the edge midscale value <b>604</b>, the R channel became saturated, but not the G or B channels. As a result, the R channel only increased by 15, while the G and B channels increased by 50. Saturation of some but not all channels may result in a difference in chrominance between the edge midscale value <b>604</b> and the modified VL image <b>606</b>.

0084<figref idref="DRAWINGS">FIG. 6B</figref> is a schematic diagram illustrating the combining of an edge factor value with an edge midscale value in YCbCr color space to determine a modified VL image pixel. In the illustrated embodiment, the edge factor value <b>616</b> is scaled and added to the luminance (Y) channel of the edge midscale value <b>604</b> to determine the luminance channel of the modified VL image <b>606</b> (Y_VL′). The chrominance channels (Cb, Cr) of the edge midscale value <b>604</b> are unaffected by the edge factor value <b>616</b>, and become the chrominance channels of the modified VL image (Cb_VL′ and Cr_VL′).

0085As discussed, adding the edge factor value <b>616</b> to the edge midscale value <b>604</b> only affects the chrominance of the edge midscale value <b>604</b> if one or more, but not all, of the R, G, and B channels of the edge midscale value <b>604</b> become saturated. When this is not the case, chrominance of the edge midscale value <b>604</b> is unaffected by the adding of the edge factor value <b>616</b>, as illustrated in <figref idref="DRAWINGS">FIG. 6B</figref>. Thus, the chrominance channels of the edge midscale value <b>604</b> become the chrominance channels of the modified VL image.

0086In some embodiments, the edge midscale value <b>604</b> may be a shade of gray, for example, if R_emv=B_emv=G_emv. It will be appreciated that if such values are scaled to different sizes (e.g., 5-bit vs. 6-bit), the values may not be literally equal, though may represent equivalent values relative to their respective bit depths. Thus, as used herein, being “equal” may refer to numbers that are equivalent in value (e.g., relative to the number of bits) without necessarily being equal in size (e.g., the number of bits). In such an embodiment, with reference to <figref idref="DRAWINGS">FIG. 6A</figref>, adding the edge factor value <b>616</b> to each channel will not result in saturation of one channel without also saturating the others. That is, each of the R, G, and B channels of the edge midscale value <b>604</b> are affected uniformly by the edge factor value. Thus, the chrominance channels of the edge midscale value <b>604</b> are unaffected by the edge factor value, and the representation of <figref idref="DRAWINGS">FIG. 6B</figref> applies.

0087In an exemplary calculation, if R_emv=G_emv=B_emv, then the edge midscale value is gray and Cb_emv=Cr_emv. In an exemplary technique incorporating 24-bit color depth, Cb_emv=Cr_emv=128. Adding the edge factor value <b>616</b> to each of the R, G, and B channels of the edge midscale value <b>604</b> will not affect the equivalence relationship between the channels, and the chrominance components will be unaffected. Accordingly, after adding the edge factor value <b>616</b> to the edge midscale value <b>604</b>, only the luminance value will be affected. That is:

0000<br /><i>Y</i>_<i>VL′=Y</i>_emv+EFV

0000<br /><i>Cb</i>_<i>VL′=Cb</i>_emv=128

0000<br /><i>Cr</i>_<i>VL′=Cr</i>_emv=128

0000As shown, since the edge midscale value <b>604</b> started out in grayscale (R=G=B; equivalently Cb=Cr=128), the resulting modified VL image <b>606</b> will also be grayscale. The luminance of the modified VL image <b>606</b> will be the luminance component of the edge midscale value <b>604</b> offset (plus or minus) by edge factor value <b>616</b>.

0088It will be appreciated that, while <figref idref="DRAWINGS">FIG. 6B</figref> has been discussed in terms of a grayscale edge midscale value <b>604</b>, in some examples, generating the modified VL image <b>606</b> may be performed in the YCbCr color space according to the embodiment of <figref idref="DRAWINGS">FIG. 6B</figref> whether or not the edge midscale value <b>604</b> is gray. That is, in some examples, the modified VL image <b>606</b> may be constructed by adding the edge factor value <b>616</b> to the luminance channel of the edge midscale value <b>604</b> and defining the chrominance channels of the modified VL image <b>606</b> to be equal to the chrominance channels of the edge midscale value <b>604</b>.

0089Various combinations of an edge factor value <b>616</b> and a generic (e.g., gray, or any color that may be programmed into memory or selected by the user) edge midscale value <b>604</b> have been described. It will be appreciated that the combination may be performed in a variety of ways in any appropriate color space. RGB and YCbCr color spaces are described herein by way of example, but similar methods may be performed using, for example, CYMK, HSL, or HSV color space representations. Adding the edge factor value <b>616</b> to an edge midscale value <b>604</b> may be done in a single channel (e.g., the luminance channel in <figref idref="DRAWINGS">FIG. 6B</figref>) or a plurality of channels (e.g., each of the R, G, and B channels in <figref idref="DRAWINGS">FIG. 6A</figref>).

0090While shown in <figref idref="DRAWINGS">FIGS. 6A and 6B</figref> as being a scalar value added to one or more channels of the edge midscale value, various other options are possible. For example, with reference to <figref idref="DRAWINGS">FIG. 6A</figref>, in some embodiments, the edge factor value may be added to fewer than all of the R, G, and B channels, such as according to the selection of a user. In doing so, pixels enhanced by large-magnitude edge factor values may be selectively shifted toward a certain color in the modified VL image. Similarly, with reference to <figref idref="DRAWINGS">FIG. 6B</figref>, the edge factor value may be added to one or more chrominance channels, in addition to or instead of the luminance channel, of the edge midscale value.

0091In some embodiments, the edge factor value may be a multi-channel color value as opposed to a single scalar value. For example, instead of converting the VL image to a scalar pseudo-luminance value prior to determining the edge factor value, the same processing techniques may be performed on one or more color channels of the VL image. As a result, the edge factor value may comprise a plurality of color channels for combining with the edge midscale value. Combining of pixels in multiple color channels can be performed in a variety of ways (e.g., addition, blending, etc.).

0092In other embodiments, scalar edge factor values may be determined such as described above, but scalar values may be mapped to a multiple-channel color space representation. Such a mapping may be performed in a variety of ways. In some embodiments, scalar edge factor values are effectively palettized in a similar way to palettization schemes of IR image data. The resulting multi-channel edge factor values may be combined with multiple channels of the edge midscale value to generate the modified VL image. In some such examples, the EFV color palette may be chosen to be complementary to the IR palettization scheme. Examples may include a grayscale IR image with an amber EFV palette in order to ultimately show visible edges as a degree of amber (yellow/red) within a grayscale IR image. In some embodiments, both palettization schemes are selectable by a user. In other embodiments, a user may select one of such palettization schemes and a complementary palettization scheme is automatically used for the other. In some examples, while both palettization schemes may be selectable by a user, the user may be alerted via the user interface, after the selection of one palettization scheme, of possible complimentary palettization schemes.

0093The result of the combination is a modified VL image <b>606</b> in which each pixel includes the edge midscale value <b>604</b> affected by the edge factor value of the corresponding pixel from the VL image. As noted above, the magnitude of the edge factor value <b>616</b> generally corresponds to the “strength” of an edge at a corresponding VL pixel. As such, the resulting modified VL image <b>606</b> generally includes the edge midscale value <b>604</b> offset by the edge factor value, and wherein pixels corresponding to edges in the VL image are most affected by the addition of the edge factor value <b>616</b>. Accordingly, pixels corresponding to the edges of the VL image are generally the most departed from the edge midscale value <b>604</b> in the modified VL image <b>606</b>.

0094With reference to <figref idref="DRAWINGS">FIG. 4</figref>, the modified VL image <b>406</b> may be combined with a corresponding IR image <b>408</b> to create a blended final image <b>410</b>. <figref idref="DRAWINGS">FIGS. 7A and 7B</figref> are schematic diagrams illustrating exemplary techniques for combining the modified VL image and a corresponding IR image. As shown in <figref idref="DRAWINGS">FIG. 7A</figref>, the R, G, and B channels of a pixel in the modified VL image are averaged with the respective R, G, and B channels of a corresponding IR image pixel. The RGB values associated with pixels in the IR image <b>708</b> may be defined according to a palettization scheme, in which pixels are assigned to colors within a palette based on associated IR image data. As described above, corresponding pixels need not have a one-to-one relationship, but rather can include one or more pixels in the IR image <b>708</b> corresponding to one or more pixels in the modified VL image <b>706</b>.

0095Averaging can include alpha-blending, in which a weighting factor determines the contribution from each of the IR <b>708</b> and modified VL image <b>706</b> pixels. In an exemplary blending procedure, for a blending ratio, alpha (α),

0000<br /><i>R</i>_Blend=α×<i>R</i>_IR+(1−α)×<i>R</i>_<i>VL′</i>

0000<br /><i>G</i>_Blend=α×<i>G</i>_IR+(1−α)×<i>G</i>_<i>VL′</i>

0000<br /><i>B</i>_Blend=α×<i>B</i>_IR+(1−α)×<i>B</i>_<i>VL′</i>

0096The blending ratio may be user-selectable or adjustable to enable a user to select the amount of blending. As previously described, in some embodiments, alpha can range from 0 to 1 (i.e., 0% IR to 100% IR). In other embodiments, there may be a minimum and/or maximum blending ratio between 0 and 1. In various embodiments the blending ratio may be constant among all pixels in the image, or may vary from pixel to pixel. In general, the edges from the VL image emphasized by the VL image processing and present in the modified VL image will be accentuated in and effectively enhance the blended image.

0097In some embodiments, the blending ratio for each pixel may be determined automatically based on one or more parameters. In some examples, the blending ratio for each pixel may be a function of the edge factor value (e.g., <b>616</b>) associated with that pixel. For example, the blending ratio may vary inversely with the magnitude of the edge factor value. That is, if the magnitude of the edge factor value for a pixel in the VL image is small, the blending ratio may be large, resulting in a corresponding pixel in the blended image <b>710</b> with a relatively large contribution from the corresponding IR pixel <b>708</b>. On the other hand, if the magnitude of the edge factor value for a pixel in the VL image is large, the blending ratio may be comparatively small, resulting in a corresponding pixel in the blended image with a small contribution from the corresponding IR pixel.

0098The result is a blended image <b>710</b> in which pixels not corresponding to edges in the VL image (i.e., a low edge factor value magnitude) have a large contribution from the IR image. By contrast, pixels corresponding to edges in the VL image (i.e., a large edge factor value magnitude) will have stronger contributions from the modified VL image. As a result, pixels that do not correspond to edges in the VL image appear similar to the corresponding IR pixels from the corresponding IR image.

0099In an exemplary embodiment, when the edge factor value is below a certain threshold, the blending ratio, a, is set to 1. That is, the when there are no edges (or edges below a predetermined or user-adjustable threshold “strength”) in a pixel in the VL image, the corresponding pixel in the blended image is 100% the corresponding pixel in the IR image <b>708</b>. Such pixels retain the RGB values from the original IR image <b>708</b>, and are therefore recognizable on the original RGB palette. By contrast, the “stronger” the edge in the VL image, the greater the contribution of the modified VL image <b>706</b> to the blended image <b>710</b>. The colors of the pixels corresponding to “stronger” edges are affected more by blending than non-edges, and generally do not retain the same RGB values from the original IR image. Thus, the edges emphasized by the VL image processing process can be easily seen in the blended image <b>710</b> due to the blended contribution of the modified VL image <b>706</b>.

0100In other embodiments, the blended image <b>710</b> will include a minimum amount of IR image contribution and a minimum amount of VL image contribution. For example, the blending ratio, a, may be precluded from being at least one of 0 and 1, but instead may be limited to a smaller range of values. It will be appreciated that any limitations on the blending ratio are possible. In addition, other blending schemes may be employed. For example, the blending ratio may reflect the amount of VL image contribution to the blended image instead of the amount of IR image contribution. In some such embodiments, the blending ratio may be a function of the edge factor value, and may increase with increasing edge factor values.

0101It will be appreciated that, additionally or alternatively, the blending ratio for each pixel may be a function of other parameters, such as parameters of the IR image pixel. For example, in some embodiments, the blending ratio for a given pixel is a function of a magnitude of the pixel in the IR image data (e.g., a scalar value such as a luminance or pseudo-luminance or one or more of associated R, G, or B values in the palettized IR image). In such examples, various aspects of the thermal scene may contribute to determining the amount of IR image information and modified VL image information is imported into the blended image. In some examples, the blending ratio as described above may be inversely related to an IR pixel magnitude, or similarly, the temperature of the target scene represented by the IR pixel. That is, in some embodiments, a blended image may include a larger contribution of the modified VL image at higher temperatures than lower temperatures, generally emphasizing visible edges in the higher-temperature regions. It will be appreciated that, in general, individual pixel blending ratios dependent on IR intensity may have any functional relationship with the IR intensity values, such as having higher temperature, lower temperature, or a certain range of temperature including a higher or lower modified VL image contribution to the blended image.

0102While a variety of blending techniques have been described, the overall blending ratio may be determined based on a combination of these. For example, the overall blending ratio for a given pixel may be a combination of blending ratio determined by parameters such as the edge factor value and the intensity of corresponding IR pixels. In further embodiments, such pixel-dependent blending ratios may also be modified by an adjustable global blending radio. In an exemplary embodiment, blending ratios, α_i, are defined per pixel based on the corresponding edge factor value (α_EFV) and the corresponding IR pixel intensity (α_IR), as well as a global blending ratio (α_global). The overall blending ratio for a given pixel may include a combination of such blending ratios, such as a weighted average:

0000<br />α_net=<i>c</i>_1×α_EFV+<i>c</i>_2×α_IR+<i>c</i>_3×α_global

0000wherein c1, c2, and c3 are weighting factors. In various embodiments, any parameters involved in the blending process, such as the blending options (e.g., EFV- and/or IR-dependent), values (e.g., global blending) or the relative contribution of each blending technique (e.g., c_i) may be selectable or adjustable by a user.

0103It should be noted that, in some embodiments, other factors may be dependent on the IR image intensity in order to selectively emphasize details in areas having certain thermal characteristics. For instance, the edge gain value used in determining edge factor values for VL image pixels may be dependent on the IR image data for corresponding IR image pixels. That is, VL edges emphasized by the VL image processing may be more or less emphasized due to the thermal characteristics of a corresponding portion of the target scene. Accordingly, pixels in the blended image having a certain thermal profile (e.g., high temperature, low temperature, within a temperature range, etc.) may include more strongly emphasized edges than other pixels.

0104<figref idref="DRAWINGS">FIG. 7B</figref> illustrates an exemplary blending process in YCbCr color space. In the illustrated embodiment, the luminance, Y, and the chrominance, Cr and Cr, channels of a modified VL image pixel <b>706</b> are averaged with corresponding channels of a corresponding IR image pixel <b>708</b>. As described above, averaging may include weighted averaging using a blending ratio, a. Similar to the RGB blending, in an exemplary blending process,

0000<br /><i>Y</i>_Blend=α×<i>Y</i>_IR+(1−α)×<i>Y</i>_<i>VL′</i>

0000<br /><i>Cb</i>_Blend=α×<i>Cb</i>_IR+(1−α)×<i>Cb</i>_<i>VL′</i>

0000<br /><i>Cr</i>_Blend=α×<i>Cr</i>_IR+(1−α)×<i>Cr</i>_<i>VL′</i>

0000As described above with regard to blending in the RGB color space, for a given pixel, the blending ratio, α, may be a function of the edge factor value associated with the corresponding VL pixel.

0105As described with regard to <figref idref="DRAWINGS">FIG. 6B</figref>, in some instances, the chrominance components of the modified VL image pixel (Cb_VL′, Cr_VL′) may correspond to the chrominance components of the edge midscale value (Cb_emv, Cr_emv). In some such instances, the chrominance components of a pixel in the blended image <b>710</b> may include a weighted blend of the chrominance of the IR image pixel <b>708</b> and the chrominance components of the edge midscale value. Additionally, in some such instances, the luminance component of the modified VL image pixel (Y_VL′) is the luminance component of the edge midscale value offset by the edge factor value associated with the corresponding VL image pixel. Accordingly, the luminance component of a pixel in the resulting blended image <b>710</b> may include a weighted blend of the luminance component of the corresponding IR pixel <b>708</b> and the luminance component of the edge midscale value offset by the edge factor value. It will be appreciated that, while two examples are given in <figref idref="DRAWINGS">FIGS. 7A and 7B</figref>, the blending processes such as those described may be performed in any appropriate color space.

0106In some examples, none of the images are palettized until the final blended image. That is, the VL image may be converted into a scalar pseudo-luminance value used to determine a scalar edge factor value. The scalar edge factor value may be combined with a gray edge midscale value to create a modified VL image having essentially scalar pixel values. These values may be combined with scalar (grayscale) IR image data to generate scalar values for pixels in the blended image. A single palettization scheme may be applied to the scalar blended image to create a colorized image.

0107In a similar embodiment, pseudo-luminance values may be used to determine a scalar edge factor value. The scalar edge vector value may be combined with a gray edge midscale value to create a modified VL image having essentially scalar pixel values. The resulting modified VL image may then be palettized according to any appropriate palettization scheme prior to blending with grayscale or color-palettized IR image data.

0108<figref idref="DRAWINGS">FIGS. 8A and 8B</figref> are schematic diagrams illustrating a process for generating a blended image. In <figref idref="DRAWINGS">FIG. 8A</figref>, R, G, and B channels of a VL image <b>800</b> (R_VL, G_VL, B_VL) are converted into a pseudo-luminance <b>812</b> (Y′). Pseudo-luminance values <b>812</b> are processed to determine an edge factor value <b>816</b> for each pixel, for example, as described above with reference to <figref idref="DRAWINGS">FIG. 5A</figref>. In the illustrated example of <figref idref="DRAWINGS">FIG. 8A</figref>, the edge factor value <b>816</b> is combined with each of the R, G, and B channels (R_emv, G_emv, and B_emv, respectively) of an edge midscale value <b>804</b>, for example, as described above with reference to <figref idref="DRAWINGS">FIG. 6A</figref>, to generate a modified VL image <b>806</b> with channels R_VL′, G_VL′, and B_VL′. As described above, the combination of the edge factor value <b>816</b> with the edge midscale value <b>804</b> generally emphasizes pixels corresponding to edges present in the VL image <b>800</b>.

0109The R, G, and B channels of the modified VL image <b>806</b> (R_VL′, G_VL′, B_VL′) are averaged with corresponding R, G, and B channels of the IR image <b>808</b> (R_IR, G_IR, B_IR) to generate the blended image <b>810</b>. Blending can be performed as described, for example, with reference to <figref idref="DRAWINGS">FIG. 7A</figref>. In some embodiments, the averaging may include a weighted average using a blending ratio, a, as described above. In some such embodiments, the blending ratio for each pixel may be dependent on the edge factor value <b>816</b> associated with a corresponding VL pixel, as illustrated by the broken lines in <figref idref="DRAWINGS">FIG. 8A</figref>. The resulting blended image <b>810</b> generally includes IR image information and emphasized edges from the contribution of the modified VL image.

0110<figref idref="DRAWINGS">FIG. 8B</figref> is a schematic diagram illustrating a blending process in which the blending is performed in YCbCr color space. The process illustrated in <figref idref="DRAWINGS">FIG. 8B</figref> is similar to that of <figref idref="DRAWINGS">FIG. 8A</figref>, in which the R, G, and B values of the VL image <b>800</b> are used to determine a pseudo-luminance value <b>812</b>. It will be appreciated that other methods for determining pseudo-luminance values using other color space values of the VL image <b>800</b> may be used. Pseudo-luminance values <b>812</b> are used to determine an edge factor value <b>816</b> for each pixel. In the illustrated embodiment, as described above with reference to <figref idref="DRAWINGS">FIG. 6B</figref>, the luminance component (Y_emv) of the edge midscale value <b>804</b> may be combined with the edge factor value in order to determine the luminance component of the modified VL image <b>806</b>. The chrominance components of the edge midscale value <b>804</b> (Cb_emv, Cr_emv) may be unaffected by the edge factor value and generally used as the chrominance components of the modified VL image.

0111As shown, the Y, Cb, and Cr components of the modified VL image <b>806</b> are averaged with corresponding components of the IR image <b>808</b>, for example, as described above with reference to <figref idref="DRAWINGS">FIG. 7B</figref>. In such an embodiment, the resulting blended image <b>810</b> includes chrominance values that are a blend of the chrominance values of the edge midscale value <b>804</b> and the IR image <b>808</b>. However, the luminance component of the blended image <b>810</b> includes the luminance component of the IR image <b>808</b> and the luminance component of the edge midscale value <b>804</b> offset by the edge factor value <b>816</b>. Thus, edge factor values <b>816</b> representative of edges from the VL image <b>800</b> are reflected in the luminance channel of the blended image <b>810</b> and may therefore be visible in the blended image <b>810</b>. As described above, in some examples, the blending ratio of the IR and modified VL images at a given pixel may be a function of the edge factor value associated with a corresponding pixel in the VL image as illustrated by the broken lines in <figref idref="DRAWINGS">FIG. 8B</figref>.

0112In general, as described above with reference to <figref idref="DRAWINGS">FIG. 5A</figref>, the edge factor value for a given pixel of the VL image may correspond to the “strength” of an edge at that pixel. According to some embodiments as described above, the edge factor value may be used to generate the modified VL image, which is then blended with the IR image. As a result, the “strength” of edges in the VL image is ultimately represented in the blended image. This provides for added clarity and context of items in the imaged scene in the resulting blended image.

0113<figref idref="DRAWINGS">FIG. 9A</figref> is a flow diagram showing a series of example images used in creating a blended image. As shown, a VL image <b>900</b> is processed <b>902</b> to determine edge factor values associated with the VL pixels. In the illustrated embodiment, the VL image <b>900</b> is shown in grayscale. In some examples, the scalar grayscale values of the grayscale pixels may be used as pseudo-luminance values described elsewhere herein. That is, in some such examples, determining a pseudo-luminance value may be implicitly built in to the VL image.

0114The edge factor value is then combined with an edge midscale value <b>904</b> to create a modified VL image <b>906</b>. As shown, the modified VL image <b>806</b> appears to be similar to the edge midscale value <b>904</b>, but darkened or lightened at various locations corresponding to edges seen in the VL image <b>900</b>. According to some embodiments, a greater departure from the edge midscale value <b>904</b> corresponds to a larger edge factor value (i.e., a “stronger” edge). Thus, the VL image processing (e.g., determining the edge factor value and combining with the edge midscale value) causes edges within the VL image to stand out from a background that is approximately the edge midscale value <b>904</b> in the modified VL image <b>906</b>. As described above, in some examples, the edge factor value includes a user-adjustable edge gain parameter, allowing a user to adjust the amount the edges stand out due to the VL image processing.

0115The resulting modified VL image <b>906</b> is subsequently blended with a corresponding IR image <b>908</b> to produce a blended image <b>910</b>. In the illustrated embodiment, the IR image <b>908</b> is representative of a generally uniform thermal scene. That is, there is little temperature variation across the scene. As a result, the IR image <b>908</b> is generally constant across the entire image. However, because of the blending of the IR image <b>908</b> and the modified VL image <b>906</b>, the edges emphasized in the modified VL image <b>906</b> are visible in the blended image <b>910</b>. Additionally, the blended image <b>910</b> includes information from the IR image <b>908</b> as well. That is, it is apparent in the blended image <b>910</b> that the thermal scene is relatively uniform due to little variation outside of the enhanced edges. In addition, it will be appreciated that while the illustrated example demonstrates the IR image being represented by a grayscale palette, any palettization scheme may be used. Similarly, while the VL image is shown in grayscale, other known VL visualization schemes may be used.

0116In general, the combination of the IR image data with the modified VL image data will tend to preserve at least relative temperature information between pixels in the blended image in locations having similar edge factor values (e.g., locations having few or no VL edges). Because the modified VL image data is generally the edge midscale value modified by the edge factor value, only those pixels corresponding to edges in the VL image depart significantly from the edge midscale value in the modified VL image <b>906</b>. Accordingly, additional detail from the VL image <b>900</b> (e.g., colors or gradual shifts in color or shading, such as due to variations in scene illumination or light intensity, etc.) does not obscure IR image data when creating the blended image <b>910</b>. Instead, in some embodiments, pixels not corresponding to strong edges in the VL image (and thus having an associated edge factor value having a low magnitude) are generally represented by the edge midscale value <b>904</b> in the modified VL image <b>906</b>. Such pixels in the modified VL image <b>906</b> therefore have similar effects on corresponding IR pixels due to the blending process, and the relative relationship between IR pixels with respect to the original IR palette is generally maintained.

0117As described above, in some embodiments, the blending ratio for combining the modified VL image <b>906</b> and the IR image <b>908</b> at each pixel is a function of the edge factor value associated with the corresponding VL pixel. Thus, in some such embodiments, pixels in the blended image <b>910</b> not corresponding to strong edges in the VL image <b>900</b> may largely or entirely include the IR image data, thereby entirely or nearly entirely preserving the IR image information at pixels corresponding to no or weak edges in the VL image <b>900</b>. By contrast, pixels corresponding to edges in the VL image <b>900</b> (having a high-magnitude edge factor value) will include the modified VL image data incorporating such edge factor values. Thus, the edges in the VL image corresponding to pixels having edge factor values of a large magnitude will ultimately be incorporated into the blended image <b>910</b> by way of the edge factor value contribution to the modified VL image <b>906</b>.

0118<figref idref="DRAWINGS">FIG. 9B</figref> is another flow diagram showing a series of example images used in creating a blended image. Generally, the process follows as described above with respect to <figref idref="DRAWINGS">FIG. 9A</figref>, in which the VL image <b>900</b> is processed and combined with the edge midscale value <b>904</b> to generate a modified VL image <b>906</b>. The modified VL image <b>906</b> includes edges showing the outline of a person as well as background details (whiteboard and writing). The modified VL image <b>906</b> is combined with the IR image <b>908</b> to create a blended image <b>910</b>. While the thermal scene represented in <figref idref="DRAWINGS">FIG. 9A</figref> is relatively uniform, the thermal scene represented in <figref idref="DRAWINGS">FIG. 9B</figref> includes thermal variations. That is, the IR image <b>908</b> of <figref idref="DRAWINGS">FIG. 9B</figref> includes contents having a wide variety of temperatures (a relatively uniform background with higher-temperature person in the foreground). As shown, the blended image <b>910</b> of <figref idref="DRAWINGS">FIG. 9B</figref> includes both the thermal information from IR image <b>908</b> (the person stands out from the background) as well as visible details from the VL image (the whiteboard and writing details). Thus, the resulting blended image <b>910</b> effectively combines thermal information from the IR image data as well as edges and detail information from the VL image data.

0119As previously described, the VL image processing may include an adjustable (e.g., user adjustable) edge gain parameter. Thus, the resulting modified VL image may include an adjustable amount of offset or other enhancement at pixels corresponding to VL edges. <figref idref="DRAWINGS">FIG. 10</figref> provides a series of modified VL images including adjusted edge gain parameters. As discussed previously, the VL image <b>1000</b> may be processed to generate edge factor values for each pixel which may depend on an edge gain parameter. Modified VL images <b>1006</b><i>a</i>, <b>1006</b><i>b</i>, <b>1006</b><i>c</i>, and <b>1006</b><i>d </i>are exemplary images resulting from VL image processing having different edge gain parameters. For example, image <b>1006</b><i>a </i>is created using a relatively small edge gain parameter, with low but visible contrast present between pixels corresponding to VL edges and those not corresponding to VL edges. By contrast, image <b>1006</b><i>b </i>is a modified VL image generated using an edge gain parameter that is larger than that of image <b>1006</b><i>a</i>, and the resulting image <b>1006</b><i>b </i>includes increased visible contrast at pixels corresponding to VL edges. Similarly, image <b>1006</b><i>c </i>is a modified VL image generated with a still larger edge gain parameter when compared to image <b>1006</b><i>b</i>. Accordingly, pixels corresponding to VL edges are further emphasized in image <b>1006</b><i>c </i>when compared to images <b>1006</b><i>a </i>and <b>1006</b><i>b</i>. Image <b>1006</b><i>d </i>is a modified VL image generated using a still greater edge gain parameter than image <b>1006</b><i>c</i>, therefore having even further offset and/or enhanced pixels corresponding to VL edges.

0120In various embodiments, a user may adjust the edge gain parameter in order to affect the degree of enhancement of pixels corresponding to VL edges, for example, via a user interface. In some embodiments, the user may adjust the edge gain parameter in real time and observe the effect of the adjustment on the display. In some examples, the edge gain parameter may be selectable from a list of predetermined values or percentages. In other examples, the edge gain parameter may be continuously adjusted between minimum and maximum values.

0121<figref idref="DRAWINGS">FIG. 11</figref> is a process-flow diagram illustrating an exemplary process for creating a blended image. One or more cameras may be used to acquire corresponding IR and VL images (<b>1130</b>). The images may be captured by the same or different cameras, and may be subsequently processed and/or analyzed on one or both of the cameras, or transferred to an external processing device. The method can include registering the acquired IR and VL images (<b>1132</b>). As previously discussed, one or both of the IR or VL images may be scaled (upsampled, downsampled, etc.) to achieve a one-to-one pixel mapping between the images. Additionally or alternatively, in some examples, a plurality of pixels in one of the images may correspond to a single pixel or a different-sized plurality of pixels in the other image. Next, an edge factor value may be determined for each pixel in the VL image (<b>1134</b>). The determined edge factor values from a plurality of pixels can be added to or otherwise combined with an edge midscale value in order to create a modified VL image (<b>1136</b>). Finally, the method can include blending the modified VL image with the IR image in order to generate a final blended image (<b>1138</b>). In some examples, the blending may be performed on a pixel-by-pixel basis, with the blending ratio at each pixel being based on the edge factor value associated with that pixel. As previously described, in some examples, pixel processing is done on an entire-image scale, in which edge factor values are combined with an entire “frame” of the edge midscale value. In alternative embodiments, several processing steps may be performed on a subset of pixels within the processing kernel. Accordingly, in some embodiments, after performing a series of processing steps the step of moving the kernel (<b>1150</b>) may be performed. As shown by the broken lines, in an exemplary process, modified VL image pixels may be blended with corresponding IR pixels (<b>1138</b>) for pixels within the processing kernel. Then the processing kernel may be adjusted (<b>1150</b>) to perform processing steps (e.g., <b>1134</b>-<b>1138</b>) on a new set of pixels. After generating the blended image (<b>1138</b>), the method can include displaying the blended image (<b>1140</b>), for example on display <b>108</b>.

0122In various examples, the display of the blended image may take a variety of forms. For instance, in some examples, the entire displayed image includes a blended image having a combination of IR image data and modified VL image data. In other examples, the blended image may be presented within a larger VL image, for example, in a picture-in-picture display. In general, the blended image may be presented in any way appropriate for presenting IR image data on a display, such as those described, for example, in U.S. patent application Ser. No. 12/828,442, filed Jul. 1, 2010, and entitled “THERMOGRAPHY METHODS,” U.S. Pat. No. 7,994,480, filed Jul. 19, 2006, and entitled “VISIBLE LIGHT AND IR COMBINED IMAGE CAMERA,” and U.S. patent application Ser. No. 13/833,853, filed Mar. 15, 2013, and entitled “THERMAL IMAGE ANIMATION,” each of which is assigned to the assignee of the instant application and is hereby incorporated by reference herein in its entirety.

0123Methods as described herein may be performed in a variety of ways. In some examples, image processing and blending techniques may be carried out, for example, by a processor (e.g., <b>222</b>) in a thermal imaging camera. In some such examples, one or both of the VL and IR images may be captured by the same camera or a different camera. In some embodiments, IR and VL images captured by the same or different cameras may be transferred to an external processing device, such as a computer, tablet, smartphone, or the like, which may perform any number of processing steps (e.g., registration, generating modified VL images, blending, etc.). In some examples, corresponding IR and VL images may be captured and processed to generate a blended image as described herein by a single device. In some embodiments, such image capturing and processing may be performed in substantially real time, for instance, using a live stream of images to generate a video signal comprising a series of blended images. In general, as used herein, unless specifically stated otherwise, the term “image” may refer to a single image frame, such as an image store in and recalled from memory, a still frame from a video feed, a live or prerecorded video feed comprising a series of captured still images, a live video feed in which some portions of the display are dynamically updated at different time than others, or other known image data presentation schemes.

0124In some embodiments, one or both of IR and VL image data comprises a stream of data received by the processor. In some such embodiments, the processor can receive a stream of image data and generate a video file therefrom. For example, in some embodiments, the processor can receive a stream of IR image data and VL image data, and generate therefrom a video file. In various embodiments, the generated video file can comprise IR image data, VL image data, or a combination of blended IR and VL image data. In some embodiments, processing techniques described herein may be performed to the image data stream as it is received by the processor.

0125As used herein, “IR” may refer to wavelengths in any portion of the infrared spectrum, such as LWIR (between approximately 8 microns and 14 microns), MWIR (between approximately 3 microns and 5 microns), SWIR (between approximately 1 micron and approximately 2 microns), or any combination of these ranges or wavelengths therebetween. “VL” images typically refer to wavelengths in the visible spectrum (e.g., between approximately 400 nanometers and approximately 700 nanometers). However, the processes as described herein for use with VL images may be performed using alternative wavelengths, such as NIR (e.g., between approximately 700 nm and 1000 nm) or UV (e.g., between approximately 200 nm and 400 nm). In general, processes for combining IR and VL images described herein may be performed on any set of two or more images.

0126Various aspects of methods described herein may be adjusted by a user. For example, a thermal imaging camera capable of executing such methods may include a user interface (e.g., <b>108</b>, <b>112</b>, <b>114</b>, <b>116</b>) for receiving one or more inputs from a user. In some examples, a user may adjust at least one of, for example, the IR palettization scheme, an edge gain used for determining edge factor values, the edge midscale value, a type of image blending (e.g., constant across image, EFV-dependent, a combination thereof, and the like), the amount of image blending (e.g., blending ratios). In other embodiments, one or more of such parameters may be fixed.

0127In some embodiments, various edge detection techniques may be included in the VL image processing, such as described in U.S. patent application Ser. No. 14/222,153, filed Mar. 21, 2014 and entitled “VISIBLE LIGHT IMAGE WITH EDGE MARKING FOR ENHANCING IR IMAGERY,” which is assigned to the assignee of the instant application and is hereby incorporated by reference in its entirety. In some such embodiments, pixels determined to correspond to edges in the VL image may be used in a variety of ways. For instance, in some examples, blending ratios used for each pixel in creating the blended image may be dependent on whether or not the corresponding VL pixel is determined to be an edge pixel. Additionally or alternatively, edge factor values may be determined for one or more VL pixels based on whether or not the pixel is determined to be an edge pixel.

0128Example thermal image cameras and related techniques have been described. The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a non-transitory computer-readable storage medium containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), a hard disk, optical media, or other computer readable media.

0129For example, an external computer comprising such computer readable medium can receive corresponding visible light and infrared images from a thermal imaging camera or from memory and perform edge detection and/or process the VL and IR images to generate display images as described herein. In some embodiments, various portions of the techniques can be embodied in multiple components. For example, a thermal imaging camera can detect edges in a visible light image and pass detected edge information to an external computing device for generating the display image incorporating the detected edges. Additionally or alternatively, the external computing device may assist in or otherwise perform edge detection and/or enhancement techniques.

0130In further examples, embodiments of the invention can be embodied in a display system. The display system can be configured to receive VL and IR image data and carry out processes such as those herein described. Exemplary display systems can include one or more processors, a display and a user interface for carrying out such processes. A display system can be incorporated into any appropriate device or system capable of receiving and processing image data. In some embodiments, the display system can include a portable, hand-held thermal imaging camera such as those described elsewhere herein in order to capture corresponding VL and IR images and provide VL and IR image data to other components of the imaging system. In further embodiments, the imaging system is fully incorporated into such a camera, or can consist essentially of a camera capable of carrying out any of the various processes described.

0131Various embodiments have been described. Such examples are non-limiting, and do not define or limit the scope of the invention in any way. Rather, these and other examples are within the scope of the following claims.