Evaluating system performance with sparse principal component analysis and a test statistic

BACKGROUND

0001This invention was made with government support under the Department of Homeland Security/Transportation Security Administration, Contract No. HTS04-10-D-ST3006. The government of the United States of America has certain rights in the invention.

0002A determination of system performance for a complex system may involve the determination and evaluation of large number of metrics or system variables. The system variables of an evaluated system may be compared to the system variables of a baseline system using statistical methods. For example, the statistical method may generate a pass or fail criteria. However, standard statistical methods may generate an unreliable pass fail or fail criteria, in particular when evaluating a large number of system variables. For example, the standard statistical methods may generate noisy pass or fail criteria, i.e., the response of the pass or fail criteria to known faults may be unpredictable. Moreover, the standard statistical methods may lead to an unacceptable number of erroneous failures.

0003A conventional method is described in the publication "<nplcit id="ncit0001" npl-type="s"><text>A method for fault detection on synchronous generators using modified principal component analysis" by Doorsamy W. and Cronje W. A., 2015 IEEE International Conference on Industrial Technology (ICIT), 2015-03-17, IEEEasd, XP032785613</text></nplcit>.

0004The present application appreciates that evaluating system performance may be a challenging endeavor.

SUMMARY

0005According to one aspect of the present invention, there is provided a method as defined in claim 1 hereinafter.

0006According to another aspect of the present invention, there is provided a system as defined in claim 10 hereinafter.

0007In one embodiment, a method for evaluating system performance may include collecting test measurements of a calibration standard with a sensor of a system. The test measurements may be transformed, automatically with one or more processors, into a test data set. The test data set may include instances of test system variables. Each of the instances of the test system variables may correspond to the test measurements. A test average of the instances of a variable of the test system variables may be compared to a baseline average of a baseline variable, automatically with the one or more processors. A shift amount may be determined based upon the test average and the baseline average. Each of the instances of the variable of the test system variables may be shifted by the shift amount, i.e., for a subset of the test system variables.. A modified test data set may be generated from the shifted test data set. The modified test data set can be transformed, automatically with the one or more processors, with a sparse principal component analysis (SPCA) into test components. The test components can be compared to baseline components using a Hotelling T² test, automatically with the one or more processors. A test statistic can be generated by the Hotelling T² test. Performance of the system can be quantified based upon the test statistic. Alternatively or additionally, the shifted variables of the modified test data set can have substantially the same mean as the corresponding variables in the baseline data set.

0008In another embodiment, a method for evaluating system performance may include providing a test data set. The test data set may include instances of test system variables. Each of the instances of the test system variables may correspond to test measurement. A variable of the test system variables may be selected, automatically with one or more processors, when the variable of the test system variables is indicative of improved quality, or when a test average of the instances of the variable of the test system variables differs from a baseline average by less than a practically significant difference. A shift amount may be determined based upon the test average and the baseline average. Each of the instances of the variable of the test system variables may be shifted, automatically with the one or more processors, by the shift amount to generate a modified test data set. The modified test data set may be transformed, automatically with the one or more processors, with a sparse principal component analysis into test components. The test components may be compared, automatically with the one or more processors, to baseline components using a Hotelling T² test to generate a test statistic. Performance of the system may be quantified based upon the test statistic.

0009In a further embodiment, a system capable of evaluating system performance may include a sensor and memory communicatively coupled to one or more processors. The memory may include machine readable instructions that are executed by the one or more processors to collect test measurements of a calibration standard with the sensor. The test measurements may be transformed into a test data set. The test data set may include instances of test system variables. Each of the instances of the test system variables may correspond to the test measurements. A test average of the instances of a variable of the test system variables may be compared to a baseline average of a baseline variable. A shift amount may be determined based upon the test average and the baseline average. Each of the instances of the variable of the test system variables may be shifted by the shift amount to generate a modified test data set from the test data set. The modified test data set may be transformed with a sparse principal component analysis into test components. The test components may be compared to baseline components using a Hotelling T² test to generate a test statistic. Performance of the system may be quantified based upon the test statistic.

0010These and additional features provided by the embodiments described herein will be more fully understood in view of the following detailed description, in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

0011The embodiments set forth in the drawings are illustrative and exemplary in nature and not intended to limit the subject matter defined by the claims. The following detailed description of the illustrative embodiments may be understood when read in conjunction with the following drawings, where like structure is indicated with like reference numerals and in which: <ul id="ul0001" list-style="none"><li><figref idref="f0001"><b>FIG. 1</b></figref> is a schematic depicting a system for evaluating system performance according to one or more embodiments shown and described herein;</li><li><figref idref="f0001"><b>FIG. 2</b></figref> is a schematic depicting a method for collecting baseline data according to one or more embodiments shown and described herein;</li><li><figref idref="f0002"><b>FIG. 3</b></figref> is a schematic depicting a baseline data set according to one or more embodiments shown and described herein;</li><li><figref idref="f0002"><b>FIG. 4</b></figref> is a schematic depicting a baseline components according to one or more embodiments shown and described herein;</li><li><figref idref="f0003"><b>FIG. 5</b></figref> is a schematic depicting a method for collecting test data according to one or more embodiments shown and described herein;</li><li><figref idref="f0003"><b>FIG. 6</b></figref> is a schematic depicting a test data set according to one or more embodiments shown and described herein;</li><li><figref idref="f0004"><b>FIG. 7</b></figref> is a schematic depicting test components according to one or more embodiments shown and described herein;</li><li><figref idref="f0004"><b>FIG. 8</b></figref> is a schematic depicting a method for performing a statistical comparison according to one or more embodiments shown and described herein; and</li><li><figref idref="f0005"><b>FIG. 9</b></figref> is a graphic depicting a plurality of data sets compared to an empirically determined threshold for a test statistic according to one or more embodiments shown and described herein.</li></ul>

DETAILED DESCRIPTION

0012The present specification generally relates to systems and methods for evaluating system performance and, more specifically, to systems and methods for evaluating system performance using component analysis and a test statistic.

0013The embodiments described herein generally relate to computerized systems and methods for evaluating the performance of a system such as, but not limited to, an X-ray computed tomography (CT) system or other detection system. For example, the performance of the system may be quantified by a test statistic that is indicative of the performance of an output generated by the system such as, but not limited to, image quality of the detection system. In some embodiments, the test statistic may be evaluated or generated using a plurality of system variables. Specifically, test results may be compared to baseline measurements for each of the system variables. Various embodiments of the system and the method for evaluating system performance will be described in more detail herein.

0014Referring now to <figref idref="f0001"><b>FIG. 1</b></figref><b>,</b> a system <b>100</b> may be configured to collect data indicative of a test article. The system <b>100</b> may include one or more processors <b>102</b> for executing machine readable instructions and memory <b>104</b> for storing the machine readable instructions. The one or more processors <b>102</b> may be communicatively coupled to the memory <b>104.</b> The one or more processors <b>102</b> may include an integrated circuit, a microchip, a computer, or any other computing device capable of executing machine readable instructions. The memory <b>104</b> may include RAM, ROM, a flash memory, a hard drive, or any device capable of storing machine readable instructions. As used herein, the phrase "communicatively coupled" may mean that components are capable of exchanging data signals with one another such as, for example, electrical signals via conductive medium, electromagnetic signals via air, optical signals via optical waveguides, and the like.

0015Thus, embodiments of the present disclosure may include logic or an algorithm written in any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as, e.g., machine language that may be directly executed by the processor, or assembly language, object-oriented programming (OOP), scripting languages, microcode, etc., that may be compiled or assembled into machine readable instructions and stored on a machine readable medium. Alternatively, the logic or algorithm may be written in a hardware description language (HDL), such as implemented via either a field-programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC), and their equivalents.

0016The system <b>100</b> may include a sensor <b>106</b> for collecting measurements of a test article. The sensor <b>106</b> may be communicatively coupled to the one or more processors <b>102,</b> the memory <b>104,</b> or both. It is noted that the term "sensor," as used herein, may mean a device that measures a physical quantity and converts it into a signal, which is correlated to the measured value of the physical quantity. In some embodiments, the system <b>100</b> may be configured as an X-ray CT system such as, but not limited to, an X-ray CT Explosives Detection System (EDS). Accordingly, the sensor <b>106</b> may be an X-ray detector that is configured to detect photons such as, for example, a point detector, a linear detector, or a planar detector.

0017In some embodiments, the system <b>100</b> may include a source <b>108</b> that is configured to generate excitation energy that is detectable by the sensor <b>106.</b> The sensor <b>108</b> may be communicatively coupled to the one or more processors <b>102,</b> the memory <b>104,</b> or both. In embodiments where the system <b>100</b> is configured as an X-ray CT system, the source <b>106</b> may be an X-ray source configured to emit photons along a path. Specifically, the path may begin at the source <b>108</b> and terminate at the sensor <b>106.</b> Generally, the test article is placed along the path and between the source <b>108</b> and the sensor <b>106</b> such that a portion of the photons are absorbed by the test article while measurements are collected by the system <b>100.</b>

0018Referring still to <figref idref="f0001"><b>FIG. 1</b></figref><b>,</b> the system <b>100</b> may include an actuation assembly <b>110</b> configured to manipulate the test article, the sensor <b>106,</b> the source <b>108,</b> or a combination thereof. Accordingly, the actuation assembly <b>110</b> may include one or more servo-mechanism for providing a controlled amount of force for manipulating the test article, the sensor <b>106,</b> the source <b>108,</b> or a combination thereof. In the embodiments described herein, the one or more processors 102, the memory <b>104,</b> or both may be integral with any or all of the sensor <b>106,</b> the source <b>108,</b> and the actuation assembly <b>110.</b> However, it is noted that the one or more processors <b>102,</b> the memory <b>104,</b> or both may be separate components communicatively coupled with one another without departing from the scope of the present disclosure.

0019In some embodiment, the actuation assembly <b>110</b> may include a mechanical actuator, a hydraulic actuator, a pneumatic actuator, an electrical actuator, or combinations thereof. The actuation assembly <b>110</b> may be communicatively coupled to the one or more processors <b>102,</b> the memory <b>104,</b> or both. In some embodiments, the one or more processors <b>102</b> may execute machine readable instructions to direct the operation of the sensor <b>106,</b> the source <b>108,</b> and the actuation assembly <b>110.</b> For example, actuation assembly <b>110</b> may include a conveyer system for moving test articles throughout the system <b>100.</b> Alternatively or additionally, the actuation assembly may be configured to cause relative motion of the test article with respect to the sensor <b>106.</b>

0020In embodiments where the system <b>100</b> is configured as an X-ray CT system, multiple measurements of the test article may be collected by the sensor <b>106</b> while the test article moves with respect to the sensor <b>106,</b> the source <b>108,</b> or both. Each measurement may be constructed into an image having greater dimensional complexity than the measurement generated by the sensor <b>106.</b> For example, each measurements may be indicative of absorption or density of the test article that may be constructed into an image indicative of both the internal and external features of the test article. Specifically, measurements collected by a line detector may be used to produce a two-dimensional image showing a slice of the test article depicting both internal and external features. A plurality of slices may be combined to produce a full representation of the internal and external features of a three-dimensional object such as, for example, by combining slices collected along a direction orthogonal to the plane of the slices. Measurements collected by a planar detector may be constructed into a three-dimensional image of the test article. It is to be understood that, while particular variations and principles may be discussed herein with regard to X-ray CT techniques, any suitable sensing technique may be used with the present disclosure. Indeed, the embodiments described herein may be applied to evaluate system performance of any system where data preprocessing may produce a standardized table of system variables or metrics. It should further be understood that, unless otherwise stated, reference to imaging or to an imaging machine includes optical imaging devices, Magnetic Resonance Imaging (MRI), X-ray CT, and any other applicable scanning or imaging technique or machine.

0021As is explained in greater detail herein, system performance may be quantified by comparing tests measurements to baseline measurements. In some embodiments, measurements may be collected using a calibration standard <b>112</b> as the test article. It is note that, while the embodiments described herein may use the calibration standard <b>112,</b> multiple test articles that are substantially the same as the calibration standard <b>112</b> may be used to generate tests measurements and/or baseline measurements. The calibration standard <b>112</b> may be an object with standardized or predefined features that are detectable by the sensor <b>106.</b> System variables may be derived from the measurements of the calibration standard <b>112.</b> Specifically, in the case of X-ray CT EDS, two calibration standards ("test article A" and "test article B") are defined by IEEE Standards Association, American National Standard for Evaluating the Image Quality of X-ray Computed Tomography (CT) Security-Screening Systems (2011), hereinafter the "ANSI N42.45-2011 standard." The ANSI N42.45-2011 standard, which is incorporated herein by reference, further defines seventy eight (78) individual image quality metrics of X-ray images. Each image quality metric may be used as a system variable, according to the embodiments described herein.

0022Referring collectively to <figref idref="f0001"><b>FIGS. 1</b> and <b>2</b></figref><b>,</b> a method <b>120</b> may be configured to establish baseline data. The method <b>120</b> may include a process <b>122</b> for collecting baseline measurements <b>124.</b> For example, the sensor <b>106</b> of the system <b>100</b> may collect one or more measurements of the calibration standard <b>112.</b> Alternatively or additionally, the baseline measurements 124 may include system parameters such as, for example, voltage, current, belt speed, temperature, humidity, or any other metric of the system detected by one or more additional sensors. In some embodiments, multiple additional baseline systems may be used to collect measurements of the calibration standard <b>112.</b> Generally, the baseline measurements <b>124</b> may correspond to baseline systems in good working order or baseline systems with known levels of performance. Accordingly, the baseline measurements <b>124</b> may correspond to a predefined level of system performance.

0023Referring collectively to <figref idref="f0001"><b>FIGS.</b> 2</figref> and <figref idref="f0002">3</figref>, the method <b>120</b> may include a process <b>126</b> for processing or transforming the baseline measurements <b>124</b> into a baseline data set <b>128.</b> In some embodiments, the baseline data set <b>128</b> may include multiple instances <b>130</b> of baseline system variables <b>132.</b> Each instance <b>130</b> may correspond to the baseline measurements <b>124.</b> For example, in some embodiments, an instance <b>130</b> may correspond to a single measurement instance, i.e., a single image or single measurement of the calibration article. Alternatively or additionally, an instance <b>130</b> may correspond to a combination of measurement instances, i.e., an average of images or an average of measurements of the calibration standard <b>112.</b> It is noted that the baseline data set <b>128</b> is provided in matrix notation in <figref idref="f0002"><b>FIG. 3</b></figref><b>,</b> where each row corresponds to one of the baseline system variables <b>132</b> and each column corresponds to an instance <b>130.</b>

0024At process <b>126,</b> the features of the calibration standard <b>112</b> may be detected from the baseline measurements <b>124.</b> The detected features may be used to derive the baseline system variables <b>132</b> of the baseline data set <b>128.</b> For example, each feature may be quantified, the relative positioning of the features may be quantified, regions of interest may be quantified, statistical metrics (e.g., mean, standard deviation, maximum, minimum, median, and the like) may be derived using one or more features and/or regions of interest, and the like. For example, in embodiments where the baseline measurements <b>124</b> include X-ray data, the baseline system variables <b>132</b> may include a variable derived from an X-ray image. Specifically, the ANSI N42.45-2011 standard defines multiple image quality metrics that may be used as one or more of the baseline system variables <b>132.</b>

0025Referring collectively to <figref idref="f0001"><b>FIGS. 2</b></figref><b>,</b><figref idref="f0002"><b>3,</b> and <b>4</b></figref><b>,</b> at process <b>134,</b> the baseline data set <b>128</b> may be transformed into baseline components <b>136</b> having reduced dimensionality than the baseline data set <b>128.</b> For example, the baseline data set <b>128</b> may be input into Principal Component Analysis (PCA) or a Sparse Principal Component Analysis (SPCA) to generate the baseline components <b>136.</b> The PCA may transform the baseline data set <b>128</b> into the baseline components <b>136</b> having principal components <b>138,</b> which are a linear combination of each of the baseline system variables <b>132.</b> The linear combination may be represented by baseline loading <b>140.</b> Specifically, when represented in matrix notation, each column of the baseline components <b>136</b> may correspond to one of the principal components <b>138,</b> and each row of the baseline components <b>136</b> may correspond to the baseline loading <b>140</b> of a corresponding baseline system variable <b>132.</b> The dimensionality may be reduced, when the total number of principal components <b>138</b> is less than the number of the baseline system variables <b>132.</b> Generally, the principal components <b>138</b> generated by the PCA are not correlated with one another. One issue with PCA is that each of the principal components <b>138</b> is a linear combination of all of the system variables, i.e., non-zero loading. Accordingly, it may be difficult to determine which of the system variables is responsible for an observed response. Without being bound to theory, it is believed that more meaningful components may be identified when all but the important loadings are zero.

0026SPCA refines PCA by using regularization methodology, which is designed to make most of the baseline loadings <b>140</b> of the baseline components <b>136</b> have a value of zero. IN one embodiment, the regularization methodology may include imposing lasso (i.e., an elastic net) constraint on the regression coefficients. The baseline system variables <b>132</b> contributing to each of the baseline principal components <b>138</b> are sparse (i.e., fewer variables than the PCA). The baseline loadings <b>140</b> having both zero and non-zero weights may enable an better interpretation of the meaning of key components than the PCA approach. The baseline principal components <b>138</b> of the SPCA are not strictly un-correlated, but the degree of correlation is low.

0027Referring collectively to <figref idref="f0001"><b>FIGS. 1, 2</b></figref><b>,</b> and <figref idref="f0003"><b>5</b></figref><b>,</b> according to the embodiments described herein, a method <b>150</b> may be configured to characterize a test performed by the system <b>100.</b> The method <b>150</b> may include a process <b>152</b> for collecting test measurements <b>154.</b> The process <b>152</b> may operate as described above with respect to process <b>122.</b> In some embodiments, the process <b>152</b> may be configured to collect measurements of the calibration standard <b>112</b> such that the test measurements <b>154</b> correspond to substantially the same data as the baseline measurements <b>124.</b> Thus, the differences between the test measurements <b>154</b> and the baseline measurements <b>124</b> may be attributed to the differences in the systems or data collection processes. Alternatively or additionally, the test measurements <b>154</b> may be collected using the sensor <b>106</b> of a system with unknown performance. Accordingly, as explained in greater detail herein, the baseline components <b>136</b> may be used to characterize the unknown system performance of the test system.

0028Referring collectively to <figref idref="f0001"><b>FIGS. 1, 2</b></figref><b>,</b><figref idref="f0002"><b>3</b></figref><b>,</b><figref idref="f0003"><b>5,</b> and 6</figref>, the method <b>150</b> may include a process <b>156</b> for processing or transforming the test measurements <b>154</b> into a test data set <b>158,</b> which includes multiple instances <b>160</b> of test system variables <b>162.</b> In some embodiments, an instance <b>160</b> may correspond to a single measurement instance, i.e., a single image or single measurement of the calibration article <b>112.</b> The process <b>156</b> may operate as described above with respect to process <b>126,</b> such that test system variables <b>162</b> correspond to the same type of variable as one of the baseline system variables <b>132.</b> Accordingly, since the calibration article <b>112</b> is used, the test system variables <b>162</b> may provide a direct comparison with the baseline system variables <b>132</b> that may be indicative of the performance of the test system compared to the baseline systems.

0029The method <b>150</b> may further include a process <b>164</b> for shifting data and transforming the test data set <b>158</b> into a modified test data set <b>170.</b> As is explained in greater detail herein, the shift may be used constrain the test statistic to practically meaningful differences that correspond to a degraded performance. At process <b>164,</b> one or more of the test system variables <b>162</b> may be identified for shifting. Specifically, the test data set <b>158</b> may include an identified test variable <b>166</b> of the test system variables <b>162</b> that corresponds to the same type of measurement as a corresponding baseline variable <b>142</b> of the baseline system variables <b>132.</b> A test average of the instances <b>160</b> of the identified test variable <b>166</b> may be compared to a baseline average of the instances <b>130</b> of the corresponding baseline variable <b>142</b> to determine a shift amount. In some embodiments, the shift amount may correspond to the delta between the test average and the baseline average. At process <b>164,</b> each of the instances <b>160</b> of the identified test variable <b>166</b> of the test system variables <b>162</b> may be shifted by the shift amount. For example, the shift amount may be applied to each instance <b>160</b> such that the test average and the baseline average are substantially the same. Accordingly, the modified test data set <b>170</b> may be generated from the test data set <b>158</b> in a manner that preserves the variation of the test data set <b>158</b> in the modified test data set <b>170,</b> while substantially eliminating the variation in the means. Without being bound to theory, it is believed that the preservation of the variation and elimination of the mean may improve the effectiveness of a statistical comparison between the baseline and the tests.

0030Referring collectively to <figref idref="f0001"><b>FIGS. 2</b></figref><b>,</b><figref idref="f0003"><b>5</b></figref><b>,</b> and <figref idref="f0004"><b>7</b></figref><b>,</b> at process <b>172,</b> the modified test data set <b>170</b> may be transformed into test components <b>174</b> having reduced dimensionality than the modified test data set <b>170.</b> The process <b>172</b> may employ a PCA or an SPCA to generate the test components <b>174.</b> Generally, the same transformation is used in both process <b>172</b> and process <b>134,</b> in order to facilitate comparison between the baseline components <b>136</b> and the test components <b>174.</b> As is noted above, the use of the SPCA for the transformation may identify less significant system variables through zero loading and more significant system variables through non-zero loading. Accordingly, the use of the SPCA may improve the functioning of the methods providing herein when evaluating complex systems that are described by a large number of system variables.

0031Referring now to <figref idref="f0004"><b>FIG. 8</b></figref><b>,</b> the baseline components <b>136</b> and the test components <b>174</b> may be compared to using a statistical comparison to evaluate the system performance of the test system. According to the embodiments described herein, a method <b>180</b> may be executed automatically to perform a statistical comparison. The method <b>180</b> may include a process <b>182</b> for performing a Hotelling T² test. In some embodiments, the baseline components <b>136</b> and the test components <b>174</b> may be input to the Hotelling T² test in SPCA space such that the Hotelling T² test performs a multivariate statistical test to evaluate if the test system performance is consistent with the baseline performance.

0032The Hotelling T² test may generate a test statistic 184, which is given by Equation (1). <maths id="math0001" num="Equation (1)"><math display="block"><msup><mi>T</mi><mn>2</mn></msup><mo>=</mo><msup><mfenced><msub><mover accent="true"><mi>μ</mi><mo>¯</mo></mover><mi mathvariant="italic">Test</mi></msub><mo>−</mo><msub><mover accent="true"><mi>μ</mi><mo>¯</mo></mover><mi mathvariant="italic">Baseline</mi></msub></mfenced><mi>T</mi></msup><msup><mi mathvariant="normal">Σ</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mfenced><msub><mover accent="true"><mi>μ</mi><mo>¯</mo></mover><mi mathvariant="italic">Test</mi></msub><mo>−</mo><msub><mover accent="true"><mi>μ</mi><mo>¯</mo></mover><mi mathvariant="italic">Baseline</mi></msub></mfenced></math><img file="EP3365635B1_D0001.tif" /></maths>

0033In Equation (1) the test statistic 184 is given by <i>T</i>², where <i><o ostyle="single">µ</o>_Test</i> is the mean vector for the test components 174, <i><o ostyle="single">µ</o>_Baseline</i> is the mean vector for the baseline components 136, and Σ is the estimated covariance matrix. <maths id="math0002" num="Equation (2)"><math display="block"><mi mathvariant="normal">Σ</mi><mo>=</mo><mfenced><mfrac><mn>1</mn><msub><mi>n</mi><mi mathvariant="italic">Baseline</mi></msub></mfrac><mo>+</mo><mfrac><mn>1</mn><msub><mi>n</mi><mi mathvariant="italic">Test</mi></msub></mfrac></mfenced><msub><mi mathvariant="normal">Σ</mi><mi mathvariant="italic">Statistical</mi></msub><mo>+</mo><mfenced><mn>1</mn><mo>+</mo><mfrac><mn>1</mn><msub><mi>n</mi><mi>g</mi></msub></mfrac></mfenced><msub><mi mathvariant="normal">Σ</mi><mi mathvariant="italic">Systematic</mi></msub></math><img file="EP3365635B1_D0002.tif" /></maths>

0034The systematic variance may be included in Σ^-1 by defining the estimated covariance matrix according to Equation (2), where the statistical covariance matrix Σ<i>_Statistical</i> is given by Equation (3), the systematic covariance matrix Σ<i>_Systematic</i> is given by Equations (4) and (5), <i>n_Baseline</i> is the number of baseline systems, and <i>n_Test</i> is the number of test systems. <maths id="math0003" num="Equation (3)"><math display="block"><msub><mi mathvariant="normal">Σ</mi><mi mathvariant="italic">Statistical</mi></msub><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mi>n</mi><mi mathvariant="italic">Baseline</mi></msub><mo>+</mo><msub><mi>n</mi><mi mathvariant="italic">Test</mi></msub><mo>−</mo><mn>2</mn></mrow></mfrac><mfenced><mfenced><msub><mi>n</mi><mi mathvariant="italic">Baseline</mi></msub><mo>−</mo><mn>1</mn></mfenced><msub><mi mathvariant="normal">Σ</mi><mi mathvariant="italic">Baseline</mi></msub><mo>+</mo><mfenced><msub><mi>n</mi><mi mathvariant="italic">Test</mi></msub><mo>−</mo><mn>1</mn></mfenced><msub><mi mathvariant="normal">Σ</mi><mi mathvariant="italic">Test</mi></msub></mfenced></math><img file="EP3365635B1_D0003.tif" /></maths>

0035In Equation (3), the baseline covariance matrix Σ<i>_Baseline</i> may be calculated from the baseline components <b>136</b> and the test covariance matrix Σ<i>_Test</i> may be calculated from the test components <b>174.</b> For example, the components may be defined as the eigenvalues of the associated covariance matrix. <maths id="math0004" num="Equation (4)"><math display="block"><msub><mi mathvariant="normal">Σ</mi><mi mathvariant="italic">Systematic</mi></msub><mfenced><mi>k</mi><mi>k</mi></mfenced><mo>=</mo><mfrac><mstyle displaystyle="true"><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><msub><mi>n</mi><mi>g</mi></msub></msubsup><msup><mfenced><mover accent="true"><mrow><msub><mi>μ</mi><mi>i</mi></msub><mfenced><mi>k</mi></mfenced></mrow><mo>¯</mo></mover><mo>−</mo><mover accent="true"><mrow><msub><mi>μ</mi><mi>p</mi></msub><mfenced><mi>k</mi></mfenced></mrow><mo>¯</mo></mover></mfenced><mn>2</mn></msup></mstyle><mrow><msub><mi>n</mi><mi>g</mi></msub><mo>−</mo><mn>1</mn></mrow></mfrac></math><img file="EP3365635B1_D0004.tif" /></maths><maths id="math0005" num="Equation (5)"><math display="block"><msub><mi mathvariant="normal">Σ</mi><mi mathvariant="italic">Systematic</mi></msub><mfenced><mi>k</mi><mi>l</mi></mfenced><mo>=</mo><msub><mn>0</mn><mfenced><mi>k</mi><mo>≠</mo><mi>l</mi></mfenced></msub></math><img file="EP3365635B1_D0005.tif" /></maths>

0036The systematic covariance matrix Σ<i>_Systematic</i> may be assumed to be a diagonal matrix given by Equations (4) and (5), whose elements are the between group variance observed in the baseline dataset and where <i>n_g</i> is the total number of systems, and <i>µ_p</i> is the average of all of the systems. The addition of the systematic covariance matrix Σ<i>_Systematic</i> may effectively lessen the weight of observables which vary significantly between daily operations.

0037Referring collectively to <figref idref="f0002"><b>FIGS. 3</b></figref><b>,</b><figref idref="f0003"><b>6</b></figref><b>,</b> and <figref idref="f0004"><b>8</b></figref><b>,</b> the method may include a process <b>186</b> for quantifying system performance. Specifically, the test statistic <b>184</b> may be compared to a statistical distribution <b>188.</b> For example, in cases where the baseline data set <b>128</b> and the test data set <b>158</b> are normally distributed, i.e., follow a normal distribution, the test statistic <b>184</b> may be characterized by an F-distribution. The F-distribution may be given by <i>T</i>²<i>≈ F</i>(<i>p,n</i>), where <i>p</i> is the number of SPCA parameters and <i>n</i> is the number of degrees of freedom <i>n</i> given by <i>n_Baseline</i> +<i>n_Test + p</i> -1. Accordingly, statistical cutoff values may be placed on the test statistic <b>184</b> to characterize the performance of the test system. For example, as the value of the test statistic <b>184</b> increases, the probability that the test system is operating like a baseline system, i.e., like a good performing system, becomes less likely.

0038In some embodiments, the test statistic <b>184</b> may be compared to the statistical distribution <b>188</b> to generate a p-value. The p-value may be compared to one or more threshold values to characterize system performance. In one embodiment, three categories may be defined according to threshold values of about 0.01 and about 0.05. Specifically, a green category may correspond to a properly functioning test system, a yellow category may correspond to a test system that may not be operating properly, and a red category may correspond to a test system that is not operating properly. A test system generating a test statistic <b>184</b> having a p-value greater than about 0.05 may be classified as green. A test system generating a test statistic <b>184</b> having a p-value between about 0.01 and about 0.05 may be classified as yellow. Assuming that all terms are normally distributed and the statistical and systematic sources have been fully accounted for, roughly one in twenty of runs under normal operating conditions should produce a yellow result. A test system generating a test statistic <b>184</b> having a p-value less than about 0.01 may be classified as red. If all sources of variance have been correctly assigned, only one out of <b>1000</b> runs should produce such a result under normal operating conditions.

0039As is described above, the Hotelling T² test assumes that the distributions of the system variables are normally distributed and uncorrelated. In some embodiments, the system variables may be correlated and not follow a normal distribution such as, for example, metrics defined by the ANSI N42.45-2011 standard may measure similar properties and thus, be highly correlated. In practice, it was observed that the system variables demonstrated deviation from normality. It was discovered that the deviation may be attributed to skewed distributions, multimodal distributions, or distributions with a high number of outliers (i.e., observations in the tails of the distributions). To adjust for the deviation, the statistical distribution <b>188</b> may be determined empirically in the alternative of using a standard distribution. That is, the test statistic <b>184</b> may be compared to an empirical test distribution to evaluate the quality of the test system.

0040Referring collectively to <figref idref="f0001"><b>FIGS. 2</b></figref><b>,</b><figref idref="f0003"><b>5</b></figref><b>,</b> and <figref idref="f0004"><b>8</b></figref><b>,</b> according to the embodiments described herein, the empirical distribution may be derived using the baseline measurements <b>124.</b> For example, a subset of the baseline measurements <b>124</b> may be compared to the remainder of the baseline measurements <b>124.</b> Specifically, the subset of the baseline measurements <b>124</b> may be used, instead of the test measurements <b>154,</b> to generate the test components <b>174,</b> as described in method <b>150,</b> and the remainder of the baseline measurements <b>124</b> may be used to generate the baseline components <b>136,</b> as described in method <b>120.</b> A test statistic <b>184</b> based only upon the baseline measurements <b>124</b> may be generated. The process may be repeated to generate sufficient test statistics <b>184</b> from only upon baseline measurements <b>124</b> to map to an empirical dataset indicative of good performance. In some embodiments, the empirical distribution may be modified by removing large system outliers (e.g., more than ten standard deviations from nominal).

0041Alternatively or additionally, the thresholds based on the F-distribution may be replaced with new thresholds based on percentiles of the empirical distribution.. For example, the empirical distribution may be divided into groups of percentages (i.e., a histogram) that are separated by quantiles. In one embodiment, the values associated with the desired quantiles of the empirical distribution may be used as the thresholds. Generally, the empirical distribution based thresholds are less stringent than the F-distribution. However, it has been discovered that the empirical distribution based thresholds better reflect the observed distribution of the data than the F-distribution for cases where the normality assumptions behind the F-distribution are not valid.

0042Referring collectively to <figref idref="f0002"><b>FIGS. 3</b></figref><b>,</b><figref idref="f0003"><b>5</b></figref><b>,</b><figref idref="f0003"><b>6</b></figref><b>,</b> and <figref idref="f0004"><b>8</b></figref><b>,</b> the test data set <b>158</b> may be transformed into the modified test data set <b>170</b> according to process <b>164.</b> The process <b>164</b> may include selecting the identified test variable <b>166.</b> In some embodiments, the identified test variable <b>166</b> may be automatically selected, when the identified test variable <b>166</b> corresponds to a system variable indicative of improved quality. For example, when evaluating X-ray images some differences in mean metrics are indicative of improved image quality. In the example of a smaller standard deviation than the baseline, the smaller standard deviation may be indicative of less noise, i.e., better image quality. With reference to Equation (1), the Hotelling T² test does not differentiate between desired and undesired changes. Specifically, the difference in mean is squared, i.e. the inverse covariance matrix Σ^-1 is multiplied on both the left and the right side of Equation (1) by the difference in mean. By definition, the mean difference squared is non-negative. The difference in mean always adds a positive (or zero) quantity to the test statistic <b>184.</b> Accordingly, even differences indicative of improved quality penalize the test statistic <b>184</b> and make the test statistic <b>184</b> more likely to fail at process <b>186.</b> In practice, differences indicative of improved quality may cause systems with less noise, i.e., improved quality, to be rejected due to the inability of the Hotelling T² test to distinguish between differences indicative of improved performance and differences indicative of degraded performance.

0043In some embodiments, improved quality candidates and an improved quality metric (e.g., an absolute value or a delta) may be identified prior to the execution of process <b>164.</b> Process <b>164</b> may be executed automatically, to compare the test system variables <b>162</b> to the improved quality candidates. Likewise, the mean of the test system variables <b>162</b> or the delta of the mean of the test system variables <b>162</b> from the mean of the baseline system variables <b>132</b> may be compared to the improved quality metric. Accordingly, the identified test variable <b>166</b> may be automatically selected, when the identified test variable <b>166</b> corresponds to a system variable indicative of improved quality. By selecting the identified test variable <b>166</b> corresponding to a system variable indicative of improved quality and shifting each instance <b>160</b> such that the mean of the identified test variable <b>166</b> is substantially equal to the mean of the corresponding baseline variable <b>142,</b> the embodiments described herein may remove the inherent penalty the Hotelling T² test. Moreover, by shifting the data rather than simply replacing the mean, the penalty of the average may be mitigated while the impact of the variance may be maintained.

0044Referring still to <figref idref="f0002"><b>FIGS. 3</b></figref><b>,</b><figref idref="f0003"><b>5</b></figref><b>,</b><figref idref="f0003"><b>6</b></figref><b>,</b> and <figref idref="f0004"><b>8</b></figref><b>,</b> the identified test variable <b>166</b> may be automatically selected, when the mean of the identified test variable 166 differs from the mean of the corresponding baseline variable <b>142</b> by a practically insignificant difference. An issue with the t-test and by extension the Hotelling T² test is that a magnitude of a minimal detectable difference of the Hotelling T² test is dependent on the sample size, i.e., as the sample size increases the magnitude of the minimal detectable difference decreases. Accordingly, if the means of two distributions are not identically the same, the test will find a difference in means larger than the magnitude of the minimal detectable difference to be statistically significantly different if the sample size is sufficiently large. However, the magnitude of the minimal detectable difference may be less than a practically significant difference, i.e., the Hotelling T² test may be overly sensitive to changes in the data. Specifically, the Hotelling T² test may be considered as combining multiple tests into a single test. The combination of several statistically significant, but practically insignificant mean differences may result in a test statistic <b>184</b> that is more likely to fail.

0045The practically insignificant difference may be identified by determining the sensitivity of the system performance to Hotelling T² test. In some embodiments, the practically significant difference may be determined empirically. Specifically, ranges of test system variables <b>166</b> may be used to generate test components <b>174,</b> i.e., empirical data sets may be input directly to process <b>174.</b> The resulting test statistics <b>184</b> may be observed to determine a range of values that provide similar resulting test statistics <b>184.</b> For example, the practical significant difference may be defined as a pre-defined percentage of difference from the mean of the corresponding baseline variable <b>142.</b> Thus, the identified test variable <b>166</b> may be selected, when the mean of the identified test variable <b>166</b> is within the pre-defined percentage from the mean of the corresponding baseline variable <b>142.</b> In further embodiments, practically significant difference may be based on other performance measures as appropriate and available. By selecting the identified test variable <b>166</b> having a mean that differs from the mean of the corresponding baseline variable <b>142</b> by a practically insignificant difference and shifting each instance <b>160</b> such that the mean of the identified test variable <b>166</b> is substantially equal to the mean of the corresponding baseline variable <b>142,</b> the embodiments described herein may make the Hotelling T² test less sensitive to insignificant changes in data. Moreover, by shifting the data rather than simply replacing the mean, the sensitivity may be improved while the impact of the variance may be maintained.

0046Referring collectively to <figref idref="f0001"><b>FIGS. 1, 2</b></figref><b>,</b><figref idref="f0003"><b>5</b></figref><b>,</b> and <figref idref="f0004"><b>8</b></figref><b>,</b> it is noted that the method <b>120,</b> the method <b>150,</b> the method <b>180,</b> or a combination thereof may be automatically executed by the one or more processors <b>102</b> of the system <b>100.</b> Without departing from the scope of this disclosure, each of the method <b>120,</b> the method <b>150,</b> the method <b>180</b>, or any process thereof may be performed on separate appliances or systems. In one embodiment, the method <b>120</b> may be executed automatically by one or more processors <b>102</b> of one or more baseline systems including the system <b>100,</b> and the method <b>150</b> and the method <b>180</b> may be executed automatically by one or more processors <b>102</b> by a test system including the system <b>100.</b> Accordingly, the baseline measurements <b>124,</b> the baseline data set <b>128,</b> the baseline components <b>136,</b> or any combination thereof may be provided upon memory <b>104</b> of the test system. Alternatively or additionally, the test system may be communicatively coupled to the one or more baseline systems, a server, or any other device capable of providing the baseline measurements <b>124,</b> the baseline data set <b>128,</b> the baseline components <b>136</b> to the test system.

0047Referring collectively to <figref idref="f0004"><b>FIGS. 8</b></figref> and <figref idref="f0005"><b>9</b></figref><b>,</b> the performance of the embodiments described herein were validated using CT data. A plurality of data sets <b>190,</b> which were collected using a nominal X-ray CT EDS, are graphically depicted in <figref idref="f0005"><b>FIG. 9</b></figref><b>.</b> The abscissa represents the magnitude of a system variable (e.g., belt speed setting) and the ordinate represents the magnitude of the test statistic <b>188.</b> Each data set <b>190</b> was collected after incrementally modifying a system variable, the belt speed setting of the X-ray CT EDS, with respect to a nominal value. The nominal data set <b>192</b> was collected using the X-ray CT EDS with the nominal belt speed. Line <b>194</b> depicts the empirically determined threshold for the test statistic <b>188.</b> Line <b>196</b> depicts the lower bounds for nominal belt speed, and line <b>198</b> depicts the upper bounds for nominal belt speed. As depicted in <figref idref="f0005"><b>FIG. 9</b></figref><b>,</b> the nominal data set <b>192</b> was within the empirically determined threshold for the test statistic <b>188,</b> i.e., below line <b>194.</b> Each of the data sets <b>190</b> below the lower bounds (left side of line <b>196)</b> and above the upper bounds (right side of line <b>198)</b> are outside of the empirically determined threshold for the test statistic <b>188,</b> i.e., above line <b>194.</b>

0048It should now be understood that the embodiments described herein may evaluate system performance using SPCA and a modified version of the Hotelling T² test. The modifications to Hotelling T² test may include the use of empirically derived distributions for determining thresholds and shifting the data to constrain the Hotelling T² test to practically meaningful differences that degrade system performance. The embodiments described herein were validated by injecting a series of faults in voltage, current, detectors, gantry, belt speed, and voltage/current combinations into X-ray CT EDS. An unmodified or conventional Hotelling T² test produced results that were generally noisy (i.e., the statistic would both increase and decrease as fault intensity increased) and included a number of false positives (i.e., failures with no faults introduced). The embodiments described herein with the modified version of the Hotelling T² test produced a more useful and stable test statistic. Instead of producing noisy results, the test statistic exhibited little noise. Indeed, the test statistic responded rapidly with dramatic increases in response to relatively large faults. Instead of generating a large number of false positives, the test statistic generally passes on nominal inputs and small introduced faults.

0049It is noted that the terms "substantially" and "about" may be used herein to represent the inherent degree of uncertainty that may be attributed to any quantitative comparison, value, measurement, or other representation. These terms are also used herein to represent the degree by which a quantitative representation may vary from a stated reference without resulting in a change in the basic function of the subject matter at issue.

0050While particular embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter have been described herein, such aspects need not be used in combination. It is therefore intended that the appended claims cover all such changes and modifications that are within the scope of the claimed subject matter.