US10169731B2

Selecting key performance indicators for anomaly detection analytics

Summary by NHIP

KPI Selection for Anomaly Detection

The system determines anomaly detectors linked to key performance indicators from historical CPU utilization data. It creates feature profiles for multiple metric groups, ranks them by correlation to the first profile, and compares the highest-ranked profile against future metrics to identify likely anomalies.

Claim Score by NHIP

Read claim 13, the broadest

Abstract

A computer program product is provided and includes a storage medium having program instructions. The program instructions are readable and executable by a processing circuit to cause the processing circuit to determine from historical data which anomaly detectors are associated with key performance indicators (KPIs), to extract descriptors of the first metric group from the KPIs to create a first feature profile thereof, to repeat the determining and the extracting with respect to historical data of second and third metric groups to create second and third feature profiles thereof, respectively, to ascertain which of the second and third feature profiles has a greater correlation to the first feature profile and to rank the second and third feature profiles based on which one has the greater correlation and compare the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce anomalies.

US10169731B2, drawing sheet 1 — Sheet 1 of 6

Term

Projected expiry 6 March 2037.

17 claims: 3 independent, 14 dependent

1
A computer program product for selecting key performance indicators, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being readable and executable by a processing circuit to cause the processing circuit to:determine from historical data which anomaly detectors are associated with key performance indicators (KPIs) for identifying an anomalous event in a first metric group related to central processing unit (CPU) utilization in a first computing system, wherein the anomaly detectors comprise two or more of: a Gaussian detector, which generates an alarm when the historical data exhibits a predefined deviation;a flat line detector, which generates an alarm when the historical data remains at a same value for a predefined time;an increased variance detector, which generates an alarm when the historical data exhibits an increased variance from a baseline value;a de-correlation detector, which generates an alarm when the historical data exhibits a de-correlation from other data;a top out detector, which generates an alarm when the historical data achieves a maximum value from which the historical data does not recede;a bottom out detector, which generates an alarm when the historical data achieves a minimum value from which the historical data does not recede;andstop and start reporting detectors, which respectively generate alarm when historical data reporting that is normally online or offline goes offline or online;extract descriptors of the first metric group from the KPIs to create a first anomaly detector feature profile thereof;repeat the determining and the extracting with respect to historical data of second and third metric groups, which are unrelated or indirectly related to the CPU utilization in the first computing system of the first metric group, to create second and third anomaly detector feature profiles thereof, respectively;ascertain which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile;andrank the second and third anomaly detector feature profiles based on which one has the greater correlation and compare the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce anomalies.
7
A computer system for selecting key performance indicators, the computer system comprising a computer readable storage medium having instructions stored thereon that are executable by a processing circuit to cause the processing circuit to:determine from historical data which anomaly detectors are associated with key performance indicators (KPIs) for identifying an anomalous event in a first metric group related to central processing unit (CPU) utilization in a first computing system, wherein the anomaly detectors comprise two or more of: a Gaussian detector, which generates an alarm when the historical data exhibits a predefined deviation;a flat line detector, which generates an alarm when the historical data remains at a same value for a predefined time;an increased variance detector, which generates an alarm when the historical data exhibits an increased variance from a baseline value;a de-correlation detector, which generates an alarm when the historical data exhibits a de-correlation from other data;a top out detector, which generates an alarm when the historical data achieves a maximum value from which the historical data does not recede;a bottom out detector, which generates an alarm when the historical data achieves a minimum value from which the historical data does not recede;andstop and start reporting detectors, which respectively generate alarm when historical data reporting that is normally online or offline goes offline or online;extract descriptors of the first metric group from the KPIs to create a first anomaly detector feature profile thereof;repeat the determining and the extracting with respect to historical data of second and third metric groups, which are unrelated or indirectly related to the CPU utilization in the first computing system of the first metric group, to create second and third anomaly detector feature profiles thereof, respectively;ascertain which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile;andrank the second and third anomaly detector feature profiles based on which one has the greater correlation and compare the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce anomalies.
13
Broadest claimClaim Score 20, narrow(NHIP)A computer-implemented method for selecting key performance indicators, comprising:determining from historical data which anomaly detectors are associated with key performance indicators (KPIs) for identifying an anomalous event in a first metric group related to central processing unit (CPU) utilization in a first computing system, wherein the anomaly detectors comprise two or more of: a Gaussian detector, which generates an alarm when the historical data exhibits a predefined deviation;a flat line detector, which generates an alarm when the historical data remains at a same value for a predefined time;an increased variance detector, which generates an alarm when the historical data exhibits an increased variance from a baseline value;a de-correlation detector, which generates an alarm when the historical data exhibits a de-correlation from other data;a top out detector, which generates an alarm when the historical data achieves a maximum value from which the historical data does not recede;a bottom out detector, which generates an alarm when the historical data achieves a minimum value from which the historical data does not recede;andstop and start reporting detectors, which respectively generate alarm when historical data reporting that is normally online or offline goes offline or online;extracting descriptors of the first metric group from the KPIs to create a first anomaly detector feature profile thereof;repeating the determining and the extracting with respect to historical data of second and third metric groups, which are unrelated or indirectly related to the CPU utilization in the first computing system of the first metric group, to create second and third anomaly detector feature profiles thereof, respectively;ascertaining which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile;andranking the second and third anomaly detector feature profiles based on which one has the greater correlation and comparing the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce anomalies.

Independent claims3

47 paragraphs in 4 sections, as filed

BACKGROUND

The present invention relates to key performance indicator selection and, more specifically, to the selection of key performance indicators for anomaly detection analytics.

In order to maintain or secure mission-critical systems, businesses often rely on monitoring systems that can help predict, detect and/or diagnose problems. However, when incidents occur in a complex production environment, it takes a tremendous amount of effort to investigate and determine the root causes of those incidents based on information provided by the monitoring system in use. For instance, a subject matter expert might need to analyze data related to metrics involved in an incident over a time period and for any given anomaly a large amount of effort can go into building up the pattern of metrics that helps an administrator understand and address the situation at hand.

Presently, the solution is complex and requires the writing of rules or situations such as those found in products, such as the IBM Tivoli Network Manager (ITMN). There, the approach is to build a situation from a series of rules (e.g., if CPU usage goes above X and memory usage drops below Y within time period Z, then raise an alarm). This type of solution involves a significant amount of manual encoding/rule writing and can be error prone and seen as a common source of pain to users. Thus, in many organizations, it has become common to “metricize” applications to enabling monitoring and this has led to an explosion in the number of possible metrics and the different systems from which these metrics can be monitored. It therefore has also become practically impossible and infeasible to write rules for the quantity of metrics that are available.

Solutions, such as smart cloud analytics predictive insights, aim to address the problem of metricized applications by applying large scale data mining techniques to automate the “rule writing” for users. Automated rule writing results in subsets of key performance indicators (KPIs), which involve metrics that are typically organized into groups, to be selected at a group-level granularity at an entry point to the system and at stage known as mediation. The KPI organization might be, at the metric level (e.g., “Response Time”) or at the resource level (e.g., “Response Time on WebSphere Servers”).

SUMMARY

According to an embodiment of the present invention, a computer program product for selecting key performance indicators is provided. The computer program product includes a computer readable storage medium having program instructions embodied therewith. The program instructions are readable and executable by a processing circuit to cause the processing circuit to determine from historical data which anomaly detectors are associated with key performance indicators (KPIs) for identifying an anomalous event in a first metric group, to extract descriptors of the first metric group from the KPIs to create a first anomaly detector feature profile thereof, to repeat the determining and the extracting with respect to historical data of second and third metric groups to create second and third anomaly detector feature profiles thereof, respectively, to ascertain which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile and to rank the second and third anomaly detector feature profiles based on which one has the greater correlation and compare the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce anomalies.

According to another embodiment of the present invention, a computer system for selecting key performance indicators is provided. The computer system includes a computer readable storage medium having instructions stored thereon that are executable by a processing circuit to cause the processing circuit to determine from historical data which anomaly detectors are associated with key performance indicators (KPIs) for identifying an anomalous event in a first metric group, extract descriptors of the first metric group from the KPIs to create a first anomaly detector feature profile thereof, repeat the determining and the extracting with respect to historical data of second and third metric groups to create second and third anomaly detector feature profiles thereof, respectively, ascertain which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile and rank the second and third anomaly detector feature profiles based on which one has the greater correlation and compare the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce anomalies.

According to yet another embodiment of the present invention, a computer-implemented method for selecting key performance indicators is provided. The computer-implemented method includes determining from historical data which anomaly detectors are associated with key performance indicators (KPIs) for identifying an anomalous event in a first metric group, extracting descriptors of the metric group from the KPIs to create a first anomaly detector feature profile thereof, repeating the determining and the extracting with respect to historical data of second and third metric groups to create second and third anomaly detector feature profiles thereof, respectively, ascertaining which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile and ranking the second and third anomaly detector feature profiles based on which one has the greater correlation and comparing the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce anomalies.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The forgoing and other features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

<figref idref="DRAWINGS">FIG. 1</figref> is a schematic diagram of a first computing system in accordance with embodiments;

<figref idref="DRAWINGS">FIG. 2</figref> is a schematic diagram of a second computing system in accordance with embodiments;

<figref idref="DRAWINGS">FIG. 3</figref> is a schematic diagram of a computer program product of the computing system of <figref idref="DRAWINGS">FIG. 1</figref> in accordance with embodiments;

<figref idref="DRAWINGS">FIG. 4</figref> is a flow diagram illustrating a deployment process for the computer program product of <figref idref="DRAWINGS">FIG. 3</figref> in accordance with embodiments; and

<figref idref="DRAWINGS">FIG. 5</figref> is a flow diagram illustrating a computer-implemented method for selecting key performance indicators in accordance with embodiments.

DETAILED DESCRIPTION

As will be described below, a computer program product, a system and a computer-implemented method are provided to enable the selection of KPIs to provide optimal value to users given a known set of deployable anomaly detectors and the environment or technical domain into which the computer program product, the system and the computer-implemented method are deployed. The deployable anomaly detectors may include, but are not limited to, detectors that detect large deviations from an expected Gaussian behavior, detectors that detect sudden halts or flat lines, detectors that detect slow leaks or drifts upwards and detectors that detect slow degradation or drifts downwards. Other anomaly detectors are, of course, possible and are included in the scope of this description. The environment or technical domain may include, but are not limited to, telecommunications and banking environments.

With reference to <figref idref="DRAWINGS">FIG. 1</figref>, a first computing system 10 is provided and may be configured for example as an enterprise computing system or as a personal computing system. In either case, the first computing system 10 includes first, second and third computing devices 11, 12 and 13 and servers 15 which are configured to be networked together for communication purposes. Each of the first, second and third computing devices 11, 12 and 13 and the servers 15 may include among other features a processing circuit 20, a display 30, user input devices 40 and a networking unit 50 as well as a computer program product 100 for selecting key performance indicators (KPIs). The processing circuit 20 may be provided as a micro-processor, a central processing unit (CPU) or any other suitable processing device. The display 30 may be provided as a monitor and is configured to display data and information as well as a graphical user interface to an administrator or user. The user input devices 40 may be provided as a mouse and a keyboard combination and are configured to allow the administrator or user to input commands to the processing circuit 20. The networking unit 50 may be provided as an Ethernet or other suitable networking device by which the first, second and third computing devices 11, 12 and 13 and servers 15 are communicative.

With reference to <figref idref="DRAWINGS">FIG. 2</figref>, a second computing system 10′ is provided and may be configured for example as an enterprise computing system or as a personal computing system. In either case, the first computing system 10′ includes first, second and third computing devices 11′, 12′ and 13′ and servers 15′ which are configured to be networked together for communication purposes. Each of the first, second and third computing devices 11′, 12′ and 13′ and the servers 15′ may include among other features a processing circuit 20′, a display 30′, user input devices 40′ and a networking unit 50.′ The processing circuit 20′ may be provided as a micro-processor, a central processing unit (CPU) or any other suitable processing device. The display 30′ may be provided as a monitor and is configured to display data and information as well as a graphical user interface to an administrator or user. The user input devices 40′ may be provided as a mouse and a keyboard combination and are configured to allow the administrator or user to input commands to the processing circuit 20′. The networking unit 50′ may be provided as an Ethernet or other suitable networking device by which the first, second and third computing devices 11′, 12′ and 13′ and servers 15′ are communicative.

With reference to <figref idref="DRAWINGS">FIG. 3</figref>, the computer program product 100 includes a computer readable storage medium 110 and may be installed in a memory unit 150 of one of the servers 15 (see <figref idref="DRAWINGS">FIG. 1</figref>) or any of the first, second and third computing devices 11, 12 and 13. For clarity and brevity, the following description will relate to only the case where the computer readable storage medium 110 of the computer program product 100 is installed in the memory unit 150 of the one of the servers 15 although it is to be understood that this configuration is merely exemplary and is not intended to be limiting of the disclosure in any way.

The computer program product 100 further includes a historical time-series database 120, in which historical time-series data is grouped into multiple metric groups 121, and a plurality of anomaly detectors 130. Each metric group 121, such as a central processing unit (CPU) utilization metric group, is made up of time-series data sets from different hosts in a distributed environment (i.e., the first, second and third computing devices 11, 12 and 13 in the first computing system 10). The anomaly detectors 130 may be provided with access to the multiple metric groups 121 of the historical time-series database and may include, but are not limited to, a Gaussian detector 131, a flat line detector 132, an increased variance detector 133, a de-correlation detector 134, a top out detector 135, a bottom out detector 136, a stop reporting detector 137 and a start reporting detector 138.

In accordance with embodiments, the Gaussian detector 131 generates an alarm for a given metric group 121 when the corresponding data exhibits, for example, a greater than three-sigma deviations. The flat line detector 132 generates an alarm for a given metric group 121 when the corresponding data flattens or reports a same value for an extended period of time. The increased variance detector 133 generates an alarm for a given metric group 121 when the corresponding data exhibits an increased tendency to vary from a baseline value. The de-correlation detector 134 generates an alarm for a given metric group 121 when the corresponding data exhibits a de-correlation with respect to data of another metric group 121. The top out detector 135 generates an alarm for a given metric group 121 when the corresponding data achieves a maximum value and then does not recede from that maximum value. The bottom out detector 136 generates an alarm for a given metric group 121 when the corresponding data achieves a minimum value and then does increase from that minimum value. The stop reporting detector 137 generates an alarm for a given metric group 121 when a metric group 121 that is normally online goes offline and the start reporting detector 138 generates an alarm for a given metric group 121 when a metric group 121 that is normally online goes offline.

The computer readable storage medium 110 has first, second, third and fourth program instructions 111, 112, 113 and 114 stored thereon. The first program instructions 111 are executable by the processing circuit 20 of the server 15 to cause the processing circuit 20 to determine from the historical time-series data which ones or more of the anomaly detectors 130 are associated with or produce from their respective operations key performance indicators (KPIs) for identifying an anomalous event in a first one of the metric groups 121 (hereinafter referred to as “first metric group 121”). The second program instructions 112 are executable by the processing circuit 20 of the server 15 to cause the processing circuit 20 to extract descriptors of the first metric group 121 from the KPIs to create a first anomaly detector feature profile thereof. The third program instructions 113 are executable by the processing circuit 20 to cause the processing circuit 20 (or the processing circuit 20′ of the server 15′ of the second computing system 10′) to repeat the determining and the extracting with respect to historical data of second and third metric groups 121/121′ to create second and third anomaly detector feature profiles thereof, respectively. The fourth program instructions 114 executable by the processing circuit 20 to cause the processing circuit 20 (or the processing circuit 20′ of the server 15′ of the second computing system 10′) to ascertain which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile.

The computer readable storage medium 110 also has fifth program instructions 115 stored thereon and may have sixth and seventh program instructions 116 and 117 stored thereon. The fifth program instructions 115 cause the processing unit 20 to rank the second and third anomaly detector feature profiles based on which one has the greater correlation and compare the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce or to be usable in the identification of anomalies.

The sixth program instructions 116 cause the processing unit 20 to deploy the one of the second and third metric groups 121/121′ whose anomaly detector feature profile has a greater correlation to the first anomaly detector feature profile. The seventh program instructions 117 are executable by the processing circuit 20 to cause the processing circuit 20 to update the first anomaly detector feature profile. More particularly, the seventh program instructions 117 may be executable by the processing circuit 20 to cause the processing circuit 20 to update the first anomaly detector feature profile based on a performance of the deployed one of the second and third metric groups 121/121′. In the latter case, the computer readable storage medium 110 may be installed in and deployed from the memory unit 150 of the server 15 of the first computing system 10 or the memory unit 150′ of the server 15′ of the second computing system 10′.

The second and third metric groups 121 may be made up of time-series data sets from same or different hosts in a distributed environment, such as the first, second and third computing devices 11, 12 and 13 in the first computing system 10. Alternatively, the second and third metric groups 121′ may be made up of time-series data sets from different hosts in a distributed environment, such as the first, second and third computing devices 11′, 12′ and 13′ in the second computing system 10′. In the latter case, the

In an exemplary deployment, let it be assumed that the Gaussian detector 131 and the flat line detector 132 are active and given access to the historical time-series data associated with the first metric group 121 and that the first metric group 121 relates to CPU utilization in the first computing system 10. Here, the execution of the first program instructions 111 may leads to a first determination that 10% of the metrics of the first metric group 121 generated an alarm on the Gaussian detector 131 and that 3% of the metrics of the first metric group 121 generated an alarm on the flat line detector 132 and to a second determination that the generated alarms on the Gaussian and flat line detectors 131 and 132 sufficiently identifies an anomalous event in the second computing system 10. Based on these determinations, the execution of the second program instructions 112 leads to an extraction of a (10, 3) descriptor of the first metric group 121 (i.e., a first anomaly detector feature profile).

Subsequently, by executing the third program instructions 113 on the second and third metric groups 121/121′, second and third feature profiles thereof may be respectively created even if the second and third metric groups 121/121′ are groups that have not yet been seen or considered. That is, it may be determined that the second metric group 121/121′ has a (10, 4) second anomaly detector feature profile since 10% of its metrics generate an alarm on the Gaussian detector 131 and 4% of its metrics generate an alarm on the flat line detector 132. Similarly, it may be determined that the third metric group 121/121′ has a (30, 20) third anomaly detector feature profile since 30% of its metrics generate an alarm on the Gaussian detector 131 and 20% of its metrics generate an alarm on the flat line detector 132.

Thus, since the second anomaly detector feature profile of the second metric group 121/121′ has a greater correlation to the (10, 3) descriptor of the first anomaly detector feature profile than that of the third metric group 121/121′, by way of execution of the fourth, fifth and sixth program instructions 114, 115 and 116, it may be ascertained that the second anomaly detector feature profile has a greater correlation to the first anomaly detector feature profile and is to be deployed for identification of future anomalous events in the first computing system 10 or the second computing system 10′. That is, given a higher degree of correlation between KPI metric groups, features from detectors and anomalies, a profile of the KPI group is assigned and ranked. This generated profile of KPI metric groups is then compared to future metrics either in the current environment or in subsequent environments to determine if specific KPIs are likely to produce or be usable in the identification of anomalies.

While it is understood that the first-seventh program instructions 111-117 may be deployed by manual loading thereof directly into a client, server and/or proxy computer by way of a loadable storage medium, such as a CD, DVD, etc., or by being manually inserted into each of the first, second and third computing devices 11, 12 and 13 and the servers 15, the first-seventh program instructions 111-117 may also be automatically or semi-automatically deployed into the first computing system 10 and the second computing system 10′ by way of the servers 15 and 15′. In such cases, the first-seventh program instructions 111-117 may be downloadable into client computers that will then execute the first-seventh program instructions 111-117.

In accordance with alternative embodiments, the first-seventh program instructions 111-117 may be sent directly to a client system via e-mail with the first-seventh program instructions 111-117 then being detached to or loaded into a directory. Another alternative would be that the first-seventh program instructions 111-117 be sent directly to a directory on a client computer hard drive. When there are proxy servers, however, loading processes will select proxy server codes, determine on which computers to place the proxy servers' codes, transmit the proxy server codes and then install the proxy server codes on proxy computers. The first-seventh program instructions 111-117 will then be transmitted to the proxy server and subsequently stored thereon.

In accordance with embodiments and, with reference to <figref idref="DRAWINGS">FIG. 4</figref>, a deployment process of the computer program product described above is provided. The process begins at block 300 and at block 101 with a determination of whether the first-seventh program instructions 111-117 will reside on the servers 15 when executed. If so, then the servers 15 that will contain the executables are identified at block 209. The first-seventh program instructions 111-117 for the servers 15 are then transferred directly to the servers' storage via FTP or some other protocol or by copying though the use of a shared file system at block 210 such that the first-seventh program instructions 111-117 are installed on the servers 15 at block 211.

Next, a determination is made on whether the first-seventh program instructions 111-117 are to be deployed by having users access the first-seventh program instructions 111-117 on servers 15 at block 102. If so, the server addresses that will store the first-seventh program instructions 111-117 are identified at block 103 and a determination is made if a proxy server is to be built at block 200 to store the first-seventh program instructions 111-117. A proxy server is a server that sits between a client application, such as a Web browser, and a real server and operates by intercepting all requests to the real server to see if it can fulfill the requests itself. If not, the proxy server forwards the request to the real server. The two primary benefits of a proxy server are to improve performance and to filter requests.

If a proxy server is required, then the proxy server is installed at block 201 and the first-seventh program instructions 111-117 are sent to the servers 15 via a protocol, such as FTP, or by being copied directly from the source files to the server files via file sharing at block 202. Another embodiment involves sending a transaction to the (one or more) servers that contained the process software, and have the server process the transaction and then receive and copy the process software to the server's file system. Once the process software is stored at the servers, the users may then access the first-seventh program instructions 111-117 on the servers 15 and copy to the same to their respective client computer file systems at block 203. Alternatively, the servers 16 may automatically copy the first-seventh program instructions 111-117 to each client and then run an installation program for the fir first-seventh program instructions 111-117 at each client computer whereby the user executes the program that installs the first-seventh program instructions 111-117 on his client computer at block 212 and then exits the process at block 108.

At block 104, a determination is made as to whether the first-seventh program instructions 111-117 are to be deployed by sending the first-seventh program instructions 111-117 to users via e-mail. If a result of the determination is affirmative, the set of users where the first-seventh program instructions 111-117 will be deployed are identified together with the addresses of the user client computers at block 105 and the first-seventh program instructions 111-117 are sent via e-mail to each of the users' client computers. The users then receive the e-mail at block 205 and then detach the first-seventh program instructions 111-117 from the e-mail to a directory on their client computers at block 206. The user executes the program that installs the first-seventh program instructions 111-117 on his client computer at block 212 and then exits the process at block 108.

Lastly, a determination is made on whether the first-seventh program instructions 111-117 will be sent directly to user directories on their client computers at block 106. If so, the user directories are identified at block 107 and the process software is transferred directly to the user's client computer directories at block 207. This can be done in several ways such as, but not limited to, sharing the file system directories and then copying from the sender's file system to the recipient user's file system or, alternatively, using a transfer protocol such as File Transfer Protocol (FTP). The users access the directories on their client file systems in preparation for installing the first-seventh program instructions 111-117 at block 208, execute the program that installs the first-seventh program instructions 111-117 at block 212 and then exit the process at block 108.

With reference to <figref idref="DRAWINGS">FIG. 5</figref>, a method for selecting key performance indicators is provided. The method includes determining from historical data which anomaly detectors are associated with key performance indicators (KPIs) for identifying an anomalous event in a first metric group at block 501, extracting descriptors of the metric group from the KPIs to create a first anomaly detector feature profile thereof at block 502, repeating the determining and the extracting with respect to historical data of second and third metric groups to create second and third anomaly detector feature profiles thereof, respectively, at block 503, and ascertaining which of the second and third anomaly detector feature profiles has a greater correlation to the first anomaly detector feature profile at block 504. The method further includes ranking the second and third anomaly detector feature profiles based on which one has the greater correlation and comparing the higher ranked one to future metrics in current or subsequent environments to determine if specific KPIs are likely to produce or be usable in the identification of anomalies at block 505.

A potential use of the computer program product and the method described above would, in some cases, be to generate point tooling or “data mediation tooling” at potential data sources and databases and, for time-series data stored therein, to analyze the time-series data and to present ranked lists to users for inclusion or exclusion with respect to downstream analysis. As noted above, aspects of the disclosure can be automated whereby, for example, only time-series data scoring above a certain level might be flagged for subsequent inclusion in downstream analysis. Moreover, users will have an opportunity to adjust weighting values assigned to various factors and to build upon those supplied by a pre-existing development team. For example, the user may decide that particular time-series data very directly measures key customer experience aspects and are more important to include in downstream analytics then the corresponding score might otherwise indicate. Here, an example of a binary score would be if the time-series is directly associated with customer experience measurements (e.g., it is a well-known metric type) it may automatically be scored/flagged for inclusion based upon the best practice.

Advantages associated with the computer program product and method described above include, but are not limited to, the fact that the features described herein can work without explicit notions of ‘best-practices’ or pre-existing lists of metrics since important factors that are generally included in the creation of such best-practices or lists are codified within the time series scoring mechanisms. Thus, from day 1 of a product deployment, data selection can be explicitly facilitated and can deal with vagaries of a particular environment (i.e., where a metric type X, based upon observation and analysis, is determined to be acceptable in one environment but would not be in another). In addition, administrators or users of the computer program product and method will be able to select the top-N sets of the time-series data and have confidence that the set is a “best” fit for a given case.

The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one more other features, integers, steps, operations, element components, and/or groups thereof.

Contents4

Every citation, both waysCites: the store holds 18 of 19

Citations
Document	Relation	Office	Cited during
US11888597B2	Cited by	United States of America	Applicant
US11513930B2	Cited by	United States of America	Applicant
US11995562B2	Cited by	United States of America	Applicant
US11243835B1	Cited by	United States of America	Applicant
US11388077B2	Cited by	United States of America	Applicant
US11797538B2	Cited by	United States of America	Applicant
US11474892B2	Cited by	United States of America	Applicant
US11599404B2	Cited by	United States of America	Applicant
US11403326B2	Cited by	United States of America	Applicant
US2009248722A1	Cites	United States of America	Applicant
US2012051642A1	Cites	United States of America	Search report
US2013110761A1	Cites	United States of America	Search report
US2013151907A1	Cites	United States of America	Search report
WO2014200458A1	Cites	World Intellectual Property Organization (WIPO)	Applicant
US2014280142A1	Cites	United States of America	Applicant
US2016103838A1	Cites	United States of America	Search report
US2016147585A1	Cites	United States of America	Search report
US7831464B1	Cites	United States of America	Search report
US8458090B1	Cites	United States of America	Search report
US20090248722A1	Cites	United States of America	Applicant
US20120051642A1	Cites	United States of America	Search report
US20130110761A1	Cites	United States of America	Search report
US20130151907A1	Cites	United States of America	Search report
US20140280142A1	Cites	United States of America	Applicant
US20160103838A1	Cites	United States of America	Search report
US20160147585A1	Cites	United States of America	Search report
WO2014200458	Cites	World Intellectual Property Organization (WIPO)	Applicant

2 members in 1 office

This recordMember, by publication datePriority claim

Priority claims2

Priority claims
Document	Office	Kind	Date
201514929982	United States of America	A	2015-11-02
US201514929982	–	–	–

Members2

Family members, oldest first
Document	Office	Kind
US2017124502A1	United States of America	A1	2017-05-04
US10169731B2This record	United States of America	B2	2019-01-01

51 transactions on the USPTO file

Allowed after 1 non-final rejection.

Non-final rejections: 1
Final rejections: 0
RCEs: 0
Appeals: 0

Over time

Point at a mark for the transaction

Transactions

Prosecution transactions, newest first
	Event	Code
2023-02-06	Expire PatentEXP.	EXP.
2022-08-22	Maintenance Fee Reminder MailedREM.	REM.
2019-01-01	Recordation of Patent Grant MailedPGM/	PGM/
2019-01-01	Patent Issue Date Used in PTA CalculationAllowedPTAC	PTAC
2018-12-13	Email NotificationEML_NTR	EML_NTR
2018-12-12	Issue Notification MailedAllowedWPIR	WPIR
2018-12-05	Dispatch to FDCD1935	D1935
2018-12-05	Application Is Considered Ready for IssuePILS	PILS
2018-12-05	Correspondence Address ChangeC.AD	C.AD
2018-11-27	Issue Fee Payment VerifiedN084	N084
2018-11-27	Issue Fee Payment ReceivedIFEE	IFEE
2018-09-12	Electronic ReviewELC_RVW	ELC_RVW
2018-09-12	Email NotificationEML_NTF	EML_NTF
2018-09-12	Mail Notice of AllowanceAllowedMN/=.	MN/=.
2018-09-08	Notice of Allowance Data Verification CompletedAllowedN/=.	N/=.
2018-09-03	Reasons for AllowanceEX.R	EX.R
2018-09-03	Examiner's Amendment CommunicationEX.A	EX.A
2018-06-21	Date Forwarded to ExaminerFWDX	FWDX
2018-06-12	Email NotificationEML_NTR	EML_NTR
2018-06-12	Mail Applicant Initiated Interview SummaryMEXIA	MEXIA
2018-06-07	Response after Non-Final ActionA...	A...
2018-06-06	Interview Summary - Applicant Initiated - TelephonicEXAT	EXAT
2018-06-06	Interview Summary- Applicant InitiatedEXIA	EXIA
2018-03-08	Electronic ReviewELC_RVW	ELC_RVW
2018-03-08	Email NotificationEML_NTF	EML_NTF
2018-03-08	Mail Non-Final RejectionNon-final rejectionMCTNF	MCTNF
2018-03-05	Non-Final RejectionNon-final rejectionCTNF	CTNF
2018-03-04	Information Disclosure Statement consideredIDSC	IDSC
2018-03-04	Information Disclosure Statement consideredIDSC	IDSC
2017-12-13	Case Docketed to Examiner in GAUDOCK	DOCK
2017-05-05	Email NotificationEML_NTR	EML_NTR
2017-05-04	Application ready for PDX access by participating foreign officesCCRDY	CCRDY
2017-05-04	PG-Pub Issue NotificationPG-ISSUE	PG-ISSUE
2016-06-17	Case Docketed to Examiner in GAUDOCK	DOCK
2016-06-10	Application Dispatched from OIPEOIPE	OIPE
2016-04-07	Correspondence Address ChangeC.AD	C.AD
2016-01-12	Electronic Information Disclosure StatementEIDS.	EIDS.
2016-01-12	Information Disclosure Statement (IDS) FiledWIDS	WIDS
2015-11-17	Email NotificationEML_NTR	EML_NTR
2015-11-17	Application Is Now CompleteCOMP	COMP
2015-11-17	Filing ReceiptFLRCPT.O	FLRCPT.O
2015-11-16	Sent to Classification ContractorPGPC	PGPC
2015-11-16	FITF set to YES - revise initial settingFTFS	FTFS
2015-11-05	Cleared by OIPE CSRL194	L194
2015-11-02	Electronic Information Disclosure StatementEIDS.	EIDS.
2015-11-02	Patent Term Adjustment - Ready for ExaminationPTA.RFE	PTA.RFE
2015-11-02	Applicants have given acceptable permission for participating foreignAPPERMS	APPERMS
2015-11-02	Information Disclosure Statement (IDS) FiledWIDS	WIDS
2015-11-02	IFW Scan & PACR Auto Security ReviewSCAN	SCAN
2015-11-02	Entity Status Set To Undiscounted (Initial Default Setting or Status Change)BIG.	BIG.
2015-11-02	Initial Exam Team nnIEXX	IEXX

7 legal events, as the office reported them to INPADOC

Over the term

Point at a mark for the event

Events

Legal events, newest first
	Event	Code
2023-02-28	Lapsed due to failure to pay maintenance feeLapsedFP	FP
2023-02-06	Lapse for failure to pay maintenance feesLapsedLAPS	LAPS
2023-02-06	Information on status: patent discontinuationSTCH	STCH
2022-08-22	Fee payment procedureFEPP	FEPP
2018-12-12	Information on status: patent grantGrantedSTCF	STCF
2018-12-12	Information on status: patent grantGrantedSTCF	STCF
2015-11-02	AssignmentAS	AS