Systems and methods for behavior-based automated malware analysis and classification

[001] The present teachings relate to systems and methods for behavior-based automated malware analysis and classification, and more particularly, to platforms and techniques for identifying malware family groups and generating clusters of similar malware samples based on low-level artifacts captured during execution of those malicious entities.

Background

[002] In the field of software security, platforms have been known which examine samples of viruses or other malware, and classify those samples into one of a predefined set of known malware families. In the most general terms, those classifiers can operate by examining the actual code of the malware to locate unique sequences or bytes, or they can instead examine the behavior of those entities while executing. In either of those respective signature-based and behavior-based classifiers, the logic assigns a generic malware family name or “label” to the best match from a library of known malware entities.

[003] In addition, platforms are also known in the software security field in which samples of malware objects can be scanned for signature-based and behaviorbased attributes, and assigned to malware groups having similar characteristics.

[004] However, in known classification and/or clustering platforms, the system generally relies upon relatively high-level attributes or characteristics in narrowing down the potential classes or clusters into which a given malware sample will be placed. This limits the precision with which classes or groups can be assigned. Likewise, existing platforms, in particular for clustering purposes, rely upon a single chosen algorithm to identify similar malware groups, which can limit the effectiveness of the results. Further, existing platforms typically capture the (relatively high-level) attributes which they analyze from a fairly small sample set, which can also lead to inconsistencies or other shortcomings in the results.

[005] It may be desirable to provide methods and systems for behavior-based automated malware analysis and classification, in which greater granularity in captured attributes, larger sample sets, and flexibility in applied algorithms can be leveraged to produce better malware identification results.

DESCRIPTION OF DRAWINGS

[006] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present teachings and together with the description, serve to explain the principles of the present teachings. In the figures:

[007] FIG. 1 illustrates an overall environment which can be used in systems and methods for behavior-based automated malware analysis and classification, according to various embodiments;

[008] FIG. 2 illustrates components of the system used for algorithm selection and other actions, according to various embodiments;

[009] FIG. 3 illustrates a flowchart of processing that can be used in behaviorbased automated malware analysis and classification, according to various embodiments; and

[010] FIG. 4 illustrates exemplary hardware, software, and other resources that can be used in behavior-based automated malware analysis and classification, according to various embodiments.

DESCRIPTION OF EMBODIMENTS

[011] Embodiments of the present teachings relate to systems and methods for behavior-based automated malware analysis and classification. More particularly, embodiments relate to platforms and techniques for automated malware analysis, classification, and characterization, wherein the sample set used to drive that analysis can comprise a comparatively large set of source attributes and other information, and that information can be extracted from relatively low-level operational artifacts of a computer or other device hosting the sample under analysis. In aspects, classification and/or clustering operations can be carried out on those artifacts using an extensible set of algorithms which can be tuned and/or selected based on accuracy and other parameters that are selectable or adjustable by the user. The ability to vary, examine, and change the operative algorithms can allow an analyst or other user to tune the performance or results achieved.

[012] Reference will now be made in detail to exemplary embodiments of the present teachings, which are illustrated in the accompanying drawings. Where possible the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[013] FIG. 1 illustrates an overall environment in which systems and methods for behavior-based automated malware analysis and classification can operate, according to aspects. In aspects as shown, a set of samples 102 can be assembled and placed into a sample queue 104, as part of the preparation for analysis and identification according to techniques described herein. The set of samples 102 can be or include program code, applications, routines, scripts, modules, data, files, text, media, and/or other information or content. In implementations, the set of samples 102 can be collected automatically from one or more source, such as, for instance, through anti -viral or other software of a Web site operator, network operator, service provider, Internet company including an infrastructure provider, and/or other entity or source.

[014] In implementations, the set of samples 102 can in addition or instead be manually collected by a systems administrator for those or other entities, after reviewing traffic records, logs, and/or other files or data. In cases, the set of samples 102 can include samples which are known to contain viruses, Trojans, worms, and/or other malware or malicious code or content. In cases, the set of samples 102 can include samples whose content is not known or identified. The set of samples 102 can be provided to the sample queue 104 via the Internet or other network connection or channel. The samples contained in sample queue 104 can consist of binary files that are kept in static form and not executed. After being assembled in the sample queue 104, the set of samples 102 can be provided to a submitter 148 for transmission to an extractor 106. The submitter 148 is responsible for feeding samples from the set of samples 102 to the extractor 106. The samples can be selected by the submitter 148 based on their priority in the sample queue 104. Given that the set of samples 102 can be drawn from multiple sources including, once again for example, customer submissions, internal submissions, and software vendor samples, prioritization of the samples can be used.

[015] Each of the samples in the set of samples 102 can be ranked by the submitter 148 with different priority. For instance, customer submissions can be assigned the highest priority followed by internal submissions from a network operator, and then software vendor feeds (such as from anti-virus providers). Other priorities, ranks, sources, and/or ways of ordering the set of samples 102 can be used.

[016] Once passed to the extractor 106, the selected samples can be managed by a controller 110 for distribution to a set of virtual machines 108 and subsequent analysis. The set of virtual machines 108 are functionally independent of the controller 110, which allows the system to add and remove VMs without affecting the overall operation of the system. The set of virtual machines 108 can consist of images that have different versions of OSes or families of OSes, with different patch levels, such as the Windows™ family of operating systems, Linux distributions, and others.

[017] In general, the extractor 106 can operate to examine the set of samples 102 and capture, extract, enumerate, and/or otherwise identify a set of artifacts 136 related to the subject samples.

[018] The controller 110 can fetch highest priority samples from the sample queue 104 with the earliest smallest submission time, and begin to process them. The processing can begin by the sample being copied into an available virtual machine (VM) in the set of virtual machines 108, applying custom settings to the VM, if there are any, and running the sample. The configuration for each VM can be applied via a python agent installed on each VM, allowing the submitter 148 and/or other logic to modify the VM environment as necessary. For example, if an analyst identifies that a malware sample is not running because it checks a specific registry key for an environment artifact to detect the virtual environment, the analyst can submit a script with the sample that will adjust the registry key so the malware sample fails to detect the virtual environment and proceed to infect the system.

[019] The controller 110 also detects the type of file being submitted and ensures it runs correctly. For example, if a dynamic link library (DLL) file is submitted, the controller 110 can install the DLL as a Windows™ Service and start the service to identify the behavior of the subject sample. If a universal resource locator (URL) is submitted, the controller 110 can launch a browser application, and visit the URL. After the sample is run for an allotted time, the controller 110 pauses the VM and begins artifact collection.

[020] The controller 110 can run one or more tools, logic, and/or processes to capture any one or more of the following artifacts for inclusion in the set of artifacts

150:

1) File system information: attributes related to when files have been created, modified, and deleted; file content; and file meta data.

2) Registry information: attributes related to when a registry has been created, modified, and deleted; registry content; and registry meta data.

3) Network information: attributes related to domain name system (DNS) resolution; outgoing and incoming content; and meta data.

4) Volatile Memory information: This artifact can, in implementations, only be stored for relatively short intervals, such as one week or other period. The volatile memory information can be used to run malware detection framework tools, such as YARA signatures, on the memory to identify malware of interest.

[021] The set of artifacts 150 including file system, registry, and network artifacts and their semantics can be extracted from source in the set of virtual machines 108 including any virtual disks and/or packet capture resources, such as a packet capture (PCAP) file. The artifacts and their semantics are then parsed by the controller 110 and stored in the artifact database 112, for instance, in corresponding tables for each artifact. The packet capture files can also stored in the artifact database 112 for record keeping. The virtual machines in the set of virtual machines 108 can also be configured to save a copy of the virtual memory to hard disk or other storage, when paused. All of the artifacts captured in the set of artifacts 150 can, once more, be stored to the artifact database 112, and/or other local or remote storage for further analysis and decomposition into classes and/or clusters.

[022] More specifically and as for instance illustrated in FIG. 2, in implementations, in general, a feature extractor 114 can access the artifact database 112, and being the process of reading out desired artifacts for purposes of classification and/or clustering of the malware or other malicious content contained in the original set of samples 102. In implementations as shown, the feature extractor 114 can communicate with a set of software, logic, and/or processes including a labeling module 116, a testing selector module 118, and a training selector module 120 to conduct those operations. The feature extractor 114 can analyze the set of artifacts 150 stored in the artifact database 112 and extract representative features from them. In general, features can represent selected groupings of artifacts, which may relate to given performance or execution characteristics of the sample.

[023] The features can be broken down based on the class of artifacts used for generating them, for instance, into several groups. A number are described, although it will be appreciated that other numbers or types of features or groups can be used. First, file system features can be used which are derived from file system artifacts created by the sample when run in the virtual environment of the set of virtual machines 108. Counts can be used both for files created, deleted, and modified, as well as counts for files created in predefined paths like %APPDATA%, %TEMP%, %PROGRAMFILES%, and other common locations. Counts can be kept for files created with unique extensions, for example if a malware sample creates four files on the system, a batch file (.BAT), two executable files (.EXE), and a configuration file can be tracked.

[024] Second, class features can include registry-related information, such as [025] registry created keys, modified keys, deleted keys, and/or a count of keys with certain type. Third, the feature classes can include network features, such as IP and port Unique destination IP, counts over certain ports, connections such as TCP, UDP, RAW.

[026] Request type (e.g., POST, GET, HEAD) and response types or codes (200s through 500s) can also be tracked, as can size request and response distribution (e.g. DNS MX, NS, A records, PTR, SOA, CNAME). In terms of file size, the file size of created files can be used, including particularly the distribution of the files’ size (rather than raw file size). The file size range can be partitioned, corresponding to the difference between the size of the largest and smallest files generated by a malware sample, into multiple

[027] ranges. Four ranges can be used, one for each quartile, and create counts for files with size falling into each range or quartile.

[028] Registry features can also be tracked, and can be similar to the file features since counts can be tracked for registries created, modified, and deleted, registry type like REG_SZ, REG_BIN, and REG_DW0RD.

[029] Third, network features can be tracked, broken into groups. The first

Network group is raw network features, which includes count of unique IP addresses, count of connections established for 18 different port numbers, quartile count of request size, and type of protocol (attention was limited to three popular protocols, namely the TCP, UDP, RAW). The second group is the HTTP features which include counts for POST, GET, and HEAD request; the distribution of the size of reply packets (using the quartile distribution format explained earlier), and counts for HTTP response codes, namely 200, 300, 400, and 500. The third category includes DNS (domain name system)-related features such as counts for A, PTR, CNAME, and MX record lookups.

[030] For safety during working analysis, the controller 110 and/or other logic can block a list of wormable ports, including port 25, 587, 445, 139, and 137 at the router level when evaluating samples. The run time of samples can also be limited, and bandwidth can be limited.

[031] In implementations, memory features may or may not be used in the evaluation of samples, but the system can be configured to collect and archive memory artifacts, as desired.

[032] It may also be noted that in implementations, different features’ values can be normalized or mapped into the range of 0 to 1 or other scales, thus not biasing the feature selection process towards any feature except of its true importance.

[033] In terms of analyzing a sample based on its artifacts and/or features, an algorithm selector 122, communicating with the labeling modules, testing selector module 118, and/or training selector module 120 can build classifiers and/or clustering algorithms for behavior-based group and labeling of malware samples. Based on the class of algorithm to be used, whether it is binary classification or clustering, the training (if applicable) and testing data can be determined by the administrator or other user.

[034] If the data is to be classified, the system can train a model using a verified and labeled data subset, and use unlabeled data for classification. The algorithm selector 122 can access an algorithm database 128 to access a set of classifier algorithms 132, which can contain or access multiple classification algorithms, including support vector machines (SVMs), decision trees, linear regression, and knearest-neighbor, among others. With any one or more choice of classification algorithm, the user can be presented with multiple settings and optimization options.

[035] In terms of those classification algorithms and their selection, the system can implement several binary classification algorithms, and is not restricted to any one particular classifier. Examples of classification algorithms that can be employed include the support vector machine (SVM), linear regression (LR), classification trees, k-nearest-neighbor (KNN), and the perceptron method.

[036] Although KNN is not strictly a binary classifier, according to implementations it can be modified by providing it with proper (odd) k, then voting is performed over which class a sample belongs to. To understand how different classification algorithms perform on the set of features and malware samples we had, testing of the classification of the malware samples across multiple algorithms has been performed, providing certain suggestions. For the SVM, and LR, several parameters for regularization, loss, and kernel functions have been tested. For this experiment, the same Zeus malware dataset was used as one class. A second class was generated using a dataset with the same size as Zeus from the total population that excludes ZBot in Table 3.

Table 3: Malware samples, and their corresponding labels, used in the classification training and testing.

<td> Size</td><td> %</td><td> Family</td><td> Description</td>

<td> 1,077</td><td> 0.94</td><td> Ramnit</td><td> File infector and a Trojan with pur-pose of stealing financial, personal, and system information</td>

<td> 1,090</td><td> 1.0</td><td> Bredolab</td><td> Spam and malware distribution bot</td>

<td> 1,091</td><td> 1.0</td><td> ZAccess</td><td> Rootkit Trojan for bitcoin mining, click fraud, and paid install.</td>

<td> 1,205</td><td> 1.1</td><td> Autorun</td><td> Generic detection of autorun fucntionality in malware.</td>

<td> 1,336</td><td> 1.2</td><td> Spyeye</td><td> Banking trojan for stealing personal and financial information.</td>

<td> 1,652</td><td> 1.4</td><td> SillyFDC</td><td> An autorun worm that spreads via portable devices and capable of downloading other malware.</td>

<td> 2,086</td><td> 1.8</td><td> Zbot</td><td> Banking trojan for stealing personal and financial information.</td>

<td> 2,422</td><td> 2.1</td><td> TDSS</td><td> Rootkit trojan for monetizing resources of infected machines.</td>

<td> 5,460</td><td> 4.7</td><td> Virut</td><td> Polymorphic file infector virus with trojan capability.</td>

<td> 7,691</td><td> 6.7</td><td> Sality</td><td> same as above, with rootkit, trojan, and worm capability.</td>

<td> 21,047</td><td> 18.3</td><td> Fakealert</td><td> Fake antivirus malware with purpose to scam victims.</td>

<td> 46,157</td><td> 40.1</td><td> Subtotal</td><td></td>

<td> 69,000</td><td> 59.9</td><td> Others</td><td> Small mal, &lt; 1k samples each</td>

<td> 115,157</td><td> 100</td><td> Total</td><td></td>

[037] Using 10-fold cross validation, the classifier was trained on part of both datasets using a group of features, and combined the remaining of each set for testing. The algorithms shown in Table 4 were run to label the testing set. For the performance of the different algorithms, the accuracy, precision, recall, and F-score were used. The results are shown in Table 4. First of all, while all algorithms perform fairly well on all measures of performance by achieving a precision and recall above 85%, it may be noted that SVM (with polynomial kernel for a degree of 2) performs best, achieving more than 99% of precision and recall, followed by decision trees, which is slightly lagged by SVM (with linear kernel). Interestingly, and despite being simple and lightweight, the logistic regression model achieves close to 90% on all performance measures, also providing competitive results. While some provide less accuracy than the best performing algorithms, all of those algorithms can be used as a building block in platforms and techniques herein, which can ultimately make use of all classifiers to achieve better results.

Table 4: Results of binary classification using several algorithms in terms of their accuracy, precision, recall, and F-score.

<td> Algorithm</td><td></td><td> A</td><td> P</td><td> R</td><td> F</td>

<td> SVM Polynomial Kemal</td><td></td><td> 99.22%</td><td> 98.92%</td><td> 99.53%</td><td> 99.22%</td>

<td> Classification Trees</td><td></td><td> 99.13%</td><td> 99.19%</td><td> 99.06%</td><td> 99.13%</td>

<td> SVM Linear Kernal</td><td></td><td> 97.93%</td><td> 98.53%</td><td> 97.30%</td><td> 97.92%</td>

<td> SVM Dual (L2R, L2L)</td><td></td><td> 95.64%</td><td> 96.35%</td><td> 94.86%</td><td> 95.60%</td>

<td> Log, Regression (L2R)</td><td></td><td> 89.11%</td><td> 92.71%</td><td> 94.90%</td><td> 88.63%</td>

<td> K-Nearest Neighbor</td><td></td><td> 88.56%</td><td> 93.29%</td><td> 83.11%</td><td> 87.90%</td>

<td> Log, Regression (L1R)</td><td></td><td> 86.98%</td><td> 84.81%</td><td> 90.09%</td><td> 87.37%</td>

<td> Perceptron</td><td></td><td> 86.15%</td><td> 84.93%</td><td> 87.89%</td><td> 86.39%</td>

[038] As for the cost of running the different algorithms, it may be noted that the SVM with polynomial kernel is relatively slow, while the decision trees require the most number of features to achieve high accuracy (details are omitted). On the other hand, while the dual SVM provides over 95% of performance on all measures, it runs relatively quickly. For that, and to demonstrate other aspects in our evaluation, attention may be restricted to the dual SVM, where possible. SVM is in aspects known for its generalization and resistance to noise.

[039] To understand causes for the relatively high false alarms (causing part of the degradation in precision and recall) with some of the algorithms, mislabeled Zeus and non-Zeus malware samples were examined. It was noted that distance in the feature vector between misclassified samples is far from the majority of other samples

[040] within the class. This is however understandable, given that a single class of malware (Zeus and non-Zeus) includes within itself multiple sub-classes that the high-level label would sometimes miss. This observation is further highlighted in the clustering application (described below), where those mislabeled samples are grouped in the same group, representing their own sub-class of samples.

[041] While, in implementations, the number of features used is relatively small when compared to known systems, not all features are equally important for distinguishing a certain malware family. Accordingly, this number can be perhaps be reduced while not affecting the accuracy of the classification algorithms. The reduction in the number of samples can be a crucial factor in reducing the cost of running the classification algorithm on large-scale datasets.

[042] In order to understand the relative importance of each feature, with respect to the (linear) classification algorithms, the recursive feature elimination (RFE) algorithm was run, which ranks all features from the most important to the least important feature.

[043] Given a set of weights of features, the RFE selects the set of features to prune recursively (from the least to the most important) until reaching the optimal number of features to achieve the best performance. In the linear classification algorithms, weights used for ranking features are the coefficients in the prediction model associated with each feature (variable).

[044] Table 5 shows the performance measures for the SVM using different numbers of features. It may be noted that, while the best performance is achieved at the largest number of features, indicating the importance of all features together, the improvement in the performance is very small, particularly for the SVM. The lowest 50 features in rank improve the accuracy, precision, and recall by less than 2%. However, this improvement is as high as 20% with decision trees (results not shown). To this end, and for the algorithm of choice (SVM), it can be noted that a minimal set of features can be used to achieve a high accuracy while maintaining efficiency for the studied malware family, Zeus.

Table 5: The accuracy measures versus the number of features used for classification (SVM with L2R and L2L).

<td> Features</td><td></td><td> A</td><td> P</td><td> R</td><td> F</td>

<td> 3</td><td></td><td> 65.3%</td><td> 66.9%</td><td> 60.5%</td><td> 63.6%</td>

<td> 6</td><td></td><td> 73.2%</td><td> 76.1%</td><td> 67.6%</td><td> 71.6%</td>

<td> 9</td><td></td><td> 89.6%</td><td> 87.6%</td><td> 92.3%</td><td> 89.9%</td>

<td> 15</td><td></td><td> 94.1%</td><td> 94.0%</td><td> 94.1%</td><td> 94.1%</td>

<td> 25</td><td></td><td> 94.4%</td><td> 94.9%</td><td> 93.9%</td><td> 94.4%</td>

<td> 35</td><td></td><td> 94.6%</td><td> 95.3%</td><td> 93.8%</td><td> 94.6%</td>

<td> 45</td><td></td><td> 94.9%</td><td> 95.6%</td><td> 94.0%</td><td> 94.8%</td>

<td> 65</td><td></td><td> 95.6%</td><td> 95.8%</td><td> 95.3%</td><td> 95.5%</td>

[045] In aspects, it is also possible to rank the different features by their highlevel category. The classifier logic was run on the file system, memory (where available), registry, and network features independently. For the network features, the connection type, IP and port, request/response type and size, and DNS were further ranked as sub-classes of features. From this measurement, it was found that while the file system features are the most important for classification for the studied malware family, Zeus—they collectively achieve more than 90% of precision and recall for classification, while the port features are the least important. It was not clear how would the memory feature rank for the entire population of samples, but using them where available, they provide competitive and comparable results to the file system features. Finally, the rest of the features were ranked as network request/response and size, DNS features, then registry features.

[046] It may be noted that the platforms and techniques herein do not only feature several algorithms, but also use several parameters for the same algorithm. For example, regularization and loss functions are widely used to improve estimating unknown variables in linear classification. For that, regularization imposes penalty for complexity and reduces over-fitting, while loss function penalizes incorrect classification. Widely used function types of parameters for linear classification are the L1 and L2 functions known in the art. In addition, since linear classification or programming problems can be stated as primal problems, they can also be converted to dual problems, which try to solve the primal problem by providing an upper bound on the optimal solution for the original (primal) problem. In the following the choice of the proper set of parameters—problem representation into primal or dual and the choice of regularization and loss functions— is examined as it affects classification by considering SVM and LR as two examples with a select set of parameters. The same dataset as above is used in this experiment as well. The results of this measurement are shown in Table 6.

Table 6: Selection of the support vector classifier with the best performing parameters. A, P, R, and F correspond to the accuracy, precision, recall, and Fscore, respectively.

<td> Algorithm</td><td></td><td> A</td><td> P</td><td> R</td><td> F</td>

<td> L1-reg. log. regression (/)</td><td></td><td> 93.7%</td><td> 93.7%</td><td> 93.7%</td><td> 93.7%</td>

<td> L2־reg. log. regression (p)</td><td></td><td> 92.3%</td><td> 91.4%</td><td> 93.4%</td><td> 92.4%</td>

<td> L2-reg. L2-I0SS SVM (d)</td><td></td><td> 95.6%</td><td> 95.8%</td><td> 95.3%</td><td> 95.5%</td>

<td> L2-reg. L2-I0SS SVM (p)</td><td></td><td> 89.1%</td><td> 84.5%</td><td> 95.7%</td><td> 89.7%</td>

<td> L2-reg. Ll-loss SVM (d)</td><td></td><td> 94.1%</td><td> 95.6%</td><td> 92.5%</td><td> 94.0%</td>

<td> L1־reg. L2-loss SVM (/)</td><td></td><td> 94.0%</td><td> 94.0%</td><td> 94.0%</td><td> 94.0%</td>

<td> L2-reg. log. regression (d)</td><td></td><td> 94.3%</td><td> 94.5%</td><td> 94.1%</td><td> 94.3%</td>

[047] It may be observed that while all algorithm perform reasonably well on all measures of performance (namely, above 90% for all measures, for most of them), and can be used as a building block for the inventive system, the L2-regularization L2־loss functions, when combined with the dual optimization representation, provides the highest performance with all accuracy measures above 95%. All algorithms do not use kernel methods, and execute efficiently, even on large datasets.

[048] In implementations, the system can leave the final decision of which algorithm to choose to the user, based on factors such as, for example, the 17 classification accuracy and cost (both run-time and memory consumption). The system can also provide the ability to tune algorithms by using a feature selector module 124 and/or parameter selector module 126. Once the user selects a desired algorithm, the system can learn the best set of parameters for that algorithm based on the training set, and can use a trained model to output one or more labels 130 for the unlabeled data. The one or more labels 130 serve as the output or results of classification operations by the system, although they can also be used to re-train classification algorithms for future runs.

[049] In terms of clustering rather than classification, using the same features extracted by the feature extractor 114 that are used for classification, the system can use unsupervised clustering algorithms to group malware samples into clusters. This can be accomplished by selecting one or more clustering algorithms from a set of clustering algorithms 134, which can be or include a set of hierarchal clustering algorithms, with variations and settings for clustering, cutting, and linkage, which settings can be by the user. Unlike classification operations, clustering operations can be unsupervised and do not require a training set to cluster the samples into appropriate clusters. The testing selector module 118 can run hierarchal clustering with several settings to present the user with preliminary cluster sizes, and number of clusters created using the different settings.

[050] In terms of clustering operations, platforms and techniques herein can employ several distance metrics within a feature spaced or other abstract space with which to group samples, like the Jaccard index, cosine similarity, Hamming distance, Euclidean distance, and correlation. On the other hand, options for linkage include average, complete (maximum), median, single (minimum), ward, and centroid, among others.

[051] Using various options noted above, the system gives the choice to the user to pick the best clustering size based on the user’s data and, when available, the performance measures. Multiple cut thresholds are calculated for each distance and method to give an overview of how each clustering method performed. The user then makes a judgment to choose the most relevant results to fit the data to. To evaluate the performance of the clustering, a known set such as the manually labeled Zeus family can be used. In those regards, further tags can be used in the manual labeling that divide the Zeus family into multiple sub-families (also called Zeus variations), and consider that as a reference (or ground-truth) clustering. To add variety of families to the problem, and challenging the clustering algorithms, an equal number of samples can be picked, for instance from the families shown in Table 3. However, this time, selection can be limited to samples for which a correct label is known. Manually labeled malware data set was run against the clustering algorithms, evaluating the performance using the precision and recall defined herein. Table 8 shows the precision, recall, and cutting threshold for several distance metrics. First of all, it may be noticed that one can achieve high performance using easy-to-set parameters. While one can brute-force the cutting parameter range to achieve the highest possible accuracy, this option might not be always available with partially labeled data. Second, and more notably, it may be noticed that the achieved precision and recall outperform the classification algorithms noted above. This in part is attributed to the power of the clustering algorithm in distinguishing subclasses into distinct clusters, whereas subclasses in the binary classification that are close to each other in the feature vector space are grouped erroneously with each other. To this end, in implementations, one may use the results of the clustering to guide the binary classification, and to reduce its false alarms, thus improving its performance.

Table 8: Clustering Precision and recall for several linkage and cutting criteria and parameter valuses.

<td></td><td></td><td> Linkage</td><td colspan="2"> Cutting</td><td> Precision</td><td> Recall</td>

<td> Correlation</td><td></td><td> Average ϋ Complete Median Single Ward</td><td> 0.40 0.70 0.25 0.40 0.25</td><td></td><td> 93.4% ii 89.7% 89.6% 90.2% 93.5%</td><td> 100% iiH ϊδό% 96.6% 100% 98.2%</td>

<td> Cosine</td><td></td><td> Average Centroid Complete Median Single M</td><td> 0.25 0.25 0.40 0.25 0.40 ®s</td><td></td><td> 84.1% 84.6% 85.5% 94.4% 91.2% MIK</td><td> 100% 100% 97.1% 95.2% 100% ill</td>

<td> Hamming</td><td></td><td> Average Centroid Complete isiai Single Ward</td><td> 0.25 0.25 0.25 0:25 0.25 0.25</td><td></td><td> 98.9% 98.5% 98.7% it® 98.3% 99.3%</td><td> 97.6% 100% 97.5% Ml 98.8% 97.6%</td>

<td> Jaccard</td><td></td><td> Centroid Complete Median Single Ward</td><td> 05 0.25 0.25 0.40 0.25 0.40</td><td></td><td> SiiB 99.9% 99.9% 99.9% 99.9% 99.9%</td><td> iisa 100% 100% 99.8% 100% 100%</td>

[052] Based on the preliminary results of one or more clustering results, the user can pick which setting fits the data set provided, and can proceed to downstream labeling and verification processes. While the clustering capability of the system is not primarily intended for labeling malware samples, but rather for grouping different samples that are similar in their behavior, the system can be configured with logic required for malware labeling. In such implementations, after the clustering algorithm runs, one of the following options can be used to label the data.

[053] Specifically, first, using analyst-vetted samples the resulting clusters can be augmented with labels, and the labels on unlabeled samples falling with the same clusters can be extrapolated to the same label. Second, for those clusters that do not have significant analyst-vetted data, other signatures or artifacts such memory signatures, where available, and further manual inspections can be used to generate candidate labels. Finally, in cases where none of the two preceding options above are available, a census of labels of antivirus scans for clusters’ members can be used to identify appropriate label choices. Thus, the system including components shown in FIG. 2 can carry out classification operations, clustering operations, and/or both types of operations.

[054] In both classification and clustering operations noted, various metrics can be used to determine a degree of accuracy or other performance. To evaluate the different algorithms in each application group, several accuracy measures can be used to highlight the performance of various algorithms. Considering a class of interest, S, the true positive (t_p) for classification is defined as all samples in S that are labeled correctly, while the true negative (t_n) is all samples that are correctly rejected. The false positive (f_p) is defined as all samples that are labeled in S while they are not, whereas the false negative (f_n) is all samples that are rejected while they belong to S. For validating the performance of the classifiers, precision can be defined as P = t_p/(t_p + f_p), the recall as R = t_p/(t_p + f_n), the accuracy as A = (t_p + t_n)/ (f_p + f_n + t_p + t_n), and the F-score defined as F = 2(p x ή/(ρ + r).

[055] For clustering, definitions of accuracy, precision, and recall can be used as known in the art. In short, the precision measures the ability of the clustering algorithm to distinguish between different samples and associate them to different clusters, whereas the recall measures how well the clustering algorithm assigns samples of the same type to the same cluster. To that end, given a reference (ground truth) clustering setting T = {T] for 0 &lt; i &lt; n_t and a set of learned clusters L = {Q for 0 &lt; i &lt; n!, the precision for the j - th learned cluster is computed as Pj = max {\Lj Π 7ך|} for 0 &lt; i &lt; n! while the recall for the j-th reference cluster is computed as Rj = max {\L! Π Tj} for 0 &lt; i &lt; n!. The total precision and recall of the algorithm are computed as«t and L respectively.

[056] FIG. 3 illustrates a flowchart of classification, clustering, and other processing that can be performed in systems and methods for behavior-based automated malware analysis and classification, according to aspects. In 302, processing can begin. In 304, the controller 110 can/or other logic can extract one or more artifacts form the set of samples 102, and generate or update the artifact database 112 with that data. In 306, the controller 110, feature extractor 114, and/or other logic can access the set of artifacts 150 in or from the artifact database 112. In 306, the feature extractor 114 and/or other logic can perform feature extraction on the set of artifacts 150, for instance, reading out artifacts to define various desired features of the samples. In 310, the labeling module, testing selector module 118, and/or training selector module 120 can be applied to the features extracted from the set of artifacts 150.

[057] In 312, an administrator and/or other user can provide inputs to the algorithm selector module 122 using the feature selector module 124, the parameter selection module 126, and/or other input or interface. In 314, the administrator and/or other user can select a classification and/or clustering algorithm, for instance, an algorithm from the set of classifier algorithms 132 and/or set of clustering algorithms 134. In 316, the system can generate and/or identify a malware classification family and/or sample cluster(s), using the selected algorithm. In the case of a classification action, the system can output one or more labels 130 to associate with a given sample. In 318, the system can store classification, cluster, and/or related data, as appropriate. In 320, processing can repeat, return to a prior processing point, jump to a further processing point, or end.

[058] FIG. 4 illustrates various hardware, software, and other resources that can be used in implementations of behavior-based automated malware analysis and classification, according to embodiments. In embodiments as shown, the extractor 106 can comprise a platform including processor 138 communicating with memory 136, such as electronic random access memory, operating under control of or in conjunction with an operating system 142. The processor 138 in embodiments can be incorporated in one or more servers, clusters, and/or other computers or hardware resources, and/or can be implemented using cloud-based resources. The operating system 142 can be, for example, a distribution of the Linux™ operating system, the Unix™ operating system, the Windows™ family of operating systems, or other open-source or proprietary operating system or platform. The processor 138 can communicate with the artifact database 112, such as a database stored on a local hard drive or drive array, to access or store the set of samples 102, the set of artifacts 150, and/or subsets of selections thereof, along with other content, media, or other data. The processor 138 can in addition further communicate with a network interface 140, such as an Ethernet or wireless data connection, which in turn communicates with the one or more networks 144, such as the Internet or other public or private networks. The processor 138 can, in general, be programmed or configured to execute control logic and to control various processing operations, including to generate artifacts, features, classifications, clusters, and/or other information. In aspects, the controller 110, algorithm selector 122, and/or other components or elements can be or include resources similar to those of the extractor 106, and/or can include additional or different hardware, software, and/or other resources. Other configurations of the extractor 106, controller 110, and algorithm selector 122, associated network connections, and other hardware, software, and service resources are possible.

[059] The foregoing description is illustrative, and variations in configuration and implementation may occur to persons skilled in the art. For example, while embodiments have been described in which artifacts and other data are stored in a single artifact database 112, in implementations, that data or other data can be stored in multiple data stores, including cloud-based data stores. Similarly, while embodiments have been described in which the extractor 106 can comprise a single system or element which accesses the set of samples 102, in implementations, the extractor 106 can be or include multiple or distributed processors, servers, services, and/or other resources. Other resources described as singular or integrated can in embodiments be plural or distributed, and resources described as multiple or distributed can in embodiments be combined. The scope of the present teachings is accordingly intended to be limited only by the following claims.