Method, a computer program and apparatus for analyzing symbols in a computer

RELATED APPLICATIONS

0001This application a U.S. National Stage Application of PCT/GB2010/050812 filed on May 18, 2010, that claims priority to U.S. provisional patent application No. 61/219,520 filed on Jun. 23, 2009, the contents which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

0002The present invention relates to a method, a computer program and apparatus for analyzing symbols in a computer.

0003There are many examples of computer systems in which it is useful to be able to analyze symbols passing through or stored in the computer system. As will be appreciated from the following, the term “symbols” in this context is to be construed broadly. In general, the term “symbols” is used herein in the broad sense, for example, including computer messages, which term is also to be construed broadly and includes for example computer messages in a computer language (including computer instructions, such as executable programs), natural languages in computer-readable form (such as in documents, emails, etc.). “Symbols” also includes computer data in the conventional sense, i.e., typically, abstractions of real world artifacts, etc.

0004In one example of computer systems in which it is useful to be able to analyze symbols passing through or stored in the computer system, third parties can attempt to take control of a computer by “attacking” the computer system. Such “attacks” can be carried out by exploiting the well-known buffer overflow vulnerabilities of some computer systems. In another example, hacking can take place by the third party sending commands to the computer system in which the commands are correctly structured in the context of the language of the computer system, but which are intended to cause the computer system to perform undesirable actions including to return an error message that can be used by the third party for reconnaissance or to return inappropriate information to a third party or to gain illegal access to the computer system. Attacks of this type on SQL databases are well known and yet are difficult to defend against. SQL databases are widely used, and are used for example by e-commerce and many other websites to hold user data (such as login name and password, address and credit card details, etc.).

0005In another example, it may be desirable to monitor computer symbols or messages to ensure that the computer system is being used properly and that for example it is not being used inappropriately. For example, in an organization, a user may be using a computer system inappropriately, for example by using the system for purposes for which the user is not authorised, and yet which is not intended by the user to be an “attack” on the computer system as such.

0006In our co-pending US and European patent applications entitled “A method, A Computer Program and Apparatus for Analyzing Symbols in a Computer” having application numbers U.S. Ser. No. 11/672,253 and EP-A-1,830,253, respectively, there is described and disclosed a method for analyzing symbols in a computer system. The method and apparatus, referred to herein as “Efficient Grammatical Clustering” (“EGC”), described in the aforementioned patent applications, provides a means to understand usage patterns based on messages entering (or leaving) computer systems. For example, EGC provides a means of being able to recognise messages that are the different database commands entering a relational database system. This enables a baseline of normal behavior to be determined. EGC enables all new messages, i.e. messages that have not been seen by the system previously, to be recognised so that a proactive device can determine whether a new message (command) should be allowed to pass to the database or not.

0007The EGC patent applications, the entire contents of which are hereby incorporated by reference, relate to a method by which a unique execution path of any instrumented computer program can be determined. Such an execution path provides the basis for generalization of execution paths into semantically similar clusters. Each execution path can be assigned a unique cluster identifier.

0008In one embodiment of the EGC patent applications, grammatical clustering of messages which are sentences within a computer language is performed. In this case, the computer program is a parser and the execution path is generated by the operation of the parser on some input sentence messages (in the appropriate computer language). In the EGC method, parsing of the incoming messages is thus extremely important as it is the parsing of the messages that enables the respective execution paths to be determined from which the unique cluster identifier can be assigned to the message.

0009Referring to <figref idref="DRAWINGS">FIGS. 1 and 2</figref>, the EGC method is briefly described. There is shown in <figref idref="DRAWINGS">FIG. 1</figref> an example of a computer system <b>106</b> connected to a network <b>105</b>.

0010The computer system <b>106</b> has a computer resource <b>103</b> which might be, for example, an SQL database. The computer system <b>106</b> makes its computer resource <b>103</b> available to applications <b>102</b> interacting directly or across the computer network <b>105</b> to support one or more users <b>101</b>. The interaction with the computer resource <b>103</b> is mediated through a computer language via the transmission of Messages MSG <b>104</b> within the message language. A process <b>202</b> is provided operating within or on the computer system <b>106</b> to observe messages. The message and the intent of the message can be determined via another process <b>201</b> explained below.

0011Typically, the messages MSG <b>104</b> might be used to specify the desired operational behavior of components in the computer system <b>106</b>. Thus, messages are used between components within the computer system, and messages are used by users to gain access to the computer system <b>106</b>. Computer languages are used to facilitate the use of messages in the computer system. Each computer language is defined by a grammar so that messages conform to a known syntax. The grammar of such languages is published so that software developers can ensure that the messages of the software conform to the correct syntax.

0012The grammar of the computer language of the messages that are to be analyzed is defined, e.g. using first order logic. This may be carried out in a manner that is known per se. For example, the programming language Prolog can be used to describe the grammar of the language as a set of first order logic. This logic is then applied initially to a set of training examples of messages. Such messages are defined so as to be correct syntactically in the context of the language and appropriate in the sense that they are messages that are deemed to be acceptable in the context of usage of the system around which the messages pass. The logic contains clauses. When the logic is applied to the messages, the identity of the clauses along a successful path through the logic is noted.

0013In this way, paths of acceptable messages being parsed via the logic are obtained. These paths can then be grouped according to similarity. In turn, the messages that follow the respective paths can be grouped according to similarity in this sense, so that patterns of similar messages can be discerned. This means that new messages, which are different from messages used in the training, can then be allocated to patterns of messages that are known to be acceptable, or rejected.

0014The EGC method works by analyzing symbols into patterns, so that new symbols can be analyzed more efficiently than in other known techniques. This enables the EGC methodology to be implemented in real-time with relatively little computational overhead. In one example, the method is carried out on new symbols to determine whether the new symbols fit a pattern of data that is known or constitute a new pattern. Patterns may also be referred to as “clusters” as they represent a cluster of similar paths through the computer logic. In practice, if the new symbols fit a pattern that is known, then a decision will already have been made as to whether symbols fitting that known pattern are to be deemed acceptable or not. If the symbols constitute a new pattern, in practice a decision will have been made what to do with symbols that constitute a new pattern, such as “always deem not acceptable” or “send error report”, etc.

0015The EGC system and method is not concerned with generating new rules for new messages. Instead, it is concerned with determining patterns for computer messages. In one embodiment, the patterns that are obtained can then be considered, for example “manually” by a human user, to determine whether a computer system has been compromised.

0016Referring to <figref idref="DRAWINGS">FIG. 2</figref>, there is shown a simplified schematic flow chart for the process <b>201</b> by which messages are classified or clustered, using the EGC method, in dependence on semantic intent of the messages. Messages MSG <b>104</b> received by a computer are clustered using the EGC process <b>401</b> which produces a classification MSG CLASSIFICATION <b>402</b> of the message. The message classifications are stored, along with a copy of the respective messages, in a message store MSG STORE <b>403</b>. As well as the message, other attributes about the message can be included in the message store. For example, these attributes could include, amongst others; the date &amp; time the message was received; the username or application name that sent the message; network addressing information about the source and destination of the message; etc.

0017The EGC system works well. In particular, by analyzing the symbols into patterns, new symbols can be analyzed more efficiently than in previous known techniques, which makes it possible to implement the method in real-time with relatively little computational overhead. However, although the EGC system does work well, a full parse of each message is needed, which can be computationally intensive. Indeed, the process typically involves <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0018">1) lexical analysis of a received message, in which the message is tokenised;</li><li id="ul0002-0002" num="0019">2) parsing of the tokenised message through a grammar, e.g. an instrumented grammar;</li><li id="ul0002-0003" num="0020">3) extracting the summarised execution path and finally;</li><li id="ul0002-0004" num="0021">4) mapping the extracted summarised execution path to a unique cluster identifier.</li></ul></li></ul>

0022With reference to <figref idref="DRAWINGS">FIG. 2</figref>, for each message MSG <b>104</b>, the Clustering Process <b>401</b> provides a unique classification MSG CLASSIFICATION <b>402</b> of the semantic intent of the message. This uniqueness allows syntactically different messages to be classified in the same way because their class of semantic intent is identical.

0023In the context of a computer resource that is a relational database, the messages are received at the computer resource in the language of Structured Query Language (SQL). As examples, the unique message classification <b>402</b> for 7 specific messages is shown in <figref idref="DRAWINGS">FIG. 3</figref>. Performing a full parse of each message MSG <b>104</b> through the instrumented grammar so as to determine the semantic intent of the message can be extremely computationally intensive. Therefore, as data rates and volumes of processed traffic and data increase, a method to reduce the computational intensity whilst still providing the required or desired performance levels and accurate determining of semantic intent is sought.

0024According to a first aspect of the present invention, there is provided a computer-implemented method of analyzing symbols in a computer system, the symbols conforming to a specification for the symbols, in which the specification has been codified into a set of computer-readable rules; and, the symbols analyzed using the computer-readable rules to obtains patterns of the symbols by determining the path that is taken by the symbols through the rules that successfully terminates, and grouping the symbols according to said paths, The computer-implemented method comprising;

0025upon receipt of a message at a computer, performing a lexical analysis of the message; and,

0026in dependence on lexical analysis of the message assigning the message to one of the groups identified according to said paths.

0027The invention provides a method by which the repeated full execution of a parser is rendered unnecessary and replaced by a more efficient process that determines the appropriate cluster identifier to associate with a message. A lexical analysis is performed on a received message and, in dependence on this, the message may be successfully allocated to an appropriate cluster. Thus, an efficient and quick method is provided by which a message may be allocated to the appropriate message cluster. As compared to the basic EGC method quicker and more efficient message allocation may be achieved since it is in on dependence on the lexical analysis that the assignment is determined.

0028In an embodiment, in the step of performing a lexical analysis of the message a sequence of tokens is generated corresponding to the message. It is preferred that in dependence on the sequence of tokens, a message digest is assigned to the message, the message digest corresponding to the said one of the groups.

0029Preferably, the tokens are tokens that are directly related to some of the tokens that would have been used in a full parse of the message. Preferably, the token sequence is a syntactic sequence, thereby enabling semantic grouping of the messages based on message syntax.

0030Thus, the tokens are not the full language based tokens that would normally be generated by a tokenizer. “Selective Message Digest” (SMD) of an SMD token sequence produced at the lexical analysis stage may be used to allocate new messages to clusters. Thus, a full parse of a received message is not required for every message and so the process is quicker and more efficient than the EGC method described above.

0031An embodiment of the invention provides a method of going directly from a lexical analysis phase to the cluster identifier without the computational complexity of a full parse of the message. In the case where the full parse has already been performed and the cluster identifier determined, then any repeat parse which would have generated such a cluster identifier can be determined solely by the SMD value generated at the lexical analysis stage. Embodiments of the invention provide a simple and robust method by which the beneficial effects of the EGC patent applications, discussed above, can be achieved in a significantly more computationally efficient and quick manner.

0032In embodiments, the invention provides a method whereby the repeated full execution of a parser is rendered unnecessary for previously processed message types and replaced by a much more efficient process that determines the appropriate cluster identifier to associate with a message by the “Selective Message Digest” (SMD) of the SMD token sequence produced at the lexical analysis stage.

0033Preferably, the token sequence is a syntactic sequence, thereby enabling semantic grouping of the messages based on message syntax. This enables a subsequent syntactic token sequence that has the same semantic grouping to be quickly identified.

0034Preferably, a message digest is calculated for a token sequence using a method selected from the group consisting of shift and rotate on the entire token sequence, SHA family or Message-Digest algorithm 5 (MD5) algorithms on the entire token sequence and an interleaved Message Digest method integrated into the tokenization process. Thus, commonly available and reliable algorithms may be used to generate the message digest.

0035In one example, not all tokens of a standard input language are used to create the message identifier. This provides the advantage that it is possible to group messages that are syntactically different into the same message cluster.

0036Preferably, the computer system includes a computer resource and the messages are directed to the computer resource. The method enables a determination to be made as to whether or not usage of the computer resource is changing at a semantic level.

0037In some cases, the messages directed to the computer system are attempts to inappropriately utilise the computer resource. In such cases, the method enables a system administrator or manager effectively to utilise detection to stop inappropriate use of the resource.

0038In a preferred embodiment, the computer system includes a computer resource and the messages are sequences of machine instructions that are about to run through a micro-processor within the computer system. Thus, the method can be used to detect buffer overflow exploits.

0039In one example, the computer resource is a relational database and messages are submitted in a language such as Structured Query Language. The method can therefore be used to detect and monitor inappropriate access and database attack techniques such as SQL injection.

0040Preferably, the process is performed progressively in that tokens are formed progressively as the message is received. This enables effective use of RAM to be made and enables the method to operate with messages that are non-grammatical structures.

0041In an embodiment, upon receipt of a message a check is made as to whether or not there already exists an identified group for messages with the token sequence of the received message and, if there is, assigning the message to the identified group. In such cases it is possible or probable that actions will have been identified for performance upon recognition of a message belonging to a particular cluster. Accordingly, the method enables the appropriate action to be identified and triggered quickly and efficiently.

0042In one embodiment, if it is ascertained that there is not already an existing identified group for a received message a full parse of the message is performed and a group is established for the message and for subsequent messages having the same token sequence. Thus, a full parse may be done only when it is necessary, thus increasing efficiency as unnecessary parsing may be avoided.

0043Preferably, the method comprises generating, e.g. automatically, statistical data in respect of the groups to which messages belong. The generation of such statistical data is useful since it can be used for applications such as accounting, charging, monitoring performance and detecting inefficiencies and the like.

0044According to a second aspect of the present invention, there is provided a computer program arranged such that when run on a computer it causes the computer to perform the method of the first aspect of the present invention.

0045According to a third aspect of the present invention, there is provided a computer programmed to carry out a method according to the first aspect of the present invention.

0046According to a fourth aspect of the present invention, there is provided a computer-implemented method of analyzing symbols in a computer system, the symbols conforming to a specification for the symbols, in which the specification has been codified into a set of computer-readable rules; and, the symbols analyzed using the computer-readable rules to obtains patterns of the symbols by determining the path that is taken by the symbols through the rules that successfully terminates, and grouping the symbols according to said paths, the method comprising; upon receipt of a message at a computer, performing a lexical analysis of the message to generate one or more tokens corresponding to the message; and, in dependence on the sequence of tokens assigning the message to one of the groups identified according to said paths.

0047From a mathematical perspective, selective message digest (SMD), in which a message digest is performed based on a sequence of tokens in which it is the form of the tokens that is considered, and EGC as described above are hierarchical abstractions between the raw message, i.e. the textual sequence of characters and to the actual semantics or meaning of the message (in the context of its operating environment).

0048For example, starting from the sequence of characters and moving in an order in which each step represents a generalization, the following stages might be included:

00491) A {sequence of characters}

00502) A {sequence of tokens}

00513) A {sequence of SMD tokens}

00524) A {SMD Value}

00535) One or more {SMD value} maps to a {Cluster ID}

0054At each level of generalization in the hierarchy specific detail is lost, whilst managing to track the “essence”, “intent”, or “motive” of the original message. This “essence” can be recognised in another non-identical message. The computer or computer program can thus easily be trained to respond to this other non-identical message in the same way that has been specified for the previously observed message. SMD is a much finer-detail method of tracking message “essence” than that provided by EGC.

0055For the SMD calculation it is only necessary that the tokenization succeeds. Even if the message is not valid (with respect to the language) an SMD value will still be generated. This property is particularly useful in the case in which the EGC grammar is deficient and the message is valid but not recognised (by parsing) by the grammar and in the case in which the message is not valid and correctly determined as such, but a (buggy) system sending the invalid message continues to do so and an action needs to be taken for each of these invalid messages. One typical action might be blocking the message arriving at the resource. In the context of an SQL Database, if the database receives the message it will attempt to process the message, consuming resources needlessly, eventually failing and returning an error to the sender. Another typical action might be translating the invalid message into a valid message. Both cases require the next invalid message to be determined to be of a form that has already been observed so that the appropriate action can be taken. SMD can be utilized to do just this.

0056Accordingly, in a fifth aspect, the invention provides a computer-implemented method of analyzing symbols in a computer, the method comprising;

0057upon receipt of a message at a computer, performing a lexical analysis of the message to determine tokens for the message;

0058selecting a form of the tokens to generate a sequence of generalised tokens representative of the message. A computer program is also provided, the computer program being arranged such that when run on a computer it causes the computer to perform the method of the fifth aspect of the present invention.

0059A method is provided that provides for a simple and robust way of processing messages when tokenization has succeeded irrespective of whether or not the message is valid.

0060Preferably, a message digest is assigned to the sequence of generalised tokens. Thus, a simple and robust method is provided by which a selectively tokenised message can be assigned a means of identification and/or qualification.

0061Thus, the present method provides that even if a message is not valid (with respect to the language) an SMD value will still be generated. As mentioned above, this property is particularly useful in the case in which the EGC grammar is deficient and the message is valid but not recognised (by parsing) by the grammar and in the case in which the message is not valid and correctly determined as such, but a (buggy) system sending the invalid message continues to do so and an action needs to be taken for each of these invalid messages.

BRIEF DESCRIPTION OF THE DRAWINGS

0062Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

0063<figref idref="DRAWINGS">FIG. 1</figref> shows a schematic representation of an example of a networked computer system arranged and configured to perform Efficient Grammatical Clustering;

0064<figref idref="DRAWINGS">FIG. 2</figref> shows a schematic representation of some steps of Efficient Grammatical Clustering as might be used in the computer system of <figref idref="DRAWINGS">FIG. 1</figref>;

0065<figref idref="DRAWINGS">FIG. 3</figref> shows a table of an example of messages that might be used in classification using Efficient Grammatical Clustering;

0066<figref idref="DRAWINGS">FIG. 4</figref> shows a schematic representation of the steps and/or components in the process or system for selective message digest;

0067<figref idref="DRAWINGS">FIG. 5</figref> shows a schematic representation of an example of steps in an Efficient Grammatical Clustering process;

0068<figref idref="DRAWINGS">FIG. 6</figref> shows a schematic representation of an example of steps in a Selective Message Digest process;

0069<figref idref="DRAWINGS">FIG. 7</figref> shows an example of a table of Selective Message Token sequences; and

0070<figref idref="DRAWINGS">FIG. 8</figref> shows an example of an interleaved Selective Message Digest process.

DETAILED DESCRIPTION OF THE INVENTION

0071Using the present method, initially a raw data stream is received at a computer from a source, e.g. a network, to which the computer is connected. A tokenizer, described in detail below, operates on the data to produce Selective Message Digest (SMD) tokens corresponding to the data stream. Once the SMD tokens have been identified, if the sequence of tokens has been seen before by the computer as determined by their SMD value, then the sequence of tokens is allocated an existing message digest. If the sequence of tokens has not been seen before by the computer then a full parse of the message is performed. Based on the assigned selective message digest the message can then be grouped or assigned to an appropriate cluster as would previously have happened in the EGC method after a full parse of the message had been performed.

0072Thus, the message digest may be allocated based on the sequence of tokens without a full and computationally intensive parse of the message being performed. Using the present method, there is no need to determine the execution path of the message through the program logic in order to assign to it a cluster identifier. Rather, the cluster identifier may be assigned based on the SMD token sequence and an associated selective message digest SMD value.

0073<figref idref="DRAWINGS">FIG. 4</figref> shows a schematic representation of the steps and/or components in the process or system for Selective Message Digest within an EGC method. A method is provided by which the full execution of a parser is rendered unnecessary and replaced by a more efficient process that determines the appropriate cluster identifier to associate with a message. A “Selective Message Digest” (SMD) of the SMD token sequence is produced at the lexical analysis stage, i.e. at the stage at which the received character stream is split into meaningful symbols defined by a grammar. The message digest is said to be “selective” since the actual value of any token within a sequence of tokens is not significant in allocating a message digest to a message. Rather, the syntax of the token sequence is used as the basis for the determination. Properties of the tokens are selected (from none to all) and it is the tokens, with the properties selected, on which the message digest is performed. This is explained in more detail below.

0074The invention thus provides a method and apparatus by which a semantic cluster ID for a message in a given language can be determined without recourse to execute a full parse of the message. Thus, the benefits of the EGC methods described above are achieved with a reduction in the computational intensity.

0075The present method, as will now be described with reference to <figref idref="DRAWINGS">FIG. 4</figref>, can replace the process described above with reference to <figref idref="DRAWINGS">FIG. 2</figref>. Referring to <figref idref="DRAWINGS">FIG. 4</figref>, a sequence of messages MSG <b>104</b> is received at a computer or computer system. The computer system includes an SMD process <b>601</b> whose function is to determine the SMD values for each of the messages MSG <b>104</b>. An SMD cache <b>602</b> is provided. The determined SMD value MSG SMD <b>603</b> for each message is stored in the SMD CACHE <b>602</b>. If a corresponding message classification MSG CLASSIFICATION <b>402</b> is already stored in the SMD CACHE <b>602</b> then the classification for this message may be determined. If the message classification MSG CLASSIFICATION <b>402</b> is not already stored in the SMD CACHE <b>602</b> then the value MSG SMD <b>603</b> is stored and the message classification is determined using the EGC Clustering Process <b>401</b> described above. In other words a parse of the message is performed and the execution path of the message through the program logic is determined. This produces the classification of the message MSG CLASSIFICATION <b>402</b> which is stored along with a copy of the message in the message store MSG STORE <b>403</b>.

0076As well as the message, other attributes about the message, i.e. metadata in respect of the message, can be included in the message store MSG STORE <b>403</b>. For example such metadata might include, the date &amp; time the message was received, the username or application name that sent the message, network addressing information about the source and destination of the message and so on. Each time a new message is received, the SMD CACHE <b>602</b> is updated so that it contains a record of the MSG SMD <b>603</b> and the corresponding MSG CLASSIFICATION <b>402</b> which has been determined by the CLUSTERING PROCESS <b>401</b>. Thus, there are two routes by which a MSG CLASSIFICATION <b>402</b> can be determined. Either the classification is determined based on a Selective Message Digest <b>603</b>, or if the message digest is not recognised then a full clustering process <b>401</b> according to the EGC method may be performed. Once a full clustering process <b>401</b> has been performed and a new cluster formed, this new cluster will be assigned its own message digest so that subsequent messages belonging to the new cluster will not have to be processed using the full EGC method but rather can be grouped based on a simple lexical analysis as described above.

0077<figref idref="DRAWINGS">FIG. 5</figref> shows a schematic representation of an example of an embodiment of a grammatical clustering process such as might have been used in the EGC patent applications referred to above. In the example of <figref idref="DRAWINGS">FIG. 5</figref>, the messages are in the form of Structured Query Language (SQL) commands destined for a database resource (not shown). A message MSG <b>104</b> is extracted from the input (stream) and delivered to the CLUSTERING PROCESS <b>401</b>. A Language Aware Tokenizer <b>501</b> produces TOKENS <b>502</b> from the message MSG <b>104</b>. The TOKENS <b>502</b> are then parsed by an Instrumented Language Parser <b>503</b>, which determines the Execution Path <b>504</b> on which the MSG CLASSIFICATION <b>402</b> can be determined. The parsing process is often a significant computer program containing logic to determine whether the sequence of tokens is grammatically correct with respect to the language. The search space of such a process is very large and can have high computational complexity.

0078The Instrumented Language Parser <b>503</b> is an embodiment of the EGC system in which the instrumented computer program is a parser. The instrumentation is points in the original program source code which provide records each time they are successfully executed. The information recorded varies and the different forms include: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0079">(a) no information (some functions are not deemed sufficiently important);</li><li id="ul0004-0002" num="0080">(b) simply an index of the function or computer program branch that was executed; or</li><li id="ul0004-0003" num="0081">(c) information about what data and what processing outcomes occurred when the function was executed.</li></ul></li></ul>

0082This process is analogous to setting debug “spy points” and running a computer program in debugging mode, whilst collecting the sequence of spy points that were triggered in the execution. An example of a sequence of recorded instrumentation information (analogous to the sequence of spy points) is shown in <figref idref="DRAWINGS">FIG. 5</figref> and is the “path” [keyword(select), <b>611</b>, <b>643</b>, <b>642</b>, <b>641</b>, <b>640</b>, <b>639</b>, <b>638</b>, <b>618</b>, function(count), <b>645</b>, keyword(from), table(tbl_log), cluster set(manipulation)]. This is to be understood as during the processing of the successful parse of the SQL message “SELECT COUNT (*) FROM tbl_Log t29” the instrumented SQL Language parser was measured first processing “keyword(select)” (a type (c) form of instrumentation), then successfully executing the sequence of parsing functions denoted by indexes <b>611</b>, <b>643</b>, <b>642</b>, <b>641</b>, <b>640</b>, <b>639</b>, <b>638</b>, <b>618</b>—each being a type (b) form of instrumentation. The next instrumented function to be recorded was “function(count)” followed by parsing function denoted by index <b>645</b>, then the two records “keyword(from)” and “table(tbl_log)”. The final recorded instrumentation is “cluster set(manipulation)”, which provides information about the overall type of message that was parsed. Many other functions will have been executed in the successful parse of the message but as they are type (a) forms of instrumentation they are not recorded. Furthermore, there are a large number of functions within the instrumented language parser that were not used in any manner in the successful parse of the SQL message. For a more detailed description of the Efficient Grammatical Clustering process refer to our co-pending US and European patent applications entitled “A method, A Computer Program and Apparatus for Analyzing Symbols in a Computer” having application numbers U.S. Ser. No. 11/672,253 and EP-A-1,830,253, respectively.

0083An example of the SMD PROCESS <b>601</b> is shown in <figref idref="DRAWINGS">FIG. 6</figref>. In this example, the messages are again in the form of SQL commands destined for a database resource. As in the example of <figref idref="DRAWINGS">FIG. 5</figref>, the message MSG <b>104</b> is extracted from the input (stream). In this case though, instead of being delivered to the language aware tokenizer and parser of <figref idref="DRAWINGS">FIG. 5</figref>, the message is delivered to the SMD PROCESS <b>601</b>. The SMD PROCESS <b>601</b> includes an SMD Tokenizer <b>604</b> and a DIGEST PROCESS <b>701</b>.

0084When a message is received, a lexical analysis is done by the SMD Tokenizer <b>604</b> so as to produce SMD TOKENS <b>605</b> from the message. The SMD TOKENS <b>605</b> are then sent to the DIGEST PROCESS <b>701</b> which serves to determine the message SMD (MSG SMD <b>606</b>). The MSG SMD <b>606</b> is then checked against stored MSG CLASSIFICATIONs <b>402</b> determined from a prior CLUSTERING PROCESS <b>401</b>. Thus, the cluster identification to which a particular message belongs, may be performed based on SMD tokens generated by lexical analysis without a full parse of the message having been performed.

0085In contrast to the system and method described with respect to <figref idref="DRAWINGS">FIG. 5</figref>, in this case, instead of an execution path (through program logic) being determined by an Instrumented Language Parser <b>503</b> leading to the MSG classification <b>402</b>, this is achieved by the use of a Selective Message Digest based on the SMD tokens <b>605</b>. There is therefore, in many cases no need to determine the execution path of the message through the logic of the program, for which a full parse would be necessary, but rather the cluster to which the message belongs can be determined based only on the input SMD token stream.

0086As in the EGC method, in an embodiment, the method is initially carried out on training examples of symbols. This allows a base set of patterns of symbols to be built up and message digests to be assigned to sets or clusters of the patterns. These can be analyzed by a human domain expert who can determine which patterns relate to acceptable or normal behavior, so that new symbols can be classified accordingly. In principle, the training examples may be examples of symbols that are known to be acceptable thereby to obtain patterns of symbols that are known to be acceptable. However, more likely in practice is that the training examples will be general and a decision will be made later, after the patterns have been produced and based on the patterns, as to which patterns are to be deemed acceptable or not.

0087To determine a Selective Message Digest process that maps a message SMD <b>606</b> to a message classification <b>402</b> requires that the appropriate tokens are determined for inclusion in the SMD Token sequence <b>605</b>. Not all tokens of the standard input language are required to create a MSG SMD <b>606</b> that maps directly to a MSG CLASSIFICATION <b>402</b>. In the present method, what is important is that appropriate token forms are utilised in the SMD Tokenization process when determining or generating an SMD Token sequence <b>605</b>. Appropriate tokens are selected on which the SMD is performed that allows a direct mapping to the cluster ID produced by the EGC process described above.

0088Token forms that are related to the part in the instrumented parse provided by the EGC process are considered in the SMD Token sequence, which can then be subsequently used to develop the message SMD. Examples of participating token forms include identifiers, operators, some punctuation and value types (but not values themselves). For example a ‘number’ may be used as a token but not its value or a ‘string’ of characters may be used but not the actual spelling. Thus, no full parse of the statement is required as it is not relevant what the actual spelling or number is. What matters is the token form. Values within the statement can be ignored at this stage and are not needed to identify the cluster to which a message belongs. Previously, in the EGC method after the language aware tokenizer had acted to identify tokens within the message, an instrumented parser might typically have been utilised so as to fully parse the message and then allocate it to an appropriate cluster. In the present system, a Selective Message Digest is produced based on the token type and not the values themselves.

0089There may well be cases in which the generated SMD token sequence is not recognised and so on occasion full instrumented parse of the message will be done. However, in most cases this will not be necessary and so significant savings in terms of computing and processing time may be achieved. As compared to the basic EGC method, the present method operates significantly faster. For example, it might operate up to 10 times faster than the basic EGC method.

0090The Selective Message Digest token sequence for sample SQL messages are shown in the table in <figref idref="DRAWINGS">FIG. 7</figref>. Taking the first example, the message

0091‘SELECT COUNT (*) FROM tblLog t29’

0000has been broken down in a lexical analysis to the sequence of Selective Message Digest tokens

0000<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="offset" colwidth="77pt" align="left" /><colspec colname="1" colwidth="140pt" align="left" /><thead><row><entry /><entry namest="offset" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /><entry>identifier(“select”)</entry></row><row><entry /><entry>identifier(“count”)</entry></row><row><entry /><entry>self(“(“)</entry></row><row><entry /><entry>operator(“*”)</entry></row><row><entry /><entry>self(“)”)</entry></row><row><entry /><entry>identifier(“from”)</entry></row><row><entry /><entry>identifier(“tbl Jog”)</entry></row><row><entry /><entry>identifier(“t29”).</entry></row><row><entry /><entry namest="offset" nameend="1" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0092These tokens are a coarse grouping pertaining to characters.

0093As compared to the parse of the message ‘SELECT COUNT (*) FROM tbl_Log t29’ performed in <figref idref="DRAWINGS">FIG. 5</figref>, the SMD tokens are at the lower semantic level with coarser discrimination to the meaning of the tokens. For example, when using the full EGC method, for the same message, the language aware tokenizer <b>501</b> generates the following sequence of tokens:

0000<tables id="TABLE-US-00002" num="00002"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="offset" colwidth="77pt" align="left" /><colspec colname="1" colwidth="140pt" align="left" /><thead><row><entry /><entry namest="offset" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /><entry>keyword (select)</entry></row><row><entry /><entry>keyword (count)</entry></row><row><entry /><entry>left_paren</entry></row><row><entry /><entry>asterisk</entry></row><row><entry /><entry>rightparen</entry></row><row><entry /><entry>keyword (from)</entry></row><row><entry /><entry>identifier(tbl_Log)</entry></row><row><entry /><entry>identifier(log_t29).</entry></row><row><entry /><entry namest="offset" nameend="1" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0094Instead of the first token being of the general category “identifier”, it is classified more specifically as a “keyword”. Instead of the 3rd and 5th tokens in the sequence being of the category “self”, they are specified as being the left and right parentheses.

0095Thus, the full EGC tokenisaton generates tokens pertaining to the properties of the meaning of the language.

0096The sequence of Selective Message Digest tokens provides the basis for calculating a unique value for each sequence. There are many appropriate methods for calculating such a unique value in which that an arbitrary block of data is taken and a fixed-size bit string is returned. This process is often referred to as “hashing” and the hash value in this context is often called the message digest. Any suitable means may be used. Some specific examples of the Selective Message Digest Process are Shift &amp; Rotate methods on the entire Selective Message Digest token sequence, SHA family or Message-Digest algorithm 5 (MD5) family of hash methods on the entire Selective Message Digest token sequence and an interleaved Message Digest method integrated into the Selective Message Digest tokenization process.

0097Referring again to <figref idref="DRAWINGS">FIG. 6</figref>, and using the example of ‘SELECT COUNT (*) FROM tbl_Log t29’ discussed above, the sequence of tokens is provided to the Digest Process <b>701</b> and from this the MSG SMD <b>606</b> for the message is determined. The Selective Message Digest values are a way to provide unique cluster identifiers to messages that might have been identified using the EGC method referred to above. In the example shown, the sequence of tokens in this case provides a selective message digest of 134707047 which is then associated with the cluster ID (MSG Classification <b>402</b>) value 286739302.

0098<figref idref="DRAWINGS">FIG. 8</figref> shows an example of an interleaved Selective Message Digest process. With reference to <figref idref="DRAWINGS">FIG. 8</figref> and with respect to the same specific message used as an example above, i.e. the SQL message ‘SELECT COUNT (*) FROM tbl_Log t29’, the process by which an interleaved Message Digest may be integrated into the Selective Message Digest described above, is explained as follows.

0099First, message MSG <b>104</b> [‘SELECT COUNT (*) FROM tbl_Log t29’] is progressively tokenised by the SMD Tokenizer <b>604</b>. The SMD Tokenizer <b>604</b> performs lexical analysis of the message to generate the tokens within the message. As the first data forming the message is received, the initial sequence of Tokens Processed so far <b>640</b> is empty. The progressive nature of the tokenization means that the analysis of the message processes one token at a time, each time producing a Next SMD Token <b>641</b>

0100The Interleaved Digest Process <b>701</b> uses the Next SMD Token <b>641</b> and combines this with a Running Digest Value <b>740</b> to produce an updated Running Digest Value <b>740</b>.

0101Initially the Running Digest Value <b>740</b> is set to null. A test, labeled in the figure as MSG PROCESSED <b>745</b>, is performed to determine whether the entire message MSG <b>104</b> has been processed. If the entire message has been processed (the Yes branch) then the MSG SMD <b>606</b> value is set to the Running Digest Value <b>740</b>. If the entire message has not yet been processed (the No branch), then flow of control returns to the SMD

0102Tokenizer <b>604</b> to produce the next token in the message sequence. After two iterations of this flowchart for the case of the message MSG <b>104</b> ‘SELECT COUNT (*) FROM tbl_Log t29’ the data stores contain the following values:

0000<tables id="TABLE-US-00003" num="00003"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="1" colwidth="91pt" align="left" /><colspec colname="2" colwidth="126pt" align="left" /><thead><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row><row><entry>Data Store</entry><entry>Value</entry></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>640 Tokens Processed so far</entry><entry>identifier(“ select”), identifier(“ count”)</entry></row><row><entry>641 Next SMD Token</entry><entry>self(“(“)</entry></row><row><entry>740 Running Digest Value</entry><entry>0xf3db7465</entry></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0103Thus, there are two tokens in the “tokens processed so far” category, which are the identifiers (“select”) and (“count”). The next SMD token as stored in block <b>641</b> in <figref idref="DRAWINGS">FIG. 8</figref> is self(“(“). After digesting the next SMD Token self(“(“) the Running Digest Value <b>740</b> becomes 0x27bf8cd5. The final MSG SMD <b>606</b> value for the message <b>104</b> MSG ‘SELECT COUNT (*) FROM tbl_Log t29’ is 0x8077767. Using an interleaved process such as that described with reference to <figref idref="DRAWINGS">FIG. 8</figref> makes efficient use of RAM.

0104It will be understood that the methods described herein will typically be carried out by appropriate software running on appropriate computer equipment. The term “computer” is to be construed broadly. The term “a computer” or similar may include several distributed discrete computing devices or components thereof. The computer program may be in the form of source code, object code, a code intermediate source and object code such as in partially compiled form, or in any other form suitable for use in the implementation of the processes according to the invention. The carrier can be any entity or device capable of carrying the program. For example, the carrier may comprise a storage medium, such as a ROM, for example a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example a floppy disk or hard disk. Further, the carrier may be a transmissible carrier such as an electrical or optical signal, which may be conveyed via electrical or optical cable or by radio or other means.

0105The methods described herein have application wherever it is useful to be able to analyze symbols passing through or stored in a computer system. This can include cases where third parties attempt to utilise a computer inappropriately, e.g. to take control of a computer by “attacking” the computer system, and other cases too. For example, the same approach may be used within a running computer system and is not restricted to externally transmitted messages. The messages can be, for example, sequences of machine instructions that are about to run through the microprocessor. Thus, the method has application wherever it is generally desired or required to know if a system is operating appropriately.

0106Embodiments of the present invention have been described with particular reference to the examples illustrated. However, it will be appreciated that variations and modifications may be made to the examples described within the scope of the present invention. For example different languages, different instrumented computer programs, differing message formats, different input/output formats to computer programs may all be used as appropriate.