Configuring state machines used to order and select matching operations for determining whether an input string matches any of at least one regular expression using lookahead finite automata based regular expression detection

PRIORITY CLAIM

0001This application claims the benefit of U.S. Provisional Patent Application Ser. No. 61/252,651 (incorporated herein by reference and referred to as “the '651 provisional”), titled “LaFA: LOOKAHEAD FINITE AUTOMATA REGULAR EXPRESSION DETECTION SYSTEM,” filed on Oct. 17, 2009, and listing Masanori Bando, (Nabi) Sertac Artan and Hung-Hsiang Jonathan Chao as inventors. The present invention in not limited to requirements of the particular embodiments described in the '651 provisional.

1. BACKGROUND OF THE INVENTION

00021.1 Field of the Invention

0003The present invention concerns matching an arbitrary length string with one of a number of known regular expressions, and/or configuring state machines for this purpose. Such matching may be used for network intrusion detection and prevention.

00041.2 Background Information

0005Although Regular Expressions (referred to as “RegExes”) have been widely used in network security applications, their inherent complexity often limits the total number of RegExes that can be detected using a single chip for a reasonable throughput. This limit on the number of RegExes impairs the scalability of today's RegEx detection systems. The scalability of existing schemes is generally limited by traditional per character state processing and state transition detection paradigm.

0006RegExes are used to flexibly represent complex string patterns in many applications ranging from Network Intrusion Detection and Prevention Systems (“NIDPS”) (See, e.g., “Bro Intrusion Detection System,” http://www.bro-ids.org, and “Snort Network Intrusion Detection System,” http://www.snort.org, both incorporated herein by reference.) to compilers (See, e.g., A. V. Aho, M. S. Lam, R. Sethi, and J. D. Ullman, <i>Compilers. Principles, Techniques, and Tools </i>(2nd Edition) (Addison Wesley, 2006), incorporated herein by reference.) and DNA multiple sequence alignment (See, e.g., the articles, A. N. Arslan, “Multiple Sequence Alignment Containing a Sequence of Regular Expressions,” <i>CIBCB</i>, pp. 1-7 (2005), and Y. S. Chung, W. H. Lee, C. Y. Tang, and C. L. Lu, “RE-MuSiC: a Tool for Multiple Sequence Alignment with Regular Expression Constraints,” <i>Nucleic Acids Res </i>(<i>Web Server issue</i>), no. 35, pp. W639-644, 2007, both incorporated herein by reference.). For instance, RegExes are used for representing programming language tokens in compilers or constraint formulation for DNA multiple sequence alignment. In particular, NIDPSs Bro and Snort, and Linux Application Level Packet Classifier (L7 filter) (See, e.g., J. Levandoski, E. Sommer, and M. Strait, “Application Layer Packet Classifier for Linux,” http://l7-filter.sourceforge.net, incorporated herein by reference.) use RegExes to represent attack signatures or packet classifiers.

0007A NIDPS RegEx detection system for high speed networks should satisfy the following two requirements—(1) scalability and (2) high throughput. A scalable detection system can accommodate more policies (rules) that increase the flexibility of the NIDPS. NIDPS should also support ever increasing traffic demand. The number of RegExes in the Snort signature set increased from 1131 RegExes in 2006, to 2290 RegExes in 2009, and the typical RegEx length keeps growing.

0008Internet carriers and vendors have successfully experimented with 100-Gbps equipment. In September 2008, Verizon successfully performed a 100-Gbps transmission for more than 646 miles. (See, e.g., “Verizon and Nokia Siemens Networks Set New Record for 100 Gbps Optical Transmission,” http://newscenter.verizon.com, incorporated herein by reference.) Cisco's global IP traffic forecast report states that global IP traffic will double nearly every two (2) years by the end of 2012 and, consequently, annual global IP traffic will exceed half a zetta (10²¹) bytes in 2013. (See, e.g., “Networking Solutions White Paper,” http://www.cisco.com, incorporated herein by reference.) In the art, there is no current state-of-the-art regular expression detection system capable of supporting such large RegEx sets (i.e., RegEx sets of 1000 RegExes or more, such as Snort which includes at least 2290 RegExes at this time), and even larger RegEx sets expected in the future, with high speed (e.g., 40 Gbps to 100 Gbps, or higher) demand. Thus, there is a need to support current and expected future RegEx sets with small memory requirements. Detection at line-speed is another desirable attribute of NIDPS. It would be desirable to be able to use very high speed on-chip memory, even for large RegEx sets, thereby supporting a higher throughput required for today's and tomorrow's line speeds.

0009Finite Automata (“FA”) are the de-facto tools to address the RegEx detection problem. For RegEx detection on an input, the FA starts at an initial state. Then, for each character in the input, the FA transitions to the new state determined by the previous state and the current input character. If the resulting state is unique, the FA is called a Deterministic Finite Automaton (“DFA”); otherwise, it is called a Non-Deterministic Finite Automaton (“NFA”). (See, e.g., N. Wirth, <i>Compiler Construction </i>(Addison Wesley, 1996), incorporated herein by reference.)

0010NFA and DFA represent two extreme cases of the tradeoff between performance and resource usage. DFA has constant time complexity since it guarantees only one state transition per character by definition. The constant time complexity per character allows DFA to achieve high-speed RegEx detection, which makes DFA the approach preferred currently. This is especially the case for software solutions, since DFA can be executed fast, serially on commodity CPUs. NFA, on the other hand, typically requires massive parallelism, making it harder to implement in software. NFA also allows multiple simultaneous state transitions, which leads to a higher time complexity. However, the price paid for the high speed of DFA is its prohibitively large memory requirement.

0011Recent applications, most notably deep packet inspection for NIDPS, require RegEx detection scalable to high quantities of complex RegExes. This need for scalability makes the DFA impractical for such applications, due to its large memory requirement. Since NFA is not fast, RegEx detection at high speeds, scalable to large sets of complex RegExes, is still an open issue. Thus, a better NIDPS is needed.

0012The present inventors note that three characteristics common to most of today's state-of-the-art RegEx detection systems limit the scalability of these systems. First, RegExes consist of a variety of different “components,” such as character classes or repetitions. (See, e.g., “Perl Compatible Regular Expressions,” http://www.pcre.org/, incorporated herein by reference.) This variety makes it hard to identify a method for concurrently detecting all of these different components of a RegEx efficiently. However, in most cases today, these heterogeneous components are detected by a state machine that consists of homogeneous states. This leads to inefficiencies that limit the scalability of such systems. Second, the order of components in a RegEx is preserved in the state machine detecting this RegEx. However, it may be beneficial to change the order of the detection of components in a RegEx. For instance, it is easy to detect exact strings (simple strings) as they are expected to appear less frequently, whereas others may appear more frequently (for instance, character classes). (The present inventors found that by reordering the detection of the components, the trade-off between detection complexity and appearance frequency can be exploited for better scalability.) Finally, most RegExes share similar components. In the traditional FA approaches, a state machine is used to detect a component in a RegEx. This state machine is duplicated since the similar component may appear multiple times in different RegExes. Furthermore, most of the time, these RegExes sharing this component cannot appear at the same time in the input. As a result, the duplication of the same state machine for different RegExes introduces redundancies, which limit the scalability of the RegEx detection system.

0013Based on the foregoing three observations of the inventors, a LaFA scheme is introduced—a novel and inventive detection method that resolves scalability issue of current RegEx detection paradigm. The scalable compact Look-ahead Finite Automata (“LaFA”) data structure requires small memory footprint and makes it feasible to implement large RegEx sets using only very fast, small on-chip memory.

2. SUMMARY OF THE INVENTION

0014The main focus of existing RegEx detection schemes is optimizing the number of states and the required transitions, but not the suboptimal character-based detection method. Furthermore, the potential benefits of reduced number of operations and states using out-of-sequence detection methods have not been explored.

0015In at least some embodiments consistent with the present invention, state machines used to order and select matching operations for determining whether an input string matches any of at least one regular expression are configured by (1) accepting the set of regular expression(s), and (2) for each of the regular expression(s) of the set accepted, (A) identifying any look-ahead type strings within the given regular expression, (B) identifying any sequential type strings within the given regular expression, (C) partitioning the regular expression based on any identified simple strings, any identified look-ahead type variable strings, and any sequential type variable strings to generate partitioned parts of the given regular expression, (D) reordering the partitioned parts of the given regular expression using optimization policies to generate reordered partitioned parts of the regular expression, and (E) configuring nodes of a state machine corresponding to the given regular expression, by recording configured information of the nodes on a tangible storage medium, using (i) an order of the reordered partitioned parts of the regular expression, and (ii) a string type of the partitioned parts of the regular expression. Once configured, the state machines may accept an input string, and for each of the regular expression(s), check for a match between the input string accepted and the given regular expression using the configured nodes of the state machine corresponding to the given regular expression. Checking for a match between the input string accepted and the given regular expression using configured nodes of a state machine corresponding to the given regular expression by using the configured nodes of the state machine may include (1) checking detection events from a simple string detector, (2) submitting queries to identified modules of a variable string detector, and (3) receiving detection events from the identified modules of the variable string detector.

0016Given previously configured state machines, embodiments consistent with the present invention can determine whether an input string matches at least one regular expression by: (1) accepting the input string; and (2) for each of the at least one regular expression, checking for a match between the input string accepted and the given regular expression using configured nodes of a state machine corresponding to the given regular expression by using the configured nodes of the state machine to (i) check detection events from a simple string detector, (ii) submit queries to identified modules of a variable string detector, and (iii) receive detection events from the identified modules of the variable string detector.

0017Embodiments consistent with the present invention use LaFA to perform scalable RegEx detection using a very small amount of memory. LaFA's memory requirement can be made very small due to the following features. First, different parts of a RegEx (namely RegEx “components”) are detected using different detectors, each of which may be specialized and optimized for the detection of a certain RegEx component. Second, the RegEx component detection sequence is systematically reordered. This provides new opportunities for memory optimization. Third, many redundant states in classical finite automata are identified and eliminated. Simulations have shown that embodiments consistent with the present invention require an order of magnitude less memory compared to today's state-of-the-art RegEx detection systems. A single commodity Field Programmable Gate Array (“FPGA”) chip can accommodate up to twenty-five thousand (25 k) RegExes. Based on the throughput of an exemplary LaFA embodied on an FPGA, it is estimated that a 34-Gbps throughput can be achieved.

3. BRIEF DESCRIPTION OF THE DRAWINGS

0018<figref idrefs="DRAWINGS">FIG. 1</figref> is a bubble diagram illustrating general operations of configuring and querying a LaFA state machine for determining a possible match between an arbitrary-length input string to a set of one or more regular expression(s), in a manner consistent with the present invention.

0019<figref idrefs="DRAWINGS">FIGS. 2A-2F</figref> illustrate a set of regular expressions being transformed according to a LaFA representation consistent with the present invention.

0020<figref idrefs="DRAWINGS">FIG. 3A</figref> is a matrix depicting the depth and the detection complexity of example components, while <figref idrefs="DRAWINGS">FIG. 3B</figref> is a matrix illustrating a string classification scheme that may be used by embodiments consistent with the present invention.

0021<figref idrefs="DRAWINGS">FIG. 4</figref> is a flow diagram of an exemplary method for configuring a LaFA state machine to check an arbitrary input sting against a set of at least one or more regular expression(s), in a manner consistent with the present invention.

0022<figref idrefs="DRAWINGS">FIG. 5</figref> is a block diagram of an exemplary LaFA architecture, including a detection block and the correlation block, consistent with the present invention.

0023<figref idrefs="DRAWINGS">FIG. 6</figref> illustrates the component reordering and transition states of the LaFA scheme, consistent with the present invention, as well as the storage of nodes in memory of a regular expression.

0024<figref idrefs="DRAWINGS">FIG. 7</figref> is a flow diagram of an exemplary method for querying a LaFA state machine with an arbitrary input string to determine a possible match against a set of at least one or more regular expression(s), in a manner consistent with the present invention.

0025<figref idrefs="DRAWINGS">FIGS. 8A-8D</figref> illustrate how multiple RegEx(es) are organized in a correlation block, along with node activation and timing verification operations, in an exemplary embodiment consistent with the present invention.

0026<figref idrefs="DRAWINGS">FIGS. 9A-9C</figref> illustrate configured LaFA state machine detection systems consistent with the present invention in which arbitrary input strings are checked for a match against regular expressions.

0027<figref idrefs="DRAWINGS">FIG. 10</figref> is a block diagram of an exemplary apparatus consistent with the present invention that may perform various operations, and store various information generated and/or used by such operations.

0028<figref idrefs="DRAWINGS">FIG. 11</figref> illustrates tracks of a LaFA architecture consistent with the present invention in which simultaneous detection of multiple regular expressions occurs in parallel.

0029<figref idrefs="DRAWINGS">FIGS. 12A-12C</figref> illustrate block detection optimization within an exemplary LaFA architecture consistent with the present invention.

0030<figref idrefs="DRAWINGS">FIGS. 13A-13C</figref> illustrate a “time-event” occurrence within a LaFA architecture consistent with the present invention.

0031<figref idrefs="DRAWINGS">FIG. 14</figref>, which includes <figref idrefs="DRAWINGS">FIGS. 14A-14C</figref>, illustrates a flow diagram of an exemplary string matching method, consistent with the present invention.

4. DETAILED DESCRIPTION

0032The present invention may involve novel methods, apparatus, message formats, and/or data structures for determining whether or not an arbitrary-length bit string matches one or more of a number of known regular expressions, as well as generating a data structure (e.g., a LaFA state machine) used in such a determination. The following description is presented to enable one skilled in the art to make and use the invention, and is provided in the context of particular applications and their requirements. Thus, the following description of embodiments consistent with the present invention provides illustration and description, but is not intended to be exhaustive or to limit the present invention to the precise form disclosed. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles set forth below may be applied to other embodiments and applications. For example, although a series of acts may be described with reference to a flow diagram, the order of acts may differ in other implementations when the performance of one act is not dependent on the completion of another act. Further, non-dependent acts may be performed in parallel. No element, act or instruction used in the description should be construed as critical or essential to the present invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. Thus, the present invention is not intended to be limited to the embodiments shown and the inventors regard their invention as any patentable subject matter described.

4.1 OVERVIEW—CONFIGURATION OF STATE MACHINES USING REGULAR EXPRESSIONS AND MATCHING AN INPUT STRING TO A REGULAR EXPRESSION USING A CONFIGURED STATE MACHINE

0033<figref idrefs="DRAWINGS">FIG. 1</figref> is a bubble diagram illustrating general operations of configuring and querying a LaFA state machine for determining a possible match between an arbitrary-length input string and a regular expression(s) belonging to a set of one or more regular expression(s). Specifically, the proposed LaFA state machine configuration operations <b>120</b> are performed first using a set of one or more regular expressions <b>110</b>. Once the operations <b>120</b> are completed, a highly optimized and scalable LaFA state machine data structure <b>130</b> results. The LaFA state machine data structure <b>130</b> is configured such that an arbitrary-length input string may be checked against the set of one or more regular expressions <b>110</b>. More specifically, given an input string(s) <b>150</b>, the configured LaFA state machine data structure <b>130</b> may be used by input string(s) to regular expression(s) match determination operations <b>140</b>. If a match is determined by such operations <b>140</b>, then the LaFA system may return a list of matching RegEx(es) <b>160</b>.

0034The proposed LaFA system is a finite automata state machine optimized for scalable RegEx detection. The LaFA system is used for representing a set of n RegExes, R={r₁, r₂, . . . r_n}, which can also be called a RegEx Database. For instance, this set of RegExes can be a set of Snort signatures, or Bro or Linux Layer 7 rules. An associated LaFA RegEx detection system can be queried with an input I, such as a network packet. The system may return a match along with a list of matching RegExes if input I matches one or more of the RegExes in R. Otherwise, a “no match” result is returned.

0035The following illustrates advantageous features and provides an initial examination of the proposed LaFA system with the help of a simple example.

0036Consider, the RegEx set, R including three RegEx(es), r₁, r₂, and r₃as shown in <figref idrefs="DRAWINGS">FIG. 2A</figref>. The traditional NFA representation of R is shown in <figref idrefs="DRAWINGS">FIG. 2A</figref>. The first step in constructing the LaFA data structure is the detection and separation of simple strings from variable strings within a RegEx for each of the RegEx(es) in the set R. A simple string is a fixed sequence of characters such as “abc” and “op” in r₁. In practice, it can be a word in the dictionary or an excerpt from an executable file. Consider the state diagram of the traditional NFA in <figref idrefs="DRAWINGS">FIG. 2B</figref>. A separate state is defined for each character in simple strings. Although a per-character state transition looks simple, it may require many state transitions per character when multiple RegEx(es) are to be matched against the RegEx database. Consider the input characters “abc”. The “a” state of r₁is first matched followed by the “b” states of both r₁and r₂. Then, as the input character “c” arrives, the “c” states of all FAs are matched. All three FAs concurrently transition to their [a-z] states (i.e., variable string component [a-z]). Hence, one can appreciate the resulting large number of concurrent active states for a database of hundreds of RegEx(es). A large number of concurrent active states often necessitate high resource requirements and low throughput.

0037Note that the detection of simple strings (namely, the exact string matching) is a well-studied problem with many optimized solutions. In particular, there are high throughput hardware solutions with very small memory footprints. Exemplary embodiments consistent with the present invention exploit the fact that neither NFA, nor DFA, is the most efficient method to detect these simple strings. Rather, such embodiments merge the states of each simple string portion of the automata (i.e., “abc”, “bc”, “op”, “qr”, and “st”) into a single super state. These simple strings can then be detected using high-speed exact string matching methods. The resulting FA, as shown in <figref idrefs="DRAWINGS">FIG. 2C</figref>, has fewer states than the traditional NFA in <figref idrefs="DRAWINGS">FIG. 2B</figref>. By generating one detection event per simple string, rather than making many state transitions for each input character as in an NFA or DFA, the LaFA system significantly reduces the number of operations.

0038By definition, NFA and DFA are constructed based on the order of components in the RegEx. Thus, the appeared order of components in the RegEx is preserved as the order of state in the FA. As a result, the reordering of the sequence of detection is difficult. In <figref idrefs="DRAWINGS">FIG. 2C</figref>, each FA consists of three components: (1) a simple string, (2) a character class [a-z] (i.e., variable string [a-z]), and (3) another simple string. Consider matching the example input string “abcxop” against the states in <figref idrefs="DRAWINGS">FIG. 2C</figref>. For this particular input, after matching earlier states for “abc”, “be”, and “c”, all three FAs go to their respective super states of [a-z] for matching “x”, concurrently. As the input continues to “op”, however, the input matches only to the first RegEx r₁. Note that, in fact, there is no possible input that can match to more than one RegEx for this example set of RegEx(es). Observe that the FAs for r₂and r₃made the unnecessary [a-z] detection just to abandon the search at the end when neither of the simple strings “qr” or “st” is found in the input.

0039In the exemplary LaFA system, the reordering of the detection sequence as shown in <figref idrefs="DRAWINGS">FIG. 2D</figref> is proposed. For instance, as depicted in <figref idrefs="DRAWINGS">FIG. 2D</figref>, the third component in each RegEx, the simple string, is swapped with the second component, the variable string (character class) [a-z]. As a result, considering the RegEx r₁for instance, the [a-z] detection is only performed only if both “abc” and “op” are detected in order (not necessarily consecutively). Consequently, referring to <figref idrefs="DRAWINGS">FIG. 2E</figref>, for the example input of “abcxop”, exactly one [a-z] detection is performed rather than three concurrent [a-z] detections as in the case before reordering in <figref idrefs="DRAWINGS">FIG. 2C</figref>. In general, components in RegExes may be classified based on the information revealed about a RegEx, if a certain component in that RegEx is found. Considering r₁as an example again, (r₁: abc[a-z]op), detecting the simple string “abc” in the input reveals more information than detecting the variable string [a-z]; namely a range of characters from a to z. Intuitively, one expects that more restrictive matching components (like “abc”) should occur much less frequently in inputs than less restrictive components (like [a-z]). In the present embodiment, components that reveal more information (such as “abc”) are called “deep” components, while components that do not reveal much information (such as [a-z]) are called “shallow” components.

0040Furthermore, there are deep components that are more complex to detect than simple strings. For example, a string repetition element “\ n{x, y}” is deep but requires more complex operations than an exact string matching. It would be advantageous to detect simple components (requiring lower detection complexity) before the more complex components (requiring higher detection complexity). In this way, the probability of requiring evaluation of the latter components is reduced. Therefore, components may be further classified according to how complex it is to detect that component.

0041<figref idrefs="DRAWINGS">FIG. 3A</figref> shows a matrix of the depth (or restrictiveness) and the detection complexity of components with examples. At least some exemplary LaFA schemes consistent with the present invention detect simple and deep components before complex and/or shallow ones. This aspect of the LaFA scheme is referred to as “Look-ahead” in the following.

0042A typical detection operation involving NFA or DFA requires at least one state transition for each input character. As a result, the handling of state transitions saves a prominent portion of the operation costs by using the exemplary LaFA super states. The number of state transitions can be reduced, on average, to less than one per input character using an exemplary LaFA scheme consistent with the present invention. This is basically a highly optimized regular expression detection system. Every time a component such as a simple string is detected, a detection event is generated. These events are then correlated to determine if they correspond to one or more RegExes in the RegEx set. The details of this aspect of the exemplary LaFA are covered in later.

0043As can be appreciated from the foregoing, detection of some components (e.g., the [a-z] component which is a variable string) only needs to be done once at any given time. Instead of duplicating those components in multiple RegEx(es), resources and memory can be saved by sharing those rarely used components. The resulting LaFA representation is shown in <figref idrefs="DRAWINGS">FIG. 2E</figref>. Furthermore, exemplary special modules are used that are specifically designed to efficiently detect certain types of variable strings to replace these states. Details of such exemplary modules are described below.

0044Each unique component in a RegEx is assigned an identification code called a string ID. The ID includes two parts. The one part (e.g., the first part) identifies the component type either as S (simple string) or V (variable string). The notation is presented in <figref idrefs="DRAWINGS">FIG. 2F</figref> for the example discussed above. Simple string components such as “abc”, and “st” are each assigned unique string IDs from S₁to S₆in <figref idrefs="DRAWINGS">FIG. 2F</figref>. Variable string component “[a-z]” is assigned V₁as a string ID.

0045The foregoing example, referring to <figref idrefs="DRAWINGS">FIGS. 2A-2E</figref>, illustrates that embodiments consistent with the present invention can be used to produce a highly optimized regular expression detection system. In general, the exemplary LaFA scheme achieves such a highly optimized system by (1) identifying simple strings and variable strings from a set of given RegEx(es), and subsequently (2) reordering such components according to a set of optimization policies which will be discussed below. Based on the resulted reordering of components, a state machine is configured accordingly allowing for a highly optimized regular expression detection system to be produced. There exist many types of variable strings—some may be reordered (i.e., allowing the “Look-ahead” operations), while others may not (i.e., requiring sequential operations). As such, look-ahead-type variable strings will be denoted as V_1aand variable strings that may not be reordered are denoted as V_seq. Even simple strings may be further classified as “normal” simple strings or “short” simple strings. <figref idrefs="DRAWINGS">FIG. 3B</figref> is a matrix illustrating a possible classification of strings normal simple strings, short simple strings, sequential variable strings and look-ahead variable strings

0046Note that each of the variable strings V_1aand V_seqmay be further classified into sub-types of variable strings. The variable string V₁mentioned in the above-discussed example is a look-ahead type variable string V_1a. A table listing sub-types of variable strings will be described below.

0047<figref idrefs="DRAWINGS">FIG. 4</figref> is a flow diagram of an exemplary method <b>400</b> for configuring a LaFA state machine (which may then be used to check an arbitrary input sting against a set of at least one or more regular expression(s)) in a manner consistent with the present invention. (Recall operations <b>120</b> of <figref idrefs="DRAWINGS">FIG. 1</figref>.) Specifically, the method <b>400</b> may accept a set of at least one or more regular expression(s). (Block <b>410</b>). Thereafter, for each of the at least one or more regular expressions of the accepted set, the method <b>400</b> may identify any simple strings, look-ahead type variable strings (i.e., V_1a), and sequential type variable strings (i.e., V_seq) within the given regular expression. (Blocks <b>420</b> and <b>430</b>) Next, the method <b>400</b> may partition the regular expression based on any identified simple strings, any identified look-ahead type variable strings, and any sequential type variable strings to generate partitioned parts of the given regular expression. (Block <b>440</b>) Then, the method <b>400</b>, reorders the partitioned parts of the given regular expression using optimization policies (described in more detail below) to generate reordered partitioned parts of the regular expression. (Block <b>450</b>) Subsequently, the method <b>400</b> configures elements of a LaFA state machine corresponding to the given regular expression (by recording configured information of the elements on a tangible storage medium) using (A) an order of the reordered partitioned parts of the regular expression, and (B) a component/string type of the partitioned parts of the regular expression. (Block <b>460</b>) Finally, if there is another regular expression available from the accepted set, the method <b>400</b> repeats the actions performed by Blocks <b>430</b>-<b>460</b> within loop <b>420</b>-<b>470</b>. Otherwise the method <b>400</b> simply terminates. (Blocks <b>470</b> and <b>480</b>)

0048A configured state machine data structure generated by the exemplary method <b>400</b> of <figref idrefs="DRAWINGS">FIG. 4</figref> may be used to determine quickly whether an arbitrary-length bit string matches (or might match) one of a large number of known RegExe(s) of a set. This characteristic makes such a data structure useful, for example, in communications network security. Specifically, this characteristic makes such a data structure useful to determine whether an input arbitrary-length bit string matches one of a number of known network intrusion signatures quickly. As another example, this characteristic also makes such a data structure useful to determine whether an input arbitrary-length bit string matches one of a number of known nucleotide sequences, such as known DNA and/or RNA sequences.

0049In the aforementioned, the core ideas behind LaFA (Look-ahead Finite Automata) that facilitate the reduction of memory requirements and detection complexity using simple examples have been illustrated. To reiterate, first the set of RegExes were first represented in NFA format. Then, the states of simple strings were identified and merged into super states. The components were reordered according to their depth and complexity. Finally, the states of components that can potentially be shared among RegExe(s) were merged. In the next section, details of the LaFA data structure and the associated scalable RegEx detection architecture are described.

4.2 ARCHITECTURE AND CONFIGURATION OF STATE MACHINES FOR CHECKING AN INPUT STRING AGAINST REGULAR EXPRESSIONS

0050<figref idrefs="DRAWINGS">FIG. 5</figref> is a simple block diagram of the LaFA architecture. The LaFA architecture includes a detection block <b>510</b> and a correlation block <b>520</b>. The detection block <b>510</b> is responsible for detecting the components in the input string. The simple-sting detector <b>530</b> is an (e.g., highly optimized) exact string matching system used to detect simple strings. The variable-string detector <b>540</b> includes (e.g., highly optimized) variable string detection modules. The correlation block <b>520</b> is responsible for keeping track of state transitions. The correlation block <b>520</b> inspects the sequence of components in the input string (i.e., order and timing) to determine whether the input matches any of the RegEx(es) in its RegEx database.

0051The detectors <b>530</b> and <b>540</b> in the detection block <b>510</b> communicate their findings to the correlation block <b>520</b> by sending detection events <b>535</b><i>a </i>and <b>535</b><i>b </i>A detector in the detection block <b>510</b> generates a detection event when it detects a component in the input <b>505</b>. Each detection event includes a unique ID for the detected component, and its location in the input packet. Extra information, such as the length of the detected string, may accompany the event if desired. There can be two sources of an event. First, a “string event” is generated when a simple string is detected. Second an “in-line” event is generated by inline lookup modules. The inline event is described in detail below and specifically refers to the sequential type of variable string (V_seq) which does not allow for reordering of RegExe(s) (i.e., no “look-ahead” operations) and uses inline variable string detection modules. The generated events prompt a detection sequence in the correlation block <b>520</b> to proceed to the detection state. The details of individual blocks in the LaFA architecture and their interactions are described below. The LaFA architecture may be implemented in hardware.

0052The following describes the formal construction of an exemplary LaFA data structure. Given a set of RegEx(es), R, the exemplary LaFA data structure is constructed in three steps: (1) RegEx partitioning, (2) component reordering, and (3) node structure construction. Each is described below.

0053The first step of the exemplary LaFA data structure construction is to partition each RegEx into simple strings and variable strings. This partitioning facilitates the detection of simple strings and variable strings. In this application, each simple string is represented by the letter S and each variable string is represented by the letter V for simple representations of components. Note that the variable strings V are assumed to be the “look-ahead” type (i.e., V_1a) in the following.

0054The second step of the exemplary LaFA data structure construction is the component reordering according to a set of optimization policies (described below). Component reordering was introduced in section 4.1. However, a detailed discussion of exemplary optimization policies that may be used to reorder the components will be provided in following. This section further introduces the new term “node”. Nodes are depicted as squares in <figref idrefs="DRAWINGS">FIG. 6</figref>. Specifically, <figref idrefs="DRAWINGS">FIG. 6</figref> illustrates the component reordering and transition states of an exemplary LaFA scheme, as well as the storage of nodes in memory of RegEx r₁: abc[a-z]op→S₁:abc, V₁[a-z], S₂:op. In the traditional sequence <b>610</b> shown in <figref idrefs="DRAWINGS">FIG. 6</figref>, components are activated one by one following the sequence of the RegEx r₁. In contrast, the exemplary LaFA sequence (i.e., “look-ahead approach) <b>620</b> shown in <figref idrefs="DRAWINGS">FIG. 6</figref> skips (look-ahead) variable string V₁and activates S₂first. The LaFA scheme <b>620</b> verifies variable strings (e.g., V₁) using special variable string modules for which specific information related to individual components is also used. The location to store this component verification information (squares in the <figref idrefs="DRAWINGS">FIG. 6</figref>) is called a “node”. Nodes are stored in memory <b>630</b>. Verification of S₂and V₁will be performed successively. The nodes of S₂and V₁are stored at consecutive locations in the node memory <b>630</b>. As long as S₂and V₁are allocated to consecutive memory locations, S₁and S₂can be stored anywhere in the memory. This helps in balancing the nodes in the node memory and the locations of simple strings are resolved by pointers. An exemplary node structure is described next.

0055The third step of the exemplary LaFA data structure construction is to identify the contents of a node. As in any other state machine, the LaFA state machine also has basic information in every state. As discussed, the LaFA states are associated with nodes and these nodes contain the verification information specific to each component. The node also contains the current state, which indicates whether the current state is active or inactive Conventional FAs, such as NFA and DFA, change states constantly, with every character received. Hence, tracking sequence timing is not an issue in NFA and DFA. (That is, since NFA and DFA don't reorder strings, and since they detect strings character by character, sequence timing does not need to be tracked in NFA and DFA.) In the LaFA state machine, however, the input (event) is generated by a simple string detector. Thus, detection timing (detection sequence) may need to be verified. Given this potential need, timing information is stored in the node A component can have variable length as well. Therefore, a node keeps two timing information pieces—namely, minimum time and maximum time. A simple string must be detected within this time range. Summarizing the information stored in a node, each node may include the following: (1) a pointer to the next state; (2) the status of the current state; (3) detection timing values (e.g., minimum time (distance) and maximum time (distance)); (4) module ID; (5) component ID; (6) minimum repetition; and (7) maximum repetition. The following sections will demonstrate why this information is stored and how it is used. More specifically, a node for processing simple string may store a time range and a pointer to a next node. A node for processing a short simple string may store a time range, a pointer to a next node, a pattern being searched for, and an SDM module identifier. A node for processing a look-ahead type variable string may store a time range, a pointer to a next node, a pattern being searched for, and a TLM module identifier. A node for processing a sequential type variable string may store a time range, a pointer to a next node, a minimum repetition, a maximum repetition, and an RDM module identifier. A node for processing a look-ahead type variable string may store a time range, a pointer to a next node, a minimum repetition, a maximum repetition, and an FRM module identifier. Finally, a node for processing a look-ahead type variable string may store a time range, a pointer to a next node, a minimum repetition, a maximum repetition, and a CLM module identifier.

4.2.1 Architecture and Configuration of the Correlation Block for Checking an Input String Against Regular Expressions

0056<figref idrefs="DRAWINGS">FIG. 7</figref> is a flow diagram of an exemplary method <b>700</b> for querying a LaFA state machine with an arbitrary input string against a set of at least one or more regular expression(s) for determining a possible match, in a manner consistent with the present invention. (Recall, e.g., operation <b>140</b> of <figref idrefs="DRAWINGS">FIG. 1</figref>.) Specifically, the method <b>700</b> may accept an arbitrary-length input string. (Block <b>710</b>) Subsequently, for each of the regular expressions, the method <b>700</b> may check for a match between the accepted arbitrary-length input string and the regular expression(s) using the configured elements of the state machine corresponding to the given regular expression(s). (Block <b>720</b>) The method <b>730</b> is then left. (Node <b>730</b>)

0057Referring back to blocks <b>710</b> and <b>720</b>, <figref idrefs="DRAWINGS">FIGS. 8A-8D</figref> illustrate an example of querying a LaFA state machine according to the method <b>700</b>. Practical RegEx sets include hundreds of RegEx(es) that are programmed in the correlation block. <figref idrefs="DRAWINGS">FIGS. 8A-8D</figref> illustrate how multiple RegEx(es) are organized in a correlation block, along with node activation and timing verification operations. The correlation block is organized into columns of nodes. Each node is similar to a state in a traditional FA, but includes more information than a state (e.g., next node pointer, min time, max time, component ID, etc.). A node in the LaFA system is also likely to correspond to multiple states in a traditional FA. Each column corresponds to one component, and has one node for each RegEx that this component appears in. Once the correlation block receives a detection event for a particular component, all the nodes in the corresponding column are checked to determine (1) whether the detection of this RegEx is in progress, and (2) which component is expected next. The exemplary look-ahead detection flow satisfies certain timing constraints. Only when the detection of contents with the proper timing is confirmed, is the next node activated. To determine this, each node has a static pointer to its next node.

0058In the example illustrated by <figref idrefs="DRAWINGS">FIGS. 8A-8D</figref>, there are two RegEx(es) in the RegEx set; namely, (r₁) RegEx<b>1</b> (abc[a-z]op: S₁V₁S₂) and (r₄) RegEx<b>4</b> (abd[̂x]{3}xyz: S₁V₂S₃). After reordering the RegExes accordingly (RegEx<b>1</b>: S₁S₂V₁and RegEx<b>4</b>: S₁S₃V₂), the nodes corresponding to the simple strings are set in the correlation block along with pertinent node information. RegEx<b>1</b> and RegEx<b>4</b> contain the same component, simple string S₁. Two nodes are examined once S₁is detected. (See <figref idrefs="DRAWINGS">FIG. 8A</figref>.) When the nodes corresponding to S₁are activated as indicated by a set bit in the node and string S₁is detected, then a node transition occurs to the next node corresponding to the next component in the given RegEx. (See <figref idrefs="DRAWINGS">FIG. 8B</figref>.)

0059The sequence of input character and arrival time is discussed below along with the two example RegExes; RegEx<b>1</b> and RegEx<b>4</b>. The unit of time is defined as the interval required to receive one character. Initially, the active RegEx list is as shown in <figref idrefs="DRAWINGS">FIG. 8A</figref>. As shown, nodes are represented as boxes containing a “0” or “1”. A node containing a “0” is inactive, while a node containing a “1” is active. A received event is processed only when the target state is active. Otherwise (inactive case) the event will be ignored. The first component of every RegEx is always active to start the state transition (otherwise state transition never occurs/starts). For instance, as illustrated in <figref idrefs="DRAWINGS">FIG. 8A</figref>, the simple string, S₁: “abc” is the first component in both RegEx<b>1</b> and RegEx<b>4</b>. Therefore, they are always active to start the detection of these RegExes, once S₁is detected.

0060Upon detecting the simple string, S₁: “abc” at time <b>3</b> (See <figref idrefs="DRAWINGS">FIG. 8B</figref>.), the simple-string detector (not shown in <figref idrefs="DRAWINGS">FIGS. 8A-8D</figref>) generates an event and passes it to the correlation block. When the event is received, based on the RegEx stored in the node, the next simple string, S₂: “op” of RegEx<b>1</b> and S₃: “xyz” of RegEx<b>4</b> is activated by setting the bit corresponding to RegEx<b>1</b> and RegEx<b>4</b> to active “1” as shown in <figref idrefs="DRAWINGS">FIG. 8B</figref>.

0061Next, at time <b>9</b>, S₃is detected and the detection time of S₃is verified based on the distance between S₃and S₁. If the timing verification is successful then, as illustrated in <figref idrefs="DRAWINGS">FIG. 8C</figref>, a query is sent to the variable string detection module to verify the RegEx component V₂. The processing time of the detection module is denoted by Δ. Hence, after finishing the processing and successful verification by the detection module, a RegEx match is generated as shown in <figref idrefs="DRAWINGS">FIG. 8D</figref>.

0062The RegEx detection is performed on the exemplary LaFA data structure that was constructed using the procedure described in the previous section. An exemplary process for the detection procedure provided here:

0000<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="left" /><thead><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>1:   LaFA (Input Event)</entry></row><row><entry>2:   if (The node is active) then</entry></row><row><entry>3:     [verify detection timing]</entry></row><row><entry>4     for (Number of Lookup Nodes) do</entry></row><row><entry>5:        if (Buffered Lookup) then</entry></row><row><entry>6:         [Access one of special modules and verify]</entry></row><row><entry>7:        end if</entry></row><row><entry>8:        if (In-Line Lookup) then</entry></row><row><entry>9:         [Set RegEx detection pattern in an in-line module]</entry></row><row><entry>10:        end if</entry></row><row><entry>11:      end for</entry></row><row><entry>12:      if (All Lookup Matched) then</entry></row><row><entry>13:        if (Following Simple-String Node Exist) then</entry></row><row><entry>14:         [Activate next simple-string node]</entry></row><row><entry>15:        else</entry></row><row><entry>16:         [generate match signal]</entry></row><row><entry>17:        end if</entry></row><row><entry>18:      end if</entry></row><row><entry>19:  end if</entry></row><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

4.2.2 Reordering Strings Using Optimization Policies

0063As mentioned above, in order to achieve a highly optimized RegEx detection system, it is important to use optimization policies when configuring the LaFA state machine. The proposed LaFA scheme achieves such a highly optimized system by identifying simple strings and variable strings from a set of given RegEx(es), and subsequently reordering such components according to the set of optimization policies. Based on the resulted reordering of components, a state machine is configured accordingly, thereby allowing a highly optimized regular expression detection system to be produced.

0064The reordering of a RegEx depends on whether the RegEx includes any look-ahead type variable strings denoted as V_1a. Consider, for example, the following cases.

Case 1:

0065If a RegEx includes only simple strings S and look-ahead type variable string(s) V_1a, then the rule for reordering is as follows. Examining each component and starting from the first component (leftmost component) of the RegEx and moving/scanning towards the right components, for every variable string V_1a(or group of adjacent V_1aonly) that is directly followed by a simple string S, a swap in component positions between the encountered V_1a(or group of adjacent V_1aonly) and the directly following simple string S shall be performed, resulting in a reordered RegEx.

0066As an example given for the above case 1, consider the regular expression: RegEx: abc[a-z]xyz[0-9]op→S₁:abc, V₁: [a-z], S₂: xyz, V₂[0-9], S₃: op→RegEx: S₁V₁S₂V₂S₃. Note that V₁and V₂are look-ahead-type variable strings. Then, applying the reordering rule of case 1, discussed above, to the RegEx: S₁V₁S₂V₂S₃→ results in the reordered RegEx: S₁S₂V₁S₃V₂.

0067The information obtained from this reordered RegEx is used for configuring the LaFA state machine, such as configuring the nodes and their information in the correlation block. (See FIGS. <b>5</b> and <b>8</b>A-<b>8</b>D.) Note that the simple strings S₂and S₃are to be detected ahead of the variable strings V₁and V₂in accordance with the look-ahead notion of the LaFA system.

Case 2:

0068If a RegEx includes simple strings S and “sequential type” variable string(s) V_seqonly, then the RegEx are not reordered. For example, the RegEx: abc[a-z]{3}op→S₁: abc, V₃: [a-z]{3}, S₃: op→RegEx: S₁V₃S₃should not be reordered since the V₃is a sequential type variable string V_seq. The information obtained directly from this RegEx (non-reordered) is used for configuring the LaFA state machine such as configuring the nodes and their information in the correlation block.

Case 3:

0069If a RegEx includes simple strings S and both look-ahead type variable string(s) V_1aand sequential type variable string(s) V_seq, then the rule for reordering is as follows. Examining each component and starting from the first component (leftmost component) of the RegEx and moving/scanning towards the right components, for every variable string V_1a(or group of adjacent V_1aonly) that is directly followed by a simple string S or variable string V_seq, a swap in component positions between the encountered V_1a(or group of adjacent V_1aonly) and the directly following simple string S or variable string V_seqshall be performed, resulting in a reordered RegEx. Basically, case 3 is similar to case 1, with the variable string V_seqtreated like another simple string S (while applying the rules of case 1).

0070As an example given for the above case 3, consider the regular expression: RegEx: abc[a-z][a-z]{3}[0-9]{2}op→S₁: abc, V₁: [a-z], V₃: [a-z]{3}, V₄: [0-9]{2}, S₃: op→RegEx: S₁V₁V₃V₄S₃. Note that V₃and V₄are sequential type variable string(s) V_seq, whereas V₁is a look-ahead type variable string(s) V_1a. Applying the reordering rule of case 3, discussed above, to the RegEx: S₁V₁V₃V₄S₃→ results in the reordered RegEx: S₁V₃V₁V₄S₃.

0071The information obtained from this reordered RegEx is used for configuring the LaFA state machine such as configuring the nodes and their information in the correlation block. (See FIGS. <b>5</b> and <b>8</b>A-<b>8</b>D.) Note that variable string V₃is to be detected ahead of the variable strings V₁in accordance with the look-ahead notion of the LaFA system.

0072It is worth noting that variable strings V_1aand V_seqmay be further classified into subtypes of variable strings (which will be provided as a table below). The exemplary LaFA system identifies the subtype of variable strings in a RegEx. These subtypes belong either to the V_1aor V_seqof variable strings. The LaFA system subsequently reorders the RegEx as discussed with the cases provided above. Information, such as component type (simple string or variable string), variable string subtype (belonging to V_1aor V_seq), and any reordered components of a RegEx are used when configuring elements of the exemplary LaFA system.

4.2.3 Architecture and Configuration of the Detection Block for Checking an Input String Against Regular Expressions

0073As mentioned previously, a RegEx set is partitioned into simple strings and variable strings. Simple strings are detected using simple-string detectors <b>530</b> within the detection module <b>510</b>, whereas variable strings are detected using a variable string detector <b>540</b> within the detection module <b>510</b>. (Recall <figref idrefs="DRAWINGS">FIG. 5</figref>.) Also the variable string detector <b>540</b> within the detection block <b>510</b> may include two main types of modules based on the detection approach in the LaFA architecture: (1) buffered lookup modules; and (2) inline lookup modules. The buffered lookup modules are used for detecting look-ahead type variable strings (V_1a) whereas the in-line lookup modules are used for detecting sequential type variable strings (V_seq). Each type of lookup module is described in more detail below.

0074Since a wide variety of variable strings exists, in at least some exemplary embodiments consistent with the present invention, they are further classified into eight subtypes of components as shown in Table 1 below. Each subtype has a qualifier symbol next to the letter V as shown in the first column of Table 1. An example of each type is provided in the right-most column of the table. Each component is also assigned a unique ID, but components that are exactly the same are assigned the same ID even if they belong to different RegExes. Next, various detection modules will be described based on the buffered lookup approach. Referring back to <figref idrefs="DRAWINGS">FIG. 3B</figref>, different optimized modules may be used to detect different string types.

0000<tables id="TABLE-US-00002" num="00002"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="1" colwidth="42pt" align="left" /><colspec colname="2" colwidth="91pt" align="left" /><colspec colname="3" colwidth="84pt" align="left" /><thead><row><entry namest="1" nameend="3" rowsep="1">TABLE 1</entry></row><row><entry namest="1" nameend="3" align="center" rowsep="1" /></row><row><entry>Symbol</entry><entry>Component type</entry><entry>Example</entry></row><row><entry namest="1" nameend="3" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>V_A</entry><entry>Character class (CC)</entry><entry>[a-z], \w</entry></row><row><entry>V_B</entry><entry>Negated CC</entry><entry>[{circumflex over ( )}a-z], [\W]</entry></row><row><entry>V_C</entry><entry>Negated character</entry><entry>[{circumflex over ( )}a], [{circumflex over ( )}\n]</entry></row><row><entry>V_D</entry><entry>Character repetition</entry><entry>[\n]{3, 10}</entry></row><row><entry>V_E</entry><entry>Simple string repetition</entry><entry>abcd{3, 10}</entry></row><row><entry>V_F</entry><entry>CC repetition</entry><entry>[a-z]{3, 10}, [\w]{3, 10}</entry></row><row><entry>V_G</entry><entry>Negated CC repetition</entry><entry>[{circumflex over ( )}a-z]{3, 10}, [\W]{3, 10}</entry></row><row><entry>V_H</entry><entry>Negated character repetition</entry><entry>[{circumflex over ( )}a]{3, 10}</entry></row><row><entry namest="1" nameend="3" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

4.2.3.1 Timestamps Lookup Module (“TLM”)

0075The buffered lookup modules are useful detection modules in the LaFA architecture. The look-ahead operation can be realized by these modules. These modules use history stored in buffers to verify past activity. A TLM stores the incoming character with respect to its time of arrival. This module can detect non-repetition types of variable strings, such as V_A, V_B, and V_c(character class, negated character class and negated character). The TLM may incorporate an input buffer, which stores characters recently received from a packet in chronological order. Using this buffer, the TLM can answer queries such as “Does the character at time t belong to a (negated) character class C?”. For instance, consider the detection process of RegEx<b>1</b>: abc[a-z]lop as an example, and assume the input “abcxop”. For the detection of RegEx<b>1</b>, a lowercase alphabetical character (i.e., from a to z) must be detected between simple strings “abc” and “op”. Detection of a simple string “op” is a trigger to access the TLM buffer memory and verify the character fetched for a particular time (one character before character “0” in this example) against the one received at the input: <ul><li id="ul0001-0001" num="0000"><ul><li id="ul0002-0001" num="0076">RegEx<b>1</b>: abc[a-z]op|S₁V₁S₂</li><li id="ul0002-0002" num="0077">Input: a b c <u>x</u> o p</li><li id="ul0002-0003" num="0078">Time: 1 2 3 <u>4</u> 5 6</li></ul></li></ul>

0079Also, assume for the time being that detection sequence and detection timing have already been verified by the correlation block. The last portion that needs to be verified is whether the input character between “abc” and “op” (i.e., the input character at time <b>4</b>) was a lowercase alphabetical character or not. In the example above, it can be seen that since at time <b>4</b> a lowercase letter “x” appears in the input, RegEx<b>1</b> is detected.

0080In a formal notation, let T_TLM=(T₁, T₂, T₃. . . T_l) be an ordered set of the TLM contents while l is the length of the character buffer. In the example, variable string [a-z] is located at time <b>4</b>. Receiving character in time <b>4</b> is verified. The query is in a statement in the form of [a,z]∩T₄≠0” to check for its validity.

4.2.3.2 Character Lookup Module (“CLM”)

0081A CLM is responsible for the detection of V_Htype (negated character repetition) variable strings. This is one type of variable string that causes traditional FAs to degrade to impractical architectures. An example follows to illustrate the detection procedure of the CLM using RegEx<b>5</b>: abc[̂x]{3,5}op and assuming the input “abcdefxop”: <ul><li id="ul0003-0001" num="0000"><ul><li id="ul0004-0001" num="0082">RegEx<b>5</b>: abc[̂x]{3,5}op|S₁V₂S₂</li><li id="ul0004-0002" num="0083">Input: a b c <u>d</u><u>e</u><u>f</u><u>x</u> o p</li><li id="ul0004-0003" num="0084">Time: 1 2 3 <u>4</u><u>5</u><u>6</u><u>7</u> 8 9</li></ul></li></ul>

0085Also assume that the detection order and timing have already been verified at the correlation block. The CLM has multiple memory locations to store detection timestamps of each character of the input. For example, the first character “a” of the input is detected at time <b>1</b>, so timestamp <b>1</b> is stored for character “a”. Timestamp <b>2</b> will be stored for character “b”, and so on. To match the input to RegEx<b>5</b>, character “x” should not appear from time <b>4</b> to time <b>7</b>. Assume that this time period is to be called the “inviolable period”. To verify that information, the CLM uses timestamps stored previously. In the example above, “x” was detected at time <b>7</b> and the time is stored at character timestamp memory in the CLM. From the above information, it is clear that the example input“abcdefxop” does not match RegEx<b>5</b> because “x” was detected during the inviolable period at time <b>7</b>.

0086In a formal notation of the ASCII values, let C_CLM={C₀, C₁, . . . C_i} (i can be 0 to 255) be an unordered set. Some components can be empty (C_x=0). Each element C_xstores timestamps (e.g., C₁₂₀={t₆} shows component C₁₂₀(lowercase “x”) appeared at time <b>6</b>). The query is in a statement in the form of “Is [t₄, t₆]∩C₁₂₀=0” to check for its validity.

0087For some RegExes, more than one timestamp needs to be stored per ASCII character. To clarify this need, consider the following example: <ul><li id="ul0005-0001" num="0000"><ul><li id="ul0006-0001" num="0088">RegEx<b>6</b>: abc[̂x]{3}xyz|S₁V₄S₃</li><li id="ul0006-0002" num="0089">Input: a b c <u>x</u><u>x</u><u>x</u> x y z</li><li id="ul0006-0003" num="0090">Time: 1 2 3 <u>4</u><u>5</u><u>6</u> 7 8 9</li></ul></li></ul>

0091This example illustrates a situation that requires more than one timestamp entry. Character “x” appears four times at times <b>4</b>, <b>5</b>, <b>6</b>, and <b>7</b>. The CLM stores the timestamps. However, timestamps <b>4</b>, <b>5</b>, and <b>6</b> are overwritten and only timestamp <b>7</b> is stored because only one timestamp entry is assigned. In this situation, RegEx<b>6</b> element [̂x]{3} cannot be verified correctly. At least two timestamp entries must be reserved in CLM for this example RegEx<b>6</b>.

4.2.3.3 Repetition Detection Module (“RDM”)

0092An RDM is an in-line lookup module used for detecting sequential-type variable strings V_seq. The RDM is responsible for detecting repetitions that are not detected by the CLM (variable strings of type V_D, V_E, V_F, and V_G(character repetition, simple string repetition, character class repetition and negated character class repetition)). More formally, RDM detects components in the form base {x,y} by accepting consecutive repetitions of the base, x to y times in the input. Here, base can be a single character, a character class, or a simple string. The {x,y} shows a range of repetition, where x is the minimum and y is the maximum repetitions. The RDM is the only inline detection module. The RDM includes several identical sub-modules, and each sub-module can detect character classes and negated character classes with repetitions. These sub-modules operate in an on-demand manner.

0093Assume a RegEx detection process is in progress and the following component to be inspected is a character class or negated character class repetition. The correlation block sends a request to the RDM for the detection of this component. The request consists of Base ID and the Minimum and Maximum repetition boundaries, where the base is represented by a Pattern ID. The RDM assigns one of the available sub-modules for detecting this component. The assigned sub-module then inspects the corresponding input characters to see whether they all belong to the base range and if the number of repetitions is between the minimum and maximum repetition boundaries. Once the number of repetitions reaches the minimum repetition value, a next simple string is activated. The next simple string is inactivated when number of repetitions reaches the maximum repetition value.

0094As an example of the RDM detection procedures, consider an example of detecting RegEx<b>7</b>: abc[a-z]{3,5}op against the character input “abcxxxxop”. <ul><li id="ul0007-0001" num="0000"><ul><li id="ul0008-0001" num="0095">RegEx<b>7</b>: abc[a-z]{3,5 }op|S₁V₃S₂</li><li id="ul0008-0002" num="0096">Input a b c <u>x</u><u>x</u><u>x</u><u>x</u> o p</li><li id="ul0008-0003" num="0097">Time: 1 2 3 <u>4</u><u>5</u><u>6</u><u>7</u> 8 9</li></ul></li></ul>

0098At time <b>3</b>, simple string S₁: “abc” is detected. The following repetition component [a-z]{3,5} is programmed in the RDM sub-module immediately and starts checking the input character. From time <b>4</b> to time <b>6</b> consecutively, the module receives three characters that match the base ([a-z]). Therefore, the minimum repetition value is satisfied and subsequently the sub-module sends an activate signal to the next component (simple string S₂). Now simple string S₂: “op” is ready to receive an event. However, this is still not the end. The sub-module continues checking the input characters. The following characters continuously match the base until the maximum repetition value is reached (i.e., V₃is detected). If there is no match in this interval, the sub-module deactivates the next component (S₂: “op” in the example case). However, in the given example all components are detected and the input matched the RegEx<b>7</b>.

0099An optimized module that may be used to improved the performance of the RDM is described in §4.2.3.3.1 below.

4.2.3.3.1 Frequently Appearing Repetition Detection Module (“FRM”)

0100In practice, there are frequently used repetition bases (frequent bases). This creates an opportunity for sharing the detection effort for these frequent bases, further reducing resource usage. A frequently appearing repetition detection module (“FRM”) can detect any type of repetition using a buffered lookup approach (variable strings of type V_D, V_E, V_F, V_G). When analyzing the RegEx sets, popularity of base in the RegEx set is ranked. For instance, [̂\n\r] is the base that most frequently appears in the Snort RegEx set. Both FRM and RDM verify repetition components, but FRM can be shared among multiple RegExes. This improves resource efficiency.

0101Thus, in the configuration phase, popular bases may be programmed in FRMs rather then RDMs. The base is programmed in memory, so updating the base does not require logic modification. Each FRM has a counter that counts the number of consecutive input characters/simple strings that match the base. When the sequence breaks, the count and timestamp are stored in a history memory inside of the FRM. Thus, for the query operation, repetition behavior can be found in a particular time range from the number of repetitions with the timestamp.

4.2.3.4 Short Simple String Detection Module (“SDM”)

0102Referring to the left column of <figref idrefs="DRAWINGS">FIG. 3B</figref>, some exemplary embodiments employing LaFA may classify simple strings even further as “regular” or “normal” simple strings (referred to as “normal simple strings”) and “short simple strings”. Short strings are classified as “shallow” components. (See <figref idrefs="DRAWINGS">FIGS. 3A and 3B</figref>.) Shorter strings naturally are more likely to generate more “events” as described in the following section. To reduce occurrence of events generated by the simple string, some exemplary embodiments employing LaFA may use another buffered lookup type of module called Short Simple String Detection Module (“SDM”). To maximize the advantage of the event-based detection approach, it is useful to select an appropriate signature size (the number of characters in a simple string signature). If one imagines that the signature table in the simple string detector contains many one character signatures, then a simple-string detector may generate an event for every single character input. (This is the reason that traditional RegEx detection architectures suffer.) Assume all characters arrive with the same probability and the signature table has s kinds of signatures. The probability to generate an event by receiving one character is

0000<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><mfrac><mi>s</mi><mn>256</mn></mfrac><mo>.</mo></mrow></math></maths>

0000The probability will be reduced exponentially,

0000<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mfrac><mi>s</mi><msup><mn>256</mn><mi>l</mi></msup></mfrac></math></maths>

0000as the number of characters, l, increases. Strings with at least a predetermined number (e.g., length=5) characters may be classified as a “normal simple strings”, and strings with less than that may be classified as a “short simple strings”.

4.3 CHECKING AN INPUT STRING AGAINST REGULAR EXPRESSIONS FOR A MATCH USING CONFIGURED STATE MACHINES

0103<figref idrefs="DRAWINGS">FIG. 14</figref>, which includes <figref idrefs="DRAWINGS">FIGS. 14A-14C</figref>, illustrates a flow diagram of an exemplary string matching method <b>1400</b>, consistent with the present invention. (Recall method <b>700</b> of <figref idrefs="DRAWINGS">FIG. 7</figref> and operation <b>140</b> of <figref idrefs="DRAWINGS">FIG. 1</figref>.) Referring first to <figref idrefs="DRAWINGS">FIG. 14A</figref>, an input character string is received and provided to a simple string detector. (Blocks <b>1402</b> and <b>1404</b>) Referring to event block <b>1406</b>, if a match is received from the simple string detector, the exemplary method <b>1400</b> continues to <figref idrefs="DRAWINGS">FIG. 14B</figref> via “A” <b>1408</b>. (Recall, for example, <b>535</b><i>a </i>of <figref idrefs="DRAWINGS">FIG. 5</figref>. The other event is described later.)

0104Referring to <figref idrefs="DRAWINGS">FIG. 14B</figref>, as indicated by the loop <b>1412</b>-<b>1432</b>, a number of acts are performed for each node corresponding to the matched simple string. More specifically, it is first determined if the node is active. (Block <b>1414</b>) If not, the method <b>1400</b> proceeds to <b>1432</b> where it is determined whether or not there is another node corresponding to the matched simple string. Referring back to decision block <b>1414</b>, if, on the other hand, it is determined that the node is active, then the timing is checked using the time range information (minimum time and maximum time) stored in the node. (Block <b>1416</b>) If the timing is not valid, then the violation type (e.g., <minimum or >maximum) is determined. If the timing violation is because the time is less than the minimum time value stored, the exemplary method <b>1400</b> proceeds to block <b>1432</b> (already discussed). If, on the other hand, the timing violation is because the time is greater than the maximum time value stored, the exemplary method <b>1400</b> determines if the node is the first node of the regular expression. (Decision block <b>1422</b>) If so, the exemplary method <b>1400</b> proceeds to block <b>1432</b> (already discussed); otherwise the exemplary method <b>1400</b> deactivates the node (Block <b>1424</b>) before proceeding to block <b>1432</b>.

0105Referring back to decision block <b>1418</b>, if the timing is valid, then it is determined whether or not the node is the first node of the regular expression. If not, the node is deactivated (Block <b>1428</b>) before the exemplary method <b>1400</b> proceeds to block <b>1430</b>; otherwise, the exemplary method <b>1400</b> proceeds directly to block <b>1430</b>. At block <b>1430</b>, the method <b>1400</b> activates the node identified by a stored pointer (and any necessary information is passed from the current node to the node identified by the pointer). The exemplary method then proceeds to <b>1432</b>.

0106At <b>1432</b>, it is determined if there are any more nodes corresponding to the matched simple string. If so, the method <b>1400</b> loops back to <b>1412</b>; otherwise, the method is left. (Via return <b>1434</b>)

0107Referring to both block <b>1430</b> of <figref idrefs="DRAWINGS">FIG. 14B</figref> and event block <b>1406</b> of <figref idrefs="DRAWINGS">FIG. 14A</figref> (as well as block <b>1460</b> of <figref idrefs="DRAWINGS">FIG. 14C</figref>, to be discussed below), if a node is activated by another node (Recall, e.g., <b>535</b><i>b </i>of <figref idrefs="DRAWINGS">FIG. 5</figref>.), the exemplary method <b>1400</b> continues to <figref idrefs="DRAWINGS">FIG. 14C</figref> via “B” <b>1410</b>. First, it is determined whether the activated node is a sequential string node. (Decision block <b>1440</b>) If so, the method <b>1400</b> is left. (Return <b>1434</b>). Otherwise, the exemplary method <b>1400</b> then determines a module identifier stored in the activated node. (Block <b>1442</b>) As indicated by block <b>1444</b>, various branches of the exemplary method <b>1400</b> are performed depending upon the module identifier determined. More specifically, if the module identifier identifies the TLM, the exemplary method <b>1400</b> queries the TLM with information stored in the activated node and information (e.g., time stamp of detection) passed from the node that activated the activated node, before the method <b>1400</b> continues to decision block <b>1456</b> (described later). (Block <b>1446</b>) If, however, the module identifier identifies the CLM, the exemplary method <b>1400</b> queries the CLM with information stored in the activated node and information (e.g., time stamp of detection) passed from the node that activated the activated node, before the method <b>1400</b> continues to decision block <b>1456</b> (described later). (Block <b>1448</b>) If, however, the module identifier identifies the RDM, the exemplary method <b>1400</b> queries the RDM with information stored in the activated node and information (e.g., time stamp of detection) passed from the node that activated the activated node, before the method <b>1400</b> continues to decision block <b>1456</b> (described later). (Block <b>1450</b>) If, however, the module identifier identifies the FRM, the exemplary method <b>1400</b> queries the FRM with information stored in the activated node and information (e.g., time stamp of detection) passed from the node that activated the activated node, before the method <b>1400</b> continues to decision block <b>1456</b> (described later). (Block <b>1452</b>) Finally, if the module identifier identifies the SDM, the exemplary method <b>1400</b> queries the SDM with information stored in the activated node and information (e.g., time stamp of detection) passed from the node that activated the activated node, before the method <b>1400</b> continues to decision block <b>1456</b> (described later). (Block <b>1454</b>) Note that exemplary TLMs, CLMs, RDMs, FRMs and SDMs were described in §4.3.2 above.

0108Referring to decision block <b>1456</b>, it is determined whether the response to the query has been received. If not, the exemplary method <b>1400</b> may loop back to wait. Otherwise, if the query response is received, it is determined whether or not the query response is positive. (Decision block <b>1458</b>) If not, the exemplary method <b>1400</b> is left. (Via return <b>1434</b>) Otherwise (if the query response is positive), then the node identified by the currently active node's pointer is activated, and any necessary information is passed form the currently active node to the newly activated node (Block <b>1460</b>) before the method <b>1400</b> is left. (Return <b>1434</b>) Note that the node activation at block <b>1462</b> will trigger an event indicating a node activated by another node. (Event block <b>1406</b> of <figref idrefs="DRAWINGS">FIG. 14A</figref>)

4.3.1 Examples of Operations

0109<figref idrefs="DRAWINGS">FIGS. 9A-9C</figref> illustrate configured LaFA state machine detection systems <b>500</b>′, <b>500</b>″, <b>500</b>′″, wherein arbitrary input strings are checked for a match against regular expressions. In particular, LaFA detection systems using TLM, CLM and RDM detection modules, respectively, are illustrated.

0110<figref idrefs="DRAWINGS">FIG. 9A</figref> illustrates a configured LaFA detection system employing a TLM buffered lookup-module wherein an input string: “abcxop” is checked against (r₁) RegEx<b>1</b>: abc[a-z]op. The input string is inserted in the detection block <b>510</b>′ wherein both the simple string detector (e.g., a TriBiCa detector, such as those described in U.S. patent application Ser. Nos. 11/978,216 (which issued as U.S. Pat. No. 7,805,460 on Sep. 28, 2010) and 11/978,185, both incorporated herein by reference) <b>530</b>′ and the variable string detector <b>540</b>′ receive the input “abcxop”<b>505</b>′. The node corresponding to S₁has been activated and awaiting a match. At time <b>3</b>, the simple string detector <b>530</b>′ has detected simple string S₁: “abc” and an event <b>535</b><i>a</i>′ is generated notifying the match to the correlation block <b>520</b>′. Next, the node corresponding to the simple string S₂: “op” is activated and awaiting a match. At time <b>6</b>, the simple string detector <b>530</b>′ has detected simple string S₂and an event <b>535</b><i>a</i>′ is generated notifying the match to the correlation block <b>520</b>′. Next, since S₂has been detected, the correlation block <b>520</b>′ must query the variable string detector <b>540</b>′ to determine if variable string V₁has been detected at the appropriate time, namely time <b>4</b>. Note, that the correlation block <b>520</b>′ includes nodes having information such as: (1) a pointer to the next state; (2) the status of the current state; (3) detection timing values (e.g., minimum time (distance) and maximum time (distance)); (4) module ID; (5) component ID; (6) minimum repetition; and (7) maximum repetition. Such information is used to verify the correct component detection at the correct timing. The variable string detector <b>540</b>′ in this example employs a TLM detector module for the given RegEx<b>1</b>. The TLM module stores the incoming character with respect to its time of arrival as illustrated in the variable string detector of the detection block. The TLM module checks the buffer and confirms that the lowercase character “x” appears at time <b>4</b>. Hence, V₁: “[a-z]” is satisfied and the detection block <b>510</b>′ sends the results to the correlation block wherein a match is finally confirmed.

0111<figref idrefs="DRAWINGS">FIG. 9B</figref> illustrates a configured LaFA detection system <b>500</b>″ employing a CLM buffered lookup-module wherein an input string: “abcdefxyz” <b>505</b>″ is checked against (r₂) RegEx<b>2</b>: abc[̂x]{3}xyz. The input string <b>505</b>″ is inserted in the detection block <b>510</b>″ wherein both the simple string detector (e.g., a TriBiCa detector) <b>530</b>″ and the variable string detector <b>540</b>″ receive the input “abcdefxyz” <b>505</b>′. The node corresponding to S₁has been activated and awaiting a match. At time <b>3</b>, the simple string detector <b>530</b>″ has detected simple string S₁: “abc” and an event <b>535</b><i>a</i>″ is generated notifying the match to the correlation block <b>520</b>″. Next, the node corresponding to the simple string S₃: “xyz” is activated and awaiting a match. At time <b>9</b>, the simple string detector <b>530</b>″ has detected simple string S₃and an event <b>535</b><i>a</i>″ is generated notifying the match to the correlation block <b>520</b>″. Next, since S₃has been detected, the correlation block <b>520</b>″ must query the variable string detector <b>540</b>″ to determine if variable string V₂has been detected at the appropriate time (namely character “x” should not appear from time <b>4</b> to time <b>6</b>). Note, that the correlation block <b>520</b>″ comprises nodes having information such as: (1) a pointer to the next state; (2) the status of the current state; (3) detection timing values (e.g., minimum time (distance) and maximum time (distance)); (4) module ID; (5) component ID; (6) minimum repetition; and (7) maximum repetition. Such information is used to verify the correct component detection at the correct timing. The variable string detector in this example case employs a CLM detector module for the given RegEx<b>2</b>. The CLM module has multiple memory locations to store detection timestamps of each character of the input as illustrated in the variable string detector of the detection block. The CLM module checks the buffer and confirms that the lowercase character “x” does not appears at from time <b>4</b> to time <b>6</b>. In particular, the CLM module determines from the character look-up buffer that “x” was detected at time. Hence, V₂: “[̂x]{3}” is satisfied. The detection block <b>510</b>″ sends the results to the correlation block <b>520</b>″, where a match is finally confirmed.

0112<figref idrefs="DRAWINGS">FIG. 9C</figref> illustrates a configured LaFA detection system <b>500</b>′″ employing a RDM in-line lookup-module wherein an input string: “abcxxxop” <b>505</b>′″ is checked against (r₃) RegEx<b>3</b>: abc[a-z]{3}op. The input string <b>505</b>′″ is inserted in the detection block <b>510</b>′″ wherein both the simple string detector (e.g., a TriBiCa detector) <b>530</b>′″ and the variable string detector <b>540</b>′″ receive the input “abcxxxop” <b>505</b>′″. The node corresponding to S₁has been activated and awaiting a match. At time <b>3</b>, the simple string detector <b>530</b>′″ has detected simple string S₁: “abc” and an event <b>535</b><i>a</i>′″ is generated notifying the match to the correlation block <b>520</b>′″. Note that RegEx<b>3</b>: abc[a-z] {3}op has a V_seq-type variable string V₃: “[a-z]{3}” only. Therefore, RegEx<b>3</b> cannot be reordered and a “look-ahead” cannot be employed. Next, since S₁has been detected, the correlation block <b>520</b>′″ queries the variable string detector <b>540</b>′″ to determine if the V₃has been detected. Note, that the correlation block <b>520</b>′″ includes nodes having information such as: (1) a pointer to the next state; (2) the status of the current state; (3) detection timing values (e.g., minimum time (distance) and maximum time (distance)); (4) module ID; (5) component ID; (6) minimum repetition; and (7) maximum repetition. Such information is used to verify the correct component detection at the correct timing. The variable string detector <b>540</b>′″ in this example case employs an RDM detector module for the given RegEx<b>1</b>. The repetition detection sub-module in the variable string detector checks the incoming input characters in order to verify V₃: “[a-z]{3}”. Since, the input character “x” repeats 3 times at times <b>4</b>, <b>5</b> and <b>6</b>, component V₃is verified and the detection block <b>510</b>′″ sends the event/results to the correlation block <b>520</b>′″, where the node corresponding to simple string S₂is activated and awaiting a match. Next, at time <b>8</b>, the simple string detector <b>530</b>′″ has detected simple string S₂:“op” and an event <b>535</b><i>a</i>′″ is generated, notifying the match to the correlation block <b>520</b>′″ where a match between the input and RegEx<b>3</b> is finally confirmed.

4.4 EXEMPLARY APPARATUS

0113<figref idrefs="DRAWINGS">FIG. 10</figref> is high-level block diagram of a machine <b>1000</b> that may perform one or more of the processes described above, and/or store information used and/or generated by such processes. The machine <b>1000</b> basically includes one or more processors <b>1010</b>, one or more input/output interface units <b>1030</b>, one or more storage devices <b>1020</b>, and one or more system buses and/or networks <b>1040</b> for facilitating the communication of information among the coupled elements. One or more input devices <b>1032</b> and one or more output devices <b>1034</b> may be coupled with the one or more input/output interfaces <b>1030</b>. The one or more processors <b>1010</b> may execute machine-executable instructions (e.g., C or C++ running on the Solaris operating system available from Sun Microsystems Inc. of Palo Alto, Calif. or the Linux operating system widely available from a number of vendors such as Red Hat, Inc. of Durham, N.C.) to effect one or more aspects of the present invention. At least a portion of the machine executable instructions may be stored (temporarily or more permanently) on the one or more storage devices <b>1020</b> and/or may be received from an external source via one or more input interface units <b>1030</b>.

0114In one embodiment, the machine <b>1000</b> may be one or more conventional personal computers, servers, or routers. In this case, the processing units <b>1010</b> may be one or more microprocessors. The bus <b>1040</b> may include a system bus. The storage devices <b>1020</b> may include system memory, such as read only memory (ROM) and/or random access memory (RAM). The storage devices <b>1020</b> may also include a hard disk drive for reading from and writing to a hard disk, a magnetic disk drive for reading from or writing to a (e.g., removable) magnetic disk, and an optical disk drive for reading from or writing to a removable (magneto-) optical disk such as a compact disk or other (magneto-) optical media.

0115A user may enter commands and information into the personal computer through input devices <b>1032</b>, such as a keyboard and pointing device (e.g., a mouse) for example. Other input devices such as a microphone, a joystick, a game pad, a satellite dish, a scanner, or the like, may also (or alternatively) be included. These and other input devices are often connected to the processing unit(s) <b>1010</b> through an appropriate interface <b>1030</b> coupled to the system bus <b>1040</b>. The output devices <b>1034</b> may include a monitor or other type of display device, which may also be connected to the system bus <b>1040</b> via an appropriate interface. In addition to (or instead of) the monitor, the personal computer may include other (peripheral) output devices (not shown), such as speakers and printers for example.

4.6 REFINEMENTS, ALTERNATIVES AND EXTENSIONS

4.6.1 Multiple Detection Tracks

0116In the previous sections, the exemplary LaFA data structures and architectures were illustrated with one set of detection modules (a detection block and a correlation block), which is referred to as a “track” in the LaFA architecture. For some large RegEx sets, having more than one track allows constant detection time regardless of the input traffic. As illustrated in <figref idrefs="DRAWINGS">FIG. 11</figref>, two simultaneous activation processes are invoked for S₂and S₃, in response to a detection event of S₁. This can slow down the detection speed if those operations are performed sequentially. The LaFA scheme avoids this drawback by spreading the two RegExes, RegEx<b>1</b> and RegEx<b>4</b>, into two tracks, and performing the detection operation in parallel as shown in <figref idrefs="DRAWINGS">FIG. 11</figref>. As a result, regardless of the input traffic, the LaFA scheme will always need at most one operation per each track and all tracks work in parallel. <figref idrefs="DRAWINGS">FIG. 11</figref> also illustrates how an exemplary LaFA architecture stores component information in a node memory. The nodes of S₃and V₂are stored at consecutive locations in the node memory since they will be verified successively, thereby achieving higher access speeds. Since simple strings like S₁and S₃are independent from each other and any other component, they can be stored anywhere in the memory. Such independent nodes are stored in a way that helps balance the memory size. Locations of simple strings are resolved by pointers.

4.6.2 Block Detection Optimization

0117In this section, an optimized detection method referred to as “block detection” is described. Using block detection can reduce bottlenecks due to multiple concurrent operations significantly, and can also reduce requirements of node information memories. <figref idrefs="DRAWINGS">FIG. 12A</figref> illustrates three RegExes and <figref idrefs="DRAWINGS">FIG. 12B</figref> shows the same RegExes in LaFA notation. All the RegExes in this example start with S₇followed by V₄. The detection of S₇activates three simple strings, S₈, S₉, and S₁₀, and causes multiple concurrent activation operations. In <figref idrefs="DRAWINGS">FIG. 12B</figref>, the three RegExes share the same components, back-to-back, (S₇and V₄). This is represented as B₁(a buffered lookup type) in <figref idrefs="DRAWINGS">FIG. 12C</figref>. Assume that this pair of components is called block B₁, and that the detection sequence is reordered such that S₈, S₉, and S₁₀are detected first. Only if S₈, S₉, or S₁₀is detected then whether B₁is detected is verified. This can be implemented as follows. The detection time of B₁will be stored in a block detection module having the same architecture as a CLM. When the following simple string is detected (for example S₈), the B₁is verified. By using in such a detection, three concurrent operations are reduced to one, and the number of tracks can be reduced more than 60% for these RegExes. It is more effective to reduce the concurrent operation at the architecture level so that number of tracks is minimized. However, some exemplary embodiments of LaFA use multiple parallel tracks, with each track having (at most) one operation at any given time.

4.6.3 Time Event

0118<figref idrefs="DRAWINGS">FIGS. 13A and 13C</figref> illustrate an exemplary LaFA representation of two RegExes—RegEx<b>10</b> and RegEx<b>11</b>. All variable strings, V₄, V₅, V₆, are buffered lookup type (or V_1a). RegEx<b>10</b> is a normal case and LaFA can perform verification operations after detecting simple string S₁₀. On the other hand, RegEx<b>11</b> does not have any simple string following to the two variable strings V₅, V₆, so LaFA does not know the right timing to verify the variable strings. Supporting this type of RegExes, a new type of event, referred to as “time event” and symbolized as T₀in <figref idrefs="DRAWINGS">FIG. 13C</figref>, is exploited. In the example case, variable strings V₅and V₆need the verification of 3001 character after detection of S₁₀, which comes from the sum of the two variable strings, 301 (=1+300) LaFA activates a timer to execute this type of event, after detecting S₁₀. The timer generates a time-event after a defined time. At this time, LaFA checks to verify V₅and V₆. Even if V₆is not an exact repetition type of a component, LaFA can be configured to issue a time event in minimum repetition timing. If the intermediate variable string (e g, V₅) is not an exact repetition, an in-line lookup module is assigned instead of a buffered lookup to verify the variable string so that this type of RegEx is also supported.

4.6.4 Online Regular Expression Update

0119An important property of the exemplary LaFA schemes is that they permit fast updates to be made. Traditionally, when a new RegEx is available, the RegEx Detection System should be taken offline and updated with this new RegEx. This requires either deploying another system for backup or leaving the network “open” to attacks during the updates. The former solution is costly, whereas the latter is not secure. Thus, fast online updates are very desirable.

0120At least some embodiments of LaFA treat each RegEx independently. Thus, updating, deleting, or adding RegExes does not affect the other memory contents or the detection process of any other RegExes. The only places that need to be updated are the simple string detector and a node memory that is related to the RegExes. In addition, offline node construction is performed locally. In DFA-based methods, the data structures are very strongly coupled, so that an update in one RegEx may affect many states/transitions in the DFA. Thus, with DFA, any update has a potential to change most of the data structure, taking longer to complete. In traditional NFA-based approaches, the data structures are hard-coded on the logic gates. Thus, with NFA, any update requires a hardware reconfiguration, which also takes long and performance of the updated logic is not guaranteed. Thus, the update time of LaFA is significantly less than other FA schemes, and LaFA is more flexible with adding new RegExes.

4.7 CONCLUSIONS

0121Embodiments consistent with the present invention provide a novel data structure (namely, a highly optimized “look-ahead” finite automata) which provides a scalable and highly optimized regular expression detection system and which supports low-cost set membership queries. By using such a data structure, matching can be checked at high speed.