A method and system for enforcing a security policy via a security virtual machine

The present invention generally relates to enforcing security policies to block unwanted behavior by computer programs.

Software systems, such as operating systems and file systems, provide an application programming interface through which application programs can access the services of the software system. The application program interface can provide functions with parameters to execute the specified service. For example, a file system can have a function (also called a "system call") for creating a file. The function can have parameters that specify the file location, file name, file type, file size, etc. of the created file. When the application program calls a function, it passes the actual parameters that correspond to the formal parameters defined for the function. The function can perform certain checks on the actual parameters to ensure that those parameters are valid. For example, a file creation function can guarantee that the specified file type is valid. If the parameter is not valid, the function returns the error to the application program.

Software systems can be extremely complex for a variety of reasons. A software system may attempt to be backward compatible with all previous versions of that software system. In such cases, the software system may need to support all functions of the previous version of the application program interface. Since new functions are usually added to each version, the number of functions can be very large and the interaction of functions can be complicated. Some software systems may also contain software components developed by different departments of the same company or by different companies. The interaction of these components can also be very complex.

<p> It is becoming increasingly important for software systems to ensure that their application programming interfaces are not vulnerable to either inadvertent misuse or intentional attack. One vulnerability in a software system may be through a parameter in its application programming interface. When an earlier version of a function is combined with a newer version of the function, or when components from different developers are integrated, validation of existing parameters performed by the function will ensure that the function behaves correctly. It may not be enough to guarantee. For example, a file system application programming interface developed by one company may be integrated with a file server developed by another company. The maximum file size parameter of the application programming interface may be larger than the size supported by the file server. In such cases, the file size applicable to the application programming interface can cause problems with the file server. As another example, a system administrator may want to limit the maximum file size even further, but the system administrator may not have the means available to enforce that limit. There is.</p><p> It would be desirable to have a security policy, and in particular a mechanism for enforcing a security policy that specifies valid conditions for the parameters of the application program interface.</p>

<p> A security virtual machine provides a method and system for enforcing a security policy. Security virtual machines include a processor engine, an instruction store, a data store and an instruction pointer. The security virtual engine runs a specified security program using an intermediate language compiled from a high-level language representation of a security policy. The security program is loaded into the instruction store for execution. When a security enforcement event occurs, such as an application programming interface function call, the data from the security enforcement event is stored in the datastore and the processor engine begins fetching and executing instructions for the security program from the instruction store. To do. The instruction specifies the action to be taken to enforce the security policy, based on the data of the security enforcement event.</p>

Security Provides methods and systems in computer systems for enforcing encoded security policies within a virtual machine's instruction set. In one embodiment, the security system provides a security virtual machine that executes a security program represented within the security virtual machine's instruction set (ie, an intermediate language). A security system can include a compiler that receives a security policy expressed in a high-level language and generates a security program that is an expression of the security policy in the intermediate language of a security virtual machine. The security system stores the security program in the instruction store of the security virtual machine. When a security enforcement event occurs (ie, an action that needs to be checked to ensure that it compiles with the security policy), the security virtual machine uses the data from the security enforcement event to store its instructions. Execute the command for the security program from and implement the security policy. If a security enforcement event indicates that an attempt is being made to perform an undesired behavior (eg, an action that could exploit an operating system vulnerability), the security program blocks the attempt. be able to. In one embodiment, the security virtual machine of the security system runs in kernel mode of the operating system and attempts by applications and external systems to perform undesired behavior for the computer system on which the security system is running. To identify and prevent.

In one embodiment, the security system identifies when the parameters of a system call issued by an application to a system service (eg, a file system and a memory management system) can lead to undesired behavior. A security policy contains conditions based on system call parameters and rules that specify actions to take when the conditions are met. For example, the conditions of a rule may be met when a file creation system call is issued with a parameter that specifies a file size greater than 1GB. The action associated with that rule can be made to block the creation of the file. High-level languages can be XML-based languages, and each rule can be identified by rules, conditions, and action tags. When the rules of a security policy expressed in a high-level language are compiled, each rule is translated into an intermediate language instruction to form a security program. For example, a rule can be transformed into a set of instructions. Print a system call to compare a value indicating that the function is "create file", a file size parameter to a constant with a value of 1GB, and instructions to block the system call. It is a thing.

In one embodiment, the security virtual machine includes a processor engine, an instruction store, an instruction pointer and a data store. When you initialize a security virtual machine and enforce a security policy, the security system loads the instruction store with a security program that enforces the security policy. The security system also loads the security program data into the data store. When a system call is received, the security system stores the system call parameters (including values that identify the system call) in the data store. The security system can also store process control information for the process that issued the system call in the data store. The security system initializes the instruction pointer to point to the instruction to start the security program. The processor engine starts a security program by fetching and executing the start instruction pointed to by the instruction pointer. Each instruction to be executed modifies the instruction pointer to point to the next instruction to be executed. The instruction refers to the data in the data store and enforces the security policy. Execution of a security program produces an output action set that specifies the actions that should be taken to enforce the security policy (eg, blocking system calls).

In one embodiment, the security policy can include a sub-policy for each system call whose parameters should be checked. The security system can compile each subpolicy into a separate security subprogram, and these separate security subprograms can be loaded into the instruction store independently of the other security subprograms. Each subpolicy may correspond to a security enforcement for a single system call. The security system can maintain a mapping of each system call to the start instruction pointer in the instruction store and to the start data pointer to the data in the data store of the corresponding security subprogram. When implementing a security policy in a system call, the security system initializes the instruction pointer to the corresponding start instruction pointer and the data pointer to the corresponding start data pointer. Security subprograms can use instruction and data reference techniques related to instruction pointers and data pointers. In this way, the instructions and data of each security subprogram can be rearranged. In the following description, a single security program handles all system calls by decoding each system call and jumping to the appropriate instruction location to handle that system call. It will be appreciated by those skilled in the art that this described system can be adapted to support subprograms for each system call.

FIG. 1 is a block diagram illustrating the components of the security system of one embodiment. The security system includes some components that run in user mode 100 and other components that run in kernel mode 120. Policy compiler 102 is a component that runs in user mode and compiles the security policy of policy store 103 into a security program that can be executed by a security virtual machine. Kernel mode components include a system call intercept component 121, a program load component 122, and a security virtual machine 125. The security virtual machine includes an instruction pointer 126, a processor engine 127, an instruction store 128, and a data store 129. The program load component loads the security program compiled by the policy compiler into the instruction store of the security virtual machine, stores the program data in the data store, and sets the start instruction pointer. The system call intercept component intercepts the system call issued by the application program 101, and stores the parameters of each system call and the process control information of the application program in the data store. After intercepting the system call and storing its parameters, the system call intercept component instructs the processor engine to execute the instruction store security program. The processor engine executes a security program by fetching the instruction pointed to by the instruction pointer and performing the operation specified by the instruction. The operation involves storing the actions to be taken to enforce the security policy in the datastore's output action set. Security

Computing devices that implement security systems include central processing units, memory, input devices (eg keyboards and pointing devices), output devices (eg display devices), and storage devices (eg disk drives). Can be done. A memory and storage device is a computer-readable medium that can contain instructions that implement a security system. In addition, data structures and message structures can be stored or transmitted via data transmission media such as signals on communication links. Various communication links can be used, such as the Internet, local area networks, wide area networks, or point-to-point dial-up connections.

Figure 1 illustrates an example of a suitable operating environment in which a security system can be implemented. This operating environment is only one embodiment of a suitable operating environment and is not intended to imply any limitation on the use or scope of functionality of the security system. Other well-known computing systems, environments and configurations that may be suitable for use include personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, etc. Includes networked PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and more.

Security systems can generally be described in relation to computer executable instructions executed by one or more computers or other devices, such as program modules. Program modules typically include routines, programs, objects, components, data structures, etc., which either perform a particular task or implement a particular abstract data type. Generally, the functionality of a program module can be combined or distributed as desired in various embodiments.

FIG. 2 is a block diagram illustrating the data structure of the data store of one embodiment. The data store includes a literal data structure 201, a dynamic data structure 211, a system call parameter data structure 221, a process control block data structure 231 and an action output set 241. The literal data structure includes a literal data table 202 and a literal data store 203. The literal data table contains fixed-size entries that reference literal data stored in the literal data store. "Literal data" refers to data in a security program that is constant during the life of the currently loaded security policy. Literal data can only be modified by loading a new security policy. Dynamic data structures have a similar organization to literal data structures, but store "dynamic" data rather than "literal data". "Dynamic data" refers to security program data that can be modified without loading a new security policy. For example, dynamic data may contain the names of applications that are exempt from security policies. Each entry in the literal and dynamic data tables points to the data in the corresponding data store. Data in literal and dynamic data stores is represented in a self-describing format. The format contains data type and data size information. In one embodiment, the security virtual machine supports Boolean, integer, string and raw binary (also called "blobs") data types in addition to arrays of integers, strings and blobs. The instruction references literal and dynamic data using indexes on literal and dynamic tables. When executing an instruction, the security virtual machine searches for indexes on literal and dynamic tables, and its values for data items in literal or dynamic data stores. Dereference by using it as a pointer to the start. Since the data is self-describing, the type and size of the data can be determined. The system call parameter data structure includes a string table 222, an integer table 223, and a raw binary table 224. System call parameters are stored in these tables according to their data type. A process control block data structure is a table containing entries for each piece of process control information available to a security virtual machine. In one embodiment, the process control information includes an application identifier and a thread identifier. An action output set can be a set of action flags that are generated during the execution of a security program and dictate the actions that should be taken to enforce the security policy in a system call. Actions may include blocking system calls and notifying the user. , Can be a set of action flags. Actions may include blocking system calls and notifying the user. , Can be a set of action flags. Actions may include blocking system calls and notifying the user.

FIG. 3 is a diagram illustrating the layout of the instructions of the security virtual machine of one embodiment. Each instruction includes an operation field 310, a parameter 1 field 320, a true branch field 330, a false branch field 340, and a parameter 2 field 350. The parameter 1 field is the data associated with the system call (ie, stored within the system call data structure or process control data structure) and the parameter 2 field reference data associated with the security program (ie, literal). Or refer to immediate data or data stored in a dynamic data structure). The operation field contains parameter 2 descriptor 311 (m) and operation code 312. The Parameter 2 descriptor specifies how to interpret Parameter 2. Parameter 2 (also called p2) can include a reference to immediate data, a reference to dynamic data, or a reference to literal data. When a reference is specified, the security virtual machine dereferences parameter 2 before performing the operation of the instruction. The operation code identifies the operation of the instruction, as defined in more detail below. The Parameter 1 field contains the Parameter 1 descriptor 321 (s) and the index 322. The parameter 1 descriptor indicates whether parameter 1 refers to system call parameters or process control information. If the parameter 1 descriptor points to a system call parameter, the high-order bits of the index should be indexed on the string table of the system call parameter data structure or on the integer table. , Specifies whether the raw binary table should be indexed, the lower bits indicate the indexed entry of the table. If the parameter 1 descriptor points to process control information, the index is the specific process control. Instruct information. True and false branching fields contain offsets that specify the next instruction to execute, depending on whether the condition code for this instruction evaluates to be true or false. The offset is added to the current instruction pointer to point to the next instruction to be executed.

Tables 1 and 2 illustrate the instructions for the security virtual machine of one embodiment. Instructions can be categorized as those that perform tests on the data and those that do not. Table 1 exemplifies non-test instructions.

<tables num="1"><img file="JP4902129B2_D0001.tif" /></tables>

Table 2 exemplifies the test instructions. The pattern instruction can define a regular expression to compare two strings, which is useful for wildcard type comparisons of filenames (eg "* .DAT").

<tables num="2"><img file="JP4902129B2_D0002.tif" /></tables>

FIG. 4 is a flow chart illustrating the processing of the program load component of one embodiment. The component is called when the security program should be loaded into the security virtual machine. Security programs include virtual instructions, literal data blocks, and dynamic data blocks. The literal data block contains information in the literal data table followed by information in the literal data store. The component starts the literal data block at the starting point and copies it to the literal store. The component then adds its base location to each offset in the literal data table and converts the offset to a pointer. Components process dynamic data blocks in a similar way. Virtual instructions include multiple offsets to the start of literal and dynamic stores. At block 401, the component identifies the starting location in the literal store, starts at that starting location, and copies the literal data block of the security program. At block 402, the component adds a starting location to each offset in the literal data table and converts the security program offset into a pointer. At block 403, the component identifies the starting location in the dynamic store and starts at that starting location to copy the dynamic data block of the security program. At block 404, the component adds a starting location to each offset in the dynamic data table and converts the security program offset into a pointer. At block 405, the component copies the security program's virtual instructions to the instruction store and then compiles them.

FIG. 5 is a flow chart illustrating the processing of the system call intercept component of one embodiment. The component is called when the system call is intercepted. The component initializes the system call parameter data structure and process control data structure, and then starts the security virtual machine. The component can also provide instruction pointers for instructions to initiate execution and process system calls. When a security subprogram is used, a component can have mappings of system calls to start instruction pointers and to start data pointers to literal and dynamic data structures for that system call. You can call the program load component multiple times to load different subprograms of your security program to handle different system calls. Because the instructions and data are offset-based, the instructions and data can be stored in the next available location after the instruction store and data structure. At block 501, the component sets the current instruction pointer (also called ip) to the start instruction pointer of the security program and resets the action output set to its initial state (eg, empty). In blocks 502 to 504, the component loops while storing the system call parameters in the system call parameter data structure. At block 502, the component selects the following parameters for the system call: In decision block 503, if all parameters of the system call are already selected, the component follows block 505, otherwise the component follows block 504. At block 504, the component stores the selected parameters in the appropriate table of the system call parameter data structure and then loops into block 502 to select the next parameter for the system call. In blocks 505 ~ 507, component Loops while storing string process control information in the process control data structure. At block 505, the component selects the next process control information for the process that made the system call. In decision block 506, if all process control information is already selected, the component follows block 508, otherwise the component follows block 507. At block 507, the component stores the selected process control (also called pc) information in the process control data structure and then loops into block 505 to select the next process control information. Alternatively, instead of using and initializing internal process control data structures, the security system can retrieve process control information directly from the process control block when needed. Those skilled in the art will appreciate that process control blocks are created by the operating system when a process is created. At block 508, the component launches a security virtual machine (also known as vm) to enforce the security policy on the intercepted system call. After the security program has been executed by the security virtual machine, the component executes the actions in the action output set. Store and then loop into block 505 to select the next process control information. Alternatively, instead of using and initializing internal process control data structures, the security system can retrieve process control information directly from the process control block when needed. Those skilled in the art will appreciate that process control blocks are created by the operating system when a process is created. At block 508, the component launches a security virtual machine (also known as vm) to enforce the security policy on the intercepted system call. After the security program has been executed by the security virtual machine, the component executes the actions in the action output set. Store and then loop into block 505 to select the next process control information. Alternatively, instead of using and initializing internal process control data structures, the security system can retrieve process control information directly from the process control block when needed. Those skilled in the art will appreciate that process control blocks are created by the operating system when a process is created. At block 508, the component launches a security virtual machine (also known as vm) to enforce the security policy on the intercepted system call. After the security program has been executed by the security virtual machine, the component executes the actions in the action output set.

FIG. 6 is a flow chart illustrating the processing of the processor engine of the security virtual machine of one embodiment. The processor engine initializes the system call parameter data structure based on the intercepted system call, launches the security virtual machine, and performs the actions of the output action set. In blocks 601 to 607, the processor engine loops while loading and executing the instructions of the security program stored in the instruction store until the stop instruction is executed. In one embodiment, a security policy can define that no other rule is tested whenever the conditions of the rule are met. At block 601, the processor engine fetches the instruction pointed to by the current instruction pointer. At block 602, the processor engine calls the component to dereference parameter 1 and parameter 2. In decision block 603, if the instruction operation code is a test operation, the processor engine follows block 604. If the instruction's operation code is a non-test operation (except for a stop operation), the processor engine follows block 608. If the operation code of the instruction is a stop operation, the processor engine continues to block 609. At block 604, the processor engine calls the evaluation test component to determine whether the test operation evaluates to true or false. The called component sets the condition code flag to true or false. In decision block 605, if the condition code is true, the processor engine follows block 607, otherwise the processor engine follows block 606. At block 606, the processor engine adds a fake branch field offset to the current instruction pointer, then loops into block 601 to fetch the next instruction. Is the processor engine a true branch in block 607? Adds the offset of this field to the current instruction pointer, then loops into block 601 and fetches the next instruction. At block 608, the processor engine calls the non-test execution component, then loops into block 601 to fetch the next instruction. The non-test execution component performs an instruction operation and sets the current instruction pointer to point to the next instruction to be executed. The non-test execution component adds a true branching offset of the instruction (or parameter 2 for jump instructions) to the instruction pointer. At block 609, the component executes a stop instruction and then completes.

FIG. 7 is a flow chart illustrating the processing of the indirect reference component of the security virtual machine of one embodiment. This component dereferences parameter 1 and parameter 2 of the fetched instruction. In blocks 701-708, the component dereferences parameter 1. In decision block 701, if parameter 1 is the system call parameter indicated by the parameter 1 descriptor, the component follows block 703, otherwise the component follows block 702. At block 702, the component sets the dereferenced parameter 1 to the process control information specified by the index of parameter 1 and then follows block 709. In decision block 703, if the index of parameter 1 indicates that the system call parameter is an integer, the component follows block 704, otherwise the component follows block 705. At block 704, the component sets the dereferenced parameter 1 to the integer specified by the index, following block 709. In decision block 705, if the index of parameter 1 indicates that the system call parameter is a string, the component follows block 706, otherwise the component follows block 707. At block 706, the component sets the dereferenced parameter 1 to the string specified by the index, following block 709. In decision block 707, if the index of parameter 1 indicates that the system call parameter is low binary, the component follows block 708, otherwise an error has occurred. At block 708, the component sets the dereferenced parameter 1 to the raw binary specified by the index, following block 709. In blocks 709-714, the component dereferences parameter 2. Judgment block 709, para If the meter 2 descriptor points to immediate data, the component follows block 710, otherwise the component follows block 711. At block 710, the component sets the dereferenced parameter 2 to the value of parameter 2 in the fetched instruction and then returns. In decision block 711, if the parameter 2 descriptor points to literal data, the component follows block 712, otherwise the component follows block 713. At block 712, the component sets the dereferenced parameter 2 to the literal data specified by parameter 2 in the fetched instruction and then returns. In decision block 713, if the parameter 2 descriptor points to dynamic data, the component follows block 714, otherwise an error has occurred. At block 714, the component sets the dereferenced parameter 2 to the dynamic data specified by parameter 2 in the fetched instruction and then returns.

FIG. 8 is a flow chart illustrating the processing of the evaluation test component of the security virtual machine of one embodiment. The component decodes the operation code of the loaded instruction and calls the component to test the operation code. The called component sets the condition code to true or false. In decision blocks 801 to 803, the component decodes the operation code of the loaded instruction. In blocks 804-806, the component calls the component to implement the decoded operation code and then returns.

FIG. 9 is a flow chart illustrating the processing of the test component of the security virtual machine of one embodiment. The test component implements the test operation code. At block 901, the component sets the condition code to the value of the dereferenced parameter 2 and then returns.

FIG. 10 is a flow chart illustrating the processing of the character string matching component of the security virtual machine of one embodiment. The component determines if the dereferenced parameter 1 matches the dereferenced parameter 2. In one embodiment, the security virtual machine can use pattern matching. For example, the parameter can contain a "wildcard" designation or, more generally, a regular expression. In decision block 1001, if the lengths of parameter 1 and parameter 2 match, the component follows block 1003, otherwise these dedirected parameters cannot match and the component follows block 1002. .. At block 1002, the component sets the condition code to false and then returns. In blocks 1003-1006, the component loops while checking each character in the string of these dereferenced parameters. At block 1003, the component selects the next character in each string. In decision block 1004, if all characters in the string are already selected, the component follows block 1008, otherwise the component follows block 1005. At block 1005, the component normalizes the selected character. For example, a component can perform a case insensitive comparison with each letter in lowercase. In decision block 1006, if the selected characters match, the component loops into block 1003 and selects the next character in the string, otherwise the component continues to block 1007. At block 1007, the component sets the condition code (also called cc) to false and then returns. At block 1008, all characters in the string match, the component truly sets the condition code, and then returns.

FIG. 11 is a flow chart illustrating the processing of the non-test component of the security virtual machine of one embodiment. The component performs an operation on the untested operation code of the fetched instruction, including setting the current instruction pointer to point to the next instruction to execute. In decision block 1101, if the operation code is an action operation, the component follows block 1102, otherwise the component follows block 1103. At block 1102, the component adds parameter 2 of the fetched instruction to the action output set, and then follows block 1107. In decision block 1103, if the operation code is a reset operation, the component follows block 1104, otherwise the component follows block 1105. At block 1104, the component clears the action output set and then continues to block 1107. In decision block 1105, if the operation code is a jump operation, the component follows 1106, otherwise the component continues to decode further operation code. At block 1106, the component adds parameter 2 to the current instruction pointer and then returns. At block 1107, the component adds the value of the true branch field (also called tb) to the current instruction pointer and then returns.

Although specific embodiments of the security system have been described herein for illustration purposes, it will be appreciated by those skilled in the art that various modifications can be made without departing from the spirit and scope of the invention. It will be appreciated by those skilled in the art that security systems that use security virtual machines can be used to enforce a wide variety of security policies. For example, a security system can be used to secure messages received over a network, transactions received by a transaction processor, and more generally any application that provides an application programming interface. Therefore, the present invention is not limited to the scope of the appended claims.

<figref num="1">It is a block diagram which illustrates the component of the security system of one Embodiment.</figref><figref num="2">It is a block diagram which illustrates the data structure of the data store of one Embodiment.</figref><figref num="3">It is a figure which illustrates the layout of the instruction of the security virtual machine of one embodiment.</figref><figref num="4">It is a flow diagram which illustrates the processing of the program load component of one Embodiment.</figref><figref num="5">It is a flow diagram which illustrates the processing of the system call intercept component of one Embodiment.</figref><figref num="6">It is a flow diagram which illustrates the processing of the processor engine of the security virtual machine of one Embodiment.</figref><figref num="7">It is a flow diagram which illustrates the processing of the indirect reference component of the security virtual machine of one Embodiment.</figref><figref num="8">It is a flow chart which illustrates the process of the evaluation test component of the security virtual machine of one Embodiment.</figref><figref num="9">It is a flow diagram which illustrates the processing of the test component of the security virtual machine of one Embodiment.</figref><figref num="10">It is a flow diagram which illustrates the processing of the character string match component of the security virtual machine of one Embodiment.</figref><figref num="11">It is a flow diagram which illustrates the processing of the non-test component of the security virtual machine of one Embodiment.</figref>

Code description

100 user mode 101 application 102 Policy compiler 103 Policy Store 110 security system 120 kernel mode 121 System call intercept 122 Program load 127 processor engine 128 instruction store 129 data store 202 Literal data table 203 Literal data store 211 Dynamic data table 221 system call parameters 222 string table 223 Integer table 224 low binary table 231 Process control block table 241 Action output set