Method of surveiling internet communication

BACKGROUND OF THE INVENTION

0001The present invention is directed to a method and system for enabling surveillance and monitoring of networks communications by analysis of data traversing therethrough.

0002A huge amount of traffic is flowing through today's computer networks, not all of which is benign. Thus, an owner or supervisor of a given network may be most interested to be able to track or "listen in" in real time in order to effectively monitor and/or secure the network. Such monitoring or surveillance can be achieved by connecting a probe to the network in order to monitor data traveling between two of more nodes (e.g., user workstations) on the network.

0003In a system where communication between two nodes is in a form of discrete packets, the network probe can "read" a packet of data in order to gather, information, such as regarding the sources and the destination addresses of the packet, or the protocol of the packet. In addition, statistical and related information can be computed such as the average or total amount of traffic of a certain protocol type during a given period of time, or the total number of packets being sent to or from a node. This information may be reported to a system administrator in real-time, or may be stored for later analysis.

0004Various attempts have already been made in this direction. For example, Clear View Network Window, a software program available from Clear Communications Corporation, of Lincolnshire, III., U.S.A, allegedly offers predictive/proactive maintenance, intelligent root-cause analysis, and proof-of-quality reports. However, the output is designed for network fault management, which is not the same as "tapping" into a communication between nodes in the network. Thus, the Clearview system does not allow monitoring of data transferred between two nodes in the network with regard to contents or characteristics.

0005Livermore National Laboratory, Livermore, Cal., U.S.A, developed a group of computer programs to protect the computers of the U.S. Department of Energy by "sniffing" data packets that travel across a local area network. The United States Navy used one of these programs, known as the "iWatch" program, in order to wiretap on communications of a suspected computer hacker who had been breaking into computer systems at the U.S. Department of Defense and NASA. The iWatch program uses a network probe to read all packets that travel over a network and then "stores" this information in a common database. A simple computer program can then be written to read through the stored data, and to display only predefined "interesting" pieces of information.

0006Whenever an interesting piece of information is found, the stored data is rescanned and a specific number of characters located at both sides of the "interesting" piece are reported. These interesting characters are then reviewed in order to determine the content of the message and used as a guide to future monitoring activity.

0007This system is restricted to history analyze of user activities and does not enable complete "tapping" of all user activities and full simulation of the users surfing activity.

0008Three major problems are encountered in the way of achieving continuous and reliable tracking: <ol id="ol0001"><li>(a) Individual browsers do not report all the activities performed to a web server. For example, when a browser loads web pages from its browser cache space or from a proxy server, it does not send requests to any "remote" web server through the cyberspace autostrade;</li><li>(b) Application programs designed to perform certain features by web browser of one manufacturer are usually not compatible with those manufactured by another vendor because browser interface mechanisms are different and proprietary to each one of them; and</li><li>(c) Individual browsers send their requests to web servers in a non-systematic order In other words, with regard to a given web server, a preceding request has no relation to a subsequent request. In processing of requests, a web site has no control over the sequences of the requests.</li></ol>

0009In an attempt to overcome these problems, <patcit id="pcit0001" dnum="US5951643A"><text>US Patent No. 5,95'1,643</text></patcit> refers to a mechanism for dependably organizing and managing information for web synchronization and tracking among multiple consumer browsers.

0010However, this solution is limited to tracking activities of identified users, who agreed to be "tapped" and willingly cooperated and be connected to the host with designated application.

0011It is thus the prime object of the invention to provide a monitoring and surveillance method and system enabling network communication suppliers to tap any user connected to the network.

0012It is a further object of the invention to provide a tapping methodology enabling network communication suppliers to watch in real time all user activities while communicating a net work.

0013It is a still further object of the invention to enable web-site owner to monitor and tap users contacting their web site

SUMMARY OF THE INVENTION

0014Thus provided according to the present invention is a method of tracking a network communication line by network probe terminal ("terminal agent") simulating a browser ("original browser") activity of a given terminal comprising the steps of accessing the network communication line, tracing TCP/IP data packets routed through the communication line, selecting TCP/IP data packets relating to a given IP address; ("identified data packets"), selecting from the identified data packets current requests for new connections ("original requests"), selecting from the identified data packets current web-page components indicating new addresses ("new navigation components"), dividing the new navigation components into two categories, embedded objects or frames ("false new components"), hyperlinks ("true new components"), dividing the original requests into original requests matching true the new components, or original requests failing to match any new connection components and belonging to HTTP or POST type as "primary requests", original requests matching the false components as "secondary requests", selecting from identified data packets, HTML data files relating to primary requests; ("respective primary responses"), generating "virtual" secondary requests according to the respective secondary responses, selecting from identified data packets responses relating to secondary virtual requests, ("respective secondary responses") and simulating web page presentation on the terminal agent according to the respective secondary responses.

BRIEF DESCRIPTION OF THE DRAWINGS

0015These and further features and advantages of the invention will become more clearly understood in the light of the ensuing description of a few preferred embodiments thereof, given by way of example only, with reference to the accompanying drawings, wherein- <ul id="ul0001" list-style="none"><li><figref idref="f0001">Fig. 1</figref> illustrates a typical network configuration, in which the present invention can be implemented;</li><li><figref idref="f0002">Fig. 2</figref> illustrates the terminal agent scheme of operation;</li><li><figref idref="f0003">Fig. 3</figref> illustrates the process of tracing and identifying TCP/IP data packets;</li><li><figref idref="f0004">Fig. 4</figref> is a flowchart of classifying TCPIIP requests;</li><li><figref idref="f0005">Fig. 5</figref> is a flowchart of simulating the creation of virtual secondary TCP/IP requests; and</li><li><figref idref="f0006">Fig. 6</figref> illustrates the process of simulating original browser activities.</li></ul>

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

0016Referring to <figref idref="f0001">Fig. 1</figref>, let us assume that terminals 01, 02... are connected to the same communication line, where the communication line is used as internal network ("Intranet"), or external network such as the Internet. According to the present invention it is proposed to connect a designated network probe (hereinafter called "the Terminal Agent") to the data communication line. Alternatively, the terminals 01,02 etc., and the terminal agent may be connected to different data communication lines, or located at different local networks.

0017The general scheme of the terminal Agent operation is illustrated in <figref idref="f0002">Fig. 2</figref>.

0018The Terminal Agent is exposed to all data frames passing through the communication line. The data frames may contain information transferred between the terminals or external data transmission to external sources such as Internet servers.

0019Let us further assume that the "Owner" of the data communication line, such as ISP or network of a private organization, is interested in monitoring in real time, the actual communication activities of a given terminal when surfing the internet.

0020The operation of the Terminal Agent is to first analyze the data frames for tracing TCP/IP data packets. As illustrated in <figref idref="f0003">Fig 3</figref>, the data analysis is processed according to the different protocol hierarchy (see RFC 0793 of the internet protocol), namely, first to analyze the local network protocol, filtering external data transmission ("gateway level "), then identifying internet protocol (IP) data frames, and finally detecting TCP("Transition Control Protocol") data packets of the "host level".

0021Upon analyzing the IP HEADER of the data packets, the IP addresses of the requesting terminal and of the message destination are identified. The owner of the communication line can easily relate the IP address to the users terminals. Therefore it is possible to filter out all other irrelevant data packets and restrict further processing to data transmission of one selected terminal (hereinafter called "the identified data packets").

0022The identified data packets are further analyzed according to the RFC 079 specification enabling full management and control of data communication ports.

0023According to known routines of managing TCP data communication ports, as processed by conventional browsers, e.g. the Internet Explorer, the terminal which operates the browser is the original source of all data transmission. For example let us assume that the terminal placed a request for YAHOO! home page, which request is delivered through the network to YAHOO! server. In response, the server sends an HTML data file containing all information of yahoo home web page components. Accordingly the browser sends new requests for receiving all components of the web-page by opening new communication "virtual" ports, where each port is used for transmitting different components of the same web-page. An "outsider" terminal, exposed to all data requests and respective responses is unable to differentiate between initial "primary" requests, e.g. requesting the complete YAHOO! home page and "secondary" requests for receiving the components thereof. For simulating the activity of the original browser by an "outsider" probe terminal it is essential to identify the primary requests as such.

0024<figref idref="f0004">Fig. 4</figref> illustrates the process for differentiating the primary requests from the secondary requests. Primary requests are originated from different operations such as entering a new URL by the user, choosing a hyperlink, etc. Therefore, in order to detect same one must analyze the previous information transmitted to the same IP address. All new navigation components (addressing the browser to new location) of the web page received by the terminal are sorted according to their type, all embedded objects, frames, etc., are marked as "false" components, while hyperlinks are marked as "true" components. All data is stored in the incoming buffer responses database for later use.

0025When identifying a request for a new connection according to TCP analysis, the request is examined according to the respective navigation components (RNC) in the incoming respond buffer. If the RNC is marked as "false" the request is ignored; if the RNC is marked as "true" the request is classified as primary; otherwise, if there is no RNC relating the said request, the connection type should be identified. If the connection is of an HTML type, or "post" type, it is classified as a primary request.

0026In order to view and monitor the activities of a terminal, all "original" browser activities must be reconstructed. For that purpose it is suggested to use a "virtual" browser. This virtual browser possesses all the capabilities of a "real" browser to download in real time web pages from the Internet. However its connection with the Internet is virtual in the sense that no actual data exchange with the Internet servers is preformed, but only simulating the activities of the original "real" browser.

0027The first function of the virtual browser is illustrated in <figref idref="f0005">Fig. 5</figref>. The browser is receiving all primary requests of the "real" browser. These primary requests and the respective primary responses from the Internet are analyzed and processed according to the conventional browser operation. However the outcome of secondary virtual requests (in conventional browser used to complete the process of downloading web page components) are not transferred directly as usual through the Internet to the appropriate server but stored in a the virtual "secondary" requests buffer database

0028Although the virtual browser connection is not "real", all TCP protocol management of opening and controlling ports connection is processed by the terminal agent as if the connections are "real" ones.

0029The final process of simulating and presenting the web pages in the virtual browser is further illustrated in <figref idref="f0006">Fig. 6</figref>. All original secondary responses coming through the communication line are analyzed and recorded in the incoming responses buffer database. The virtual requests are compared to the respective secondary responses stored in the incoming responses buffer database, by the order of their arrival. If the respective secondary responses already exists in the buffer, these responses are transferred to the virtual browser, and processed (according to conventional browser operation) to present the visual picture of the respective web page components. As a result, the terminal agent is simulating in real time the exact process of downloading Internet web pages as it has been performed by the original terminal.

0030In case the respective responses do not appear in the incoming responses buffer database, activity of an original local cache is deduced. If the original local cache was not used with respect to said virtual request, it is suspend in the buffer database until the original secondary respective responses arrive. Otherwise, if the real local cache was used relating to this respond, the local cache of the virtual browser is examined, and if respective secondary responses exist in the local cache, then the respective respond is transferred to the virtual browser and processed as described above. In case the respective responses do not exist in the virtual cache, either of the following alternatives may be applied. According to one, "passive" version of the terminal agents, no further action is taken to find the "missing" respond, and an "error" message will appear at the agent terminal instead of the web page component which appeared in the real terminal. According to this version, the simulation of the real terminal is not complete but the tapping activity is undetectable. According to another, "active" version, the terminal agent addresses the web page server to request the "missing" respond. Although this version enables the terminal agent to present more exact picture of the real terminal activities, it is traceable for more experienced terminal users, who are able to detect the tapping activity.

0031According to a further mode of implementation of the of the present invention, it is proposed to tap not only to related web page data packets, but to trace also related messages data packets e.g. e-mail or chats. To enables such tapping, the same method and principals as described above are applied at request for receiving and sending messages through the network other than requests for web pages. The process of analyzing such requests and the respective responses is more streamlined since there is no need to check the cache memory activity, as by definition such information is always new.

0032Finally, it should be appreciated that the above-described embodiments are directed to Internet communication environment. However, the invention in its broad aspect is equally applicable to computerized network communication in general, such as satellite, cellular and others.

0033While the above description contains many specificities, these should not be construed as limitations on the scope of the invention, but rather as exemplification of the preferred embodiments. Those skilled in the art will envision other possible variations that are within its scope. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their legal equivalents.

0034Other embodiments of the present invention are: <ol id="ol0002"><li>1. A method of tracking a network communication line by network probe terminal ("terminal agent") simulating a browser ("original browser") activity of a given terminal comprising the steps of: <ul id="ul0002" list-style="none"><li>I. Accessing the network communication line;</li><li>II. Tracing TCP/IP data packets routed through the communication line;</li><li>III. Selecting TCP/IP data packets relating to a given IP address ("identified data packets");</li><li>IV.Selecting from the identified data packets current requests for new connections ("original requests");</li><li>V. Selecting from the identified data packets current web page components indicating new addresses ("new navigation components");</li><li>VI. Dividing the new navigation components into two categories; <ul id="ul0003" list-style="none" compact="compact"><li>(f1) embedded objects or frames ("false new components");</li><li>(f2) hyperlinks ("true new components");</li></ul></li><li>VII. Dividing the original requests into "primary" or "secondary" requests according to the following criteria, respectively: <ul id="ul0004" list-style="none" compact="compact"><li>(g1) original requests matching true the new components, or original requests failing to match any new connection components and belonging to HTTP or POST type;</li><li>(g2) original requests matching the false components or original requests failing to match any new connection components and not belonging to HTTP or post type;</li></ul></li><li>VIII. Selecting from the identified data packets, HTML data files relating to primary requests ("respective primary responses");</li><li>IX. Generating "virtual" secondary requests according to the respective secondary responses;</li><li>X. Selecting from the identified data packets responses relating to secondary virtual requests; ("respective secondary responses"); and</li><li>XI. Simulating web page presentation on the terminal agent according to the respective secondary responses.</li></ul></li><li>2. The method of point 1 further comprising the steps of: <ul id="ul0005" list-style="dash"><li>Selecting virtual secondary requests currently not matching any respective original responses ("unanswered secondary requests");</li><li>Retrieving content form local cache of terminal agent ("Virtual cache") if original browser local cache ("original cache") was used relating to unanswered secondary requests;</li><li>Simulating web page presentation on agent terminal according to data relating to the unanswered secondary requests if the virtual cache contains such data; and</li><li>Displaying error massages at the respective places on the agent terminal if the virtual cache does not contain data relating to the unanswered secondary requests;</li></ul></li><li>3. The method of point 2 further comprising the step of: <ul id="ul0006" list-style="dash" compact="compact"><li>Addressing via the communication line to appropriate Internet server for receiving respective simulated responses relating to unanswered secondary requests if the virtual cache does not contain data, and displaying thereof.</li></ul></li><li>4. The method of point 1 further comprising the steps of: <ul id="ul0007" list-style="dash"><li>Selecting from identified data packets data relating network messages e.g. e-mail ("messages data");</li><li>Transforming the message data into text data file; and</li><li>Displaying text data file on terminal agent.</li></ul></li><li>5. The method of point 1 where in the net work is local-area network (LAN) and the terminal is connected to the extended communication line.</li><li>6.The method of point 1 wherein the communication line an is an external communication line e.g. telephone line, IZDN line, optical lines, etc.</li><li>7. The method of point 1 wherein the net work is a local-area network (LAN), and the terminal agent is situated in a location different from that of the given terminal.</li><li>8. The method of point 1 wherein the given IP address is identified by the communication line provider (ISP).</li><li>9. The method of point 1 wherein the given IP address is identified by the communication line owner.</li><li>10. The method of point 1 wherein the given IP address are addresses of web site visitors and identified by a web site owner.</li><li>11. A network probe terminal for tracking a network communication line and simulating a browser ("original browser") activity of a given terminal comprising of: <ul id="ul0008" list-style="none"><li>I. Connection means for accessing the network communication line;</li><li>II. Monitoring means for tracing TCP/IP data packets routed through the communication line;</li><li>III. First filtering module for selecting new connection requests ("original requests") and web page components indicating new addresses ("new navigation components") out of TCP/IP data packets relating to a given IP address ("identified data packets");</li><li>IV. First Sorting means for dividing the new navigation components into two categories; <ul id="ul0009" list-style="none" compact="compact"><li>(f1) embedded objects or frames ("false new components");</li><li>(f2) hyperlinks ("true new components");</li></ul></li><li>V. Second Sorting means for dividing the original requests into "primary" or "secondary" requests according to the following criteria, respectively: <ul id="ul0010" list-style="none" compact="compact"><li>(g1) original requests matching the true new components, or original requests failing to match any new connection components and belonging to HTTP or POST type;</li><li>(g2) original requests matching the false components or original requests failing to match any new connection components and not belonging to HTTP or post type;</li></ul></li><li>VI. Classifying module for selecting HTML data files relating to primary requests ("primary responses") from the identified data packets;</li><li>VII. Request generating module for creating "virtual" secondary requests according to the respective secondary responses;</li><li>VIII. Second filtering module for selecting responses relating to secondary virtual requests; ("secondary responses") from identified data packets; and</li><li>IX. Displaying means for simulating web page presentation on the terminal agent according to the secondary responses.</li></ul></li><li>12. The network probe terminal of point 11 wherein the filtering module comprises means for selecting virtual secondary requests currently not matching any respective original responses ("unanswered secondary requests");</li><li>13. The network probe terminal of point 11 further comprising cache module for activating terminal agent local cache in case the original browser local cache ("original cache") has been used in response to unanswered secondary requests;</li><li>14. The network probe terminal of point 13 further comprising a retrieval module for addressing via the communication line to appropriate Internet server and receiving respective simulated responses relating to unanswered secondary requests if the virtual local cache does not contain data.</li><li>15. The network probe terminal of point 11 further comprising an electronic message module for selecting identified data packets data relating to network messages e.g. e-mail ("messages data"), transforming the message data into text data file, and displaying text data file on terminal.</li><li>16. The network probe terminal of point 11 wherein the given IP address is identified by the communication line provider (ISP).</li><li>17. The network probe terminal of point 11 wherein the given IP address is identified by the communication line owner.</li><li>18. The network probe terminal of point 11 wherein the given IP addresses are addresses of web site visitors and identified by the web site owner.</li></ol>