CN102316074A - HTTP (hyper text transfer protocol) multithreading restoration method based on libnids - Google Patents

Abstract

The invention designs a libnids-based http protocol multi-thread restoration method. The system needs to run in the Linux operating system, and the computer running the system is required to function normally and run stably. In addition, it is required that the computer installed with the software of the present invention be in a local area network. In the present invention, firstly, the network card of the host computer is set to a promiscuous mode, and then all data packets passing through this network segment are monitored. Determine whether it is a data packet of the http protocol. If so, name the data packet through the URL, and record the source IP, source port, and destination IP, and store them in a data structure as a quadruple. Continue to judge whether the data packet is a multi-threaded downloaded data packet, and if so, record the two variables content_range and content_length. Calculate the position of the packet in the original file through content_range and content_length. Write the content in the data packet to the corresponding location. The present invention brings the following beneficial effects: (1) completeness, can capture most of the data packets in the local area network, and can restore files including multi-thread download. (2) Continuity, the system can operate stably in a stable environment. (3) High efficiency, using multi-thread buffer technology to process different application layer protocols separately, improving processing efficiency, and ensuring that most of the data can still be captured when the network traffic is heavy.

Description

Multi-thread restoration method of http protocol based on libnids

technical field

The present invention relates to malicious code.

Background technique

Now network-based attacks are becoming more and more common and diverse, and these network-based intrusion behaviors are difficult to be discovered by the attacking host. Therefore, in recent years, the network-based intrusion monitoring system is different from the host-based intrusion detection system. A network intrusion detection system can directly obtain various information of the host, such as system operation logs, etc., while a network intrusion detection system can only obtain data packets of network communication, and cannot directly obtain the specific content of communication. It turns out that the specific content of network communication is obtained, and most network intrusions can only be detected based on these restored content. Due to the large-scale use of TCP/IP networks, most current intrusion detection systems are aimed at TCP/IP networks. For TCP/IP network, the restoration of TCP/IP protocol becomes one of the key technologies of network intrusion detection system.

At present, there are also some related patents, which will be introduced below.

Patent 200610125451.3 is used for HTTP data restoration method. A method for restoring HTTP data, characterized in that: the request data sent by the client to the server and the response data requested by the server to the client are intercepted, and then filtered, parsed and cached to form the overall data in html format, if When the data is transmitted through the transfer encoding, the formed overall data is decompressed; if the data is compressed before transmission, the formed overall data is decompressed, so that the data that can be directly displayed by the browser is formed, and the HTTP protocol data The restore is complete. By intercepting and processing the HTTP packet data of these requests and responses, the data of the HTTP packet can be restored, and the user's online behavior can also be reproduced. However, with the development of the Internet, multi-threaded downloading tools represented by Thunder, Flashget, Download Tornado, etc. are increasingly rampant. In HTTP data restoration, there are more and more situations where multiple IP addresses and multiple ports download the same file. However, the patent 200610125451.3 only implements protocol restoration for one-to-one situations, that is, one client downloads a file at one server, without considering one-to-many (one client downloads the same file at multiple servers), And many-to-many (multiple clients download the same file on multiple servers), no further research has been done. In contrast, this patent can not only completely capture most of the data packets in the local area network, but also completely restore files including multi-threaded downloads, and also use multi-threaded buffer technology to process different application layer protocols separately to improve processing efficiency , to ensure that most of the data can still be captured when the network traffic is heavy.

Contents of the invention

The invention provides a libnids-based http protocol multi-thread restoration method, which has integrity, continuity and high efficiency.

First, set the host network card to promiscuous mode, and then monitor all data packets passing through this network segment to judge each data packet. First determine whether the packet is from the client or the server. If it is client data, check if there is an HTTP characteristic value in the get field, and if it is the server side, judge whether there is an HTTP characteristic value in the data packet header. If it is determined to be an HTTP data packet, the URL in the data packet is extracted and the data packet is named by analyzing the URL, and the source IP, source port, and destination IP are recorded at the same time, and stored in a data structure as a five-tuple. All five-tuple data structures are concatenated with a linked list, and the data package name is used as a key to uniquely determine all data packages of a file. Continue to judge whether the data packet is a multi-threaded downloaded data packet. If so, record the two variables content_range and content_length plus the data name and entity content to form a second linked list. Its function is to write for the entity when writing the file File content for positioning. When writing data to the same file, a third data structure needs to be established, including filename, content_range, and long file_position to locate the specific writing position. In the process of multi-threaded download, the way the data packets arrive at the customer service end is to divide the file into multiple segments by content_range and transmit them at the same time. Therefore, the order in which the data packets in each segment arrive at the customer service end is out of order. The segments, and the orderly combination of the positions in the segments need to be co-located through three data structures. Write the content in the data packet to the corresponding location. Data structures and linked lists are dynamically growing, and if they are not deleted in time, the memory will be exhausted. This makes the program unable to run stably for a long time, but deleting its data structure in the linked list before all the data of a file reaches the customer service end will cause the file to fail to be successfully restored. Based on the above considerations, we use Libnids to provide nids_state during the deletion process of the data structure in the linked list. If the state of nids_state is NIDS_CLOSE and NIDS_RESET, we consider that all the data packets of a file have arrived, and use the filename keyword in the linked list to find the data structure with the same filename, and delete all the data structures with the same filename. The program is divided into two threads. The main thread is responsible for capturing packets, analyzing protocols, and assembling data. Secondary threads are responsible for creating files, writing data, content control, etc.

This patent system includes the following five functional modules:

libnids: The main functions include capturing network data packets, IP fragment reassembly, TCP data stream reassembly, port scanning attack testing and abnormal data packet testing, etc.

Filter data packet module: analyze the data packets captured by libnids, and filter out data packets other than the http protocol.

Protocol analysis module: extract the information to be used in the http protocol, mainly including: file name, destination ip, port, etc.

Data writing module: write data packets belonging to the same quadruple to the same file.

File deletion module: determine whether the data packets of the same file have been transmitted, and if the transmission is completed, delete the quadruple in the memory.

When this patent was implemented, the Libnids module was mainly responsible for packet capture, the filter data packet module was responsible for packet filtering, the protocol analysis module was responsible for extracting key data from the packet, the data writing module was responsible for writing data packets into files, and the file deletion module was responsible for judging whether the file After the transfer is complete, the expired data structure has been deleted.

The system runs on the Linux operating system, and the computer running the system is required to function normally and run stably. In addition, it is required that the computer with this patented software installed is in a local area network. And set the network card of this computer to promiscuous mode.

Description of drawings

Figure 1 is a flow chart of protocol analysis;

Fig. 2 is a data stream filtering module;

Fig. 3 is a data writing module;

Figure 4 is the file deletion module

Fig. 5 is the concrete flowchart of this patent;

Detailed ways

The technical scheme of the present invention will be described in detail below in conjunction with the accompanying drawings.

Figure 1 shows the process of protocol analysis. The main functions of protocol analysis include writing the http protocol and extracting key data. The steps to refine Figure 1 are as follows:

Step S101: Determine whether the data packet belongs to the customer service end or the server end by analyzing the first line of the data packet header.

Step S102: If it is the server data extraction status code, it is used to determine what status the connection belongs to.

Step S103: If it is the URL provided in the server-side data extraction packet.

Step S104: Analyzing the URL and extracting the file name of the restored file through regular expression matching.

Step S105: Extract the destination IP, destination port, source IP, and source port for link list positioning.

Step S106: Extracting the content_length and content_range fields for locating the offset when writing the file.

Step S107: Extracting entity content for file restoration.

Figure 2 shows filtering data packets, analyzing data packet message headers, and judging whether there are http characteristic values. If not filtered, otherwise handed over to the protocol analysis module. It contains the following steps:

Step S201: Determine whether the data packet belongs to the customer service end or the server end by analyzing the first line of the data packet header.

Step S202: The customer service end requests the header, judges the get field, and discards the packet if there is no

Step S203: the customer service terminal judges whether there is a characteristic value HTTP. Drop the packet if not

Step S204: If it is server data, extract the data packet status line, and judge whether the beginning part is HTTP. If there is an HTTP characteristic value, it is regarded as an HTTP packet.

Figure 3 shows the data writing module, which writes the extracted entity content into the file according to its position in the file. The specific process is as follows:

Step S301: Since the file name is unique through the analysis of the protocol analysis module, firstly judge whether the file has been created or not by the file name.

Step S302: Determine whether the file belongs to a multi-threaded download file through the status code extracted by the protocol analysis module

Step S303: If it is a multi-thread download file, extract content_range and content_length attributes. Used for file positioning.

Step S304: Write the entity content and file into the corresponding location of the file.

Figure 4 shows the data deletion module. Determine the data connection status and delete the corresponding data structure in the linked list.

Step S401: Monitor the state of the nids_state variable.

Step S402: Determine whether the nids_state is NIDS_CLOSE or NIDS_RESET. These two states represent the completion of file transfer.

Step S403: Find the data structure corresponding to the linked list through the destination ip, destination port, source ip, and source port. This four-tuple can identify all packets of the same file in the linked list

Step S404: extract the file name in the data structure. The file name can not only determine all packages of the same file in a linked list, but also uniquely determine all data packages of the same file in three linked lists at the same time.

Step S405: Find all the data structures in the three linked lists according to the file names.

Step S406: Delete the data structure. Free up memory space.

Although this instruction manual has only described the detail of described method, and does not talk about the application of the present invention more, but because of the important value of http protocol multi-thread restoration method based on libnids in malicious code research, its application area is very extensive Therefore, the spirit and scope of the present invention should not be limited to the embodiments described herein.

Claims (10)

1. the http agreement multithreading method of reducing based on libnids is characterized in that and will host network card be made as promiscuous mode, monitors all packets through this network segment then.Judge whether it is the packet of http agreement.If then be this packet name, and note source IP through URL, source port, purpose IP deposits a data structure in as a four-tuple.Continue whether the judgment data bag is a multithreading data downloaded bag, if then note content_range and these two variablees of content_length.Calculate the position of packet in original document through content_range and content_length.Content in the packet is write correspondence position.

2. like right 1 described http agreement multithreading method of reducing, it is characterized in that native system moves in (SuSE) Linux OS, require the computer function of operation native system normal, stable based on libnids.In addition, the computer that requires this patent software is installed is in the local area network (LAN).And the network interface card of this computer is set to promiscuous mode.

3. like right 1 described http agreement multithreading method of reducing based on libnids; It is characterized in that, adopt the multithreading buffer technology, improve treatment effeciency to different application layer protocol individual processing; Can guarantee under the very big situation of network traffics, still can capture most data.

4. like right 1 described http agreement multithreading method of reducing, it is characterized in that libnids will be responsible for catching network packet, ip fragmentation reorganization, the reorganization of tcp data stream and Port Scan Attacks test and the test of abnormal data bag etc. based on libnids.

5. like right 1 described http agreement multithreading method of reducing, it is characterized in that the packet that filtering data bag module will capture libnids is analyzed, and filters the packet that drops out beyond the http agreement based on libnids.

6. like right 1 described http agreement multithreading method of reducing, it is characterized in that based on libnids, protocol-analysis model must extract in the http agreement the information that will use, mainly comprise: filename, purpose ip, port etc.

7. like right 1 described http agreement multithreading method of reducing, it is characterized in that the data writing module is responsible for the packet that belongs to same four-tuple is write identical file based on libnids.

8. like right 1 described http agreement multithreading method of reducing, it is characterized in that the packet that the file delete module will be judged identical file is end of transmission whether, if end of transmission its four-tuple of deletion in internal memory just based on libnids.

9. like right 2 described http agreement multithreading method of reducing based on libnids, it is characterized in that having continuity, system can stable operation in a stable environment.

10. like right 3 described http agreement multithreading method of reducing, it is characterized in that based on libnids, can be complete catch most packet in the local area network (LAN), can complete reduction comprise the multithreading downloaded files.

Images