CN102137022B - Method for identifying information of data packet, crawler engine and network system - Google Patents

Abstract

The embodiment of the present invention discloses a method for providing information for identifying data packets, a crawler engine, and a network system, wherein the method for providing information for identifying data packets includes: using a crawler program of an application protocol to establish the application The corresponding relationship between the protocol and the network entity information using the application protocol; sending the identification information including the corresponding relationship to the deep packet inspection DPI device, so that the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs; wherein The information about the network entity using the application protocol includes: the address of the network entity using the application protocol and the identifier of the transport layer protocol used by the network entity using the application protocol. Using the technical solution provided by the embodiment of the present invention can reduce the identification time and performance overhead of the DPI device in the process of identifying the code stream.

Description

Method for providing information for identifying data packets, crawler engine and network system

technical field

The invention relates to the field of communication technology, in particular to a method for providing information for identifying data packets, a crawler engine and a network system.

Background technique

Deep Packet Inspection (DPI) technology is a DPI device that identifies the specific application protocol to which the code stream belongs through the characteristic characters or characteristic behaviors in the code stream. Subsequently, various upper-layer services can be performed according to the identification results, such as billing, flow control operations, etc. The network deployment location of the DPI device is the same as that of the gateway. All code streams of all users under the jurisdiction of the gateway need to pass through the DPI device. The number of users may reach millions or more, but it is necessary to ensure that the code stream transmission cannot be caused by code stream identification. Delay in subsequent processing.

In order to enable the DPI device to identify the code stream, it is necessary to analyze the specific character characteristics of some application protocol code streams offline, generate a knowledge base based on the analysis results, and load the knowledge base to the DPI device. The subsequent DPI device uses the knowledge base to determine the received data. The specific application protocol to which the stream belongs.

The prior art has the following problems:

When an application protocol in the network is updated, it is necessary to analyze the specific character characteristics of the code stream of the updated application protocol offline and generate a new knowledge base. The code stream can only be identified, so that in the process of generating a new knowledge base and loading the new knowledge base into the DPI device, the code stream of the application protocol cannot be recognized, which increases the recognition time; moreover, the DPI device recognizes the code stream During the process, some encrypted code streams need to be decrypted first, and then identified according to the characteristic characters or characteristic behaviors in the decrypted code stream, which requires a lot of performance overhead.

Contents of the invention

Embodiments of the present invention provide a method for providing information for identifying data packets, a crawler engine, and a network system, so as to reduce DPI device identification time and performance overhead in the process of identifying code streams.

In view of this, embodiments of the present invention provide:

A method of providing information for identifying a data packet comprising:

Using the crawler program of the application protocol to establish a corresponding relationship between the application protocol and the network entity information using the application protocol;

Sending identification information including the corresponding relationship to the deep packet inspection DPI device, so that the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs;

Wherein, the information of the network entity using the application protocol includes: the address of the network entity using the application protocol and the identifier of the transport layer protocol used by the network entity using the application protocol.

A web crawler comprising:

An establishing unit, configured to use a crawler program of an application protocol to establish a corresponding relationship between the application protocol and the network entity information using the application protocol;

A sending unit, configured to send identification information including the corresponding relationship to the DPI device, so that the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs;

Wherein, the information of the network entity using the application protocol includes: the address of the network entity using the application protocol and the identifier of the transport layer protocol used by the network entity using the application protocol.

A network system, comprising: the above-mentioned web crawler and a deep packet inspection DPI device, wherein the DPI device is configured to receive the identification information sent by the web crawler, and use the corresponding relationship to identify the application protocol to which the data packet belongs.

In the embodiment of the present invention, the crawler program of the application protocol is used to establish the corresponding relationship between the application protocol and the information of the network entity using the application protocol, and send it to the DPI device, so that the DPI device can use the corresponding relationship to identify the application protocol to which the data packet belongs In this way, the DPI device does not need to decrypt the data packet, which reduces the performance overhead, and the DPI device does not need to analyze the specific character characteristics of the code stream of the application protocol offline, which reduces the time for code stream identification.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort.

FIG. 1 is a flowchart of a method for providing information for identifying a data packet to a DPI device according to an embodiment of the present invention;

2A is a flowchart of a method for providing information for identifying a data packet to a DPI device according to another embodiment of the present invention;

Fig. 2B is a schematic diagram of obtaining Peer information by a crawler engine provided by another embodiment of the present invention;

3 is a flowchart of a method for providing information for identifying a data packet to a DPI device according to another embodiment of the present invention;

FIG. 4 is a flow chart of a method for a DPI device to identify an application protocol to which a data packet belongs according to an embodiment of the present invention;

Fig. 5 is a schematic diagram of an identification application protocol provided by an embodiment of the present invention;

Fig. 6 is the method flowchart that crawler engine obtains Peer information when the application agreement that the embodiment of the present invention provides is Bittorent non-DHT agreement;

Fig. 7 is the method flowchart that reptile engine obtains Peer information when the application agreement that the embodiment of the present invention provides is Bittorent DHT agreement;

Fig. 8 is a schematic diagram of identification information sharing provided by an embodiment of the present invention;

Fig. 9 is a schematic diagram of another identification information sharing provided by an embodiment of the present invention;

FIG. 10 is a structural diagram of a web crawler provided by an embodiment of the present invention;

Fig. 11 is another kind of web crawler structural diagram provided by the embodiment of the present invention;

Fig. 12 is another kind of web crawler structural diagram provided by the embodiment of the present invention;

FIG. 13 is a structural diagram of a network system provided by an embodiment of the present invention;

FIG. 14 is a structural diagram of another network system provided by an embodiment of the present invention.

Detailed ways

Referring to Fig. 1, an embodiment of the present invention provides a method for providing DPI with information for identifying a data packet, which includes:

101. Using a crawler program of an application protocol, establish a correspondence between the application protocol and network entity information using the application protocol.

Wherein, the execution subject of this embodiment may be a crawler engine, and the crawler engine may be located on a DPI device or a device that exists independently from the DPI device, which does not affect the implementation of the present invention.

Wherein, the information of the network entity includes: the address of the network entity and the identifier of the transport layer protocol used by the network entity. The address of the network entity includes: the IP address and port number of the network entity; or, the address of the network entity includes: the domain name of the network entity. Wherein, the corresponding relationship between the application protocol and the network entity information using the application protocol may specifically be: the corresponding relationship between the identifier of the application protocol and the network entity information using the application protocol. Wherein, the identifier of the application protocol may be an ID of the application protocol or a name of the application protocol.

Specifically, each network entity in the network may use different application protocols, or multiple network entities may use the same application protocol, each application protocol corresponds to a crawler program, and different application protocols correspond to different crawler programs. The crawler program of the protocol can include: BT program, eDonkey program, QQ program, etc.

This step can be achieved in the following two ways:

The first method: the crawler engine sequentially invokes the crawler programs of the application protocols in the crawler program set to send detection request messages to the network entities in the network, until the response message returned by the network entity indicating successful detection is received, and a corresponding response is established The corresponding relationship between the application protocol used by the probe request message and the information of the network entity.

Among them, there are crawler programs of various application protocols in the crawler program collection, such as BT program, eDonkey program, QQ program, etc., and the crawler engine sequentially calls the crawler programs of each application protocol to detect the network entity in the network to detect the network entity Whether to use the application protocol. If the network entity uses a certain application protocol, the network entity will feed back a response message indicating successful detection to the crawler engine after receiving the detection request message sent by the crawler program of the crawler engine using the application protocol.

Preferably, in the case of adopting this implementation method, before this step, it may also include: the crawler engine receives the information of the network entity that needs to recognize the application protocol sent by the DPI device; wherein, the network entity that needs to recognize the application protocol is The DPI device failed to identify the network entity for the application protocol used. Then in this step, the crawler engine sequentially invokes the crawler program of the application protocol in the crawler program set to send a detection request message to the network entity that needs to identify the application protocol, until it receives a response indicating that the detection is successful returned by the network entity that needs to identify the application protocol news.

The second method includes the following steps:

A. Determine a network entity serving as a query source; wherein, the network entity serving as a query source uses the application protocol.

B. The crawler program using the application protocol obtains the information of the network entity associated with the query source from the query source; wherein, the network entity associated with the query source is a network that uses the application protocol and has the same resources as the query source entity.

Wherein, the network entity having the same resource as the query source may be a network entity sharing the same file as the query source. For example, a network entity that downloads the same file as the query source, or a network entity that downloads the file uploaded by the query source while the query source is uploading a file.

C. Use a network entity that has not been used as a query source among the network entities associated with the query source as an updated query source. When the query end condition is not met, return to step B. At this time, the query source in step B is all The updated query source; when the query end condition is reached, the corresponding relationship between the application protocol and the information of the network entity using the application protocol is established. Wherein, the network entity using the application protocol includes: the network entity used as the query source in step A, and the network entity obtained in step B.

Wherein, the query end condition may be that the number of acquired network entities reaches a predetermined number, or the timer expires, that is, the time for performing the query operation (that is, the time for performing the above steps A, B, and C) exceeds the predetermined time .

Wherein, the network entity used as the query source in step A may be a predetermined network entity using the application protocol; it may also be a network entity extracted from the seed file of a specific resource and using the application protocol, or from a specific The network entity tracked by the tracker tracker extracted from the seed file of the resource and using the application protocol; it can also be that the crawler engine sequentially calls the crawler program in the crawler program set to try to send a network entity to a certain network entity in the network Send a probe request message until receiving a response message indicating that the probe is successful returned by the network entity. After determining that the application protocol used in the probe request message corresponding to the response message is the application protocol used by the network entity, the network entity As a query source; the DPI device can also notify the crawler engine. Specifically, after receiving a data packet, the DPI device will use deep packet inspection technology to determine the application protocol to which the data packet belongs, and establish the source end and /or the mapping relationship between the destination end and the application protocol identifier and send it to the crawler engine; the crawler engine determines that the network entity corresponding to the application protocol identifier in the mapping relationship is the network entity as the query source according to the mapping relationship sent by the DPI device. Wherein, the DPI device uses the deep packet inspection technology to determine the application protocol to which the data packet belongs specifically may be: the DPI device uses a feature recognition method, a behavior recognition method, a heuristic recognition method or an association recognition method to identify the application protocol to which the data packet belongs. Wherein, the specific implementation manner of identifying the service type of the data packet by means of feature recognition, behavior recognition, heuristic recognition or association recognition can refer to existing solutions, and will not be repeated here.

Among them, the application protocol crawler program applicable to this method is a point-to-point application protocol crawler program, such as: BT program, eDonkey program and so on.

102. Send identification information including the correspondence to the DPI device, so that the DPI device uses the correspondence to identify the application protocol to which the data packet belongs.

Wherein, the identification information further includes: the aging suggestion time of the corresponding relationship, so that the DPI device can control the corresponding relationship to be invalid after the aging suggested time of the corresponding relationship arrives, for example, directly delete the corresponding relationship, etc., so that After the aging suggestion time of the corresponding relationship arrives, the DPI device can no longer determine the application protocol to which the data packet belongs according to the corresponding relationship.

Wherein, the network entity in the embodiment of the present invention may be a peer-to-peer network entity (Peer), or other network entities, which does not affect the implementation of the present invention.

In the embodiment of the present invention, the crawler program of the application protocol is used to establish the corresponding relationship between the identification of the application protocol and the information of the network entity using the application protocol, and send it to the DPI device, so that the DPI device can use the corresponding relationship to identify the information packet to which the data packet belongs. In this way, the DPI device does not need to decrypt the data packet, which reduces the performance overhead, and the DPI device does not need to analyze the specific character characteristics of the code stream of the application protocol offline, which reduces the time for code stream identification.

In order to make the above-mentioned technical solutions of the present invention clearer, the following embodiments take the network entity as a peer (peer-to-peer network entity) as an example, and describe in detail the method for providing DPI with information for identifying data packets provided by the present invention. include:

201A. The crawler engine in the DPI device determines that the peer-to-peer network entity as the query source is Peer1, and uses the crawler program of the application protocol to send a query request message to Peer1.

In this embodiment, the crawler engine saves the crawler program developed for not using the application protocol. When Peer needs to be accessed, it determines the peer-to-peer network entity as the query source according to the predetermined information, calls the crawler program of the application protocol, and sends a query to the query source. ask. Wherein, the reservation information can be predetermined information corresponding to the query source of the specific application protocol. For example, the query source corresponding to the Bittorent_DHT protocol can be predetermined to be Peer1, and then the reservation information can be the address information of the Peer1 and the transport layer applied by the Peer1 protocol.

Optionally, the crawler engine in the DPI device may also determine the peer-to-peer network entity as the query source according to the information reported by the identification engine of the DPI device. Specifically, after receiving a data packet, the recognition engine in the DPI device uses deep packet inspection technology to determine the application protocol to which the data packet belongs, and establishes a mapping between the source end and/or destination end of the data packet and the application protocol identifier The relationship is sent to the crawler engine, and the crawler engine uses the source and/or destination corresponding to the application protocol identifier in the mapping relationship as the query source.

202A. Peer1 sends a query response message to the crawler engine, which includes: information about the Peer associated with Peer1.

Wherein, the information of the Peer associated with Peer1 includes: the address information of the Peer associated with Peer1 and the transport layer protocol used by the Peer associated with Peer1, wherein the address information of the Peer can include an IP address and a port number, or include a domain name wait.

Wherein, the Peer associated with Peer1 is a Peer that uses the application protocol and has the same resource as the Peer1, such as the same media resource, for example, a Peer that is downloading the same movie as the Peer1. In this embodiment, it is assumed that the Peers associated with Peer1 are Peer2 and Peer3.

203A. The crawler engine in the DPI device uses Peer2 as a query source, and uses the crawler program of the application protocol to send a query request message to Peer2.

204A, Peer2 sends a query response message to the crawler engine, which includes: information about the Peer associated with Peer2, wherein the Peer associated with Peer2 is a Peer that uses the application protocol and has the same resources as the Peer2.

205A. The crawler engine uses Peer3 as a query source, and uses the crawler program of the application protocol to send a query request message to Peer3.

206A. Peer3 sends a query response message to the crawler engine, which includes: information about the Peer associated with Peer3, wherein the Peer associated with Peer3 is a Peer that uses the application protocol and has the same resources as the Peer3.

It should be noted that step 203A and step 204A are executed sequentially, and step 205A and step 206A are executed sequentially, but there is no sequence of execution between step 203A-step 204A and step 205A-step 206A, and they can also be executed in parallel , does not affect the realization of the present invention.

207A. The crawler engine in the DPI device judges whether the query end condition is met, if not, execute step 208A, and if yes, execute step 210A.

Wherein, judging whether the query end condition is met may be judging whether the number of peers using the application protocol found in the query meets the predetermined number, or judging whether the timer expires, that is, whether the query operation time reaches the predetermined time.

Before this step, when the crawler engine determines that there are duplicate nodes in the peers using the application protocol fed back by each query source according to the query response messages returned by each query source, remove the nodes in the peers using the application protocol fed back by each query source. Duplicate Peer, then the above-mentioned judged "queried peers using the application protocol" does not include the duplicate Peer.

Wherein, this step is to remove duplicate Peers from each Peer associated with Peer2 and each Peer associated with Peer3. Wherein, this step may also remove the invalid Peer among the Peers using the application protocol fed back by each query source, where the invalid Peer may be a node currently in a fault state.

208A. After the crawler engine in the DPI device removes the repeated Peers among the Peers using the application protocol fed back by each query source, the remaining Peers are used as the updated query source, and the crawler program of the application protocol is used to update the query source. Send query request message.

In this embodiment, it is assumed that the Peers associated with Peer2 are Peer4, Peer5, and Peer6, and the Peers associated with Peer3 are Peer5, Peer6, then the query sources in step 208 are Peer4, Peer5, and Peer6.

209A. The crawler engine in the DPI device will receive the query response message for the query request message, which includes the Peer information associated with the updated query source, and return to step 207A.

Wherein, the Peer information associated with the above-mentioned updated query source is: the address of the Peer that uses the application protocol and has the same resources as the above-mentioned updated query source, and the transport layer protocol used by the Peer.

210A. The crawler engine in the DPI device reports identification information to the identification engine in the DPI device, where the identification information includes: the correspondence between the information of the peer using the application protocol and the application protocol identifier.

Wherein, the information of the Peer using the application protocol includes: the address of the Peer using the application protocol and the transport layer protocol used by the Peer using the application protocol.

Wherein, the crawler engine in the DPI device can also save the aging suggestion time corresponding to each application protocol, so in this step, the identification information reported by the crawler engine to the recognition engine can also include: aging suggestion time, aging suggestion time Indicates when the correspondence is aged out. After the suggested time arrives, the corresponding relationship between the Peer information using the application protocol and the application protocol identifier is no longer valid.

211A. The recognition engine in the DPI device receives and saves the recognition information sent by the crawler engine.

Fig. 2B shows a schematic diagram of the crawler engine corresponding to the embodiment shown in Fig. 2A above obtaining Peer information, wherein, assuming that Peer1 is a predetermined query source, "1" in Fig. 2B represents the first stage query operation, that is, querying Peer1 , the Peers associated with Peer1 obtained by querying Peer1 are Peer2 and Peer3; "2" indicates the second-stage query operation, that is, querying Peer2 and Peer3, where the Peers associated with Peer2 obtained by querying Peer2 are Peer4, Peer5, and Peer6 , the Peers associated with Peer3 obtained by querying Peer3 are Peer4 and Peer5; "3" indicates the third-stage query operation, that is, querying Peer4, Peer5, and Peer6.

Optionally, the crawler engine simulates a real Peer. After obtaining the information of a certain Peer from the query response, the crawler engine can simulate the Peer and actively provide the information of the Peer to other Peers. After receiving related requests from other Peers, provide the information of the Peer to other Peers.

In the embodiment of the present invention, the crawler program using the application protocol obtains the Peer information associated with the query source from the network entity as the query source, and then associates the obtained updated query source with the updated query source In this way, the Peer information using the application protocol in the network is obtained, and the corresponding relationship between the identification of the application protocol and the Peer information using the application protocol is sent to the identification engine in the DPI device, so that the DPI device The identification engine in the application uses the corresponding relationship to identify the application protocol to which the data packet belongs. In this way, the DPI device does not need to decrypt the data packet, reducing performance overhead, and the DPI device does not need to analyze the specific character characteristics of the code stream of the application protocol offline, which reduces the The time of stream identification.

Referring to FIG. 3, the following embodiment is another method for providing DPI with information for identifying a data packet provided by an embodiment of the present invention. The method specifically includes:

301. After receiving the data packet, the identification engine in the DPI device queries the correspondence between the application protocol identifier and the network entity information stored locally. If the corresponding application of the network entity corresponding to the data packet cannot be determined according to the stored correspondence protocol identification, send the network entity information corresponding to the data packet to the crawler engine in the DPI device, wherein the network entity corresponding to the data packet is a network entity that needs to identify the application protocol.

Wherein, the network entity corresponding to the data packet may be the source end and/or the destination end of the data packet.

Optionally, the identification engine in the DPI device can also use the deep packet inspection technology to identify the application to which the data packet belongs after it cannot determine the application protocol identifier corresponding to the network entity corresponding to the data packet according to the stored correspondence. Protocol, that is, to identify according to the characteristic characters or characteristic behaviors in the data packet. If the application protocol to which the data packet belongs cannot be successfully identified within a predetermined time, the network entity corresponding to the data packet will be sent to the crawler engine in the DPI device. information.

302. The crawler engine in the DPI device sequentially invokes the crawler program of the application protocol in the crawler program set to send a detection request message to the network entity that needs to identify the application protocol, until it receives a response message indicating that the detection is successful returned by the network entity.

For example, the crawler engine in the DPI device can call the BT program in the crawler program set to send a detection request message to the network entity that needs to identify the application protocol. Then perform the follow-up 303; if the response message indicating that the detection is successful sent by the network entity that needs to identify the application protocol is not received, continue to call the QQ program to send a detection request message to the network entity that needs to identify the application protocol until the need to identify Until the response message sent by the network entity of the application protocol indicating that the detection is successful.

303. The crawler engine in the DPI device establishes a corresponding relationship between the identifier of the application protocol used in the detection request message corresponding to the response message and the information of the network entity that needs to identify the application protocol.

Continuing the above example, if the QQ program is invoked to send a detection request message to the network entity that needs to identify the application protocol, and a response message indicating that the detection is successful is received from the network entity that needs to identify the application protocol, then establish the QQ program ID and the application protocol that needs to be identified The corresponding relationship between the information of the network entity, that is, the establishment of the corresponding relationship between the QQ program identifier, the address of the network entity that needs to identify the application protocol, and the identifier of the transport layer protocol used by it.

304. The crawler engine in the DPI device sends identification information to the identification engine in the DPI device, where the identification information includes the corresponding relationship. Optionally, the identification information may also include an aging suggestion time for the corresponding relationship.

In the embodiment of the present invention, the crawler program of the application protocol in the crawler program set is sequentially called to send a detection request message to the network entity that needs to identify the application protocol, until the response message returned by the network entity indicating that the detection is successful is received, and the information of the network entity is established. The corresponding relationship with the application protocol identifier used in the detection request message of the corresponding response message is sent to the DPI device, so that the identification engine in the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs, so that the DPI device does not need to decrypt The data packet reduces the performance overhead, and the DPI device does not need to analyze the specific character characteristics of the code stream of the application protocol offline, which reduces the time for code stream identification.

FIG. 4 shows a method for a DPI device to identify the application protocol to which a data packet belongs according to an embodiment of the present invention, wherein the network entity in this embodiment may be a Peer (peer-to-peer network entity), which includes:

401. The recognition engine in the DPI device receives a data packet, and the data packet carries a quintuple.

The five-tuple includes: the IP address of the source network entity, the IP address of the destination network entity, the port number of the source network entity, the port number of the destination network entity, and the transport layer protocol identifier.

Wherein, the source network entity is the network entity that sends out the data packet, and the destination network entity is the network entity that receives the data packet.

402. The identification engine in the DPI device determines the application protocol used by the data packet according to the stored correspondence.

Wherein, the saved corresponding relationship includes: the corresponding relationship between the identifier of the application protocol and the information of the network entity using the application protocol, wherein the information of the network entity using the application protocol includes: the address of the network entity using the application protocol, and the identifier of the transport layer protocol used by the network entity, the address of the network entity includes the IP address and port number of the network entity.

Specifically, this step compares the five-tuple with the information of the network entity, if the transport layer protocol identifier in the information of the network entity is the same as the transport layer protocol represented by the transport layer protocol identifier in the five-tuple, and, if the network The address of the entity is the same as the IP address and port number of the source network entity in the quintuple, or the same as the IP address and port number of the destination network entity, then it is determined that the application protocol used by the data packet is the network entity in the corresponding relationship The corresponding application protocol.

It should be noted that if the transport layer protocol identifier in the information of the network entity is different from the transport layer protocol indicated by the transport layer protocol identifier in the quintuple, or if the address of the network entity is the same as the source network in the quintuple If the IP address and port number of the entity and the destination network entity are different, the application protocol applicable to the data packet can be identified according to the characteristic characters or characteristic behaviors in the data packet. Behavior recognition is common knowledge in the art, and will not be repeated here.

In order to make the above-mentioned technical solution clearer, the following examples are given to illustrate: Assume that the corresponding relationship established by the recognition engine is:

TCP 192.168.0.1:5566<-->Bittorent_DATA 1800;

UDP 192.168.0.1:5566<-->Bittorent_DHT_Control 1801;

UDP 192.168.0.16:5566<-->Bittorent_DHT_Control 1801;

TCP 192.168.0.16:5566<-->Bittorent_DATA 1800;

Among them, TCP and UDP are the names of the transport layer protocols; Bittorent_DATA and DHT_Control are the names of the application protocols respectively, and 1800 and 1801 are the IDs of the application protocols respectively.

Assume that the transport layer protocol used by the network entity carried in the received data packet is the TCP protocol, the IP address of the source network entity is 192.168. The name of the application protocol corresponding to the entity is Bittorent_DATA, and the ID of the application protocol is 1800.

It should be noted that the above recognition engine and crawler engine can be deployed on the same device, that is, the DPI device, or they can be deployed on different devices, that is, the recognition engine is deployed on the DPI device, and the crawler engine is deployed on different devices. . When the recognition engine and the crawler engine are deployed on different devices, they can be located in the same network or in different networks. The recognition engine and crawler engine can be deployed according to some special requirements. For example, because the recognition engine will be deployed on the DPI device, and the DPI device also has the function of decrypting encrypted data packets. If the DPI is deployed in the operator's computer room, it may be Sensitive information of some operators is collected, so some operators do not allow DPI devices to privately access the external network of operators to prevent DPI devices from leaking sensitive information. In this case, if DPI devices want to collect crawler engine information, It needs to be deployed in the operator's external network.

It should be noted that protocols such as Bittorent can be subdivided into multiple sub-protocols, such as Bittorent non-DHT (Distributed Hash Table, distributed hash table) protocol and Bittorent DHT protocol. The following two embodiments take the application protocol as Bittorent non-DHT (Distributed Hash Table, Distributed Hash Table) protocol and Bittorent DHT protocol as examples respectively to describe the process of crawler engine obtaining Peer information and reporting identification information to identification engine.

Referring to Fig. 6, this embodiment describes the process that the crawler engine acquires Peer information and reports the identification information to the identification engine by taking the application protocol as the Bittorent non-DHT protocol and the network entity as Peer (corresponding network entity) as an example. In this embodiment, based on specific resources To search for Peer information, specific resources can be pre-configured or obtained from other devices. The process of obtaining Peer information and reporting identification information to the identification engine specifically includes:

601. The crawler engine uses the crawler program of the application protocol to create a search request, uses the search request to search for a specific resource, and searches for a torrent file of the specific resource.

Wherein, the specific resource may be a video resource, such as the film The Founding of a Nation, or an audio resource, which does not affect the realization of the present invention.

602. The crawler engine extracts Tracker information and Peer information by parsing the seed file, and writes the Peer information into the Peer list.

Wherein, the Peer information extracted in this step is the information of the Peer using the application protocol. Specifically, the Peer information includes: the address of the Peer and the identification of the transport layer protocol used by the Peer.

603. The crawler engine uses the crawler program of the application protocol to create a query request, sends a query request to the corresponding Tracker according to the extracted Tracker information, and the Tracker returns a query response to the crawler engine, which includes: use the application protocol and be tracked by the Tracker The address of the Peer and the transport layer protocol identifier used by the Peer.

Optionally, in this step, the crawler engine can also judge the validity of the received query response, and when the query response is valid, execute step 605, otherwise end the process. Wherein, the specific method for the crawler engine to judge the validity of the received query response may be: judge whether the query response is valid through the message format of the received query response; or judge the validity of the query response through the content of the received query response Whether the query response is valid; it can also be judged whether the query response is valid according to the interaction process of the query request and the query response, or it can be judged whether the query response is valid by other means, which does not affect the realization of the present invention.

604. The crawler engine uses the Peer extracted in step 602 as a query source, uses the crawler program of the application protocol to create a query request, and sends a query request to the Peer as the query source, and the Peer returns a query response to the crawler engine, which includes: The address of the Peer associated with the Peer and the transport layer protocol used by the Peer.

Wherein, the Peer associated with the Peer is a Peer that uses the application protocol and has the same resources as the Peer.

Optionally, in this step, the crawler engine can also judge the validity of the received query response, and when the query response is valid, subsequently write the address of the Peer associated with the Peer and the transport layer protocol used into the Peer list . Wherein, the specific manner of judging the validity of the received query response by the crawler engine is the same as that in step 603 and will not be repeated here.

605. The crawler engine removes the repeated Peer information and invalid Peer information in steps 603 and 604, and writes the remaining Peer information into the Peer list.

606. The crawler engine judges whether the number of Peers written in the Peer list has reached the threshold, if yes, execute 608, if not, execute 607;

607. The crawler engine takes the above remaining Peers (ie effective Peers) as the updated query source, returns to step 604, uses the crawler program of the application protocol to create a query request, and sends the query request to the updated query source.

608. The crawler engine sends identification information to the identification engine, and the identification information includes: the Peer information in the Peer list, the identification of the application protocol, and the aging suggestion time.

Wherein, the Peer information in the Peer list includes: the address of the Peer and the identification of the transport layer protocol used by the Peer.

Wherein, this step and the previous steps can be implemented in the same thread, or in different threads or processes, without affecting the implementation of the present invention.

Referring to Fig. 7, this embodiment is that the application protocol is the Bittorent DHT protocol, and the network entity is Peer (corresponding network entity) as an example to describe the process that the crawler engine obtains Peer information and reports identification information to the identification engine and includes:

701. The crawler engine uses the crawler program of the application protocol to create a query request, and sends the query request to a known Peer, and the Peer returns a query response to the crawler engine, which includes: information about the Peer associated with the Peer.

Wherein, the Peer associated with the Peer is: the Peer that uses the application protocol and has the same resources as the Peer; the information of the Peer associated with the Peer includes: the address of the Peer associated with the Peer and the Peer associated with the Peer The identifier of the transport layer protocol used.

Wherein, the known Peer in this step is a predetermined Peer, or a valid Peer carried in the received query response message and has not been a query source. Wherein, the predetermined Peer is the Peer that is predetermined as the query source of the Bittorent DHT protocol.

Optionally, after the crawler engine receives the query response, it can judge the validity of the query response. When the query response is valid, the method of judging the validity of the query response is the same as the corresponding description in the above embodiment, and will not be repeated here. . When the query response is valid, perform subsequent steps.

702. The crawler engine removes the invalid Peer information in step 701, and writes the remaining valid Peer address information into the Peer list.

703. The crawler engine judges whether the number of Peers written in the Peer list has reached the threshold, if yes, execute 705, and if not, execute 704.

704. The crawler engine takes the Peer that is valid among the peers fed back by the query response and has not been used as a query source as an updated query source, and returns to step 701 to create a query request using the crawler program of the application protocol, and sends a query request to the updated query source .

705. The crawler engine sends identification information to the identification engine, and the identification information includes: the Peer information in the Peer list, the identification of the application protocol, and the aging suggestion time.

Wherein, this step and the previous steps can be implemented in the same thread, or in different threads or processes, without affecting the implementation of the present invention.

Optionally, after the identification engine obtains the identification information, it can share it with other identification engines, wherein the identification information includes: the correspondence between the identification of the application protocol and the information of the network entity using the application protocol, and the identification information also It may include: the aging suggestion time of the corresponding relationship. Specifically, there are two sharing methods as follows:

1. The recognition engine in each DPI device reports the recognition information to the information sharing control center, so that the recognition engines in other DPI devices can obtain the recognition information from the information sharing control center, as shown in FIG. 8 .

2. The recognition engines in each DPI device notify each other of the recognition information, as shown in FIG. 9 .

Among them, the recognition engine in the DPI device can obtain the recognition information in the following ways: the first way: the recognition information sent by the crawler engine to the recognition engine, as described in the above-mentioned embodiments; the second way: the recognition information in the DPI device The engine identifies the application protocol applicable to the data packet according to the characteristic characters or characteristic behaviors in the data packet, and records the identification information, which includes: the correspondence between the identification of the application protocol and the information of the network entity that sends and/or receives the data packet relation. Optionally, the identification information may also include: aging suggestion time.

Referring to Fig. 10, the embodiment of the present invention provides a kind of web crawler, and it comprises:

The establishing unit 50 is configured to use a crawler program of an application protocol to establish a corresponding relationship between the application protocol and the network entity information using the application protocol;

The sending unit 60 is configured to send identification information including the corresponding relationship to the DPI device, so that the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs; wherein, the network entity information using the application protocol includes: The address of the network entity using the application protocol and the identifier of the transport layer protocol used by the network entity using the application protocol.

For the description of the information of the network entity, etc., refer to the relevant description under step 101 of the method embodiment, and details are not repeated here.

In the embodiment of the present invention, the crawler program of the application protocol is used to establish the corresponding relationship between the application protocol and the information of the network entity using the application protocol, and send it to the DPI device, so that the DPI device can use the corresponding relationship to identify the application protocol to which the data packet belongs In this way, the DPI device does not need to decrypt the data packets, reducing performance overhead, and the DPI device does not need to analyze the specific character characteristics of the code stream of the application protocol offline, reducing the time for code stream identification.

Referring to FIG. 11 , in one embodiment, the establishment unit 50 specifically includes: a query source determination unit 51, configured to determine a network entity as a query source; wherein, the network entity as a query source uses the application protocol; query Unit 52, configured to use the crawler program of the application protocol to obtain information of the network entity associated with the query source from the query source; wherein, the network entity associated with the query source uses the application protocol and is related to the query source. The query source has a network entity with the same resource; and a query source updating unit 53, configured to use a network entity that has not been a query source among network entities associated with the query source as an updated query when the query end condition is not reached source, sending the updated query source to the query unit, triggering the query unit to use the crawler program of the application protocol to acquire information of the network entity associated with the updated query source from the network entity as the updated query source The corresponding relationship establishment unit 54 is used to establish the application protocol and the information of the network entity obtained by the query unit and the information of the network entity determined by the query source determination unit as the query source when the query end condition is reached corresponding relationship. Specifically, the query source determining unit 51 is configured to determine, according to the mapping relationship between the network entity information sent by the DPI device and the application protocol identifier, that the network entity corresponding to the application protocol identifier in the mapping relationship is the network entity serving as the query source, wherein , the DPI device is a mapping relationship between network entity information and application protocol identifiers determined by using a deep packet inspection technology. Or, the query source determining unit 51 determines that the network entity extracted from the seed file of the specific resource and using the application protocol is the network entity as the query source; or, the query source determining unit 51 determines that the network entity extracted from the seed file of the specific resource The network entity tracked by the tracker Tracker and using the application protocol is the network entity serving as the query source.

This implementation method uses the crawler program of the application protocol in order to obtain the Peer information associated with the query source from the query source, and then uses the obtained query source as an updated query source, and then obtains the peer information associated with the updated query source. Peer information, in this way, the Peer information using the application protocol in the network is obtained, and the corresponding relationship between the application protocol and the Peer information using the application protocol is sent to the identification engine in the DPI device, so that the identification engine in the DPI device The engine uses this correspondence to identify the application protocol to which the data packet belongs.

Referring to FIG. 12, in another embodiment, the establishing unit 50 specifically includes: a calling unit 56, configured to sequentially call the crawler program of the application protocol in the crawler program set to send a detection request message to the network entity in the network until the received until the response message returned by the network entity indicating that the detection is successful; the correspondence relationship establishing unit 57 is configured to establish a correspondence relationship between the application protocol used in the detection request message corresponding to the response message and the information of the network entity. In this way, the web crawler also includes: a receiving unit 61, configured to receive the information sent by the DPI device that needs to identify the network entity of the application protocol; a calling unit 56, used to sequentially invoke the crawler program of the application protocol in the crawler program set Sending a probe request message to the network entity that needs to identify the application protocol until receiving a response message indicating that the probe is successful returned by the network entity that needs to identify the application protocol. In this embodiment, the crawler program of the application protocol in the crawler program set is sequentially called to send a detection request message to the network entity that needs to identify the application protocol, until the response message returned by the network entity indicating that the detection is successful is received, and the information of the network entity is established. The corresponding relationship with the application protocol used by the detection request message corresponding to the response message is sent to the DPI device, so that the identification engine in the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs.

In this embodiment, the crawler program of the application protocol in the crawler program set is sequentially invoked to send a detection request message to the network entity that needs to identify the application protocol, until the response message returned by the network entity indicating that the detection is successful is received. The corresponding relationship between the network entity information and the application protocol used in the detection request message corresponding to the response message is sent to the DPI device, so that the identification engine in the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs.

Referring to FIG. 13 , an embodiment of the present invention provides a network system, which is characterized in that it includes: a web crawler 70 and a DPI device, wherein,

The web crawler 70 is configured to use a crawler program of an application protocol to establish a corresponding relationship between the application protocol and the information of the network entity using the application protocol; wherein, for the description of the information of the network entity, etc., refer to the step 101 of the method embodiment Relevant descriptions will not be repeated here.

The DPI device 80 is configured to receive the identification information sent by the crawler engine, and use the corresponding relationship to identify the application protocol to which the data packet belongs.

Wherein, the web crawler 70 can be integrated on the DPI device 80 . The structure of the web crawler 70 is similar to the above-mentioned embodiments shown in FIG. 10 , FIG. 11 and FIG. 12 , and will not be repeated here.

In order to realize the above identification information sharing, in one embodiment, the network system further includes: an information sharing control center 90, wherein,

The DPI device 70 is further configured to send the identification information sent by the web crawler to the information sharing control center 90 . The information sharing control center 90 is configured to receive the identification information sent by the DPI device 70 for other DPI devices in the network except the DPI device 70 to obtain the identification information from the information sharing control center 90 . Optionally, the DPI device 70 may also send the identification information obtained through the deep packet inspection technology to the information sharing control center, wherein the identification information obtained through the deep packet inspection technology includes: the address of the source end and/or destination end of the data packet and the correspondence between the transport layer protocol used and the application protocol to which the data packet belongs, optionally, it may also include: an aging suggestion time for the correspondence.

In another implementation manner, the DPI device 70 is further configured to send the identification information sent by the web crawler to other DPI devices in the network except the DPI device.

In the embodiment of the present invention, the crawler program of the application protocol is used to establish the corresponding relationship between the application protocol and the information of the network entity using the application protocol, and send it to the DPI device, so that the DPI device can use the corresponding relationship to identify the application protocol to which the data packet belongs In this way, the DPI device does not need to decrypt the data packets, reducing performance overhead, and the DPI device does not need to analyze the specific character characteristics of the code stream of the application protocol offline, reducing the time for code stream identification.

Wherein, above-mentioned DPI device, can be used in the network system shown in Figure 14, wherein, DPI device can exist independently with GGSN (Gateway GPRS Support Node, gateway GPRS support node), also can be integrated on a device with GGSN, does not Affect the realization of the present invention. In the figure, GGSN is connected to GPRS (General Packet Radio Service)/UMTS (Universal Mobile Telecommunications System) through SGSN (Serving GPRS Support Node, serving GPRS support node), and the DPI device is connected to the Internet through a firewall. Connection, Peer is located in GPRS/UMTS or Internet.

Those of ordinary skill in the art can understand that all or part of the steps in the methods of the above embodiments can be implemented through a program to instruct related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, Disk or CD, etc.

The method for providing information for identifying data packets to DPI provided by the embodiments of the present invention, the web crawler and the network system have been introduced in detail above. In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The above The description of the embodiment is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application. As mentioned above, the contents of this specification should not be construed as limiting the present invention.

Claims (11)

1. A method of providing information for identifying a data packet, comprising: Using the crawler program of the application protocol to establish a corresponding relationship between the application protocol and the network entity information using the application protocol; Sending identification information including the corresponding relationship to the deep packet inspection DPI device, so that the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs; Wherein, the network entity information using the application protocol includes: the address of the network entity using the application protocol and the transport layer protocol identifier used by the network entity using the application protocol; the crawler program using the application protocol establishes The corresponding relationship between the application protocol and the information of the network entity using the application protocol includes: A. Determine a network entity serving as a query source; wherein, the network entity serving as a query source uses the application protocol; B. Using the crawler program of the application protocol to obtain the information of the network entity associated with the query source from the query source; wherein, the network entity associated with the query source uses the application protocol and is connected to the query source Network entities that have the same resources; C. Use the network entity associated with the query source that has not been used as a query source as an updated query source. When the query end condition is not met, return to step B. At this time, the query in step B The source is the query source of the update; when the query end condition is reached, the corresponding relationship between the application protocol and the information of the network entity using the application protocol is established. 2. The method according to claim 1, wherein said determining the query source comprises: According to the mapping relationship between the network entity information sent by the DPI device and the application protocol identifier, it is determined that the network entity corresponding to the application protocol identifier in the mapping relationship is the network entity serving as the query source. 3. The method according to claim 1, wherein, using the crawler program of the application protocol, establishing the corresponding relationship between the application protocol and the information of the network entity using the application protocol comprises: Sequentially call the crawler program of the application protocol in the crawler program set to send the detection request message to the network entity until the response message returned by the network entity indicating that the detection is successful is received, and the application protocol and the detection request message corresponding to the response message are established. The corresponding relationship of the information of the network entities. 4. The method according to claim 3, characterized in that, before the crawler program of the application protocol in the crawler program set is called sequentially to send the detection request message to the network entity, the method further comprises: Receive the information sent by the DPI device that needs to identify the network entity of the application protocol; The sequentially invoking the crawler program of the application protocol in the crawler program set to send the detection request message to the network entity includes: The crawler program that sequentially invokes the application protocol in the crawler program set sends a detection request message to the network entity that needs to recognize the application protocol. 5. A web crawler providing information for identifying packets, comprising: An establishing unit, configured to use a crawler program of an application protocol to establish a corresponding relationship between the application protocol and the network entity information using the application protocol; A sending unit, configured to send identification information including the corresponding relationship to the DPI device, so that the DPI device uses the corresponding relationship to identify the application protocol to which the data packet belongs; Wherein, the network entity information using the application protocol includes: the address of the network entity using the application protocol and the transport layer protocol identifier used by the network entity using the application protocol; The building unit includes: A query source determining unit, configured to determine a network entity serving as a query source; wherein, the network entity serving as a query source uses the application protocol; The query unit is configured to use a crawler program of an application protocol to acquire information of a network entity associated with the query source from the query source; wherein, the network entity associated with the query source uses the application protocol and is related to the query source. A network entity that owns the same resource as the source of the query; The query source updating unit is configured to use a network entity associated with the query source that has not been used as a query source as an updated query source when the query end condition is not reached, and send the updated query source to all The query unit is triggered to use the crawler program of the application protocol to obtain the information of the network entity associated with the updated query source from the network entity as the updated query source; A correspondence relationship establishing unit, configured to establish a correspondence between the application protocol and the information of the network entity acquired by the query unit and the information of the network entity as the query source determined by the query source determination unit when the query end condition is reached relation. 6. web crawler according to claim 5, is characterized in that, The query source determining unit is configured to determine, according to the mapping relationship between the network entity information sent by the DPI device and the application protocol identifier, that the network entity corresponding to the application protocol identifier in the mapping relationship is the network entity serving as the query source. 7. web crawler according to claim 5, is characterized in that, The building unit includes: The calling unit is used to sequentially call the crawler program of the application protocol in the crawler program set to send a detection request message to the network entity in the network until a response message indicating that the detection is successful returned by the network entity is received; The correspondence relationship establishment unit is configured to establish a correspondence relationship between the application protocol used by the probe request message corresponding to the response message and the information of the network entity. 8. web crawler according to claim 7, is characterized in that, Also includes: The receiving unit is configured to receive the information sent by the DPI device that needs to identify the network entity of the application protocol; The calling unit is used to sequentially call the crawler program of the application protocol in the crawler program set to send a detection request message to the network entity that needs to identify the application protocol, until the network entity that needs to identify the application protocol returns a response indicating that the detection is successful response message. 9. A network system providing information for identifying data packets, comprising: the web crawler and deep packet inspection DPI device according to any one of claims 5-8, wherein, The DPI device is configured to receive the identification information sent by the web crawler, and use the corresponding relationship to identify the application protocol to which the data packet belongs. 10. The network system according to claim 9, characterized in that, The DPI device is further configured to send the identification information sent by the web crawler to an information sharing control center, so that other DPI devices in the network other than the DPI device can obtain the information from the information sharing control center. identifying information; Alternatively, the DPI device is further configured to send the identification information sent by the web crawler to other DPI devices in the network except the DPI device. 11. The network system according to claim 9, characterized in that, The identification information also includes: the aging suggestion time of the corresponding relationship; The DPI device is further configured to control the corresponding relationship to be invalid when the aging suggestion time arrives.

Images