CN105095070A - Method and system for obtaining QQ group data base on test assembly of browser - Google Patents

Abstract

The invention discloses a method and system for acquiring QQ group data based on a browser test component. By providing the browser test component, user behavior simulation is realized, and in this way, the login operation of QQ on the Web side is realized, and combined with parallel design, it is oriented to QQ. The group realizes the data acquisition of instant chat information and non-instant information. The present invention can avoid manual login operation, take into account the integrity of the acquired data while quickly acquiring data, and at the same time, the cost investment in later maintenance is smaller, and can be quickly modified and put into use according to version changes.

Description

QQ group data acquisition method and system based on browser test component

technical field

The invention relates to the field of social network data acquisition, and is a method and system for acquiring QQ group data based on a browser test component to realize more targeted and accurate QQ group data acquisition.

Background technique

Tencent QQ is currently the most influential social network in China. QQ registered users have grown from 2 people (Ma Huateng and Zhang Zhidong) in 1999 to hundreds of millions of users. In August 2013, the number of QQ active users reached 818.5 million. At 21:11 on April 11, the number of online users exceeded 200 million. QQ users can post text and picture messages, upload album pictures, upload files, organize event voting and other operations in the QQ group. Now it has become a representative work of Tencent. . The development and popularization of IM are necessarily related to its own characteristics. IM has the characteristics of real-time, online and text interaction, and meets people's needs for communication and collaboration in daily life and office environment.

Currently, there are two main methods for obtaining information on QQ:

(1) Database cracking, that is, cracking the embedded database accessed after the instant messaging software runs. This method is very difficult to implement technically for QQ group information collection, because the local database of the QQ client has a lot to do with the change of the QQ version. Usually, the database file is encrypted by MD5, or the data and the database file are double-encrypted. The difficulty is quite high, and the cracking cost cannot be controlled.

(2) Simulate the communication process of the web version of QQ, that is, simulate HTTP get and post requests to those instant messaging software that provides web access to obtain data. Although QQ has a web access method (such as SmartQQ), this method is heavy and cumbersome for collecting QQ. It is necessary to capture and analyze each instruction accessing QQ, and obtain the corresponding get or post address and content. In addition, its communication process essentially uses asynchronous loading technology. In the Web communication process, there are often multiple request dependencies for one data request, and the next request often needs the result of the previous request as input. The analysis of this communication process is complicated and difficult to cover all the returned result. With the upgrade of the WebQQ protocol version, the acquisition system also needs to undergo corresponding potential large-scale modification accordingly. The later work is heavy and the maintenance is not easy.

Due to the private nature of the QQ client protocol, the frequent changes of the QQ protocol on the WEB side, and the limitations of the QQ open API, it is not suitable for large-scale collection.

The browser testing component (browserautomation) is a testing tool based on the browser kernel, which can be used without special settings. It is mainly oriented to web applications at first, and conducts compatibility tests, function tests, and stress tests between web applications and browsers by simulating user behavior. Based on the data acquisition method of the browser test component, the browser test interface is driven by a high-level program to realize automated web browsing. After the browser completes JS parsing and webpage rendering, the data loaded by AJAX can be obtained by locating and parsing the DOM elements of the webpage. This method can obtain all social network data and avoid the restrictions of the official API method.

To sum up, the current data acquisition technologies and methods are difficult to be universal due to different platform characteristics. At the same time, data acquisition versions for some social networks are highly dependent, less flexible, and less robust. In the long run, maintenance The workload of upgrading is huge.

Contents of the invention

Based on the above analysis, in order to realize the collection of QQ group information and messages, the present invention proposes a method and system for acquiring QQ group data based on a browser test component.

The present invention mainly comprises two aspects:

(1) Obtain the data information and instant chat messages of the QQ group through the browser test component.

(2) Parallelized QQ group message acquisition based on the browser test component.

Specifically, the present invention includes the following:

A method for acquiring QQ group data based on a browser testing component, comprising the following steps:

1) On the client side, use the browser to test the port of the component, start the browser, and jump to the QQ page on the Web side;

2) Simulate user behavior and log in to the QQ collection account that wants to obtain QQ group data;

3) By monitoring the DOM tree of the page where the QQ collection account is logged in, real-time information and non-real-time information are continuously obtained and stored in the MySQL database.

Further, the browser testing component is a core component that is initialized before starting the browser, and injects a JavaScript library into the browser to execute the command request sent by the client.

In the present invention, we introduce the application of a headless browser. The advantage of a headless browser is that it includes a browser kernel but has no UI interface, so that computer resource occupation can be reduced by avoiding graphics rendering and loading. However, the present invention supports browsers with interfaces (such as Chrome, Firefox, etc.) at the same time. For the asynchronous loading technology commonly used at present, better performance can also be obtained based on browsers with interfaces.

Further, step 1) comprises: realize a plurality of browsers to start by different ports of the Hub component in the browser test component, and the client starts a thread for each browser (can be written for calling the browser test component according to actual needs) and data processing and other functional programs), that is, to realize the control of multiple browsers through multi-threading, and each thread sends command requests to the browsers started separately. Through this process, the parallelized acquisition of QQ group messages based on the browser test component can be realized.

Further, the user behavior in step 2) includes behavior operations such as clicking, inputting, and dragging.

Further, step 2) includes: inputting the account number and password of the QQ collection account when logging in, and judging whether a manual input of the verification code is required by locating the page verification code element node, and prompting the operator to manually input the verification code if necessary, and waiting for the above process to be completed , click the login button to log in.

Further, in step 3), the client sends the corresponding parameters specified by the browser test component by calling different interfaces provided by the browser test component, and the browser test component receives the parameters, parses the client request, and then sends a JS command notification through HttpProxy The JavaScript library performs response operations to monitor the DOM tree of the browser page.

Further, in step 3), obtaining instant information includes the following steps:

1‐1) After successful login and redirection, wait for the page elements to be loaded, simulate user behavior and click to enter the QQ group interface.

1-2) Allocate the time slice of a single group through the task scheduling strategy (the strategy can be set according to actual needs), poll according to the strategy, monitor and obtain new messages.

1‐3) Repeat steps 1-2) until all groups are polled.

Further, in step 3), obtaining non-instant information includes the following steps:

2‐1) After the login jump is successful, wait for the page elements to be loaded, obtain the list of groups that the current account has joined on the page, and fully load all groups by simulating user clicks.

2‐2) To poll according to the group list, first enter the group member list page. On this page, group announcements and group member lists can be obtained by locating page elements. Determine whether the group announcement has been updated, if so, write it into the database, if not, do not write into the database. It is judged that the list of group members has changed. If a new member joins, it will be written into the database. If a member has left the group, the status of the member in the database will be modified.

2-3) Jump to the group shared file page, obtain the shared file information through page element positioning, judge whether there is an update, write it into the database if there is one, and skip it if not.

2-4) Jump to the group album page, obtain the group album information through page element positioning, judge whether there is an update, write it into the database if there is one, and skip it if not.

2-5) If the polling of the group list has not been completed, then jump to step 2-2). If the polling of the group list is completed, the browser is closed.

Further, the ways of locating the page elements in steps 2-2), 2-3) and 2-4) include XPath way or CSS selector way. Since XPath is a language used to determine elements in an XML document, and a DOM tree is a tree structure of XML, which has a more comprehensive function of locating nodes, it is preferable to use XPath to locate page elements.

A QQ group data acquisition system based on browser testing components, including:

The browser test component is used to start the browser through the port and simulate the user to log in to the QQ on the Web side;

The data collection module is used to obtain instant information and non-instant information by monitoring the page elements that have logged into QQ;

The data storage module is used for storing the obtained QQ group data.

The positive effect of the present invention is as follows:

The invention is based on the browser test component, combined with the parallel design, and realizes data acquisition of instant chat information and non-instant information for QQ groups. The present invention can avoid manual login operation and avoid a considerable part of manual operation.

The traditional collection of browser-based test components needs to open the browser UI, which consumes a lot of system resources, slows down the loading process and affects the collection efficiency, and at the same time, the scale of parallel collection is affected by the huge resource overhead. By operating the headless browser (headless browser) based on browser test component social network data acquisition, reduce system resource consumption, increase the number of parallel collection points under certain hardware resource conditions, reduce unnecessary time expenditure, and reduce system resources Overhead, accelerating the speed and scale of data acquisition. Then set the monitoring mechanism to monitor the changes of DOM elements to perform operations such as data positioning and analysis to obtain data. Cooperate with parallel distributed technology and message bus task scheduling to achieve large-scale social platform data acquisition on different platforms.

Realize user behavior simulation based on browser test components, including click, input, drag and drop and other behavioral operations. In this way, the login operations such as QQ on the Web side are realized. Compared with the QQ client cracking, this method avoids the problems of long cracking cycle and high cracking cost, and the implementation difficulty is low, and no reverse analysis and a large number of debugging are required. Compared with the communication process simulation based on AJAX technology, etc., it not only avoids the complicated analysis of the underlying request process, but also reduces the packet capture analysis process, because usually, a data request in the communication process may contain multiple front and back dependencies, and some requests The format parameters are randomly generated by the server, and it is difficult to determine the generated algorithm. During the simulation request process, there is a possibility that the server may detect anomalies and freeze them; and avoid large-scale modification of subsequent system codes brought about by the update of the protocol version, and can According to the version change, it only needs to modify the user operation logic to adapt to the new version, and put it into the actual data acquisition application quickly and effectively.

This system does not depend on the underlying communication protocol of WebQQ, and it can run across versions.

The present invention does not depend on the lower-level protocol, and can quickly realize the upgrade along with the version upgrade, adjust the acquisition mode, and adapt to the new version for data acquisition.

According to the characteristics of WebQQ, the present invention can quickly obtain QQ related information, and at the same time, the present invention supports cross-platform data acquisition and can adapt to different system environments.

Multiple browsers can be started through different interfaces of the browser test component. The client starts a thread for each browser and sends instructions separately. Since the browsers are independent of each other, data acquisition between browsers will not interfere. Since the use of browser test components to simulate user behavior has certain general behaviors in web page operations and certain general functions in data acquisition, by encapsulating these general behaviors and functions, it is possible to obtain data for social networks. It is easy to reuse and reduce engineering time overhead. Currently, multiple group information can be obtained concurrently.

This method highlights the feature of "what you see is what you get". As long as it can be seen on the web page, the system can obtain it. Moreover, this method only depends on the structure of web page elements. As long as the structure remains unchanged, there is no need for large-scale modification in the future. Easy maintenance.

To sum up, the present invention not only quickly acquires data, but also takes into account the integrity of the acquired data, and at the same time requires less investment in later maintenance, and can be quickly modified and put into use according to version changes.

Description of drawings

Fig. 1 is a flow chart of QQ group data collection in the embodiment of the present invention.

FIG. 2 is a frame diagram of a QQ group data acquisition system based on a browser testing component according to an embodiment of the present invention.

Fig. 3 is a schematic diagram of a web-side SmartQQ page according to an embodiment of the present invention.

Fig. 4 is a schematic diagram of logging in to the SmartQQ page at the web end according to the embodiment of the present invention.

Fig. 5 is a schematic diagram of a group interface according to an embodiment of the present invention.

FIG. 6 is a schematic diagram of a group message of a certain group according to an embodiment of the present invention.

Fig. 7 is a schematic diagram of outputting and obtaining the information shown in Fig. 6 in the console according to the embodiment of the present invention.

Fig. 8 is a schematic diagram of a group list in a webpage according to an embodiment of the present invention.

Fig. 9 is a schematic diagram of a list of group members according to an embodiment of the present invention.

Fig. 10 is a schematic diagram of a group shared file according to an embodiment of the present invention.

Fig. 11 is a schematic diagram of a group photo album according to an embodiment of the present invention.

Fig. 12 is a design diagram of parallelization of QQ group data collection in the embodiment of the present invention.

Detailed ways

The flow of QQ group data collection in the present invention is shown in Figure 1, and the system frame diagram is shown in Figure 2, wherein the acquisition of QQ group data includes instant information acquisition and non-instant information acquisition. details as follows:

1) The instant information acquisition based on the browser test component can be divided into the following steps:

a) Through the interface of the browser test component, start the browser, and jump to the QQ page on the Web side. As shown in Figure 3, open SmartQQ (this is a version of qq on the web side).

b) Simulate user behavior, input and collect account number and password, and judge whether manual input of verification code is required, and if necessary, prompt the operator to manually input (the verification code used by QQ on the web side has been processed, and it is currently difficult to obtain correct verification through image comparison code), after the above is completed, click Login. As shown in Figure 4, automatically enter the collection account number and password to log in.

c) After the login jump is successful, wait for the page elements to be loaded, simulate user behavior and click to enter the group interface. As shown in Figure 5, after the jump is successful, automatically click on the corresponding label (respectively "Contacts" -> "Group") to enter the group page, and you can get the group list and so on.

d) Allocate the time slice of a single group through a certain task scheduling strategy, poll according to the strategy, monitor and obtain new messages. As shown by the arrow in Figure 6, a user in a random group sent a chat message "233333", and the obtained information was output on the console, including the speaker, the content of the speech, and the group to which he belonged, as shown in Figure 7 indicated by the arrow.

e) Repeat step d) until all groups are polled.

f) It is worth mentioning that the task scheduling strategy in step d) can be chosen by oneself. Generally, it is based on the time slice polling method. In order to reduce the delay of data acquisition, it can be based on a certain strategy or ratio according to the amount of group information generated. Allocate time slices.

2) Acquisition of non-instant information based on the browser test component can be specifically divided into the following steps:

a) Test the component port of the component through the browser, start the browser, and jump to the homepage of the QQ group on the Web side.

b) Simulate user behavior, input and collect account number and password, and judge whether manual input of verification code is required, and if necessary, prompt the operator to manually input (the verification code used by QQ on the web side has been processed, and it is currently difficult to obtain correct verification through image comparison code), after the above is completed, click Login.

c) After the login jump is successful, wait for the page elements to be loaded, obtain the list of groups that the current collection account has joined on the page, check whether there are labels such as "more groups" on the page, and fully load all groups by simulating user clicks, as shown in the figure 8.

d) Perform polling according to the group list, first enter the group member list page, as shown in Figure 9. On this page, the group announcement and group member list can be obtained by locating the XPath element. Determine whether the group announcement has been updated, if so, write it into the database, if not, do not write into the database. It is judged that the list of group members has changed. If a new member joins, it will be written into the database. If a member has left the group, the status of the member in the database will be modified.

e) Jump to the group shared file page, as shown in Figure 10, locate the page elements through XPath, obtain the shared file information, judge whether there is an update, write it into the database if there is one, and skip it if not.

f) Jump to the group album page, as shown in Figure 11, use XPath to locate page elements, obtain group album information, judge whether there is an update, write it into the database if there is one, and skip it if not.

g) If the polling of the group list has not been completed, then jump to step d). If the polling of the group list is completed, close the browser, switch the collection account, and jump to step a).

In addition, the QQ group data collection in the present invention adopts a parallel design, as shown in FIG. 12 . The browser test component provides a component called Hub, which accepts client code requests through the Hub component, and can start browser processes that are isolated from each other, and then subsequent client code requests are made by the Hub component through the sessionID configured when starting an independent process Distribute commands and interact with the DOM tree of different browser pages respectively. In this way, parallel data acquisition is realized. The message bus is mainly implemented by client programs written on demand, including the acquisition and distribution of collection accounts, and data collection is performed by assigning collection accounts to different threads.

Claims (10)

1., based on QQ group's data capture method of browser testing assembly, comprise the following steps:

1) at the port of client by browser testing assembly, start browser, redirect enters Web and holds the QQ page;

2) modelling customer behavior, the QQ logged in for obtaining QQ group's data gathers account;

3) by gathering the monitoring of the page dom tree that account logs in QQ, constantly obtain instant messages and non-instant information, and be stored into MySQL database.

2. as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, described browser testing assembly carried out initialization before startup browser, and inject JavaScript storehouse to browser, for performing the core component of the command request that client sends.

3. as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, described browser is without interface browser or has interface browser.

4. as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, step 1) comprising: realize multiple browser by the Hub assembly different port in browser testing assembly and start, client starts a thread for each browser, and the browser of each thread to each self-starting sends command request respectively.

5. as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, step 2) comprising: input account and password that QQ gathers account when logging in, by positioning webpage identifying code node element, judge whether to need artificial input validation code, if desired, then alert manually inputs, complete etc. above flow process, click Button Login and log in.

6. as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, step 3) in client send by the distinct interface calling browser testing assembly and provide the relevant parameter that browser testing assembly specifies, browser testing assembly receiving parameter, client-requested is resolved, then sends JS order notice JavaScript storehouse execution operation response by HttpProxy and browser page dom tree is monitored.

7., as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, step 3) in, obtain instant messages and comprise the following steps:

1 ?1) log in redirect success after, wait for page elements loaded, modelling customer behavior is clicked and is entered QQ group interface;

1 ?2) timeslice of single group is distributed by task scheduling strategy, carry out poll by strategy, monitor and obtain new information;

1 ?3) until all group's polls are complete, repeat step 1-2).

8., as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, step 3) in, obtain non-instant information and comprise the following steps:

2 ?1) log in redirect success after, wait for page elements loaded, obtain the group-list that in the page, current account has added, clicked by analog subscriber, all groups of complete loading;

2 ?2) carry out poll according to group-list, first enter the group member list page, at this page, by page elements location obtain group bulletin and group member list; Judge whether group's bulletin has renewal, if any, then write into Databasce, if do not have, does not then do write into Databasce operation; Judge that group member list changes, add if any newcomer, then write into Databasce, if there is member to exit this group, then this member condition in Update Table storehouse;

2 ?3) redirect enters group shared file page, obtain shared file information by page elements location, judged whether renewal, had then write into Databasce, then do not skip;

2 ?4) redirect enters faciation album of paintings or calligraphy face, obtain faciation volume information by page elements location, judged whether renewal, had then write into Databasce, then do not skip;

2 ?5) if the non-poll of group-list is complete, then jump to step 2-2), if group-list poll is complete, then close browser.

9., as claimed in claim 1 based on QQ group's data capture method of browser testing assembly, it is characterized in that, step 2-2), 2-3) and 2-4) described in the mode of page elements location comprise XPath mode or CSS selector mode.

10., based on QQ group's data-acquisition system of browser testing assembly, comprising:

Browser testing assembly, for starting browser by port, and analog subscriber logs in Web end QQ;

Data acquisition module, for by monitoring the page elements having logged in QQ, obtains instant messages and non-instant information;

Data memory module, for storing QQ group's data of acquisition.