CN106598991A

CN106598991A - Web crawler system capable of realizing website interaction and automatic form extraction by conversational mode

Info

Publication number: CN106598991A
Application number: CN201510675362.5A
Authority: CN
Inventors: 不公告发明人
Original assignee: SHANGHAI INTPLE TECHNOLOGY CO LTD
Current assignee: SHANGHAI INTPLE TECHNOLOGY CO LTD
Priority date: 2015-10-19
Filing date: 2015-10-19
Publication date: 2017-04-26

Abstract

The invention discloses a web crawler system capable of realizing website interaction and automatic form extraction by a conversational mode. Considerable information on the Internet can be obtained by an account login way and through specific mouse clicks and keyboard input operations. A traditional web crawler development technology is realized in a way that a browser development tool is used for monitoring the interaction request and response of a browser and a server side during manual operation, intercepted request and response contents are subjected to manual analysis, and codes are developed. The invention provides one set of system which is feasible through verification, and the above construction work of crawler interaction information is realized by a mode close to full automation. The system consists of three elements including an HTML parser, a script transcribing engine and a script operation simulator, wherein the HTML parser is used for parsing tags including a form (<FORM/>), a link (<A/>) and the like from an HTML webpage, the script transcribing engine is used for providing access to a target website for users through an agency mode and recording the data interaction of the browser and the server side, and the script operation simulator operates a script transcribed in the previous step in an explanation way and can realize the interaction of a crawler and the website and information capture by script playing.

Description

A kind of use conversational mode realizes the network crawler system automatically extracted with website interactive form

The present invention provides a set of system feasible through checking, and the work of above-mentioned construction reptile interactive information is realized close to full automatic mode.This system is made up of three key elements：

1, a html parser, for parse from html web page list (<FORM/>) and link (<A/>) label；

2, a script recording engine,, by way of agency, any data interaction for providing the user the access to targeted website, user browser and server end is all recorded for it；

3, script runs simulator, runs the script that previous step is recorded with interpretive mode, play script be capable of achieving reptile and website interact and information crawl.

System has following working condition：

1, record script；

2, script argument, this is an artificial process, the parameter value for being needed setting and being obtained by artificial control；

3, script is played, reptile function is realized；

2nd point, when parametrization is done to script, can also be made some simple modifications to realize that some need the work that exploitation code can just be completed originally.For example, the Row control of circulation is controlled using cumulative variations, calls external function to realize automatic identification to picture validation code, etc..

Description of the drawings

Fig. 1 is the structure chart and workflow schematic diagram of the present invention.

Fig. 2 is to automatically generate and the script sample after artificial parameter.

Specific embodiment

The constitution content and working method of three key elements mentioned above described below.

Html parser is used for parsing html web page content, generates DOM (Document of Model) tree.For analysis reptile with server interaction this task, the form tags being only concerned in dom tree<FORM/>And link label<A/>.

Html parser in the systems described in the present invention, has two effects.First, the task in order to complete script recording engine, during html parser needs the former webpage of replacement<FORM/>With<A/>List is submitted to address or link to point to the URL corresponding to the server address for being changed to script recording engine by the sensing link of label.Second, html parser is responsible for analyzing web page, as the variable of reptile interactive operation next time, or as the final content returning result for needing crawl.The former such as extracts hits in webpage article, thumb up number etc. as extracted option names and option value in the combobox in webpage, the latter.

Script recording engine is the critical component for realizing automatically extracting with website interactive form.It has the WEB interface of a similar search engine, and station address can be input in input frame.

After user submits station address to, recording engine can connect website and obtain webpage.Webpage to returning does following process：

A) webpage is added<Base href=' $ URL) '/>Code, wherein $ URL) it is station address.This forces browser to the relative link on webpage by the network address relative localization specified, and shows the related resources such as picture such that it is able to normal；

B) it is right<A href=" $ { URL } "/>、<FORM action=" $ { URL } "/>、<FRAME src=" $ { URL } "/>、<IFRAME sre=" $ { URL } "/>Label, needs to distort the chained address $ { URL } in attribute.{ URL } is replaced with into the address of engine itself, and the value of former { URL } is carried on this address；

Now, the list done on recording engine by user is submitted to or clickthrough operation, will all be indicated that browser sends GET/POST requests to engine, and will be instead of directly in the originating website.So, the request of browser and original web and response series are all recorded engine and record.

After recording engine records the interaction data of request and response, script source code is translated into, similar to " GET http：// ... Param1=Value1 | Param2=Value2 | ... | ParamN=ValueN { ControlBits } " as code.Wherein, { ControlBits } e.g., distinguishes GET/POST requests, if remove Cookie, etc. for recording some auxiliary informations.The form parameters that Param1=Value1 is submitted to when being then POST modes.

The script that recording engine is generated can artificial further parametrization.Username and password is characterized than Param1=Value1 as described above and Param2=Value2, general shape is as username=jackey and password=123456, then the parameter that can be set when jackey and 123456 can be substituted for operation.Row control can also further be added, for example, add a circulation and cycle counter, with reference to the parameter modification that can be set during operation, batch operation can be completed.

After the completion of script argument, it is possible to run simulator " broadcasting " this section of script by script, realize the actual motion of reptile.Simulator mainly to be completed the work of three types：

A) explain and perform " GET http：// ... Param1=$ { Value1 } | Param2=$ { Value1 } | ... | ParamN=$ { ValueN } { ControlBits } " script; by $ { Value1 }, $ { Value1 }, $ { ValueN } replaces with operation when value, to website send ask and receive response；

B) explain and perform " FIND ... " script：Call html parser, analyzing web page, as the variable of reptile interactive operation next time, or as the final content returning result for needing crawl.

C) Row control：The Row control order included in processing script.

Claims

1. a kind of use conversational mode realizes the network crawler system automatically extracted with website interactive form, and which is special Levy and be：

1) system includes three main composition parts：One html parser；One script recording engine； One script runs simulator.

2) system has three working conditions：One, record script；Two, can manpower intervention script argument； Three, play script and realize reptile function.

2. the use conversational mode as described in right 1 realizes the web crawlers automatically extracted with website interactive form The composition part " html parser " of system, it is characterised in that：Html parser is used for parsing HTML nets Page content, generates a dom tree.

3. the use conversational mode as described in right 1 realizes the web crawlers automatically extracted with website interactive form The composition part " script recording engine " of system, it is characterised in that：There is the WEB of a similar search engine Interface, can be input into station address in input frame；Recording engine can connect website and obtain webpage；User exists The list done on recording engine is submitted to or clickthrough operation, and recording engine records request and the friendship for responding Mutually after data, script source code is translated into.

4. the use conversational mode as described in right 1 realizes the web crawlers automatically extracted with website interactive form The composition part " script operation simulator " of system, it is characterised in that：Can be with described in " broadcasting " right 3 Script source code, complete the function of web crawlers.