CN110096666A - The method and device of data processing - Google Patents
The method and device of data processing Download PDFInfo
- Publication number
- CN110096666A CN110096666A CN201910382348.4A CN201910382348A CN110096666A CN 110096666 A CN110096666 A CN 110096666A CN 201910382348 A CN201910382348 A CN 201910382348A CN 110096666 A CN110096666 A CN 110096666A
- Authority
- CN
- China
- Prior art keywords
- data
- relational database
- source
- crawl
- source website
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000012545 processing Methods 0.000 title claims abstract description 38
- 230000004044 response Effects 0.000 claims abstract description 42
- 230000002159 abnormal effect Effects 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 5
- 230000010354 integration Effects 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004146 energy storage Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000446 fuel Substances 0.000 description 2
- 238000010248 power generation Methods 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 229910052799 carbon Inorganic materials 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000004065 wastewater treatment Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
- G06F16/9558—Details of hyperlinks; Management of linked annotations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/986—Document structures and storage, e.g. HTML extensions
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The present invention provides a kind of method and devices of data processing, are related to the technical field of electronic information, can obtain pre-stored source list of websites, wherein source list of websites includes the link information of multiple source websites；Request of data is sent to the corresponding source website of link information according to the pre-set period；If receiving the response message of source website return, pre-set field information is extracted；Crawl and the matched data of field information from the source website of returning response information；The data of crawl are stored to non-relational database, when user needs data, is directly inquired to obtain query result from non-relational database, improves the response time of server, and the user experience is improved.
Description
Technical field
The present invention relates to electronic information technical fields, more particularly, to a kind of method and device of data processing.
Background technique
Internet includes various knowledge, including music, books, animation, TV play, animation, open class, and speech etc. is various
Mass data, it is many kinds of, it is abundant in content.With the rapid development of Internet technology, network data is also being skyrocketed through.
Conventionally, as in internet the type of data and scale it is huge, for traditional business processing side
Formula, in particular for the business for handling a large amount of parallel datas, it is often necessary to the downloading data from internet, then retrieved from memory
Then process instruction carries out Data Management Analysis in calculating machine.Since the request number of times for sending internet is more, service will lead to
The response time of device is too long, and this business processing mode needs to be connected in real time with network data, if network environment is unstable,
Reduce the experience of user.
Summary of the invention
In view of this, the object of the present invention is to provide a kind of method and device of data processing, to alleviate above-mentioned skill
Art problem.
In a first aspect, the embodiment of the invention provides a kind of methods of data processing, wherein this method comprises: obtaining pre-
The source list of websites first stored, wherein source list of websites includes the link information of multiple source websites；According to presetting
Period send request of data to the corresponding source website of link information；If receiving the response message of source website return,
Extract pre-set field information；Crawl and the matched data of field information from the source website of returning response information；It will
The data of crawl are stored to non-relational database.
With reference to first aspect, the embodiment of the invention provides the first possible embodiments of first aspect, wherein presses
After sending request of data to the corresponding source website of link information according to the pre-set period, the above method further include: judgement
The response message of source website return whether is received in first time threshold；If not, sending number to source website again
According to request；If not receiving the response message of source website return in second time threshold, will interrupt and source website
Communication request, wherein second time threshold is greater than first time threshold.
With reference to first aspect, the embodiment of the invention provides second of possible embodiments of first aspect, wherein will
It includes: to judge whether non-relational database communicates normally that the data of crawl, which were stored to the step of non-relational database,；If
It is to store the data of crawl to non-relational database；If not, generating the abnormal log of non-relational database, save
The exception information of non-relational database.
The possible embodiment of second with reference to first aspect, the embodiment of the invention provides the third of first aspect
Possible embodiment, wherein by the data of crawl store to the step of non-relational database include: by the data of crawl with
Data in non-relational database are matched, if not being matched to identical data, the data of crawl are stored to non-pass
It is type database.
The third possible embodiment with reference to first aspect, the embodiment of the invention provides the 4th kind of first aspect
Possible embodiment, this method further include: the data saved in non-relational database are subjected to classification processing, to data
It is integrated；The data volume that each classification includes data is counted, the data volume of classification and each classification after integration is carried out
Storage.
The 4th kind of possible embodiment with reference to first aspect, the embodiment of the invention provides the 5th kind of first aspect
Possible embodiment, the above method further include: the data volume of each classification and each classification after classification processing is carried out
The page shows, the content that the page is shown include at least the icon of each classification is shown, title is shown and quantity is shown.
The 5th kind of possible embodiment with reference to first aspect, the embodiment of the invention provides the 6th kind of first aspect
Possible embodiment, the above method further include: provide data search control in the page；When monitoring that legitimate user searches in data
When rope control inputs keyword, keyword is shown in the data of non-relational database search and keyword match, and in the page
Search result.
Second aspect, the embodiment of the present invention also provide a kind of device of data processing, comprising: module are obtained, for obtaining
Pre-stored source list of websites, wherein source list of websites includes the link information of multiple source websites；Sending module,
For sending request of data to the corresponding source website of link information according to the pre-set period；Extraction module, if for
The response message for receiving the return of source website, extracts pre-set field information；Handling module, for believing from returning response
Crawl and the matched data of field information in the source website of breath；Memory module, for storing the data of crawl to non-relationship
Type database.
In conjunction with second aspect, the embodiment of the invention provides the first possible embodiments of second aspect, wherein should
Device further include: first judgment module, for sending number to the corresponding source website of link information according to the pre-set period
After request, the response message that the return of source website whether is received in first time threshold is judged；Module is retransmitted,
When judging result for first judgment module is no, request of data is sent to source website again；Setup module, if for
The response message for not receiving the return of source website in second time threshold, sets invalid for the link information of source website
Information, wherein second time threshold is greater than first time threshold.
In conjunction with second aspect, the embodiment of the invention provides second of possible embodiments of second aspect, wherein should
Memory module is also used to: judging whether non-relational database communicates normally；If so, storing the data of crawl to non-relationship
Type database；If not, generating the abnormal log of non-relational database, the exception information of non-relational database is saved.
The embodiment of the present invention bring it is following the utility model has the advantages that
A kind of method and device of data processing provided in an embodiment of the present invention can obtain pre-stored source website
List, wherein source list of websites includes the link information of multiple source websites；According to the pre-set period to link information
Corresponding source website sends request of data；If receiving the response message of source website return, pre-set word is extracted
Segment information；Crawl and the matched data of field information from the source website of returning response information；By the data of crawl store to
Non-relational database is directly inquired to obtain query result from non-relational database, be mentioned when user needs data
The high response time of server, and the user experience is improved.
Other features and advantages of the present invention will illustrate in the following description, also, partly become from specification
It obtains it is clear that understand through the implementation of the invention.The objectives and other advantages of the invention are in specification and attached drawing
Specifically noted structure is achieved and obtained.
To enable the above objects, features and advantages of the present invention to be clearer and more comprehensible, preferred embodiment is cited below particularly, and cooperate
Appended attached drawing, is described in detail below.
Detailed description of the invention
It, below will be to specific in order to illustrate more clearly of the specific embodiment of the invention or technical solution in the prior art
Embodiment or attached drawing needed to be used in the description of the prior art be briefly described, it should be apparent that, it is described below
Attached drawing is some embodiments of the present invention, for those skilled in the art, without creative efforts,
It is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow chart of the method for data processing provided in an embodiment of the present invention；
Fig. 2 is the flow chart of the method for another data processing provided in an embodiment of the present invention；
Fig. 3 is a kind of structural schematic diagram of the device of data processing provided in an embodiment of the present invention；
Fig. 4 is the structural schematic diagram of the device of another data processing provided in an embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with attached drawing to the present invention
Technical solution be clearly and completely described, it is clear that described embodiments are some of the embodiments of the present invention, rather than
Whole embodiments.Based on the embodiments of the present invention, those skilled in the art institute without making creative work
The every other embodiment obtained, shall fall within the protection scope of the present invention.
Currently, the method that traditional concurrent service is handled, needs to send repeatedly request to internet and is just able to achieve to business
Processing, the response time that this will lead to server is too long, and need be connected in real time with network data, if network environment is unstable
It is fixed, reduce the experience of user.Based on this, a kind of method and device of data processing provided in an embodiment of the present invention be may be implemented
When user needs data, it can directly be inquired to obtain query result from non-relational database, clothes have been effectively relieved
The technical issues of business device response time is long, poor user experience.
For the method convenient for understanding the present embodiment, first to a kind of data processing disclosed in the embodiment of the present invention
It describes in detail,
Embodiment one:
The embodiment of the invention provides a kind of method of data processing, a kind of method of data processing as shown in Figure 1
Flow chart, method includes the following steps:
Step S102 obtains pre-stored source list of websites, wherein source list of websites includes multiple source websites
Link information.
When specific implementation, the relatively good source website for meeting business need of quality, example can be picked out according to business demand
Such as, business demand is to seek energy-related company and product, can pick out the source of satisfactory company and product
Website, and the source website that will be singled out is stored in acquisition list.
Step S104 sends request of data to the corresponding source website of link information according to the pre-set period.
In general, the timer when starting server in meeting loading configuration file carries out timing, when timer timing reaches
When the preset time, by the link information of the source website in list and it is sent to server.
Step S106 extracts pre-set field information if receiving the response message of source website return.
Step S108, crawl and the matched data of field information from the source website of returning response information.
Step S110 stores the data of crawl to non-relational database.
Specifically, if responded to the link information that server is sent, it will return to the corresponding source of link information
The full content data of website grab in the full content data of the source website of response according to preset field information
The data to match with field information, and the data grabbed are stored into non-relational database.Wherein, field information can
To be set according to user demand, for example, field information includes the letters such as power generation information, energy storage information and fuel information provision
Breath.
A kind of method of data processing provided in an embodiment of the present invention can obtain pre-stored source list of websites,
Wherein, source list of websites includes the link information of multiple source websites；It is corresponding to link information according to the pre-set period
Source website send request of data；If receiving the response message of source website return, pre-set field letter is extracted
Breath；Crawl and the matched data of field information from the source website of returning response information；The data of crawl are stored to non-pass
It is type database, when user needs data, is directly inquired to obtain query result from non-relational database, be improved
The response time of server, and the user experience is improved.
When specific implementation, according to the pre-set period to the corresponding source website of link information send request of data it
Afterwards, also judge the response message that the return of source website whether is received in first time threshold；If not, again to source net
It stands and sends request of data；It, will interruption and source if not receiving the response message of source website return in second time threshold
The communication request of head website, wherein second time threshold is greater than first time threshold.
Specifically, above-mentioned judgment method is that opposite server sends request and the process of response is monitored, setting first
Threshold time is 300s, and the second threshold time is 400s, if 300s server be not received by transmission request or
Do not obtain response returning response information, then again to server send request communicated, if 400s server also
It is not received by the request of transmission or does not obtain response returning response information, then disconnect current link information and sent out to server
Send request, behind can also continue to server send current link information.Meanwhile exception information is generated, it is stored in log text
In part, convenient for accurately inquiry exception information.
In general, storing the data of crawl to before non-relational database, also judge whether non-relational database leads to
Letter is normal；If so, storing the data of crawl to non-relational database；If not, generating the different of non-relational database
Chang Zhi saves the exception information of non-relational database.
Specifically, the time for presetting connection non-relational database, if non-relational number within the set time
Data can be received according to library, then shows to communicate between non-relational database and server normally, the data grabbed is deposited
Storage, if non-relational database does not receive data within the set time, shows non-pass into non-relational database
It is communication abnormality between type database and server, at this moment, generates the exception information of non-relational database, and save to log
In file.
Further, the data that will be grabbed are needed to carry out duplicate removal processing, in the data and non-relational database of crawl
Data are matched；The data of crawl are matched with the data in non-relational database, if not being matched to identical number
According to then the data of crawl are stored to non-relational database.
When specific implementation, the data grabbed are cleaned, delete unwanted field, fill missing values and transformation
After the processing such as data format, the data grabbed are matched with the data in database, if be matched in database
Data, then the data that will match to are without storage, if being not matched to the data in database, the number that will be matched to
According to being stored into database, so as to subsequent arithmetic, data analysis and use.
Further, the process of the method based on above-mentioned data processing, Fig. 2 shows the streams of the method for another data processing
Cheng Tu, wherein the process of step S202 to step S210 can be with reference to step S102 in above-mentioned Fig. 1 to the corresponding mistake of step S110
Journey, details are not described herein.As shown in Fig. 2, this method is further comprising the steps of:
The data saved in non-relational database are carried out classification processing, to integrate to data by step S212.
Step S214 counts the data volume for the data that each classification includes, by the classification and each classification after integration
Data volume is stored.
In actual use, according to customer service demand and field information, non-relational database will can be stored in
In data be divided into fuel supply, hot/cold, energy storage, with can and efficiency, traffic, water resource or wastewater treatment, power generation, electric energy it is defeated
Match, carbon, energy market and transaction, derived energy chemical, resource circulation utilization and environmental protection industry and other service this 13 classifications, it is whole
The data after classification are closed, by the data volume of data included in 13 classifications of counters count, by 13 after integration
The data volume of classification and each classification is stored into new non-relational database for users to use.
Step S216 carries out the page to the data volume of each classification and each classification after classification processing and shows, the page
The content of display include at least the icon of each classification is shown, title is shown and quantity is shown.
In actual use, for intuitive and convenient for users to use, it can be carried out to sorted Various types of data is grabbed
Page presentation is able to use the cheer and bright quantity for recognizing the specific field name of above-mentioned 13 classifications and statistics in family.
Further, data search control is provided in the above-mentioned page；When monitor legitimate user data search control input
When keyword, the search result of keyword is shown in the data of non-relational database search and keyword match, and in the page.
When specific implementation, user, which needs to carry out Login Register, first just can enter page display interface later by verifying,
User can input keyword in the search column at interface, click search option and start to search for, at this point, server is first defeated according to user
The keyword entered is searched in new non-relational database and the data of keyword match, when the number of the related data searched out
Amount stops search when reaching pre-set number of searches, at this point, can be shown on page display interface it is searching as a result, with
Family can click search result and carry out checking reading.
Embodiment two:
On the basis of the above embodiments, the embodiment of the invention also provides a kind of devices of data processing, as shown in Figure 3
A kind of data processing device structural schematic diagram, which includes:
Module 302 is obtained, for obtaining pre-stored source list of websites, wherein source list of websites includes multiple
The link information of source website；
Sending module 304 is asked for sending data to the corresponding source website of link information according to the pre-set period
It asks；
Extraction module 306 extracts pre-set field letter if the response message for receiving the return of source website
Breath；
Handling module 308, for the crawl from the source website of returning response information and the matched data of field information；
Memory module 310, for storing the data of crawl to non-relational database.
Further, on the basis of Fig. 3, Fig. 4 shows the structural schematic diagram of the device of another data processing, the device
Further include:
First judgment module 402, for sending number to the corresponding source website of link information according to the pre-set period
After request, the response message that the return of source website whether is received in first time threshold is judged；
Module 404 is retransmitted, when the judging result for first judgment module is no, sends number to source website again
According to request；
Setup module 406 will if the response message for not receiving the return of source website in second time threshold
The link information of source website is set as invalid information, wherein second time threshold is greater than first time threshold.
Specifically, memory module is also used to:
Judge whether non-relational database communicates normally；If so, storing the data of crawl to non-relational data
Library；If not, generating the abnormal log of non-relational database, the exception information of non-relational database is saved.
The device of data processing provided in an embodiment of the present invention has with the method for data processing provided by the above embodiment
Identical technical characteristic reaches identical technical effect so also can solve identical technical problem.
The computer program product of the method and device of data processing provided by the embodiment of the present invention, including store journey
The computer readable storage medium of sequence code, the instruction that said program code includes can be used for executing institute in previous methods embodiment
The method stated, specific implementation can be found in embodiment of the method, and details are not described herein.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description
It with the specific work process of device, can refer to corresponding processes in the foregoing method embodiment, details are not described herein.
It, can be with if the function is realized in the form of SFU software functional unit and when sold or used as an independent product
It is stored in a computer readable storage medium.Based on this understanding, technical solution of the present invention is substantially in other words
The part of the part that contributes to existing technology or the technical solution can be embodied in the form of software products, the meter
Calculation machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be a
People's computer, server or network equipment etc.) it performs all or part of the steps of the method described in the various embodiments of the present invention.
And storage medium above-mentioned includes: that USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), arbitrary access are deposited
The various media that can store program code such as reservoir (RAM, Random Access Memory), magnetic or disk.
In the description of the present invention, it should be noted that term " center ", "upper", "lower", "left", "right", "vertical",
The orientation or positional relationship of the instructions such as "horizontal", "inner", "outside" be based on the orientation or positional relationship shown in the drawings, merely to
Convenient for description the present invention and simplify description, rather than the device or element of indication or suggestion meaning must have a particular orientation,
It is constructed and operated in a specific orientation, therefore is not considered as limiting the invention.In addition, term " first ", " second ",
" third " is used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance.
Finally, it should be noted that above embodiments, only a specific embodiment of the invention, to illustrate skill of the invention
Art scheme, rather than its limitations, scope of protection of the present invention is not limited thereto, although with reference to the foregoing embodiments to the present invention into
Go detailed description, it should be understood by those skilled in the art that: anyone skilled in the art takes off in the present invention
In the technical scope of dew, it can still modify to technical solution documented by previous embodiment or can readily occur in change
Change or equivalent replacement of some of the technical features；And these modifications, variation or replacement, do not make relevant art
Scheme essence be detached from technical solution of the embodiment of the present invention spirit and scope, should all cover protection scope of the present invention it
It is interior.Therefore, protection scope of the present invention should be subject to the protection scope in claims.
Claims (10)
1. a kind of method of data processing, which is characterized in that the described method includes:
Obtain pre-stored source list of websites, wherein the source list of websites includes the link letter of multiple source websites
Breath；
Request of data is sent to the corresponding source website of the link information according to the pre-set period；
If receiving the response message that the source website returns, pre-set field information is extracted；
Crawl and the matched data of the field information from the source website for returning to the response message；
The data of crawl are stored to non-relational database.
2. method according to claim 1, which is characterized in that it is described according to the pre-set period to the link information pair
After the source website answered sends request of data, the method also includes:
Judge the response message that the source website returns whether is received in first time threshold；
If not, sending request of data to the source website again；
If not receiving the response message that the source website returns in second time threshold, will interrupt and the source net
The communication request stood, wherein the second time threshold is greater than the first time threshold.
3. method according to claim 1, which is characterized in that described to store the data of crawl to non-relational data
The step of library includes:
Judge whether the non-relational database communicates normally；
If so, storing the data of crawl to non-relational database；
If not, generating the abnormal log of the non-relational database, the exception information of the non-relational database is saved.
4. method according to claim 3, which is characterized in that described to store the data of crawl to non-relational data
The step of library includes:
The data of crawl are matched with the data in the non-relational database, if not being matched to identical number
According to then the data of crawl are stored to the non-relational database.
5. method according to claim 4, which is characterized in that the method also includes:
The data saved in the non-relational database are subjected to classification processing, to integrate to the data；
The data volume for counting the data that each classification includes, by the classification and each classification after integration
Data volume is stored.
6. method according to claim 5, which is characterized in that the method also includes:
It carries out the page to the data volume of classification described each of after classification processing and each classification to show, the page
The content of display is included at least to the icon of each classification is shown, title is shown and quantity is shown.
7. method according to claim 6, which is characterized in that the method also includes:
Data search control is provided in the page；
When monitor legitimate user the data search control input keyword when, the non-relational database search with
The data of the keyword match, and the search result of the keyword is shown in the page.
8. a kind of device of data processing, which is characterized in that described device includes:
Module is obtained, for obtaining pre-stored source list of websites, wherein the source list of websites includes multiple sources
The link information of website；
Sending module is asked for sending data to the corresponding source website of the link information according to the pre-set period
It asks；
Extraction module, if the response message returned for receiving the source website, extracts pre-set field information；
Handling module, for the crawl from the source website for returning to the response message and the matched number of the field information
According to；
Memory module, for storing the data of crawl to non-relational database.
9. device according to claim 8, which is characterized in that described device further include:
First judgment module, for sending number to the corresponding source website of the link information according to the pre-set period
After request, judge the response message that the source website returns whether is received in first time threshold；
Module is retransmitted, when the judging result for the first judgment module is no, is sent again to the source website
Request of data；
Setup module, if the response message returned for not receiving the source website in second time threshold, will in
The disconnected communication request with the source website, wherein the second time threshold is greater than the first time threshold.
10. device according to claim 8, which is characterized in that the memory module is also used to:
Judge whether the non-relational database communicates normally；
If so, storing the data of crawl to non-relational database；
If not, generating the abnormal log of the non-relational database, the exception information of the non-relational database is saved.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910382348.4A CN110096666A (en) | 2019-05-08 | 2019-05-08 | The method and device of data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910382348.4A CN110096666A (en) | 2019-05-08 | 2019-05-08 | The method and device of data processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110096666A true CN110096666A (en) | 2019-08-06 |
Family
ID=67447480
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910382348.4A Pending CN110096666A (en) | 2019-05-08 | 2019-05-08 | The method and device of data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110096666A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434205A (en) * | 2020-11-30 | 2021-03-02 | 北京秒针人工智能科技有限公司 | Data integration capturing method and system based on data site and computer equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101551813A (en) * | 2009-05-13 | 2009-10-07 | 腾讯科技（深圳）有限公司 | Network connection apparatus, search equipment and method for collecting search engine data source |
| CN104166729A (en) * | 2014-08-28 | 2014-11-26 | 四川长虹电器股份有限公司 | Timing multi-task webpage data capturing system and method |
| US8983931B2 (en) * | 2011-11-29 | 2015-03-17 | Sybase, Inc. | Index-based evaluation of path-based queries |
| CN106776787A (en) * | 2016-11-24 | 2017-05-31 | 山东浪潮云服务信息科技有限公司 | A kind of method being acquired to internet data |
| CN107885777A (en) * | 2017-10-11 | 2018-04-06 | 北京智慧星光信息技术有限公司 | A kind of control method and system of the crawl web data based on collaborative reptile |
| CN108021604A (en) * | 2017-10-24 | 2018-05-11 | 山东科技大学 | A kind of web crawlers method for crawling barrage in Dou Yu webcast websites main broadcaster room |
| CN109657120A (en) * | 2018-11-26 | 2019-04-19 | 河南大瑞物联网科技有限公司 | A kind of internet data acquisition method that matching degree is high |
-
2019
- 2019-05-08 CN CN201910382348.4A patent/CN110096666A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101551813A (en) * | 2009-05-13 | 2009-10-07 | 腾讯科技（深圳）有限公司 | Network connection apparatus, search equipment and method for collecting search engine data source |
| US8983931B2 (en) * | 2011-11-29 | 2015-03-17 | Sybase, Inc. | Index-based evaluation of path-based queries |
| CN104166729A (en) * | 2014-08-28 | 2014-11-26 | 四川长虹电器股份有限公司 | Timing multi-task webpage data capturing system and method |
| CN106776787A (en) * | 2016-11-24 | 2017-05-31 | 山东浪潮云服务信息科技有限公司 | A kind of method being acquired to internet data |
| CN107885777A (en) * | 2017-10-11 | 2018-04-06 | 北京智慧星光信息技术有限公司 | A kind of control method and system of the crawl web data based on collaborative reptile |
| CN108021604A (en) * | 2017-10-24 | 2018-05-11 | 山东科技大学 | A kind of web crawlers method for crawling barrage in Dou Yu webcast websites main broadcaster room |
| CN109657120A (en) * | 2018-11-26 | 2019-04-19 | 河南大瑞物联网科技有限公司 | A kind of internet data acquisition method that matching degree is high |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434205A (en) * | 2020-11-30 | 2021-03-02 | 北京秒针人工智能科技有限公司 | Data integration capturing method and system based on data site and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108776671A (en) | A kind of network public sentiment monitoring system and method | |
| CN113051147A (en) | Database cluster monitoring method, device, system and equipment | |
| CN107104840A (en) | A kind of daily record monitoring method, apparatus and system | |
| CN109120429B (en) | Risk identification method and system | |
| CN106815125A (en) | A kind of log audit method and platform | |
| CN105511727B (en) | A kind of message treatment method and device | |
| CN104750795A (en) | Intelligent semantic searching system and method | |
| CN103875015A (en) | Multi-factor identity fingerprinting with user behavior | |
| CN108280115A (en) | Identify the method and device of customer relationship | |
| CN110321352A (en) | Production line monitoring method and device, electronic equipment and readable storage medium | |
| CN104462314A (en) | Power grid data processing method and device | |
| CN105205686A (en) | Method and system for obtaining product price information | |
| CN102147816A (en) | System for counting cases and analyzing tendency | |
| CN110458296A (en) | The labeling method and device of object event, storage medium and electronic device | |
| CN104077293A (en) | Webpage acquisition method and device | |
| CN105260365B (en) | The treating method and apparatus of end message | |
| CN104951553B (en) | A kind of accurate content of data processing is collected and data mining platform and its implementation | |
| CN114090380A (en) | Terminal monitoring method, device, equipment and storage medium | |
| KR20210063879A (en) | Computer program and recording medium for providing chatbot services of analyzing marketing information | |
| CN113111261A (en) | Data processing method of cloud platform, cloud platform and panoramic analysis system | |
| CN116756330A (en) | Knowledge graph construction method and device, electronic equipment and storage medium | |
| KR20220074574A (en) | A method and an apparatus for analyzing real-time chat content of live stream | |
| CN110096666A (en) | The method and device of data processing | |
| CN118885971A (en) | A heterogeneous data fusion method, device, equipment and storage medium | |
| CN106533728A (en) | Server information collecting method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190806 |