KR102275135B1 - Data collection and analyzing system and construction method of the same for project - Google Patents

Abstract

The present invention relates to a method of constructing a data collection and analysis system for a project, and more particularly, to a method of constructing a system for performing data collection and analysis by automatically performing a mechanical operation when a user searches a website for a project on a network. . The present invention provides a data collection and analysis system for a project by generating a mind map by crawling the web, comprising: a mind map generator for generating a mind map according to a crawler setting; and when a web crawling is requested by selecting a specific part of the web site according to a user's operation, outputting a web crawling setting interface, storing web crawling setting information input through the web crawling setting interface, and the web crawling and a web crawling unit that crawls web information according to setting information and displays the crawled web information to the arbitrary node, wherein the web crawling unit generates a mind map using the web crawling, and then sets the crawled web information According to the information, it may be summarized based on the user search term.
According to the present invention having the above-described configuration, there is an advantage in efficient management of data sources, since another person derives a mind map based on the memo process of one of the users when collecting data during the project execution process.

Description

DATA COLLECTION AND ANALYZING SYSTEM AND CONSTRUCTION METHOD OF THE SAME FOR PROJECT

The present invention relates to a method of constructing a data collection and analysis system for a project, and more particularly, to a method of constructing a system for performing data collection and analysis by automatically performing a mechanical operation when a user searches a website for a project on a network. .

With the explosive growth of the Internet environment and the ease of web accessibility of portable mobile devices, individuals collect various information, and the sources and recorded contents of the information can be stored in the form of a list.

However, since these records are stored in the form of a list, it is very inefficient in delivering data if one's own records are checked at once or a large number of users try to share data. In addition, the existing data collection and visualization method consists only of a method in which the user directly inputs, thus inefficient projects are being carried out.

In recent years, rather than an individual project being carried out, a large number of people gather to collect data and share their opinions in the planning stage, and the source of the collected data must be maintained.

In this regard, the prior Korean Patent Application Laid-Open No. 10-2017-0041523 (a method for providing a scraping service, a server device and system therefor) includes the steps of: receiving a scraping request including user personal information from a mobile terminal; using the user's personal information received from the mobile terminal to access a web server having information to be scraped; collecting information to be scraped from the accessed web server; and transmitting the information collected from the web server to the mobile terminal or storing the information in a database associated with the mobile terminal.

However, the prior art discloses only the information collection process of one user, and there is a problem in that it cannot provide a method for suggesting traceability, correlation, and direction of data while the user proceeds with a project.

In addition, when collecting data and trying to produce a single result, there are problems in that it is difficult to see at a glance the collected data report in the form of a document, and it is difficult to continuously analyze each other by approaching each other.

Korean Patent Publication No. 10-2017-0041523

An object of the present invention is to provide a method of constructing a data collection and analysis system that can easily categorize and rearrange data, minimize time and cost loss, and efficiently perform a project.

In order to achieve the above object, the present invention provides a method of constructing a data collection and analysis system for a project using web crawling, (a) a mind map in a system for collecting and analyzing data based on a user's web access a step of requesting the web crawling by generating a random node on the; (b) storing setting information including access information or usage information of the web crawling according to information input in response to the request; and (c) crawling web information on the web according to the setting information, and displaying the crawled web information on the arbitrary node, after generating the mind map, setting the crawled web information An object of the present invention is to provide a method of constructing a data collection and analysis system for a project, characterized in that the user's setting information is stored according to the information.

According to an embodiment, the step (b) includes: (b1) calling a web browser; (b2) outputting a specific web page through the web browser based on the input keyword; (b3) setting a predetermined area of the web page as a selection area according to the setting information; and (b4) generating or storing web crawling setting information including address information of the web page, data extracted from the selection area, and the user's memo.

According to an embodiment, after generating the mind map, (d1) searching for or determining whether there is a request for “open” for the mind map; and (d2) displaying the web information to which the “open” request is input and the user's memo on the arbitrary node.

According to an embodiment, the generation of the mind map may include: (e1) determining whether a specific part containing the crawled web information is clicked in the mind map including the crawled web information; and (e2) when the click is determined, the click information related to the click is converted according to the setting information of the web crawling, and the mouse event information is transmitted to a separate page.

According to an embodiment, in the step (e2), the web information may be displayed as an image or text on a node where the web information is displayed.

According to an embodiment, in step (e2), the web information may be displayed in a separate web browser.

According to an embodiment, it may be implemented as a computer-readable recording medium in which a program is recorded to execute the method of steps (a) to (e).

In addition, the present invention provides a data collection and analysis system for a project by generating a mind map by crawling the web, comprising: a mind map generator for generating a mind map according to a crawler setting; and when a web crawling is requested by selecting a specific part of the web site according to a user's operation, outputting a web crawling setting interface, storing web crawling setting information input through the web crawling setting interface, and the web crawling and a web crawling unit for crawling web information according to setting information and displaying the crawled web information to the arbitrary node, wherein the web crawling unit generates a mind map using the web crawling, and then sets the crawled web information Another object of the present invention is to provide a data collection and analysis system for a project, characterized in that the information is summarized based on user search terms.

According to an embodiment, the web crawling unit may include: a web browser processing module for calling and outputting a web browser displaying a web page of a web address according to the user's request; a user information processing module for providing a user with a setting interface including the importance of the web page, user memo, and image capture, and for storing setting information set through the setting interface; a web crawling setting module for generating or storing web crawling setting information including the address information of the web page and the setting information; and a web crawling processing module that crawls web information according to the setting information and displays the crawled web information on a selected specific node.

According to an embodiment, when a specific part on the mind map is clicked, the web browser processing module converts click information related to the click according to the setting information and transmits the mouse event information to the web server, and the mouse event According to the information, it is possible to receive and output web information transmitted from the web server.

According to an embodiment, the web browser processing module may display the web information as an image or text on a node where the crawled web information is displayed.

According to an embodiment, the web browser processing module may display the web information by calling the web browser.

According to the present invention having the above-described configuration, there is an advantage in efficient management of data sources because, when data is collected in the project execution process, another person derives a mind map based on the memo process of one of the users.

In addition, the present invention has the advantage of being able to provide a service of coordinating opinions and drawing consensus among project parties, presenting a project vision, and delivering a data flow chart as a mind map.

1 shows a flow chart for a method of constructing a data collection and analysis system for the project of the present invention.
2 shows a block diagram of the data collection and analysis system of the present invention.
3 shows a state of an integration process for a user DB inside the DB server shown in FIG. 1 according to an embodiment of the present invention.
4 shows a format of a user DB according to an embodiment of the present invention.
5 shows a result of a visualization module in a data collection and analysis system according to an embodiment of the present invention.
6 shows a multi-visit bar and a minimum-visit bar according to an embodiment of the present invention, and shows guidelines for the thoughts of each user.
7 shows a state in which an edited node is provided based on data collected by each user for a project according to an embodiment of the present invention.
8 shows a state of a related search word module arranged when a key word module is selected according to an embodiment of the present invention.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. . Also, throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being connected "with another configuration in between".

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily carry out the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In an embodiment of the present invention, the term "mind map" refers to information that is visually structured using nodes and branches, and includes all information having the characteristics of a mind map regardless of the display format. is a concept that

In addition, the term "web information" refers to all types of information included in a web page and output to a user terminal, and as an embodiment, specific text information, image information, or It may be video information.

1 shows a flow chart for a method of constructing a data collection and analysis system for the project of the present invention.

Referring to Figure 1, the present invention comprises the steps of (a) generating a random node on a mind map in a system that performs data collection and analysis based on a user's web access and requesting the web crawling; (b) storing setting information including access information or usage information of the web crawling according to information input in response to the request; and (c) crawling web information on the web according to the setting information, and displaying the crawled web information on the arbitrary node, after generating the mind map, setting the crawled web information It provides a method of constructing a data collection and analysis system for a project, characterized in that the user's setting information is stored according to the information.

In the present invention, there are four key functional elements. It is divided into four parts: web crawling, classification, connection, and visualization.

In particular, the step (b) includes: (b1) calling a web browser; (b2) outputting a specific web page through the web browser based on the input keyword; (b3) setting a predetermined area of the web page as a selection area according to the setting information; and (b4) generating or storing web crawling setting information including address information of the web page, data extracted from the selection area, and the user's memo.

In the case of web crawling, according to the embodiment, when a user finds data through a search engine while a web page is open, a separate program operates to collect information on the web page visited by the user, and stores it in a separate database will do In the case of an individual, a separate integrated database is not required. However, if a large number of people find information while proceeding with a project, an integrated database is set up that integrates the information found by many people.

In classification, the information stored by each user is brought to the database and integrated using a separate classification algorithm. When integration is attempted, a separate table created by each user is combined after a separate unique number is assigned to each user. Here, when the integration is carried out, the classification is performed again according to the level of the node of the web page visited by each user, thereby making it easy to create a result with high user visibility. It also accompanies these basic tasks to make the linking part easier.

In connectionization, each user's information collected from web crawling and the data in the state in which classification is completed through the classification are brought, and the corresponding integrated database that has been classified according to the level of each node is connected according to the node. make in the form

In the connection part, a map is first created using a basic integrated database, and then transferred to the visualization part through a separate program. In addition, a separate algorithm can be applied later to re-examine the correlation between data and provide a separate map.

In addition, after generating the mind map, (d1) searching or determining whether there is an “open” request for the mind map; and (d2) displaying the web information to which the “open” request is input and the user's memo on the arbitrary node.

The generation of the mind map may include: (e1) determining whether a specific part containing the crawled web information is clicked in the mind map including the crawled web information; and (e2) when the click is determined, the click information related to the click is converted according to the setting information of the web crawling, and the mouse event information is transmitted to a separate page.

In the visualization part, after crawling the web pages visited and recorded by basically three users, the crawled data are reclassified to create an integrated database, and then the integrated database is connected to create a single map. will be shown in the form.

In the visualization part, data is not statically collected and displayed as a single map, but the shape of the map may vary according to a user's separate setting. Depending on the nature and direction of the project, the final data structure desired by the user looking at the final data is different, so a separate algorithm is applied to reflect this.

In particular, in step (e2), the web information may be displayed as an image or text on a node where the web information is displayed. In step (e2), the web information may be displayed with a separate web browser.

In addition, according to an embodiment, the present invention may be implemented as a computer-readable recording medium in which a program is recorded to execute the method of steps (a) to (e).

Hereinafter, a data collection and analysis system implementing the above-described method will be described.

2 shows a block diagram of the data collection and analysis system of the present invention.

Referring to FIG. 2 , the entire system includes a terminal device (hereinafter referred to as a 'data construction generating device') 100, a network 110, a DB server 120, and a visualization for a data collection and analysis system construction method for a project. A module 130 may be included.

The terminal device 100 according to the present invention is a device that generates a node according to user data using information input according to a user's operation, and opens (executes) a group of generated nodes.

The terminal device 100 is provided with memory means such as a desktop computer as well as a notebook computer, a workstation, a personal digital assistant, a web pad, a mobile communication terminal including a smart phone, and a microprocessor-equipped computing power and network. If it is a digital device having data communication capability through the present invention, it is included in the web server 120 according to the present invention, accesses the Internet 110A, collects information on the web page visited by the user, and stores the information in the DB server 120 .

The terminal device 100 is logically loaded with a tree node and an analysis algorithm in the visualization module 130 , is expressed in the terminal device 100 according to the present invention, and can perform a separate operation.

The function of generating a node by accessing the Internet 110A from the terminal device 100 through the web server 110 and collecting information by the user, that is, the first-level central node and the central node branched once in the form of a tree. It may include primary nodes located on the branch.

In the case of a secondary node located on a branch again branched from the primary nodes in a tree form, it is a node that branches again when passing based on a web page redirected from the primary node, and input information A mind map may be generated by structuring/visualizing .

The terminal device 100 according to an embodiment of the present invention connects to a specific web server 120 according to a user's operation when creating a node, focusing on the web crawling function, and the web page provided by the web server 120 is displayed. Crawl some

The terminal device 100 sets the selected setting area on the target system, crawls the web information set in the corresponding area as images/text, and attaches/attaches to the node (or note attached to the node) selected by the user on the mind map. It performs the display function.

In the data collection and analysis system for a project by generating a mind map by crawling the web, the terminal device 100 includes: a mind map generator for generating a mind map according to a crawler setting; and when a web crawling is requested by selecting a specific part of the web site according to a user's operation, outputting a web crawling setting interface, storing web crawling setting information input through the web crawling setting interface, and the web crawling It may include a web crawling unit that crawls web information according to the setting information and displays the crawled web information to the arbitrary node.

After generating the mind map using the web crawling, the web crawling unit may summarize the crawled web information based on a user search word according to setting information.

The web crawling unit may include: a web browser processing module for calling and outputting a web browser displaying a web page of a web address according to the user's request; a user information processing module for providing a user with a setting interface including the importance of the web page, user memo, and image capture, and for storing setting information set through the setting interface; a web crawling setting module for generating or storing web crawling setting information including the address information of the web page and the setting information; and a web crawling processing module that crawls web information according to the setting information and displays the crawled web information on a selected specific node.

In addition, the more preferable terminal device 100 according to the present invention may mark the crawled web information included in the mind map separately according to the user's function or write a memo, so that the user can click the corresponding node and re-create it. It can be configured so that you know right away when you have access to information.

In addition, there are a user DB 120A and a preprocessing word DB 120B inside the DB server 120. In the case of the user DB 120A, information of web pages visited by each user is included, which will be described in detail in FIG. 3 . it has become

The preprocessing word DB 120B is a separate DB module for processing a word by applying a technology different from the vector value of the word.

According to an embodiment of the present invention, the network is used as a data path between the mind map generating apparatus 100 according to the present invention, the web server 120 and the DB server 130, and the communication mode such as wired and wireless is provided. It can be configured regardless of whether it is a short-range network (PAN), a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). can be configured.

The web browser processing module, when a specific part on the mind map is clicked, converts click information related to the click according to the setting information and transmits mouse event information to the web server,

The web information transmitted from the web server may be received and output according to the mouse event information. The web browser processing module may display the web information as an image or text on a node where the crawled web information is displayed.

The web browser processing module may display the web information by calling the web browser.

The web server 120 may include all general configurations for providing web services to personal information and communication devices accessed through a network. In an embodiment of the present invention, the web server 120 may perform a function of providing a web page including web information to the connected terminal device 100 according to a user's manipulation.

In the case of the terminal device 100 of the present invention, since there are many access paths, it facilitates accessibility by creating a single integrated web server rather than developing a system suitable for each generation device, and It is solved by connecting the existing web server rather than being distributed.

3 shows a state of an integration process for a user DB inside the DB server shown in FIG. 1 according to an embodiment of the present invention.

Referring to FIG. 3 , in the case of a corresponding user DB 120B, each user DB is divided into a user table 200 , and integration into the integrated table 210 may be performed.

Each user has a unique number, and the information of the web page visited by the user includes information such as keywords, node level, URL, and external reference URL. When integration is carried out, the node level is set as the main key. Integration can proceed.

4 shows a format of a user DB according to an embodiment of the present invention.

5 shows a result of a visualization module in a data collection and analysis system according to an embodiment of the present invention.

Referring to FIG. 5 , it is an example result output by the visualization module 130 . In the case of the corresponding main node 400, it is a module indicating the keywords first searched for in the corresponding project. In the case of the user visit count display module 410, when the corresponding users repeatedly visit, the number of visits is expressed as a module and a classification algorithm is applied later to provide an analysis function to the user. In the case of the tag module 420, a module that a separate user thinks is important from among the materials found while visiting can be separately displayed, so that there is no trouble finding it again.

6 shows a multi-visit bar and a minimum-visit bar according to an embodiment of the present invention, and shows guidelines for the thoughts of each user.

Referring to FIG. 6 , through the classification algorithm shown in FIG. 5 , through the multi-visit bar 500 and the minimum-visit bar 510, which data are found by many people and represent the plurality of thoughts, and the minimum-visited bar It provides a guideline to check if there are any unusual thoughts that few have thought through.

7 shows a state in which an edited node is provided based on data collected by each user for a project according to an embodiment of the present invention.

Referring to FIG. 7 , it is a service to be separately provided to the user in order to view the overall flow. In the case of the service, after providing a separate node based on the data collected by the user, it is displayed as the corresponding node by the user's editing.

8 shows a state of a related search word module arranged when a key word module is selected according to an embodiment of the present invention.

Referring to FIG. 8 , the key word module 700 is analyzed using the preprocessed word DB 120B in FIG. 1 through analysis of the data found by users to create a related search word module 710 and the corresponding key word module 700 ) is pressed, the related search word module 710 is set to appear.

All of the above drawings can be edited and analyzed through the terminal device 100 of the configuration diagram, and a separate service is provided by configuring such a system in the past.

Although the present invention has been described in detail through representative embodiments above, those of ordinary skill in the art will understand that various modifications are possible within the limits without departing from the scope of the present invention with respect to the above-described embodiments. will be. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by all changes or modifications derived from the claims and equivalent concepts as well as the claims to be described later.

100: terminal device 110: web server
110A: Internet 120: DB Server
120A: User DB 120B: Preprocessed word DB
130: visualization module 200: user table
210: unified table 400: main node
410: user visit count display module 420: tag module
500: multi-visit bar 510: minimum visit bar
600: project name 610: idea process module
700: key word module 710: related search word module

Claims (12)

In the method of constructing a data collection and analysis system for a project using web crawling,
(a) generating a mind map according to the crawler settings in the mind map generation unit;
(b) a step in which a random node is created on a mind map in a system that performs data collection and analysis based on a user's web access, a specific part in the web site is selected according to the user's operation, and the web crawling is requested ;
(c) storing web crawling setting information including access information or usage information of the web crawling according to information input in response to the request; and
(d) crawling web information on the web according to the setting information, and displaying the crawled web information on the arbitrary node,
After generating the mind map, the method of constructing a data collection and analysis system for a project, characterized in that the crawled web information is summarized based on a user search word according to the setting information.
The method of claim 1,
Step (b) is,
(b1) calling a web browser;
(b2) outputting a specific web page through the web browser based on the input keyword;
(b3) setting a predetermined area of the web page as a selection area according to the web crawling setting information; and
(b4) The method of constructing a data collection and analysis system for a project, comprising the step of generating or storing the web crawling setting information including the address information of the web page, the data extracted from the selection area, and the user's memo.
The method of claim 1,
After generating the mind map,
(d1) retrieving or determining whether there is an “open” request for the mind map; and
(d2) The method of constructing a data collection and analysis system for a project comprising the step of displaying the web information to which the "open" request is input and the user's memo on the arbitrary node.
The method of claim 1,
The generation of the mind map is,
(e1) determining whether a specific part containing the crawled web information is clicked in the mind map including the crawled web information; and
(e2) when it is determined that the click is converted according to the setting information of the web crawling, the method of constructing a data collection and analysis system for a project comprising the step of transmitting mouse event information to a separate page.
5. The method of claim 4,
The step (e2) is,
A method of constructing a data collection and analysis system for a project, characterized in that the web information is displayed as an image or text on a node where the web information is displayed.
5. The method of claim 4,
The step (e2) is,
A method of constructing a data collection and analysis system for a project, characterized in that the web information is displayed in a separate web browser.
A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 6.
In a data collection and analysis system for a project by creating a mind map by web crawling,
a mind map generator that generates a mind map according to the crawler settings; and
When a specific part in the website is selected according to a user's operation and web crawling is requested, a web crawling setting interface is outputted, web crawling setting information input through the web crawling setting interface is stored, and the web crawling setting A web crawling unit that crawls web information according to the information and displays the crawled web information on the arbitrary node,
The web crawling unit,
After generating the mind map using the web crawling, the data collection and analysis system for a project, characterized in that the crawled web information is summarized based on a user search word according to setting information.
9. The method of claim 8,
The web crawling unit,
a web browser processing module for calling and outputting a web browser displaying a web page of a web address according to the user's request;
a user information processing module for providing a user with a setting interface including the importance of the web page, user memo, and image capture, and for storing setting information set through the setting interface;
a web crawling setting module for generating or storing web crawling setting information including the address information of the web page and the setting information; and
A data collection and analysis system for a project comprising a web crawling processing module that crawls web information according to the setting information and displays the crawled web information on a selected specific node.
10. The method of claim 9,
The web browser processing module,
When a specific part on the mind map is clicked, it converts according to the setting information and transmits mouse event information to the web server,
Data collection and analysis system for a project, characterized in that receiving and outputting web information transmitted from the web server according to the mouse event information.
11. The method of claim 10,
The web browser processing module,
Data collection and analysis system for a project, characterized in that the web information is displayed as an image or text on a node where the crawled web information is displayed.
11. The method of claim 10,
The web browser processing module,
Data collection and analysis system for a project, characterized in that the web information is displayed by calling the web browser.

Images