CN110471917A - It is a kind of based on historical data excavate customs declaration list intelligently make a report on method - Google Patents

Abstract

The invention provides a method for intelligently filling in a customs declaration form based on historical data mining. The method includes: preprocessing the historical data of the customs declaration form, merging the header and body data, and removing irrelevant fields; designing and implementing a correlation analysis algorithm for each field based on specific values; designing a tree structure, based on each field Correlation builds a dynamic tree and stores it; based on the generated dynamic tree, intelligently recommends the content to be filled in; regularly and automatically performs data maintenance and update based on newly entered data. The correlation analysis algorithm and the dynamic tree designed by the present invention can better dynamically perform intelligent filling of other filling fields according to the current input content of the user, and the accuracy rate is high. With strong generalization and self-learning ability, it can greatly improve the efficiency of customs declaration, and save manpower and material resources for customs declaration agencies and customs declaration enterprises.

Description

An Intelligent Filling Method of Customs Declaration Form Based on Historical Data Mining

Technical field:

The invention relates to the field of customs declaration information filling, in particular to a method for intelligently filling customs declaration forms based on historical data mining.

Background technique:

The Customs of the People's Republic of my country is the entry and exit supervision and management agency of our country, and the declaration management of import and export commodities is an important and basic task in its daily business. At present, the import and export commodity declaration mainly requires the declaration company to sort out all relevant paper documents and enter them into the relevant customs system. Due to the large number of input fields and the low connection between the contents, the input efficiency is low, the error rate is high, and manpower and material resources are wasted.

Invention content:

The invention proposes a method for intelligently filling in customs declaration forms based on historical data mining. The present invention fully considers the correlation between each field in the reporting information based on specific values, provides an intelligent reporting strategy, and has the characteristics of higher efficiency and higher accuracy than traditional complete manual input.

The present invention provides the following scheme, a method for intelligently filling customs declaration forms based on historical data mining, comprising the following steps:

S1: Preprocess the historical data of the customs declaration form, merge the header and body data, and remove irrelevant fields.

S1.1: Use Spark distributed computing to merge data

The real customs declaration data is divided into header data and body data, which are stored in two data tables respectively. The header data describes a certain order information, such as import and export ports, declaration units, transaction methods, etc.; the body data describes a certain order information. The specific information of the product in the order, such as product number, product name, declared unit price, etc. The present invention uses the Spark distributed computing method to connect the two tables through the order number as the primary key to obtain a data table containing all information fields to be filled in, and store it in the Hive database.

S1.2: Count the null values of each field and remove irrelevant fields.

In the process of customs declaration, some fields belong to the items that users choose to fill in, so there will be cases where the fields are empty. In the preprocessing process, the ratio of the number of empty value data items in a certain field to the total number of data items is counted. If it is greater than 90%, then Remove this field. At the same time, fields without actual recommendation value such as time, serial number, and manual number are removed.

Through the above preprocessing, a field data table with recommended value is obtained.

S2: Design and implement a correlation analysis algorithm for each field based on specific values.

The traditional correlation discovery algorithm only relies on the internal relationship between the fields themselves to determine the correlation between certain fields. The present invention combines real customs declaration data with fields, and judges the correlation between the field and other fields when the value of a certain field is determined, thereby simulating the continuous interaction process with users in the actual entry scene.

The definition of correlation is, given field A and its value a, when the value of a certain field B is determined, so that the values of fields other than A and B that need to be entered are unique or the options are the fewest, it is called the relationship between field A and field B. The greatest correlation among them.

The input of the algorithm is the historical data set, user input field A and field value a, and the output is the most relevant field B under the current input situation. The algorithm is implemented as follows:

5. Segment the historical data set according to the input field A and field value a to obtain the sub-data set under the specific input situation;

6. Deduplicate the sub-dataset;

7. Calculate the correlation between the remaining fields that need to be entered and field A, and select field B that has the greatest correlation with field A

8. Output field B.

S3: Design a tree structure, construct a dynamic tree based on the correlation of each field and store it.

The present invention implements the intelligent customs declaration filling method based on historical data mining. After discovering the value-based field correlation relationship through S2, a tree structure is designed to build a tree with the shortest historical filling path.

S3.1: Tree structure design

A tree structure includes nodes and edges. The nodes are divided into division nodes (non-leaf nodes) and recommendation nodes (leaf nodes). Divide the nodes into the attribute name of a field and the attribute value with the highest frequency of occurrence in this field. The recommended node is a Map structure, which stores the field name and the corresponding attribute value; each node is connected by an edge, and the corresponding value of the parent node field is stored at the same time. attribute value.

S3.2: Dynamic tree generation

First read the field data table preprocessed by S1, define the field of the first-level division node (root node) as the declaration line, and define the second and third-level division nodes as the business unit and commodity name respectively. Nodes are connected through the values of the corresponding fields of the parent nodes.

The nodes of the fourth layer and subsequent layers are selected according to the corresponding attribute values of the upper layer nodes and edges as input, and the field with the highest correlation is selected as the lower layer node until the value of other input fields after a certain node is unique or all fields are entered. Generate recommended nodes (leaf nodes), store other input fields and field values or generate empty nodes.

Through the above steps, a dynamic tree with the shortest reporting path in history is generated.

S3.3: Dynamic Tree Storage

Store the dynamic tree generated by S3.2 in the MySql database, and the table structure is shown in the following table.

Table 1 Divide node table structure

field name Field Type illustrate id int self-growth, primary key Field_name String Fill in the field name Best_value String Most frequent field value name String Node name, unique Level int tree structure level Agent_code String Corresponding reporting line name date date The record was inserted at

Table 2 Leaf node table structure

field name Field Type illustrate id int self-growth, primary key value String Field name + field value splicing Level int tree structure level Agent_code String Corresponding reporting line name date date The record was inserted at

Table 3 Edge table structure

field name Field Type illustrate id int self-growth, primary key Father String Parent node field name Son String Child node field name value String A certain value of the parent node Agent_code String Corresponding reporting line name date date The record was inserted at

S4: Based on the generated dynamic tree, intelligently recommend filling content.

Read each structure of the dynamic tree stored in the MySql database into the memory, and re-build the corresponding dynamic tree in the memory. First, search the dynamic tree according to the declaration line, business unit and product name entered by the user, then perform a depth-first search according to the BestValue attribute stored in each node, and output all other fields and recommended filling values in an orderly manner; then interact with the user and perform partial Field modification or input, and finally perform the above search and recommendation again based on the new input value until it meets the input expectations.

S5: Regularly and automatically perform data maintenance and update based on newly entered data.

First, store each entry of the user in the database and add a time tag, then add new data to the historical data set at a fixed time every week according to the time tag, and construct a new dynamic tree according to the newly generated historical data set and store it, so as to achieve Data is automatically maintained and updated.

The present invention has the following technical effects:

1. The present invention can better dynamically perform intelligent filling of other filling fields according to the user's current input content, and the accuracy rate is high;

2. With strong generalization and self-learning ability, it can greatly improve the efficiency of customs declaration, and save manpower and material resources for customs declaration agencies and customs declaration enterprises;

3. The present invention has a regular automatic update function.

Description of drawings:

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort.

Fig. 1 is an implementation flow chart of an intelligent customs declaration form filling method based on historical data mining provided by an embodiment of the present invention.

FIG. 2 is a dynamic tree structure diagram of a method for intelligently filling customs declaration forms based on historical data mining provided by an embodiment of the present invention.

Detailed ways:

The invention provides an intelligent customs declaration form filling method based on the mining of customs declaration history data. The overall plan includes:

Preprocess the historical data of the customs declaration form, merge the header and body data, and remove irrelevant fields; design and implement a correlation analysis algorithm for each field based on specific values; design a tree structure, and build dynamics based on the correlation of each field Tree and store; based on the generated dynamic tree, intelligently recommend filling content; regularly and automatically perform data maintenance and update based on newly entered data. Fig. 1 is the implementation flowchart of the present invention.

The correlation analysis algorithm and the dynamic tree designed by the present invention can better fill in the rest of the filling fields according to the user's input intelligence and dynamics, and the accuracy rate is high. With strong generalization and self-learning ability, it can greatly improve the efficiency of customs declaration, and save manpower and material resources for customs declaration agencies and customs declaration enterprises.

To illustrate the embodiment of the present invention:

Take the real data set provided by a domestic customs port as an example, which includes all the declared data of the port in 2015 and 2016. In 2015, there were 14,423,930 records in the header data table, and 36,578,318 records in the body data table; in 2016, there were 14,832,866 records in the header data table, and 38,673,224 records in the body data table. The data involved more than 7,000 reporting banks. Among them, the header data has a total of 43 fields, which describe an order information, such as the import and export port, declaration unit, transaction method, etc., and the table body data has a total of 15 fields, which describe the specific information of many commodities in an order, such as commodity number, commodity Name, declared unit price, etc.

Step S1: Preprocessing the historical data of the customs declaration form, merging the header and body data, and removing irrelevant fields.

S1.1: Use Spark distributed computing to merge data

Using the Spark distributed computing method, the order number is used as the primary key to connect the two tables to obtain a data table containing all the information fields to be filled in, and store it in the Hive database.

S1.2: Count the null values of each field and remove irrelevant fields.

Count the fields with a null value ratio greater than 90% and remove them, and remove the time, serial number, manual number, etc. to obtain a field data table with recommended value.

Step S2: Design and implement a correlation analysis algorithm for each field based on specific values.

Combining real customs declaration data with fields, judging the correlation between this field and other fields when the value of a certain field has been determined, so as to simulate the continuous interaction process with users in the actual entry scene.

Implement the field correlation analysis algorithm, design a program to input a certain field name A, field value a and the data set under the current input, and output the field B with the highest correlation under the specific input situation.

Step S3: Design a tree structure, construct a dynamic tree based on the correlation of each field and store it.

S3.1: Tree structure design

A tree structure includes nodes and edges. The nodes are divided into division nodes (non-leaf nodes) and recommendation nodes (leaf nodes). Divide the nodes into the attribute name of a field and the attribute value with the highest frequency of occurrence in this field. The recommended node is a Map structure, which stores the field name and the corresponding attribute value; each node is connected by an edge, and the corresponding value of the parent node field is stored at the same time. attribute value.

S3.2: Dynamic tree generation

Read in the data table generated after step S1, construct a dynamic tree in which the division nodes of the first layer are declaration lines, and the division nodes of the second and third layers are business units and declared commodities, and the nodes of each layer are connected through the parent node Connect each filled value of the corresponding field. The remaining layer division node selects the most relevant field obtained in step S2, and the recommended node is the only or empty remaining entry.

S3.3: Dynamic Tree Storage

The division nodes, recommended nodes and edges of the generated dynamic tree are respectively stored in the Mysql division node table, leaf node table and edge table.

Step S4: Intelligently recommend filling content according to the generated dynamic tree.

After the above steps, the dynamic tree obtained based on the historical data mining and correlation discovery algorithm is stored in the Mysql database. This step reads database data into memory and reconstructs the dynamic tree. First search the dynamic tree according to the declaration line, business unit and product name entered by the user, then perform a depth-first search according to the BestValue attribute stored in each node, and output all other fields and recommended filling values in an orderly manner; then interact with the user and perform some Field modification or input, and finally perform the above search and recommendation again based on the new input value until it meets the input expectations.

Step S5: Periodically and automatically perform data maintenance and update according to newly entered data.

First, store each entry of the user into the database and add a time tag, then add new data to the historical data set at a fixed time every week according to the time tag, and construct a new dynamic tree according to the newly generated historical data set and store it.

Claims (2)

1. A method for intelligently filling customs declaration forms based on historical data mining, characterized in that it comprises the following steps: S1: Preprocess the historical data of the customs declaration form, merge the header and body data, and remove irrelevant fields: S1.1: Use Spark distributed computing to merge data The real customs declaration data is divided into header data and body data, which are stored in two data tables respectively. The header data describes an order information, such as the import and export port, declaration unit, and transaction method; the body data describes an order Commodity specific information, such as commodity number, commodity name, declared unit price; the present invention uses the Spark distributed computing method to connect the two tables through the order number as the primary key to obtain a data table containing all the information fields to be filled in, and store it in the Hive database middle; S1.2: Count the null values of each field and remove irrelevant fields During the preprocessing process, count the ratio of the number of empty value data items in a certain field to the total number of data items. If it is greater than 90%, remove the field; at the same time, remove the fields that have no actual recommendation value such as time, serial number, and manual number; Through the above preprocessing, a field data table with recommended value is obtained; S2: Design and implement a correlation analysis algorithm for each field based on specific values By combining real customs declaration data with fields, it is judged that the value of a certain field is determined, and the correlation between the field and other fields is determined, thereby simulating the continuous interaction process with the user in the actual entry scene; The definition of correlation is, given field A and its value a, when the value of a certain field B is determined, so that the values of fields other than A and B that need to be entered are unique or the options are the fewest, it is called the relationship between field A and field B. the greatest correlation between The input of the algorithm is the historical data set, user input field A and field value a, and the output is the most relevant field B under the current input situation. The algorithm is implemented as follows: 1. Segment the historical data set according to the input field A and field value a to obtain the sub-data set under the specific input situation; 2. Deduplicate the sub-dataset; 3. Calculate the correlation between the remaining fields that need to be entered and field A, and select field B that has the greatest correlation with field A; 4. Output field B; S3: Design a tree structure, construct a dynamic tree based on the correlation of each field and store it After discovering the field correlation relationship based on specific values through S2, design a tree structure and build a tree with the shortest historical reporting path: S3.1: Tree structure design The tree structure includes nodes and edges. The nodes are divided into partition nodes and recommended nodes. The partition nodes are the attribute name of a certain field and the attribute value with the highest frequency of occurrence of the field. The recommended node is a Map structure, which stores the field name and Corresponding attribute value; each node is connected by an edge, and the attribute value corresponding to the parent node field is stored on the edge; S3.2: Dynamic tree generation First, read in the field data table preprocessed by S1, define the first-level division node fields as declaration lines, define the second and third-level division nodes as business units and commodity names respectively, and define the nodes and nodes of each layer They are connected by filling in the values of the corresponding fields of the parent node; The nodes of the fourth layer and subsequent layers are selected according to the corresponding attribute values of the upper layer nodes and edges as input, and the field with the highest correlation is selected as the lower layer node until the value of other input fields after a certain node is unique or all fields are entered. Generate recommended nodes, store other input fields and field values or generate empty nodes; Through the above steps, a dynamic tree with the shortest reporting path in history is generated; S3.3: Dynamic Tree Storage Store the dynamic tree generated by S3.2 in the MySql database; S4: Based on the generated dynamic tree, intelligently recommend filling content Read each structure of the dynamic tree stored in the MySql database into the memory, and re-build the corresponding dynamic tree in the memory: firstly search the dynamic tree according to the declaration line, business unit and product name entered by the user, and then perform a search based on the BestValue attribute stored in each node Depth-first search, orderly output all other fields and recommended filling values; then interact with the user, modify or input some fields, and finally perform the above search and recommendation again based on the new input values until the input expectations are met; S5: Periodically and automatically perform data maintenance and update based on newly entered data First, store each entry of the user in the database and add a time tag, then add new data to the historical data set at a fixed time every week according to the time tag, and construct a new dynamic tree according to the newly generated historical data set and store it, so as to achieve Data is automatically maintained and updated. 2. A customs declaration form intelligent filling method based on historical data mining as claimed in claim 1, characterized in that: the table structure in S3.3 is as shown in the following table: Table 1 Divide node table structure field name Field Type illustrate id int self-growth, primary key Field_name String Fill in the field name Best_value String Most frequent field value name String Node name, unique Level int tree structure level Agent_code String Corresponding reporting line name date date The record was inserted at Table 2 Leaf node table structure field name Field Type illustrate id int self-growth, primary key value String Field name + field value splicing Level int tree structure level Agent_code String Corresponding reporting line name date date The record was inserted at Table 3 Edge table structure