CN111459994A - A method and system for analyzing big data for disabled people - Google Patents

Abstract

The invention provides an analysis method for disabled big data, which includes: decomposing attributes of a dataset corresponding to the disabled big data, decomposing all selected attributes into a plurality of broken attributes according to their options, and discretizing continuous attributes; and then based on the decomposition The latter judgment attribute uses one-hot encoding to represent each piece of data. In each code, each judgment attribute corresponds to an attribute value. Among them, the selected attribute value of the option corresponding to the judgment attribute is 1. Otherwise, it is 0. The data is converted into one-hot encoding to form a data set matrix; based on the data set matrix, a rule set is generated for the judgment attribute whose attribute value is 1 in the data set matrix; A set of association rules, which calculates the weight of the influence of other attributes in the dataset on the target attribute, and the weights of the influence of all other attributes on the target attribute form the weight set of the target attribute.

Description

A method and system for analyzing big data for disabled people

technical field

The invention relates to the field of data mining, in particular to the field of intelligent regression analysis of big data, and more particularly, to a method and system for analyzing big data for disabled persons.

Background technique

The Basic Big Data of Disabled Persons is a government-funded survey and statistical data set, which aims to investigate and register the current economic status, employment and poverty alleviation, accessible community services and education of disabled persons across the country, as well as basic information on individual subjects. These registered information It is updated annually by non-profit professional organizations (such as the Disabled Persons' Federation). The process of implementing a registration survey includes: designing a disability registration or survey form with a series of systematic questions; distributing the form to grass-roots communities; grass-roots communities organizing disability reporting; validating and integrating survey feedback into a data set of forms; publishing the data set . Because the basic big data of disabled people usually has great industry effect and is funded by the government, it is considered as one of the authoritative sources for government and social organization practitioners to formulate data-driven policies. That is to say, through the analysis of the disabled data set, the practitioners of the government and social organizations can better understand the current situation and needs of the disabled group, so as to formulate corresponding reasonable policies.

Data regression analysis aims to discover a range of other attributes that influence changes in the value of the target attribute, and to quantify the impact of these attributes. Through the attribute value to describe the evolution in a targeted manner, data regression analysis can know which attributes will have a positive or negative impact on the target attribute, and which of these attributes have a greater influence.

Regression analysis research on disabled big data includes classic sociological research methods and currently popular data mining analysis methods:

Classical methods combine expert knowledge with statistics. In general, these methods determine the variable inputs for each analysis based on expert knowledge and random inference, which are then analyzed using statistical algorithms. Classical methods have some inherent defects. On the one hand, the research of classical methods is time-consuming and labor-intensive, requiring many experts to help researchers create models based on their knowledge and inferences. On the other hand, the results generated by classical methods are simple and limited. It is difficult to obtain good analytical results.

Data mining methods generally use model-based algorithms. For regression analysis, most models (such as neural networks) are black box models, and the results are not very interpretable and cannot be traced back when something goes wrong. For interpretable models (such as linear regression), the analytical accuracy is generally not guaranteed. For example, linear regression models assume that the relationship between the target value y and the attribute x = (x ₁ , x ₂ , ..., x _n ) is linear , the target value is calculated as y = α ₀ +α ₁ x ₁ +α ₂ x ₂ +...+α _n x _n , where α ₀ is the error term for all other factors affecting variable y except attribute x. The linear regression model assumes that the attributes are independent of each other, does not consider the correlation and role between the attributes, and requires the attributes to be as continuous as possible. However, the disabled big data is a data set based on a questionnaire, and there are often some attribute items that are missed by the respondents. The data attributes generally include two forms of true and false questions and multiple choice questions, both of which are discrete. And some attributes are interrelated and interact with each other. In addition, for the selection attribute with M options, the one-hot encoding mode is generally used in data mining to decompose the selection attribute into M judgment attributes, each judgment attribute corresponds to an option of the selection attribute, obviously this M The judgment attributes are also interrelated (M judgment attributes have and only one is 1). Therefore, the big data of disabled persons contains a large number of interrelated discrete data attributes, which is not suitable for regression analysis by using a linear regression model.

SUMMARY OF THE INVENTION

Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide a new method for analyzing big data of disabled persons, to realize data correlation analysis and regression analysis, and to mine the internal connection of data.

According to a first aspect of the present invention, the present invention provides a data analysis method for disabled big data, comprising the following steps:

S1. Decompose the attributes of the dataset corresponding to the disabled big data, decompose all selected attributes into multiple judgment attributes according to their options, and discretize the continuous attributes; then use one-hot encoding to represent each piece of data based on the decomposed judgment attributes. In each code, each judgment attribute corresponds to an attribute value, wherein the attribute value of the selected option corresponding to the judgment attribute is 1, otherwise it is 0, and all data in the data set is converted into one-hot encoding to form a data set matrix; among them, The continuous attributes are discretized by means of data segmentation.

S2. Based on the data set matrix, a rule set is generated for the judgment attribute with the attribute value of 1 in the data set matrix, wherein a rule is generated between each judgment attribute and other attributes, preferably, including:

S21, take a judgment attribute as an analysis object, and form a data sample matrix of the analysis object with all the data in the data set whose attribute value of the judgment attribute is 1;

S22. Sort the judgment attributes with the attribute value of 1 in the data sample matrix according to the frequency of occurrence, and quantify by the support degree, wherein the support of each judgment attribute other than the analysis object in the data sample in the data sample matrix is calculated respectively. degree and the support degree in the data set matrix, and calculate the ratio of the support degree of each judgment attribute in the data sample matrix to its support degree in the data set matrix to obtain the regular relationship between each judgment attribute and the analysis object. Confidence, the attribute value whose support is less than the support threshold or the confidence is less than the confidence threshold is removed from the data sample matrix, and the support threshold and the confidence threshold are preset based on historical data analysis results; in some of the present invention In the embodiment, the support degree of the attribute value xik=1 is calculated in the following manner:

The confidence level of the attribute value x _jk =1 is calculated as follows:

是一条规则。where T is the dataset matrix, X is the data sample matrix composed of all the data in T containing the attribute value x _uv = 1, x is a row in the matrix X, |x∈X, x _jk ∈x| is the matrix X contains the number of rows of the attribute value x _jk = 1, |X| is the number of rows of the matrix X, N is the number of rows of the dataset matrix,

is a rule.

S23, construct a frequent pattern tree of the analysis object according to the remaining data of the dataset samples in step S22, take the analysis object and the data sample matrix with the attribute value of the analysis object as 1 as the top layer of the tree, and use other attributes in the data sample matrix of the analysis object and the data matrix composed of the data whose attribute value is 1 in the data sample matrix is used as the second layer of the tree, and each other attribute corresponds to a node; the data sample of this analysis object is used as the new data set matrix, and the second The attribute corresponding to the layer node is used as a new analysis object, and the third layer of the frequent pattern tree is constructed, and then the data matrix corresponding to the third layer node is used as the new data set matrix to construct the fourth layer of the frequent pattern tree, and so on, until Stop building the frequent pattern tree when any of the following conditions are met: the height of the frequent pattern tree meets the preset requirements, all attributes with an attribute value of 1 are stored in the tree, the support of attribute values at a certain level is lower than the support threshold or The confidence level is below the confidence level threshold;

S24, combining the rules based on the frequent pattern tree constructed in step S23, so that each judgment attribute corresponds to a rule.

S3. Perform regression analysis on the matrix of the data set, take a judgment attribute as the target attribute in turn, and calculate the weight of the influence of other attributes in the data set on the target attribute based on the association rule set, and the weight of the influence of all other attributes on the target attribute forms the A collection of weights for the target attribute. Each time a judgment attribute is used as the target attribute, the column corresponding to the target attribute in the data set matrix is used as the independent variable set, the other judgment attributes in the data set matrix are used as a series of dependent variable sets, and the rule set of all judgment attributes is used as the dependent variable. The attribute value in the independent variable set is the attribute value set corresponding to each row of the target attribute in the data set matrix, and the target attribute is subjected to regression analysis. In some embodiments of the present invention, the weight value set of the target attribute is obtained by performing regression analysis in the following manner:

Data set matrix: T _N×(M+1) ;

Set of independent variables: y=(y ₁ , y ₂ , ..., y _n ), where n=N;

其中，

表示第i个M维的因变量，上标T代表矩阵的转置；Set of dependent variables:

in,

Represents the i-th M-dimensional dependent variable, and the superscript T represents the transpose of the matrix;

Rule set: r ^T , where r ^T is a rule of dimension M+1;

The relationship between the independent variable and the dependent variable is expressed as a regression equation:

y _i =α ₀ +α ₁ x _i1 +...+α _M x _iM +β ₁ r ₁ (x _i )+β ₂ r ₂ (x _i )+...+β _M r _M (x _i )-β ₀ r ₀ (y _i )

Among them, θ=(α ₀ ... α _M , β ₀ ... β _M ) is the normalized weight of the independent variable set; including the weight of the target attribute itself and the weight based on the rule;

Based on the given evaluation function, the regression equation is solved to obtain the weight value set of the target attribute: γ=(γ ₀ =α ₀ -β ₀ , γ ₁ =α ₁ +β ₁ …γ _M =α _M +β _M ) .

Preferably, the weight value is obtained by gradually approximating the weight parameter by using the gradient descent method to solve the regression equation.

S4. Perform a visual graph operation on the weight set in step S3.

According to a second aspect of the present invention, an analysis system for disabled big data is provided, including: an attribute decomposition module, which is used to decompose the attributes of the data set to be analyzed, discretize continuous attributes, and use one-hot encoding to represent each piece of data To convert the data set into a data set matrix, the number of rows of the matrix is the number of data pieces of the data set, and the number of columns of the matrix is the number of discrete attributes after decomposition; the data association analysis module is used to analyze the attribute decomposition module. Analyze the decomposed data set, and output the rule set corresponding to each attribute; the data regression analysis module is used to perform regression analysis on the data set decomposed by the attribute decomposition module based on the rule set, and output the weight set of each attribute; The result visualization module is used to visualize the weight set of attributes output by the data regression analysis module.

Compared with the prior art, the present invention has the following advantages: (1) At the macro-application level, the present invention can strengthen the overall planning and management of disabled persons' data resources, improve scientific decision-making ability, and improve career management services for disabled persons. Therefore, the present invention can provide assistance for policy formulation of precise disability assistance, and has important engineering application value. (2) The mathematical model of regression analysis produced by the present invention is interpretable, and the influence evaluation is performed on other attributes that affect the change of the target attribute value. (3) By digging out which attribute values often appear in a piece of data, the present invention describes the internal correlation of different attributes in the form of rules, applies the rules to regression equations, and fully considers the correlation characteristics between attributes. (4) The realization of data association requires the data set to meet a major requirement, that is, the type of the specified attribute in the data set needs to be a discrete value rather than a continuous value. The vast majority of disabled big data is discrete data (that is, the data values represent the selection results of answering options to a question), which makes it particularly feasible to apply data correlation to regression analysis. (5) By generating a regression equation describing the change trend of the target attribute, and outputting the weight of the attribute and the rule as its influencing factor in the equation, the present invention can mine a series of factors that affect the changes of various important indicators of the data set, At the same time, each factor is accompanied by a weighting factor describing its influence. The weight factor includes not only the influence of the attribute itself on the target attribute, but also the joint influence of multiple association rules on the target attribute. In addition, the attributes that do not appear in the regression equation have no dominant influence on the target attribute.

Description of drawings

The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

1 is a schematic diagram of a frequent pattern tree according to an example of the present invention;

Fig. 2 is a schematic diagram of the analysis results of the disabled big data of three provinces A, B, and C according to an example of the present invention;

3 is a schematic diagram of the analysis results of the disabled big data in province A according to an example of the present invention;

FIG. 4 is a schematic diagram of the analysis results of the disabled big data in province B according to an example of the present invention;

5 is a schematic diagram of the analysis results of the disabled big data in province C according to an example of the present invention;

FIG. 6 is a schematic diagram of an analysis system for disabled big data according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

As the disabled big data is a government-funded survey and statistical data set, it is unique compared to other big data. Fully understanding and analyzing the disabled big data will help to formulate more reasonable policies and protect the livelihood of the disabled.

According to an embodiment of the present invention, the present invention provides an analysis method for disabled big data, including steps S1, S2, S3 and S4, each step will be described in detail below.

In step S1, attribute decomposition is performed on the dataset corresponding to the disabled big data:

First of all, for the continuous attributes existing in the data set, the continuous attributes are first discretized and then the attributes are decomposed. For example, for the continuous attribute "age", it can be discretized into four age groups: teenagers (0-18), youth (19-35) , middle-aged (36-55), old (56-).

其中每一个判断属性与选择问题中的每一个选项所对应。因此当被调查者勾选第k个选项时，所对应的X_jk被赋值为1，反之则被赋值为0，则该被调查者的数据用独热编码表示为{0，0，…，1，0，0}的一条数据，其中第X_jk判断属性的属性值为1，其他判断属性的属性值为0。通过属性分解，数据集中的每一条数据均被表示为一条编码，在每条编码中每个判断属性对应一个属性值，其中，对应的选项被选中的判断属性的属性值为1，反之为0，将数据集中所有数据转换成独热编码形成数据集矩阵，其中，矩阵的行数是数据集中数据的条目数，矩阵的列数是数据集中对应的判断属性的个数。Secondly, for each selection attribute in the dataset, one-hot encoding is used to represent each option. For example, assuming that the jth attribute X _j corresponds to the result of the respondent's selection of M _j options, then select Attribute X _j will be decomposed into M _j judgment attributes:

Each of these judgment attributes corresponds to each option in the choice question. Therefore, when the respondent selects the kth option, the corresponding X _jk is assigned a value of 1, otherwise, it is assigned a value of 0, and the respondent's data is represented by one-hot encoding as {0, 0, ..., A piece of data of 1, 0, 0}, wherein the attribute value of the X _jk judgment attribute is 1, and the attribute value of other judgment attributes is 0. Through attribute decomposition, each piece of data in the data set is represented as a code, and each judgment attribute in each code corresponds to an attribute value, wherein the attribute value of the judgment attribute whose corresponding option is selected is 1, otherwise it is 0 , convert all data in the dataset into one-hot encoding to form a dataset matrix, where the number of rows in the matrix is the number of entries in the dataset, and the number of columns in the matrix is the number of corresponding judgment attributes in the dataset.

According to an example of the present invention, each data survey statistical table in a large dataset of disabled persons includes the following selection attributes, that is, options:

Poverty and status of file and card establishment: poverty population, other poor population, and others;

Household housing status: in good condition, identified as dangerous, not identified as dangerous, rented, rented or without fixed residence;

Education level: never attended school, primary school, junior high school, high school (including technical secondary school), college junior college, college undergraduate, graduate student;

Current employment and poverty alleviation needs: vocational skills training, job introduction, rural practical technical training, capital and credit support, other assistance, no demand;

Social assistance and housing improvement: minimum living guarantee, assistance and support for the extremely poor, medical assistance, other assistance (education assistance, housing assistance, employment assistance and other temporary assistance), enjoyment of the housing and construction sector’s rural dilapidated house renovation policy (only agricultural household registration is optional ),none.

Attribute decomposition is performed on the data in this dataset. The first selection attribute contains 3 options and is decomposed into 3 judgment attributes; the second selection attribute contains 5 options and is decomposed into 5 judgment attributes; the third selection attribute contains 7 options, decomposed into 7 judgment attributes; the fourth selection attribute contains 6 options, decomposed into 6 judgment attributes; the fifth selection attribute contains 6 options, decomposed into 6 judgment attributes, therefore, in the data set After the selection attribute is decomposed, the judgment attributes of 3+5+7+6+6=27 single-choice questions are obtained. Each piece of data is represented by one-hot encoding as a code containing 27 attribute values. If the user selects an option, the attribute value of the corresponding judgment attribute is 1, and the other is 0.

In step S2, based on the data set matrix of step S1, a rule set is generated for the judgment attribute whose attribute value is 1 in the data set, wherein a rule is generated between each judgment attribute and other attributes, including:

S21, take a judgment attribute as an analysis object, and form a data sample matrix of the analysis object with all the data in the data set whose attribute value of the judgment attribute is 1;

S22. Rank other attributes with the attribute value of 1 in the data sample matrix according to the frequency of occurrence, and quantify by the support degree, wherein the support of each judgment attribute other than the analysis object in the data sample in the data sample matrix is calculated respectively. degree and the support degree in the data set matrix, and calculate the ratio of the support degree of each judgment attribute in the data sample matrix to its support degree in the data set matrix to obtain the regular relationship between each judgment attribute and the analysis object. Confidence, remove the attribute values whose support is less than the support threshold or whose confidence is less than the confidence threshold from the data sample matrix;

S23, construct a frequent pattern tree of the analysis object according to the remaining data of the dataset samples in step S22, take the analysis object and the data sample matrix with the attribute value of the analysis object as 1 as the top layer of the tree, and use other attributes in the data sample matrix of the analysis object and the data matrix composed of the data whose attribute value is 1 in the data sample matrix is used as the second layer of the tree, and each other attribute corresponds to a node; the data sample of this analysis object is used as the new data set matrix, and the second The attribute corresponding to the layer node is used as a new analysis object, and the third layer of the frequent pattern tree is constructed, and then the data matrix corresponding to the third layer node is used as the new data set matrix to construct the fourth layer of the frequent pattern tree, and so on. Frequent pattern tree, stop building the frequent pattern tree until any of the following conditions are met: the height of the frequent pattern tree meets the preset requirements, all attributes with an attribute value of 1 are stored in the tree, and the support of attribute values at a certain layer is low at the support threshold or the confidence is lower than the confidence threshold;

S24, based on the frequent pattern tree constructed in step S23, merge the rules so that each judgment attribute corresponds to a rule;

S25. Taking another judgment attribute as the analysis object, repeat steps S21 to S24 until a rule set of all judgment attributes is generated.

It should be noted that a rule refers to deriving another condition from one condition. For example, deriving Y from X is a rule, which is expressed as X=>Y, and the mathematical formula can be expressed as: r(x)=if x ₁ ∈ { 1, 2, 3} and x ₂ <4 then 1else 0, in a frequent pattern tree for Y, the top level of the tree is Y, and the second and related lower levels are X.

According to an embodiment of the present invention, the data set matrix is T _N×(M+1) , wherein the data set matrix consists of N entries, and each entry contains M+1 discrete attributes that have been attribute decomposed. A rule set is generated based on each option value of each judgment attribute (except for the one-hot encoding of 0), that is, a rule set of judgment attributes is generated based on the data whose attribute value of each judgment attribute is 1:

P1. Taking a judgment attribute x _uv as the analysis object, select all data including the attribute value x _uv =1 from the data set T _N×(M+1) to form a data sample matrix X.

P2. Sort the frequently occurring attribute values in the data sample matrix X according to the frequency of occurrence of attribute value 1 of the judgment attribute, and quantify according to their support degrees. The support degree of each attribute value x _jk =1 in the data sample is calculated as follows :

的置信度计算如下：where x is a row in the matrix X of data samples, and |x∈X, x _jk ∈ x| is the number of rows in the matrix of data samples containing the attribute value x _jk = 1, |X| is the number of rows in matrix X, and M _j Indicates the number of options contained in the jth selection attribute to which x _jk belongs, that is, the number of judgment attributes contained in the jth selection attribute after attribute decomposition, and M _u represents the options contained in the uth selection attribute to which x _uv belongs The number is the number of judgment attributes included in the u-th selection attribute after attribute decomposition.

The confidence level of is calculated as follows:

The higher the support in the data sample matrix, the higher the frequency of the attribute value in the data sample; the higher the confidence, the higher the credibility of the corresponding rule. By setting predefined support threshold and confidence threshold, all attribute values with support or confidence below the threshold will be considered infrequent and removed from the corresponding data samples. Support threshold and confidence threshold can be preset through historical data analysis results.

P3. According to the data samples processed in step P2, a frequent pattern tree of the analysis object is constructed. The top level of the tree is the attribute value with the highest support (one-hot encoding is 1) and the data sample containing the attribute value. Obviously, the attribute value with the highest support in the data sample matrix is the analysis object x _uv , that is, supp(x _uv ) =M _u , the corresponding data sample matrix is X. The second level of the tree is the attribute value above the support threshold and confidence threshold in order, and the data samples containing the attribute value. Take the data set contained in each node of the second layer of the pattern tree as a new data sample matrix X, calculate the support and confidence of each attribute value as the third layer node, and recursively construct the frequent pattern tree until the rule length ( The height of the tree) or all attribute values (one-hot encoded as 1) are stored into the tree or the support or confidence of the attribute value is below a threshold.

P4. Merge the rules in the frequent pattern tree of the analysis object based on the subsequent attributes corresponding to the lower nodes of the tree, so that the rules corresponding to the paths from all the bottom nodes of the frequent pattern tree to the top root node are merged into a rule corresponding to the analysis object r _j (x).

The present invention will be further described below with reference to an example.

Each data survey statistical table in a large data set of disabled persons contains the following selection attributes:

Poverty and status of file and card establishment: poverty population, other poor population, and others;

Household housing status: in good condition, identified as dangerous, not identified as dangerous, rented, rented or without fixed residence;

Education level: never attended school, primary school, junior high school, high school (including technical secondary school), college junior college, college undergraduate, graduate student;

Current employment and poverty alleviation needs: vocational skills training, job introduction, rural practical technical training, capital and credit support, other assistance, no demand;

Social assistance and housing improvement: minimum living guarantee, assistance and support for the extremely poor, medical assistance, other assistance (education assistance, housing assistance, employment assistance and other temporary assistance), enjoyment of the housing and construction sector’s rural dilapidated house renovation policy (only agricultural household registration is optional ),none.

There are 100 pieces of data in the data set (that is, N=100, the one-hot encoding of these 100 pieces of data forms a data set matrix T _100x(26+1) ), M ₁ =3, M ₂ =5, M ₃ =7, M ₄ =6, M ₅ =6, the data of the jth row in the data set matrix can be expressed as: x _j ={x _j1 , x _j2 ,...x _j27 }, in this example, x _uv is used to represent "poverty and documentation Situation: "The poor population with national files and cards" is used as the analysis object. Among the 100 pieces of data, 70 pieces of data have selected x _uv (these 70 pieces of data are the data sample matrix X), and establish an association rule for x _uv :

First, assuming that x _jk represents "family housing status: it is identified as a dilapidated house", among the 70 pieces of data in the data sample, 30 pieces of data have selected x _jk , and among the 100 pieces of data in the data set, 50 pieces of data have been chosen x _jk , then

(x_jk所属的选择属性具有5个可选项)

(The selection attribute to which x _jk belongs has 5 optionals)

(x_uv所属的选择属性具有3个可选项)

(The selection attribute to which x _uv belongs has 3 optionals)

Secondly, assuming that x _jk represents "educational level: never attended school", among the 70 pieces of data in the data sample, 20 pieces of data have selected x _jk , and among the 100 pieces of data in the dataset, there are 40 pieces of data x _jk is chosen, then

(x_jk所属的选择属性具有7个可选项)

(The selection attribute to which x _jk belongs has 7 optionals)

(x_uv所属的选择属性具有3个可选项)

(The selection attribute to which x _uv belongs has 3 optionals)

In this way, the first and second layers of the frequent pattern tree are constructed. The first layer (root node) is "Poverty and Filed Card Status: National Filed and Registered Poor Population", and the second layer (leaf node) has 2 The nodes are "family housing status: identified as dangerous" and "educational level: never attended school" (current tree length is 2).

Then, take the 70 pieces of data in the data sample as a new data set matrix T, and "family housing status: identified as a dilapidated house" as a new analysis object x _uv , so as to continue to build the third layer of the tree, assuming that the third layer ( Leaf node) has 1 node, which is "current employment poverty alleviation needs: vocational skills training", then this rule can be simply expressed as "current employment poverty alleviation needs: vocational skills training and family housing status: identified as dilapidated buildings => poverty and construction File status: the poor population on file with the national file”, and so on, recursively construct the frequent pattern tree based on “Poverty and file status: the poor population on file with the national file” as shown in Figure 1, in which , the top layer of the tree is "Poverty and the status of filing and setting up a card: the poverty-stricken population registered by the state", and the second layer of the tree is "the status of the family housing: it has been identified as a dilapidated house", "education level: never went to school"", etc., each judgment attribute corresponds to a rule, each rule corresponds to the path from all the bottom nodes in the tree to the root node, a rule corresponding to a frequent pattern tree is the rule of an analysis object, and all the rules of the analysis object form a rule set. If a rule is contrary to common sense, it is convenient to backtrack this path and find out which node has doubts (for example, the confidence or support is inappropriate, and the quality of some fields in the data set is poor).

In step S3, a regression analysis is performed on the data set matrix, one judgment attribute is used as the target attribute in turn, and based on the association rule set, the weight of the influence of other attributes in the data set on the target attribute is calculated, and the influence of all other attributes on the target attribute is calculated. The weights form the set of weights for this target attribute.

所有判断属性对应的规则集合也作为因变量r^T，其中，上标T代表矩阵的转置，

表示第i个M维的因变量，r^T是M+1维的规则，如果单次规则分析中，某个判断属性的选项值之间互相冲突，那么忽略该属性对应的规则。According to an embodiment of the present invention, in a single analysis, first take a judgment attribute as a target attribute, and use the column corresponding to the target attribute in the data set matrix as the independent variable set y=(y ₁ , y ₂ ,..., y _n ), other attributes in the data set matrix form a set of dependent variables as a series of dependent variables

The set of rules corresponding to all judgment attributes is also used as the dependent variable r ^T , where the superscript T represents the transpose of the matrix,

Represents the i-th M-dimensional dependent variable, and r ^T is the M+1-dimensional rule. If the option values of a certain judgment attribute conflict with each other in a single rule analysis, the rule corresponding to this attribute is ignored.

Second, the correlation between the independent variable and the set of dependent variables is described by a linear equation:

y _i =α ₀ +α ₁ x _i1 +...+α _M x _iM +β ₁ r ₁ (x _i )+β ₂ r ₂ (x _i )+...+β _M r _M (x _i )-β ₀ r ₀ (y _i )

where θ=(α ₀ ... α _M , β ₀ ... β _M ) is the normalized weight of the set of independent variables, by unifying the set of dependent variables and the rule results with matrix X, and the set of independent variables with matrix Y, The regression equation can be simplified as: Y=h _θ (X)=θ ^T X.

根据本发明的前述一个示例，数据集中共100条数据，经过属性分解后得到27个单选问题属性，则回归方程可以表示为：The weight set can be obtained by solving the regression equation according to the preset evaluation function. According to an embodiment of the present invention, a gradient descent method can be used to gradually approximate the weight parameters, or it can be directly calculated

According to the foregoing example of the present invention, there are 100 pieces of data in the data set, and 27 single-choice question attributes are obtained after attribute decomposition, then the regression equation can be expressed as:

For the least squares method, the evaluation function is:

The evaluation function takes the partial derivative with respect to θ:

By solving the system of equations, the process of determining the weight value set by data regression analysis can be expressed by the following formula:

θ=(X ^T X) ^-1 X ^T Y

Finally, γ=(γ ₀ =α ₀ -β ₀ , γ ₁ =α ₁ +β ₁ ...γ _M =α _M +β _M ) is to describe the relationship between other judgment attributes (including the attributes themselves and their relationship) to the target attribute A collection of weight values for influence. According to the aforementioned example of the present invention, it is possible to select “Poverty and status of filing and setting up cards: the poverty-stricken population registered by the state”, which has a weight α ₁ . training” and “family housing status: identified as a dilapidated house”, then according to the rules, it can be deduced that “poor and file status: the state file and file the poor”, it also has a weight β ₁ , and γ ₁ = α ₁ + β ₁ is the final weight value of "Poverty and the status of file registration: the poor population with national file and registration".

S4. Perform a visual graphic operation on the weight set in step S3, so as to intuitively analyze the influence of other attributes on the target attribute. The result of the data regression analysis can be visualized in the form of a histogram, where a large amplitude value indicates that the attribute has an impact on the target attribute. greater impact.

In order to illustrate the effect of the present invention, the inventor selects a total of 3,798,462 pieces of data from three provinces A, B, and C from the national basic database of disabled persons for analysis experiments. Each piece of data contains 28 non-privacy attributes, covering basic personal information, economy, housing, Education, employment and poverty alleviation, social security, basic medical care and rehabilitation, barrier-free, culture and sports. Using the analysis method of the present invention, the 3,798,462 pieces of data are analyzed as a whole with the target attribute of "poverty and the status of file and card registration: the poor population with national file and card". As shown in Figure 2, the data of the three provinces are analyzed respectively, and the visualization results of regression analysis are obtained as shown in Figures 3 to 5, wherein Figure 3 is the visualization result of the data in Province A, and Figure 4 is the visualization result of the data in Province B. 5 is the visualization result of the data in Province C, from which the present invention can obtain valid information examples as follows:

(1) For the three provinces as a whole, compared with the non-poor disabled persons, the top three influencing factors of the poor disabled persons are the minimum living security, the enjoyment of the dilapidated house renovation policy of the housing construction department, and the certified dilapidated house.

(2) There are significant differences in influencing factors among provinces.

(3) The salient characteristics of the poor population in province A are as follows: it has been identified as a dilapidated house; the housing and construction department has a policy of renovating dilapidated houses; it has not been identified as a dangerous house.

(4) The salient characteristics of the impoverished population in province B are as follows: it has been identified as a dilapidated house;

(5) The salient characteristics of the poor population in province C are as follows: minimum living guarantee; assistance and support for the extremely poor; enjoying the dilapidated house renovation policy.

According to an embodiment of the present invention, as shown in FIG. 6 , the present invention provides an analysis system for disabled big data, including: an attribute decomposition module, which is used to decompose the attributes of the data set to be analyzed, discretize the continuous attributes, and One-hot encoding is used to represent each piece of data to convert the data set into a data set matrix. The number of rows of the matrix is the number of data pieces of the data set, and the number of columns of the matrix is the number of discrete attributes after decomposition; data association analysis module , used to analyze the data set decomposed by the attribute decomposition module, and output the rule set corresponding to each attribute; the data regression analysis module is used to perform regression analysis on the data set decomposed by the attribute decomposition module based on the rule set, Output the weight set of each attribute; the result visualization module is used to visualize the weight set of attributes output by the data regression analysis module.

The beneficial effects of the present invention are as follows: (1) At the macro-application level, the present invention can strengthen the overall planning and management of disabled persons' data resources, improve scientific decision-making ability, and improve career management services for disabled persons. Therefore, the present invention can provide assistance for policy formulation of precise disability assistance, and has important engineering application value. (2) The mathematical model of regression analysis produced by the present invention is interpretable, and the influence evaluation is performed on other attributes that affect the change of the target attribute value. (3) By digging out which attribute values often appear in a piece of data, the present invention describes the internal correlation of different attributes in the form of rules, applies the rules to regression equations, and fully considers the correlation characteristics between attributes. (4) The realization of data association requires the data set to meet a major requirement, that is, the type of the specified attribute in the data set needs to be a discrete value rather than a continuous value. The vast majority of disabled big data is discrete data (that is, the data values represent the selection results of answering options to a question), which makes it particularly feasible to apply data correlation to regression analysis. (5) By generating a regression equation describing the change trend of the target attribute, and outputting the weight of the attribute and the rule as its influencing factor in the equation, the present invention can mine a series of factors that affect the changes of various important indicators of the data set, At the same time, each factor is accompanied by a weighting factor describing its influence. The weight factor includes not only the influence of the attribute itself on the target attribute, but also the joint influence of multiple association rules on the target attribute. In addition, attributes that do not appear in the regression equation have no dominant influence on the target attribute.

It should be noted that although the steps are described above in a specific order, it does not mean that the steps must be executed in the above-mentioned specific order. In fact, some of these steps can be executed concurrently, or even change the order, as long as it can be achieved The required function can be.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that retains and stores instructions for use by the instruction execution device. Computer-readable storage media may include, but are not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing, for example. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (10)

1. A method for analyzing big data for disabled people is characterized by comprising the following steps:

s1, performing attribute decomposition on the data set corresponding to the big data of the disabled, decomposing all selected attributes into a plurality of judgment attributes according to options of the selected attributes, and discretizing continuous attributes; then, expressing each piece of data by adopting one-hot coding based on the decomposed judgment attributes, wherein each judgment attribute corresponds to one attribute value in each code, the selected attribute value of the option corresponding to the judgment attribute is 1, otherwise, the selected attribute value is 0, and converting all data in the data set into one-hot coding to form a data set matrix;

s2, generating a rule set for the judgment attributes with the attribute value of 1 in the data set matrix based on the data set matrix, wherein a rule is generated between each judgment attribute and other attributes;

and S3, carrying out regression analysis on the data set matrix, sequentially taking a judgment attribute as a target attribute, calculating the weight of the influence of other attributes in the data set on the target attribute based on the association rule set, and forming a weight set of the target attribute by the weights of all other attributes on the influence of the target attribute.

2. The method for analyzing big data of the disabled person according to claim 1, further comprising:

and S4, performing visual graphic operation on the weight set in the step S3.

3. The method for analyzing big data of the disabled as claimed in claim 1, wherein the continuous attribute is discretized by means of data segmentation.

4. The method for analyzing big data of disabled people according to claim 1, wherein the step S2 comprises:

s21, forming a data sample matrix of the analysis object by taking a judgment attribute as the analysis object and all data of which the attribute value of the judgment attribute is 1 in the data set;

s22, sorting the judgment attributes with the attribute value of 1 in the data sample matrix according to the occurrence frequency and quantizing the judgment attributes through the support degree, wherein the support degree of each judgment attribute except for the analysis object in the data sample matrix and the support degree in the data set matrix are respectively calculated, the ratio of the support degree of each judgment attribute in the data sample matrix to the support degree of each judgment attribute in the data set matrix is calculated to obtain the confidence coefficient of the rule between each judgment attribute and the analysis object, and the attribute value with the support degree smaller than the support degree threshold or the confidence coefficient smaller than the confidence coefficient threshold is removed from the data sample matrix;

s23, constructing a frequent pattern tree of the analysis object according to the residual data of the data set samples in the step S22, taking the analysis object and the data sample matrix with the attribute value of 1 of the analysis object as the top layer of the tree, taking a data matrix formed by other attributes in the data sample matrix of the analysis object and the data with the attribute value of 1 in the data sample matrix as the second layer of the tree, and enabling each other attribute to correspond to a node; taking a data sample of the analysis object as a new data set matrix, taking the attribute corresponding to the node of the second layer as a new analysis object, constructing the third layer of the frequent pattern tree, then taking the data matrix corresponding to the node of the third layer as a new data set matrix to construct the fourth layer of the frequent pattern tree, and so on, and stopping constructing the frequent pattern tree until any one of the following conditions is met: the height of the frequent pattern tree reaches a preset requirement, all attributes with the attribute value of 1 are stored in the tree, and the support degree of the attribute value at a certain layer is lower than a support degree threshold value or the confidence degree is lower than a confidence degree threshold value; wherein the support degree threshold value and the confidence degree threshold value are preset based on the historical data analysis result.

And S24, merging rules based on the frequent pattern tree constructed in the step S23, and enabling each judgment attribute to correspond to one rule.

5. The method for analyzing big data of disabled people according to claim 4,

the attribute value x is calculated as follows_jkSupport of 1:

the attribute value x is calculated as follows_jkConfidence of 1:

wherein T is a data set matrix, and X is a matrix formed by all the contained attribute values X in T_uvA data sample matrix of 1 data, X being a row within matrix X, | X ∈ X, X_jk∈ X | is the matrix X containing the attribute value X_jk1, | X | is the number of rows of matrix X, N is the dataset matrixThe number of rows of (a) to (b),

is a rule.

6. The method for analyzing big data for disabled persons according to claim 1, wherein in step S3, each time one judgment attribute is used as the target attribute, the column corresponding to the target attribute in the data set matrix is used as an independent variable set, the other judgment attributes in the data set matrix are used as a series of dependent variable sets, and the rule sets of all judgment attributes are used as dependent variables, wherein the attribute values in the independent variable sets are the attribute value sets corresponding to each row of the target attribute in the data set matrix, and the regression analysis is performed on the target attribute.

7. The method for analyzing big data of disabled people according to claim 6, wherein the step S3 is performed by performing regression analysis to obtain the weight value set of target attributes as follows:

a data set matrix: t is_N×(M+1)；

Self-variable set: y ═ y₁，y₂，...，y_n) Wherein N is N;

and (4) dependent variable set:

wherein,

representing the dependent variable of the ith M dimension, and the superscript T represents the transposition of the matrix;

rule set: r is^TWherein r is^TIs a rule of dimension M + 1;

the relationship of independent variables to dependent variables is expressed as a regression equation:

y_i＝α₀+α₁x_i1+…+α_Mx_iMten β₁r₁(x_i) Ten β₂r₂(x_i)+…+β_Mr_M(x_i)-β₀r₀(y_i)

Wherein θ ═ α₀...α_M，β₀...β_M) Is the normalized weight of the set of arguments; including the weight of the target attribute itself and the rule-based weight;

solving the regression equation based on a given evaluation function to obtain a weight value set of the target attribute: γ ═ y (γ)₀＝α₀-β₀，γ₁＝α₁+β₁...γ_M＝α_MTen β_M)。

8. The method for analyzing big data of disabled people according to claim 7,

and gradually approaching the weight parameters by adopting a gradient descent method to solve the regression equation to obtain the weight value.

9. An analysis system for the big data of the disabled based on any one of claims 1 to 8, comprising:

the attribute decomposition module is used for performing attribute decomposition on a data set to be analyzed, discretizing continuous attributes and representing each piece of data by adopting one-hot coding so as to convert the data set into a data set matrix, wherein the row number of the matrix is the number of the data sets, and the column number of the matrix is the number of the decomposed discrete attributes;

the data association analysis module is used for analyzing the data set decomposed by the attribute decomposition module and outputting a rule set corresponding to each attribute;

the data regression analysis module is used for carrying out regression analysis on the data set decomposed by the attribute decomposition module on the basis of the rule set and outputting a weight set of each attribute;

and the result visualization module is used for visually displaying the weight set of the attributes output by the data regression analysis module.

10. An electronic device comprising a memory and a processor, on which a computer program is stored which is executable on the processor, characterized in that the steps of the method according to any of claims 1 to 8 are implemented when the processor executes the program.

Images