CN118626758A - A data processing method and device based on least squares method of clustering - Google Patents

Abstract

The invention discloses a data processing method and device based on clustering least squares method, which belongs to the field of data processing, and comprises the steps of: using a computer program to merge a clustering algorithm with a least squares algorithm, and controlling the J(m _k ) value selection range of the clustering algorithm program module by error feedback between the output result of the least squares algorithm program module and the actual data in the computer, thereby using the clustering algorithm program module to eliminate abnormal data, so that the least squares method fitting result is closer to the target value; wherein J(m _k ) represents the square sum of the distances from the object m _k of class k to the cluster center; the data comprises aviation integrated electronic information equipment test data, and the aviation integrated electronic information equipment test data is input into the computer running the program through a computer input unit. The method of the invention is simple, highly feasible, and can make data processing more accurate.

Description

A data processing method and device based on least squares method of clustering

Technical Field

The present invention relates to the field of data processing, and more specifically, to a data processing method and device based on clustering least squares method.

Background Art

In many research areas such as scientific experiments, computer image processing, and signal processing, data processing is involved. In order to analyze the relationship between things, predict future trends, and make reasonable decisions, data fitting must be performed. Data fitting is widely used in many fields, so it has always been a hot topic. Its essence is to fit the curves and surfaces of data through given discrete data points, and to approximate these discrete points with smooth curves and surfaces, so as to achieve better fitting results. It is an important research topic in mathematics and other disciplines.

The least square method of curve fitting uses a polynomial function for fitting, and abandons the requirement that the desired function strictly passes through each given point ( _xi , _yi ). It only requires that the fitting curve can reflect the basic relationship of the data. This can eliminate the errors in the measured data itself to a certain extent, and is widely used in actual engineering data processing. The core content of the least square method of data fitting is: according to the given data set ( _xi , _yi ) (i = 0, 1, ..., m), select the approximate function form, that is, given the function class Φ, find the function f(x)∈Φ, so that the sum of the squares of the error _vi = f( _xi ) _-yi (i = 0, 1, ..., m) is the smallest, that is:

In geometric terms, it is to find the curve y = f(x) with the smallest sum of squares of distances from a given point (x _i , y _i ) (i = 0, 1, ..., m), as shown in Figure 1. The function f(x) is called a fitting function or a least squares solution, and the method for finding the fitting function f(x) is called the least squares method of curve fitting. As can be seen from Figure 1, the original data points are evenly distributed on both sides of the fitting curve f(x), and reflect the trend and characteristics of data changes. By using the least squares method to analyze the measured data, an approximate function form can be fitted based on the measured data to obtain a more accurate solution. In practical problems, due to the different observation accuracy of each point, weighted variance is often introduced.

The advantages of the least squares method are: Least squares curve fitting is a common method for measuring data processing. The best square approximation can approximate the function more evenly in an interval, and has the advantages of being simple and easy to implement, time-effective, and widely used, and can be easily implemented through a simple computer program. Under the conditions that the regression model meets the assumptions, the least squares method is used to estimate its regression parameters with good statistical properties, such as unbiasedness, consistency, and minimum variance.

The disadvantage of the least squares method is that since the amount of data to be processed in actual engineering tests is very large, if the least squares method is applied to the entire measurement data for one-time fitting, its fitting accuracy will be difficult to guarantee. The least squares method based on gradient information can easily fall into the local optimum, and the fitted function sometimes cannot meet the actual requirements. The actual test data varies greatly, and the purpose of data fitting for the test data is also different, so the results of data fitting using the least squares method often fail to meet the expected requirements. For example, due to occasional gross errors, outliers appear, or the probability distribution of the data deviates from the normal distribution. At this time, the fitting results using the least squares method will lose their good statistical properties. On the other hand, when the order of the normal equation is high, it often becomes ill-conditioned, so it must be treated with caution and handled skillfully. One of the effective methods is to introduce orthogonal polynomials to improve its ill-conditioned properties.

However, the prior art has the following technical problems: the accuracy needs to be improved.

Summary of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a data processing method and device based on the least squares method of clustering, which can make data processing more accurate.

The object of the present invention is achieved through the following solutions:

A data processing method based on the least squares method of clustering comprises the following steps:

A clustering algorithm and a least squares algorithm are integrated by using a computer program, and the J(m _k ) value selection range of the clustering algorithm program module is controlled by error feedback between the output result of the least squares algorithm program module and actual data in the computer, so that the clustering algorithm program module is used to eliminate abnormal data, so that the least squares fitting result is closer to the target value; wherein J(m _k ) represents the sum of squares of distances from objects m _k of class k to the cluster center; the data includes aviation integrated electronic information equipment test data, and the aviation integrated electronic information equipment test data is input into a computer running the program through a computer input unit.

Furthermore, the clustering algorithm and the least squares algorithm are integrated by a computer program, and the J(m _k ) value selection range of the clustering algorithm program module is controlled by error feedback between the output result of the least squares algorithm program module and the actual data in the computer, so that the clustering algorithm program module is used to eliminate abnormal data, so that the least squares fitting result is closer to the target value, which specifically includes the following sub-steps:

Step 1, assuming that the input data of n samples are (x1, p1, ..., q1), (x2, p2, ..., q2), ..., (xn, pn, ..., qn), and the test output data are y1, y2, ...yn, where n = [1, 2, ...] represents the sampling point; x, p, ..., q represents m different input feature parameters;

Step 2, the input data (x, p, ..., q) enters the clustering algorithm program module for processing through the computer input unit;

Step 3, the clustering algorithm program module outputs data to the least squares algorithm program module, and the least squares algorithm program module obtains the corresponding fitting function z=f(x, p, ..., q) according to the algorithm principle, and outputs the test results z ₁ , z ₂ , ... z _n ;

Step 4: z ₁ , z ₂ , ... z _n and test data y ₁ , y ₂ , ... y _n are input into the mean square error calculation program module to calculate the mean square error E:

Step 5, when the E value meets the error requirement, the least squares method results z ₁ , z ₂ , ... z _n are directly output through the computer output unit; if the error requirement is not met, E is fed back to the clustering algorithm program module to re-correct the required range of J(m _k ). If the E value is large, the applicable range of J(m _k ) is reduced and the elimination points are increased; if the E value is small, the applicable range of J(m _k ) is expanded so that more useful data can enter the least squares method program module for fitting.

Furthermore, the input data (x, p, ..., q) enters the clustering algorithm module through the computer input unit for processing, which specifically includes the following sub-steps:

Step a), clustering each parameter set (x, p, ..., q) separately, i.e. selecting k1, k2, ..., km data points respectively and recording them as [C ₁ , C ₂ , ..., C _ki ], i = 1, ..., m, m represents the number of different input feature parameters of x, p, ..., q, as the center of each of the ki categories;

Step b), for each parameter set (x, p, ..., q), calculate the Euclidean distance from the remaining elements to the center of the ki class, and each data is assigned to the class closest to it. The Euclidean distance calculation formula is:

d _ji =|a _j -C _ki |a∈[x,p,...,q];

i＝1,…,m, represents the number of different input feature parameters; j＝1,…,n, represents the number of sampling points;

Step c), for each parameter set (x, p, …, q), first calculate the mean of all objects in each category as the new cluster center r _ki according to the clustering results, and then calculate the sum of the squares of the distances of all objects from the cluster center of their category, that is, the J value:

Step d), calculate the effect evaluation function:

in:

d _kj = 1, a _j ∈ _{r ki} ;

d _kj = 0, a _j does not belong to r _ki ;

kx＝[k1,k2,...km];

Step f), determine whether J(m _k ) meets the user's requirements, if so, the data is output from the module; if not, sort according to D = || a _j - r _ki || ² , remove the data points with a large D = || a _j - r _ki || ² , and then re-enter step b) until J(m _k ) meets the requirements and the data is output from the clustering algorithm module.

Furthermore, in step 1, the collected data of the corresponding scene is input for the application scenario requiring data fitting; the corresponding scene includes machine learning, digital twin system, artificial intelligence and machine vision.

Further, in step a), the ki data points [C ₁ , C ₂ , …, C _ki ] are not required to be from sample data points.

Further, in step f), the user requirement may set a requirement range.

A data processing device based on clustering least squares method comprises a processor and a memory. A program is stored in the memory. When the program is loaded by the processor, the data processing method based on clustering least squares method as described in any one of the above items is executed.

The beneficial effects of the present invention include:

The method of the present invention is simple and highly feasible. The least squares algorithm and the clustering algorithm are closely combined by introducing the mechanism of error feedback control, and the output result of the least squares algorithm is closer to the actual demand by cyclically adjusting the J(m _k ) value range, so that the final data processing can be more accurate.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

Figure 1 is a least squares fitting curve diagram;

FIG2 is a schematic diagram of a method according to an embodiment of the present invention;

Figure 3 is a UVPA noise coefficient test data curve;

Figure 4 shows the first output result of the least squares method;

FIG5 is a schematic diagram of using a clustering algorithm to eliminate points with large errors in UVPA noise coefficient test data;

Figure 6 is the second output result of the least squares method;

Figure 7 shows the final output result of the least squares method;

Figure 8 is the mean square error curve after 7 iterations.

DETAILED DESCRIPTION

All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and/or steps, can be combined and/or expanded or replaced in any manner.

In view of the problems in the background, the inventors of the present invention further discovered after creative thinking:

K-means clustering algorithm is a classic and effective algorithm in cluster analysis. This algorithm is based on the idea of partitioning. Assume that the original data set is d-dimensional, and the original data set is X = { _xi | _xi∈Rd ^, i = 1, 2, ..., N}. Through cluster analysis, the original data set is clustered into k classes, and the k classes are denoted by _mk (k = 1, 2, ..., K). Each class has a cluster center, and the cluster center of each class is denoted by _ck (k = 1, 2, ..., K). Then the sum of the squares of the distances from the objects in each class to the cluster center is:

Cluster analysis is to minimize the sum of squares of distances from objects of all classes to the cluster center, so the function for effect evaluation is:

in:

_dki ＝ ₁ ， _xi∈ci

d _ki = 0, _xi is other

By preprocessing the original data set through the K-means clustering algorithm process, the data will be divided into k categories, and each data will be assigned to the class closest to it. Then, we can calculate the Euclidean distance from each object to the cluster center and select the largest distance as the outlier. Therefore, the data is clustered into classes through the clustering algorithm, and those data that do not belong to the task class are outliers.

Therefore, the scheme of the present invention essentially proposes a novel data processing method based on the least squares algorithm framework of the clustering algorithm, and the method is implemented based on a computer hardware running program. As shown in Figure 2, the method deeply integrates the clustering algorithm and the least squares algorithm based on a computer program, and controls the J(m _k ) value selection range of the clustering algorithm program module through the error feedback between the output result of the least squares algorithm program module and the actual data, thereby using the clustering algorithm program module to eliminate abnormal data, so that the least squares fitting result is closer to the target value, thereby improving the accuracy of the method.

The solution of the present invention can innovatively add a feedback mechanism to implement the solution based on the clustering algorithm plus the least squares method. The overall workflow is designed as follows:

Step 1, for application scenarios that require data fitting, etc., assume that the input data of n samples is (x1, p1, ..., q1), (x2, p2, ..., q2), ..., (xn, pn, ..., qn), and the test output data is y1, y2, ...yn, where n = [1, 2, ...] represents the sampling points, and x, p, ..., q represents m different input feature parameters. In order to obtain an accurate fitting curve for predicting the output of other sampling points, it is hoped to find a suitable function expression so that the input x, p, ..., q, the output corresponds to the test output data y1, y2, ...yn;

Step 2: Input data (x, p, ..., q) into the clustering algorithm program module. The workflow of the clustering algorithm program module is as follows:

Step a), clustering each parameter set (x, p, …, q) separately, i.e. selecting k1, k2, …, km data points denoted as [C ₁ , C ₂ , …, C _ki ] (i=1, …, m), where m represents the number of different input feature parameters of x, p, …, q, as the centers of the ki categories, wherein the ki data points [C ₁ , C ₂ , …, C _ki ] are not required to come from sample data points;

Step b), for each parameter set (x, p, ..., q), calculate the Euclidean distance from the remaining elements to the center of the ki class, and each data is assigned to the class closest to it. The Euclidean distance calculation formula is:

d _ji =|a _j -C _ki |a∈[x,p,...,q];

i=1,…,m (number of different input feature parameters), j=1,…,n (number of sampling points).

Step c), for each parameter set (x, p, …, q), first calculate the mean of all objects in each category as the new cluster center r _ki according to the clustering results, and then calculate the sum of the squares of the distances of all objects from the cluster center of their category, that is, the J value:

Step d), calculate the effect evaluation function

in:

d _kj = 1, a _j ∈ _{r ki} ;

d _kj = 0, a _j does not belong to r _ki ;

kx＝[k1,k2,...km];

Step f), determine whether J(m _k ) meets the user's requirements (the user can set the requirement range), if so, the data is output from the module; if not, sort according to D = || a _j - _{r ki} || ² , remove the data points with a large D = || a _j - r _ki || ² , and then re-enter step b), until J(m _k ) meets the requirements, the data is output from the clustering algorithm module;

Step 3, the clustering algorithm module outputs data to the least squares algorithm module, and the least squares method obtains the corresponding fitting function z=f(x, p, ..., q) according to the algorithm principle, and the test results z ₁ , z ₂ , ... z _n are output;

Step 4: z ₁ , z ₂ , ... z _n and test data y ₁ , y ₂ , ... y _n are input into the mean square error calculation module to calculate the mean square error E:

Step 5, when the E value meets the error requirement, the least squares method results z ₁ , z ₂ , ... z _n are directly output; if it does not meet the error requirement, E is fed back to the clustering algorithm module to re-correct the required range of J(m _k ). If the E value is large, the applicable range of J(m _k ) is reduced and the elimination points are increased; if the E value is small, the applicable range of J(m _k ) is expanded so that more useful data can enter the least squares module for fitting.

Since aviation integrated electronic information equipment is integrated equipment, and different functional units are realized by multiple modules, the core indicators of most functional units of the whole machine are jointly determined by the indicators of multiple cascade modules. Due to the complex indicator parameters of each module, there are large differences in the working mode of modules in different functional units, and the influence of multiple factors such as the working mode, working environment, and test instruments, the mapping relationship between core functional indicators and module indicators is complex and diverse. Therefore, the relationship between core functional indicators and module indicators is deeply analyzed, and their physical characteristics are studied and discussed. Functions are constructed to calculate core functional indicators through module indicators. On the basis of existing knowledge, there is a further understanding of the connection between modules. Combined with the characteristics of different modules, further exploration is carried out between core indicators and module indicators.

The following is further explained in conjunction with the accompanying drawings and example scenario data. For the sensitivity test data, the collected data is shown in Table 1. The target data of the fitting curve is sensitivity (represented by y), and the input test data is frequency (represented by a), UVPA gain data (represented by b), UVPA noise coefficient data (represented by c), and UVRT noise coefficient data (represented by d).

Table 1 Correspondence between UV function sensitivity index and module index

Regarding the clustering algorithm, the input test data UVPA gain data (represented by b), UVPA noise coefficient data (represented by c) and UVRT noise coefficient data (represented by d) are clustered respectively, as shown in FIG3 for the UVPA noise coefficient data. In the first calculation, the point in the circle that is farthest from the cluster center and has the largest value of D = ||a _j - _{r ki} || ² is removed.

Regarding the least squares method, FIG4 shows the least squares method output result and compares the output result with the test result.

Regarding the mean square error calculation, the mean square error is calculated to be 0.2709. If it does not meet the requirements, the error is fed back to the clustering algorithm module.

Regarding the clustering algorithm, the clustering algorithm will remove the second point that is farthest from the cluster center and has the largest value of D = || a _j - _{r ki} || ^2. The data is sent to the least squares module. As shown in Figure 5.

Regarding the least squares method, FIG6 shows the second calculation result of the least squares method and compares the output result with the test result.

Regarding the mean square error calculation, the mean square error is calculated to be 0.113. If it does not meet the requirements, the error is fed back to the clustering algorithm module.

After 7 iterations, the least squares method output result is shown in Figure 7, which is consistent with the measured data. The mean square error curve is shown in Figure 8, indicating that the calculated results are getting closer and closer to the measured data, and the model is becoming more and more accurate.

The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.

In addition to the above examples, those skilled in the art may obtain other embodiments based on the above disclosure or by using the knowledge or technology in the relevant field to make changes. The features of each embodiment may be interchangeable or replaced. The changes and modifications made by those skilled in the art do not depart from the spirit and scope of the present invention and should be within the scope of protection of the claims attached to the present invention.

Claims (7)

1. The data processing method based on the clustering least square method is characterized by comprising the following steps of:

The clustering algorithm and the least square algorithm are fused by using a computer program, and the J (m _k) value selection range of the clustering algorithm program module is controlled by the error feedback of the output result of the least square algorithm program module and the actual data in the computer, so that the clustering algorithm program module is used for eliminating abnormal data, and the fitting result of the least square method is closer to the target value; wherein J (m _k) represents the sum of squares of distances from the object m _k of class k to the cluster center; the data includes avionics equipment test data, and the avionics equipment test data is input to a computer running the program through a computer input unit.

2. The data processing method based on the clustering least square method according to claim 1, wherein the clustering algorithm and the least square algorithm are fused by using a computer program, and the error feedback of the output result of the least square algorithm program module and the actual data controls the J (m _k) value selection range of the clustering algorithm program module in the computer, so that the abnormal data is removed by using the clustering algorithm program module, and the fitting result of the least square method is closer to the target value, and the method specifically comprises the following sub-steps:

Step 1, assuming that the input data of n samples is (x 1, p1, …, q 1), (x 2, p2, …, q 2), …, (xn, pn, …, qn), and the test output data is y1, y2, … yn, where n= [1,2, … ] represents the sampling point; x, p, …, q represent m different input characteristic parameters;

Step 2, inputting data (x, p, …, q) into a clustering algorithm program module for processing through a computer input unit;

Step 3, the output data of the clustering algorithm program module enters a least square algorithm program module, the least square algorithm program module obtains a corresponding fitting function z=f (x, p, q) according to an algorithm principle, and a test result z ₁,z₂,...z_n is output;

Step 4, z ₁,z₂,...z_n and test data y ₁,y₂,...y_n are input into a mean square error calculation program module to calculate a mean square error E:

Step 5, when the E value meets the error requirement, directly outputting a result z ₁,z₂,...z_n of the least square method through a computer output unit; if the error requirement is not met, feeding E back to a clustering algorithm program module, revising the requirement range of J (m _k), if the E value is larger, reducing the application range of J (m _k), and increasing the reject point; if the E value is smaller, the application range of J (m _k) is expanded, so that more useful data enter a least square program module for fitting.

3. The data processing method based on the clustering least square method according to claim 2, wherein the input data (x, p, …, q) is processed by a clustering algorithm module through a computer input unit, and specifically comprises the following sub-steps:

step a), clustering each parameter set of (x, p, …, q) respectively, namely respectively selecting k1, k2, …, and marking km data points as [ C ₁,C₂,…,C_ki ], wherein i=1, …, m, m represents the number of different input characteristic parameters of x, p, …, q, and taking the number as the center of each of ki categories;

step b), for each parameter set of (x, p, …, q), respectively calculating the Euclidean distance from the rest element to the center of the ki class, wherein each data is classified into the nearest class, and the Euclidean distance calculation formula is as follows:

d_ji＝|a_j-C_ki|a∈[x,p,...,q]；

i=1, …, m, representing the number of different input characteristic parameters; j=1, …, n, representing the number of sampling points;

Step c), for each parameter set (x, p, …, q), respectively according to the clustering result, firstly calculating the average value of all objects in each class as a new clustering center r _ki, and then calculating the square sum of the distances of all objects from the clustering center of the class in which the objects are located, namely J value:

Step d), calculating a function of the effect judgment:

wherein:

d_kj＝1,a_j∈r_ki；

d _kj＝0,a_j does not belong to r _ki;

kx＝[k1,k2,...km]；

Step f), judging whether J (m _k) meets the user requirement or not, and outputting data from the module if the J (m _k) meets the user requirement; if not, sorting according to D= |a _j-r_ki||², eliminating the data points with the size of D= |a _j-r_ki||², and then re-entering the step b) until J (m _k) meets the requirement, and outputting the data from the clustering algorithm module.

4. The data processing method based on the cluster least square method according to claim 2, wherein in step 1, acquisition data of a corresponding scene is input for an application scene requiring data fitting; the corresponding scenarios include machine learning, digital twinning systems, artificial intelligence, and machine vision.

5. A data processing method based on a cluster least squares method according to claim 3, wherein in step a) ki data point [ C ₁,C₂,…,C_ki ] is not required from sample data points.

6. The data processing method based on the cluster least square method according to claim 2, wherein in step f), the user request can set a request range.

7. A data processing apparatus based on a cluster least square method, comprising a processor and a memory, wherein a program is stored in the memory, and the data processing method based on the cluster least square method according to any one of claims 1 to 6 is executed when the program is loaded by the processor.