CN115618265A - Data integration method and system based on big data and edge computing - Google Patents

Abstract

The invention relates to a data processing technology, and discloses a data integration method and a system based on big data and edge calculation, wherein the data integration method comprises the following steps: acquiring edge nodes of a client, and collecting data by using the edge nodes to obtain data to be processed; carrying out standardization processing on the data to be processed to obtain standard data; performing feature analysis on the standard data to obtain an optimal feature subset; performing integrated trust calculation on the optimal feature subset to obtain a trust interval; and performing feature synthesis on the standard data according to the trust interval to obtain integrated data. The invention can improve the efficiency of data integration.

Description

Data integration method and system based on big data and edge computing

technical field

The present invention relates to the technical field of data processing, in particular to a data integration method and system based on big data and edge computing.

Background technique

In the initial process of enterprise informatization construction, due to the lack of a unified plan for informatization construction, an information system constructed by different core technologies was established. Due to the huge data types and data scale of the system, enterprise information data cannot be effectively circulated and utilized. With the emergence of enterprise information integration needs, data integration, as a concept and method of resource integration, is constantly developing and improving. Most of the existing technologies use the traditional cloud computing mode for data integration. When the cloud computing mode performs data integration, multiple data calculation programs need to be decomposed and then analyzed and processed. Therefore, there will be higher requirements for resource allocation, and the big data Consolidation often overloads traditional cloud computing models, resulting in slow data consolidation. To sum up, the prior art has the problem of low data integration efficiency.

Contents of the invention

The present invention provides a data integration method and system based on big data and edge computing, and its main purpose is to solve the problem of low data integration efficiency.

In order to achieve the above purpose, the present invention provides a data integration method based on big data and edge computing, including:

Obtaining an edge node of the client, using the edge node to collect data, and obtaining data to be processed;

Standardizing the data to be processed to obtain standard data;

Perform feature analysis on the standard data to obtain an optimal feature subset;

performing integrated trust calculation on the optimal feature subset to obtain a trust interval;

Perform feature synthesis on the standard data according to the trust interval to obtain integrated data.

Optionally, the collecting data by using the edge node to obtain the data to be processed includes:

setting an edge node device at the edge node to obtain a plurality of edge data sensors;

Using the edge data sensor to connect the edge node and the cloud center to obtain a data transmission channel;

The edge data sensor uses the data transmission channel to receive target data;

storing the target data in the edge node to obtain data to be processed.

Optionally, the standardization processing of the data to be processed to obtain standard data includes:

judging the existence form of the data to be processed;

If it is determined that the existence form of the data to be processed is an array, then execute the preset conversion scheme 1 to obtain the first data;

If it is determined that the existence form of the data to be processed is a matrix, then execute the second preset conversion scheme to obtain the second data;

Summarizing the first data and the second data to obtain standard data.

Optionally, the execution of the preset conversion scheme one to obtain the first data includes:

performing multidimensional decomposition on the data to be processed to obtain multiple sample vectors;

calculating the mean and standard deviation of a plurality of said sample vectors;

The plurality of sample vectors are calculated according to the mean value and the standard deviation to obtain the first data.

Using the following formula to calculate a plurality of the sample vectors according to the mean and standard deviation:

Z=(X-M)/S

Wherein, Z represents the first data; X represents the sample vector; M represents the mean value of the sample vector; S represents the standard deviation of the sample vector.

Optionally, the execution of the second preset conversion scheme to obtain the second data includes:

Performing column vector extraction on the data to be processed to obtain multiple column vectors;

Calculate the mean and standard deviation of the column vectors, respectively;

Utilize the following formula to calculate the mean and standard deviation of the column vectors in the feature matrix respectively:

表示为所述待处理数据中的第i行第j列的标准数据值(i＝1,2,,…,n)；H_ij表示为所述待处理数据中的第i行第j列的向量；

表示为所述待处理数据中第j列的均值；L_j表示为所述待处理数据中第j列的标准差；n表示为所述待处理数据中的行数；in,

Expressed as the standard data value of row i and column j in the data to be processed (i=1,2,,...,n); H _ij is represented as the value of row i and column j in the data to be processed vector;

Expressed as the mean value of the j column in the data to be processed; Lj is represented as the standard deviation of the _j column in the data to be processed; n is represented as the number of rows in the data to be processed;

Processing the column vectors one by one by using the mean value and standard deviation of the column vectors to obtain second data.

Optionally, performing integrated trust calculation on the optimal feature subset to obtain a trust interval includes:

calculating a belief function value for the optimal feature subset;

Use the following formula to calculate the trust function value of the optimal feature subset:

B(A)×2 ^v →[O,11

Among them, B(A) represents the trust function value of the optimal feature subset A; v represents the model framework of the preset feature subset; m(Q) represents all of the optimal feature subset A The probability function of the subset Q;

calculating a likelihood function value for the optimal feature subset based on the belief function value;

The likelihood function value of the optimal feature subset is calculated using the following formula:

P(A)＝1-B(A)＝∑Q∩A≠φm(Q)

Among them, P(A) is expressed as the likelihood function value of the optimal feature subset A; B(A) is expressed as the trust function value of the optimal feature subset A; m(Q) is expressed as the optimal feature subset A The probability function of all subsets Q of the superior feature subset A;

The trust function value and the likelihood function value are integrated and expressed to obtain a trust interval.

Optionally, performing feature analysis on the standard data to obtain an optimal feature subset, including:

performing feature extraction on the standard data to obtain a feature set;

A weight evaluation is performed on the feature set, and an optimal feature subset is obtained according to a result of the weight evaluation.

Optionally, performing a weight evaluation on the feature set, and obtaining an optimal feature subset according to a result of the weight evaluation includes:

Performing weight calculation on the feature set to obtain the feature weight of the feature set;

calculating the dimensional difference degree of the feature set according to the feature weight;

Use the following formula to calculate the dimensional difference degree of the feature set:

Among them, diff(a ₁ ,a ₂ ,i) is expressed as the dimensional difference between feature data a ₁ and feature data a ₂ in the feature set on the i-th dimension; a _1i and a _2i are expressed as individual data in The difference value on the i-th dimension; max(f _i ) represents the maximum weight value of the feature weight corresponding to the feature data contained in the i-th dimension; min(f _i ) represents the feature data contained in the i-th dimension The minimum weight value of the corresponding feature weight;

The feature set is adjusted according to the dimension difference to obtain an optimal feature subset.

Optionally, performing feature synthesis on the standard data according to the trust interval to obtain integrated data includes:

Using the trust interval as a constraint condition to filter the standard data to obtain valid data;

The effective data is combined according to a preset synthesis rule to obtain integrated data.

In order to solve the above problems, the present invention also provides a data integration system based on big data and edge computing, said system comprising:

The data collection module to be processed is used to obtain the edge node of the client, and collect data by using the edge node to obtain the data to be processed;

a standardized processing module, configured to perform standardized processing on the data to be processed to obtain standard data;

A feature subset generating module, configured to perform feature analysis on the standard data to obtain an optimal feature subset;

A trust calculation module, configured to perform integrated trust calculation on the optimal feature subset to obtain a trust interval;

An integrated data generating module, configured to perform feature synthesis on the standard data according to the trust interval to obtain integrated data.

The embodiment of the present invention proposes a data integration method and system based on big data and edge computing. By using edge nodes to collect data, the data generated by the data source does not need to be transmitted to the cloud data center for processing, but can be processed on the client side nearby. The edge side completes data analysis and processing, which improves the efficiency of data transmission; standard data is obtained by standardizing the data to be processed, which eliminates the influence of the variation and numerical value of the data to be processed, and improves the accuracy of data processing; The feature analysis of the data obtains the optimal feature subset, which avoids high-dimensional data processing, reduces the number of features and data processing time, and improves the efficiency of data integration. Therefore, the data integration method, system, electronic device, and computer-readable storage medium proposed by the present invention based on big data and edge computing can solve the problem of low efficiency in data integration.

Description of drawings

FIG. 1 is a schematic flow diagram of a data integration method based on big data and edge computing provided by an embodiment of the present invention;

FIG. 2 is a schematic flow diagram of using the edge node to collect data and obtain data to be processed according to an embodiment of the present invention;

FIG. 3 is a schematic flow diagram of performing integrated trust calculation on the optimal feature subset to obtain a trust interval according to an embodiment of the present invention;

FIG. 4 is a functional block diagram of a data integration system based on big data and edge computing provided by an embodiment of the present invention;

The realization of the purpose of the present invention, functional characteristics and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

detailed description

It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The embodiment of the present application provides a data integration method based on big data and edge computing. The execution subject of the data integration method based on big data and edge computing includes but is not limited to at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the data integration method based on big data and edge computing can be executed by software or hardware installed on the terminal device or server device, and the software can be a block chain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like. The server can be an independent server, or it can provide cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (ContentDelivery Network) , CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

Referring to FIG. 1 , it is a schematic flowchart of a data integration method based on big data and edge computing provided by an embodiment of the present invention. In this embodiment, the data integration method based on big data and edge computing includes:

S1. Obtain an edge node of the client, use the edge node to collect data, and obtain data to be processed;

In the embodiment of the present invention, the client includes a router, through which the data transmission between the cloud and the data source can be realized; wherein, the edge is a relative concept, mainly referring to the path between the data source and the cloud computing center. Any computing, storage and network-related resources between, so the edge node refers to any node between the source of data generation and the cloud center with computing resources and network resources. For example, a mobile phone can be an edge node between a person and a cloud center, and a gateway can be an edge node between a smart home and a cloud center.

Please refer to FIG. 2, in the embodiment of the present invention, the use of the edge node to collect data to obtain the data to be processed includes:

S21. Setting an edge node device at the edge node to obtain a plurality of edge data sensors;

S22. Using the edge data sensor to connect the edge node and the cloud center to obtain a data transmission channel;

S23. The edge data sensor uses the data transmission channel to receive target data;

S24. Store the target data in the edge node to obtain data to be processed.

In the embodiment of the present invention, the edge data sensor is used to perceive the state of the edge node and collect the data information required by the edge node; the cloud center ensures the communication between the edge data sensor and the target data Interaction specifications and protocols, the existence of the cloud center can be a database, a file system, etc., the data source defines the location information and also encapsulates the interface connecting the edge data sensor to the data source; The data transmission channel is used to transmit the data generated by the data source to the edge data sensor, through which the key status in the data can be initially screened, for example, for blank values in the data Delete, which can reduce the time of data processing.

S2. Standardize the data to be processed to obtain standard data;

In the embodiment of the present invention, the standardization processing of the data to be processed to obtain standard data includes:

judging the existence form of the data to be processed;

If it is determined that the existence form of the data to be processed is an array, then execute the preset conversion scheme 1 to obtain the first data;

If it is determined that the existence form of the data to be processed is a matrix, then execute the second preset conversion scheme to obtain the second data;

Summarizing the first data and the second data to obtain standard data.

In the embodiment of the present invention, since the data to be processed is a large amount of unstructured or structured data from various sources, there are multiple types of data sources. In order to improve the accuracy of data standardization, the Different transformation schemes corresponding to the source data format, thereby ensuring that different source data formats can be standardized.

In the embodiment of the present invention, the execution of the preset conversion scheme 1 to obtain the first data includes:

performing multidimensional decomposition on the data to be processed to obtain multiple sample vectors;

calculating the mean and standard deviation of a plurality of said sample vectors;

The plurality of sample vectors are calculated according to the mean value and the standard deviation to obtain the first data.

In the embodiment of the present invention, the data to be processed may be a multidimensional array, for example, the number to be processed is X=[X ₁ , X ₂ , X ₃ ,...,X _n ] (n is the number of samples), where, X ₁ , X ₂ , X ₃ ,…,X _n are multi-dimensional sample vectors, for example, vector X ₁ can be X ₁ =[X ₁₁ ,X ₂₁ ,…,X _d1 ] (d is the number of dimensions), equivalent to The multidimensional array X contains n sample vectors, and each sample vector contains d dimensions.

In the embodiment of the present invention, the following formula can be used to calculate a plurality of said sample vectors according to said mean value and standard deviation:

Z=(X-M)/S

Wherein, Z represents the first data; X represents the sample vector; M represents the mean value of the sample vector; S represents the standard deviation of the sample vector.

In the embodiment of the present invention, the execution of the second preset conversion scheme to obtain the second data includes:

Performing column vector extraction on the data to be processed to obtain multiple column vectors;

Compute the mean and standard deviation of the column vectors separately

Processing the column vectors one by one by using the mean value and standard deviation of the column vectors to obtain second data.

In the embodiment of the present invention, the mean value and standard deviation of the column vectors in the feature matrix can be calculated respectively by using the following formula:

表示为所述待处理数据中的第i行第j列的标准数据值(i＝1,2,,…,n)；H_ij表示为所述待处理数据中的第i行第j列的向量；

表示为所述待处理数据中第j列的均值；L_j表示为所述待处理数据中第j列的标准差；n表示为所述待处理数据中的行数。In the embodiment of the present invention,

Expressed as the standard data value of row i and column j in the data to be processed (i=1,2,,...,n); H _ij is represented as the value of row i and column j in the data to be processed vector;

Expressed as the mean value of column _j in the data to be processed; Lj is represented as the standard deviation of column j in the data to be processed; n is represented as the number of rows in the data to be processed.

In the embodiment of the present invention, the data to be processed is a multi-dimensional matrix, and extracting the column vector of the data to be processed can perform a dimensionality reduction operation on the data to be processed, simplifying the complexity of data processing; the second The data is a standard matrix where each column satisfies

S3. Perform feature analysis on the standard data to obtain an optimal feature subset;

In the embodiment of the present invention, performing feature analysis on the standard data to obtain an optimal feature subset includes:

performing feature extraction on the standard data to obtain a feature set;

A weight evaluation is performed on the feature set, and an optimal feature subset is obtained according to a result of the weight evaluation.

In the embodiment of the present invention, the feature extraction can use a classifier to perform online feature extraction on the standard data, for example, assuming that the set of the standard data is D=(b _t , c _t )(t=1,...,T, T is the number of elements; b _t and c _t are multi-dimensional vectors in the standard data, which are classified in turn according to the order of the elements, and the binary classifier can be used to pass the linear function sgn(w _t ^T x _t ) (w _t is the classification implement) to obtain the feature set; the set of the standard data can also be D=(b _t , c _t , e _t ), (b _t , c _t and e _t are all multidimensional vectors in the standard data ), can use three classifiers to perform feature extraction through linear functions to obtain feature sets, and so on, the classifier is determined according to the number of categories of multidimensional vectors in the set of standard data.

In the embodiment of the present invention, the weight evaluation of the feature set is performed, and the optimal feature subset is obtained according to the result of the weight evaluation, including:

Performing weight calculation on the feature set to obtain the feature weight of the feature set;

calculating the dimensional difference degree of the feature set according to the feature weight;

The feature set is adjusted according to the dimension difference to obtain an optimal feature subset.

In the embodiment of the present invention, the weight calculation can use the prioritization graph method to calculate according to the relative size of the data in the feature set, for example, the number 0 means relatively unimportant, the number 1 means relatively more important, and the number 0.5 means equally important , if two sets in the feature sets are equally important, record 0.5 points for the two sets respectively, and after all feature sets have been compared for a round, summarize the scoring situation of each feature set, A feature weight corresponding to each feature set is obtained.

In the embodiment of the present invention, the dimension difference degree of the feature set can be calculated by using the following formula:

Among them, diff(a ₁ ,a ₂ ,i) is expressed as the dimensional difference between feature data a ₁ and feature data a ₂ in the feature set on the i-th dimension; a _1i and a _2i are expressed as individual data in The difference value on the i-th dimension; max(f _i ) represents the maximum weight value of the feature weight corresponding to the feature data contained in the i-th dimension; min(f _i ) represents the feature data contained in the i-th dimension The minimum weight value of the corresponding feature weight.

In the embodiment of the present invention, the adjustment of the feature set is to filter according to the numerical value of the dimensional difference degree, and retain the feature set corresponding to the dimensional difference degree smaller than the preset dimensional difference degree threshold, and the dimensional difference degree The threshold may be 0.2; the optimal subset is composed of the last k feature sets ranked by the degree of difference in the dimensions, and k generally takes a value of 6.

S4. Perform integrated trust calculation on the optimal feature subset to obtain a trust interval;

Please refer to FIG. 3, in the embodiment of the present invention, the integrated trust calculation is performed on the optimal feature subset to obtain a trust interval, including:

S31. Calculate the trust function value of the optimal feature subset;

S32. Calculate the likelihood function value of the optimal feature subset according to the trust function value;

S33. Integrate and express the trust function value and the likelihood function value to obtain a trust interval. In the embodiment of the present invention, the following formula can be used to calculate the trust function value of the optimal feature subset:

B(A)×2 ^v →[0,1]

Among them, B(A) represents the trust function value of the optimal feature subset A; v represents the model framework of the preset feature subset; m(Q) represents all of the optimal feature subset A Probability function for subset Q.

Specifically, the probability function is the probability of a discrete random variable at a certain value. For example, the probability of heads occurring after tossing an even coin 3 times is calculated, and 3 is the feature value.

In the embodiment of the present invention, the likelihood function value of the optimal feature subset can be calculated using the following formula:

P(A)＝1-B(A)＝∑Q∩A≠φm(Q)

Among them, P(A) is expressed as the likelihood function value of the optimal feature subset A; B(A) is expressed as the trust function value of the optimal feature subset A; m(Q) is expressed as the optimal feature subset A The probability function of all subsets Q of the superior feature subset A.

Specifically, the likelihood function represents the degree of confidence of not denying A, which is the sum of the basic probability distributions of all the feature subsets intersecting with A.

In the embodiment of the present invention, the trust interval may be expressed as μ(A)=[B(A), P(A)]; wherein, μ(A) is expressed as the trust interval of the optimal feature subset A; B(A) is expressed as the likelihood function value of the optimal feature subset A, which is the lower limit of the confidence interval; P(A) is expressed as the arbitrary function value of the optimal feature subset A, which is the The upper limit of the trust interval; for example, the trust interval is assumed to be (0.25,0.85), which means that A is true with a trust degree of 0.25, A is false with a trust degree of 0.15, and A is uncertain with a trust degree of 0.6.

S5. Perform feature synthesis on the standard data according to the trust interval to obtain integrated data.

In the embodiment of the present invention, the feature synthesis of the standard data according to the trust interval to obtain the integrated data includes:

Using the trust interval as a constraint condition to filter the standard data to obtain valid data;

The effective data is combined according to a preset synthesis rule to obtain integrated data.

In the embodiment of the present invention, screening the standard data is to screen the feature subsets whose trust function value of the feature subset corresponding to the standard data is in the trust interval, for example, the trust function value of the feature subset R is 0.56 , the trust interval is (0.25,0.85), and the trust function value of the feature subset R is located in this interval, then the standard data corresponding to this feature subset R is reserved; the composition rule can use the D-S evidence theory Combination rules, which solve the information fusion problem by integrating characteristic information and using upper and lower bound probabilities (the confidence interval), to obtain the final result, that is, data integration, are a kind of combination rules mainly aimed at multi-source information.

The present invention proposes a data integration method and system based on big data and edge computing. By using edge nodes to collect data, the data generated by the data source does not need to be transmitted to the cloud data center for processing, but is nearby at the edge of the client. Completing data analysis and processing improves the efficiency of data transmission; standard data is obtained by standardizing the data to be processed, which eliminates the influence of the variation and numerical value of the data to be processed and improves the accuracy of data processing; Feature analysis, to obtain the optimal feature subset, avoid high-dimensional data processing, reduce the number of features and data processing time, and improve the efficiency of data integration. Therefore, the data integration method based on big data and edge computing proposed by the present invention can solve the problem of low data integration efficiency.

As shown in FIG. 4 , it is a functional block diagram of a data integration system based on big data and edge computing provided by an embodiment of the present invention.

The data integration system 100 based on big data and edge computing in the present invention can be installed in electronic devices. According to the realized functions, the data integration system 100 based on big data and edge computing may include a data collection module 101 to be processed, a standardized processing module 102, a feature subset generation module 103, a trust calculation module 104 and an integrated data generation module 105. The module in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of the electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The data-to-be-processed collection module 101 is configured to acquire edge nodes of the client, and use the edge nodes to collect data to obtain data to be processed;

The standardization processing module 102 is configured to perform standardization processing on the data to be processed to obtain standard data;

The feature subset generating module 103 is configured to perform feature analysis on the standard data to obtain an optimal feature subset;

The trust calculation module 104 is configured to perform integrated trust calculation on the optimal feature subset to obtain a trust interval;

The integrated data generation module 105 is configured to perform feature synthesis on the standard data according to the trust interval to obtain integrated data.

In detail, each module described in the data integration system 100 based on big data and edge computing described in the embodiment of the present invention uses the same method as the data integration method based on big data and edge computing described in the accompanying drawings. Technical means, and can produce the same technical effect, will not repeat them here.

This embodiment also provides an electronic device, which may include a processor, a memory, a communication bus, and a communication interface, and may also include a computer program stored in the memory and operable on the processor, such as A protection upgrade program based on information security big data.

Wherein, the processor may be composed of integrated circuits in some embodiments, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions, including one or more A combination of a central processing unit (Central Processing Unit, CPU), a microprocessor, a digital processing chip, a graphics processor, and various control chips, etc. The processor is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, and runs or executes programs or modules stored in the memory 11 (for example, executes based on data integration programs for big data and edge computing, etc.), and call data stored in the memory to perform various functions of electronic devices and process data.

The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card type memory (for example: SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage may be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in mobile hard disk equipped on the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc. Further, the memory may also include both an internal storage unit of the electronic device and an external storage device. The memory can not only be used to store application software and various data installed in electronic equipment, such as codes of data integration programs based on big data and edge computing, but also can be used to temporarily store data that has been output or will be output.

The communication bus may be a Peripheral Component Interconnect (PCI for short) bus or an Extended Industry Standard Architecture (EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to implement communication between the memory and at least one processor.

The communication interface is used for communication between the electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which are generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display) or an input unit (such as a keyboard (Keyboard)). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, Organic Light-Emitting Diode) touch panel, and the like. Wherein, the display may also be properly referred to as a display screen or a display unit, and is used for displaying information processed in the electronic device and for displaying a visualized user interface.

The data integration program based on big data and edge computing stored in the memory in the electronic device is a combination of multiple instructions. When running in the processor, the above-mentioned big data and edge computing-based data integration program can be realized. The steps of the data integration method.

Specifically, for the specific implementation method of the above instructions by the processor, reference may be made to the description of relevant steps in the corresponding embodiments in the accompanying drawings, and details are not repeated here.

Furthermore, if the integrated module/unit of the electronic device is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or system capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory).

The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of the above-mentioned data integration method based on big data and edge computing are realized.

These program codes can also be loaded into a computer or other programmable data processing device, so that a series of operational steps are executed on the computer or other programmable device to produce computer-implemented processing, thereby executing instructions on the computer or other programmable device Provides the steps to implement a function specified in a flowchart flow or flows.

Storage media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media may include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM) ), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cartridges Magnetic tape, disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may physically exist separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the invention is not limited to the details of the above-described exemplary embodiments, but that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention.

Accordingly, the embodiments should be regarded in all points of view as exemplary and not restrictive, the scope of the invention being defined by the appended claims rather than the foregoing description, and it is therefore intended that the scope of the invention be defined by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalents of the elements are embraced in the present invention. Any reference sign in a claim should not be construed as limiting the claim concerned.

The embodiments of the present application may acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. .

In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or systems stated in the system claims may also be realized by one unit or system through software or hardware. The terms first, second, etc. are used to denote names and do not imply any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention without limitation. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent replacements can be made without departing from the spirit and scope of the technical solutions of the present invention.

Claims (10)

1. A data integration method based on big data and edge calculation is characterized by comprising the following steps:

acquiring edge nodes of a client, and collecting data by using the edge nodes to obtain data to be processed;

carrying out standardization processing on the data to be processed to obtain standard data;

performing feature analysis on the standard data to obtain an optimal feature subset;

performing integrated trust calculation on the optimal feature subset to obtain a trust interval;

and performing characteristic synthesis on the standard data according to the trust interval to obtain integrated data.

2. The data integration method based on big data and edge computing as claimed in claim 1, wherein said collecting data by using the edge node to obtain the data to be processed comprises:

edge node equipment is arranged at the edge nodes to obtain a plurality of edge data sensors;

connecting the edge nodes and the cloud center by using the edge data sensor to obtain a data transmission channel;

the edge data sensor receives target data by using the data transmission channel;

and storing the target data to the edge node to obtain data to be processed.

3. The data integration method based on big data and edge calculation according to claim 1, wherein the normalizing the data to be processed to obtain standard data comprises:

judging the existence form of the data to be processed;

if the existence form of the data to be processed is judged to be an array, executing a preset first transformation scheme to obtain first data;

if the existence form of the data to be processed is judged to be a matrix, executing a preset second transformation scheme to obtain second data;

and summarizing the first data and the second data to obtain standard data.

4. The method for data integration based on big data and edge calculation according to claim 3, wherein the step of executing a first predetermined transformation scheme to obtain the first data comprises:

carrying out multidimensional decomposition on the data to be processed to obtain a plurality of sample vectors;

calculating a mean and a standard deviation of a plurality of the sample vectors;

and calculating the plurality of sample vectors according to the mean value and the standard deviation to obtain first data.

Calculating a plurality of said sample vectors from said mean and standard deviation using:

Z＝(X-M)/S

wherein Z is represented as the first data; x is represented as the sample vector; m is expressed as the mean of the sample vectors; s is expressed as the standard deviation of the sample vector.

5. The method for data integration based on big data and edge calculation according to claim 3, wherein the step of executing a second predetermined transformation scheme to obtain second data comprises:

performing column vector extraction on the data to be processed to obtain a plurality of column vectors;

respectively calculating the mean value and the standard deviation of the column vectors;

calculating the mean and standard deviation of the column vectors in the feature matrix using the following equations, respectively:

wherein,

a standard data value (i =1,2, 8230; n) expressed as an ith row and a jth column in the data to be processed; h _ij A vector which is represented as the ith row and the jth column in the data to be processed;

expressed as the mean value of the jth column in the data to be processed; l is _j Expressed as the standard deviation of the jth column in the data to be processed; n is expressed as the number of lines in the data to be processed;

and processing the column vectors one by using the mean value and the standard deviation of the column vectors to obtain second data.

6. The big data and edge computing-based data integration method according to claim 1, wherein the performing integration trust computation on the optimal feature subset to obtain a trust interval comprises:

calculating a belief function value for the optimal feature subset;

calculating a belief function value for the optimal subset of features using the following equation:

B(A)×2 ^v →[0,1]

wherein B (A) is expressed as a belief function value for the optimal feature subset A; v represents a model framework of a preset feature subset; m (Q) is represented as a probability function of the entire subset Q of the optimal feature subset A;

calculating a likelihood function value for the optimal subset of features from the belief function values;

calculating likelihood function values for the optimal subset of features using:

P(A)＝1-B(A)＝∑Q∩A≠φm(Q)

wherein P (A) is expressed as a likelihood function value for the optimal feature subset A; b (A) is expressed as a belief function value for the optimal feature subset A; m (Q) is expressed as a probability function of all subsets Q of the optimal feature subset A;

and integrating and representing the trust function values and the likelihood function values to obtain a trust interval.

7. The big data and edge calculation based data integration method of claim 1, wherein the performing feature analysis on the standard data to obtain an optimal feature subset comprises:

performing feature extraction on the standard data to obtain a feature set;

and performing weight evaluation on the feature set, and obtaining an optimal feature subset according to the result of the weight evaluation.

8. The method for data integration based on big data and edge calculation according to claim 7, wherein the performing weight evaluation on the feature set to obtain an optimal feature subset according to the result of weight evaluation comprises:

carrying out weight calculation on the feature set to obtain the feature weight of the feature set;

calculating the dimension difference degree of the feature set according to the feature weight;

calculating the dimension difference degree of the feature set by using the following formula:

wherein, diff (a) ₁ ,a ₂ I) expressed as feature data a in the feature set ₁ And characteristic data a ₂ The dimension difference degree in the ith dimension; a is a _1i And a _2i Expressed as the difference value of the individual data in the ith dimension; max (f) _i ) The weight maximum value of the characteristic weight corresponding to the characteristic data contained in the ith dimension is represented; min (f) _i ) The weight minimum value is expressed as the weight of the characteristic corresponding to the characteristic data contained in the ith dimension;

and adjusting the feature set according to the dimension difference degree to obtain an optimal feature subset.

9. The big data and edge calculation based data integration method according to any one of claims 1 to 8, wherein the performing feature synthesis on the standard data according to the trust interval to obtain integrated data comprises:

screening the standard data by taking the trust interval as a constraint condition to obtain effective data;

and combining the effective data according to a preset synthesis rule to obtain integrated data.

10. A data consolidation system based on big data and edge computation, the system comprising:

the to-be-processed data collection module is used for acquiring edge nodes of the client and collecting data by utilizing the edge nodes to obtain to-be-processed data;

the standardization processing module is used for standardizing the data to be processed to obtain standard data;

the characteristic subset generating module is used for carrying out characteristic analysis on the standard data to obtain an optimal characteristic subset;

the trust calculation module is used for carrying out integrated trust calculation on the optimal characteristic subset to obtain a trust interval;

and the integrated data generation module is used for carrying out feature synthesis on the standard data according to the trust interval to obtain integrated data.

Images