CN111563436B - Infrared spectrum measuring instrument calibration migration method based on CT-CDD - Google Patents

Abstract

The invention relates to the technical field of migration learning under a machine learning module, and provides a CT-CDD-based infrared spectrum measuring instrument calibration and migration method. First, collect the source domain and target domain data sets {X ^m , y ^m }, {X ^s }, and use the KS algorithm to divide the source domain calibration set

and centralize it; then, calibrate the centralized source domain set

Establish a PLS calibration model; then, calculate the characteristic spectrum ^Tm of the master instrument and the pseudo-characteristic spectrum of the slave instrument

Using OLS and the dataset ^{ T ^m , y ^m } to determine the number of clusters K by cross- ^validation

Cluster separately, for subdatasets

Establish the kth OLS model and calculate the transformation matrix M _k ; finally, predict the substance concentration variable of the measured object set. The invention does not need to use a standard sample to construct a migration model, and can greatly improve the precision and efficiency of the migration calibration of the infrared spectrum measuring instrument.

Description

A method of calibration and migration of infrared spectroscopy measuring instruments based on CT-CDD

technical field

The invention relates to the technical field of migration learning under a machine learning module, in particular to a method for calibration and migration of an infrared spectrum measuring instrument based on CT-CDD.

Background technique

The near-infrared region is a type of electromagnetic wave between visible light and mid-infrared light, and it is the non-visible light region first discovered by William Herschel in the 19th century. The American Society for Testing and Materials (ASTM) defined it in October 1985 as a spectral region with a wavelength of 780nm-2526nm and a wavenumber of 12820-3959cm-1. By the 1950s, near-infrared spectroscopy techniques were available in some areas. Then in the 1960s, due to the continuous emergence of some novel analytical techniques, coupled with some shortcomings of near-infrared spectroscopy, people's interest in the near-infrared region gradually diminished except for some users of specific analytical applications.

Since then, the research on near-infrared spectroscopy has entered a long period of silence. With the deepening of the research and discussion of chemometrics and the continuous improvement of the manufacturing technology of spectroscopic instruments, the infrared spectroscopic analysis technology was further developed in the mid-1980s. Different from traditional analysis techniques, near-infrared spectroscopy is an indirect analysis technique and cannot directly obtain information such as substance content. A calibration model must be established through known samples to predict the concentration information of unknown samples to complete quantification or quantification. Qualitative analysis. The analysis process of near-infrared spectroscopy is shown in Figure 1, and the main steps are as follows:

(1) Select representative samples to form a calibration set, and test the near-infrared spectrum of the calibration set samples. In this link, it is necessary to pay attention to the representativeness of the collected calibration set samples;

(2) After collecting the samples of the calibration set, measure the concentration information of the substances of interest in the samples by standard analytical chemical means;

(3) Select an appropriate algorithm to model the spectrum of the measured calibration set samples and the corresponding substance concentration information. This step is the core step in the quantitative analysis of near-infrared spectroscopy. A calibration model is established for the pre-processed near-infrared spectroscopy and concentration information. Generally speaking, the parameters of the model are determined by cross-validation. Finally, the performance of the model needs to be checked;

(4) After the multivariate calibration model is established, the near-infrared spectrum of the current test sample can be measured, and the established calibration model can be used to predict the substance content of the test sample.

Modern near-infrared spectroscopic analysis technology has rich theoretical basis and technical practical experience. Different from other analytical techniques, near-infrared spectroscopy involves theoretical knowledge of many different disciplines such as spectroscopy, chemometrics and computer technology.

Near-infrared spectroscopy has many advantages, as it can determine the chemical composition and properties of a sample within minutes. Only by collecting and measuring the near-infrared spectrum of the tested sample once, it is possible to analyze various components of the sample at the same time, up to more than ten indicators. Near-infrared spectroscopic analysis technology can directly analyze the sample after a simple pretreatment, without causing damage to the sample, and realizing non-destructive testing; it does not need to use any chemical reagents, which greatly reduces the cost of analysis, and has no effect on the environment. Cause pollution, which is a "green analysis" technology. Near-infrared spectroscopy mainly reflects the information of chemical bonds such as C-H, O-H, N-H, S-H and other hydrogen-containing groups in organic molecules in the sample, and is very suitable for quantitative or qualitative analysis of hydrogen-containing organic compounds. The analysis range of near-infrared spectroscopy analysis technology includes most organic mixtures and compounds, and the unique advantages of near-infrared spectroscopy technology make its application field extremely broad, and it plays an indispensable role in many industries. The component content of substances, such as measuring the moisture or protein content in corn, etc.; in the field of medicine, measuring the component content of medicines, as well as biological, food and environmental testing, etc.

Machine learning and data mining techniques have achieved significant success in many fields of knowledge engineering including classification, regression, and clustering. For traditional machine learning methods, the distribution of training data and the distribution of test data should be the same, so that the test data can be predicted using the model established by the training data. In practical application scenarios, there will be slight differences between their data distributions. In some cases, training data is expensive or impossible to collect. In this case, if there is a significant difference in the data distribution of the training data and the test data, there will be a large difference between the predicted results of the test data and the real results, and most statistical models need to be remodeled using the newly collected training data . In this case, knowledge transfer between task domains is desirable, an approach called transfer learning. Transfer learning is the enhancement of the abilities of learners in another domain through information transferred from a related domain.

Multivariate calibration is a very useful tool for extracting chemical information from spectral signals, and the established multivariate calibration model is crucial for many analytical measurements. It has been used in various analytical techniques, but its importance has been demonstrated in near-infrared (NIR) spectroscopy. Usually, a lot of manpower and resources are invested in building a robust calibration model. Problems can arise when measuring samples on different instruments or under different environmental factors. Even if the same sample is measured, the two spectral matrices measured by different instruments are different, and the model established will produce differences. A model built on one instrument usually cannot make predictions about the spectrum measured by a second instrument. One solution to this problem is to remeasure each sample and build a new model for the newly acquired spectrum, but this is not a practical solution. Building a robust calibration model requires a lot of cost and time, and another acceptable way to save these unnecessary expenses is to perform model transfer. This way of dealing with problems in the field of machine learning is called transfer learning, and more specifically, the situation where the task is the same but the domain is different is called domain adaptation. And in the field of chemometrics, they are called calibration shifts.

Most of the calibration migration methods use a set of standard samples to build a migration model, which requires the measurement of a set of standard samples on the master instrument and the slave instrument respectively, and a variety of standard migration methods have been proposed. For example, Direct Standardization (DS) and Piecewise Direct Standardization (PDS) correct for spectral differences between master and slave instruments through a set of standard samples. In DS, each wavelength of the master is related to all wavelengths of the slaves. In PDS, each wavelength of the master instrument is related to the wavelength window of the slave instrument, and finally a band-like migration matrix is formed by the regression coefficients of each window. The experimental results are consistent with the hypothesis that the spectral correlation between master and slave instruments is restricted to a small area in various migration methods. The key to PDS is the selection of the window size and the determination of the number of standard samples. This process builds multiple regression models, which will result in a lot of computation. PDS is one of the most widely used migration methods and is often used as a comparison to other new technologies. In slope and bias correction (SBC), a linear relationship is assumed between the predicted values of different instruments. First, the regression coefficients between the spectra and the response values are calculated; the predicted values of the master and slave instruments are calculated from the regression coefficients; finally, a linear fit is performed between the predicted values. Liang et al. proposed a calibration transfer method based on canonical correlation analysis that successfully corrected the differences between different spectra. First, use the calibration set of the master instrument to build the PLS model; select a part of the calibration set of the master instrument and the slave instrument as a standard sample; extract features through canonical correlation analysis (CCA). The relationship between the master spectrum and the slave spectrum was established by the method of least squares (Ordinary least squares, OLS), and finally the difference of the spectrum was successfully corrected. In addition, other calibration migration methods have been proposed, such as Spectral regression (SR), Transfer by orthogonal projection (TOP), Single wavelength standardization (SWS), Multispectral calibration transfer, Generalized least squares weighting (GLSW) method, and other methods that require standard samples.

It can be seen from the above that there are many methods to develop a relatively robust calibration model in the prior art, but changes in environmental conditions and adjustment of measuring instruments will make the prediction performance of the calibration model poor or even lead to model failure. The relevant knowledge of the calibration model is transferred to the measured spectrum, which helps the measured spectrum to predict and saves a lot of overhead. Among the existing calibration transfer methods that can significantly improve the model prediction performance, most of them need to use standard samples to build a transfer model. The standard sample should closely match the sample from which the calibration model was constructed and must exhibit sufficient variability to account for differences between the two instruments. The volatility and reactivity of components make maintaining the integrity of standard samples a challenge. Even, in some practical applications, it is difficult or even impossible to obtain standard samples, that is, it is difficult to measure their spectra simultaneously on the master and slave instruments. Although there are a few calibration transfer methods that do not require standard samples, the prediction performance of these methods is quite different from that of transfer methods with standard samples.

SUMMARY OF THE INVENTION

Aiming at the problems existing in the prior art, the present invention provides a method for calibrating migration of an infrared spectrometer measuring instrument based on CT-CDD, which does not need to use a standard sample to construct a migration model, and can greatly improve the accuracy and efficiency of calibrating and migrating an infrared spectrometer measuring instrument.

The technical scheme of the present invention is:

A kind of infrared spectroscopic measuring instrument calibration migration method based on CT-CDD, is characterized in that, comprises the following steps:

Step 1: Correspond the main infrared spectrum measurement instrument to the source domain and the infrared spectrum measurement slave instrument to the target domain, use the infrared spectrum measurement master instrument and the infrared spectrum measurement slave instrument to collect the spectrum of each sample, and obtain the main spectrum and the slave instrument respectively. spectrum, extract spectral data at intervals of anm in the wavelength range for the main spectrum and the slave spectrum, and collect the variable value of the substance concentration of each sample to obtain the source domain data set {X ^m , y ^m } and the target domain data set {X ^s };

分别为第i个样本的主光谱向量、从光谱向量，i＝1,2,...,I，I为样本总数，

分别为第i个样本的第j个主光谱数据、从光谱数据，j＝1,2,...,J，J为提取的光谱数据点总数；y^m为物质浓度变量值矩阵，

为第i个样本的物质浓度变量的值；Among them, X ^m , X ^s are the main spectral matrix and the secondary spectral matrix, respectively,

are the master spectrum vector and slave spectrum vector of the ith sample respectively, i=1,2,...,I, where I is the total number of samples,

are the jth main spectral data and slave spectral data of the ith sample respectively, j=1,2,...,J, J is the total number of extracted spectral data points; y ^m is the substance concentration variable value matrix,

is the value of the substance concentration variable of the ith sample;

与源域测试集

Step 2: Use the KS algorithm to divide the source domain dataset {X ^m , y ^m } into source domain calibration sets

Test set with source domain

进行中心化处理，得到中心化处理后的源域标定集

Step 3: Calibration set for source domain

Perform centralized processing to obtain the centrally processed source domain calibration set

建立标定模型

计算得到

的权重矩阵W^m、

的载荷矩阵P^m、回归系数矩阵β^m；Step 4: Align the dataset based on the PLS algorithm

Build a calibration model

Calculated

The weight matrix W ^m ,

The load matrix P ^m and the regression coefficient matrix β ^m of ;

Step 5: Build the migration model:

Step 5.1: Calculate the characteristic spectrum matrix of the main instrument of infrared spectrum measurement as

T ^m =X ^m W ^m (P ^m W ^m ) ^-1

Calculate the pseudo-eigenspectral matrix of the infrared spectrometer from the instrument as

l＝1,2,...,L；Step 5.2: For each number of clusters L∈L ^* , use the k-means clustering algorithm to cluster the eigenspectral vectors of the dataset {T ^m , y ^m }, and divide the dataset {T ^m , y ^m } for L sub-datasets

l=1,2,...,L;

建立初始最小二乘模型

l＝1,2,...,L；Based on the OLS algorithm, the l-th sub-data set is

Build an initial least squares model

l=1,2,...,L;

Calculate the cross-validation error RMSECV _L of the L initial least squares models under each number of clusters, and determine the number of clusters corresponding to min{RMSECV _L |L∈L ^* } as the final number of clusters K;

为第l个初始子特征光谱矩阵，

为

对应的样本的物质浓度变量值组成的矩阵，β_{0_l}为第l个初始回归系数矩阵；Among them, L ^* is the cluster number set,

is the l-th initial sub-feature spectrum matrix,

for

The matrix composed of the substance concentration variable values of the corresponding sample, β _{0_l} is the l-th initial regression coefficient matrix;

k＝1,2,...,K；Step 5.3: Use the k-means clustering algorithm to cluster the characteristic spectral vectors of the dataset {T ^m , y ^m } with the number of clusters K, and divide the dataset {T ^m , y ^m } into K sub-data sets

k=1,2,...,K;

的伪特征光谱向量进行聚类，将数据集

划分为K个子数据集

k＝1,2,...,K；With the number of clusters K, use the k-means clustering algorithm to classify the data set

The pseudo eigenspectral vector of the clustering, the dataset

Divide into K sub-datasets

k=1,2,...,K;

的行向量，

分别为第k个子特征光谱矩阵、子伪特征光谱矩阵，

为

对应的样本的物质浓度变量值组成的矩阵；Among them, the characteristic spectral vector and pseudo characteristic spectral vector are T ^m ,

the row vector of ,

are the k-th sub-feature spectrum matrix and sub-pseudo-feature spectrum matrix, respectively,

for

A matrix consisting of the substance concentration variable values of the corresponding samples;

建立第k个最小二乘模型

计算得到第k个回归系数矩阵β_k；Step 5.4: Based on the OLS algorithm for the kth subdataset

Build the kth least squares model

Calculate the kth regression coefficient matrix β _k ;

其中，

分别为

的协方差矩阵；Step 5.5: Calculate the k-th transformation matrix

in,

respectively

The covariance matrix of ;

Step 6: Predict the substance concentration variable of the measured object set:

Step 6.1: Use infrared spectrum measurement to collect the spectrum of each measured object in the measured object set from the instrument, extract the spectral data using the same method as in step 1, and obtain the slave spectrum matrix of the measured object set

Step 6.2: Calculate the pseudo-feature spectrum matrix of the measured object collection from the infrared spectrum measurement as

的伪特征光谱向量进行聚类，将数据集

划分为K个子数据集

k＝1,2,...,K；其中，

为被测对象集合的第k个子伪特征光谱矩阵，Step 6.3: With the number of clusters K, use the k-means clustering algorithm to classify the data set

The pseudo eigenspectral vector of the clustering, the dataset

Divide into K sub-datasets

k=1,2,...,K; where,

is the kth sub-pseudo-feature spectral matrix of the measured object set,

进行变换校正，得到第k个变换校正过的子伪特征光谱矩阵为

Step 6.4: Use the k-th transformation matrix _Mk pair

Perform transformation correction, and obtain the sub-pseudo-feature spectral matrix corrected by the kth transformation as

对应的被测对象的物质浓度变量预测值矩阵为

Step 6.5: Calculate the k-th transformation-corrected sub-pseudo-feature spectral matrix

The predicted value matrix of the substance concentration variable corresponding to the measured object is:

The beneficial effects of the present invention are:

The invention performs calibration and migration by correcting the data distribution difference (CT-CDD) of the PLS subspace. Specifically, by establishing the PLS model of the master instrument, and simultaneously projecting the spectra of the master instrument and the slave instrument into the PLS subspace, the latent variables of different spectra are analyzed. Cluster analysis was performed separately, and the regression model between the latent variables of the main instrument and the concentration information was established by the ordinary least square method, and the characteristic spectrum with the closest data distribution between the two instruments was found, and the conversion function was calculated separately to compare the measured data. The substance concentration variables of the object are predicted, and the prediction results can be corrected by their respective conversion functions. The whole process does not need to use standard samples to build a migration model, which greatly improves the accuracy and efficiency of the migration calibration of the infrared spectroscopy measuring instrument, and solves the problem of the existing technology. There are calibration migration methods that can significantly improve the prediction performance of the model. Standard samples are needed to build the migration model, and the standard samples are difficult or even impossible to obtain, and its integrity is difficult to guarantee. A small number of calibration migration methods that do not require standard samples have poor predictive performance. question.

Description of drawings

Figure 1 is a schematic diagram of the analysis process of near-infrared spectroscopy.

FIG. 2 is a flow chart of a method for calibrating migration of an infrared spectrometer measuring instrument based on CT-CDD of the present invention.

FIG. 3 is a schematic diagram of the spectral difference between different instruments in Example 1 and Example 2

4 is a schematic diagram of the prediction results of the CT-CDD-based infrared spectroscopy measuring instrument calibration migration method of the present invention and the other five calibration migration methods on M5*-MP5 in Example 1.

5 is a schematic diagram of the prediction results of the CT-CDD-based infrared spectroscopy measuring instrument calibration migration method of the present invention and the other five calibration migration methods on M5*-MP6 in Example 1.

6 is a schematic diagram showing the prediction results of the CT-CDD-based infrared spectroscopy measuring instrument calibration migration method of the present invention and the other five calibration migration methods on MP5*-MP6 in Example 1.

FIG. 7 is a schematic diagram showing the prediction results on B1*-B2 of the CT-CDD-based infrared spectroscopy measuring instrument calibration migration method of the present invention and the other five calibration migration methods in the second embodiment.

FIG. 8 is a schematic diagram showing the prediction results on B1*-B3 of the CT-CDD-based infrared spectroscopy measuring instrument calibration migration method of the present invention and the other five calibration migration methods in the second embodiment.

FIG. 9 is a schematic diagram showing the prediction results on B3*-B2 of the CT-CDD-based infrared spectroscopy measuring instrument calibration migration method of the present invention and the other five calibration migration methods in the second embodiment.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

The present invention aims at the calibration migration method existing in the prior art that can significantly improve the model prediction performance, which requires standard samples to construct a migration model, and the standard samples are difficult or even impossible to obtain, its integrity is difficult to guarantee, and a small number of calibration migration methods do not require standard samples. For the technical problem of poor prediction performance, in view of the characteristics of high dimension and multicollinearity of spectral data, the transfer learning method in machine learning is used, and a calibration transfer method without transfer standard is proposed. The performance of the CT-CDD of the present invention was compared with the predicted performance of SBC, PDS, CCACT, TCR and CTAI through two NIR spectral datasets. In the absence of standard samples, the prediction performance achieved by the present invention is superior to the classical standard calibration transfer method.

Example 1

As shown in Figure 2, the CT-CDD-based infrared spectroscopy measuring instrument calibration migration method of the present invention comprises the following steps:

Step 1: Correspond the main infrared spectrum measurement instrument to the source domain and the infrared spectrum measurement slave instrument to the target domain, use the infrared spectrum measurement master instrument and the infrared spectrum measurement slave instrument to collect the spectrum of each sample, and obtain the main spectrum and the slave instrument respectively. spectrum, extract spectral data at intervals of anm in the wavelength range for the main spectrum and the slave spectrum, and collect the variable value of the substance concentration of each sample to obtain the source domain data set {X ^m , y ^m } and the target domain data set {X ^s };

分别为第i个样本的主光谱向量、从光谱向量，i＝1,2,...,I，I为样本总数，

分别为第i个样本的第j个主光谱数据、从光谱数据，j＝1,2,...,J，J为提取的光谱数据点总数；y^m为物质浓度变量值矩阵，

为第i个样本的物质浓度变量的值。Among them, X ^m , X ^s are the main spectral matrix and the secondary spectral matrix, respectively,

are the master spectrum vector and slave spectrum vector of the ith sample respectively, i=1,2,...,I, where I is the total number of samples,

are the jth main spectral data and slave spectral data of the ith sample respectively, j=1,2,...,J, J is the total number of extracted spectral data points; y ^m is the substance concentration variable value matrix,

is the value of the substance concentration variable of the ith sample.

In the first embodiment, the sample is corn, and the spectral data is the absorbance. The substance concentration variables can be moisture content, oil content, protein content, and starch content. In this embodiment, the moisture content is used to verify the method of the present invention. The corn data set was composed of data measured on the same I=80 samples using three near-infrared spectroscopy instruments (M5, MP5, MP6). Use the near-infrared spectrum measuring instruments M5, MP5, and MP6 to measure the infrared spectrum every a=2nm in the wavelength range of 1100-2498nm, with a total of J=700 channels, and obtain between M5-MP5, M5-MP6, MP5-MP6 The spectral differences between them are shown in Figure 3A, Figure 3B, and Figure 3C, respectively.

与源域测试集

Step 2: Use the KS algorithm to divide the source domain dataset {X ^m , y ^m } into source domain calibration sets

Test set with source domain

In the first embodiment, the Kennard-Stone (KS) algorithm divides 80 corn samples into two groups: the source domain data of the first group of 64 samples constitutes the source domain calibration set; the source domain data of the second group of 16 samples Constitute the source domain test set.

进行中心化处理，得到中心化处理后的源域标定集

Step 3: Calibration set for source domain

Perform centralized processing to obtain the centrally processed source domain calibration set

建立标定模型

计算得到

的权重矩阵W^m、

的载荷矩阵P^m、回归系数矩阵β^m。Step 4: Align the dataset based on the PLS algorithm

Build a calibration model

Calculated

The weight matrix W ^m ,

The loading matrix P ^m , the regression coefficient matrix β ^m .

Step 5: Build the migration model:

Step 5.1: Calculate the characteristic spectrum matrix of the main instrument of infrared spectrum measurement as

T ^m =X ^m W ^m (P ^m W ^m ) ^-1

Calculate the pseudo-eigenspectral matrix of the infrared spectrometer from the instrument as

l＝1,2,...,L；Step 5.2: For each number of clusters L∈L ^* , use the k-means clustering algorithm to cluster the eigenspectral vectors of the dataset {T ^m , y ^m }, and divide the dataset {T ^m , y ^m } for L sub-datasets

l=1,2,...,L;

建立初始最小二乘模型

l＝1,2,...,L；Based on the OLS algorithm, the l-th sub-data set is

Build an initial least squares model

l=1,2,...,L;

Calculate the cross-validation error RMSECV _L of the L initial least squares models under each number of clusters, and determine the number of clusters corresponding to min{RMSECV _L |L∈L ^* } as the final number of clusters K;

为第l个初始子特征光谱矩阵，

为

对应的样本的物质浓度变量值组成的矩阵，β_{0_l}为第l个初始回归系数矩阵；Among them, L ^* is the cluster number set,

is the l-th initial sub-feature spectrum matrix,

for

The matrix composed of the substance concentration variable values of the corresponding sample, β _{0_l} is the l-th initial regression coefficient matrix;

k＝1,2,...,K；Step 5.3: Use the k-means clustering algorithm to cluster the characteristic spectral vectors of the dataset {T ^m , y ^m } with the number of clusters K, and divide the dataset {T ^m , y ^m } into K sub-data sets

k=1,2,...,K;

的伪特征光谱向量进行聚类，将数据集

划分为K个子数据集

k＝1,2,...,K；With the number of clusters K, use the k-means clustering algorithm to classify the data set

The pseudo eigenspectral vector of the clustering, the dataset

Divide into K sub-datasets

k=1,2,...,K;

的行向量，

分别为第k个子特征光谱矩阵、子伪特征光谱矩阵，

为

对应的样本的物质浓度变量值组成的矩阵；Among them, the characteristic spectral vector and pseudo characteristic spectral vector are T ^m ,

the row vector of ,

are the k-th sub-feature spectrum matrix and sub-pseudo-feature spectrum matrix, respectively,

for

A matrix consisting of the substance concentration variable values of the corresponding samples;

建立第k个最小二乘模型

计算得到第k个回归系数矩阵β_k；Step 5.4: Based on the OLS algorithm for the kth subdataset

Build the kth least squares model

Calculate the kth regression coefficient matrix β _k ;

其中，

分别为

的协方差矩阵。Step 5.5: Calculate the k-th transformation matrix

in,

respectively

The covariance matrix of .

In the process of constructing the least squares model, the master instrument model closest to the characteristic spectrum after clustering of the slave instruments is found, and the transformation matrix is calculated separately. details as follows:

The master instrument and the slave instrument each correspond to a domain. A domain consists of two main parts: the input space X, and its corresponding marginal probability distribution P(X). Relative entropy or KL divergence can represent the distance between the data distributions of two domains, expressed using formula (1):

Among them, p and q are the probability density functions of the data distribution of the source domain and the target domain, respectively.

p(x) cannot be obtained directly, but assuming that a finite set of training points xn has been observed, _n =1,...,N can be derived from p(x). Then the expectation about p(x) can be approximated by a finite sum over these points with the formula:

不同仪器测量的光谱不同，导致两个仪器之间的数据分布是不同的。公式(3)使用绝对值的形式来表示数据分布之间的距离，两者都是随机向量，分别遵循各自的数据分布：Given the labeled spectrum {X ^m ,y ^m } of the master instrument and the unlabeled spectrum {X ^s } of the slave instrument, the goal is to predict the output of the spectrum to be measured by the slave instrument

The spectra measured by different instruments are different, resulting in different data distributions between the two instruments. Formula (3) uses the form of absolute value to represent the distance between data distributions, both of which are random vectors and follow their respective data distributions:

KL(P||Q)≈|lnP(X ^m )-lnQ(X ^s )| (3)

Among them, P and Q are the probability density functions of the data distribution of the source domain and the target domain, respectively;

Considering the multicollinearity of the spectral data, all spectra are mapped to the PLS subspace of the main instrument, which simplifies the model (3) while reducing the dimensionality of the data. The characteristic spectrum of the master instrument and the pseudo characteristic spectrum of the slave instrument are calculated using formula (4) in the following form:

分别是被提取的A个得分向量组成的矩阵。where ^Tm and

are a matrix composed of the extracted A score vectors, respectively.

At this point, the KL distance between the data distributions of the two instruments is:

KL(P||Q)≈|lnP(t ^m )-lnQ(t ^s )| (5)

where t ^m and t ^s are random vectors that follow the data distributions T ^m and t s, respectively

和

可以通过对各个聚类的均值和协方差分别校正，降低公式(6)中的分布差异。先中心化数据校正均值，两个仪器的第i个特征光谱的数据分布分别为

和

和

分别是两个仪器的第i个高斯分布；

是第i个转换函数；

和

是两个随机向量，分别服从两个仪器的第i个特征光谱的数据分布。公式(5)可以写成公式(6)的形式。The spectral data of the master and slave instruments are all mixed distribution, assuming that the data distribution after clustering is a single Gaussian distribution, and the data distributions of the characteristic spectra of the master and slave instruments are

and

The distribution difference in equation (6) can be reduced by separately correcting the mean and covariance of each cluster. Firstly, the data is adjusted to the mean value, and the data distribution of the i-th characteristic spectrum of the two instruments is

and

and

are the ith Gaussian distribution of the two instruments, respectively;

is the ith conversion function;

and

are two random vectors, respectively obeying the data distribution of the i-th characteristic spectrum of the two instruments. Equation (5) can be written in the form of Equation (6).

成立，有

使得校正的从光谱与主光谱之间的距离最小，相对熵(6)可以改写成如下：Assuming that there is a linear transformation matrix M _i can make the above

established, have

To minimize the distance between the corrected slave spectrum and the master spectrum, the relative entropy (6) can be rewritten as follows:

In formula (7), the solution process of the linear transformation matrix _Mi is as follows:

由公式(8.1)给出，从仪器的概率密度函数

由公式(8.2)给出。Each group of data after clustering is approximately normally distributed, the mean of the data is 0, and the probability density function of the main feature spectrum

is given by equation (8.1), from the probability density function of the instrument

is given by equation (8.2).

变换之后，从仪器的随机向量t^s能转化为主仪器的随机向量t^m，公式如下所示：through the function

After transformation, the random vector t ^s of the slave instrument can be converted to the random vector t ^m of the master instrument with the following formula:

Assuming _Mi is a non-singular matrix, then the random vector of the master instrument can be converted to the random vector of the slave instrument using equation (10):

可用公式(9)进行变换如下

其可变换如下

从而有：

According to the properties of the probability density function, the probability density function of the main instrument

It can be transformed by formula (9) as follows

It can be transformed as follows

Thus there are:

Equation (11) is the probability density function after transformation from the instrument characteristic spectrum to the main instrument through the transformation matrix _Mi , and the expanded form is as follows:

M_i的解为：Formula (12) and formula (8.1) are the same probability density function, so the covariance of the two is the same, so there is

The solution of _Mi is:

Step 6: Predict the substance concentration variable of the measured object set:

Step 6.1: Use infrared spectrum measurement to collect the spectrum of each measured object in the measured object set from the instrument, extract the spectral data using the same method as in step 1, and obtain the slave spectrum matrix of the measured object set

Step 6.2: Calculate the pseudo-feature spectrum matrix of the measured object collection from the infrared spectrum measurement as

的伪特征光谱向量进行聚类，将数据集

划分为K个子数据集

k＝1,2,...,K；其中，

为被测对象集合的第k个子伪特征光谱矩阵，Step 6.3: With the number of clusters K, use the k-means clustering algorithm to classify the data set

The pseudo eigenspectral vector of the clustering, the dataset

Divide into K sub-datasets

k=1,2,...,K; where,

is the kth sub-pseudo-feature spectral matrix of the measured object set,

进行变换校正，得到第k个变换校正过的子伪特征光谱矩阵为

Step 6.4: Use the k-th transformation matrix _Mk pair

Perform transformation and correction to obtain the k-th transformation-corrected sub-pseudo-feature spectral matrix as

对应的被测对象的物质浓度变量预测值矩阵为

k＝1,2,...,K。Step 6.5: Calculate the k-th transformation-corrected sub-pseudo-feature spectral matrix

The predicted value matrix of the substance concentration variable corresponding to the measured object is:

k=1,2,...,K.

In this embodiment, the method for calibrating the migration of an infrared spectrometer measuring instrument based on CT-CDD of the present invention and the traditional method for calibrating and migrating an infrared spectrometer measuring instrument based on SBC, PDS, CCACT, TCR and CTAI are respectively used for the substances collected by the measured object. Concentration variables are predicted. The PLS model of the main instrument of the present invention is established on the calibration set. For other migration methods with migration standards, the Kennard-Stone method is used to select several standard samples on the calibration set. For the SBC, PDS, CCACT, and CTAI algorithms, the PLS algorithm is used as the main algorithm, and the spectral data of the master instrument is used to establish a multivariate calibration model as a reference model to predict the samples to be tested from the slave instrument.

The parameter selection criteria for different transfer methods are similar to CTAI. The optimal number of latent variables of the PLS model is in the range [1, 15], which is determined by ten-fold cross-validation. According to the minimum cross-validation error criterion, the optimal number of latent variables is selected.

The PLS modeling method and parameter optimization of the main instruments of SBC and PDS are the same as those of CTAI. The window size in the PDS starts from 3, and the increment is 2 to 16. The parameters are selected through 5-fold cross-validation, the RMSECV of each window model is calculated separately, and the window with the smallest RMSECV is selected as the optimal parameter. Poor performance, use F-test to determine optimal window size during window selection.

In this embodiment, the root mean square error RMSE is used as an indicator for parameter selection and model evaluation. Furthermore, RMSEC represents the training error on the calibration set, RMSECV represents the cross-validation error, and RMSEP represents the prediction error on the test set. The RMSE calculation method is written as

是预测值，y是测量值，n表示样本的数目。in,

is the predicted value, y is the measured value, and n is the number of samples.

On the corn dataset, the RMSEC, RMSEP, RMSECVmin, and LV of the PLS models of the M5, MP5, and MP6 instruments are shown in Table 1. Among them, RMSECVmin is the minimum cross-validation error, and LV is the number of latent variables corresponding to the minimum cross-validation error. It can be seen from Table 1 that the RMSECVmin, RMSEC and RMSEP of the PLS model of the instrument M5 are 0.01066, 0.00599 and 0.00764 respectively. It can be seen that the three root mean square errors are not much different, the PLS model is relatively stable, and there is no overfitting and underfitting phenomenon. The RMSECVmin, RMSEC, and RMSEP of the PLS model of the instrument MP5 are 0.13035, 0.09458, and 0.12445, respectively. Similar to the PLS model of the M5, there is no underfitting and overfitting. The same conclusion is obtained on the PLS model of the MP6. The selection of parameters is selected by 10-fold cross-validation, and the optimal number of latent variables is determined based on the lowest RMSECV criterion. The optimal number of latent variables in the PLS models of the three instruments of M5, MP5, and MP6 are 14, 15, and 10, respectively. . It is very important for the main instrument to establish a model with better prediction performance. In this embodiment, an instrument with good prediction performance is selected as the main instrument. As can be seen from Table 1, the prediction error of instrument MP6 > the prediction error of instrument MP5 > the prediction error of instrument M5, so that the three combinations (M5*-MP5, M5*-MP6 and MP5*-MP6) are used for model checking. Best and reasonable; where the superscript * indicates the master instrument and the other indicates the slave instrument.

Table 1

instrument Reference RMSEC RMSEP RMSECV_min LV M5 moisture 0.00599 0.00764 0.01066 14 MP5 moisture 0.09458 0.12445 0.13035 15 MP6 moisture 0.09991 0.15637 0.14775 10

CT-CDD was compared with five calibration migration methods, SBC, PDS, CCACT, TCR and CTAI. In CT-CDD, the number of clusters was determined by ten-fold cross-validation. The corn dataset contains 80 samples, and the maximum number of sub-models after clustering is set to 3. Otherwise, the calculated migration matrix is under-ranked, which will lead to the final prediction result being infinite. Due to the limitation of the number of samples, when the number of clusters is large, the characteristic spectrum after clustering does not have enough samples to establish a stable model. In the first embodiment, it is found through calculation that when the number of clusters is 2, the minimum cross-validation error is obtained.

The prediction errors of CT-CDD and the other five calibration transfer methods are shown in Table 2. In Table 2, N is the number of standard samples in the migration method that requires standard samples, a is the optimal window size in PDS, and b is the dimension of the corresponding optimal subspace in TCR.

It can be seen from Table 2 that:

(1) For the spectral transfer from instrument MP5 to instrument M5: when the number of standard samples is 35, the SBC reaches the lowest RMSEP (0.28872); when the number of standard samples is 5, the PDS reaches the lowest RMSEP (0.18828); When the number of samples is 25, CCACT reaches the lowest RMSEP (0.18699); it can be seen that the RMSEP (0.15024) of CT-CDD is smaller than the minimum RMSEP of the three methods of PDS, SBC and CCACT, and is also smaller than TCR (0.47391) and CTAI ( 0.17511).

(2) For the spectral transfer from MP6 to M5, the lowest RMSEP obtained by SBC, PDS, CCACT are 0.33240, 0.27901, 0.17862, respectively, CT-CDD has lower RMSEP than the other five methods.

(3) For the spectral transfer from MP6 to MP5, the lowest RMSEP corresponding to SBC, PDS, and CCACT are 0.20481, 0.18409 and 0.13722, respectively, the prediction errors of TCR and CTAI are 0.46124 and 0.16563, respectively, and CT-CDD reaches the minimum again. RMSEP(0.12357).

From these three sets of experiments, it can be seen that the CT-CDD model achieves the best prediction performance in general and has better generalization ability.

Table 2

Figure 4, Figure 5, and Figure 6 show the relationship between the predicted and measured values obtained by 6 different calibration migration methods on the combinations M5*-MP5, M5*-MP6, and MP5*-MP6, respectively. Zero difference between predicted and measured concentrations will place the sample points on a straight line. For the calibration transfer method with standard samples, under different standard samples, when the prediction performance is the best, this group of experiments is selected for comparison, in order to more fully reflect that CT-CDD can achieve good prediction performance.

The prediction results of CT-CDD, CTAI, TCR, CCACT, SBC, and PDS on M5*-MP5 are shown in Fig. 4A, Fig. 4B, Fig. 4C, Fig. 4D, Fig. 4E, Fig. 4F, respectively, on M5*-MP6 The prediction results are shown in Figure 5A, Figure 5B, Figure 5C, Figure 5D, Figure 5E, Figure 5F, respectively, and the prediction results on MP5*-MP6 are shown in Figure 6A, Figure 6B, Figure 6C, Figure 6D, Figure 6E, respectively , as shown in Figure 6F. It can be seen from Figure 4 that the sample points of CT-CDD are closer to a straight line; TCR and SBC have poor fitting effects under this group of experiments. As can be seen from Figure 5, in the spectral transmission from instrument MP6 to instrument M5, CT-CDD is closer to a straight line than the other five methods, SBC and TCR once again achieve the worst prediction performance, PDS, CCACT and CTAI three The prediction error of the method is smaller but its prediction performance is still poor compared to CT-CDD. As can be seen from Figure 6, the same conclusions as in Figures 4 and 5 were achieved in the spectral transmission from instrument MP6 to instrument MP5, which confirms that CT-CDD achieves the best predictive performance. It can be seen that CT-CDD achieves more satisfactory results in comparison with the other five models, achieving the best prediction performance.

Embodiment 2

In the second embodiment, the sample is wheat. The wheat dataset is the "Shootout" dataset released at the 2016 International Diffuse Reflectance Conference (IDRC). The wheat dataset contains data measured by 3 different NIR spectrometers (B1, B2, B3) on the same I=248 samples, with protein content selected as the substance concentration variable. Use NIR spectrometers B1, B2, B3 to measure the infrared spectrum at every interval a=0.5nm in the wavelength range of 570-1100nm, and obtain the spectral differences between B1-B2, B1-B3, B3-B2 as shown in Figure 3D, Figure 3E, shown in Figure 3F.

In the first embodiment, the Kennard-Stone (KS) algorithm divides 248 wheat samples into two groups: the source domain data of 198 samples in the first group constitutes the source domain calibration set; the source domain data of 50 samples in the second group Constitute the source domain test set.

On the wheat dataset, the RMSEC, RMSEP, RMSECVmin, and LV of the PLS models of the three instruments B1, B2, and B3 are shown in Table 3. It can be seen from Table 3 that the RMSECVmin, RMSEC, and RMSEP of the PLS model established on the instrument B1 are 0.50337, 0.32880, and 0.33254, respectively, and there is no overfitting or underfitting. The same situation is also observed on instrument B2 and instrument B3. There is no overfitting and underfitting in the PLS models established by the three instruments, which explains the reasonable selection of the optimal latent variables. For the wheat dataset, the parameter selection criteria for the PLS model were similar to those for the corn dataset, and the number of latent variables was set to a maximum of 15. It can be observed from Table 3 that the prediction error of the instrument B1 < the prediction error of the instrument B3 < the prediction error of the instrument B2, so that the three combinations (B1*-B2, B1*-B3 and B3*-B2) are used to test the model performance .

table 3

instrument Reference RMSEC RMSEP RMSECV_min LV B1 protein 0.32880 0.33254 0.50337 15 B2 protein 0.21636 0.83755 0.32441 15 B3 protein 0.30288 0.51567 0.43896 15

In CT-CDD, the method for determining the number of characteristic spectral clusters is similar to that of the corn dataset. When the number of samples in the wheat dataset is relatively sufficient, the under-rank situation that occurs when calculating the migration matrix in the corn dataset does not occur. The number of clusters is set between 2 and 5. In the second embodiment, it is found through calculation that when the number of clusters is 2, the minimum cross-validation error is obtained.

The parameter selection criteria for other calibration transfer methods are also similar to the corn dataset. In PDS, the optimal window size is shown in Table 4. When B1 is the master instrument and B2 is the slave instrument, the optimal window sizes of PDS are 11, 15, 15, and 15, respectively. When B1 is the master instrument and B3 is the slave instrument, the optimal window sizes of the PDS are 15, 11, 5, and 5, respectively. When B3 is the master instrument and B2 is the slave instrument, the optimal window sizes of PDS are 7, 15, 15, and 15, respectively. In TCR, the optimal dimensions of the subspace are 7, 12, and 21, respectively.

It can be seen from Table 4 that:

(1) When the instrument B1 is the master instrument and the instrument B2 is the slave instrument, when the number of standard samples is 5, the SBC produces the lowest RMSEP (0.45225). In PDS and CCACT, the RMSEP decreased significantly when the standard sample was increased. When the number of standard samples was 35, PDS and CCACT achieved the lowest RMSEP of 0.47222 and 0.80448, respectively. Compared with SBC, PDS, and CCACT, CT-CDD achieved the lowest RMSEP (0.43007). The prediction errors of TCR and CTAI are 0.86884 and 0.41419, respectively. Compared with them, CT-CDD is second only to CTAI in prediction effect under the current experimental group.

(2) When the instrument B1 is used as the master instrument and the instrument B3 is used as the slave instrument, the corresponding lowest RMSEPs of SBC, PDS and CCACT under different numbers of standard samples are 0.79919, 0.41235 and 0.83440 respectively. The RMSEP of TCR and CTAI were 0.72987 and 0.68215, respectively. The results show that the RMSEP (0.35160) of CT-CDD is significantly lower than other calibration transfer methods, achieving the best prediction performance.

(3) When the instrument B3 is the master instrument and the instrument B2 is the slave instrument, the lowest RMSEP corresponding to SBC, PDS and CCACT are 0.47177, 0.33707 and 0.75119 respectively. The prediction errors of the two methods, TCR and CTAI, are 0.63708 and 0.38446, respectively. The same situation recurs, CT-CDD achieves the best prediction performance (RMSEP of 0.31856). SBC, PDS and CCACT require standard samples, TCR requires reference values from the instrument, and in the absence of standard samples, CT-CDD achieves better predictive performance. Obviously, this means that CT-CDD is the more acceptable method.

Table 4

Figures 7, 8, and 9 show the relationship between the predicted and measured values obtained by 6 different calibration migration methods on the combinations B1*-B2, B1*-B3, and B3*-B2, respectively.

The prediction results of CT-CDD, CTAI, TCR, CCACT, SBC, and PDS on B1*-B2 are shown in Fig. 7A, Fig. 7B, Fig. 7C, Fig. 7D, Fig. 7E, Fig. 7F, respectively, and on B1*-B3 The prediction results are shown in Figure 8A, Figure 8B, Figure 8C, Figure 8D, Figure 8E, Figure 8F, respectively, and the prediction results on B3*-B2 are shown in Figure 9A, Figure 9B, Figure 9C, Figure 9D, Figure 9E , as shown in Figure 9F. It can be clearly seen from Figure 7C and Figure 7D that the correlation between TCR and CCACT is poor, and the prediction error of the model is large. It can be observed from Figure 7A that the CT-CDD has a better fitting effect, and the prediction error of the corresponding model is smaller. Figure 8 shows that CT-CDD and PDS achieve good fitting results, while the other four methods have relatively poor fitting results. Figure 9 shows that TCR and CCACT found a rather poor correlation between substance concentrations and predicted results, the other four methods fit better, but among them CT-CDD achieved the best fit. It can be seen that CT-CDD provides more satisfactory results in comparison with the other five transfer methods.

It can be seen from the above-mentioned Embodiment 1 and Embodiment 2 of the present invention that in the process of using CTAI, TCR, CCACT, SBC, and PDS as comparative experiments and using two NIR data sets to test the performance of the CT-CDD method, the present invention The CT-CDD-based calibration transfer method achieves the best RMSEP (minimum). The results clearly show that CT-CDD successfully corrects for differences between spectra measured on different instruments. For SBC, PDS and CCACT, they require standard samples to model migration. In TCR, a small amount of reference value is also required from the instrument sample. Both of these conditions are very expensive in practical applications, and even this condition cannot be met. Therefore, when standard samples are not available in practical applications, the CT-CDD-based method of the present invention is a cost-effective method for calibrating migration.

The standard-free calibration migration method of the present invention, by Correcting for PLS Subspace Data Distribution Difference (CT-CDD), attempts to find a transfer function that ensures that when the data from the slave is projected into this space, the data between the master and slave The distribution distance can be reduced. The data distribution of the characteristic spectra is a mixture distribution, and it is necessary to cluster the spectra and minimize the distance of each sub-distribution between the two instruments by their respective transfer functions. The present invention preserves the important properties of the two instruments in the same PLS subspace and eliminates spectral multicollinearity, while the data differences between the master instrument's features and the slave's pseudo-features can be narrowed more precisely. Differences in data distributions were further corrected for by correcting for the mean and variance of each component of the latent variables from different instruments.

Obviously, the above-mentioned embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. The above embodiments are only used to explain the present invention, and do not constitute a limitation on the protection scope of the present invention. Based on the above embodiments, all other embodiments obtained by those skilled in the art without creative work, that is, all modifications, equivalent replacements and improvements made within the spirit and principle of the present application, are fall within the scope of protection claimed by the present invention.

Claims (1)

1. A CT-CDD-based infrared spectrum measuring instrument calibration migration method is characterized by comprising the following steps:

step 1: the method comprises the steps of enabling an infrared spectrum measurement master instrument to correspond to a source domain, enabling an infrared spectrum measurement slave instrument to correspond to a target domain, collecting the spectrum of each sample by using the infrared spectrum measurement master instrument and the infrared spectrum measurement slave instrument, respectively obtaining a master spectrum and a slave spectrum, respectively extracting spectral data of the master spectrum and the slave spectrum at intervals anm within a wavelength range, collecting material concentration variable values of each sample, and obtaining a source domain data set { X^m,y^mAnd a target domain data set { X }^s}；

Wherein, X^m、X^sRespectively a master spectral matrix and a slave spectral matrix,

i is the main spectral vector and the slave spectral vector of the ith sample, I is 1, 2.

J, J is the total number of extracted spectral data points, i.e. the jth primary spectral data and the jth secondary spectral data of the ith sample respectively; y is^mIs a matrix of values of the concentration of the substance,

is the value of the substance concentration variable for the ith sample;

step 2: source domain data set { X using KS algorithm^m,y^mDividing into a source domain calibration set

And source domain test set

And step 3: set of source domain calibrations

Performing centralization treatment to obtain a source domain calibration set after centralization treatment

And 4, step 4: PLS algorithm based on data sets

Establishing a calibration model

Is calculated to obtain

Weight matrix W of^m、

Load matrix P^mRegression coefficient matrix beta^m；

And 5: constructing a migration model:

step 5.1: calculating a characteristic spectrum matrix of a main instrument for infrared spectrum measurement

T^m＝X^mW^m(P^mW^m)^-1

Calculating a pseudo-characteristic spectral matrix of the infrared spectrometric slave instrument

Step 5.2: for each cluster number L belongs to L^*Using k-means clustering algorithm to data set { T }^m,y^mThe characteristic spectrum vectors of the data set are clustered, and the data set is subjected to the clustering of the characteristic spectrum vectors of the data set T^m,y^mDivide into L sub-datasets

l＝1,2,...,L；

On the basis of OLS algorithm, the first sub-data set

Establishing an initial least squares model

l＝1,2,...,L；

Calculating the cross validation error RMSECV of L initial least square models under each cluster number_LDetermining min { RMSECV }_L|L∈L^*The corresponding cluster number is the final cluster number K;

wherein L is^*To be a set of the number of clusters,

is the l-th initial sub-feature spectral matrix,

is composed of

Matrix of values of variables of the concentration of substance of the corresponding sample, beta_{0_l}Is the first initial regression coefficient matrix;

step 5.3: using K-means clustering algorithm to perform data set { T) according to clustering number K^m,y^mThe characteristic spectrum vectors of the data set are clustered, and the data set is subjected to the clustering of the characteristic spectrum vectors of the data set T^m,y^mDivide into K sub-datasets

k＝1,2,...,K；

Using K-means clustering algorithm to perform data set according to clustering number K

The pseudo characteristic spectral vectors are clustered, and the data set is obtained

Partitioning into K sub-datasets

k＝1,2,...,K；

Wherein the characteristic spectrum vector and the pseudo characteristic spectrum vector are respectively T^m、

The line vectors of (a) are,

respectively a k-th sub characteristic spectrum matrix and a sub pseudo characteristic spectrum matrix,

is composed of

A matrix formed by the variable values of the substance concentration of the corresponding sample;

step 5.4: on the basis of OLS algorithm, the kth sub-data set

Establishing a kth least squares model

Calculating to obtain a k-th regression coefficient matrix beta_k；

Step 5.5: computing the kth transformation matrix

Wherein,

are respectively as

The covariance matrix of (a);

step 6: and predicting the substance concentration variable of the measured object set:

step 6.1: collecting the spectrum of each measured object in the measured object set from the instrument by using infrared spectrum measurement, and extracting the spectrum data by using the same method as the step 1 to obtain a secondary spectrum matrix of the measured object set

Step 6.2: calculating a pseudo characteristic spectrum matrix of a measured object set under an infrared spectrum measuring slave instrument as

Step 6.3: using K-means clustering algorithm to perform data set according to clustering number K

The pseudo characteristic spectral vectors are clustered, and the data set is obtained

Partitioning into K sub-datasets

K1, 2,. K; wherein,

for the kth sub-pseudo characteristic spectrum matrix of the measured object set,

step 6.4: using the kth transformation matrix M_kTo pair

Carrying out transformation correction to obtain a k transformation corrected sub-pseudo characteristic spectrum matrix of

Step 6.5: computing a k-th transform corrected sub-pseudo feature spectrum matrix

Predicted value of substance concentration variation of corresponding measured objectThe matrix is

k＝1,2,...,K。

Images