CN113239829B - Cross-dimension remote sensing data target identification method based on space occupation probability characteristics - Google Patents

Abstract

A cross-dimensional remote sensing data target recognition method based on space occupancy probability features belongs to the technical field of remote sensing image target recognition. The present invention is to solve the problem that there is currently no effective cross-dimensional feature description method that can realize the association between two-dimensional image data and three-dimensional point cloud data of the same object in remote sensing data. The invention firstly trains the space occupancy probability feature extraction network of two-dimensional images and the space occupancy probability feature extraction network of three-dimensional point cloud, and then extracts the same kind of features for the two-dimensional image data and three-dimensional point cloud data in the remote sensing data, namely the space occupancy probability Finally, based on the space occupancy probability feature, the cross-dimensional target recognition between the two-dimensional and three-dimensional data of the same object in the remote sensing data is realized, that is, the association between the two-dimensional image data of the same object and the three-dimensional point cloud data is realized. Mainly used for target recognition of remote sensing data.

Description

Target recognition method of cross-dimensional remote sensing data based on spatial occupancy probability feature

technical field

The invention relates to a remote sensing data target recognition method, and belongs to the technical field of remote sensing image target recognition.

Background technique

For a long time, the extraction and recognition technology of ground object information based on remote sensing images has a wide range of application requirements in the fields of people's livelihood and national defense. With the development of technology in the field of remote sensing and the improvement of application requirements, compared with the two-dimensional image data obtained by traditional visible light band cameras, the three-dimensional point cloud data obtained by remote sensing scanning equipment such as lidar can perform remote sensing targets in three-dimensional space. A more complete three-dimensional description has greater significance in both civil and military fields. Therefore, in order to make full use of all the ground object information obtained by equipment such as lidars and cameras in the traditional visible light band, it is necessary to associate the two-dimensional image data of the same ground object with the three-dimensional point cloud data. Since two-dimensional images and three-dimensional point clouds have different dimensional features, there is currently no effective cross-dimensional feature description method to achieve the correlation between the two. Dimensional target recognition is of great significance.

SUMMARY OF THE INVENTION

The present invention is to solve the problem that there is currently no effective cross-dimensional feature description method that can realize the association between two-dimensional image data and three-dimensional point cloud data in remote sensing data.

The target recognition method of cross-dimensional remote sensing data based on space occupancy probability features includes the following steps:

S1: Preprocess the two-dimensional image remote sensing data: input the two-dimensional image remote sensing data into the instance segmentation network, and perform target extraction on the remote sensing data according to the instance segmentation result;

S2: Preprocess the 3D point cloud remote sensing data: input the 3D point cloud into the point cloud target detection network, and perform target segmentation on the 3D point cloud remote sensing data according to the target detection result;

S3: Input the image processed by S1 into the deep learning network of the space occupancy probability feature of the two-dimensional image, and extract the space occupancy probability F _test-2D of the two-dimensional image;

S4: input the 3D point cloud processed by S2 into a deep learning network of the space occupancy probability feature of the 3D point cloud; extract the space occupancy probability F _test-3D of the 3D point cloud;

S5: Input the space occupancy probability features F _test-2D and F _test-3D obtained in S3 and S4 into the classifier for target recognition, so as to realize the association between the two-dimensional image data and the three-dimensional point cloud data.

Further, the instance segmentation network described in S1 adopts PANET.

Further, the point cloud target detection network described in S2 adopts 3D-BONET.

Further, the deep learning network of the space occupancy probability feature of the two-dimensional image described in S3 is the OccupancyNetwork-2D network, namely Onet-2D, and its training process includes the following steps:

S301, constructing a two-dimensional image data set M _pre-2D , the two-dimensional image data set M _pre-2D includes a two-dimensional image training data set M _tr-2D and a two-dimensional image test data set M _test-2D ;

S302, training Onet-2D:

The two-dimensional image data in the two-dimensional image training dataset M _tr-2D is input into Onet-2D, and Onet-2D first uses the RESNET18 residual network with the super channel attention module ECA to characterize the input two-dimensional image data. Extraction to obtain a feature F ₁ of 1*256;

Second, randomly generate a sampling point cloud cube of unit volume, input the x, y, and z coordinates of each point in the point cloud cube into a three-layer mlp multi-layer neural network, and transpose to obtain a feature F of 256*N ₂ ;

Then F ₁ and F ₂ are respectively input into at least one conditional batch normalization module, the conditional batch normalization module is the CBN module; the specific process includes the following steps:

The 1*256 feature F ₁ extracted from the 2D image is input into the mlp multi-layer neural network to obtain the N*256 feature F ₃ , and the .* operation is performed with the feature F ₂ extracted from the 3D point cloud to obtain Feature F ₄ , then F ₄ and two-dimensional image feature F ₁ are passed through the mlp multi-layer neural network to obtain N*256 feature F ₃ for addition operation to obtain N*256 feature F ₅ ; then F ₅ is maximized The pooling operation is performed to obtain a 1*256 feature F ₆ , and then the convolution operation and the sigmoid operation are performed to obtain the final N*256 feature F ₇ ;

When the conditional batch normalization module is larger than one, the ₁ *256 feature F1 extracted from the two-dimensional image is input into the mlp multi _- layer neural network, and the N*256 feature F3 is obtained, which is compared with the one obtained from the previous conditional batch normalization module. The feature F ₇ performs the .* operation to obtain the feature F ₄ , and then the F ₄ and the two-dimensional image feature F ₁ are added through the mlp multi-layer neural network to obtain the N*256 feature F ₃ , and the N*256 feature F is obtained. ₅ ; then perform the maximum pooling operation on F ₅ to obtain a 1*256 feature F ₆ , and then perform a convolution operation and sigmoid to obtain a new N*256 feature F ₇ ;

Finally, input F ₇ into the mlp multi-layer neural network to obtain the prediction result in the form of N*3 point cloud; process the three-dimensional model drawn according to the actual size of the object in the training set, and obtain the true value point cloud. Compare the point cloud with the prediction result and calculate the loss value;

After iteration, the training is finally completed to obtain the trained Onet-2D.

Further, the process of constructing a two-dimensional image dataset M _pre-2D includes the following steps:

Input the obtained two-dimensional image remote sensing data into the instance segmentation network, and extract objects from the remote sensing data according to the instance segmentation results, and obtain a two-dimensional image data set M _pre-2D with only one object in each image. M _pre-2D is divided into a two-dimensional image training data set M _tr-2D and a two-dimensional image test data set M _test-2D ; the training data set M _tr-2D contains two-dimensional images of ground objects and corresponding The three-dimensional model drawn by the actual size of the object target, the test data set M _test-2D only contains the two-dimensional image of the object target.

Further, the deep learning network of the space occupancy probability feature of the three-dimensional point cloud described in S4 is the OccupancyNetwork-3D network, namely Onet-3D, and its training process includes the following steps:

S401, constructing a three-dimensional point cloud data set M _pre-3D , the three-dimensional point cloud data set M _pre-3D includes a three-dimensional point cloud training data set M _tr-3D and a three-dimensional point cloud test data set M _test-3D ;

S402. Training Onet-3D:

Input the 3D point cloud data in the 3D point cloud training data set M _tr-3D into Onet-3D; Onet-3D first uses the pointnet point cloud feature extraction network to extract features from the input 3D point cloud data, and obtain 1*256 feature f ₁ ;

Second, randomly generate a sampling point cloud cube of unit volume, input the x, y, and z coordinates of each point in the point cloud cube into a three-layer mlp multi-layer neural network, and transpose to obtain a feature f of 256*N ₂ ;

Then f ₁ and f ₂ are respectively input into at least one conditional batch normalization module, the conditional batch normalization module is the CBN module; the specific process includes the following steps:

The 1*256 feature f ₁ extracted from the two-dimensional image is input into the mlp multi-layer neural network to obtain the N*256 feature f ₃ , and the .* operation is performed with the feature f ₂ extracted from the three-dimensional point cloud to obtain Feature f ₄ , and then add f ₄ and two-dimensional image feature f ₁ to obtain N*256 feature f ₃ through the mlp multi-layer neural network to obtain N*256 feature f ₅ ; then perform maximum pooling on f ₅ 1*256 feature f ₆ is obtained, and then the convolution operation and sigmoid operation are performed to obtain the final N*256 feature f ₇ ;

When the conditional batch normalization module is larger than one, the ₁ *256 feature f1 extracted from the two-dimensional image is input into the mlp multi-layer neural network, and the N*256 feature _f3 is obtained, which is compared with the one obtained from the previous conditional batch normalization module. The feature f ₇ performs the .* operation to obtain the feature f ₄ , and then the f ₄ and the two-dimensional image feature f ₁ are added to the N*256 feature f ₃ through the mlp multi-layer neural network to obtain the N*256 feature f ₅ ; then perform the maximum pooling operation on f ₅ to obtain a feature f ₆ of 1*256, and then perform a convolution operation and sigmoid to obtain a new N*256 feature f ₇ ;

Finally, input f ₇ into the mlp multi-layer neural network to obtain the prediction result in the form of N*3 point cloud; process the three-dimensional model drawn according to the actual size of the object in the training set to obtain the true value point cloud, The loss value is obtained by comparing the true point cloud with the prediction result;

After iteration, the training is finally completed to obtain the trained Onet-3D.

Further, the process that the Occupancy Network-3D network adopts the pointnet point cloud feature extraction network to perform feature extraction on the input 3D point cloud data includes the following steps:

Pass the N*3 real point cloud data of the input network through the input transform module, and then pass through a two-layer mlp multi-layer neural network to obtain the N*64 feature F'₁;

Input the feature F' ₁ into the feature transform module, and then pass through a three-layer mlp multi-layer neural network to obtain the feature F' ₂ of N*1024;

Perform a maximum pooling operation on the N*1024 feature F' ₂ to obtain a 1*1024 feature F' ₃ , and then pass a two-layer mlp multi-layer neural network to obtain a 1*256 feature f ₁ .

Further, the process of constructing the 3D point cloud dataset M _pre-3D includes the following steps:

Input the obtained 3D point cloud remote sensing data into the point cloud target detection network, and perform target segmentation on the 3D point cloud remote sensing data according to the target detection result, and obtain a 3D point cloud data set M _pre-3D that contains only one object in each file. The point cloud dataset M _pre-3D is divided into a 3D point cloud training dataset M _tr-3D and a 3D point cloud testing dataset M _test-3D . The training data set M _tr-3D contains the 3D point cloud of the object and the corresponding 3D model drawn according to the actual size of the object, and the test data set M _test-3D only contains the 3D point cloud of the object.

Further, the classifier described in S5 adopts the pointnet++ point cloud classification network, and the process of using the pointnet++ point cloud classification network to perform target recognition includes the following steps:

(1) Convert the space occupation probability of the array form obtained by S3 and S4 into a point cloud form, and set the number of points in the point cloud to a fixed value m, if the number of points in the point cloud > m, perform downsampling operation, if The upsampling operation is performed when the number of points in the point cloud < m.

(2) Input the preprocessed point cloud data into the pointnet++ point cloud classification network; classify according to the finally extracted 1*k features, that is, to realize the association between the two-dimensional image data and the three-dimensional point cloud data.

Further, the training process of the classifier includes the following steps:

S501, extracting the two-dimensional image test data set M _test- 2D in the two-dimensional image data set M _pre- 2D, and the three-dimensional point cloud test data set M _test- 3D in the three-dimensional point cloud data set M _pre- 3D;

S502: Input the two-dimensional image test data set M _test-2D into Onet-2D, extract the space occupation probability feature of the two-dimensional image data, and obtain the space occupation probability F _test-2D in the form of an array.

Input the three-dimensional point cloud test data set M _test-3D into Onet-3D, extract the space occupation probability feature of the three-dimensional point cloud data, and obtain the space occupation probability F _test-3D in the form of an array;

S503. Input the point cloud corresponding to the space occupancy probability feature extracted from the two-dimensional image as input data into the pointnet++ network, and use the extracted feature as a class feature; use the space occupancy probability point extracted from the three-dimensional point cloud The cloud is input into the pointnet++ network as the target data, the extracted features are matched with the class features, the accuracy is calculated, and the classifier is trained iteratively;

or,

The point cloud corresponding to the space occupancy probability feature extracted from the 3D point cloud is input into the pointnet++ network as input data, and the extracted features are used as class features; the space occupancy probability point cloud extracted from the 2D image is used as the input data. The target data is input into the pointnet++ network, the extracted features are matched with the class features, the accuracy is calculated, and the classifier is trained iteratively.

Beneficial effects:

The invention firstly trains the space occupancy probability feature extraction network of two-dimensional images and the space occupancy probability feature extraction network of three-dimensional point cloud, and then extracts the same kind of features for the two-dimensional image data and three-dimensional point cloud data in the remote sensing data, namely the space occupancy probability Finally, based on the space occupancy probability feature, the cross-dimensional target recognition between the two-dimensional and three-dimensional data of the same object in the remote sensing data is realized, that is, the association between the two-dimensional image data of the same object and the three-dimensional point cloud data is realized. The invention can well solve the problem that the two-dimensional image data and the three-dimensional point cloud data cannot be effectively correlated at present, and the correlation accuracy rate can reach 80%.

Description of drawings

1 is a schematic flow chart of Embodiment 1;

Fig. 2 is the schematic diagram of two-dimensional image instance segmentation method PANET;

3 is a schematic diagram of a three-dimensional point cloud target detection method 3D-BONET;

Fig. 4 is the network schematic diagram of Occupancy Network-2D, a space occupancy probability feature extraction method based on two-dimensional images;

Fig. 5 is the network schematic diagram of Occupancy Network-3D, a space occupancy probability feature extraction method based on three-dimensional point cloud;

Figure 6 is a network diagram of the point cloud classification network pointnet++.

Detailed ways

Embodiment 1: This embodiment is described with reference to FIG. 1 ,

The cross-dimensional remote sensing data target recognition method based on the space occupancy probability feature described in this embodiment includes the following steps:

Step 1: Preprocess the two-dimensional image remote sensing data.

Firstly, preprocess the obtained two-dimensional image remote sensing data: input the two-dimensional image remote sensing data into the instance segmentation network, and extract the target from the remote sensing data according to the instance segmentation result, and obtain the two-dimensional image data of only one building in each picture. The set M _pre-2D is further divided into a two-dimensional image training data set M _tr-2D and a two-dimensional image testing data set M _test-2D . The training data set M _tr-2D contains the two-dimensional images of the ground objects and the corresponding three-dimensional models drawn according to the actual size of the building, and the test data set M _test-2D only contains the two-dimensional images of the ground objects.

Step 2: Preprocess the 3D point cloud remote sensing data.

Preprocess the 3D point cloud remote sensing data obtained through lidar and other means: input the 3D point cloud into the point cloud target detection network, and perform target segmentation on the 3D point cloud remote sensing data according to the target detection result, and obtain that each file contains only one file. The three-dimensional point cloud data set M _pre-3D of the point cloud data of the building is further divided into a three-dimensional point cloud training data set M _tr-3D and a three-dimensional point cloud test data set M _test-3D . The training data set M _tr-3D contains the 3D point cloud of the object and the corresponding 3D model drawn according to the actual size of the building, and the test data set M _test-3D only contains the 3D point cloud of the object.

Step 3: Train the spatial occupancy probability feature extraction network of the two-dimensional image.

Input the images in the two-dimensional image training dataset M _tr-2D into the deep learning network, and train the deep learning network for extracting the spatial occupancy probability features of the two-dimensional images;

The training data of the two-dimensional image training data set includes the real two-dimensional remote sensing image of the building and the three-dimensional simulation model drawn according to the actual size of the building.

Step 4: Train the spatial occupancy probability feature extraction network of the 3D point cloud.

Input the point cloud files in the 3D point cloud training data set M _tr-3D into the deep learning network, and train the deep learning network for extracting the spatial occupancy probability features of the 3D point cloud;

The training data of the three-dimensional point cloud training data set includes the real three-dimensional point cloud data of the building and the three-dimensional simulation model drawn according to the actual size of the building;

Step 5: Feature extraction of space occupancy probability based on two-dimensional images.

Input the two-dimensional image test data set M _test-2D into the network trained in step 3, extract the space occupation probability feature of the two-dimensional image data, and obtain the space occupation probability F _test-2D in the form of an array.

Step 6: Feature extraction of space occupancy probability based on 3D point cloud.

Input the 3D point cloud test data set M _test-3D into the network trained in step 4, extract the space occupancy probability feature of the 3D point cloud data, and obtain the space occupancy probability F _test-3D in the form of an array.

Step 7: Cross-dimensional target recognition. The spatial occupancy probability features F _test-2D and F _test-3D obtained from remote sensing data of different dimensions obtained in steps 5 and 6 are input into the classifier for target recognition.

In fact, the preprocessing link of remote sensing data includes step 1 and step 2: step 1 is the preprocessing step of 2D image remote sensing data; step 2 is the preprocessing step of 3D point cloud remote sensing data.

In step 1, the PANET method is used to segment the instances of the two-dimensional image. As shown in Figure 2, the structure of the PANET method is mainly divided into a feature pyramid module, a dynamic feature pooling module and a fully connected layer module, which takes two-dimensional image data as input for instance segmentation. The loss function L used by the instance segmentation network includes classification error, detection error and segmentation error:

L=L _cls +L _box +L _mask

Among them, for each ROI, the mask branch defines a K*m*2-dimensional matrix representing K different categories. For each m*m region, there is one for each class. For each pixel, the relative entropy is calculated using the sigmod function to obtain the average relative entropy error Lmask. For each ROI, if it is detected to which category the ROI belongs, only the relative entropy error of which branch is used as the error value for calculation.

In step 2, the 3D-BONET method is used to detect objects on the 3D point cloud. As shown in Figure 3, 3D-BONET provides a new framework for instance segmentation on 3D point cloud data, and takes 3D point cloud as input for object detection. The loss function of 3D-BONET is defined as L:

L=L _sem +L _bb o _x +L _bbs +L _pmask

in

The preliminary network training includes step 3 and step 4: step 3 is the process of training the network for spatial occupancy probability feature extraction based on 2D images; step 4 is the process of training the spatial occupancy probability feature extraction network based on 3D point cloud.

In step 3, the Occupancy Network-2D (Onet-2D) method is adopted, and the training data M _tr-2D obtained in step 1 is used to train a two-dimensional image-based spatial occupancy probability feature extraction network, as shown in Figure 4. ^The space _{occupancy probability} is the _probability of whether each point is a point in the model under ideal conditions. 2D network to get this 3D function. The neural network assigns each position p an occupancy probability between 0 and 1, which is equivalent to a neural network for binary classification, while the present invention focuses on the decision boundary of the object surface. The specific steps of Onet-2D parameter training are:

First, input the 2D image data in the training set into the network, and use the RESNET18 residual network with the super channel attention module ECA to perform feature extraction on the input 2D image data to obtain a 1*256 feature F ₁ . The super channel attention module ECA is a module that avoids dimension reduction and effectively captures cross-channel interactions: the H*W*C features extracted by the RESNET18 residual network are processed channel by channel without reducing the dimension. Global average pooling followed by fast 1D convolutions of size k ₁ to achieve cross-channel interactions for each channel and its k ₁ nearest neighbors, where the kernel size k ₁ represents the coverage of local cross-channel interactions, i.e. how many Nearby neighbors participate in the attention prediction of a channel, and the output is also the feature of H*W*C.

Second, randomly generate a sampling point cloud cube of unit volume (this step first randomly generates a sampling point cloud cube, and then configure the space occupancy probability for each sampling point through the subsequent training steps, and gradually train it as a "3D simulation model" shape), input the x, y, z coordinates (N*3) of each point in the point cloud cube into a three-layer mlp multi-layer neural network (3→64→256), and transpose to get 256*N the feature F ₂ .

Then the above two features are respectively input into at least one conditional batch normalization module (CBN module), preferably 5 CBNs. Conditional batch normalization module (CBN module): The ₁ *256 feature F1 extracted from the 2D image is input into the mlp multi _- layer neural network, and the N*256 feature F3 is obtained, which is compared with the feature extracted from the 3D point cloud. The feature F ₂ performs the .* operation (multiplying the corresponding elements of the two matrices) to obtain the feature F ₄ , and then adds the F ₄ and the two-dimensional image feature F ₁ through the mlp multi-layer neural network to obtain the N*256 feature F ₃ for addition. operation to obtain N*256 feature F ₅ ; then perform maximum pooling operation on F ₅ to obtain 1*256 feature F ₆ , and then perform convolution operation and sigmoid operation to obtain final N*256 feature F ₇ ;

When the conditional batch normalization module is larger than one, the ₁ *256 feature F1 extracted from the two-dimensional image is input into the mlp multi _- layer neural network, and the N*256 feature F3 is obtained, which is compared with the one obtained from the previous conditional batch normalization module. The feature F ₇ performs the .* operation to obtain the feature F ₄ , and then the F ₄ and the two-dimensional image feature F ₁ are added through the mlp multi-layer neural network to obtain the N*256 feature F ₃ , and the N*256 feature F is obtained. ₅ ; then perform the maximum pooling operation on F ₅ to obtain a 1*256 feature F ₆ , and then perform a convolution operation and sigmoid to obtain a new N*256 feature F ₇ ;

Finally, input F ₇ into the mlp multi-layer neural network to get the prediction result in the form of N*3 point cloud. The three-dimensional model drawn according to the actual size of the building in the training set is processed to obtain the true point cloud of the building, and the loss value can be obtained by comparing with the prediction result.

To learn the parameters of the neural network, consider randomly sampling points in the generated point cloud cube of unit volume, for the ith sample, sample K points, and then evaluate the mini-batch loss L _B (θ) for these positions as follows:

where f _θ (p _ij , x _i ) is the space occupancy probability function, with x _i and p _ij as inputs, through the set threshold, it is judged whether the j-th sampling point of the i-th sample is a point in the model, x _i is the observed value of the i-th sample, p _ij is the probability that the j-th sampling point of the i-th sample is a point in the model, o _ij is the true position of the point cloud, and L is the calculated cross-entropy loss.

In step 4, the Occupancy Network-3D (Onet-3D) method is adopted, and the training data M _tr-3D obtained in step 2 is used to train the spatial occupancy probability feature extraction network based on 3D point cloud, as shown in Figure 5. The training steps in step four are the same as step three, the difference is that since the network input is changed from two-dimensional image to three-dimensional point cloud, the input coding part has changed: in step four, the pointnet point cloud feature extraction method is used:

First, pass the N*3 real point cloud data of the input network through the input transform module, and then pass through a two-layer mlp multi-layer neural network (3→64) to obtain the N*64 feature F' ₁ .

Then, the feature F' ₁ is input into the feature transform module, and then through a three-layer mlp multi-layer neural network (64→128→1024), the N*1024 feature F' ₂ is obtained. The structure of the input transform module and featuretransform module is shown in Figure 5. First, pass the input through a T-Net network to obtain a 3*3 or 64*64 matrix, and then perform matrix multiplication with the input to obtain N*3 or N*64 Its functions are to enhance the feature extraction ability. The maximum pooling operation is performed on the N*1024 feature F' ₂ to obtain the 1*1024 feature F' ₃ , and then a two-layer mlp multi-layer neural network (512→256) is used to obtain the 1*256 feature.

Steps 5 and 6 use the Occupancy Network-2D and Occupancy Network-3D networks trained in Steps 3 and 4, respectively, to extract the spatial occupancy probability feature of remote sensing data of different dimensions, and use 2D image remote sensing data and 3D point cloud remote sensing. The data are used as input, respectively, and the space occupancy probability feature of the two-dimensional image in the form of an array and the space occupancy probability feature of the three-dimensional point cloud in the form of an array are output.

The classifier in step 7 is implemented by the pointnet++ point cloud classification network:

(1) First, preprocess the point cloud to be classified. Convert the space occupancy probability of data of different dimensions in the form of arrays obtained in steps 5 and 6 into point cloud form, and set the number of points in the point cloud to a fixed value m, if the number of points in the point cloud > m, perform downsampling operation, if the number of points in the point cloud is less than m, the upsampling operation is performed (the points in the point cloud are simply copied until the target number m is reached, which has no effect on the point cloud information).

(2) Input the above preprocessed point cloud data into the point cloud classification network shown in FIG. 6 . The structure of the point cloud classification network is the same as that of the point cloud encoder in step 4, and the classification is performed according to the finally extracted 1*k features, that is, the association between the two-dimensional image data and the three-dimensional point cloud data is realized.

In this process, the point cloud corresponding to the space occupancy probability feature extracted from the two-dimensional image is input as input data into the point cloud classification network pointnet++ network, and the extracted features are used as class features; The space occupancy probability point cloud is input into the pointnet++ network as the target data, the extracted features are matched with the class features, and the accuracy rate is calculated, and the classifier is trained iteratively. (In this step, the spatial occupancy probability feature extracted from the 2D image and the spatial occupancy probability feature extracted from the 3D point cloud can be interchanged, that is, both can be used as input data and target data);

The pointnet++ point cloud feature extraction network used in the classifier performs feature extraction on the input 3D point cloud data, including the following steps:

Pass the N*3 point cloud data of the input network through the input transform module, and then pass through a two-layer mlp multi-layer neural network to obtain the N*64 feature F"₁;

Input the feature F" ₁ into the feature transform module, and then pass through a three-layer mlp multi-layer neural network to obtain the feature F" ₂ of N*1024;

Perform a maximum pooling operation on the N*1024 feature F" ₂ to obtain a 1*1024 feature F" ₃ , and then pass a three-layer mlp multi-layer neural network to obtain a 1*k feature F"₄; k is a point The number of categories for cloud classification.

The above classifier based on the pointnet++ point cloud classification network performs feature extraction on the space occupancy probability in the form of point clouds obtained by extracting cross-dimensional features from remote sensing data of different dimensions, and then realizes the data of different dimensions of the same object in the remote sensing data through feature similarity ranking. identification. In actual use, the two-dimensional image data is preprocessed in step 1, and then the spatial occupancy probability feature extraction network of the two-dimensional image is input to extract the space occupation probability of the two-dimensional image; at the same time, the three-dimensional point cloud data is preprocessed in step two. , and then input the space occupancy probability of the 3D point cloud. The feature extraction network extracts the space occupancy probability of the 3D point cloud; Classification to realize the association of 2D image data and 3D point cloud data.

The present invention can also have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations are all It should belong to the protection scope of the appended claims of the present invention.

Claims (9)

1. A cross-dimensional remote sensing data target recognition method based on space occupancy probability feature, is characterized in that, comprises the following steps: S1: Preprocess the two-dimensional image remote sensing data: input the two-dimensional image remote sensing data into the instance segmentation network, and perform target extraction on the remote sensing data according to the instance segmentation result; S2: Preprocess the 3D point cloud remote sensing data: input the 3D point cloud into the point cloud target detection network, and perform target segmentation on the 3D point cloud remote sensing data according to the target detection result; S3: Input the image processed by S1 into the deep learning network of the space occupancy probability feature of the two-dimensional image, and extract the space occupancy probability F _test-2D of the two-dimensional image; S4: input the 3D point cloud processed by S2 into a deep learning network of the space occupancy probability feature of the 3D point cloud; extract the space occupancy probability F _test-3D of the 3D point cloud; S5: Input the space occupancy probability features F _test-2D and F _test-3D obtained in S3 and S4 into the classifier for target recognition, so as to realize the association between the two-dimensional image data and the three-dimensional point cloud data; the classifier uses pointnet++ points Cloud classification network, the process of using pointnet++ point cloud classification network for target recognition includes the following steps: (1) Convert the space occupation probability of the array form obtained by S3 and S4 into a point cloud form, and set the number of points in the point cloud to a fixed value m, if the number of points in the point cloud > m, perform downsampling operation, if If the number of points in the point cloud < m, the upsampling operation is performed; (2) Input the preprocessed point cloud data into the pointnet++ point cloud classification network; classify according to the finally extracted 1*k features, that is, to realize the association between the two-dimensional image data and the three-dimensional point cloud data. 2. The cross-dimensional remote sensing data target identification method based on the space occupancy probability feature according to claim 1, wherein the instance segmentation network described in S1 adopts PANET. 3. The cross-dimensional remote sensing data target recognition method based on the space occupancy probability feature according to claim 2, wherein the point cloud target detection network described in S2 adopts 3D-BONET. 4. the cross-dimensional remote sensing data target recognition method based on the space occupancy probability feature according to claim 1, 2 or 3, is characterized in that, the deep learning network of the space occupancy probability feature of the described two-dimensional image of S3 is OccupancyNetwork- 2D network, namely Onet-2D, its training process includes the following steps: S301, constructing a two-dimensional image data set M _pre-2D , the two-dimensional image data set M _pre-2D includes a two-dimensional image training data set M _tr-2D and a two-dimensional image test data set M _test-2D ; S302, training Onet-2D: The two-dimensional image data in the two-dimensional image training dataset M _tr-2D is input into Onet-2D, and Onet-2D first uses the RESNET18 residual network with the super channel attention module ECA to characterize the input two-dimensional image data. Extraction to obtain a feature F ₁ of 1*256; Second, randomly generate a sampling point cloud cube of unit volume, input the x, y, and z coordinates of each point in the point cloud cube into a three-layer mlp multi-layer neural network, and transpose to obtain a feature F of 256*N ₂ ; Then F ₁ and F ₂ are respectively input into at least one conditional batch normalization module, the conditional batch normalization module is the CBN module; the specific process includes the following steps: The 1*256 feature F ₁ extracted from the 2D image is input into the mlp multi-layer neural network to obtain the N*256 feature F ₃ , and the .* operation is performed with the feature F ₂ extracted from the 3D point cloud to obtain Feature F ₄ , then F ₄ and two-dimensional image feature F ₁ are passed through the mlp multi-layer neural network to obtain N*256 feature F ₃ for addition operation to obtain N*256 feature F ₅ ; then F ₅ is maximized The pooling operation is performed to obtain a 1*256 feature F ₆ , and then the convolution operation and the sigmoid operation are performed to obtain the final N*256 feature F ₇ ; When the conditional batch normalization module is larger than one, the ₁ *256 feature F1 extracted from the two-dimensional image is input into the mlp multi _- layer neural network, and the N*256 feature F3 is obtained, which is compared with the one obtained from the previous conditional batch normalization module. The feature F ₇ performs the .* operation to obtain the feature F ₄ , and then the F ₄ and the two-dimensional image feature F ₁ are added through the mlp multi-layer neural network to obtain the N*256 feature F ₃ , and the N*256 feature F is obtained. ₅ ; then perform the maximum pooling operation on F ₅ to obtain a 1*256 feature F ₆ , and then perform a convolution operation and sigmoid to obtain a new N*256 feature F ₇ ; Finally, input F ₇ into the mlp multi-layer neural network to obtain the prediction result in the form of N*3 point cloud; process the three-dimensional model drawn according to the actual size of the object in the training set, and obtain the true value point cloud. Compare the point cloud with the prediction result and calculate the loss value; After iteration, the training is finally completed to obtain the trained Onet-2D. 5. the cross-dimensional remote sensing data target recognition method based on space occupancy probability feature according to claim 4, is characterized in that, the process of constructing two-dimensional image data set M _pre-2D comprises the following steps: Input the obtained two-dimensional image remote sensing data into the instance segmentation network, and extract objects from the remote sensing data according to the instance segmentation results, and obtain a two-dimensional image data set M _pre-2D with only one object in each image. M _pre-2D is divided into a two-dimensional image training data set M _tr-2D and a two-dimensional image test data set M _test-2D ; the training data set M _tr-2D contains two-dimensional images of ground objects and corresponding The three-dimensional model drawn by the actual size of the object target, the test data set M _test-2D only contains the two-dimensional image of the object target. 6. the cross-dimensional remote sensing data target identification method based on space occupancy probability feature according to claim 4, is characterized in that, the deep learning network of the space occupancy probability feature of the described three-dimensional point cloud of S4 is OccupancyNetwork-3D network, namely Onet-3D, its training process includes the following steps: S401 , constructing a three-dimensional point cloud data set M _pre-3D , where the three-dimensional point cloud data set M _pre-3D includes a three-dimensional point cloud training data set M _tr-3D and a three-dimensional point cloud test data set M _test-3D ; S402. Training Onet-3D: Input the 3D point cloud data in the 3D point cloud training data set M _tr-3D into Onet-3D; Onet-3D first uses the pointnet point cloud feature extraction network to extract features from the input 3D point cloud data, and obtain 1*256 features f ₁ ; Second, randomly generate a sampling point cloud cube of unit volume, input the x, y, and z coordinates of each point in the point cloud cube into a three-layer mlp multi-layer neural network, and transpose to obtain a feature f of 256*N ₂ ; Then f ₁ and f ₂ are respectively input into at least one conditional batch normalization module, the conditional batch normalization module is the CBN module; the specific process includes the following steps: The 1*256 feature f ₁ extracted from the two-dimensional image is input into the mlp multi-layer neural network, and the N*256 feature f ₃ is obtained, and the .* operation is performed with the feature f ₂ extracted from the three-dimensional point cloud to obtain Feature f ₄ , and then add f ₄ and two-dimensional image feature f ₁ through the mlp multi-layer neural network to obtain N*256 feature f ₃ to obtain N*256 feature f ₅ ; then perform maximum pooling on f ₅ 1*256 feature f ₆ is obtained, and then convolution operation and sigmoid operation are performed to obtain the final N*256 feature f ₇ ; When the conditional batch normalization module is larger than one, the ₁ *256 feature f1 extracted from the two-dimensional image is input into the mlp multi-layer neural network, and the N*256 feature _f3 is obtained, which is compared with the one obtained from the previous conditional batch normalization module. The feature f ₇ performs the .* operation to obtain the feature f ₄ , and then the f ₄ and the two-dimensional image feature f ₁ are added to the N*256 feature f ₃ through the mlp multi-layer neural network to obtain the N*256 feature f ₅ ; then perform the maximum pooling operation on f ₅ to obtain a feature f ₆ of 1*256, and then perform a convolution operation and sigmoid to obtain a new N*256 feature f ₇ ; Finally, input f ₇ into the mlp multi-layer neural network to obtain the prediction result in the form of N*3 point cloud; process the three-dimensional model drawn according to the actual size of the object in the training set to obtain the true value point cloud, The loss value is obtained by comparing the true point cloud with the prediction result; After iteration, the training is finally completed to obtain the trained Onet-3D. 7. The cross-dimensional remote sensing data target recognition method based on space occupancy probability feature according to claim 6, is characterized in that, Occupancy Network-3D network adopts pointnet point cloud feature extraction network to carry out feature extraction to the input three-dimensional point cloud data. The process includes the following steps: Pass the N*3 real point cloud data of the input network through the input transform module, and then pass through a two-layer mlp multi-layer neural network to obtain the N*64 feature F'₁; Input the feature F' ₁ into the feature transform module, and then pass through a three-layer mlp multi-layer neural network to obtain the feature F' ₂ of N*1024; Perform a maximum pooling operation on the N*1024 feature F' ₂ to obtain a 1*1024 feature F' ₃ , and then pass a two-layer mlp multi-layer neural network to obtain a 1*256 feature f ₁ . 8. the cross-dimensional remote sensing data target identification method based on space occupancy probability feature according to claim 7, is characterized in that, the process of constructing three-dimensional point cloud data set M _pre-3D comprises the following steps: Input the obtained 3D point cloud remote sensing data into the point cloud target detection network, and perform target segmentation on the 3D point cloud remote sensing data according to the target detection result, and obtain a 3D point cloud data set M _pre-3D that contains only one object in each file. The point cloud data set M _pre-3D is divided into a three-dimensional point cloud training data set M _tr-3D and a three-dimensional point cloud test data set M _test-3D ; the training data set M _tr-3D contains the three-dimensional point cloud of the ground object and the corresponding The 3D model drawn according to the actual size of the object, the test data set M _test-3D only contains the 3D point cloud of the object. 9. The cross-dimensional remote sensing data target recognition method based on space occupancy probability feature according to claim 1, is characterized in that, the training process of described classifier comprises the following steps: S501, extracting the two-dimensional image test data set M _test- 2D in the two-dimensional image data set M _pre- 2D, and the three-dimensional point cloud test data set M _test- 3D in the three-dimensional point cloud data set M _pre- 3D; S502, input the two-dimensional image test data set M _test-2D into Onet-2D, extract the space occupation probability feature of two-dimensional image data, obtain the space occupation probability F _test-2D of array form; Input the three-dimensional point cloud test data set M _test-3D into Onet-3D, extract the space occupation probability feature of the three-dimensional point cloud data, and obtain the space occupation probability F _test-3D in the form of an array; S503. Input the point cloud corresponding to the space occupancy probability feature extracted from the two-dimensional image as input data into the pointnet++ network, and use the extracted feature as a class feature; use the space occupancy probability point extracted from the three-dimensional point cloud The cloud is input into the pointnet++ network as the target data, the extracted features are matched with the class features, the accuracy is calculated, and the classifier is trained iteratively; or, The point cloud corresponding to the space occupancy probability feature extracted from the 3D point cloud is input into the pointnet++ network as input data, and the extracted features are used as class features; the space occupancy probability point cloud extracted from the 2D image is used as the input data. The target data is input into the pointnet++ network, the extracted features are matched with the class features, the accuracy is calculated, and the classifier is trained iteratively.

Images