CN106570136B - Remote sensing image semantic retrieval method and device based on pixel level association rule - Google Patents

Abstract

The invention relates to the technical field of remote sensing image retrieval, in particular to a semantic retrieval method and a semantic retrieval device for remote sensing images based on pixel level association rules. The retrieval method comprises the following steps: constructing a transaction set of training samples; extracting association rules of the training samples from the transaction sets of the training samples, and establishing a training model according to the association rules of the training samples; and calculating the association rule of each image in the image library, inputting the association rule of each image into a training model to obtain the semantic vector of each image, and performing image retrieval through the image to be retrieved and the semantic vector of each image in the image library. The method comprises the steps of extracting association rules of the remote sensing images from the transaction set, establishing a training model between the association rules and the image categories, obtaining semantic vectors of each image through the training model, and realizing the retrieval of the remote sensing images through semantic vectors of the images to be retrieved and all images in the image library, so that the accuracy of the remote sensing image retrieval is improved.

Description

A remote sensing image semantic retrieval method and device based on pixel-level association rules

technical field

The invention relates to the technical field of remote sensing image retrieval, in particular to a remote sensing image semantic retrieval method and device based on pixel-level association rules.

Background technique

Remote sensing images have the characteristics of large image format and many and complex image contents. The phenomenon of "same object with different spectrum" and "different object with same spectrum" is very common, which brings great difficulty to the retrieval of remote sensing images. Image retrieval is to search for images containing specified features or similar content in the database. The current mainstream content-based image retrieval (Content-Based Image Retrieval, CBIR) method can integrate image processing, information retrieval, machine learning, computer vision, artificial intelligence, etc. The knowledge in many fields uses the visual features automatically extracted from the image as the description of the image content; at present, a large number of research results have been obtained in content-based image retrieval.

Visual feature extraction plays an important role in image retrieval. It can be divided into two research directions. One is to study the extraction and similarity measurement of low-level visual features such as spectrum, texture, and shape of images, including hyperspectral feature extraction based on spectral curve absorption features. Image retrieval, using color space and color moment to extract color features, using wavelet transform, Contourlet transform, Gabor wavelet, generalized Gaussian model, texture spectrum and other methods to describe image texture features, based on pixel shape index, PHOG (Pyramid Histogram of Oriented Gradients , hierarchical gradient direction histogram) shape and wavelet pyramid shape feature description method of remote sensing image, etc. The application of such low-level visual features is relatively mature, but it cannot describe the semantic information of the image, and the retrieval results it provides often have a considerable gap with the human brain's cognition of remote sensing images, which is not completely satisfactory.

In response to this problem, another research direction is to establish a low-level visual feature-semantic mapping model to improve the accuracy of image retrieval at the semantic level. The main research achievements include semantic retrieval methods based on statistical learning, such as Bayesian network of Bayesian classifier model context, Bayesian network and EM (maximum expectation) parameter estimation, etc.; retrieval methods based on semantic annotation, such as Language index model, concept semantic distribution model, etc.; semantic retrieval methods based on GIS (Geographic Information System, Geographic Information System), such as the method of using the space and attribute information of vector elements in GIS data to guide semantic assignment; ontology-based semantic Retrieval methods, such as methods based on visual object domain ontology, GeoIRIS, etc. This kind of method can reflect the semantic understanding process of the human brain for image retrieval to a certain extent, has a high accuracy rate, and is the development trend of image retrieval in the future. However, current semantic retrieval methods often pay too much attention to the construction process of low-level visual features and semantic mapping models, ignoring factors such as the types of low-level visual features and semantic learning methods used, which ultimately affect the precision of semantic retrieval.

In recent years, the characteristics of human visual perception have been introduced into the field of image retrieval and have received extensive attention. However, this type of method is still in its infancy, and there are still many problems to be solved: such as the physiological process of the human visual system, and the Feature description method, bottom-up perceptual model, salient feature extraction and measurement, top-down visual attention mechanism, etc. In addition, the typical results for remote sensing image data retrieval mainly include the Swiss RSIAII+III project, which studies the description and retrieval of multi-resolution remote sensing image data based on spectral and texture features; the prototype system Blobworld developed by the Berkeley digital library project, which uses aviation Images, USGS orthophotos, topographic maps, and SPOT satellite images are used as data sources to allow users to intuitively improve retrieval results; the (RS) 2I project of Nanyang Technological University in Singapore covers remote sensing image feature extraction and description, Many aspects of multidimensional indexing technology and distributed architecture design; Stanford University's SIMPLIcity uses a robust integrated region matching method (Integrated Region Matching, IRM) to define the similarity between images, in satellite remote sensing images based on association rules Good results have been obtained in the retrieval; iFind of Microsoft Research Asia constructs a semantic network through the annotation information of the image, and combines it with the visual features of the image in the relevant feedback, effectively realizing relevant feedback at two levels. These systems have achieved important results, but both in feature extraction and representative feature selection still need further research.

To sum up, whether it is pixel-based or object-oriented image retrieval methods, most of them focus on the statistical information of low-level features such as color, texture, and shape of the image as a whole or in part or in the object area. The retrieval method directly based on low-level features cannot extract the target of interest, lacks the ability to describe the spatial information of the image, has too high feature dimension, incomplete description, poor accuracy, lack of regularity, feature description and human cognition exist semantics Disadvantages such as gaps. At the same time, remote sensing image retrieval based on high-level semantic information lacks mature theories and methods. The "semantic gap" between low-level features and high-level semantic information hinders the development and application of remote sensing image retrieval.

Contents of the invention

The present invention provides a remote sensing image semantic retrieval method and device based on pixel-level association rules, aiming to solve one of the above-mentioned technical problems in the prior art at least to a certain extent.

In order to solve the above problems, the present invention provides the following technical solutions:

A remote sensing image semantic retrieval method based on pixel-level association rules, comprising the following steps:

Step a: Construct a transaction set of training samples;

Step b: extracting association rules of training samples from the transaction set of training samples, and establishing a training model according to the association rules of training samples;

Step c: Calculate the association rules of each image in the image database, input the association rules of each image into the training model, obtain the semantic vector of each image, compare the image to be retrieved with each image in the image database Image retrieval based on the similarity of semantic vectors of images.

The technical solution adopted by the embodiment of the present invention also includes: the step a further includes: selecting a certain number of remote sensing images from each image category as training samples according to the specified image category, and performing pixel graying on the training samples Degree-level compression.

The technical solution adopted by the embodiment of the present invention also includes: in the step a, the construction of the transaction set of the training sample is specifically: for each image category, respectively constructing the transaction set of the training sample; the construction method includes:

Gray value transaction set construction method: use the gray value of each band of each pixel for transaction set construction;

Edge four-direction transaction set construction method: use the canny operator to extract the edge of the image, and then extract the multiple directions of the edge point pixels, with the neighborhood as the unit, and the arrangement of the gray value of all pixels in each direction in the neighborhood One transaction in the transaction set, each edge point constitutes 4 transactions;

Pixel four-direction transaction set construction method: use the sequence of gray values in the four directions of each pixel value to construct a transaction set.

The technical solution adopted by the embodiment of the present invention also includes: in the step c, the retrieval method of performing image retrieval by comparing the similarity between the image to be retrieved and the semantic vector of each image in the image database is: respectively calculating the Retrieve the image and the association rules of all images in the image library, input the calculation results into the training model, and output the semantic vectors of all images through the training model; respectively calculate the relationship between the image to be retrieved and the semantic vectors of all images in the image library According to the distance between the semantic vectors, the similarity between the image to be retrieved and all the images in the image database is calculated, and the retrieval result is returned according to the similarity.

The technical solution adopted by the embodiment of the present invention also includes: in the step c, the closer the distance between the semantic vectors is, the higher the similarity between the two images is, and the similarity calculation formula is:

In the above formula, v1 and v2 are two semantic vectors respectively, and N is the length of the vector.

Another technical solution adopted by the embodiment of the present invention is: a remote sensing image semantic retrieval device based on pixel-level association rules, including:

Transaction set building block: transaction set used to construct training samples;

Association rule extraction module: used to extract the association rules of the training samples from the transaction set of the training samples;

Model training module: used to establish a training model according to the association rules of the training samples;

Vector calculation module: used to calculate the association rules of each image in the image database, and input the association rules of each image into the training model to obtain the semantic vector of each image;

Image retrieval module: used for image retrieval by comparing the similarity between the image to be retrieved and the semantic vector of each image in the image library.

The technical solution adopted by the embodiment of the present invention also includes a sample selection module and a grayscale compression module: the sample selection module is used to select a certain number of remote sensing images from each image category as training samples according to the specified image category; The grayscale compression module is used for performing pixel grayscale compression on the training samples.

The technical solution adopted by the embodiment of the present invention also includes: the transaction set construction module constructs the transaction set of the training sample specifically: for each image category, respectively constructs the transaction set of the training sample; the construction method includes:

Gray value transaction set construction method: use the gray value of each band of each pixel for transaction set construction;

Edge four-direction transaction set construction method: use the canny operator to extract the edge of the image, and then extract the multiple directions of the edge point pixels, with the neighborhood as the unit, and the arrangement of the gray value of all pixels in each direction in the neighborhood One transaction in the transaction set, each edge point constitutes 4 transactions;

Pixel four-direction transaction set construction method: use the sequence of gray values in the four directions of each pixel value to construct a transaction set.

The technical solution adopted by the embodiment of the present invention also includes: the image retrieval module performs image retrieval by comparing the similarity between the image to be retrieved and the semantic vector of each image in the image library: the retrieval method is: respectively calculated by the association rule extraction module The image to be retrieved and the association rules of all images in the image database are input into the training model through the vector calculation module, and the semantic vectors of all images are output through the training model; the image retrieval module calculates the image to be retrieved and the image respectively The distance between the semantic vectors of all the images in the library, calculate the similarity between the image to be retrieved and all the images in the image database according to the distance between the semantic vectors, and return the retrieval results according to the similarity.

The technical solution adopted in the embodiment of the present invention also includes: the closer the distance between the semantic vectors is, the higher the similarity between the two images is, and the similarity calculation formula of the image retrieval module is:

In the above formula, v1 and v2 are two semantic vectors respectively, and N is the length of the vector.

Compared with the prior art, the beneficial effect produced by the embodiment of the present invention lies in that the remote sensing image semantic retrieval method and device based on pixel-level association rules in the embodiment of the present invention construct a transaction set, extract the association rules of remote sensing images through the transaction set, and establish The training model between association rules and image categories obtains the semantic vector of each image through the training model, and then calculates the similarity between the image to be retrieved and all the images in the image database through the distance between the semantic vectors of the image, so as to realize remote sensing image retrieval; the invention provides a feasible new way to realize remote sensing image retrieval from low-level visual features to high-level semantic information, and improves the accuracy of remote sensing image retrieval.

Description of drawings

Fig. 1 is the flowchart of the remote sensing image semantic retrieval method based on pixel-level association rules according to an embodiment of the present invention;

Fig. 2 is a schematic diagram of pixel gray values in four directions of an edge point pixel;

Figure 3(a) is the original remote sensing image, Figure 3(b) is the compressed remote sensing image, and Figure 3(c) is the edge detection image;

4 is a schematic structural diagram of a remote sensing image semantic retrieval device based on pixel-level association rules according to an embodiment of the present invention;

Figure 5 shows 8 images of houses to be retrieved;

Figure 6 shows the first 24 images returned by multi-band association rule retrieval;

Figure 7 is the first 24 returned images of the histogram matching retrieval method;

Fig. 8 is a schematic diagram of the average precision rate of house retrieval results obtained by adopting various methods;

Figure 9 shows 8 images of squares to be retrieved;

Figure 10 is the first 24 images returned by multi-band association rule retrieval;

Figure 11 is the first 24 returned images of the color moment retrieval method;

Figure 12 is a schematic diagram of the average precision rate of square retrieval results obtained by various methods.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Please refer to FIG. 1 , which is a flowchart of a remote sensing image semantic retrieval method based on pixel-level association rules according to an embodiment of the present invention. The remote sensing image semantic retrieval method based on pixel-level association rules in the embodiment of the present invention includes the following steps:

Step 100: According to the specified image category, select a certain number of remote sensing images from each image category as training samples;

In step 100, the specified image category includes but not limited to grassland, woodland, lake, road and so on.

Step 200: Perform pixel grayscale compression on the training samples;

In step 200, the pixel-based remote sensing image has 256 gray levels. If the 256 gray levels are directly used to mine the association rules of the remote sensing image, the amount of calculation is quite large. , the support degree of frequent itemsets will be very small, which is not conducive to extracting association rules with sufficient support and confidence. Compress at least a few gray levels to reduce the calculation of association rule mining.

In the embodiment of the present invention, the method for compressing the gray level of training sample pixels is as follows: performing image compression with the neighborhood mean and variance, and calculating the mean value μ of the neighborhood for each pixel in a 3*3 neighborhood on the remote sensing image and standard deviation σ, and then use the following formula to calculate the gray level of the central pixel in the neighborhood after compression:

In formula (1), g is the original gray level of the central pixel, g' is the compressed gray level of the central pixel, and c is the proportional coefficient, and the value range is between [0.1, 0.5]. Through this method, the original remote sensing images can be compressed to three gray levels of 0, 1, and 2.

In another embodiment of the present invention, according to the attributes of the object, each attribute is quantized to the range of [1, G] by means of uniform segmentation, specifically: using the method of average compression, the 256 gray levels Evenly distributed to several gray levels,

Among them, G is the maximum gray level, G=8, ceil() is the function of rounding up, and g+1 is to compress the gray level of the image to 1~8.

According to the properties of the object, the method of uniform segmentation is used to quantize each attribute to the range of [1, G]. Specifically, the method of linear segmentation is used for compression. First, the maximum gray level gMax and the minimum gray level of the image are calculated. degree level gMin, and then use the following formula to calculate the compressed gray level:

Among them, G is the maximum gray level, G=8.

The more gray levels after compression, the greater the amount of calculation for association rule mining, but the closer the relationship between the pixels reflected is to the real; on the contrary, the less gray levels after compression, the greater the relationship between pixels after compression. The smaller the difference, the more unfavorable it is to dig out meaningful association rules, so it is very important to choose an appropriate gray level. The maximum number of gray levels in the embodiment of the present invention is selected as 8, and the compression method adopted is average compression, and the compression formula is as follows:

In formula (4), G is the maximum number of gray scales, G=8, ceil() is an upward rounding function, and g+1 is to compress the pixel gray scales of remote sensing images to 1-8.

Step 300: For each image category, construct a transaction set of training samples;

In step 300, the method of constructing the transaction set is to use the attributes of each pixel to construct the transaction set. The methods of constructing the transaction set include the construction method of the gray value transaction set, the construction method of the edge four-direction transaction set and the pixel four-direction transaction Set construction method; wherein, the construction method of the gray value transaction set is: directly use the gray value of each band of each pixel to construct the transaction set. For example, if it is an RGB three-band image, there are three gray values of R, G, and B on each pixel, and the transaction set is constructed with these gray values.

ID item 1 9, 7, 3 2 8, 6, 5 3 7,8,9 4 9,5,8

The construction method of the edge four-direction transaction set is as follows: the neighborhood is used as the unit, and the arrangement of the gray value of all pixels in the neighborhood is used as a certain transaction in the transaction set, for example, the gray value of 9 pixels in the 3*3 neighborhood is enough constitute a transaction. Then, for an image with a size of 100*100 pixels, a transaction set consisting of 98*98=9604 transactions can be formed, and each transaction contains 9 items. The larger the remote sensing image, the more transactions are formed, and the larger the transaction set is; the more items a transaction contains, the larger the frequent itemset that needs to be calculated, and the greater the amount of calculation, so it is necessary to calculate each transaction The number of items included is limited. Considering that the edge of the remote sensing image contains a lot of useful information, and the edge has directionality, the present invention first uses the canny operator to extract the edge of the remote sensing image, and then extracts the four directions of the edge point pixels, and uses the 3 pixel grayscale values as elements of an item. For an edge point pixel, its pixel gray value in four directions is specifically shown in FIG. 2 , which is a schematic diagram of the pixel gray value in four directions of an edge point pixel.

Since the edge point pixels on the remote sensing image account for a small proportion of the entire image, and each transaction only contains 3 items, the amount of calculation can be significantly reduced. For details, please refer to Figure 3(a), Figure 3(b) and Figure 3(c). Figure 3(a) is the original remote sensing image with a size of 128*128 pixels; Figure 3(b) is the compressed remote sensing image. For the convenience of display, each gray level is multiplied by 256/G. It can be seen from the compressed image that although the number of gray levels is reduced to 8 after compression, the content of the remote sensing image has not changed much; Figure 3 ( c) is an edge detection image, and the number of detected edge points is 1964, so the size of the formed transaction set is 1964*4=7856, and each transaction contains 3 items. If there are more features in the image, more edge points will be detected, and the final transaction set will be larger. Table 4-16 shows some of the transactions in this transaction set.

Table 4-16 Some items in the transaction set

The method for constructing the pixel four-direction transaction set specifically includes: directly using the gray value sequence of each pixel in the four directions to construct the transaction set.

Step 400: using an association rule mining algorithm to extract the association rules of each image category from the transaction set of the training samples;

In step 400, since the transaction set is constructed using the attributes of each pixel, the mined association rules are the association rules among the attributes of each pixel. The methods for extracting association rules specifically include:

Firstly, frequent itemsets need to be calculated; calculating frequent itemsets is a very critical step in association rule mining, and its calculation directly affects the calculation of the entire association rule mining. For an item consisting of 3 elements, set it to [a b c], if the support is not considered, the extracted association rules are 12 as shown in the following table 4-17:

Table 4-17 Association rules generated by 3 items

a=>b a=>c b=>c b=>a c=>a c=>b ab=>c ac=>b bc=>a a=>bc b=>ac c=>ab

Since the first six association rules only involve the relationship between two elements, they are not enough to express the content of remote sensing images, and will increase the amount of calculations for association rule mining and subsequent similarity calculations. Therefore, the embodiment of the present invention uses Apriori or FP -Growth and other frequent itemset algorithms for mining association rules extract 6 association rules that meet the specified confidence and support from the constructed transaction set.

Step 500: use a machine learning algorithm to train the association rules and the image categories to which they belong, and establish a training model between the association rules and the image categories;

In step 500, the embodiment of the present invention uses a support vector machine learning algorithm for model training. In other embodiments of the present invention, the machine learning algorithm may also include various clustering algorithms, expectation maximum (EM) algorithms, and the like.

Step 600: Input the association rules of all images in the image library into the training model to obtain the semantic vector of each image;

In step 600, firstly, according to step 200 to step 500, the image to be retrieved and all the images in the image database are respectively calculated for association rules, and the association rules of each image are excavated, and then the association rules are input into the training model established in step 500 , and output the membership degree of each image belonging to each category, and use the vector of membership value as the semantic vector description of the image content. For example, the semantic vector of an image is (0.8,0.1,0.05,0.05), corresponding to four categories (grassland, woodland, lake, road).

Step 700: Calculate the distances between the semantic vectors of the image to be retrieved and all the images in the image database respectively, calculate the similarity between the image to be retrieved and all the images in the image database according to the distance between the semantic vectors, and return a certain number of similarities Higher images as search results;

In step 700, after obtaining the semantic vectors of all images, the similarity between them can be calculated by calculating the distance between the semantic vectors of the two images, the closer the distance between the semantic vectors of the two images, The higher the similarity between the two images is. The embodiment of the present invention uses KL divergence (Kullback–Leibler divergence) first-order approximate distance to calculate image similarity, and the calculation formula is as follows:

In formula (5), v1 and v2 are two semantic vectors respectively, and N is the length of the vector; in other embodiments of the present invention, various distance functions such as urban street distance and Euclidean distance can also be used to calculate image similarity .

Sort all images in the image library in ascending order of distance, and output a certain number of returned images as retrieval results.

Please refer to FIG. 4 , which is a schematic structural diagram of an apparatus for semantic retrieval of remote sensing images based on pixel-level association rules according to an embodiment of the present invention. The remote sensing image semantic retrieval device based on pixel-level association rules in the embodiment of the present invention includes a sample selection module, a grayscale compression module, a transaction set construction module, an association rule extraction module, a model training module, a vector calculation module and an image retrieval module.

The sample selection module is used to select a certain number of remote sensing images from each image category as training samples according to the specified image category;

The gray level compression module is used to compress the pixel gray level of the training samples; among them, the pixel-based remote sensing image has 256 gray levels, if the 256 gray levels are directly used to mine the association rules of the remote sensing image, then The amount of calculation is quite large. At the same time, due to too many gray levels, the support of frequent itemsets will be very small, which is not conducive to extracting association rules with sufficient support and confidence. Therefore, before association rule mining, it is necessary to First, pixel gray level compression is performed on the remote sensing image, and the remote sensing image is compressed to a few gray levels, so as to reduce the calculation amount of association rule mining.

In the embodiment of the present invention, the method for compressing the gray level of training sample pixels is as follows: performing image compression with the neighborhood mean and variance, and calculating the mean value μ of the neighborhood for each pixel in a 3*3 neighborhood on the remote sensing image and standard deviation σ, and then use the following formula to calculate the gray level of the central pixel in the neighborhood after compression:

In formula (1), g is the original gray level of the central pixel, g' is the compressed gray level of the central pixel, and c is the proportional coefficient, and the value range is between [0.1, 0.5]. Through this method, the original remote sensing images can be compressed to three gray levels of 0, 1, and 2.

In another embodiment of the present invention, according to the attributes of the object, each attribute is quantized to the range of [1, G] by means of uniform segmentation, specifically: using the method of average compression, the 256 gray levels Evenly distributed to several gray levels,

Among them, G is the maximum gray level, G=8, ceil() is the function of rounding up, and g+1 is to compress the gray level of the image to 1~8.

According to the properties of the object, the method of uniform segmentation is used to quantize each attribute to the range of [1, G]. Specifically, the method of linear segmentation is used for compression. First, the maximum gray level gMax and the minimum gray level of the image are calculated. degree level gMin, and then use the following formula to calculate the compressed gray level:

Among them, G is the maximum gray level, G=8.

The more gray levels after compression, the greater the amount of calculation for association rule mining, but the closer the relationship between the pixels reflected is to the real; on the contrary, the less gray levels after compression, the greater the relationship between pixels after compression. The smaller the difference, the more unfavorable it is to dig out meaningful association rules, so it is very important to choose an appropriate gray level. The maximum number of gray levels in the embodiment of the present invention is selected as 8, and the compression method adopted is average compression, and the compression formula is as follows:

In formula (4), G is the maximum number of gray scales, G=8, ceil() is an upward rounding function, and g+1 is to compress the pixel gray scales of remote sensing images to 1-8.

The transaction set building module is used to construct the transaction set of training samples for each image category; the method of constructing the transaction set is to use the attributes of each pixel to construct the transaction set, and the method of constructing the transaction set includes the gray value The transaction set construction method, the edge four-direction transaction set construction method, and the pixel four-direction transaction set construction method; wherein, the gray value transaction set construction method is: directly use the gray value of each band of each pixel to construct the transaction set. For example, if it is an RGB three-band image, there are three gray values of R, G, and B on each pixel, and the transaction set is constructed with these gray values.

ID item 1 9, 7, 3 2 8, 6, 5 3 7,8,9 4 9,5,8

The construction method of the edge four-direction transaction set is as follows: the neighborhood is used as the unit, and the arrangement of the gray value of all pixels in the neighborhood is used as a certain transaction in the transaction set, for example, the gray value of 9 pixels in the 3*3 neighborhood is enough constitute a transaction. Then, for an image with a size of 100*100 pixels, a transaction set consisting of 98*98=9604 transactions can be formed, and each transaction contains 9 items. The larger the remote sensing image, the more transactions are formed, and the larger the transaction set is; the more items a transaction contains, the larger the frequent itemset that needs to be calculated, and the greater the amount of calculation, so it is necessary to calculate each transaction The number of items included is limited. Considering that the edge of the remote sensing image contains a lot of useful information, and the edge has directionality, the present invention first uses the canny operator to extract the edge of the remote sensing image, and then extracts the four directions of the edge point pixels, and uses the 3 pixel grayscale values as elements of an item. For an edge point pixel, its pixel gray value in four directions is specifically shown in FIG. 2 , which is a schematic diagram of the pixel gray value in four directions of an edge point pixel.

Since the edge point pixels on the remote sensing image account for a small proportion of the entire image, and each transaction only contains 3 items, the amount of calculation can be significantly reduced. For details, please refer to Figure 3(a), Figure 3(b) and Figure 3(c). Figure 3(a) is the original remote sensing image with a size of 128*128 pixels; Figure 3(b) is the compressed remote sensing image. For the convenience of display, each gray level is multiplied by 256/G. It can be seen from the compressed image that although the number of gray levels is reduced to 8 after compression, the content of the remote sensing image has not changed much; Figure 3 ( c) is an edge detection image, and the number of detected edge points is 1964, so the size of the formed transaction set is 1964*4=7856, and each transaction contains 3 items. If there are more features in the image, more edge points will be detected, and the final transaction set will be larger. Table 4-16 shows some of the transactions in this transaction set.

Table 4-16 Some items in the transaction set

The method for constructing the pixel four-direction transaction set specifically includes: directly constructing the transaction set by directly using the sequence of gray values in the four directions of each pixel value.

The association rule extraction module is used to use the association rule mining algorithm to extract the association rules of each image category from the transaction set of the training samples; where, since the transaction set is constructed using the attributes of each pixel itself, the mined association rules are Association rules between each pixel attribute.

The method of extracting association rules specifically includes: firstly, frequent item sets need to be calculated; calculating frequent item sets is a very critical step in association rule mining, and its calculation amount directly affects the calculation amount of the entire association rule mining. For an item consisting of 3 elements, set it to [a b c], if the support is not considered, the extracted association rules are 12 as shown in the following table 4-17:

Table 4-17 Association rules generated by 3 items

a=>b a=>c b=>c b=>a c=>a c=>b ab=>c ac=>b bc=>a a=>bc b=>ac c=>ab

Since the first six association rules only involve the relationship between two elements, they are not enough to express the content of remote sensing images, and will increase the amount of calculations for association rule mining and subsequent similarity calculations. Therefore, the embodiment of the present invention uses Apriori or FP -Growth and other frequent itemset algorithms for mining association rules extract 6 association rules that meet the specified confidence and support from the constructed transaction set.

The model training module is used to use the machine learning algorithm to train the association rules and the image categories to which they belong, and to establish a training model between the association rules and the image categories; wherein, the embodiment of the present invention uses a support vector machine learning algorithm for model training. In other embodiments of the present invention, the machine learning algorithm may also include various clustering algorithms, maximum expectation (EM) algorithms, and the like.

The vector calculation module is used to input the association rules of all images into the training model to obtain the semantic vector of each image; firstly, the association rule calculation is performed on the image to be retrieved and all the images in the image library, and the association rules of each image are mined. Then input the association rules into the training model established by the model training module, and output the degree of membership of each image belonging to each category, and use the vector formed by the value of the degree of membership as the semantic vector description of the image content. For example, the semantic vector of an image is (0.8,0.1,0.05,0.05), corresponding to four categories (grassland, woodland, lake, road).

The image retrieval module is used to calculate the distance between the image to be retrieved and the semantic vectors of all the images in the image library, calculate the similarity between the image to be retrieved and all the images in the image library according to the distance between the semantic vectors, and return a certain number of The images with high similarity are used as the retrieval results; among them, after obtaining the semantic vectors of all images, the similarity between them can be calculated by calculating the distance between the semantic vectors of the two images, and the semantic vectors of the two images The closer the distance is, the higher the similarity between the two images. The embodiment of the present invention uses KL divergence (Kullback–Leibler divergence) first-order approximate distance to calculate image similarity, and the calculation formula is as follows:

In formula (5), v1 and v2 are two semantic vectors respectively, and N is the length of the vector; in other embodiments of the present invention, various distance functions such as urban street distance and Euclidean distance can also be used to calculate image similarity .

Sort all images in the image library in ascending order of distance, and output a certain number of returned images as retrieval results.

In order to verify the effectiveness of the present invention, retrieval experiments are carried out on remote sensing images of different sensors through the following examples, and the remote sensing image semantic retrieval method based on pixel-level association rules of the present invention is matched with the existing histogram matching, color moments, Gabor wavelet, DT-CWT (Dual-Tree Complex Wavelet Transform) and other retrieval methods were compared.

WorldView-2 Image Retrieval Experiment

The WorldView-2 image is similar to the QuickBird image. The remote sensing image fusion method is used to fuse the true color image composed of panchromatic and red, green and blue bands. The non-overlapping block method is used to build the image library. The image library contains 3250 block images , and there are more types of features on the image. In order to facilitate statistical retrieval accuracy, the embodiment of the present invention only selects two types of easily distinguishable features, houses and squares, and randomly selects 8 block images for each type of feature, and uses these 8 images as the images to be retrieved. Since the specific number of images of each type in the image database is not known, indicators such as recall rate and missed detection rate cannot be used, and the average precision rate of the first N images can reflect the retrieval performance of the retrieval algorithm, while taking into account the user experience , so the embodiment of the present invention uses the average precision rate of the first 64 images to measure the performance of each retrieval algorithm. When calculating, the correct images in the first 8, first 16, first 24, first 32, first 40, first 48, first 56, and first 64 returned images were counted respectively, and the average precision rate of the 8 images was taken as the final precision Rate. The following are the two types of features and the corresponding search results:

(1) Housing

As shown in Figure 5, there are 8 house images to be retrieved. The houses in the image library include relatively large single-family houses, which may span two block images, and also include dense small houses. They are not distinguished during retrieval and are collectively classified as houses. Houses generally have darker roofs and may have surrounding green trees and deep black shadows. Due to space limitations, the first 24 returned images of the multi-band association rule retrieval method and the histogram matching method in the embodiment of the present invention are shown in Figure 6 and Figure 7 respectively. Figure 6 is the first 24 returned images of the multi-band association rule retrieval, and 7 is the first 24 images returned by the histogram matching retrieval method.

Please refer to Figure 8, which is a schematic diagram of the average precision rate of house retrieval results obtained by various methods. It can be seen from FIG. 8 that the average precision rate of the multi-band association rule retrieval method according to the embodiment of the present invention is relatively stable and higher than other methods. Since the color and texture information of the house and its surrounding features are relatively obvious, the average precision of other methods is also relatively stable. The average precision rate of Gabor wavelet is the lowest, because the housing structure is relatively diverse, and the information in each scale and direction is inconsistent.

(2) Square

As shown in Figure 9, there are 8 square images to be retrieved. The brightness of the square is relatively high, the distribution is relatively regular, and the ground objects contained in it are relatively simple, and may be surrounded by trees and lawns. The first 24 returned images of the multi-band association rule retrieval method and the color moment retrieval method of the embodiment of the present invention are shown in Figure 10 and Figure 11 respectively, Figure 10 is the first 24 returned images of the multi-band association rule retrieval, and Figure 11 is the color The first 24 images returned by the moment retrieval method.

Please refer to Figure 12, which is a schematic diagram of the average precision rate of square retrieval results obtained by various methods. Because the gray value of the square is relatively uniform and the content of ground objects is uncertain, the color moment feature and texture feature are not obvious, resulting in a higher average precision of histogram matching than Gabor wavelet, DT-CWT and color moment, but both The multi-band association rule method of the embodiment of the present invention is below. The average precision rate of the multi-band association rule method in the embodiment of the present invention is relatively close, because for the square, the gray value is relatively uniform, and relatively good retrieval results can be obtained.

The remote sensing image semantic retrieval method and device based on pixel-level association rules in the embodiment of the present invention constructs a transaction set, extracts the association rules of remote sensing images through the transaction set, establishes a training model between association rules and image categories, and obtains Retrieve the semantic vector of the image, and then calculate the similarity between the image to be retrieved and all the images in the image database through the distance between the semantic vectors of the two images, so as to realize the retrieval of remote sensing images; the present invention provides a feasible low-level visual A new way to realize remote sensing image retrieval from features to high-level semantic information, which improves the accuracy of remote sensing image retrieval.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (8)

1. a kind of remote sensing images semantic retrieving method based on Pixel-level correlation rule, which comprises the following steps:

Step a: the transaction set of training sample is constructed；

Step b: the correlation rule of training sample is extracted from the transaction set of the training sample, and according to the training sample Correlation rule establishes training pattern；

Step c: calculating the correlation rule of each width image in Image Database, and the correlation rule of each width image is inputted training Model obtains the semantic vector of each width image, by comparing in image to be retrieved and Image Database each width image it is semantic to The similitude of amount carries out video search；

In the step a, the transaction set of the building training sample specifically: be directed to each image classification, respectively building instruction Practice the transaction set of sample；The construction method includes:

Gray value transaction set construction method: transaction set building is carried out using the gray value of each wave band of each pixel；

Four direction transaction set construction method of edge: using the edge of canny operator extraction image, marginal point pixel is then extracted Multiple directions, as unit of neighborhood, with being arranged to make up in transaction set for all pixels gray value in the neighborhood in each direction An affairs, then each marginal point constitutes 4 affairs；

Four direction transaction set construction method of pixel: transaction set is constructed using the gray value sequence on each pixel value four direction.

2. the remote sensing images semantic retrieving method according to claim 1 based on Pixel-level correlation rule, which is characterized in that The step a further include: according to specified image classification, a certain number of remote sensing image conducts are selected from each image classification Training sample, and pixel grayscale compression is carried out to the training sample.

3. the remote sensing images semantic retrieving method according to claim 2 based on Pixel-level correlation rule, which is characterized in that In the step c, the similitude by comparing the semantic vector of each width image in image to be retrieved and Image Database into The retrieval mode of row video search are as follows: the correlation rule for calculating separately all images in image and Image Database to be retrieved will be counted It calculates result to input in the training pattern, the semantic vector of all images is exported by the training pattern；It calculates separately to be checked The distance between the semantic vector of all images in rope image and Image Database calculates to be retrieved according to the distance between semantic vector The similarity of all images in image and Image Database, and search result is returned to according to similarity.

4. the remote sensing images semantic retrieving method according to claim 3 based on Pixel-level correlation rule, which is characterized in that In the step c, the distance between described semantic vector is closer, and the similarity of two width images is higher, the similarity meter Calculate formula are as follows:

In above-mentioned formula, v1 and v2 are respectively two semantic vectors, and N is the length of vector.

5. a kind of remote sensing images semantic retrieval device based on Pixel-level correlation rule characterized by comprising

Transaction set constructs module: for constructing the transaction set of training sample；

Correlation rule extraction module: for extracting the correlation rule of training sample from the transaction set of the training sample；

Model training module: for establishing training pattern according to the correlation rule of the training sample；

Vector calculation module: for calculating the correlation rule of each width image in Image Database, by the association of each width image Rule input training pattern, obtains the semantic vector of each width image；

Video search module: for the similitude by comparing the semantic vector of each width image in image to be retrieved and Image Database Carry out video search；

Transaction set constructs the transaction set of module building training sample specifically: is directed to each image classification, constructs training sample respectively This transaction set；Construction method includes:

Gray value transaction set construction method: transaction set building is carried out using the gray value of each wave band of each pixel；

Four direction transaction set construction method of edge: using the edge of canny operator extraction image, marginal point pixel is then extracted Multiple directions, as unit of neighborhood, be arranged to make up in transaction set a affairs for all pixels gray value in the neighborhood, then Each marginal point constitutes 4 affairs；

Four direction transaction set construction method of pixel: transaction set is constructed using the gray value sequence on each pixel value four direction.

6. the remote sensing images semantic retrieval device according to claim 5 based on Pixel-level correlation rule, which is characterized in that Further include sample selection module and gray-scale compression module: the sample selection module is used for the specified image classification of basis, from A certain number of remote sensing images are selected in each image classification as training sample；The gray-scale compression module is used for described Training sample carries out pixel grayscale compression.

7. the remote sensing images semantic retrieval device according to claim 6 based on Pixel-level correlation rule, which is characterized in that The video search module is carried out by comparing the similitude of the semantic vector of each width image in image to be retrieved and Image Database The retrieval mode of video search are as follows: all shadows in image and Image Database to be retrieved are calculated separately by correlation rule extraction module Calculated result is inputted in training pattern by vector calculation module, exports institute by the training pattern by the correlation rule of picture There is the semantic vector of image；The video search module calculate separately all images in image to be retrieved and Image Database it is semantic to The distance between amount calculates the similarity of all images in image and Image Database to be retrieved according to the distance between semantic vector, And search result is returned to according to similarity.

8. the remote sensing images semantic retrieval device according to claim 7 based on Pixel-level correlation rule, which is characterized in that The distance between the semantic vector is closer, and the similarity of two width images is higher, and the video search module calculates similarity Calculation formula are as follows:

In above-mentioned formula, v1 and v2 are respectively two semantic vectors, and N is the length of vector.